Image scaling

In computer graphics and digital imaging, scaling refers to the resizing of a digital image. In video technology, the magnification of digital material is known as upscaling or resolution enhancement.

When scaling a vector graphic image, the graphic primitives which make up the image can be scaled using geometric transformations, without any loss of image quality. When scaling a raster graphics image, a new image with a higher or lower number of pixels must be generated. In the case of decreasing the pixel number (scaling down) this usually results in a visible quality loss. From the standpoint of digital signal processing, the scaling of raster graphics is a two-dimensional example of sample rate conversion, the conversion of a discrete signal from a sampling rate (in this case the local sampling rate) to another.

Mathematical

Image scaling can be interpreted as a form of image resampling or image reconstruction under consideration of the Nyquist sampling theorem. According to the theorem the down sampling to a smaller image from a higher-resolution original should be only carried out only after applying a suitable 2D anti-aliasing filter to prevent aliasing artifacts. The image is reduced to the information which can be carried by the smaller image.

In the case of up sampling, a reconstruction filter takes the place of the anti-aliasing filter.

Original image	Original image in spatial fourier domain	2D low pass filtered	filtered image in spatial fourier domain	4x downsampled	4x fourier upsampling (correct reconstruction)	4x fourier upsampling (with aliasing)

A more sophisticated approach to up scaling treats the problem as an inverse problem, solving the question of generating a plausible image which, when scaled down, would look like the input image. A wide variety of techniques can be used for this, including optimization techniques with regularization terms, and the use of machine learning from examples.

Algorithms

An image size can be changed in several ways.

Nearest-neighbor interpolation

One of the simpler ways of increasing the size is nearest-neighbor interpolation, replacing every pixel with a number of pixels of the same color: The resulting image is larger than the original, and preserves all the original detail, but has (possibly undesirable) jaggedness. Diagonal lines, for example, will show the "stairway" shape characteristic of nearest-neighbor interpolation.

Other scaling methods below are better at preserving smooth contours in the image:

Bilinear and bicubic algorithms

One common method is bilinear interpolation. This works by interpolating pixel color values, introducing a continuous transition into the output even where the original material has discrete transitions. Although this is desirable for continuous-tone images, this algorithm reduces contrast (sharp edges) in a way that may be undesirable for line art. Bicubic interpolation yields substantially better results, with only a small increase in computational complexity.

Sinc and Lanczos resampling

Sinc resampling in theory provides the best possible reconstruction for a perfectly bandlimited signal. In practice, the assumptions behind sinc resampling are not completely met by real-world digital images. Lanczos resampling, an approximation to the sinc method, yields better results. Bicubic interpolation can be regarded as a computationally efficient approximation to Lanczos resampling.

Box sampling

One weakness of bilinear, bicubic and related algorithms is that they sample a specific number of pixels. When down scaling below a certain threshold, such as more than twice for all bi-sampling algorithms, the algorithms will sample non-adjacent pixels, which results in both losing data, but also cause rough results.

The trivial solution to this issue is box sampling, which is to consider the target pixel a box on the original image, and sample all pixels inside the box. This ensures that all input pixels contribute to the output. The major weakness of this algorithm is that it is hard to optimize.

Mipmap

Another solution to the downscale problem of bi-sampling scaling are mipmaps. A mipmap is a prescaled set of downscale copies. When downscaling the nearest larger mipmap is used as the origin, to ensure no scaling below the useful threshold of bilinear scaling is used. This is algorithm is fast, and easy to optimize. It is the standard in many frameworks such as OpenGL. The cost is using more image memory, exactly one third more in the standard implementation.

Fourier transform methods

Simple Fourier transform-based interpolation based on padding of the frequency domain with zero components (a smooth window based approach would reduce the ringing). Beside the good conservation (even recovering) of details, notable is the ringing and the circular bleeding of content from the left border to right border (and way around).

Edge-directed interpolation

Edge-directed interpolation algorithms aim to preserve edges in the image after scaling, unlike other algorithms which can produce staircase artifacts around diagonal lines or curves.

Examples of algorithms for this task include New Edge-Directed Interpolation (NEDI),^[1][2] Edge-Guided Image Interpolation (EGGI),^[3] Iterative Curvature-Based Interpolation (ICBI),^[4] and Directional Cubic Convolution Interpolation (DCCI).^[5] An article from 2013 compared the four algorithms above, and found that DCCI had the best scores in PSNR and SSIM on a series of test images.^[6]

hqx

For magnifying computer graphics with low resolution and/or few colors (usually from 2 to 256 colors), better results will be achieved by hqx or other pixel art scaling algorithms. These produce sharp edges and maintain high level of detail.

Vectorization

An entirely different approach is vector extraction or vectorization. Vectorization first creates a resolution independent vector representation of the graphic to be scaled. Then the resolution-independent version is rendered as a raster image at the desired resolution. This technique is used by Adobe Illustrator Live Trace, Inkscape, and several recent papers.^[7] Scalable Vector Graphics are well suited to simple geometric images, while photographs do not fare well with vectorization due to their complexity.

Deep Convolutional Neural Networks

This method uses Machine Learning for more detailed images such as photographs and complex artwork, Programs that use this method include Waifu2x and Reshade.

Applications

General

Image scaling is used in, among other applications, web browsers, image editors , image and file viewers, software magnifiers, digital zoom, the process of generating thumbnail images, and when outputting images through screens or printers.

Video

The magnification of images for the home theater area of importance in which HDTV-ready output devices with material in PAL-Resolution, for example from a DVD player. Upscaling is here (video scaler) of special chips performed in real time, the output signal will not be saved. The upscaling is therefore contrary to the upconverting of a material in which the output signal is not mandatory to create in real time, but will be stored instead.

Pixel art scaling

As pixel art graphics are usually in very low resolutions, they rely on careful placing of individual pixels, often with a limited palette of colors. This results in graphics that rely on a high amount of stylized visual cues to define complex shapes with very little resolution, down to individual pixels. This makes scaling of pixel art a particularly difficult problem.

A number of specialized algorithms^[8] have been developed to handle pixel art graphics, as the traditional scaling algorithms do not take such perceptual cues into account.

Since a typical application of this technology is improving the appearance of fourth-generation and earlier video games on arcade and console emulators, many are designed to run in real time for sufficiently small input images at 60 frames per second.

On sufficiently fast hardware, these algorithms are suitable for gaming and other real-time image processing software. These highly optimized algorithms provide sharp, crisp graphics while minimizing blur. Scaling art algorithms have been implemented in a wide range of emulators, 2D game engines and game engine recreations like HqMAME, DOSBox, and ScummVM. They have gained wide recognition with gamers, with whom these technologies have encouraged a revival of '80s and '90s gaming experiences.

Such filters are currently used in commercial emulators on Xbox Live, Virtual Console, and PSN to allow classic low resolution games to be more visually appealing on modern HD displays. Recently released games that incorporate these filters include Sonic's Ultimate Genesis Collection, Castlevania: The Dracula X Chronicles, Castlevania: Symphony of the Night, and Akumajō Dracula X Chi no Rondo.

See also

• Bicubic interpolation
• Bilinear interpolation
• Lanczos resampling
• Spline interpolation
• Seam carving
• Image reconstruction

References

External links

• 2D Image Filters at code.google.com including hqnx-family, SaI, Eagle, EPX, XBR, XBRz, Lanczos, etc.
• xBRZ Scaler Comparison and Testing Tool at SourceForge
• C library implementing the xBR pixel art scaling algorithm at GitHub
• AVIR image resizing algorithm at GitHub