Save the dolphins with artificial intelligence
Disclaimer - This article is about improving images of dolphins and not actually about saving the animals themselves. Stay with us though, its worth a read!
Upscale images with artificial intelligence using LetsEnhance
It has always been impossible to accurately upscale bitmap images without quality loss. Until now that is.
The problem with bitmaps
Stored bitmap images (JPEGs, GIFs, PNGs, TIFFS, etc.) contain information about the position and colour of individual pixels in a graphic. When displayed at its original resolution, the image will look great but any attempt to upscale it will break the illusion and individual pixels will become visible.
In addition, the so-called 'lossy compression' algorithms used when storing some file types, including JPEGs, leads to the introduction of distortions (known as artifacts).
This is in contrast with vector images (e.g. SVGs) which store information about images as mathematical operations which can recreate the image at any size.
But what happens if you only have a bitmap original to work with?
One method of solving the bitmap upscaling problem is to fill in the gaps between pixels. The technical term for this is 'resampling.' The Photoshop interpolation function attempts to resample upscaled images using bicubic scaling and similar mathematical processes.
These functions calculate what colour the pixels in the gaps should probably be and then creates them. Although you can choose between different resampling options, the tool is quite crude and images upscaled in Photoshop generally look blurred and pixelated.
From Google Brain to Let's Enhance!
What has Google Brain got to do with upscaling images? Quite a lot actually. In 2011, the Google Brain project was set up to study deep learning (a type of machine learning) and its potential for advancing the field of AI.
Out of Google Brain came TensorFlow, a software library for data flow programming used for machine learning apps. Google open-sourced the TensorFlow code in November 2015 and a community of eager coders seized upon the chance to use machine learning, including neural networks, in their own various GitHub projects.
Among the ongoing community projects spawned by Google Brain are the Super-Resolution Generative Adversarial Network (SRGAN) and Enhanced Deep Residual Networks (EDSR) frameworks. Chemistry expert Alex Savsunenko and former CTO Vladislav Pranskevicius were researching image enhancement when they came across the papers from SRGAN and EDSR.
The pair used this knowledge to create Let's Enhance!, a free image upscaling website using sophisticated neural networks rather than basic mathematical principles to work out what should be in the gaps between the original pixels. Let's Enhance! can also remove artifacts introduced during lossy compression.
Let's Enhance v Photoshop: an examples
To appreciate the differences between Photoshop interpolation and upscaling using machine learning, it is useful to compare them side by side. We took a small image of a dolphin (200 pixels wide) and upscaled it to 800 pixels using Photoshop. The resulting image (see below) has clear image noise around the outside of the dolphin.
The original image
The Photoshop upscaled image
We then upscaled the same image using LetsEnhance. As you will see, the compression noise is almost completely absent. The water patterns on the surface of the water and the sea bed also look much more natural. This is worlds apart from the Photoshop version.
The LetsEnhance upscaled image
As you can see, the colour and textural detail added in the Let's Enhance! images is more true to life than that seen in the Photoshop versions, creating a more faithful representation of the original.
At its current stage of development, Let's Enhance! works better with some types of image than others. For example, upscaling landscapes, pictures of animals and images featuring hair work well. Upscaling pictures featuring eyes can be less predictable, producing stunning effects in some cases and disappointing results in others.
The developers are addressing this inconsistency by working on ways to set up slightly different neural networks depending on the category of image being enhanced.
Using the Let's Enhance Website
For some hands-on experience with upscaling your own images you can head on over to the free Let's Enhance! website. You can then either drag and drop or click to upload a photograph to the website. You will then have to complete the fast and free sign-up process.
Providing the image is below the 5 megapixel size limit, 'Let's Enhance!' will process it before offering you the choice of three upscaled versions to instantly download. The processing can take a while longer than indicated on screen but you can let it run in the background as you attend to other tasks.
One of the most exciting implications of the Let's Enhance project is that it illustrates the potential of smart machines to improve not just image manipulation but the entire design process, introducing new tools and concepts that would have been unthinkable only a few years ago.