Super-Resolution - Image Stacking + Sharpening [part 2 of 4]

Previously I covered a simple cubic up-rez + USM "super resolution" technique.  In this blog entry I would like to cover a second "super-resolution" technique.  This involves up-rez'd image stacking.

This all re-started for me when some folks talked about Photoshops "new" function that up-rez's an image.  Some people think it's the cat's meow.  Others point out that Topaz AI somethingorother is better.  And there have been comparisons showing how "good" an up-rez can be these days.  

In contrast, the image stacking approach took the idea that shooting a number of images handheld would cause just enough pixel to pixel displacement that a careful practitioner could average the information when up-rezing each layered stack image and then setting each layer's opacity. The idea tries to emulate in-camera multi-shot sensor displacement and image stacking.  Olympus and Sony implement this feature on some of their cameras.

Here is the approach.

1. Take a number of handheld images of a scene
   
2. Load these images as layers into Photoshop or the Gimp
   
3. Cubic up-rez - with an appropriately high interpolation filter sample rate
4. Align the layers - this can be very tricky, but there is software that can help
   
5. Set the Opacity of each layer to average the information
6. Flatten the image
   
7. Unsharp Mask sharpen or use some other image sharpening method

 

NOTE: Remember that I've chosen the Gimp specifically because the software designers have correctly implemented the Cubic interpolation function.  We will select the X/Y resolution of the interpolation filter and it will be properly applied to the image.  

This is very important as I've found that some software packages don't correctly apply the image resolution settings when they perform an up-rez and images can come out "blocky" and "pixilated" as you increase the image dimensions.

Here is what I suggest.  Using the Gimp, select...

1. Image -> Scale Image
2. Quality -> Interpolation -> Cubic
3. X/Y resolution -> 1200 - this right here is the secret to success
   

 

Here, once again, is the base scene that I will work from.

 

In the following comparison I show the base image as processed in RawTherapee and with "Capture Sharpen" applied.  

Then I show the Gimp output of a 4 image stack with Image -> Scale image from 6000 pixels on the long side to 9000 pixels (a 2x area increase in size) with light USM (unsharp mask) applied in selecting 1 pixels.  This is followed by the image stack sharpened with a sharpening function implemented in G'Mic called Richardson Lucy, which is much more aggressive than a USM.

NOTE: Some practitioners suggest using as many as 20 or more images to stack, up-rez, and then average.  I have tried this approach and after 3 or 4 images, I can see no improvement in image "resolution." YMMV.

 

 

The image stack approach really seems to struggle to add the expected "resolution" to the up-rez'd image.  The USM image is soft to my eyes, even with just a mild 2x area increase in image size.  This should be "easy", right?  Well it's not.

The Richardson Lucy sharpened image looks pretty good, but it is starting to look "artificial" and "water colory."

Unless I'm seriously missing something, the handheld multi-image stacking approach doesn't quite live up to its initial promise.  It would be interesting to see how this compares with Olympus or Sony sensor "wiggle" in-camera up-rez functions.  Should someone care to share an image or two, I'm all eyes.

What is approach does, however, is provide for very clean, noise-free output.  So, in my way of thinking, there is a definite use for this technique. I have tried this using very high ISO images where there is a ton of noise and the stacked output looked rather nice.  From what I hear, cellphones freely use this approach when making images in dim light.