Index

On Super-Resolution

Contents

Most everyone is familiar with the tantalizing idea of achieving "super-vision" by correcting higher-order optical aberrations of the eye. The prospects of success seem to be limited by the fact that the performance of the normal optical system appears to match normal neural performance quite well, at least in central vision [1, 2]. But what about "super-resolution"? The term may be unfamiliar but hints at another approach to enhancing vision, namely, the processing aspect. In its original sense, super-resolution refers to a form of computational image processing that from a set of similar images seeks to generate a single image with enhanced fineness of detail. For a beautifully illustrated presentation of several types of procedures, see T.Q. Pham's 2006 thesis, Spatiotonal Adaptivity in Super-Resolution of Under-sampled Image Sequences [3]. Major application areas include digital still and video cameras. In these devices, the resolution of the image chip generally trails behind the optical resolution, resulting in an under-sampled output. By utilizing minute, movement-induced differences between successive images, super-resolution essentially generates a merged image with more detail. The price to pay for the increased spatial resolution is a reduction in temporal resolution. Ironically, the increasing use of inbuilt image stabilizing devices in digital cameras works against super-resolution.


Biological super-resolution

Super-resolution is a newly coined term but the principle of combining sequences of images may have been employed in the natural world for hundreds of millions of years and particularly among jumping spiders. Having immobile optics, jumping spiders actively move the retinas in their principal eyes to change the direction of "gaze". Once on target, the retinal movements change to bursts of complex, small-angle oscillations [4]. The latter generate sequences of closely similar images. Their resolution is lower than the eye's optical resolution [5], setting the scene for super-resolution neural processing. In the photograph above, the dark dots in the center of this jumping spider's anteromedian eyes show that it has aligned its retinas with the camera (using large-scale retinal movements) and may be scrutinizing camera detail (using small-scale movements). Incredibly, spiders can generate retinal optokinetic nystagmus [6].

Humans, too, sweep retinal images, using a combination of microtremor, microsaccades, and drifts [7]. Microtremor cannot be seen because of its high frequency (ca. 90 Hz) and small amplitude (≤ 0.5 minutes of arc) but drifts and micro-saccades can easily be visualized with the aid of after-images. In the figure to the right, look steadily at the red dot for at least 30 seconds. Then fixate steadily on the green dot and watch how the after-image of the cross moves around. It is an enigma why these movements are not seen in daily life [8].

Microsaccades and microdrifts can be eliminated in laboratory settings by artificial image stabilization. Under such conditions, resolution capacity has been shown to drop in a statistically significant manner but the magnitude of change was small and apparently undetectable to the test subjects [9]. Sweeping is more clearly gainful in another way, namely, to prevent the fade-out that occurs if the retinal image is kept perfectly stationary relative to the neural matrix [7]. Fading-out (the Troxler effect) is still noticeable in peripheral vision and the more so the larger the eccentricity [10]. The eccentricity effect can actually be seen in the figure: after a few seconds of steady fixation on the red dot, the blue dot will disappear. Somewhat later the green one will follow suit. The fade-out effect is immediately abolished on changing fixation.


Clinical relevance

Because of the normally close match between the optical quality of the eye and the dimensions of the neural matrix, normal subjects seem to have little to gain from super-resolution. The situation may be quite different in eyes with abnormal neural matrices. A partial loss of neural elements causes under-sampling of the retinal image and under-sampling is a requisite for gainful super-resolution.

What is under-sampled vision like? A subnormal resolution acuity is a familiar characteristic, of course. When asked to explain why more difficult lines on the acuity chart cannot be read, subjects commonly describe their perceptions as "dimmed" or "blurred" and they rarely use terms suggestive of actual depletion, e g, splotchy images (as illustrated elsewhere on this site [ A, B]), presumably because of the operation of eye movements and cerebral filling-in.

A predictable consequence of under-sampling is aliasing [11]. The term signifies a false representation. Various forms of aliasing can easily be demonstrated in many settings, including digital images. The fan-like pattern in the left panel shows good detail. However, if the pattern is under-sampled (right panel), the finest components can no longer be rendered correctly and jagged lines and false curves will appear. Aliasing is commonly encountered in broadcast television where repetitive details that are near the system's resolution limit, e g, fine fabric weaves, often are rendered erroneously, with color moirés and erratic movement. Similarly, depleted neural matrices should be prone to generate aliased percepts although they might be hard to identify and to describe. Depleted matrices may also suffer from Troxler effects and particularly so if fixation is eccentric. It seems natural to ask whether subnormal resolution, aliasing, and Troxler effects due to neural depletion perhaps might be counter-acted by employing supra-normal image sweeps and recruiting neural super-resolution.

Incidentally, super-resolution needs to be distinguished from hyperacuity. The latter refers to acuity tasks that involve alignment of detail. Alignment can be done at much smaller scales than those of conventional resolution, hence the "hyper" epithet.


Simulating super-resolution in a depleted matrix


This simplistic display aims to demonstrate that under-sampled information can be summarized over paths of movement to form a more complete percept. This is a key feature of super-resolution.

The display contains a row of randomly selected test letters of fixed size. Right-click on the display to bring up a new combination of letters. Various degrees of matrix depletion are exemplified in the left panel; black dots represent non-functional matrix units or "holes". In the display, the matrix unit size is set approximately equal to the letter stroke width, mimicking the conventional resolution limit. Defective matrices can be superposed on the test letters by selecting under the Loss menu. Each defective matrix masks those parts of the test letters that coincide with matrix defects. Sparse defects can be seen to barely affect legibility whereas medium density defects have a pronounced effect. Very few, if any, letters can be read with the most severely depleted matrix. Now, try adding movement to the test letters, under the Mode menu. Movement can be toggled on and off by left-clicking on the display. Rates and amplitudes can be altered with the sliders. Generally, movement seems to improve legibility considerably and particularly when the path of movement forms a closed curve.


This browser does not support the HTML5 canvas

Super-Resolution Demo     © L. Frisén 2007




Mode Loss Chars


Formal testing


The above display is meant for demonstration only. A somewhat different display is needed for formal measurements. A second display offers adjustable letter size and an inbuilt test protocol for generating frequency-of-seeing curves. It is fair to sound a warning: formal measurements are a bit involved. Further, this second display has the format of an unsigned Java applet, so it requires the Java JRE plug-in, which can be downloaded from Oracle. Additionally, Windows versions ≥8 require release of the Java block. To release the block, open Window's Control Panel, enter Java in the search field, open the Java Control Panel, and select Security. Click the High option button. In the Exception Site List enter http://www.neuro-o.se/, tap Apply and OK, and close the control panels.

For a full presentation of results from formal testing, see [12].




Super-resolution as a visual aid

From the above it appears that the super-resolution principle might be gainfully invoked in patients who have suffered partial losses of neural elements. What seems to be required is

  1. retinal image sweeps that are large enough to envelop sufficient numbers of remaining functional units
  2. ability to suppress following eye movements
  3. learning to combine image fragments over space and time
The required, supra-normal sweeps might not need to be generated internally, by the subject. Indeed, learning to oscillate the eyes at will is a difficult task and those few who manage to generate voluntary nystagmus tire very rapidly. Further, two-dimensional movements may be more effective than linear ones. Optimum sweeps might best be generated externally, by moving objects of interest, e g, snippets of text, as exemplified in the above displays.

As to learning to extract detail over space and time, it is submitted that this is not a foreign procedure. Witness the normal behaviour during acuity testing: reading speed consistently decreases with decreasing optotype size, suggesting an increasing need of multiple re-fixations.

Interestingly, in a quite different type of setting, induced oscillations have proved useful to enhance resolution in robotic vision.



Posted on December 12, 2007
Top

End