Sensors, MSI Capture Equipment Rodeo

Sensors

What people documenting cultural heritage should know when comparing sensors used for multispectral imaging

Multispectral Imaging Capture Equipment Rodeo
Rochester, New York, June 27, 2025

Todd Hanneken, St. Mary’s University

Thank you to all joining us here in person and on Zoom. The last few days we have been testing a wide variety of components suitable for multispectral imaging at very different price points. I will be speaking about sensors, and will be followed by other presentations on lenses, lights and filters, and testing targets and procedures. The last few days have been the culmination of many years of learning about multispectral imaging, starting for me in 2009. First I learned about all the ways in which the high-end systems were optimal for multispectral imaging. In recent years we have seen a surge of interest in what can be done with systems that aim for accessible price points more than the absolute highest possible specs. The fact that we were able to test a $500 camera side-by-side with a $50,000 camera illustrates that a variety of price-points are comparable in the sense that they can be compared, though not to suggest equivalence. In the following fifteen minutes, I hope to share with you the highlights of what I have learned in the past fifteen years and the recent tests.

Overview

Sensors quantify the amount of light coming through the lens with varying degrees of spatial resolution and noise

Major Types of Sensors

Multispectral sensors vs. sensors intended to approximate the human visual system
Full-spectrum vs. visible spectrum
Monochrome (panchromatic) vs. color

Major Types of Sensors

Multispectral sensors vs. sensors intended to approximate the human visual system
- Silicon is receptive to wavelengths 200-1100 nanometers (less so at extremes, lenses also limit short wavelengths), humans perceive wavelengths 400-700 nanometers
- Silicon itself does not distinguish wavelength (without controlled lighting and filters), humans have three color receptors
- Multispectral imaging seeks to go beyond the range and/or color resolution of the human eye
Full-spectrum vs. visible spectrum
Monochrome (panchromatic) vs. color

Human Response to Color — The human visual system

Major Types of Sensors

Multispectral sensors vs. sensors intended to approximate the human visual system
Full-spectrum vs. visible spectrum
- Cameras marketed for astrophotography, machine vision, etc. do not limit the sensitivity of silicon to the human visual range
- Cameras marketed for color photography use filters to block infrared (IR)
- Relatively easy to remove the IR block filter
Monochrome (panchromatic) vs. color

So when we talk about full-spectrum sensors, we mean sensors that measure the range of the electromagnetic spectrum to which silicon is sensitive, which is greater than the range to which the human eye is sensitive. For multispectral imaging, we have two options. First we can look for cameras designed for markets other than color photography in the first place. One such market is astrophotography. The James Webb telescope is designed around the advantages of infrared for studying the cosmos, and thousands of amateur enthusiasts image galaxies from their back yards. It is a simple adapter to attach non-telescope lenses to such cameras. Another market is machine vision. When a self-driving car is deciding how to avoid colliding with a pedestrian, it takes advantage of all available wavelengths. Similarly, quality control in industry often uses invisible contrasts to detect problems before they are visible to humans. While these markets are large, they are not nearly as large as the total market for cameras made by Canon and Nikon. Canon can sell a camera so cheaply that it might be a good bargain even if we have to pay more to modify it to reverse the specialization for human perception. The simplest modification is the removal of the infrared block filter. This service is offered by several companies and might even be a do-it-yourself project if you have the option to ruin a camera or two learning the skill.

Major Types of Sensors

Multispectral sensors vs. sensors intended to approximate the human visual system
Full-spectrum vs. visible spectrum
Monochrome (panchromatic) vs. color
- Silicon counts photons, does not measure wavelength
- Panchromatic in the sense of measuring all the wavelengths
- Monochromatic in the visualization of all intensities as a shade of gray
- Digital color cameras mostly use a Bayer mosaic to divide the pixels into pixels dedicated to red, green, and blue

Although there are technically other ways to do it, most color cameras use a Bayer mosaic. This is a very tiny mosaic of filters spread over the pixels of the sensor, as close as possible to minimize error due to light arriving at low angles. This means the mosaic is physically attached to the sensor and not easily removed. The mosaic works by causing each pixel to specialize in one of the three colors distinguished by humans. Half the pixels measure green, a quarter measure red, and a quarter measure blue. In ordinary photography, it is generally safe to assume that any of the pixels not dedicated to blue can be estimated as the average of the four or eight nearest pixels. In photographs of galaxies, that is not a safe assumption at all. For documenting cultural heritage, the assumption is worth testing.

Options for using sensors with Bayer mosaics for MSI

Embrace the fact that you are getting three filters for free in one shot, accept reduced spatial resolution
Use only the green channel
Interpolate
Removing the Bayer filter is difficult, expensive, and never truly equivalent to panchromatic
A different question is how well the three micro-filters approximate human color perception.

For multispectral imaging of cultural heritage, we have a couple of options for managing the drawbacks that can come with an affordable mass-market camera. I sorted these from my most to least preferred, but I acknowledge room for disagreement here. My favorite option is to embrace the filters. By controlling the wavelength of light at the source or with additional filters, true multi-band data can be captured even with a Bayer mosaic. Even the very high-end systems still use red, green, and blue filters to distinguish ultraviolet fluorescence. When I finally bought a panchromatic astrophotography sensor, the first thing I did was to buy filters so I could capture accurate color. It takes me at least three captures to achieve color (actually more). Of course at the end of those three captures I have three times as many pixel measurements. We can embrace the free filters if we accept that a capture with a Bayer mosaic is comparable to three captures with a sensor that has a quarter of the pixels. As manufacturers constantly increase the number of megapixels in response to what consumers think they want, this might work out quite favorably. Another option is to use only the green channel, which reduces our pixel count by half rather than a quarter. If you recall the graph of the human visual system, it will make sense why the Bayer mosaic has as many green as blue and red combined. Human green perception covers a wide band of the visible spectrum, and differs only slightly from red. Another option is to interpolate, the same way pretty-picture photography does all the time. Processing software has a button to turn any color picture monochrome, which can be a nice artistic effect. For multispectral imaging, and I may be in the minority here, interpolation seems dishonest. The same methods of interpolation and upscaling could double the resolution of any image. It is worth emphasizing the difference between captured data and interpolated data. Finally, some consider it an option to try to remove the Bayer filter. This process is expensive because the failure rate is so high even for specialists. Check the warranty before considering this option. While it is possible to increase the amount of light that passes by the Bayer filter, it is not possible to eliminate the effect of the mosaic on the physical sensor and its processing firmware. Images captured in this way still need to be interpolated to be used as full resolution. Finally, I wish to note that human color perception is difficult to map to tiny filters, so some Bayer mosaics do a better job than others at color accuracy. Color accuracy is addressed in a separate presentation on lights and filters.

Sensors We Tested by Type

Full-spectrum panchromatic: QHY 411 ($50,000), QHY 600 ($4000), Pixelink ($1245), Flir Blackfly ($700), QHY miniCam ($500)
Full-spectrum with Bayer mosaic: Kolari RP ($2000), Spencer R7, Kolari Elph ($350)
Consumer color cameras: cell phones, some tests of Canon T1i

These are the cameras we tested in three categories. Note that the generalization that full-spectrum panchromatic is more expensive is not absolute. The price points of the full-spectrum panchromatic options differ by two orders of magnitude. The main differentiator is the size of the sensor and the number of pixels (the size of each pixel on the sensor is the same). The QHY cameras are marketed to astrophotography. The Pixelink and Flir cameras are marketed to machine vision applications. The Kolari and Spencer cameras are based on Canon cameras of the same model. The Kolari Elph has other advantages, particularly portability. One might question whether unmodified consumer cameras belong in a list of components suitable for MSI in the first place. First, they do very well with the principle of “the best camera for the job is the camera you have.” We all can afford them, and they might be the best option in unplanned opportunities. Second, they may not be inherently multispectral, but they can be if combined with non-white lights. A handheld or specialized ultraviolet light with a cell phone has the same underlying concept as UV fluorescence captures with filters. Cell phones are also more sensitive to infrared than the human eye, although it remains to be established if meaningful real-world results can be achieved with a cell phone and infrared light.

Sensor Size and Resolution

Sensor size is the physical size of the sensor (measured in millimeters) or the size of a single pixel (measured in micrometers)
Spatial resolution is the number of pixels measured (measured in megapixels), linked to pixels per inch
Analogous to the size of a pizza vs. the number of slices into which it is cut

I just noted that the difference between the $500 camera and the $50,000 camera is almost entirely a matter of sensor size. They are identical in manufacturer and every other metric discussed in this presentation. I think we all have a concept of spatial resolution, whether we think about it more in megapixels or pixels per inch. Sensor size, and especially the terms we use for sensor size, are considerably more confusing. The cell phone market has intensified the race for more megapixels and smaller everything. It is worth noting that packing more pixels into a sensor of the same size simply shrinks the space available to measure photons hitting any one pixel. Doubling the number of pixels also doubles the amount of time necessary to properly expose the same amount of light striking the sensor. You could think of the sensor as a pizza and the resolution (megapixels) as the number of slices into which it is cut. A small pizza is only going to feed you so much no matter how many slices into which it is cut. A small sensor is only going to measure so much light no matter how many pixels into which it is divided.

I don't know about you, but I have complex feelings when I look at this graphic of different sensor sizes. It is remarkable that cell phones can do what they do with such tiny components. The truth is that it has less to do with accuracy and more to do with tricks and processing to make pictures look pretty to the human eye. The DSLR that I use for family photos and vacations is APS-C, which always seems like plenty. Yet the APS-C sensor is less than a quarter of the area of the medium format. I spent the past year preparing a critical edition based on images captured with a medium-format sensor. I never complained that the images were too sharp. The source of my ambivalence about the larger sensors is based on the cost not only of the sensor but the lens required to use it. This image shows the relative sizes of sensor and what they are called. Although the larger sensors measure more light, they are also much more expensive to produce and require more expensive lenses.

Sensor Size

Bigger is better in that a sensor twice as large will capture twice as much light
Bigger is more expensive for the silicon, the camera, and especially for the lens to focus more light at comparable depth of field and chromatic aberration

Sensors We Tested by Size

Medium format (54x40mm): Sony IMX 411 in the QHY 411 Pro ($50,000)
Full frame (35x24mm): Sony IMX 455 in the QHY 600 ($4000), Kolari RP ($2000)
APS-C (22x15mm): Spencer’s R7, Canon T1i
1" type (13x9mm): Sony IMX 183 in the Flir Blackfly and Pixelink ($700-1200)
1/1.2" type (11x8mm): Sony IMX 585 in the QHY Minicam ($500)
1/1.31" type (9.6x7.2mm): Google Pixel 6

The sensor sizes we tested range from 9.6 to 54 millimeters wide. The names of the types are not intuitive. In the days of film, 35 mm cameras were so-called because they exposed an area 35 millimeters wide on the film. That same dimension is now considered “full frame.” Without getting into the issue of aspect ratio, the width is still the most intuitive metric. The types labeled as inch or fraction of an inch are actually considerably smaller than an inch. These are the sensors we tested grouped by physical size. Physical size correlates strongly with price, even after including the cost of removing a visible pass filter. I'm counting a cell phone camera as free because you already have it.

Sensor Spatial Resolution

150 megapixels = 14192 × 10640 = 24" x 18" at 600 PPI (QHY 411, $50,000)
62 megapixels = 9600 x 6422 = 16" x 11" (QHY 600, $3900)
20 megapixels = 5472 x 3648 = 9" x 6" (Flir Blackfly IMX 183, $700)
12 megapixels = 4032 x 3024 = 7" x 5" (Pixel 6 processed)
8 megapixels = 3856 x 2180 = 6" x 4" (QHY mini, $500)
Workarounds: tiling, stitching, lower PPI

Besides sensor size, we also have spatial resolution. Megapixels listed include all pixels. Divide by four if demosaicing to three channels. Divide by two if using only green. I have to make an effort to restrain a rant about how megapixels are a misunderstood metric of sensor quality. Many pixels without a corresponding increase in sensor size means the sensitivity of any one of those pixels is limited. A color camera that advertises thirty-two megapixels is comparable to three captures at eight megapixels with three color filters. Or perhaps it would be no less solid logic to say that an image processed from 17 captures of 61 megapixels is really a gigapixel image. The assumption that more is better is also worth re-examining. 600 PPI of a complete object may be appropriate for archival imaging, but imaging a region of an object or imaging at a lower spatial resolution may be appropriate in some contexts. It is worth remembering that a full-HD screen is only two megapixels, so all the spatial resolutions listed can only be utilized by zooming in on a region anyway.

Noise

Can appear as salt-and-pepper or lines
Consistent noise can be corrected through dark subtraction
Inconsistent noise, even if minor in one image, amplifies when multiple images are processed for enhancements or accurate color
Software “denoise” functions are not magic, effectively blur the image and reduce spatial resolution
Can be caused by thermal energy, addressed by thermoelectric cooling

Finally we come to the metric of sensor quality that I, at least, came to appreciate the most recently. Sensor noise is right up there with size and resolution for considering the quality of a sensor for a given price point or application. It is much more difficult to compare fairly based on claims from different manufacturers. One of our most important tasks in the tests this week was to quantify the differences in noise across different kinds of sensors at different price points. There is perhaps still room for discussion about the relative importance of avoiding noise for documenting cultural heritage. As already mentioned, astrophotography of a galaxy is a clear case of a situation in which noise must be avoided. In pretty-picture photography, a pixel substantially brighter than its neighbors may be safely dismissed as aberrant noise. In astrophotography, this false assumption is called the “star eater” effect. It may be less obvious that the same problem applies to human-crafted objects when imaged at a true resolution of 600 PPI. There are two reasons noise is important regardless of target object. First, noise is tied to exposure time. Almost all the problems of imaging could be solved with longer exposure times. Filters reduce light, check. Tight aperture to increase depth of field, check. Weak narrowband LEDs, check. Diffusers dissipate too much light, check. Light collimated with distance decreases intensity with distance exponentially, check. The main reason we don't have sixty-minute exposure times, besides our schedules and risk of movement, is sensor noise. If we intend to study very weak signals such as infrared fluorescence, we need very long exposure times. For those images to be meaningful, we need to minimize noise. The second problem with noise is linked to one of the most fundamental concepts of advanced digital multispectral imaging, namely registration. Registration means that each pixel represents the exact same spot on the target across a number of captures in different lighting conditions. We process those many captures into highly-accurate color or linear transformations such as PCA. The problem with noise, particularly inconsistent noise, in those stacks of registered images is that the noise amplifies from each layer. A calibrated color image based on ten or more images will be much noisier than any of the component images, easily crossing the line from visually acceptable to visually unacceptable. Some linear transformations will derive component images that show nothing but read noise from the sensor. Noise has several causes. The only noise that can be effectively addressed in processing is consistent noise. If a given pixel is consistently hot or cold we can correct for that, as long as we remember to do it. In our tests this week we distinguished every exposure time as a distinct case. We captured multiple dark images at that exposure time, and took the median of those values for each pixel as the consistent noise. Other software techniques to “denoise” an image amount to blur, reducing the true spatial resolution. This practice, combined with other practices such as interpolating red and blue channels, can quickly bring 600 claimed PPI to an effective PPI of well less than 300. Many might consider 300 PPI sufficient for casual visual appreciation, but having studied different images of palimpsests, I can affirm that 600 PPI is always worthwhile. So what is there to be done about noise? The field of astrophotography is way ahead of us on the idea of long exposures and low noise. Especially because they and we are interested in infrared, we have a problem with thermal energy introducing noise. The solution is to use thermoelectric cooling of the sensor.

Sensors We Tested with Thermoelectric (Peltier) Cooling

QHY 411 ($50,000), QHY 600 ($4000), QHY miniCam ($500)
Rated for cooling to 30-45° Celsius below ambient temperature
Tests used -10° Celsius (14° Fahrenheit)

How We Measured Noise

Dark noise: how much inconsistency comes from the sensor when there is no light at all?
White noise: how much inconsistency comes from the sensor as it reads light?

In our tests earlier this week, we measured two kinds of noise. In an ideal world, if there was no light reaching the sensor, all measurements would be consistent at zero or whatever is considered the minimum. We put a lens cap over the lens and otherwise blocked all light. The standard deviation of the pixel values told us the noise in a dark image. We also created an image which told us the consistent noise by taking the median value of multiple captures for each pixel. Subtracting the consistent noise from any dark image gave us a measure of the inconsistent noise. There is also noise from reading the signal of light we are trying to measure. In an ideal world, every photon of light would charge one electron on the silicon and measure one integer on a 16-bit scale. To compare components in the real world, we measured signal as the average values of pixels on a uniform white surface, namely spectralon. We measured noise as the variation (standard deviation) in those values that should have been uniform.

Noise Results

QHY 411 ($50,000) TBA, TBA
QHY 600 ($4000) TBA, TBA
Kolari RP ($2000) TBA, TBA
Spencer R7 TBA, TBA
Pixelink ($1245) TBA, TBA
Flir Blackfly ($700) TBA, TBA
QHY miniCam ($500) TBA, TBA
Kolari Elph ($350) TBA, TBA

Settings

Exposure time, time value (Tv)
Gain, ISO
Offset
Bits per pixel

When using sensors for multispectral imaging, we should also pay attention to the settings we use. Exposure time is the most familiar. More time measures more light, but also increases sensor noise and risk of vibration or movement in the camera or target. Astrophotography cameras have a maximum exposure time of sixty minutes. Gain refers to the amplification of the analog signal before it is converted to digital. Although excessive gain can increase noise, it is a misconception that lower gain always means lower noise. Some gain is necessary to make use of the sixteen bits of dynamic range output by most cameras. The term “unity gain” refers to the setting at which one electron on sensor is measured as one integer when converted to digital. In our tests we used gain settings recommended by the manufacturer, or ISO 400 for Canon sensors. Offset is the extent to which the signal is shifted higher, or to the right if you’re picturing a histogram. The digital conversion does not allow negative numbers, which would happen even with no light due to noise. We also want to avoid values of zero to avoid divide-by-zero errors in processing. In our tests we made sure that each sensor was set such that the minimum value was never less than 100 (for sixteen bit data). We also set all our sensors to output sixteen bits of data. Silicon does not measure quite that precisely even in the best of conditions, but gain and some gain tricks can produce that level of dynamic range. Some sensors did not measure more than 12 bits of data. Eight bits may be fine for human perception, but for processing and flexibility in exposure settings, eight bits is not enough. For this week’s tests we relied mostly on the manufacturer to determine the best gain and offset for each sensor. We used those settings consistently for all the tests. Details will be reported on the rodeo website. It remains for future work to compare thoroughly different gain settings.

Questions?

Slides and speaker notes available online at https://palimpsest.stmarytx.edu/rodeo/2025/sensors.html