Industrial autonomy is becoming increasingly popular to increase efficiency and performance. Applications such as autonomous mobile robotics (AMR), warehouse robotics, drones, agriculture, factory inspection, and security/surveillance implement machine vision-based artificial intelligence (AI) with advanced technology to perform critical functions. Improving existing object detection and identification depends on solving the challenge of image capture in difficult lighting conditions, with objects in motion and fine detail over longer distances. onsemi’s new sensor with embedded High Dynamic Range (eHDR) image sensor offers the features to solve the complicated design challenges engineers face when developing autonomous machine vision systems.

Importance of High Dynamic Range in Industrial Automation

Many industrial applications operate in scenes with harsh lighting that have both bright and dark areas. For example, an outdoor AMR must operate accurately in conditions with bright sun and dark regions in the same scene. A decision to trigger an alarm for a surveillance system relies on distinguishing the movement of an unauthorized person moving from the shadows to bright areas. While factory engineers control lighting to attain the ideal brightness levels where detail is required, specific warehouses may not be able to control lighting or require high redevelopment costs. Additionally, industrial robotics should be able to scale and operate in multiple locations, which potentially introduces non-deterministic lighting conditions. Drones carrying packages may do so under nighttime conditions or high-intensity sunlight. These are just a few instances where these autonomous systems must operate under scenes with high dynamic range.

MULTIPLE EXPOSURES = HIGHER DYNAMIC RANGE

The capacity of an image sensor to capture detail in both the darkest and brightest areas of an image is the dynamic range of the sensor, measured in decibels (dB). Generally speaking, 120 dB of dynamic range captures detail in all but the most extreme scenarios (automotive applications require an even higher dynamic range, especially regarding passenger safety). Such scenes with dark and bright areas using a single long exposure may oversaturate bright regions, and conversely, a single short exposure may not capture detail in dark places.

Figure 1. Image with Shorter Exposure Time Captures Detail in the Bright Areas and Loses Detail in the Dark

Figure 2. Image with Longer Exposure Time Captures Detail in Dark Areas and Loses Detail in Light Areas

Therefore, to capture detail in all regions, sensors can take many images with multiple exposures with short, medium, and long exposure times. These exposures are then combined into a single image to achieve a high dynamic range, a process called linearization.

Figure 3. Figure 1 and Figure 2 scene now with High Dynamic Range

Now with a range of detail from light to dark, AI recognition systems can improve identification and recognition performance.

The High-Resolution Challenge

More extended distance recognition and more detailed images require higher resolutions. Between five to ten meters away from an object, 1080p is insufficient to provide the detail needed to detect and classify objects. Therefore, there has been increasing demand for 4K and even higher-resolution sensors. With images of these sizes, bandwidth becomes a design challenge because the size per image rises significantly. Furthermore, a sensor that achieves 120 dB HDR needs three exposures, and 3x the effective frame rate as the sensor takes three distinct images for every frame. For example, if a sensor’s effective frame rate is 30 frames per second (fps) for three exposure HDR, the sensor array, circuitry, and output is effectively operating at 90 fps. A three-exposure-HDR 4K image at 30 fps with linearization executed in an ISP separate from the sensor requires 9000 Mbps, a challenge for processing and all but the fastest interfaces. Combining high-resolution and off-sensor HDR is a difficult challenge for a camera system.

Figure 4. A Typical High Dynamic Range Architecture Sends Multiple Images over the Interface between the Image Sensor and ISP, potentially Straining or Exceeding the Available Bandwidth at High Resolutions

To alleviate both issues, the onsemi AR0822 intelligently embeds the high-dynamic range by integrating real-time linearization into the sensor. By doing so, conventional processing and interfacing, even for higher resolutions, is now more achievable.

Figure 5. An AR0822 Embedded High Dynamic Range (eHDR) Image Sensor

Motion and LED Lighting Artifacts

Artifacts are introduced in the HDR images because three exposures are taken at different points in time. The locations of fast-moving objects (angular velocity) will slightly differ among the various exposures.

Figure 6. A fan rotating showing motion artifacts from multi-exposure HDR

Additionally, scenes with LED lighting can introduce additional artifacts. LEDs turn on and off at a frequency that cameras can see to conserve energy, but not the human eye. An LED might be on during the first exposure and off during the second and third, resulting in different light levels among the multiple exposures and images.

Figure 7. An LED light box with two columns lit, the difference in intensity is an LED flicker artifact

AR0822 eHDR includes “intelligent linearization” to address these artifacts caused by multi-exposure frame combination. This is possible by sensing the difference in signal level captured by different exposures within a frame and thus trying to reduce the artifact commonly caused by motion or LED flicker.

 

Figures 8 and 9. A fan without motion artifacts and an LED lightbox with two columns fully lit as a result of AR0822’s intelligent linearization

By providing high-quality performance in challenging lighting conditions, eHDR boosts object detection and identification, combined with the desired resolution that AI systems need in security/surveillance, AMR, and industrial mobile robot applications.