How Sensor Tech Will Shape Vision of the Future
Article By : Taehyun Kim
CMOS image sensors used to be for optimizing image quality; now they're being engineered to enhance human vision.
A camera, much like the human eye, relies on layers of parts — lenses, filters and electronics — to provide visual clarity, color and depth. In a smartphone camera, one of the key parts, comparable to the retina of the human eye, is known as the CMOS image sensor, or CIS.
The original purpose of CIS technology was to optimize image quality for viewing by human eyes. Now, CIS technology is being advanced for a new purpose — enhancing human vision. It’s being used as a sensor to gather digital image information that provides data for up-and-coming artificial intelligence (AI) applications. There are a range of new uses for this technology, from screening defects in manufacturing processes to detecting objects in dark environments.
As photo capabilities in mobile devices have become more enhanced and more competitive, there’s an expectation the technology will continue to improve and feed further innovation. As such, the global market for image sensors is expected to grow 6.2 percent by 2023 to become a $23.2 billion market, according to a 2018 report from Research and Markets.
Driving growth
For the most part, growth has been largely attributable to the improved performance of image sensors, but other factors are playing a role too. Smartphone manufacturers have turned to camera innovation as a way to differentiate their devices from competitors in a saturated market. New and diverse camera functions — optical zoom, for example — require advanced sensor technology. The development of low-power and compact image sensors, as well as the increased use of image sensing devices in biometric applications, has also contributed. And the trend toward multiple cameras in mobile devices has been a driver.
In smartphone cameras, the image quality is often tied to the CIS because its performance impacts key factors, such as resolution, sensitivity and signal-to-noise ratio (SNR). But it’s also allowed new product applications to surface while boosting the performance of imaging devices. As such, the CIS image quality of smartphone cameras has surpassed the level of compact cameras, and the gap from DSLRs is continually being narrowed.
A shifting focus
In its early days, the purpose of CIS technology was to optimize image quality for human eyes. As the technology advanced, its objective shifted to achieve image quality that was optimized for machine algorithms. But diffraction limits have limited the ability to miniaturize CIS pixels.
As a result, companies have been increasing the level of integration of CIS pixels through the continuous development of device and process technologies and by supporting various application fields through the development of image signal processing, or ISP technology. This has been a gradual transformation.
In the first phase, pixel engineers concentrated on making up for the sensitivity loss inevitably caused by the reduction of pixel size, developing many innovative technologies including on-chip-lens (or micro-lens), deep photodiode with thicker silicon, and backside illumination technology.
As the pixel size reached about one micrometer, the second phase, which focused more on crosstalk reduction, began. During this time, novel technologies such as metal grid structure in color filter layer and deep trench isolation process for Si photodiode were developed in order to suppress the optical and electrical crosstalk, respectively. With these new innovations, CIS platforms are expected to evolve into an information sensor that supports advanced additional functions, without being limited to image quality improvement.
The emergence of new technology
There’s another driving force behind this innovation: the emergence of stack sensor technology. Since the conventional sensor has a structure where pixels and circuits are implemented on the same substrate, it was essential to reduce the light-free area for the CIS size reduction. Therefore, only essential functions of analog/digital circuits were implemented and adding circuits for additional functions were very limited.
SK Hynix’s stack sensor is already capable of embedding a simple AI hardware engine inside the ISP on the lower substrate, based on the advanced semiconductor process. Meanwhile, new machine learning-based technologies such as super resolution, color restoration, face recognition and object recognition are also in development.
There are several areas where these new types of chips will be useful, and some of the innovations are already beginning to come to market.
Sony recently announced the release of two models of intelligent vision sensors, the first image sensors in the world to be equipped with AI processing functionality for cloud services. These products expand the opportunities to develop AI-equipped cameras, enable a diverse range of applications in the retail and industrial equipment industries and contribute to building optimal systems that link with the cloud.
For example, when installed at facility entrance, a camera with these sensors can count the number of visitors entering the facility. When installed on a retail shelf, it can detect stock shortages. Installed on a ceiling, it can be used to heat map visitors to determine areas where people gather most. Since it is possible to extract and classify various features from input images when using machine learning-based ISP technology, CIS will become a key component of information sensors that collect various data points about the image, the location, distances and other biometric information.
This becomes more critical when it applies to autonomous vehicles, which use at least ten cameras to detect their surroundings. To improve accuracy, there are various requirements that must be satisfied, such as high-resolution support for distinguishing distance objects, HDR support for recognizing objects even in dark environments, and pre-processing of the ISP to reduce the number of computations that the processor must handle.
In the security field, there’s a function that’s required to compress and encrypt image signals in the CIS’ built-in ISP and transmit them to an external processor. If the unencrypted image signal is transmitted to the outside as it is, the possibility of security vulnerability and information leakage increases. For this reason, the encryption function inside the CIS is essential.
The landscape head
Smartphone applications are leading in CIS market share, but many other applications are expected to emerge as drivers of CIS’ future growth, especially as machine vision applications grow and evolve. These emerging opportunities are pushing technology from mobile imaging into other growth areas, and we may see a shift from using vision for imaging to using vision for sensing and other interactive applications.
Moving forward, CIS will be utilized in various application fields, including smartphone cameras that will contribute to the creation of economic and social value, which will allow it to grow as a key component of information sensors in the future.
—Taehyun (Ted) Kim, Ph. D., is head of CIS ISP at SK Hynix
In smartphone cameras, the image quality is often tied to the CIS because its performance impacts key factors, such as resolution, sensitivity and signal-to-noise ratio (SNR). But it’s also allowed new product applications to surface while boosting the performance of imaging devices. As such, the CIS image quality of smartphone cameras has surpassed the level of compact cameras, and the gap from DSLRs is continually being narrowed.
A shifting focus
In its early days, the purpose of CIS technology was to optimize image quality for human eyes. As the technology advanced, its objective shifted to achieve image quality that was optimized for machine algorithms. But diffraction limits have limited the ability to miniaturize CIS pixels.
As a result, companies have been increasing the level of integration of CIS pixels through the continuous development of device and process technologies and by supporting various application fields through the development of image signal processing, or ISP technology. This has been a gradual transformation.
In the first phase, pixel engineers concentrated on making up for the sensitivity loss inevitably caused by the reduction of pixel size, developing many innovative technologies including on-chip-lens (or micro-lens), deep photodiode with thicker silicon, and backside illumination technology.
As the pixel size reached about one micrometer, the second phase, which focused more on crosstalk reduction, began. During this time, novel technologies such as metal grid structure in color filter layer and deep trench isolation process for Si photodiode were developed in order to suppress the optical and electrical crosstalk, respectively. With these new innovations, CIS platforms are expected to evolve into an information sensor that supports advanced additional functions, without being limited to image quality improvement.
The emergence of new technology
There’s another driving force behind this innovation: the emergence of stack sensor technology. Since the conventional sensor has a structure where pixels and circuits are implemented on the same substrate, it was essential to reduce the light-free area for the CIS size reduction. Therefore, only essential functions of analog/digital circuits were implemented and adding circuits for additional functions were very limited.
SK Hynix’s stack sensor is already capable of embedding a simple AI hardware engine inside the ISP on the lower substrate, based on the advanced semiconductor process. Meanwhile, new machine learning-based technologies such as super resolution, color restoration, face recognition and object recognition are also in development.
There are several areas where these new types of chips will be useful, and some of the innovations are already beginning to come to market.
Sony recently announced the release of two models of intelligent vision sensors, the first image sensors in the world to be equipped with AI processing functionality for cloud services. These products expand the opportunities to develop AI-equipped cameras, enable a diverse range of applications in the retail and industrial equipment industries and contribute to building optimal systems that link with the cloud.
For example, when installed at facility entrance, a camera with these sensors can count the number of visitors entering the facility. When installed on a retail shelf, it can detect stock shortages. Installed on a ceiling, it can be used to heat map visitors to determine areas where people gather most. Since it is possible to extract and classify various features from input images when using machine learning-based ISP technology, CIS will become a key component of information sensors that collect various data points about the image, the location, distances and other biometric information.
This becomes more critical when it applies to autonomous vehicles, which use at least ten cameras to detect their surroundings. To improve accuracy, there are various requirements that must be satisfied, such as high-resolution support for distinguishing distance objects, HDR support for recognizing objects even in dark environments, and pre-processing of the ISP to reduce the number of computations that the processor must handle.
In the security field, there’s a function that’s required to compress and encrypt image signals in the CIS’ built-in ISP and transmit them to an external processor. If the unencrypted image signal is transmitted to the outside as it is, the possibility of security vulnerability and information leakage increases. For this reason, the encryption function inside the CIS is essential.
The landscape head
Smartphone applications are leading in CIS market share, but many other applications are expected to emerge as drivers of CIS’ future growth, especially as machine vision applications grow and evolve. These emerging opportunities are pushing technology from mobile imaging into other growth areas, and we may see a shift from using vision for imaging to using vision for sensing and other interactive applications.
Moving forward, CIS will be utilized in various application fields, including smartphone cameras that will contribute to the creation of economic and social value, which will allow it to grow as a key component of information sensors in the future.
—Taehyun (Ted) Kim, Ph. D., is head of CIS ISP at SK Hynix