Sony commercializes "IMX500" and "IMX501", the world's first image sensors equipped with AI processing functions

Equipped with image analysis processing function by AI in the logic chip

On May 14, Sony announced the world's first announced that it will commercialize the intelligent vision sensors "IMX500" and "IMX501" as image sensors equipped with AI processing functions.

The IMX500 is a 1/2.3-inch (diagonal 7.857mm) bare chip product with approximately 12.3 million effective pixels.

The IMX501 is a 1/2.3-inch (diagonal 7.857mm) package product with approximately 12.3 million effective pixels.

This product is the world's first image sensor that uses a layered structure in which a pixel chip and a logic chip are layered, and the logic chip is equipped with an image analysis processing function using AI. By performing AI processing within the sensor on the signals acquired by the pixel chip, it is possible to realize an edge AI system without the need for a high-performance processor or external memory.

Furthermore, in addition to reducing the amount of data and considering privacy by outputting metadata (meaning information belonging to image data) that does not output image information, real-time tracking of objects by high-speed AI processing, built-in memory It is said that various applications will be realized by various functions such as selection of AI model according to user's usage environment and conditions by rewriting.

The main features are as follows.

　Approximately 12.3 million effective back-illuminated pixels are arranged on the pixel chip to capture information with a wide viewing angle. In addition to the normal image sensor signal processing circuit, the logic chip is equipped with Sony's proprietary DSP (Digital Signal Processor), which handles signal processing specialized for AI, and memory that can write AI models. This makes it possible to realize an edge AI system without requiring a high-performance processor or external memory.

　In the process of processing the signal obtained from the pixel chip with the logic chip, ISP (Image Signal Processor) processing and AI processing are performed to output the target object as metadata, Reduce the amount of data you handle. In addition, by not outputting image information, it is possible to reduce security risks and take privacy into consideration. In addition to normal image sensor images, select the output data format according to the user's needs and application, such as ISP output format images (YUV/RGB) and ROI (Region of Interest) images extracted from specific areas. It is also possible to

　When shooting a video with a normal image sensor, it is necessary to connect each frame of the output image to AI processing, so more data is sent and real-time It was difficult to ensure the With this product, the logic chip performs ISP processing and high-speed AI processing (processing time of 3.1 milliseconds for MobileNet V1) to complete all processing within one video frame. This enables high-precision real-time tracking of objects while shooting video.

　Users can write any AI model to memory, rewrite it according to the usage environment and conditions, and update it. For example, if multiple cameras using this product are installed in a store, one type of camera can be used according to the purpose and application, such as installation position, situation, and time. It can be used for multiple purposes, such as counting the number of people entering a store at the entrance, detecting product shortages at the shelf, and a heat map of visitors (detecting places where many people gather) at the ceiling. It is also possible to rewrite the AI ​​model used for heat map detection to an AI model used to understand consumer behavior.