Design of CMOS image acquisition system based on FPGA

Based on FPGA

Bei Yan ,Yuefeng Sun,Fengfeng Ding,Haiwen Yuan

School of Automation Science and Electrical Engineering

Beihang University Beijing, China yanbei@buaa.edu.cn

Abstract—The paper designs a common image acquisition system for digital cameras with Camera Link interface. The system uses CMOS image sensor of 1.3 million pixels as the light-sensitive chip andhigh-performance FPGA as the main controller. In order to meet ease of use and high real-time demands, the system uses high-speed USB2.0 interface to transfer image data between the acquisition system and PC. Further the system reserves a communication interface with DSP though which DSP image processing functions can be easily extended in future. Therefore FPGA can communicate with DSP to achieve an embedded real-time image acquisition and processing system. The experimental results show that the system can capture images at the rate of 25frames/s when the frame size of CMOS digital camera is set to 1280*1024 pixels, which can meet the real-time image processing system requirements.

Keywords-CMOS image sensor; image acquisition; FPGA; USB2.0

II.SYSTEM ARCHITECTURE DESIGN

Figure 1 shows the image acquisition system structure. The system mainly consists of CMOS sensor imaging module, FPGA acquisition and control module, USB2.0 data transfer module.To make imaging module can be installed in any position in use, the system adopts Camera Link interface between the imaging module and the acquisition module which enable long-distance and high-speed image data transmission. The system also designs analog video input interface that can support standard analog camera. System reserves FPGA I/O ports which can be connected with EMIF bus and control signals of DSP. With the expansion of DSP, a real-time image acquisition and processing system can be achieved based on FPGA and DSP[1].

I.INTRODUCTION

Solid image sensors currently used are mainly two kinds, CCD and CMOS. With the development of large scale integrated circuit manufacturing technology in recent years, the performance of CMOS image sensors which originally at a disadvantage have been closer to CCD such as the sensitivity, noise, etc. Further CMOS sensor has high integration, need no complex driver circuit, and has advantages of small size, low power consumption, high anti-radiation ability, and so on, so it has been widely applied in image acquisition system of many fields such as robot vision.

Currently there are some problems of robot vision to be solved, especially the improvement of real-time. So the traditional computer-based image processing system can not meet demands. Now real-time image processing system based on DSP has been used more and more widely. At the same time the requirements of the speed of image acquisition system and the ability of cooperative processing with DSP are also increasing. Therefore this paper proposes the large area CMOS sensor image acquisition system program based on FPGA and USB2.0.As a result of adopting high-performance FPGA, DSP image processing sub-board can be easily extended, which form a high real-time image acquisition and processing system.

Figure 1. Block diagram of the hardwar platform.

FPGA is the timing control center and data exchange bridge of the system. Imaging module captures image under the control of FPGA, and output LVTTL level image digital signals. Then Camera Link interface circuit converts the LVTTL level to LVDS level to meet Camera Link protocol standards. Though the Camera Link cable the LVDS level signal is transferred to FPGA acquisition module which translates LVDS to LVTTL image signal, caches the image data by SRAM ping-pong buffer structure and transfers the data to the host computer though high-speed USB2.0 interface after SRAM. RS-232 interface is employed to initialize the camera before image acquisition.

Supported by the Aeronautic Science Fundamental Funds 2010ZC51033 andthe Fundamental Research Funds for the Central Universities

c2011IEEE978-1-4244-8756-1/11/$26.00󰀁

1726

III.CMOSDIGITAL CAMERA

TTL/CMOS signals to LVDS signals in the camera side, converting from LVDS signals to TTL signals in the A.Image Sensor Module

acquisition side, driving and receiving the camera control This system is mainly designed to capture images for signals, asynchronous serial communication between the machine vision system. In order to reduce costs and easily camera and collecting cards.

upgrade to high-performance near-infrared sensitive image sensor, we use a low-cost black and white CMOS image sensor MT9M001. The active imaging pixel array is 1,280H x 1,024V. Progressive scan allows all pixel data to be output at 30 frames per second (fps). MT9M001 features breakthrough low-noise CMOS imaging technology that achieves CCD image quality while maintaining the inherent size, cost, and integration advantages of CMOS.

Imaging module schematic design is shown in Figure 2, mainly by the sensor chip and its peripheral circuits, data and control signal interface. The maximum pixel output clock of MT9M001 is 48MHz. In order to match the USB2.0 interface transfer speed, we design the sensor input clock is 40MHz. The chip output pixel clock PIXCLK is also 40MHz, thus frame rate is up to 25frames/s, which meet the real-time system requirements. MT9M001 is also can be set to windowing or column and row skip mode though configuring the registers of MT9M001 to achieve a higher frame rate. When the chip is powered on, FRAME_VALID and LINE_VALID signals are output on dedicated pins, along with the pixel clock PIXCLK that is synchronous with valid data. After power-on reset Figure 3. Camera Link interface circuit diagram.

MT9M001 works in the default mode. The acquisition and control module set the frame size, exposure, gain and other In this paper, the Camera Link interface uses basic parameters via the serial interface (I2C) bus (SCLK, SDATA) configuration. The signals of CMOS image sensor need to be to enable users to adjust the imaging module according to connected with the FPGA though Camera Link cable as outside light in use.

follows:

1)Image data signals: Pixel data D9~D0, fram effective signal FVAL, line effective signal LVAL and pixel clock PIXCLK. According to Camera Link standard, signals D9~D0, FVAL and LVAL are assigned to 12-bit of the 28-bit parallel data, and pixel clock PIXCLK is assigned to the the clock signal RxCLK/TxCLK of the Camera Link. The driver DS90CR287 and the receiver DS90CR288 of NI company are employed for serial/parallel data converting and TTL/LVDS level transition.

2)Camera control signals: The trigger control signal

Figure 2. Imaging module schematic diagram. TRIGGER of MT9M001, which is allocated to CC1 of the

four control signals. In addition, FPGA outputs the master B.Camera Link Interface Module

clock to MT9M001 to facilitate debugging in design. The Camera Link is a standard data transmission interface master clock is assigned to CC2. For the four control signals, specifically for digital cameras, jointly launched by NI, digital the driver DS90LV047A is used in the acquisition side, and camera suppliers and image acquisition card companies in the receiver DS90LV048A in the camera side.

2000. The basic idea is that the 7 channel parallel data signals

3)Serial communication signals: Serial bus SCLK and are converted into a serial signal to transport, and the conversion needs a pair of driver and receiver. The driver SDATA. As MT9M001 has no universal asynchronous serial converts the 28 CMOS/TTL signals to four pairs of LVDS data port, the controller in the camera side is required which can convert the standard serial control signals to the I2stream. A PLL clock is transferred with the data stream though C protocol the fifth pair of LVDS link. During each cycle of the

signals in compliance wich MT9M001. In this paper, transmission clock, 28-bit input data is sampled and

C8051F310 microcontroller integrated I2C interface is applied

transmitted. Then the receiver converts the data stream back to

to convert the signals. The microcontroller is small size, low 28-bit LVTTL parallel data[2].

price, and provids high flexibility and low cost of optimization.As shown in Figure 3, functions that the Camera Link The two serial signals are drivered and receivered by

interface circuit to be implemented include[3]: converting from

DS90LV019.

20116thIEEEConferenceonIndustrialElectronicsandApplications1727

Inside of the drivers and receivers applied on the Camera Link interface, a PLL is used to generate and recover the clock transmitted via LVDS interfaces. The time width of each serial LVDS data accounts for one seventh of the clock cycle, so the clock jitter will reduce the size of the window used to sample LVDS serial data stream. Therefore we must ensure that the clock into the driver is a low-noise signal without glitches. To abtain low-jitter LVDS clock, we used decoupling capacitors to reduce switching noise which affecsts the performance of the system. Three parallel decoupling capacitors (surface mount multilayer ceramic capacitors) are employed between the power and ground. The capacity values of the three capacitance are 0.1uF, 0.01uF and 0.00luF.

IV.

FPGAACQUISITION AND CONTROL MODULE

The system uses Altera Corporation's Cyclone II series FPGA chip EP2C35F484 as the main controller. It has 33,216 LE units, 105 M4K RAM blocks and 35 multipliers, and other resources, so in addition to the logic and timing control of the whole system, the FPGA acquisition module can process Large number of accumulated multiply-add time-consuming operations such as median filtering through parallel processing approach when the DSP processing board is expanded[4]. In spite of FPGA and its external circuit, the acquisition module also extends the two SDRAM chips and two SRAM chips for FPGA to cache image data.

In Quartus II development environment, FPGA program adopts modularization design by Verilog language. The modules includes: CMOS image acquisition module, SRAM cache control module, USB2.0 interface module, universal asynchronous serial port module and I2C Interface module to initialize the video decoder SAA7113H. The difficulty of FPGA programming is timing design, especially the synchronization across clock domains, so next we focus on timing design of the first three modules.

machine.Figure 5 shows the acquisition process and the transition sequence of the state machine.

Figure 4. Pixel data output timing chart of MT9M001.

Figure 5. The timing sequence process and the state transition of the

acquisition module.

The gray image is 8-bit data while the data width of SRAM and USB interface is 16 bit, so the acquisition module merges the two pixels into 16-bit data for storage and transmission. The post simulation result of the acquisition module is shown in Figure 6.

A.CMOS image acquisition module

Before images acquisition, MT9M001 should be initialized according to the actual needs, so the image acquisition module also includes initialization sub-modules. Camera Link protocol defines the communication between the collection card and the Figure 6. The post simulation of the acquisition module. camera as universal asynchronous serial communication. Therefore FPGA acquisition module just simulates universal

serial port interface to send the command word to conversion B.USB interface module

The USB2.0 protocol has four transmission methods: MCU (UART to I2C). Then MCU writes the command word to

the appropriate registers of MT9M001 by integrated hardware control transfer, interrupt transfer, isochronous transfer and

block transfer. The block transfer mode supports a maximum I2C interface to complete the initialization of the camera.

theoretical transfer rate 52MByte/s, but also ensures the

MT9M001 pixel data output timing is shown in Figure 4. reliability of the transfer data. Therefore the system uses block When FRAME_VALID and LINE_VALID signals are high, transfer mode. The resolution of MT9M001 is 1280 × 1024, the data are sequentially sent to the data bus at the rising edge and the maximum frame rate is 30frame/s. If all the pixel data of pixel clock. Therefore FPGA uses the falling edge of pixel of the image are transferred to PC, the transfer rate of the USB clock PIXCLK, to latch data when the pixel data is valid. interface is not less than 37.5MByte/s (1.25MByte/frame × Acquisition and control timing is implemented with the finite 30frame/s). In addition to protocol information, the actual state machine, and is synchronized by the pixel clock. The effective data transmission rate of the USB2.0 transmission is assignment of the output signal is written in the clock process almost impossible to meet the requirements. This image so that the glitch can be eliminated, although the output time acquisition system is designed primarily for embedded real-will be one clock cycle later than the time of Common state time processing of high-resolution images. The image transfer

172820116thIEEEConferenceonIndustrialElectronicsandApplications

to PC is only for the display of the original image, so there is no need to transfer the full-pixel frames. We adopt row skip and column skip mode to transfer the image, which is 680 * 512 pixels, so the requirement of the effective transfer rate of USB is only 9.96MByte/s which can be easily achieved. We used Cypress's EZ-USB FX2 Series CY7C68013A as the USB2.0 interface chip. Its data transfer rate is up to the maximum bandwidth of USB2.0 because of the unique Figure 8. The timing simulation diagram of the USB interface module.

architecture of FX2. The rate meets the system requirement of the image data transmission bandwidth. FX2 provides three C.SRAM cache control module

available interface modes: Port mode, Slave FIFO mode and programmable GPIF master mode. In this paper, Slave FIFO In the real-time video processing system, in order to mode is used, so the external circuit can read and write the data provide the right channel between the constant speed video buffers of FX2 endpoint 2, 4, 6, 8 like doing on normal streaming and the variable speed DSP image processing, data FIFO[5]. Under FX2 Slave FIFO mode, the connection input and output buffer circuits are required. This circuit between FPGA and FX2 is shown in Figure 7. The setting of generally has three kinds of structure: dual-port RAM structure, IFCLK determines which mode Slave FIFO works in, FIFO structure and ping-pong buffer structure. The first two synchronous or asynchronous mode. In this paper, we adopt structures usually have small storage capacity, so are not synchronous mode. The writing signal SLWR is synchronized suitable for high speed image processing systems. Ping-pong by the rising edge of IFCLK. The collected data are written into SRAM cache structure is use in this paper. Using ping-pong the FIFO and then directly transferred to the host. To take full operation, the collected image data flow is isochronously advantages of the internal FIFO of FX2, four times buffer way allocated into the two data buffers SRAM1 and SRAM2 under is adopted to cache data. In this paper, Endpoint 6 is used to the control of the input data selection unit, which completes the complete the IN operation. Endpoint 6 selected by FIFOADR seamless buffering of the image data.

register is configured to AUTO IN mode by the chip firmware. In order to save I/O pin resources of FPGA, the addresses The mode allows FPGA to send data to the FIFO continuously of the two expanded SRAM shared a set of I/O pins. The without considering how to package and transmit data, because access time of the SRAM chip is 10ns (100MHz), while the CY7C68013A will automatically send the data to the USB host. image acquisition and transfer rate is 15MHz (16 bits), so the In order to achieve maximum data bandwidth, the controller is ping-pong operation of the two SRAM can be achieved by the directly connected to the USB host by CY7C68013A and the chip selection signals.

chip’s CPU is pass out, so the timely and accurate data transmission from the acquisition system to the host computer can be guaranteed.

V.

USBSOFTWARE DESIGN

The software design of USB device development includs: firmware, device drivers and PC applications. Firmware refers to the program running in CPU of the device. It is the core of the whole software design of the USB device. The PC application communicates with the underlying device by driver to achieve the online control for the acquisition system. A.Firmware design

Based on the Firmware programming framework provided by Cypress Company, the firmware design is to add the Figure 7. The connection diagram between CY7C68013 and FPGA.

required function code. The code includes the following two parts:

Figure 8 shows the communication interface timing simulation between FPGA and USB implemented by Verilog 1)USB descriptors:According to the format of the USB language. The input clock CLK of the module is 15MHz, and descriptor sample Dscr.a51, modify the device descriptor, its reverse is used as the interface synchronous clock IFCLK configuration descriptors, interface descriptorss, endpoint from FPGA to CY7C68013A. When the writing signal SLWR descriptors and string descriptors.

is effective, 16-bit data are placed to the data bus FD[15:0] by 2)Initialization function: In the periph.c program of the FPGA at the falling edge of IFCLK, and then at the rising edge firmware framework, add the initialization function TD_Init (). of IFCLK the USB interface chip collects the data and writes In the initialization function, we set the clock, operating mode them to Endpoint 6.

and the transmission flags of the USB chip, and configure Endpoint 6 to AUTO IN mode to achieve the maximize data bandwidth.

B.Driver design

The driver is developed in VC++6.0 development environment with WinDriver device driver kit. We do not need

20116thIEEEConferenceonIndustrialElectronicsandApplications1729

to know kernel programming of the operating system. We just start DriverWizard, detect the USB device connected, read the endpoint and configuration information of the USB device, and finally automatically generate the pipe read-write function code for operating the USB device and .inf file. After reinstalling the USB device driver, we can develop PC applications with the two files (.C and .H files) generated by DriverWizard.

C.PC application design

The PC application is developed by C++ language in VC+ +6.0 development environment. The application interacts with the USB device though the driver. First the application gets a handle to access the device driver by calling the Windows API function CreateFile(), then call DeviceIOControl() function to implement I/O control of the device. As CY7C68013A without ROM, the firmware should be downloaded after power on. Therefore the function of downloading the firmware should be included in the initialization of the application.

VI.

CONCLUSION

The Camera Link standard interface between the camera and the acquisition system makes the system applicable to any Camera Link interface digital camera. In addition, with the reserved DSP communication interface, the image acquisition system can be easily extended image real-time processing functions when FPGA can cooperate with DSP to achieve underlying image processing. By synthesizing the advantages of FPGA and DSP in dealing different algorithms, we can achieve a complete real-time image acquisition and processing system. The image acquisition system designed in this paper therefore has broad application prospects.

REFERENCES

[1]

Chao LI, Yulin ZHANG, and Na Zhao ZHENG. Design of Image Acquisition and Processing Based on FPGA[]. 2009 International Forum on Information Technology and Applications, 2009, pp.113-115. Basler. Specification of the camera link interface standard for digital cameras and frame grabbers [Z]. Ahrensburg,Germany: Basler, 2000. Ning Li, and Junfa Wang. High-speed data acquisition system based on Camera Link [J]. Infrared, 2005, vol. 6, pp. 31-37.

Lei Duan, Mei Li, and Caixia Wang. Real-time image processing platform design based on DSP and FPGA [J]. Experimental Science and Technology, 2008, vol. 10, pp. 52-54, 70.

Xudong Zhou, and Yijun Lan. CMOS array image acquisition and processing system based on USB [J]. Semiconductor Optoelectronics, 2009, vol. 30, pp. 284-285, 290.

[2][3][4]

In this paper, the image data acquisition and lossless transmission of the big area CMOS image sensor is implemented by high-performance FPGA and USB2.0 bus [5]technology. The experiments prove that the system can capture real-time images of 1280 * 1024 pixels at rate of 25frames/s, and display real-time images of 640 * 512 pixels on PC.

173020116thIEEEConferenceonIndustrialElectronicsandApplications