uClinux/VisionKit developer documentation

Version:

0.30

Author:

Martin Strubel

Date:

01/2011

Introduction

The section5 VisionKit is a full blown uClinux development environment for custom high performance network camera applications. It is based on a custom DMA buffer queue driver (v4l2 compatible) for the (E)PPI video port of the Blackfin architecture.

The main components of the VisionKit are:

• ppivideo.h module: Kernel driver module, lean and mean buffer queue
• netpp library: Remote control and support of various sensor registers and properties
• videoserver: Multi thread user application for acquisision, processing, transmission
• remote display application Camasutra: Client (PC) application for remote camera display (video over IP/netpp)

The VisionKit depends on the v4l2 API but makes least use of it when it comes to video device specific configurations. The reason for this is, that the video device is no longer seen as a unit of video generator (optical sensor, etc.) and grabber device, but an independent set of devices with a manifold of configuration options and operation modes. To implement all these options using ioctls would hopelessly bloat the kernel - and is in fact not supposed to be in the kernel at all.

Also, we are not only pulling video using this interface. In fact, most of the industrial applications of the VisionKit actually just pull generic data, as this is the fastest way to get data into the Blackfin/uClinux companion. So you could say, we abuse the video interface. We would probably call it: Keep it as generic as possible!. But we plead guilty for keeping the legacy, video specific naming of most of the functions.

Therefore the strategy is to keep most of the control in userspace and use the netpp library for all (remote) device configuration. This makes adding support for new sensor devices easy:

1. Write a new XML file containing the register maps of your sensor
2. Add code on how to access the sensor registers (i2c, etc) into the sensor library
3. Register sensor ID and detection method
4. Import XML device description from above into the videoserver main XML file
5. Compile and enjoy.

The Video server

The video server consists of mainly three threads:

1. acquire: Pull next image frame delivered by the video device (sensor) off the kernel buffer queue
2. process: Process the frame
3. deliver: Transmit the processed result (image or data) to a client

Since all of these actions require some time to complete, the most performance gain is in a multi threaded application where no waiting process is blocking the entire processing pipeline. In order to not lose frames on a blocking transfer (TCP hang) or general system overload, FIFOs are implemented on driver side and on user space side.

This delivers a very high guarantee that frames or scanlines do not get lost under normal circumstances, a latency for the incoming data to be processed can be guaranteed using the RealTime extension of the PPIvideo driver.

However, frames do get lost, if a process or transmission fails, or if the processing just takes too much CPU time such that the FIFO overflows. The user will have to implement his own failure handlers to deal with these exceptions. See more about possible errors in Potential failures and troubleshooting.

The VisionKit supports various standard processing methods, see Processing threads.

Acquisition Methods

Fast data acquisition is mostly based on the typical DMA (direct memory access) buffer queue scheme. That means, the entity (data client) expecting data allocates a buffer where data can be streamed into from a peripheral interface. Once the buffer is filled, a notification "BUFFER FULL" is sent to the data client. Meanwhile, more data might arrive, so we have to provide one or a few more buffers where the data can go, while processing is happening on the client side. There are various terms for this: FIFO, buffer queue, ring buffer, Pingpong buffer (if we have only two).

The v4l2 API basically supports two modes of acquisition using buffer queues:

• MMAP: Memory mapped, the kernel owns (allocates and deallocates) the buffer space, and controls its location/memory options
• USERPTR: User pointer, the user space owns the buffers and passes them on to the kernel.

The standard method of the VisionKit is the USERPTR method which is required by the video FIFO buffer. The advantage of this method is, that the number of buffers is only limited by available memory, thus, a huge FIFO can be constructed. Since caching is always enabled for these buffers (and the user has no control over that), a certain overhead for invalidating the cache is introduced by the kernel driver. The performance loss is however compensated by caching on the user space access side (since imaging operations can occur on the cached SDRAM image chunks. The MMAP method again allows more control, but the number of buffers is limited by the kernel. Starting from Kernel v.2.6.34, the MMAP method is no longer supported.

Writing your own Video server

To implement your custom camera subsystem, you will have to implement a video generator source file, preferrably using a skeleton like "vidgen_*.c", see Video generator interface for API details. Basically, you have to follow these steps:

1. Implement video_process() function for your target. This may again call several processing handlers depending on the preprocessing mode.
2. Implement vidgen_init() and vidgen_exit()
3. Define and instanciate global configuration structures needed for video generator specific netpp configuration
4. Compile and debug :-)

Potential failures and troubleshooting

Even though striving after minimal image loss using the VisionKit framework, there are various errors than can occur on various system levels:

• Hardware framing error: A unexpected/erroneus framing signal was received from the video generator
• Hardware FIFO overrun: System bandwidth exceeded. This occurs when the pixel clock is too high with respect to general system load so that the DMA stream can not complete. This is a difficult situation, either reduce the pixel clock or trace your entire data processing flow for avoidable memory to memory copying.
• Kernel FIFO overrun: This can happen when the user process is not popping the buffers off the queue soon after a fresh frame has arrived. This is normally a deficiency of the user space software, but with line scan applications it can occur in a non realtime environment.

• videoserver FIFO overrun: This can occur under two conditions:

  • Your processing is too slow with respect to the incoming data
  • Your transmission is too slow or stuck with respect to the incoming and processed data

  In order to solve these problems, you will get in touch in depth with the FIFO code. See The video FIFO for more details.

Remote Display software

The remote display software 'Camasutra' is a front end software, specifically built for networked cameras, but can be used for all kinds of remote pixel data visualization. It is based on the netpp library and contains a netpp master control utility with XML GUI interface as well as a tiny display server.

Typically, a network camera is opened within a selection box (like a file selection dialog), scanned for available properties and can then be configured by the user. When the video stream is enabled, the camera makes a connection to the display server and displays images according to the functionality implemented in the videoserver running on the camera.

For more documentation, please refer to the Camasutra help menu.