Create your own real-time visual effects

Document options Document options requiring JavaScript are not displayed Sample code Rate this page Help us improve this content

Level: Intermediate

Nathan Harrington (harrington.nathan@gmail.com), Programmer, IBM

17 Jan 2006

Use EffecTV and Simple DirectMedia Layer (SDL) to create your own real-time visual effects on live video. Learn how to integrate geometric primitives, bitmap image loading, and simple motion tracking to create your own games, leading-edge user interfaces, or immersive environments. Explore the EffecTV and SDL architectures, and learn how to harness the power of open source video processing on Linux®.

Real-time video effects processing is becoming more common throughout major sporting events. NASCAR, the NFL, professional hockey, and other sports apply real-time video effects, such as the colored first-down line in football, based on precise geospatial data. Advanced user interfaces and immersive environments employ specialized input hardware to specify coordinates in a 3-D world with six degrees of freedom. These technologies all require investment in expensive hardware and custom data acquisition packages. You can build your own system for real-time video effects processing based on simple motion tracking and existing open source software.

EffecTV is a fantastic application written by Fukuchi Kentaro that gives you access to the raw pixel data from a video capture device. You can use its existing open source framework to modify more than two dozen effects, or add your own. With the integration of the SDL libraries, the EffecTV framework makes it easy to add your own effects with more abstracted interfaces, making them easier to create and integrate with existing code.

This article will cover the installation and setup of EffecTV, the integration with a geometrical primitives API, and the application of a simple motion-tracking algorithm on a computer running Fedora Core V3. You can use your own distribution of Linux, as the software components are not specific to Fedora distributions. When you complete the installation, you can use these basic concepts and architecture descriptions to build your own immersive environments, games, or next-generation computing interfaces.

Requirements

Hardware

Aside from a monitor, you need three pieces of hardware to use EffecTV:

1. The first requirement is a speedy main processor. The code in this article was developed and run on a 500-MHz Intel® Pentium® III, which is acceptable for full-motion video in a 320x240-pixel window. Running the same code on a 2-GHz (or faster) Pentium 4 produces full-motion video at 800x600. The lesson here: Faster CPUs yield better results.
2. The video capture card is the second vital piece needed for real-time video processing. Hauppauge WinTV cards provide excellent performance for the price and have great support in the Linux community. I used a WinTV-GO PCI model that I picked up for about $80 at the local electronics store. This is the least expensive capture card from Hauppauge, and hits the sweet spot between price and performance for this application. You can find more expensive capture cards and cards that won't work at all with Linux. What is needed at a bare minimum to make sure your card works is a Brooktree/Conexant Bt8x8 series chipset in the video capture card. See Resources to find a capture card that will work with your computer.
3. The final piece of the puzzle is a camera. I used a consumer-grade camcorder that sent its output to the video capture card using a composite (standard) video cable. You can use a USB cam, but it will almost certainly not provide usable results. In my experience, the full-motion video from a USB cam is blurry at best. However, no matter which camera you have, if the video4linux drivers can read from it, it will almost certainly work for EffecTV, and for the purposes of this article.

Software

Your kernel must include the video4linux modules. Any V2.6-level kernel will include these modules by default, including the newer Fedora distributions (Core V2 and higher). If you are using an older kernel, you have to install the video4linux software before your video capture will work. You will also need to have the SDL runtime and development libraries installed. Netwide Assembler (NASM) support (an assembly language compiler) is also required for EffecTV to process its included effects correctly.

Fear not: No assembly programming is required to understand this article.

Back to top

Installation

To begin the process of EffecTV installation, complete the steps below. The locations of each component are listed in Resources.

1. Install NASM. For Fedora Core V3, you may find it at fc3/disc2/Fedora/RPMS/nasm-0.98.38-3.i386.rpm.
2. Download and untar EffecTV V3.10 from SourceForge.
3. Enter the effectv-3.10 directory and type make.
4. Plug in your camera.

If you have installed all of your software prerequisites, the make should proceed without error. Now you are ready to see the amazing things EffecTV can do. In the current directory, type ./effectv -channel 1 and press the up arrow key to cycle through the effects. -channel 1 in this context tells your video capture card to read from line in. Your options may vary depending on your hardware.

Once you have EffecTV up and running, let's add the last package we need. SDL_gfxPrimitives from Andreas Schiffler is an excellent API for drawing alpha-blended circles and lines (among other things). We will use this code to abstract away the physical drawing of lines, circles, and bitmaps. The included EffecTV effects manipulate the pixels directly, but we will use existing code and a modular approach to reuse what others have done.

5. Download and install SDL_gfx and SDL_gfx-devel RPMS from ferzkopp.net (see Resources).

Back to top

Creation of a simple effect

Architecture modification

Our approach of reusing existing code, instead of taking on the challenge of direct pixel modification, will require a substantial change to the EffecTV architecture. We will eliminate all but one effect from the program and move the modification of the pixels displayed to the user into the screen.c file.

Begin by modifying the program to compile only the component we need. In this case, we will use the existing default effect that doesn't do anything (yet), which is called dumbTV.

1. In main.c, edit the effectRegisterFunc function and remove all the *Register lines except dumbRegister.
2. In effects/Makefile, edit EFFECTS = <effect names > to be just EFFECTS = $(DUMB).
3. In effects/Makefile, remove all the object definition entries except DUMB = dumb.o.
4. In effects/effects.h, remove all the entries except: extern effectRegisterFunc dumbRegister.

Blue circle effect

For our first simple effect, we will draw an alpha-blended (semitransparent) circle at a fixed position over the video stream. The first step is to add the drawing functions to the screen.c file:

1. In screen.c, add '#include "SDL_gfxPrimitives.h"' to the definitions.
2. Append the following function to the bottom of screen.c:

    /* add a blue circle using primitives */
    void screen_draw_blue_circle( int x, int y ){
      // screen is the SDL surface that will be rendered with a SDL_Flip operation
      filledCircleRGBA(screen, x, y, 30, 100,155,255, 150);
    }// screen_draw_circle

3. In screen.h, add the function definition:

    void screen_draw_blue_circle( int x, int y );
    

Modify the Makefile to properly compile our included GFX Primitives.

4. In Makefile, change:

  LIBS = v4lutils/libv4lutils.a -lm `sdl-config --libs` $(LIBS.extra)   

  LIBS = v4lutils/libv4lutils.a -lm `sdl-config --libs` \
  $(LIBS.extra) -L/usr/lib -lSDL -lpthread -lSDL_gfx -lSDL_image

Add the code to draw a blue circle on the image at position 50x, 50y. Adding this in the main.c file will allow for easier changing of the location of the circle.

In main.c, in the StartTV function, add screen_draw_blue_circle( 50, 50 ); just above screen_update();.

Type make and run ./effectv -channel 1, and you will see a plain video image with an alpha-blended blue circle at position 50,50. Later, we will see how to load an image and overlay it on the video stream. This is useful for adding borders, captions, or other types of static overlays to your video.

Back to top

Motion tracking

The motion-tracking algorithm implemented for this project is designed for simplicity. See Resources for starting points of applied applications for facial, human form, geometric shape, or other types of advanced motion tracking and content recognition.

In this case, we will simply determine the coordinates of a cell of pixels closest to the top of the captured image and that changed from the previous frame. Due to various issues with digital cameras involving noise, pixels will rarely maintain their exact color value between frames. Grab a magnifying glass and train your camera on a white background to see this phenomenon. An appropriate motion threshold must be selected to ignore color changes due to electronic noise. Lower the threshold if your camera has a higher color-constancy per pixel. Raise the threshold if you are using a USB cam or are in a visually noisy environment.

In the existing architecture we modified, the dumb.c draw function copies the source pixel memory area into the destination memory area. We will use this function to check every portion of the source area for motion.

1. In effects/dumb.c, add the following to the function definition section:

  /* for motion detection */
  static int get_last_motion_x(void);  // return the last x location of movement
  static int get_last_motion_y(void);  // return the last y location of movement

  #define MOTION_THRESHOLD  3000000
  int last_motion_x = 0;               // last coordinate of motion x plane
  int last_motion_y = 0;               // last coordinate of motion y plane

  // record previous frame pixels to check for motion
  int previous[DEFAULT_VIDEO_HEIGHT][DEFAULT_VIDEO_WIDTH]; 

  // process current frame of pixels
  int motion[DEFAULT_VIDEO_HEIGHT][DEFAULT_VIDEO_WIDTH];

  // process a 5x5 chunk of pixels for motion
  // Expand the chunk size to speed up the motion processing at the cost of 
  // placement precision of where the motion is in the frame.
  // Decrease the chunk size for a precise measurement of where the motion was
  // at the cost of increased processing time per frame.
  int x_chunk = 5;  
  int y_chunk = 5;

2. In effects/dumb.c, add the following functions to retrieve the motion coordinates:

  int get_last_motion_x(){ 
    int newx = last_motion_x;
    last_motion_x = 0;
    return(newx);
  }
  int get_last_motion_y(){
    int newy = last_motion_y;
    last_motion_y = 0;
            
    return(newy);
  }         

3. In effects/dumb.c, modify the dumbRegister function to include the following:

    entry->get_last_motion_x = get_last_motion_x;
    entry->get_last_motion_y = get_last_motion_y;

4. In EffecTV.h, modify the _effect struct to include the following:

    int (*get_last_motion_x)(void);
    int (*get_last_motion_y)(void);

Now add our new motion detection code by commenting out the existing draw function and adding the following code into effects/dumb.c:

/* draw with highest point on screen in motion */
static int draw(RGB32 *src, RGB32 *dest){

  int highest_y = video_width;  // highest point Y value of motion
  int highest_x = video_height; // highest point X value of motion
  int x=0;  // loop variables
  int y;
  int i;
  int temp_cell = 0;

  // for each row in the frame
  for( i=0; i < video_height; i+= y_chunk ){

    // for each column in the frame
    for( x=0; x < video_width; x++ ){

      // create a 5x5 chunk of data that is the sum of colors for all 25 pixels
      for( y=i; y < (i+y_chunk); y++ ){

        RGB32 pixel_value;                     // temporary pixel color value
        pixel_value = *(src+x+y*video_width);  // current frame pixel
        *(dest+x+y*video_width) = pixel_value; // update destination surface
        temp_cell += pixel_value;              // increment cell color sum

      }//for y

      // if 5x5 cell has been processed
      if( x % x_chunk == 0 ){

        // compute average color value of 5x5 cell
        temp_cell = temp_cell / (x_chunk * y_chunk);
        motion[i][x] = temp_cell;

        // if the difference between previous frame and current frame is
        // greater than the threshold, there was motion
        if( abs(previous[i][x] - motion[i][x]) > MOTION_THRESHOLD ){

          // if the motion was higher up than any previous recorded motion
          // update the current highest portion
          if( i < highest_y ){
            highest_y = i;
            highest_x = x;
          }// highest x part

        }//motion detect

        // set pixel in previous for the next frame, regardless if there was motion
        previous[i][x] = motion[i][x];
        temp_cell = 0;

      }// if the chunk is ready to be processed

    }// for x

  }// for i
}// end draw function with motion detection

Now we have a motion-tracking function that is run for every frame of video that comes in. Let's retrieve the coordinates and draw a blue circle where there is motion. In main.c, update the StartTV function by replacing the screen_draw_blue_circle( 50,50 ) function call with:

    int motion_x = currentEffect->get_last_motion_x();
    int motion_y = currentEffect->get_last_motion_y();
    screen_draw_blue_circle( motion_x, motion_y);

Type make and run ./effectv -channel 1, and you will see your video image with a blue circle drawn wherever the motion coordinates are. Go ahead and shake your hand. It won't fall off.

Back to top

Interactive logic development

Current framework possibilities

You now have a framework and a simple example that allows you to read in a frame of video, process it on the pixel level, and draw over the video as an SDL Surface. Anything you can do with an SDL Surface you can now do with your motion-tracked blue dot example. Fire up those 3-D SDL libraries, map the drawing surface onto a plane, and you can have full-motion video as an aspect of your 3-D world. Put in some special gesture recognition, and you have the beginnings of fun games like those available for PlayStation2. Track colorful objects (like marker tips) through your video interface, and integrate the open source Sphinx library from Carnegie Mellon University (see Resources) to cast spells based on your marker (wand) position and the word you say. Anything you can do with SDL you can do with this framework. Read on for one example of how to create an immersive environment.

Why rainbows and butterflies?

My initial ideas focused on laser beam-type duels between friends in front of the camera. I planned on taking a snapshot of the background before the players entered the field of view. Small irregular-shaped holes extracted from this background would then overlay the player's image when the opponent's laser beam struck their targets. I had planned a system of simple gestures, such as movement in quadrant 1 for shield, movement in quadrant 2 for laser beam fire, as a method of control, complete with particle effects -- pulsing laser shields and sound effects taken from previous work with SDL. I encourage you to follow similar lines of thought to make some exciting games that take advantage of the unique capabilities of a video interface.

At the same time, I had the desire to create something "magical" for my 3-year-old daughter that was related to my work with SDL. Laser beams and holes in the torso didn't quite cut it. I decided to make an interface that would allow her to simply move in front of the camera, and she could make rainbows and butterflies surround herself.

Back to top

Rainbows

To begin, start with a detailed description of the effect:

• If there was motion in the current frame, add a rainbow piece between the end of previous rainbow piece and the current motion position.
• Rainbow pieces have a lifetime, so the gradually fade out to be replaced by new rainbow pieces.

Once you've got a good idea of what you want to see, adding the code to the existing architecture is easy.

1. Drop in rainbow.h and rainbow.c from the files below (or download the whole package -- see Resources). The contents of these files are self-explanatory, and simply serve to animate the fade-out portion of the rainbow and manage the variables to make sure new components are drawn correctly.
2. Update screen.c with #include "rainbow.h".
3. Append screen.c with the following function:

void screen_draw_rainbow(){

  int i;
  // for each rainbow piece, if it's on the screen, get its coordinates
  // draw all of the bars of color, and apply the fade
  for(i=0;i < =RAINBOW_PIECES;i++){
    int end_x = get_rainbow_end_x(i);
    // if end_x = -200, then the bar is in unused state, and will not be drawn
    if(   end_x != -200 ){
      int start_x = get_rainbow_start_x(i);
      int start_y = get_rainbow_start_y(i);
      int end_y = get_rainbow_end_y(i);
      int c;
      // for each rainbow bar, 
      for( c=0;c <=RAINBOW_SIZE;c++ ){
        int fade = get_rainbow_life(i);
        fade = fade *4;
        int r;
        int g;
        int b;
        if( c<4 ){
          r=255; g=102; b=0;    // red
        }else if( c < 8 ){
          r= 255; g=204; b=0;   // orange
        }else if( c < 12 ){
          r=255; g = 255; b=0;  // yellow
        }else if( c < 16 ){
          r=51; g=255; b=0;     // green
        }else if( c < 20 ){
          r=0; g=0; b= 255;     // blue
        }else{
          r=153; g=0; b=255;    // violet
        }
        lineRGBA(screen, start_x, start_y+c, end_x, end_y+c, r,g,b, 50+fade);
      }//for chunkiness

    }//if not an empty draw
  }//for each rainbow piece
}// screen_draw_rainbow

4. In main.c, add #include "rainbow.h".
5. In main.c, replace:

   screen_draw_blue_circle( motion_x, motion_y); screen_update();

   
   with:

   screen_draw_rainbow(); screen_update(); if( motion_y != 240 ){ add_rainbow_piece(motion_x, motion_y); }// if not off the screen animate_rainbow_pieces();

   
   
6. In Makefile, append rainbow.o to COREOBJS = main.o screen.o video.o frequencies.o palette.o.

Type make and run ./effectv -channel 1, and you will see a video image with rainbow following the motion.

Back to top

Butterflies

To begin, start with a detailed description of the effect: Randomly have butterflies appear out of the rainbow and float up the screen.

Again, once you've got a good idea of what you want to see, adding the code to the existing architecture is easy.

1. Drop in butterfly.h, butterfly.c and the images directory from the files below (or download the whole package -- see Resources). The contents of these files are self-explanatory and simply serve to animate the "fluttering" of the butterflies (by cycling through different size versions of the same image), and moving the butterflies up the screen by decrementing their Y coordinate.
2. Update screen.c with #include "butterfly.h".
3. In screen.c, add #include sdl/sdl_image.h.
4. The butterflies will be drawn to the SDL surface as transparent bitmaps. I found it easiest to do this in the screen.c file.
5. In screen.c, append the load_butterfly_anims function:

void load_butterfly_anims(){

  int i=0;
  for( i=0;i <=11;i++ ){
    char filename[50] ;
    sprintf(filename, "images/bf%d.png", i);
    anims_0[i] = IMG_Load(filename);

    sprintf(filename, "images/bf%d.2.png", i);
    anims_1[i] = IMG_Load(filename);

    sprintf(filename, "images/bf%d.4.png", i);
    anims_2[i] = IMG_Load(filename);

    sprintf(filename, "images/bf%d.6.png", i);
    anims_3[i] = IMG_Load(filename);
  }// for 12 butterfly superfly

}// load_butterfly_anims

6. In screen.c, append the draw_butterflies function:

// draw the butterflies
void screen_draw_butterflies(){
  int i=0;
  for(i=0; i <=MAX_BUTTERFLIES; i++){
 
    int bx = get_butterfly_x(i);
    if( bx != -200 ){
      int by = get_butterfly_y(i);
      int bf = get_butterfly_frame(i);
      screen_butterfly_animate( bx, by, i, bf );
    }

  }//for i butterflies
}//animate_butterflies

7. In screen.c, append the draw_butterflies function:

void screen_butterfly_draw( int x, int y, int butterfly_number, int frame ){

  // draw the appropriate bitmap for animation stage
  SDL_Rect src, dest;
  src.x = 0;
  src.y = 0;
  dest.x = x;
  dest.y = y;
  if( frame == 0 ){
    src.w = anims_0[butterfly_number]->w;
    src.h = anims_0[butterfly_number]->h;
    dest.w = anims_0[butterfly_number]->w;
    dest.h = anims_0[butterfly_number]->h;
    SDL_BlitSurface(anims_0[butterfly_number], &src, screen, &dest);
  }else if(  frame == 1 ){
    src.w = anims_1[butterfly_number]->w;
    src.h = anims_1[butterfly_number]->h;
    dest.w = anims_1[butterfly_number]->w;
    dest.h = anims_1[butterfly_number]->h;
    SDL_BlitSurface(anims_1[butterfly_number], &src, screen, &dest);
  }else if(  frame == 2 ){
    src.w = anims_2[butterfly_number]->w;
    src.h = anims_2[butterfly_number]->h;
    dest.w = anims_2[butterfly_number]->w;
    dest.h = anims_2[butterfly_number]->h;
    SDL_BlitSurface(anims_2[butterfly_number], &src, screen, &dest);
  }else if(  frame == 3 ){
    src.w = anims_3[butterfly_number]->w;
    src.h = anims_3[butterfly_number]->h;
    dest.w = anims_3[butterfly_number]->w;
    dest.h = anims_3[butterfly_number]->h;
    SDL_BlitSurface(anims_3[butterfly_number], &src, screen, &dest);
  }// compress animation
}// screen_butterfly_draw

8. In screen.h, add the following function definitions:

void screen_draw_butterfly_bitmap( int x, int y, int butterfly_number);
void screen_butterfly_animate( int x, int y, int butterfly_number, int frame);
void screen_draw_butterflies(void);
void load_butterfly_anims(void);

9. In main.c, add butterfly.h to the function definitions.
10. In Makefile, append butterfly.o to COREOBJS = main.o screen.o video.o frequencies.o palette.o rainbow.o.

Type make and run ./effectv -channel 1, and you will see your video image overlaid with rainbows and butterflies following the motion on the screen.

Back to top

Conclusion

Capabilities

With your current framework and example effects, you are on your way to creating exciting visual effects and interfaces. I encourage you to follow the UNIX® way, and plug existing code and tools into the current architecture. Find the most elegant way to integrate your ideas, and you will succeed in making them a reality.

Further work

The resources below cover many example projects that cover the range of interfaces. From immersive 3-D sculpting environments to crowd interaction and simple games. You can find starting points for tracking objects in video, pattern recognition, and ideas for games. I hope you can use what you have learned in this article and from the resources below to create your own visual effects.

Back to top

Download

Description	Name	Size	Download method
Sample code for this article	os-viseffects.zip	278KB	HTTP|Download Director

	Information about download methods

Resources

• Discover the capabilities of Simple DirectMedia Layer (SDL), download source code, and see examples, and read SDL documentation.
  
  
• Read about EffecTV download the source, and see screenshots of effects.
  
  
• Learn more about the SDL GFX Primitives library and more by Andreas Schiffler.
  
  
• A fantastic resource for all things is video on Linux.
  
  
• Read the article from Ars Technica about video capture. It covers hardware compatibility, options, and tradeoffs between Hauppauge and no-name brands.
  
  
• Read the article from LinuxQuestions.org on choosing a Linux-compatible video capture card.
  
  
• Check out some examples of video-related interactive audience participation projects.
  
  
• Jarrell Pair covers visual effects, augmented reality, and immersive environments.
  
  
• PiDiP real-time video effects processing software is available for Linux distributions and Mac OS X.
  
  
• Gephex is a modular video jockey software. Download software for various Linux distributions.
  
  
• frei0r is a minimalistic plug-in API for video effects.
  
  
• This Shockwave Flash real-time video processing game provides an excellent example of what is possible with a USB cam.
  
  
• 3-D surface drawing allows artists and designers to create 3-D shapes comfortably and naturally with hand motions and physical tools.
  
  
• Learn more about the Augmented Reality user interface project.
  
  
• Carnegie Mellon University is dedicated to speech technology research, development, and deployment. The Speech at CMU Web site contains links to Sphinx and other projects.
  
  
• Stay current with developerWorks technical events and webcasts.
  
  
• Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
  
  

• Download a free trial version of WebSphere Application Server V6.0.
  
  
• Build your next development project with IBM trial software, available for download directly from developerWorks.
  
  
• Innovate your next open source development project with IBM trial software, available for download or on DVD.
  
  

• KernelNewbies.org has lots of resources for people who are new to hacking the kernel: FAQ, IRC channel, mailing list, and wiki.
  
  
• Get involved in the developerWorks community by participating in developerWorks blogs.

About the author

		Nathan Harrington is a programmer at IBM currently working with Linux and resource-locating technologies.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Back to top