Visualizing time-dependent data with distortion portals

Document options Document options requiring JavaScript are not displayed Sample code Rate this page Help us improve this content

Level: Introductory

Nathan Harrington (harrington.nathan@gmail.com), Programmer, IBM　

10 Jun 2008

Create an SDL-enabled application that allows you to create distortion portals in sequential image frames to explore the relationship of data sets through time.

There are many ways to visualize data as snapshots, or sequential images showing trending and time progressions. Few options exist for exploring the relationships between data sets through time with an interactive interface. This article demonstrates code and techniques to create what I call "animated distortion portals" in the data to provide time-dependent visualizations of various parts of the image. Additionally, certain aspects of the code are presented that allow for effective visualization on slower-computing platforms without sacrificing usefulness. The code presented here will allow new insights into application-flow models and usage patterns by exploring various data sets and how they move through time.

What is SDL? What is blitting? In this article, Simple DirectMedia Layer (SDL) provides easy cross-platform access to video modes and input devices. A certain level of familiarity is expected with SDL in general, as well as double-buffering, blitting and other 2-D image-processing techniques. Wikipedia defines bit blit (bitblt, blitting, etc.) as a computer graphics operation in which several bitmap patterns are combined into one using a "raster operator." The name derives from the BitBLT machine instruction for the Xerox Alto computer, standing for "Bit Block Transfer."

Requirements

Hardware

Modern fast hardware is required for effective utilization of the visualization algorithms in this article. Although developed on an IBM® ThinkPad running at 1.8 GHz, a much-faster processor and associated hardware data channels are recommended for per-pixel correct visualizations. To retain usefulness, the algorithms presented allow for a "chunk-drawing" effect, which retains much of the usefulness of the temporal distortions while still being able to run quickly on slower hardware. Fast hardware is still required, and a high-powered video card is recommended.

Software

The code presented here is capable of running on a variety of operating systems. We used Linux®. To follow along, you need a modern Linux distribution that includes a development environment capable of compiling C programs. You need the SDL and SDL_image libraries. And you need mplayer to extract video frames into a format usable by the application we develop here (see Resources).

Back to top

Developing a data set

Consult the demonstration video (see Resources) for some ideas on data sets for with temporal visualizations. Keep in mind that the choice to have a static or moving background image can create many issues with the clarity of the visualization. Static backgrounds are recommended for those starting with these types of visualizations, as their image progressions are easier to integrate with traditional frameworks.

Possible sources of data include video of natural phenomena, simulations, or user-built sequences of images. Consider the files in the included temporal.images/ directory, which consist of radar-like precipitation-reflectivity images created during a rainstorm.

Back to top

Basic program structure

All the functionality of the demonstration program is contained in one file called temporalVisualizer.c. Follow along with this article to build your own copy or download the complete source code. Listing 1 shows the beginning of the program.

                
//temporalVisualizer.c - display temporal distortion portals in video
#include <stdio.h>
#include <math.h>
#include "SDL.h"
#include "SDL_image.h"

#define WIDTH  1024     // screen dimensions
#define HEIGHT 768
#define MAX_IMAGES 110  // number of frames to read from disk
#define PORTAL_DIA 50   // center distortion portal size

// use 1 for per pixel correctness, multiples of ten for faster 'chunking'
int chunkSize=1;

int pixels[WIDTH][HEIGHT];      // frame number at each pixel coordinate
int animateGrid[WIDTH][HEIGHT]; // record animation position at each pixel

SDL_Surface *screen;            // surface to display to user
SDL_Event event;                // keyboard, mouse event handling
SDL_Surface* immutableImage;    // base image to overwrite each frame
SDL_Surface* frame[MAX_IMAGES]; // array of frames to animate
SDL_Rect baseRect;              // immutable image clipping rect
SDL_Rect src;                   // current frame clipping rect

int mouseX = 0;
int mouseY = 0;
int mouseIsDown = 0;
int stopMainLoop = 0;

The variables defined will be used mostly in the various functions within the program. Consider the following function declarations.

                
void initScreen();
void loadImages();
void checkEvents();
void resetPixels();
void circleSetPixels(float, float, float);   
void lineSet(int, int, int, int, int);
void topCircle(float, float, int, float);
void bottomCircle(float, float, int, float);
void drawPixels();
void animateFrames();

These definitions illustrate the overall logic flow of the program. First, the screen is set up and images are loaded. Then, each pass of the main loop checks for events, resets the draw state, draws the circular "portal" windows, then animates their distortion back to the current time frame.

Back to top

SDL configuration and loading images from disk

SDL provides a wide array of configuration options for configuring video modes and display surfaces. Listing 3 shows the display options set by the initScreen function.

                
void initScreen()
{
  const SDL_VideoInfo *info;
  Uint8  video_bpp;
  Uint32 videoflags;

  if ( SDL_Init(SDL_INIT_VIDEO) < 0 )
  {
    fprintf(stderr, "Problem initializing SDL: %s\n",SDL_GetError());
    exit(1);
  }
  atexit(SDL_Quit);

  info = SDL_GetVideoInfo();
  video_bpp = info->vfmt->BitsPerPixel;

  // store surfaces in hardware video memory where possible, and enable 
  // double buffering on the displayable surface
  videoflags = SDL_HWSURFACE | SDL_RESIZABLE | SDL_DOUBLEBUF;

  if ( (screen=SDL_SetVideoMode(WIDTH,HEIGHT,video_bpp,videoflags)) == NULL )
  {
    fprintf(stderr, "Video mode error: %s\n",SDL_GetError());
    exit(2);
  }
  SDL_WM_SetCaption("temporalVisualization","temporalVisualization");

}//initScreen

After defining variables and checking to see if the video interface is available, a display surface is initialized that uses hardware video memory. This will help reduce some of the speed issues associated with blitting multiple surfaces in a traditional 2-D context. Note that the SDL library has many options for video interfaces that are not covered here. If you have difficulty loading this basic configuration, consult the SDL documentation or one of the many available tutorials for assistance (see Resources).

With the display surface set up, the next step is to load the images from disk. Listing 4 shows the loadImages function.

                
void loadImages()
{
  int i=0;
  char filename[100];

  for( i=1; i<=MAX_IMAGES; i++ )
  {
    sprintf(filename,"temporal.images/%d.jpg",i);

    SDL_Surface *tempSurface;
    if( (tempSurface=IMG_Load(filename))==NULL )
    {
      fprintf(stderr, "Image load error for file %s\n",filename);
      exit(3);
    }

    frame[i] = SDL_ConvertSurface(tempSurface, screen->format, SDL_HWSURFACE);

    fprintf(stdout,"loaded image %d\n",i);
  }//for i

  sprintf(filename,"temporal.images/%d.jpg",1);
  immutableImage = SDL_ConvertSurface( IMG_Load(filename),
                    screen->format, SDL_HWSURFACE);

  baseRect.x = 1;
  baseRect.y = 1;
  baseRect.w = immutableImage->w;
  baseRect.h = immutableImage->h;

}//loadImages

Various image formats will use different color spaces from the display surface. After using the IMG_Load function to load each video frame into a temporary surface, the SDL_ConvertSurface function is used to make sure all the surfaces used will have the same color format. This pre-conversion is critical for maintaining color state through transitions of different frames as their portions are blitted to the buffer and, eventually, the screen.

Next up, after image loading and screen configuration, is tracking the application events.

Back to top

Handling application events

The interface strategy is to have a "distortion portal" opened at the current mouse coordinates when the mouse button is down. In addition to video-interface libraries, SDL provides a framework to do much of the event-handling work, as shown below.

                
void checkEvents()
{
  while ( SDL_PollEvent(&event) )
  {
    if( event.type == SDL_KEYDOWN && event.key.keysym.sym == SDLK_ESCAPE)
    {
      stopMainLoop = 1;
    }//if escape pressed

    if( event.type == SDL_MOUSEBUTTONUP   ){ mouseIsDown = 0; }
    if( event.type == SDL_MOUSEBUTTONDOWN ){ mouseIsDown = 1; }

    if( event.type == SDL_MOUSEMOTION )
    {
      mouseX = event.motion.x;
      mouseY = event.motion.y;

      //snap mouse to grid if not per-pixel mode
      if( chunkSize > 1 )
      {
        mouseX = (int) mouseX/chunkSize;
        mouseX *= chunkSize;
        mouseY = (int) mouseY/chunkSize;
        mouseY *= chunkSize;
      }//if not trying high precision
    }//mouse motion

  }//while pollevent
}//checkEvents

Consult the "Speed considerations" section for the ideas behind transforming mouse coordinates into a grid. Before the image components are copied to the drawable surface, their locations are recorded to ensure that they are drawn in the correct order. Listing 6 shows the resetPixels function, which re-initializes the pixels' 2-D array before each drawing pass.

                
void resetPixels()
{
  int x,y =0;
  for( x=0; x<=WIDTH; x+=chunkSize )
  {
    for( y=0; y<=HEIGHT; y+=chunkSize )
    {
      pixels[x][y]= 1;
    }//y
  }//x
}//resetPixels

Back to top

Setting the pixels blitting frame

Drawing frames of video at various positions in time that are potentially overlapping requires special consideration in the pre-rendering steps. Additionally, distorting the frames correctly as they move toward the edge of the distortion portal is implemented as a multipass set of overlapping circles. To keep these image components drawn in the correct order, each pixel's — or chunk's (more on that below) — frame is determined prior to the actual blitting steps.

circleSetPixels, topCircle, and bottomCircle functions

For a given set of coordinates and an ending frame, the circleSetPixels function will draw a series of overlapping circles. Each circle will have the appropriate frame number to show a graduated distortion zone between the start and end frames. Listing 7 shows the circleSetPixels function.

                
void circleSetPixels(float startx, float starty, float inFrame)
{
  float frameInc = (inFrame-1) / PORTAL_DIA;
  float i=0;
  float currFrame=1;

  for( i=(PORTAL_DIA*2); i>=PORTAL_DIA; i-- )
  {
    int xshift = (PORTAL_DIA-i) * 2;
    int yshift = PORTAL_DIA-i;

    topCircle(    startx+xshift, starty+yshift, i, currFrame );
    bottomCircle( startx+xshift, starty+yshift, i, currFrame );
    currFrame += frameInc;
  }//for i
}//circleSetPixels

Circles are drawn in top and bottom halves, from largest diameter to smallest. Each half-circle is offset by computed x and y coordinates in order to ensure that they are concentric. As each circle is drawn, the currFrame variable moves to the next frame between the base image (defined in the immutableImage surface) and the distortion-portal end frame. As the distortion-portal end frame moves back through time, the frames in between are spread across the available distortion zone. Thus, the frameInc variable is defined to provide the proper frame increment between 1 and (Portal Frame/2) in the blended distortion zone.

Listing 8 shows the functions topCircle and bottomCircle as called by circleSetPixels.

                
void bottomCircle(float startx, float starty, int r, float inFrame)
{
  int i = 1;
  float nexty = starty +(r*2);
  for( i=r; i>=1; i-- )
  {
    int length = (int) (sqrt( cos(0.5f * 3.14159 * (i-r)/r)) * r * 2);
    int ofs = (r*2) - (length/2);
    lineSet(startx+ofs, nexty-i, length, chunkSize, inFrame );
  }//i
}//bottomCircle

void topCircle(float startx, float starty, int r, float inFrame)
{
  int i = 1;
  for( i=1; i<=r; i++ )
  {
    int length = (int) (sqrt( cos(0.5f * 3.14159 * (i-r)/r)) * r * 2);
    int ofs = (r*2) - (length/2);
    lineSet(startx+ofs, starty+i, length, chunkSize, inFrame );
  }//i
}//topCircle

Each top- and bottom-circle half is divided into a collection of horizontal lines. The functions above determine the offset from the circle edge and length of each line. The lineSet function, as shown in Listing 9, traverses the entire line.

                
void lineSet(int inX, int inY, int inWidth, int inHeight, int frameNum )
{
  if( chunkSize > 1 )
  {
    inX = (int) inX/chunkSize;
    inX *= chunkSize;
    inY = (int) inY/chunkSize;
    inY *= chunkSize;
  }//snap to grid if not in per-pixel mode

  int x=0;
  for( x=inX; x<=(inX+inWidth); x+= chunkSize )
  {
    int y=0;
    for( y=inY; y<=(inY+inHeight); y+=chunkSize )
    {
      if( x < WIDTH && y < HEIGHT )
      {
        if( frameNum > pixels[x][y] ){ pixels[x][y] = frameNum; }
      }//if on screen

    }//for y
  }//for x
}//lineSet

After the now-familiar chunking code (see "Speed Considerations"), each pixel on each line is set to the current frame if it is more than the current frame number. This will prevent overlapping distortion windows from showing conflicting portions of video frames. Additionally, this approach to predetermining the frame numbers at each pixel (or chunk) drastically reduces the number of blitting operations required to draw the correct visual.

Back to top

Rendering the frames at each pixel and animating the frames

Pre-processing the displayable visuals to select the correct pixels from the appropriate images contains much of the complexity of the program. Actually rendering the pixels to the drawable surface is simple once the precise pixel values have been determined. Listing 10 shows the drawPixels function.

                
void drawPixels()
{
  int x,y = 0;
  for( x=0; x<=WIDTH; x+=chunkSize )
  {
    for( y=0; y<=HEIGHT; y+=chunkSize )
    {
      src.x=x;
      src.y=y;
      src.w = chunkSize;
      src.h = chunkSize;
      SDL_BlitSurface( frame[pixels[x][y]], &src, screen, &src);
    }//y
  }//x
}//drawPixels

Notice how in this function a rectangle of chunkSize by chunkSize pixels is copied from the appropriate frame to the drawable surface. At a chunkSize of 1, this will create a per-pixel correct rendering of the distortion algorithm, while each increase in chunkSize will drastically reduce the number of SDL_BlitSurface calls required. After copying the drawable visuals, the animation of each frame is controlled by the animateFrames function.

                
void animateFrames()
{
  int x,y =0;
  for( x=0; x<=WIDTH; x+=chunkSize )
  {
    for( y=0; y<=HEIGHT; y+=chunkSize )
    {
      if( animateGrid[x][y] > 1 ){ animateGrid[x][y]--; }
    }//y
  }//x
}//animateFrames

This function moves each frame back in time from the last image in the sequence to the first. Consult the next section for the code block that sets each frame when the mouse is down.

Back to top

Main logic loop and usage

Completing main()

Recall from Listing 2 the main logic flow of setting up the screen, loading images, processing events, setting pixel states, drawing, and animating the frames. Listing 12 shows the code in main() and its implementation of this order.

                
int main(int argc, char *argv[])
{
  initScreen();
  loadImages();

  while ( stopMainLoop == 0 )
  {
    checkEvents();
    if( mouseIsDown ){ animateGrid[mouseX][mouseY] = MAX_IMAGES-1; }
    resetPixels();

    int x,y = 0;
    for( x=0; x<=WIDTH; x+=chunkSize )
    {
      for( y=0; y<=HEIGHT; y+=chunkSize )
      {
        if( animateGrid[x][y] > 1 ){ circleSetPixels(x,y, animateGrid[x][y] ); }
      }//y
    }//x

    SDL_BlitSurface(immutableImage, NULL, screen, &baseRect);
    drawPixels();
    SDL_Flip(screen);

    animateFrames();

  }// while stopMainLoop == 0

  SDL_Quit();
  return(0);
}//main

Once the screen is set up and the images are loaded, each pass of the main loop runs checkEvents and resetPixels. In between, the last frame state is set for the coordinates of the current mouse position if the mouse button is pressed. The next loop section controls setting pixels coordinates only if the frame at that position is greater than the start frame. SDL_BlitSurface clears the screen with the base image (immutableImage), and the drawPixels function call fills in the drawable surface with the appropriate image components. Finally, SDL_Flip transfers the back buffer to the screen buffer, and animateFrames updates the smooth progression of the video frames.

Usage

Included in the article source code distribution is a temporal.images directory with NOAA weather service radar images suitable for experimentation. Populate the directory with images extracted from your video of choice, or simply use the existing images with the command gcc `sdl-config --cflags --libs` -lSDL_image temporalVisualizer.c && ./a.out. Note that you'll need to have the appropriate SDL and SDL_image libraries installed for the compilation command above to work. Try modifying the chunkSize parameter to 10 at line 13 and recompile to drastically increase the speed of the visualizations.

Back to top

Speed considerations

Adapting the code to support useful visualizations on computing platforms of various speeds is controlled through the use of the chunkSize variable. In the checkEvents function, the stored mouse coordinates are forced into a grid of chunkSize by chunkSize squares. The lines shown below and the similar entries in the lineSet function ensure that the drawable pixels all start and end on the correct grid. Each increase in the chunkSize variable drastically reduces the number of comparisons, as well as surface blits when drawing the portals.

                
        mouseX = (int) mouseX / chunkSize;
        mouseX *= chunkSize;
        mouseY = (int) mouseY / chunkSize;
        mouseY *= chunkSize;

Back to top

Conclusion and further examples

With the code above and your completed temporalVisualizer.c program, you now have a functional program for visualizing frames from video or other sequential data sets in a new and useful way.

The temporalVisualizer.c program was designed to be a framework for further enhancements. As such, consider making the frames play backward or forward in time depending on which button was clicked. Add the option to mouse over a portal to have it freeze its current position. Send the frames further back in time the longer the mouse button is pressed.

Use the code here and your own ideas to create new visualizations for understanding and exploring your data through time.

Back to top

Download

	Information about download methods

Resources

• View the author's demonstration video of time-dependent visualization at YouTube.com.
  
  
• Read about the Simple DirectMedia Layer project, a cross-platform multimedia library.
  
  
• Read about SDL_image, an image-loading library used with the SDL library.
  
  
• There are many excellent tutorials on using SDL. Three are available at the SDL Web site, Lazy Foo' Productions, and Jari Komppa's tutorials.
  
  
• MPlayer is a video player that supports the playback of a wide array of video file formats. Download MPlayer for Linux, Windows, and OS X. It's also available package form for some Linux distributions.
  
  
• This article was inspired by the Khronos Projector.
  
  
• To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
  
  
• Stay current with developerWorks' Technical events and webcasts.
  
  
• Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
  
  
• Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
  
  
• Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
  
  

• Innovate your next open source development project with IBM trial software, available for download or on DVD.
  
  
• Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

• Participate in developerWorks blogs and get involved in the developerWorks community.
  
  

About the author

		Nathan Harrington is a programmer at IBM currently working with Linux and resource-locating technologies.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!