 | Level: Intermediate Nathan Harrington (harrington.nathan@gmail.com), Programmer, IBM
11 Jul 2006 Use Perl, ImageMagick, and MPlayer to create mosaic movies composed of frames from other movies. Zoom out from the center of a large text-overlay image made up of sequential frames of existing movies. Disassemble, composite, and encode your own mosaic-type movies for special promotional or home video events.
Mosaic movies are popular in today's television and advertising media due to their visual appeal and suggestions of technological advancement. This article will cover the methods required to create your own mosaic movies using text and graphics as the foreground subject. The background subject images will be composed of a set of frames from your existing video library.
Using MPlayer to extract frames of any video into jpeg files, we will process the images using a Perl script to composite them into sequence. MPlayer's movie encoder mencoder is then used to assemble the movie frames into an mpeg4 or divx-encoded video stream.
Requirements
The image manipulation and compositing processes are very memory- and CPU-intensive. This article was developed on Pentium® 3 and 4 processors of at least 800 MHz, with more than 512 MB of RAM. In addition, the disassembly process can require vast amounts of disk space, depending on your source resolution. This article uses 0.7 megapixel tiles for each component frame. If you are using high-quality source movies, prepare for a large chunk of your disk space to be occupied by the tiles. In addition, the composite phase uses a brute-force approach to inserting the tiles into the "mask" image. This step is relatively time-consuming and requires about 15 seconds to create a single frame of completed video on a 2-GHz Pentium 4. Fast processors are not required, but plan on longer render times if you use dated hardware.
You need Perl, GD, The Gimp, and ImageMagick -- all of which are installed by default in the majority of Linux® distributions. MPlayer is also required, along with any of the associated video codecs you may require to read or write your desired movie formats. See Resources for more information on where to find these tools. Recommend but not required is feh, an excellent image-viewing program that will help with the processing of your mosaic tiles and viewing your finished render frames.
Creation of a foreground image with The Gimp
Choose a foreground image size that is a larger multiple of your movie image tile size. For example, if your tiles are 320x240 resolution, a foreground image size of 3520x720 will be suitable and give a nice widescreen look at final rendering. A wide format is also suitable for one-word text entries, such as ELENA or other names. Create a new file in The Gimp with a transparent background at 3520x720. The next step is to insert a white foreground "mask" image onto this canvas. Any text or graphics are acceptable at this stage, as long as they are pure white. Any white pixels in the image will be treated as a pass-through mask in subsequent steps. For this example, we will use the text ELENA in Gill Sans Bold 2nd Medium font. This provides a nice wide viewable area for the tiles at various stages of the zoom process. For this example, the foreground text overlay file is saved as 3520x720whiteGillSansBoldSecondMedium.png and used in the compositing render step.
Figure 1. The Gimp 3520x720 ELENA
Movie tile images
For this article, I used 320x240 AVI-format movies from my digital camera. Other versions of the tools and methods in this article have used higher resolution or widescreen format videos with success. However, for the purposes of this article, I recommend converting all your source videos to 320x240 to keep the demand on system resources low.
Processing tiles
To create our mosaic video, we need a frame-by-frame extraction of all the source videos. We will use MPlayer and a simple Perl script to automate the extraction of frames from the source files into jpeg files on disk.
- Create a directory to place all your movie files in. For this example, we will use the directory avis. Copy all your video files into the avis directory.
- Create a file called buildTiles.pl with the following code:
Listing 1. buildTiles.pl
#!/usr/bin/perl -w
# buildTiles.pl - create jpgs of each frame of source movies
use strict; # like a librarian
my $cmd = ""; # command line string
my $mPlayer = `which mplayer`; # get location of mplayer
chomp($mPlayer);
if( $mPlayer eq "" ){
die "no mplayer detected!\n" .
"this project requires mplayer for decoding/encoding of\n".
"video frames, please install mplayer.\n";
}# if mplayer not detected
my $dirName = $ARGV[0] or die "specify directory with video files\n";
my @files = `ls -1 $dirName`;
for my $fName ( @files ){
chomp($fName);
print "processing $fName\n";
my $tileDir = "$dirName/$fName" . ".tiles";
$cmd = `mkdir $tileDir`;
# mplayer code to extract each frame into tileDir
$cmd = `$mPlayer -vo jpeg $dirName/$fName`;
$cmd = `mv *.jpg $tileDir/`;
}#while stdin
print "all files processed \n";
|
- Run the tile generation step with the command
perl buildTiles.pl avis.
You will now have a subdirectory under avis for every filename in avis, with a jpeg for each frame in the movie stored in the subdirectory.
These tiles will be used as the background components in our next step. Note that if you do not have MPlayer installed correctly, or you are missing codecs required to decode your selected video streams, this step of the process will fail. In this example, the -vo jpeg option tells MPlayer to extract the frames as jpeg images into the current directory, then all the files are moved to the tile directory for that file.
Creation of background frames
Now that we have a frame-by-frame extraction of our source video files, we can create tiled background images in sequence for each frame of the final product. This step is linked with the final render step, so we will simply create a background image one at a time with specified parameters, then automate this generation process frame by frame.
- Create the backgrounds directory to place all of the auto-generated background grid images in.
- Create a file called buildBackgroundFrame.pl with the following code:
Listing 2. buildBackgroundFrame.pl
#!/usr/bin/perl -w
# buildBackgroundFrame.pl - create a background image from tiles in sequence
# assumes a tile resolution of 320x240
use strict; # like an English banker
use GD; # image library
my $errMsg = "specify an image width, height, frame number, file list";
my $im_width = $ARGV[0] or die $errMsg; # width of overlay image
my $im_height = $ARGV[1] or die $errMsg; # height of overlay image
my $frameNum = $ARGV[2] or die $errMsg; # frame number
shift; shift; shift; # remove image parameters
my @tileList = @ARGV or die $errMsg; # list of tile directories
# create a new image in true color for tiles to be inserted into
my $image = new GD::Image($im_width,$im_height,1);
my $tilePos = 0; # position in tile list
my $xpos = 0; # incrementer for x axis
my $ypos = 0; # incrementer for y axis
# number of rows for a 720 pixel high image, assumes a 240 tile image height
my $row_total = $im_height / 240;
# for each row
while( $ypos < $im_height ){
$xpos = 0;
#for each column
while( $xpos < $im_width ){
# use each image in tile list in order
my $currimage = $tileList[ $tilePos ];
$tilePos++;
# reset to beginning of tile list if all have been used
if( $tilePos == @tileList ){ $tilePos = 0; }
# open the image
my $tileimg = newFromJpeg GD::Image( "$currimage/$frameNum.jpg");
# insert the image into the existing large template
# at position xpos, ypos (640,480 for example), starting at
# position 0,0 in the source image, with a height and width
# of 320,240
$image->copy($tileimg,$xpos,$ypos,0,0,320,240);
# move to next column
$xpos += 320;
}#while xpos < width
# move to next row
$ypos+= 240;
}#while ypos < im_height
# write out the image
print "writing image backgrounds/back_$frameNum\n";
open( TILEOUT,"> backgrounds/back_$frameNum.png") or die
"can't write to backgrounds/back_$frameNum.png";
print TILEOUT $image->png;
close(TILEOUT);
|
- If you want to test this step, run the command
perl buildBackgroundFrame.pl 3520 720 00000001 `ls -1d avis/*.tiles`.
The command above will create a 3520x720 image filled with the tile names 00000001.jpg from each of the directories specified in the ls -1d avis/*.tiles command. Note that this step fills an entire 3520x720 grid with tiles from the video extraction process. Not all of these tiles will be visible in the final composited image, so there is a great opportunity to increase the performance of this step with a little finesse. All the current steps are designed for simplicity, so there are plenty of areas for speed increases throughout this article.
Rendering of frames
The final rendering process will require the automatic compositing, extraction, and resizing of the images created in the previous steps. For examples of how to do this in static images, please refer to the article "Create mosaic images using Perl and ImageMagick" (see Resources).
Follow these steps:
- Create the render directory to place all of temporary and finalized rendered frames in.
- Create a file called buildVideoFrames.pl with the following code:
Listing 3. buildVideoFrames.pl
#!/usr/bin/perl -w
# buildVideoFrames.pl - create frames of mosaic video
use strict; # like Captain von Trapp
use GD; # image processing library
use List::Util 'shuffle'; # to randomize the image list
my $errMsg = "specify an image width, height, frame max, frame window, ".
"transparent, tiles directory \n";
my $im_width = $ARGV[0] or die $errMsg; # width of source image
my $im_height = $ARGV[1] or die $errMsg; # height of source image
my $frameMax = $ARGV[2] or die $errMsg; # maximum frames to process
my $frameWindow = $ARGV[3] or die $errMsg; # size to increment each frame
my $transP = $ARGV[4] or die $errMsg; # transparent or white background
my $tilesDir = $ARGV[5] or die $errMsg; # tiles directory
my $currFrame = 1; # current frame counter
my $cmd = ""; # command line string
my $tileSc = ""; # space delimited list of tile names
my ($sizeX, $sizeY, $cropX, $cropY, $frSizeX, $frSizeY) ="";
$sizeX = 960; # end image size X dimension
$sizeY = 176; # end image size Y dimension
$cropX = 1280; # start of crop area X dimension
$cropY = 262; # start of crop area Y dimension
# frSizeX is actually X dimension pieces of frameWindow size
# frSizeY is computed based on frameWindow so zoom out is smooth
$frSizeX = ($im_width/2) /$frameWindow;
$frSizeY = ($im_height/2) /$frameWindow;
# if necessary, create a white background of sizeXxsizeY
if( $transP == 1 ){ createSmallWhiteBackground(); }
# build a randomized list of tile names
my @tileList = `ls -1d $tilesDir/*.tiles`;
@tileList = shuffle(@tileList); # randomize the array
# build a space delimited list of tile names for passing on the command line
for( @tileList ){ chomp($_); $tileSc .= "$_ "; }
while( $currFrame <= $frameMax ){
# create a number like 00000001
my $padFrame = sprintf( "%08d", $currFrame);
# create background grid of images - if you are happy with your background
# tiles, and just want to create mosaics with different foregrounds, comment
# out this step to increase speed
$cmd = "perl ./buildBackgroundFrame.pl " .
"$im_width $im_height $padFrame $tileSc";
print "$cmd \n";
$cmd = `$cmd`;
# composite the foreground with background images
# insert your own foreground image filename in the second step
$cmd = "composite -compose in backgrounds/back_$padFrame.png " .
"3520x720whiteGillSansBoldSecondMedium.png " .
"composite/frame_$padFrame.png";
print "$cmd\n";
$cmd = `$cmd`;
# extract the zoomed in portion of the image
$cmd = "convert -crop ${sizeX}x${sizeY}+${cropX}+${cropY} " .
"composite/frame_$padFrame.png render/render_$padFrame.png";
print "$cmd\n";
$cmd = `$cmd`;
# resize the zoomed in portion of the image
$cmd = "convert -resize 960x196! render/render_$padFrame.png " .
"render/render_$padFrame.png";
print "$cmd\n";
$cmd = `$cmd`;
if( $transP == 1 ){
# overlay on white if desired
$cmd = "composite -compose over render/render_$padFrame.png " .
"backgrounds/final_backwhiteimage.png render/render_$padFrame.png";
print "$cmd \n";
$cmd = `$cmd`;
}# if transP active
# increment frame counter
$currFrame++;
# increment the size of the image by the window size, so it extracts a
# increasingly large portion of the main image, if the border of the
# image has been reached, extract the entire image
$sizeX = $sizeX + $frSizeX;
$sizeY = $sizeY + $frSizeY;
if( $sizeX > $im_width ){ $sizeX = $im_width; }
if( $sizeY > $im_height ){ $sizeY = $im_height; }
# increment the crop area by half the window size, so it gets half the
# content above/left, and the other half below/right. Change this if
# you want to zoom out towards the upper left or lower right, etc.
$cropX = $cropX - ($frSizeX/2);
$cropY = $cropY - ($frSizeY/2);
if( $cropX < 0 ){ $cropX = 0; }
if( $cropY < 0 ){ $cropY = 0; }
}#while frameCounter <= maxFrames
sub createSmallWhiteBackground{
# create a new image in true color for background only
my $backimage = new GD::Image($sizeX,$sizeY);
# initialize some colors
my $white = $backimage->colorAllocate(255,255,255);
my $black = $backimage->colorAllocate(0,0,0);
# write the file to disk
open(PNGOUT,"> backgrounds/final_backwhiteimage.png") or die
"can't write to output file backgrounds/final_backwhiteimage.png";
print PNGOUT $backimage->png;
close(PNGOUT);
}#createSmallWhiteBackground
|
- Run the first rendering step with the command
perl buildVideoFrames.pl 3520 720 120 20 1 avis.
In the command above, the first two options specify a resolution of 3520x720 for the composited images. This includes the background images for each frame of the video and the foreground image of the text ELENA. The next two options are maximum image frames and frameWindow. The maximum image frames must be less than the lowest number of frames in any of your video source files. For example, if you only have 60 frames in one of your source files, and you specify 75 as the maximum image frames parameter, your final rendered result will have blank entries in part of the image.
The frameWindow parameter is how many pixels to expand the cropped image per frame to. For example, if you specify a frameWindow parameter of 20, the interior image that is cropped will be 20 pixels wider in each additional frame, until the width of the image is reached. The height frameWindow parameter is computed based on the specified dimensions of the composited images in order to create a smooth zoom out in both dimensions, regardless of the aspect ratio of the image.
The goal of using a relatively large frameWindow parameter was for the zoom out to happen quickly to allow the viewer to appreciate the novelty of the mosaic effect. If you have many frames in your video source files, try specifying 2 as the frameWindow parameter for an extremely slow zoom out effect.
The remaining two options are the transparency flag and the directory of image tiles. Specify a 0 for the transparency flag to have your final encoding scheme decide what the background image will be. For example, specifying a transparent background with a 0 option and encoding in mpeg4 will produce black for the background color. Use 1 to guarantee a white background color in the final movie. The avis parameter is simply the name of the directory where all the image tiles are stored, under their respective directories.
Encoding of the final movie
After the buildVideoFrames.pl command has completed, the render directory will contain each composited frame of the movie. All that is left is to encode them as a video stream with the codec of choice. Run the final rendering step with the command:
Listing 4. Final rendering step
mencoder \
"mf://render/*.png" -mf fps=17 \
-ovc lavc \
-lavcopts vcodec=mpeg4 \
-o videoMosaic.mpeg
|
mencoder takes the parameters mf://filemask -- in this case, every png file in the render directory. The -mf fps=17 parameter tells mencoder to create a movie at 17 frames per second. This seems to match well to the low frame rate of the source video files. You may have to experiment with this to achieve a natural-looking flow to the end-result animation.
The -ovc lavc parameter tells mencoder to use the libavcodec codec, and the -lavcopts vcodec=mpeg4 specifies the usage of the mpeg4 codec for video encoding. -o videoMosaic.mpeg is the file to write the resulting video to.
Figure 2 shows example frames from the final rendering process. Download the source code for an mpeg4-encoded example of the video mosaic process.
Figure 2. Example mosaic results
Summary
With your current framework and example mosaic movie, you can build interesting examples for your promotional videos or home movies. Remember, any white pixels you put in your foreground image will provide a mask over the background frames. With this approach, and with tips from the still-image mosaic article in Resources, you can create video mosaic images of the text of your choosing.
Download | Description | Name | Size | Download method |
|---|
| Source code | video_mosaic.zip | 829KB | HTTP |
|---|
Resources Learn
-
Read "Create mosaic images with Perl and ImageMagick" to explore the capabilities of ImageMagick, GD, and The Gimp, which are open source graphical editing tools.
-
Learn all about ImageMagick at ImageMagick.org.
-
Read about GD, its API, and find example code on the GD Web site.
-
If you need the best in image manipulation software, visit The Gimp.
-
Stay current with developerWorks technical events and webcasts.
-
Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
-
Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
-
To listen to interesting interviews and discussions for software developers, be sure to check out developerWorks podcasts.
Get products and technologies
-
Find Perl modules including GD.pm and more at CPAN, the comprehensive Perl archive network.
-
Download Tom Gilbert's image viewer, feh.
-
Innovate your next open source development project with IBM trial software, available for download or on DVD.
Discuss
About the author | | | Nathan Harrington is a programmer at IBM currently working with Linux and resource-locating technologies. |
Rate this page
| |
