Add multitouch gesture support to a TouchPad-equipped laptop

Document options Document options requiring JavaScript are not displayed Sample code Rate this page Help us improve this content

Level: Introductory

Nathan Harrington (harrington.nathan@gmail.com), Programmer, IBM　

03 Jun 2008

Enable swipe and pinch gestures for Linux® applications by analyzing synclient program output for a Synaptics TouchPad.

Multitouch interfaces provide a great deal of benefits for integrating new interaction modes within applications. Newer hardware and drivers on Mac OS X and Microsoft® Windows® allow for a variety of gestures beyond point and click that create more-efficient application navigation. This article provides tools and code needed to add some of this new gesture support on older Linux-enabled hardware. Building on the output of the synclient program, the Perl code presented here allows you to assign specific application functions to "Three-Finger Swipe," as well as Open- and Close-Pinch gestures.

Requirements

Hardware

The code presented here is designed for use with a computer equipped with a Synaptics TouchPad only and happened to be developed on an IBM® ThinkPad T30. You can find Synaptics touch-pads on many laptops ranging from Acer Aspires to Toshiba Tecras. Consult the Resources section for a Synaptics TouchPad software project hardware compatibility list to see if you hit the jackpot.

Software

You need a modern Linux kernel with evdev support. Fortunately, most modern distributions have this functionality built in. Many distributions come with the Synaptics package as well, which includes synclient, used for monitoring TouchPad events. Fedora Core, for example, also includes the proper X Window System configuration to enable TouchPad usage with minimal modification by the user. Other distributions, such as Ubuntu V7.10, may require further configuration before the Synaptics package — installed with the command sudo apt-get install tpconfig — will work correctly. Consult Resources for more information on achieving basic functionality with Synaptics TouchPads under Linux.

You also need the Time::HiRes module from CPAN to provide subsecond timing control for processing TouchPad events. In addition, you need the X11::GuiTest module to send synthetic X Window events to applications. See Resources for these tools.

Back to top

Ensuring basic functionality

If mouse control is enabled with the TouchPad, check for adequate multifinger detection for gesture support. Run the command synclient -m 100 and try different touches on the TouchPad. You should see output similar to the following.

                
    time     x    y   z f  w  l r u d m     multi  gl gm gr gdx gdy
  13.872  5680 4409   0 0  0  0 0 0 0 0  00000000   0  0  0   0   0
  14.891  1072 3945  28 1  4  0 0 0 0 0  00000000   0  0  0   0   0
  14.994  3529 2667 104 2  5  0 0 0 0 0  00000000   0  0  0   0   0
  15.605  3669 3667   0 0  0  0 0 0 0 0  00000000   0  0  0   0   0
  16.625  2628 2841 255 3  5  0 0 0 0 0  00000000   0  0  0   0   0
  17.951  3117 2843 255 3  5  0 0 0 0 0  00000000   0  0  0   0   0
  18.053  2902 3142   3 1 15  0 0 0 0 0  00000000   0  0  0   0   0
  18.155  2430 3062   0 0  0  0 0 0 0 0  00000000   0  0  0   0   0

Try one-, two-, and three-finger touches to make sure events are detected correctly. Make sure the TouchPad can detect three fingers, as the first gesture to be added is "Three-Finger Swipe." Notice how the TouchPad picks up zero fingers, as well as the X and Y coordinate readings when pressing two fingers at widely varying spaces. The processing script below makes use of some these characteristics to help detect open and close pinches. Press Ctrl+c to exit the synclient program.

Back to top

General program approach

Using syclient output for monitoring the TouchPad state is a simple and effective way for adding further interface options to Linux applications. The gestureListener.pl program introduced below opens a pipe to read from the synclient program, and processes the TouchPad events to detect gestures. These specific gestures are linked with keyboard commands sent to the current in-focus application in X Window System.

Back to top

Swipe gestures

The Three-Finger Swipe is a relatively simple gesture to detect, as it simply requires three fingers on the TouchPad moving left or right. Listing 2 shows the beginning of the gestureListener.pl program required to begin processing the synclient output for gesture detection.

                
#!/usr/bin/perl -w  
# gestureListener.pl listens for pinch and swipe events
use strict;
use Time::HiRes();
use X11::GUITest qw( :ALL );

my @xHist = ();               # x coordinate history
my @yHist = ();               # y coordinate history
my @xHistThree = ();          # x coordinate history (three fingers)

my $lastTime = 0;             # time monitor for TouchPad event reset
my $eventTime = 0;            # ensure enough time has passed between events
my $eventString = "default";  # the event to execute
my $centerTouchPad = 3000;    
my $openSt         = 1000;    # start of open pinch
my $openEn         = 500;     # end of open pinch
my $closeSt        = 1000;    # start of close pinch
my $closeEn        = 1000;    # end of close pinch

my $synCmd = qq{synclient TouchpadOff=1 -m 10};
my $currWind = GetInputFocus();
die "couldn't get input window" unless $currWind;
open(INFILE," $synCmd |") or die "can't read from synclient";

Note that the centerTouchPad variable and other parameters may require customization based on your specific Synaptics hardware or driver level. The TouchPadOff=1 option to the synclient command turns off regular TouchPad events. The red "mouse stick" (on ThinkPads and others) is still available, as well as PS2 and USB mouse support. Turning off the TouchPad is not necessary, but it reduces the problem of identifying nongesture-related mouse events from the swipes and pinches.

The call to GetInputFocus finds the current window identifier for the window in focus. This allows the SendKeys command (used later) to send synthetic X Window events to the currently in focus window. Listing 3 starts the main program loop and reads the synclient output.

                
while( my $line  = <INFILE>)
{
  chomp($line);
  my(  $time, $x, $y, $z, $f ) = split " ", $line;
  next if( $time =~ /time/ ); #ignore header lines
  
  if( $time - $lastTime > 1 )
  { 
    @xHist = ();
    @yHist = ();
    @xHistThree = ();
  }#if time reset
  
  $lastTime = $time;

Resetting the event-detection history arrays at each timeout is critical to eliminating carry over between TouchPad gestures. Listing 4 shows the beginning of the three-finger detection in the main program loop.

                
  # three finger swipe detection
  if( $f eq "3" )
  {
    @xHist = ();
    @yHist = ();

    push @xHistThree, $x;
        
    if( @xHistThree > 10 )
    { 
      my @srt = sort @xHistThree;
      my @revSrt = reverse sort @xHistThree;

      if( "@srt" eq "@xHistThree" )
      { 
        # alt + right arrow - forward
        $eventString = "'%({RIG})";
      }elsif( "@revSrt" eq "@xHistThree" ) 
      {   
        # alt + left arrow - back
        $eventString = "'%({LEF})";
      }#if forward or backward

      @xHistThree= ();
        
    }#if more than 10 data points in 3 finger array

After 10 points of three-finger data are collected, the X coordinates are processed to create ascending and descending sorts. If the ascending sort matches the current value of X coordinates, the right-swipe condition is set. Conversely, if the descending sort is matched, the left-swipe condition is set. The eventString variable holds this condition and will be executed as shown below.

                
  }else
  { 
    # reset all data points, yes you can have 0 fingers at x,y
    @xHist = ();
    @yHist = ();  
    @xHistThree = ();
  }# if not one or two or three fingers
  
  # only process one event per time window
  if( $eventString ne "default" )
  { 
    if( abs(time - $eventTime) > 1 )
    { 
      $eventTime = time;
      SendKeys( "$eventString");
    }#if enough time has passed
    $eventString = "default";
  }#if non default event

}#synclient line in

close(INFILE);

At this point, each data structure for pinch or swipe is reset if three fingers are not detected. If an event has been set and the current time is far enough from the last event execute time, a new event is executed. The SendKeys subroutine sends the appropriate event (Alt + Left or Right Arrow) to the currently focused application. As shown in the demo video (see Resources), these three-finger gestures are used to move forward and backward in the browsing history.

Back to top

Pinch gestures

Pinch gestures are substantially more complicated to detect, especially on the older hardware used to develop this article. Monitoring the open and close gestures in real time using a tool like kst is a helpful way to extract relevant features from the TouchPad data. Insert the code in Listing 6 at line 65 (above the else) to begin the close pinch-detection section.

                
  }elsif( $f eq "2" || $f eq "1" )
  {
    # accept 1 or 2 finger entries as part of pinch section
    @xHistThree = ();
  
    push @xHist, $x;
    push @yHist, $y;
    
    if( @xHist > 50 )
    {
      if( (getStrAvg(\@xHist) > $closeSt && getStrAvg(\@yHist) > $closeSt) &&
          (getEndAvg(\@yHist) < $closeEn && getEndAvg(\@yHist) < $closeEn) )
      {
        # wide to narrow detected, now search for enough 'wiggle'

        my $tenX = 0; 
        my $tenY = 0;
        for my $each( @xHist[40..49] ){ $tenX += $each }
        for my $each( @yHist[40..49] ){ $tenY += $each }
        $tenX = $tenX / 10;
        $tenY = $tenY / 10;

        my $diffX = 0;
        my $diffY = 0;
        for my $each( @xHist[40..49] ){ $diffX += abs( $each - $tenX ) }
        for my $each( @yHist[40..49] ){ $diffY += abs( $each - $tenY ) }

        # ctrl - decrease font size
        if( ($diffX+$diffY) > 80 ){ $eventString = "^({-})" }
        
        @xHist = ();
        @yHist = ();
        @xHistThree = ();

      }#if x and y in range

    }#if enough data for 50 close pinch detection

One or two fingers are accepted as part of the pinch-detection step to enhance the overall tracking. Slightly offset timings of touches and releases, as well as moving one finger slightly off the TouchPad during pinch movements are dealt with more easily by accepting one or two touches as part of the pinch movement.

After 50 or more data points have been collected, the average start and end positions of the past 50 data points are computed. The third if statement performs four separate checks to ensure that the start and end points are in the correct parts for a close-pinch detection. Specifically, if the X and Y averages for the start points need to be more than the close-pinch start points. That is, the positions of the fingers need to be in the corners. Conversely, the end points need to be within the close-pinch end points. getStrAvg and getEndAvg create averages of the three start and end points, which are a more reliable data point for the check.

True multitouch capability would allow for an X and Y coordinate reading of each finger position. The Synaptics hardware available does not have this capability, but does provide consistent behaviors as the two-finger touch points are "averaged" into one output for the synclient program. Monitoring the output of the synclient program shows that when two fingers touch in the southwest and northeast corners, the values rapidly go from the corners to the center of the TouchPad. If the fingers are kept in the corners, the synclient output shows the X and Y coordinates staying near the center. As the fingers are moved toward the center, this automatic averaging shows slight perturbation. The wiggle in the data is detected by the section of code above to indicate a close pinch, as opposed to the synthetic averaging of data points when the fingers are held in the corners without movement.

Ten of the trailing values of the X and Y coordinate history are averaged together. The difference between each of the last 10 trailing values and the average for the last 10 is then computed. If the overall difference is great enough, (greater than 80 in this case), enough wiggle has been detected and the close-pinch condition is set in eventString.

Add the code shown in Listing 7 at line 103 (below the close-pinch 50-point if closing bracket ) for open-pinch detection.

                
    #open pinch requires substantially fewer data points
    if( @xHist > 10 )
    {
      if( (getStrAvg(\@xHist) < $openSt && getStrAvg(\@yHist) < $openSt) &&
          (getEndAvg(\@yHist) > $openEn && getEndAvg(\@yHist) > $openEn) )
      {
        # ctrl + increase font size
        $eventString = "^({+})"; 

        @xHist = ();
        @yHist = ();
        @xHistThree = ();
      }#if absx and absy

    }#if enough data

In addition to requiring substantially fewer data points for reliable detection of an open pinch, the wiggle-detection step is not necessary. With real-time monitoring using a program like kst, synclient output shows a much broader range of data points collected for the open-pinch gesture. The automatic averaging of the two fingers positions does not produce a large influence on the synclient X and Y coordinates during an open-pinch gesture. Therefore, it is no longer necessary to recognize the wiggle for accurate gesture detection. Listing 8 shows the getStrAvg and getEndAvg averaging subroutines:

                
sub getStrAvg
{     
  my $arrRef = $_[0];
  my $val = (@$arrRef[0] + @$arrRef[1] + @$arrRef[2]) / 3;
  $val = abs( $val - $centerTouchPad ); 
  return($val);
}#getStrAvg
        
sub getEndAvg
{       
  my $arrRef = $_[0];
  my $val = (@$arrRef[@$arrRef-3] + @$arrRef[@$arrRef-2] + 
               @$arrRef[@$arrRef-1]) / 3;
  $val = abs( $val - $centerTouchPad );
  return($val);
}#getEndAvg

Place the code shown in Listing 8 at line 144 (the end of the file) to complete the gestureListener.pl program. getStrAvg and getEndAvg are responsible for returning the average of the first three array elements and the average of the last three array elements, respectively.

Usage

Activating the program is a simple matter of running the command perl gestureListener.pl. Bring your Web-browsing window into focus and try moving through the browsing history with left and right swipes. Note that due to the smaller track-pad size, as well as the inherent limitations of the ability of the hardware, it may take some practice before you can reliably trigger gesture events. Consult the demonstration video (see Resources) for examples of hand movements that work on the author's ThinkPad. If you want to re-enable TouchPad "mouseiness," run synclient TouchPadOff=0 after exiting the gestureListener.pl program.

Back to top

Conclusion and further examples

Linkage between gestures and events

Keep in mind that the SendKeys command will send the appropriate commands (Alt+Left, Ctrl+-, etc.) to the currently focused application, regardless of the functionality this triggers. A relatively simple change to consider is specifying different keystrokes or mouse events to send to an application based on the window name.

Further examples and modifications

The combination presented here of analysis and processing of synclient output and synthetic X event creation is only one path for added functionality to the Synaptics TouchPad. Consider adding additional gesture recognition, such as pinch rotate through further feature extraction from the synclient output. Alternatively, modify the Synaptics driver source code to support additional features at the kernel level and rewrite applications to take better advantage of these new input channels.

• Innovate your next open source development project with IBM trial software, available for download or on DVD.
  
  
• Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.

• Participate in developerWorks blogs and get involved in the developerWorks community.
  
  

About the author

		Nathan Harrington is a programmer at IBM currently working with Linux and resource-locating technologies.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!