 | Level: Intermediate Nathan Harrington (harrington.nathan@gmail.com), Programmer, IBM
11 Mar 2008 Use synthetic X Window System events and embedded accelerometers to control
applications by the movement of a laptop computer. Translate gestures, such as
shaking, into mode-switching commands with detection algorithms to interact with
applications in new ways. Develop tools to help build the next generation of
interfaces that use accelerometers, such as applications for laptops and iPhones.
Innovative haptic interface systems are allowing a new level of application interaction.
Inexpensive video game peripherals with embedded accelerometers and PC
devices ranging in size from laptops to hand-held phones are enabling new methods for
application control. Applications frequently lag behind in their interfaces for
control, restricting their functionality to keyboard and mouse events through a lack of APIs.
This article demonstrates techniques and code you'll need to connect
accelerometer-enabled devices to applications in a traditional desktop computing context.
Requirements
Hardware
Many IBM® ThinkPads manufactured in 2003 and later sport Hard Disk Active Protection
System (HDAPS) hardware. If you are unsure of your hardware configuration, check out the
technical details on the Lenovo Web site.
Software
You need a Linux® distribution with kernel support for HDAPS. The HDAPS
driver must be included in the kernel to enable access to the accelerometers. Newer
kernel distributions, including Red Hat, Debian, Gentoo, and Ubuntu, include HDAPS
drivers. Consult the HDAPS-related articles in the Resources
section for more information about how to get started with HDAPS kernel drivers.
You'll also need the Time::HiRes module from CPAN to provide subsecond timing control
when recording data. In addition, the X11::GUITest module is required for sending
synthetic X events to applications (see Resources).
Note that the techniques and algorithms described here should work with any
notebook computer that has on-board accelerometers. Those with MacBooks or ThinkPads
running an OS other than Linux should be able to adapt the code presented here without
extraordinary difficulty. Hand-held devices and other accelerometer-enabled systems can
also be used to employ the translation between tips, tilts, and commands. There are
many methods for sending synthetic keyboard and mouse events to applications that may
prove useful if operating on other operating systems. If you are using Microsoft®
Windows®, consider using the SendKeys class of
commands for application access.
Synthetic X events using xorg components
Control of application-specific features is most often accomplished with functional
hooks at the API level. However, many applications do not have a full set of APIs and
require mouse and keyboard input to manipulate the application fully. Other
applications have no API and provide sole interactivity through traditional human
computer-interface devices. This article will demonstrate components and algorithms that
can be linked together to control applications without a D-Bus or other API.
Sending synthetic X events to applications is the key method to enabling full control
of D-Bus and non-API applications.
X11::GUITest
You can spend a great amount of time integrating XTest library function calls into your
application. You can specify key release rates and find all the nuances in identifying
windows and keeping track of the keyboard and mouse state. Or you can let X11::GUITest
do the work for you. The X11::GUITest module provides a robust wrapper around the
XTest library for much more efficient handling of events.
Many applications that respond to user-keyboard and mouse events perform in unexpected
ways when using synthetic events. You may have to alter your keyboard repeat rate and
delay, or find a new method altogether, than standard input mechanisms. For example,
Google Earth does not always respond in a useful way to key-down events regardless of
the repeat rate and speed. An alternate method for equivalent synthetic input is used
for this article: holding down the center control stick of the navigation compass. You
may find similar inaccuracies or limitations in other programs when trying to create
synthetic X events that perform the same functions as their real-world alternatives.
When gluing together applications using programmatically triggered keyboard and mouse
events, you may need to experiment to find the input method that provides the best usability experience.
Translating HDAPS sensor data to application control
Our sensors have been enabled, and our controlling framework set up, so the next step
is to integrate them in a Perl program to translate movement into action. Listing 1
shows the header and variable declaration of the hdapsGoogleEarth.pl script.
Listing 1. hdapsGoogleEarth.pl globals and includes
#!/usr/local/bin/perl -w
# hdapsGoogleEarth.pl - control google earth by the pitch and roll of a thinkpad
use strict;
use Time::HiRes qw( usleep );
use X11::GUITest qw( :ALL );
my $threshMove = 3; # lower limit indicating motion
my $shakMin = 20; # amount of a shake required to indicate change in mode
my $timeThreshold = 1; # number of seconds between mode changes
my $maxMove = 25; # maximum movement of mouse from center point
my @avgX = (); # array of 10 recent hdaps X values for shake detection
my $lastTime = time; # record time of last mode change
my $slptim = 25000; # microseconds to pause between data reads
my $maxTip = 100; # turn thinkpad on its' side to exit the program
my $restX = 0; # `resting' positiong of X axis accelerometer
my $restY = 0; # `resting' positiong of Y axis accelerometer
my @windows; # array of window id's associated with google earth
my $hdapsFN = "/sys/devices/platform/hdaps/position"; # hdaps sensor
|
After including the required modules, global variables are set up to process the motion
and maintain the input mode. This article demonstrates controlling Google Earth in
either zoom or movement mode. Shaking the ThinkPad enables a quick switch between
modes for more intuitive navigation. Listing 2 handles the identification and focusing of the Google Earth window.
Listing 2. Identification and focus of Google Earth window
@windows = FindWindowLike("Google Earth");
die "No google earth windows found" unless( scalar(@windows) > 0 );
my ( $geX, $geY, $geW, $geH, undef, undef ) = GetWindowPos( $windows[0] );
# rose center is left 110 from geX+geW, down 135 from geY
# zoom center is left 22 from geX+geW, down 135 from geY
my $cntrCompX = ($geX + $geW) - 110;
my $cntrCompY = $geY + 135;
my $centerZoomX = ($geX + $geW) - 22;
my $centerZoomY = $geY + 135;
my $maxZoomTop = $centerZoomY - $maxMove;
my $maxZoomBottom = $centerZoomY + $maxMove;
my $maxLeft = $cntrCompX - $maxMove;
my $maxRight = $cntrCompX + $maxMove;
my $maxTop = $cntrCompY - $maxMove;
my $maxBottom = $cntrCompY + $maxMove;
my $mouseX = $cntrCompX;
my $mouseY = $cntrCompY;
my $controlMode = "movement";
MoveMouseAbs( $cntrCompX, $cntrCompY );
PressMouseButton M_LEFT;
($restX, $restY ) = readPosition();
|
The first step is to use the X11::GUITest module to locate the Google Earth window
using the FindWindowLike function. Multiple windows for
Google Earth are sometimes identified, but the first window ID is usually the main
window. Note that if the first window ID is not the main Google Earth window, the
program will not process inputs correctly. You may run into problems if you machine
does not specify the main Google Earth window as the first Google Earth window ID.
If this is the case, experiment with window IDs at different positions in the @windows array until you find the correct ID for the main Google Earth window.
After determining the correct coordinates of the Google Earth input window and moving
the mouse into place, a baseline for the accelerometers is determined through readPosition. Listing 3 shows the readPosition subroutine.
Listing 3. readPosition subroutine
sub readPosition {
my ($posX, $posY) = "";
open(FH," $hdapsFN") or die "can't open hdaps file";
while(my $hdapsLine = <FH>)
{
$hdapsLine =~ s/(\(|\))//g; # remove parens
($posX, $posY) = split ",", $hdapsLine;
}# while hdaps line read
close(FH);
return( $posX, $posY );
}#readPosition
|
Reading data from the HDAPS sensor is a straightforward process requiring an open and
close of the file at each read due to its nonseekable nature. (Check Resources for more information about working with the
particulars of the HDAPS sensors.) With the base variable declarations and subroutine
complete, the program moves on to the main program loop, shown below.
Listing 4. Main program loop, Part 1
while(1)
{
# get accelerometer values
my ($currX, $currY) = readPosition();
$currX -= $restX; # adjust for rest data state
$currY -= $restY;
# exit loop if ThinkPad tipped past 'exit' range
last if( abs($currX) > $maxTip || abs($currY) > $maxTip );
my $testX = abs($currX); # keep original values for trending
my $testY = abs($currY);
if( $controlMode eq "movement" )
{
# stay at the maximum distance to move even if severely tipped
if( $testX > $maxMove ){ $testX = $maxMove }
if( $testY > $maxMove ){ $testY = $maxMove }
$mouseX = ($cntrCompX - $testX) if( $currX < -$threshMove );
$mouseX = ($cntrCompX + $testX) if( $currX > $threshMove );
$mouseY = ($cntrCompY - $testY) if( $currY < -$threshMove );
$mouseY = ($cntrCompY + $testY) if( $currY > $threshMove );
if( $mouseX < $maxLeft ) { $mouseX = $maxLeft }
if( $mouseX > $maxRight ) { $mouseX = $maxRight }
if( $mouseY < $maxTop ) { $mouseY = $maxTop }
if( $mouseY > $maxBottom ){ $mouseY = $maxBottom }
# reset the mouse to the center if the current reading is not past
# the threshold
$mouseY = $cntrCompY if( $currY < $threshMove && $currY > -$threshMove );
$mouseX = $cntrCompX if( $currX < $threshMove && $currX > -$threshMove );
|
Each pass of the main program loop reads new values from the accelerometer sensors.
These values are subtracted from the reference state created earlier to transform
values like '-379,-380' to a more useful '0,0' (when the ThinkPad is at rest). Next,
the program checks to see if the ThinkPad has been tipped drastically from a rest state
to indicate an exit condition. This is necessary as full keyboard and mouse control
has been taken by the program and, therefore, a physical-orientation method of telling the program to exit is required.
If the program is in movement-processing mode, the next two lines set the upper
movement-boundary limits to $maxMove. The next four lines
move the mouse in the appropriate X and Y dimensions depending on how far the ThinkPad
has been tipped past the threshold of movement ($threshMove). Very few people possess the ability to consistently
hold their ThinkPads perfectly level, so it's useful to have a threshold of tip in each dimension for easier base-state detection.
The next four lines restrict the movement of the mouse to the maximum distance from the
center in each dimension. Finally, the mouse position is reset to the center if the
aforementioned threshold of movement has not been reached. Resetting the center
position can create a jump movement upon tilt past a threshold, but is preferable in
many applications to a slow drift or unknown initial state.
Listing 5 shows the comparatively simple zoom-control mode.
Listing 5. Main program loop, Part 2
}else
{
# zoom control - move up and down only
$mouseY = ( $centerZoomY - $testY ) if( $currY < -$threshMove );
$mouseY = ( $centerZoomY + $testY ) if( $currY > $threshMove );
if( $mouseY < $maxZoomTop ){ $mouseY = $maxZoomTop }
if( $mouseY > $maxZoomBottom ){ $mouseY = $maxZoomBottom }
$mouseX = $centerZoomX;
$mouseY = $centerZoomY if( $currY < $threshMove && $currY > -$threshMove )
}#if in zoom mode
MoveMouseAbs( $mouseX, $mouseY); # move the mouse after coordinates computed
|
Similar to the movement mode above, the zoom-control mode moves the mouse in the Y
dimension depending on how far the ThinkPad has been tipped past the movement threshold
forward or back. The computed movement is then restricted to the maximum distance from
the center of the zoom slider. Again, the mouse position is reset to the center if the movement threshold is not reached.
After computation of the mouse position, whether in zoom or movement mode, the mouse is
moved to control the application.
Gesture acquisition for mode switching
With the code above, you can control the movement and zoom modes of Google Earth by
moving your ThinkPad. To switch between modes, an additional check is required in each
pass of the main program loop. Listings 6 and 7 show the remainder of the main program
logic and the shakeCheck subroutine that perform the gesture acquisition and mode change.
Listing 6. Main program loop, Part 3
# mode switch by shaking, only if at least timeThreshold seconds have passed
# since last mode switch
if( shakeCheck( $currX ) == 1 && abs($lastTime - time) > $timeThreshold)
{
$lastTime = time;
if( $controlMode eq "movement" )
{
$controlMode = "zoomRotate";
$mouseX = $centerZoomX;
$mouseY = $centerZoomY;
}else
{
$controlMode = "movement";
$mouseX = $cntrCompX;
$mouseY = $cntrCompY;
}#if movement or zoomRotate mode
ReleaseMouseButton(M_LEFT) if( IsMouseButtonPressed(M_LEFT) );
MoveMouseAbs( $mouseX, $mouseY);
PressMouseButton( M_LEFT );
print "$controlMode mode\n";
}# if a large enough shake in X dimension detected
usleep($slptim);
}#while
ReleaseMouseButton M_LEFT; # let go of button on exit
|
A true value from the shakeCheck subroutine, as well as an
acceptable time difference between mode switches, will alternate between movement and
zoom mode. $timeThreshold is easily configurable to require
more or less time between mode shaking. Depending on how hyper-caffeinated you are, you
may want to increase this parameter, as well as the shakMin
variable shown in the shakeCheck subroutine, as shown in the
globals definition section and Listing 7.
Listing 7. shakeCheck subroutine
sub shakeCheck
{
my $xVal = $_[0];
push @avgX, $xVal;
# build an array of 10 samples before attempting processing
return(0) if( @avgX < 10 );
my $currAvg = 0;
for my $eachX ( @avgX ){ $currAvg += $eachX };
$currAvg = $currAvg /10;
shift @avgX;
# if current value is a big enough deviation from average value
return(1) if( abs($currAvg - $xVal) > $shakMin );
return(0);
}#shakeCheck
|
The subroutine shakeCheck builds an average of the past 10
HDAPS sensor reads in the X dimension only. The current value of the X dimension is
compared to the average, and if they diverge by at least $shakMin, a shake condition is set and the subroutine returns true.
A rapid side-to-side shaking motion is easily detected because, although the movements
are relatively small, the accelerations are large enough for useful gesture acquisition.
hdapsGoogleEarth.pl and hdapsGoogleStreeView.pl usage
To run the hdapsGoogleEarth.pl program as built above, start the Google Earth
application. When the default globe is displayed, start the synthetic X events
translator with the command perl hdapsGoogleEarth.pl. You
should see the Google Earth window brought into focus, with the mouse grabbing the
movement-control knob in the upper right-hand corner. Pick up your ThinkPad and start
navigating, and remember to tip your ThinkPad near vertical in either dimension to exit the program.
Check the code archive in the Resources section for another
Perl script to control Google street view by translating ThinkPad tips and tilts to key
presses. To use the hdapsGoogleStreetView.pl script, make sure you have a Firefox Web
browser running with Google Maps open and in street-view mode.. Start the
application with perl hdapsGoogleStreetView.pl, and you'll
be able to navigate city streets with tilts and rolls. Examine the source code to see
how the hdapsGoogleStreetView.pl application uses keystrokes instead of mouse moves for application control.
Conclusion and further examples
Continuous input applications, such as mapping software, are excellent for
applying intuitive navigation systems. With the code demonstrated here, you
can apply this next level of navigational system to any application through the use of synthetic keyboard and mouse events.
Connect your accelerometer-enabled cell phone over Bluetooth to your PC, and control
applications as you rotate your phone. Attach an inertial gyroscope to the ThinkPad
and scan the map in rotate, as well as tilt and roll.
Download | Description | Name | Size | Download method |
|---|
| Sample code | os-thinkpad-hdapsAppControl.zip | 4KB | HTTP |
|---|
Resources Learn
Get products and technologies
-
Grab the Time::HiRes module from
CPAN.
-
Learn more about Linux kernel
download mirrors for acquiring HDAPS.
-
Grab Perl from the source at Perl.org.
-
Innovate your next open source development project with IBM trial software, available for download or on DVD.
-
Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
Discuss
About the author | 
| | Nathan Harrington is a programmer at IBM currently working with Linux and resource-locating technologies. |
Rate this page
| |
