How to ungrab Firefox hotkeys from Flash players

Document options Document options requiring JavaScript are not displayed Sample code Hey there! developerWorks is using Twitter Follow us Rate this page Help us improve this content

Level: Intermediate

Nathan Harrington, Programmer, IBM

16 Dec 2008

Flash players and other embedded applications in Firefox require their own hooks for keyboard and mouse input. For years, Flash has grabbed Firefox keypresses, which stops people from using the keyboard for navigation, creating new tabs, or even exiting the Flash focus. Learn how to create a Perl program that communicates with a Firefox extension and cnee to restore your keyboard functionality.

Flash players and other embedded applications in Firefox require their own hooks for keyboard and mouse input. Since May 2001, if Flash grabbed your Firefox keypresses, you couldn't use your keyboard for navigation, creating tabs, or even exiting the Flash focus (see Resources for Mozilla bug No. 78414).

This article presents tools and code to allow a Firefox running on Linux® to respond to hotkeys, such as Ctrl+t (open new tab) even when an embedded Flash player has the focus. Use the code here to reclaim your Firefox keyboard application control. This article does not fix the underlying problem, but it provides a work-around to Mozilla bug No. 78414 for Linux users.

By using cnee to monitor system keyboard events, and with Perl to track Firefox application status, Firefox hotkey functionality can be restored even when a Flash player is focused.

Hardware and software requirements

Linux is required, with Firefox V2 or later. The libXnee and cnee components are necessary to monitor systemwide keyboard events, and Perl handles the algorithms. Certain Perl modules are required to process the cnee output and send X Window System events: threads, Thread::Queue, X11:GUITest and Time::HiRes. See Resources for information about these modules and the libXnee software package.

Although implemented on Linux, the general concepts presented here are applicable to multiple operating platforms, such as Microsoft® Windows®. All that is required is a cnee replacement that can print out systemwide keyboard events reliably, as the Firefox extension and Perl code are cross-platform. (If you are a clever developer of Windows applications and create an open source fix similar to what's presented here, please send me an e-mail with your code, and we'll amend this article with proper credit for your work.)

Familiarity with Firefox extension programming will be helpful, as will the installation of the Extension Developer's Extension (see Resources).

Back to top

Create a Firefox keyboard reporting extension

Determining when a Flash player has grabbed a Firefox hotkey (such as Ctrl+t), is performed in two parts. The first part is recording when the location bar text has changed in Firefox. The location bar text changes each time a tab is opened, a new page is visited, or a different tab is displayed. The second part is monitoring keyboard events across the entire system. If a keyboard combo (such as Ctrl+t) is recognized by cnee, yet the last location bar text change was X seconds ago, the Flash player has grabbed the keyboard focus.

A simple method for determining the location bar change is presented in the Mozilla Developer's Center Progress Listeners page. To implement similar code, download the pre-built extension from the Google Calendar Encryption article (see Resources). Extract the extension directory and change the contents of the install.rdf file, as shown below.

<?xml version="1.0"?>
<RDF:RDF xmlns:em="http://www.mozilla.org/2004/em-rdf#"
         xmlns:NC="http://home.netscape.com/NC-rdf#"
         xmlns:RDF="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
  <RDF:Description RDF:about="urn:mozilla:install-manifest"
                   em:id="flashUngrabber@devWorks_IBM.com"
                   em:name="flashUngrabber"
                   em:version="1.0.0"
                   em:creator="Nathan Harrington"
                   em:description="flashUngrabber">
    <em:targetApplication RDF:resource="rdf:#$9ttCz1"/>
  </RDF:Description>
  <RDF:Description RDF:about="rdf:#$9ttCz1"
                   em:id="{ec8030f7-c20a-464f-9b0e-13a3a9e97384}"
                   em:minVersion="1.5"
                   em:maxVersion="3.0.5" />
</RDF:RDF>

Replace the chrome.manifest file with the contents of Listing 2.

content flashUngrabber  chrome/content/

overlay chrome://browser/content/browser.xul  \
  chrome://flashUngrabber/content/overlay.xul

locale  flashUngrabber  en-US   chrome/locale/en-US/

skin    flashUngrabber  classic/1.0     chrome/skin/
style   chrome://global/content/customizeToolbar.xul  \
  chrome://flashUngrabber/skin/overlay.css

Note that the backslash characters (\) are for line continuation only and should not be placed in the file. After replacing the extension metadata as shown above, erase the chrome/content/overlay.js file, and insert the contents shown below.

//overlay.js for flash "ungrabber" borrows heavily from
//https://developer.mozilla.org/en/Code_snippets/Progress_Listeners

var myExt_urlBarListener = {
  QueryInterface: function(aIID)
  {
   if (aIID.equals(Components.interfaces.nsIWebProgressListener) ||
       aIID.equals(Components.interfaces.nsISupportsWeakReference) ||
       aIID.equals(Components.interfaces.nsISupports))
     return this;
   throw Components.results.NS_NOINTERFACE;
  },

  // switching through tabs changes the location bar
  onLocationChange: function(aProgress, aRequest, aURI)
  {
    myExtension.updateFile();
  },

};

Copied almost directly from the Progress Listeners example, the myExt_urlBarListener function queries the available interfaces to make sure the onLocationChange capability is available. Each time the location bar changes, the myExtension.updateFile() function will be called. Listing 4 shows the updateFile and init/unint functions, which are added to the bottom of the overlay.js file.

  
var myExtension = {

  init: function() {
    // add the listener on web page loaded
    gBrowser.addProgressListener(myExt_urlBarListener,
        Components.interfaces.nsIWebProgress.NOTIFY_STATE_DOCUMENT);
  },

  uninit: function() {
    // remove the listener when page is unloaded
    gBrowser.removeProgressListener(myExt_urlBarListener);
  },

  updateFile: function() {

    // write the epoch seconds + precision when the location bar was changed
    locTime = new Date().getTime();

    var fileOut = Components.classes["@mozilla.org/file/local;1"]
                        .createInstance(Components.interfaces.nsILocalFile);
    fileOut.initWithPath("/tmp/locationBarChange");
    var foStream = Components.classes["@mozilla.org/network/file-output-stream;1"]
                            .createInstance(Components.interfaces.nsIFileOutputStream);
    foStream.init(fileOut, 0x02 | 0x08 | 0x20, 0666, 0);
    foStream.write(locTime.toString(), locTime.toString().length);
    foStream.close();
  }

};

Implementing simple interprocess communication is performed by writing out the last update time (in seconds since the UNIX® epoch) to the /tmp/locationBarChange file. Add the lines in Listing 5 to ensure the extension is loaded and unloaded correctly.

  
window.addEventListener("load", function() {myExtension.init()}, false);
window.addEventListener("unload", function() {myExtension.uninit()}, false);

To finish the extension updates, replace the contents of the chrome/content/overlay.xul file with the contents below.

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://quickgooglecal/skin/overlay.css" type="text/css"?>
<!DOCTYPE overlay SYSTEM "chrome://quickgooglecal/locale/overlay.dtd">
<overlay id="helloworld-overlay"
         xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <script src="overlay.js"/>
</overlay>

The extension is now ready to be loaded into Firefox. A simple method is to create an xpi by changing to the extensions' root directory and issuing the command zip -r flashUngrabber.xpi *. Load the xpi (also available in the Download section) in Firefox and restart the browser.

When the reload is complete, issue the following command perl -e 'while(1){print `cat /tmp/locationBarChange` . "\n";sleep(1)}'. Make the Firefox window viewable, and create and load pages in different tabs. You should see increasing numbers printed once a second if you change tabs, load a different page, or otherwise change the text in the location bar.

Back to top

flashUngrabber.pl program

Now that it can be determined when Firefox responds to a hotkey press, we can write a program to detect when it should have responded, but did not. The flashUngrabber.pl program, shown in Listing 7, handles this process.

#!/usr/bin/perl -w
# flashUngrabber.pl - monitor keyboard events, send firefox key combos
use strict;  
use X11::GUITest qw( :ALL );                # make sure firefox app has focus
use Time::HiRes  qw( gettimeofday usleep ); # sub second timings
use threads;                                # for asynchronous pipe reads
use Thread::Queue;                          # for asynchronous pipe reads
  
my $padLen = 16;       # epoch significant digits
my $foundControl = 0;  # loop control variable
my $setLocalTime = 0;  # last recorded synthetic event 
my %keys = ();         # key codes and times
  
my ($currWind) = FindWindowLike( 'Mozilla Firefox' );
die "can't find Mozilla Firefox window\n" if ( !$currWind );

The code above includes necessary modules and defines variables. Certain "special" characters, such as "—" (an "m-dash" in typography parlance) present in the Firefox application name may cause the FindWindowLike function to fail. Try loading a different page or switching to a different tab if flashUngrabber.pl can't find your Firefox application ID. Listing 8 continues with the program setup.

my $cneeCmd  = qq{ cnee --record --keyboard | };
my $pipeCnee = createPipe( $cneeCmd ) or die "cannot create cnee pipe\n";

$keys{ "ctrl-t" }{ cneeCode } = '0,2,0,0,0,28';
$keys{ "ctrl-t" }{ sendKeys } = '^(t)';
$keys{ "ctrl-t" }{ event } = 0;

$keys{ "ctrl-w" }{ cneeCode } = '0,2,0,0,0,25';
$keys{ "ctrl-w" }{ sendKeys } = '^(w)';
$keys{ "ctrl-w" }{ event } = 0;

After creating a connection to the cnee program in keyboard monitor mode, special key codes are defined in the %keys hash. These keys will be searched for later in the program and their last recorded time stored as the value of the "event" hash element. Listing 9 shows the beginning of the main program loop.

while( 1 )
{
  # read all data off the cnee output queue, process each line.  cnee data 
  # needs to be control first, then the very next line be the key like: ctlr-t
  my $cneeData =  "";
  while( $pipeCnee->pending ){ $cneeData .= $pipeCnee->dequeue or next }

  for my $line ( split "\n", $cneeData )
  {
    if( $foundControl == 1 )
    { 
      $foundControl = 0;
      for my $name( keys %keys )
      { 
        next unless ( $line =~ /$keys{$name}{"cneeCode"}/ );
        $keys{$name}{"event"} = getTimeStr();

      }#for each key

    }#if control pressed

    if( ($line =~ /0,2,0,0,0,37/) || ($line =~ /0,2,0,0,0,109/) )
    {
      # control pressed
      $foundControl = 1;

    }elsif( ($line =~ /0,3,0,0,0,37/) || ($line =~ /0,3,0,0,0,109/) )
    {
      #control released
      $foundControl = 0;

    }#if control pressed

  }#for each line

Depending on system load and a variety of other factors, keyboard events can be processed by Firefox a substantial amount of time before they reach the cnee program. Conversely, it's possible for cnee to print out keyboard events before Firefox has a chance to process the keystrokes. This particular infinite loop and micro-sleep approach is designed to allow cnee and Firefox to have a chance to process events, while retaining adequate UI performance.

At each pass of the main loop, the cnee output (if it exists) is processed to find the control key. If a control key is found, and the next key pressed is specified in the %keys hash, the event time for that key is recorded. Listing 10 shows the continuation of the main processing loop after the cnee events are read.

  my $curTime = getTimeStr();

  for my $name ( keys %keys )
  { 
    # require the event to have .5 second to bubble up to cnee
    next unless ( ($curTime - $keys{$name}{"event"} ) > 500000 &&
                  $keys{$name}{"event"} != 0 );

    # reset the event time
    $keys{$name}{"event"} = 0;

    next unless ( $currWind == GetInputFocus() ); # skip if firefox not focused

    next unless( -e "/tmp/locationBarChange" );   # skip if no address bar data

    open( FFOUT, "/tmp/locationBarChange" ) or die "no location bar file";
      my $ffTime = <FFOUT>;
    close(FFOUT);

    # determine if firefox has written a location bar change recently 
    $ffTime = $ffTime . "0" x ( $padLen - length($ffTime) );
    if( $ffTime > $setLocalTime ){ $setLocalTime = $ffTime }

    # if it's been more than two seconds since last event
    next unless( ($curTime - $setLocalTime)  > 2000000 );

Each key code is processed to find if at least a half-second has gone by since the event was detected. If Firefox has the focus currently, the /tmp/locationBarChange file exists, and it has been at least two seconds since the last synthetic event was sent, processing continues as shown below.

    # record original mouse position
    my($origX,$origY) = GetMousePos();
    my( $x, $y, $width, $height ) = GetWindowPos( $currWind );

    # highly subjective, clicks in google search box on default firefox 
    # installation.  Sleeps are ugly, but help ensure inputs trigger 
    # correctly on a heavily loaded machine
    ClickWindow( $currWind, $width-150, $height-($height-40) );
    usleep(200000);
    SendKeys( $keys{$name}{"sendKeys"} );
    usleep(200000);
    MoveMouseAbs( $origX, $origY );
    usleep(200000);
    $setLocalTime = $curTime;

  }#for each key combo to look for

  usleep(100000); # wait a tenth of a second

}#while main loop

By this point, a synthetic event needs to be sent, so the current mouse position and window position data are recorded. Just sending a Ctrl+t at this point will not create the desired behavior, as the Flash player will simply absorb the keypress. Moving the mouse, clicking the window (in the Google Search box, for example), and sending the Ctrl+t is the most reliable method of ensuring that the keystrokes are processed by Firefox. Moving the mouse back to the original position prior to the keypress makes sure the mouse is put back where you left it.

Heavy system load and a variety of other factors can affect when Firefox receives the mouse move and keyboard events. Reducing or removing the usleep function calls can improve the speed of the keypress event sending, but may cause other issues when the system is responding sluggishly.

Changes to the ClickWindow coordinates may be required if you have a nonstandard Firefox toolbar setup or would like to ensure that the synthetic clicks are sent to a different location in your browser. Listing 12 shows the createPipe and getTimeStr supporting subroutines.

sub createPipe
{ 
  my $cmd = shift;
  my $queue = new Thread::Queue;
  async{
      my $pid = open my $pipe, $cmd or die $!;
      $queue->enqueue( $_ ) while <$pipe>;
      $queue->enqueue( undef );
  }->detach;

  #detach causes the threads to be silently terminated on exit (sometimes)
  return $queue;

}#createPipe

sub getTimeStr
{
  # i suppose the idea of not providing standard length time strings makes
  # sense... somewhere, this is not one of those times
  my   ($seconds, $microseconds) = gettimeofday;
  my $text = "$seconds$microseconds";
  return( $text . "0" x ($padLen - length($text)) );

}#getTimeStr

The createPipe subroutine creates a nonblocking pipe read from the cnee program, and getTimeStr provides a consistent length high-precision time string. Save the above listings as the flashUngrabber.pl program and run the program with the following command: perl flashUngrabber.pl.

Back to top

Usage

Test your configuration by loading a Flash content player, such as a YouTube video. If you click inside the Flash player — such as on the volume control — and press Ctrl+t, you should see the mouse move to the Google search box, a new tab created, then the mouse return to its previous coordinates.

Back to top

Conclusion, further examples

With the code and tools presented here, you can restore your favorite Firefox hotkeys from the grasp of Flash. Consider adding cnee key codes to the flashUngrabber.pl program to enable further keyboard navigation, such as Ctrl+tab for next tab, or Ctrl+l for address bar access. Reclaim your PgUp and PgDn keys from the Flash player to scroll the entire page, or add the cnee --record --mouse option to re-enable your scroll wheel.

Back to top

Download

	Information about download methods

Resources

• This article addresses some of the issues discussed in Mozilla bug No. 78414, first reported in May 2001: Application shortcut keys (keyboard commands like F11, Ctrl+t, Ctrl+r) fail to operate when plug-in (Flash, Acrobat, Quicktime) has focus at Bugzilla@Mozilla.
  
  
• Learn how to install Perl modules and read more about Perl.
  
  
• Read the developerWorks article "Integrate encryption into Google Calendar with Firefox extensions," download the source code archive, and extract it to a directory of your choice.
  
  
• Use Xnee and cnee for monitoring mouse events.
  
  
• Learn more about Progress Listeners, which allow extensions to be notified of events associated with documents loading in the browser and with tab-switching events, at the Mozilla Developer Center Code snippets site.
  
  
• To listen to interesting interviews and discussions for software developers, check out developerWorks podcasts.
  
  
• Stay current with developerWorks' Technical events and webcasts.
  
  
• Check out upcoming conferences, trade shows, webcasts, and other Events around the world that are of interest to IBM open source developers.
  
  
• Visit the developerWorks Open source zone for extensive how-to information, tools, and project updates to help you develop with open source technologies and use them with IBM's products.
  
  
• Watch and learn about IBM and open source technologies and product functions with the no-cost developerWorks On demand demos.
  
  
• Learn how to install Perl modules and read more about Perl.
  
  

• UNIX and Linux users: If you're new to installing Perl modules, Andreas J. Konig's CPAN module automates the installation of other modules.
  
  
• You need the threads, Thread::Queue, and Perl/Tk.
  
  
• Get the Extension Developer's Extension, a suite of tools for extension developers available at the Mozilla Developer Center Extensions site.
  
  
• Innovate your next open source development project with IBM trial software, available for download or on DVD.
  
  
• Download IBM product evaluation versions, and get your hands on application development tools and middleware products from DB2®, Lotus®, Rational®, Tivoli®, and WebSphere®.
  
  
• Download Perl, or read more about Perl at Perl.org.

• Participate in developerWorks blogs and get involved in the developerWorks community.
  
  

About the author

		Nathan Harrington is a programmer working with Linux at IBM. You can find more information about him at nathanharrington.info.

Rate this page

Please take a moment to complete this form to help us better serve you. Did the information help you to achieve your goal? Yes No Don't know   Please provide us with comments to help improve this page:   How useful is the information? 1 2 3 4 5 Not useful Extremely useful

Share this.... Digg this story del.icio.us Slashdot it!

Back to top