Use your Wii-Remote with the FlashPlayer – WiiFlash

You maybe know the Nintendo Wii and its controller the Wiimote. Joa Ebert and Thibault Imbert implement a AS3 package called WiiFlash which allows you to connect the Wiimote ( or Wii Remote Control ) with your Flash/Flex Application.

The Wiimote uses several sensors to detect different motion values. It allows you to read the acceleration value for X-,Y-, Z-axis and Roll, Pitch and Yaw, which is the rotation around the x-,y- and z-axis. So you have a lot more degrees of freedom than with a simple mouse or keyboard. A funny thing you could implement with this for example is WiiSpray, which also uses the WiiFlash Lib.

The WiiFlash package consists of two parts. The first one is the WiiFlash Server. It’s written in C++ / .NET and handles the communication with the Wiimote. The second part is the WiiFlash ActionScript API, which gives you methods and events to get and handle the Wiimote data.

The Wiimote communicates via Bluetooth with its endpoint. So the WiiFlash Server connects to your PCs or Mac Bluetooth stack and reads the data from the Bluetooth connection. This data are send via Socketconnection to the Flash Player. The WiiFlash AS3 API now converts the Wiimote binary protocoll into useable data. Which means you could directly read the values and also get update and/or change events if anything happens.

Connect the Wiimote
The WiiServer is available for PC or Mac. At first you have to connect your Wiimote with your PC or Mac. That means you have to create a Bluetooth connection between them – pair them via Bluetooth. After this step the WiiFlash Server should be able to find the Bluetooth connection and create the connection to the Flash Player. Find a detailed description how to connect your Wiimote with your PC here.

After you could pair your PC/Mac with the Wiimote, disconnect the Wiimote and start the WiiFlash Server. When the server started connect your Wiimote again ( this is done by pressing the Buttons „1“ + „2“ of your Wiimote at the same time ). After a short time you should see the connected Wiimote at the WiiFlash Server. It shows you the Batterylevel and the Wiimote-ID. Both Server versions could connect up to four Wiimotes at the same time.

I did not test the WiiFlash under Windows Vista. But wit OSX 10.5.x you’ll get an Java error when you start the server:

Cannot launch Java application.  Uncaught exception in main method:
java.lang.IllegalStateException: Bluetooth failed to initialize.
There is probably a problem with your local Bluetooth stack or API.

You just have to declare the WiiFlash Server as 32bit application. Just open the file properties ( Apple + i ) and check the 32bit-mode checkbox. This will fix the problem. ( Otherwise the application is launched as 64bit application ).

Unfortunately both server versions aren’t that stabel. From time to time the server crashes or the connection between server and Flash Player doesn’t seem to work any more. You just have to close the server, disconnect the Wiimote from your PC or Mac and restart the connection. This also helps if the rumble function or the LED setter method seem to quit their work.

Connect the Wiimote with your Flex App
The WiiFlash AS3 API gives you a very clear set of methods, which gives you fully access to the Wiimote data. At first you have to create an instance of the Wiimote class. Now you could create the connection to the Wiimote and access the data.

import org.wiiflash.Wiimote;
 
private var wiimote:Wiimote;
 
public function init():void{
	wiimote = new Wiimote();
 
	/* use Event.Connect NOT WiimoteEvent.CONTROLLER_CONNECT */
	wiimote.addEventListener( Event.CONNECT, connectHandler );
 
	wiimote.connect();
}
 
private function connectHandler( evt:Event ):void{
	trace( "Wiimote is connected" );
}

Read Wiimote data
The accelerator values could be accessed on to different ways. First you could read the data directly from the Wiimote-instance. This is helpful if you have an timed update interval, which handles the data. For example if you use an the enterFrame event to call a method which handles the Wiimote data:

public function init():void{
	wiimote = new Wiimote();
 
	/* get Wiimote data onEnterFrame */
	this.addEventListener( Event.ENTER_FRAME, enterFrameHandler );
 
	wiimote.connect();
}
 
private function enterFrameHandler( evt:Event ):void{
	trace( wiimote.sensorX );
        trace( wiimote.roll );
}

The common way is to use the Wiimote Update event. This event is fired on each update update call from the Bluetooth stack. ATTENTION: The communication works with about 100Hz. This means you will get about 100 datasets per second. This is important for how you will later work with your data, and have to be aware of performance issues if you might bad datastructures or uneffective datahandling methods.

Write Wiimote data
You could not only read values from the Wiimote. You could also write two values of the Wiimote. This are the led and the rumble attribute. With the led attribute you should be able to specify which led-light on the Wiimote shines. This is not really working well, but works sometimes …
You could also use the Force-Feedback of the Wiimote. Just set the rumble value to true and the Wiimote start vibrating. If its used this way, its vibrating all time. You could also set the rumbleTimeout value. The parameter is the time in milliseconds the Wiimote starts to rumble.

/* set LEDs */
wiimote.leds = Wiimote.LED4;
 
/* set Force Feedback - for positiv direction */
// wiimote.rumble = ( wiimote.sensorX>3 || wiimote.sensorY>3 || wiimote.sensorZ>3 );
if( wiimote.sensorX>3 || wiimote.sensorY>3 || wiimote.sensorZ>3 ){
	wiimote.rumbleTimeout = 1000;
}

Interprete the Wiimote data
The Wiimote controller returns the acceleration concerning to the gravity value g. 1g means a gravity acceleration of 9.82m/s². The controller itself is able to detect values between -3.6g and +3.6g.

If the Wiimote will not be moved and rests still on the table, you will always have on value (sensorX,sensorY,sensorZ) which is -1 or +1. This is the gravity affecting on the Wiimote. This is the criteria to detect if you Wiimote is in motion or not. Therefore you could use the Euclid Normal Form. If you take the root of the sum of all squares, this will be near +1. Normally you will define a constant values which is a bit higher than exact +1 to ignore very little movements of the Wiimote.

/* const to detect motion */
private const THRESHOLD:Number = 1.01;
 
private function wiimoteUpdateHandler( evt:WiimoteEvent ):void{
 
	_x = evt.target.sensorX;
	_y = evt.target.sensorY;
	_z = evt.target.sensorZ;
 
	var euLength:Number;
 
	/* check for "NO MOTION" */
	euLength = Math.sqrt( _x*_x + _y*_y + _z*_z );
 
	/** set Force Feedback - for positiv direction **/
	wiimote.rumble = ( _x>3 || _y>3 || _z>3 );
 
	if(euLength>THRESHOLD){
		/* do something */
	}
}

Optimize Wiimote date
Normally you won’t directly use the rough data from the Wiimote. Because the detection is relative precise you will get a lot of little peaks and a very unsmooth curve of data. If you want to detect a specific gesture or just want to transform the Wiimote motion into a motion of objects in the Flash you should parse the data. For smoothing the input data you should use a lowpass filter. It will smooth the curve by multiply the actual value with older values. In the following example a new value will only enter 10%. 90% of the value is taken from the old value.

/* const for filtering values */
private const FILTER_FACTOR:Number = 0.1;
 
/* high pass filter */
private function highPassFilter( oldValue:Number, value:Number ):Number{
	return value - lowPassFilter( oldValue, value );
}
 
/* low pass filter */
private function lowPassFilter( oldValue:Number, value:Number ):Number{
	return  value * FILTER_FACTOR + oldValue * (1.0 - FILTER_FACTOR );
}
 
private function wiimoteUpdateHandler( evt:WiimoteEvent ):void{
	_x = this.lowPassFilter( _x, evt.target.sensorX );
	_y = this.lowPassFilter( _y, evt.target.sensorY );
	_z = this.lowPassFilter( _z, evt.target.sensorZ );
 
	/* do something with the data */
}

Gesture detection
If you know have smooth data, which you only will process if the Wiimote is really in motion, you could start to detect gestures. One of the very simple gestures is a shift. If you just shake the Wiimote a little step right it should detect a rightshift and maybe open the next Powerpoint slide. If you shift to left left the application should go on step back.

For this examples you have to watch the x sensor values of the Wiimote. Because you got the accelation of the controller you will not just get on positive peak in x-direction, but you will get a positive peak in x-direction followed by a negative peak in the x-direction. This is because you start a motion in positive x-direction but than stop to motion, which is a negative accelaration. An right shift is a positive x peak followed by a negative x peak. Vice versa a left shift is a negative x peak followed by a positive x peak.

So the first thing to implement is a peak detection class. I do this by writing several values to a history array ( for example of the last 40 values ). I now compare the avarage of the last 2 or 3 values to the avarage of all history values. If the difference between the two avarage values exceed a specific value ( depending on how exact my motion detection should work ) a peak is detected.

/**
 * @classdescription: peak detection. detects a peak in an array of values
 * @author: Sebastian Martens - http://blog.sebastian-martens.de/
 * @SVN: $Id$
 *
 */
public class PeakDetect{
	private var _history:Vector.<number>;
	private var _historyLength:int;
	private var _kernelLength:int;
	private var _peakFactor:Number;
	private var _lastPeak:Boolean;
 
	/**
	 * @constructor
	 * @param kernelLength Number of values which represent the actual motion
	 * @param historyLength Number of values which are taken as history of motion
	 * @param peakFactor If distance between history and actual motion greater than this value motion is detected
	 *
	 */
	public function PeakDetect( kernelLength:int=2, historyLength:int=32, peakFactor:Number=0.2 ){
		var i:int;
 
		_history = new Vector.</number><number>();
		_kernelLength = kernelLength;
		_historyLength = historyLength;
		_peakFactor = peakFactor;
 
		_lastPeak = false;
 
		// init with 0
		for(i=0;i<_historylength ;i++){
			_history[i] = 0;
		}
	}
 
	/**
	 * calculates the intersection value of items in the array
	 * @param startIndex to get avarage value from the history list
	 * @param length number of values which should calculate to avarage value
	 * @return avarage value
	 */
	private function _getIntersection( startIndex:int, length:int ):Number{
		var i:int;
		var mValue:Number=0;
 
		for(i=startIndex;(i<(startIndex+length)&&i<_historyLength);i++){
			mValue += _history[i];
		}
 
		return (mValue/length);
	}
 
	/**
	 * tries to detect a peak in the list of value
	 * @return true if a peak was detected
	 *
	 */
	private function _checkForPeak():Number{
		var pValue:Number;
		var aValue:Number;
		var peakVal:Number;
		var multiPl:int;
 
		pValue = _getIntersection(0,_historyLength);
		aValue = _getIntersection( _historyLength-_kernelLength, _kernelLength );
 
		if( aValue>0 ){
			peakVal = (aValue-pValue);
			multiPl = 1;
		}else{
			peakVal = (pValue-aValue);
			multiPl = -1;
		}
 
		return ( peakVal>_peakFactor )?(peakVal*multiPl):0;
	}
 
	/**
	 * Adds a new value to item history. Return true if peak is detected.
	 * @param value Number-
	 * @return Boolean -
	 */
	public function addValue( value:Number ):Number{
		_history.shift();
		_history[ (_historyLength-1) ] =  value;
 
		return this._checkForPeak();
	}
}
</_historylength></number>

Finally you could now decide on the order of the different peaks which type of gesture it should be. Therefore i use a timer. If one peak is detected the timer starts. If another peak is detected within the runtime of the timer a gesture is detected if the peak order fits to a gusture.

Please find attached a example FlashBuilder Project which consists serveral steps with examples. Including the WiiFlash framework and my GestureDetection library.

WiiFlashTest – FlashBuilder Project
com.nonstatics.GestureDetection – Gesture Detection Library for WiiFlash v0.1

cheers,
Sebastian