Myo Armband controlling Lego Mindstorm EV3 with hand gestures

A couple of weeks ago I received the Myo Armband from Thalmic Labs and decided to connect it to my Lego Mindstorms EV3 and do my own proof of concept of this new technology. So I built a truck that can be controlled wirelessly with hand gestures. Following is a brief description of the project, video demonstration, source code and short user experience evaluation of Myo. Work hard, play hard 😉

Control Gestures

Raise your hand -> speed increase
Lower your hand -> speed decrease/switch to reverse gear
Fist Fist pose -> put the truck into sleep (stop moving)
Spread Fingers spread pose -> wake the truck up from sleep
Unlock Thumb to pinky pose -> close the program

Video

Hardware and Software

In order to be able to replicate my project with Myo and EV3, you will need to have:

Myo Armband (Developer Kit) – here is my previous article about Myo
Lego Mindstorms EV3

From the software dependencies:

Myo Connect (version 0.6.0)
myo-java
lejOS EV3 Runtime Library (version 0.8.1)

To build the EV3 RAC3 Truck, you can follow the official building instructions on the lego site. Then you need to install lejOS on your EV3. It’s quite simple and you will find all necesarry information and files here. I recommend you to download EV3 Eclipse plugin, which makes it very convenient to control your EV3 from Eclipse environment. Also, if you have the possibility to get NETGEAR Wifi Dongle for your EV3, it will allow you to connect to your EV3 wirelessly. Otherwise you can use Bluetooth or USB connection.

Architecture

The picture above describes the architecture I picked for this project. Although there are other options how to connect Myo to EV3:

Through PC, Mac, Android or iOS device
Install Myo drivers directly on EV3 brick with leoJS
NodeJS + websockets
more…

Following platforms are supported:

Mac, Windows, Linux
Android and iOS

User Experience Evaluation

After playing with Myo Developer Kit, I would like to share my personal experience with you. Myo provides us with two different types of data – orientation data (roll, pitch, yawl) and hand poses analyzed from muscle activity sensors. As far as orientation data is concerned, you won’t probably have any problems. The response from Myo is very accurate and fast. The problems appeared when I tried to include more hand poses into my gesture control system. Very quickly I realized that the way how I wear Myo matters a lot. It took me quite a long time to find out what position of Myo on my hand provides me with accurate results. Even slight rotation of Myo around the hand caused inaccurate results and poor responsiveness. It might be because of Myo doesn’t have muscle activity sensors along the whole inner surface, so I need to find the right position where Myo can successfully listen to the right muscles. So overally, I think Myo is a cool concept, but to be widely accepted it definitely needs to make hand poses recognition more accurate. I know that Myo is still in very early developmnent stage, so hopefully we will see some improvements in the future with the next releases.

Source Code

The source code is available at the myo-truck github repository. There are two classes in the project.

MyoTruck – Main class that handles connection to Myo, EV3 and does the main execution.
MyoDataCollector – This class collects orientation data and listens to gestures from your Myo.

MyoTruck.java

package net.havlena.myotruck;

import java.rmi.RemoteException;

import com.thalmic.myo.Hub;
import com.thalmic.myo.Myo;

import lejos.hardware.BrickFinder;
import lejos.hardware.BrickInfo;
import lejos.hardware.Button;
import lejos.hardware.Sound;
import lejos.hardware.lcd.GraphicsLCD;
import lejos.hardware.lcd.LCD;
import lejos.remote.ev3.RMIRegulatedMotor;
import lejos.remote.ev3.RemoteEV3;
import lejos.utility.Delay;

public class MyoTruck {
// Steering motor
private RMIRegulatedMotor motorS;
// Left motor
private RMIRegulatedMotor motorL;
// Right motor
private RMIRegulatedMotor motorR;
private RemoteEV3 ev3;
private GraphicsLCD lcd;
private boolean running = true;
private boolean sleeping = false;
//private int totalSteps = 0;
// In case your maximum steering angles are shifted to either left or right, you can correct it here.
private static int ANGLE_STEERING_CORRECTION = 50;
// We don't want to allow steering to maximum values
private static int STEERING_RESERVE_FROM_MAX = 20;
// The higher number, the more sensitive, 1 is the lowest
private static int STEERING_SENSITIVITY = 4;
// The higher number, the more sensitive, 1 is the lowest
private static int SPEED_SENSITIVITY = 2;
private int steeringScale;

public MyoTruck() throws RemoteException {
try {
BrickInfo[] bricks = BrickFinder.discover();
if (bricks.length==0) throw new RuntimeException("Unable to find an EV3!");
this.ev3 = new RemoteEV3(bricks[0].getIPAddress());
ev3.setDefault();
Sound.beep();
Button.LEDPattern(4);
lcd = ev3.getGraphicsLCD();
lcd.clear();
lcd.drawString("Myo Truck", LCD.SCREEN_WIDTH/2, LCD.SCREEN_HEIGHT/2, GraphicsLCD.BASELINE | GraphicsLCD.HCENTER);
lcd.drawString("Running...", LCD.SCREEN_WIDTH/2, LCD.SCREEN_HEIGHT/2+30, GraphicsLCD.BASELINE | GraphicsLCD.HCENTER);
motorS = ev3.createRegulatedMotor("A", 'M');
motorL = ev3.createRegulatedMotor("B", 'L');
motorR = ev3.createRegulatedMotor("C", 'L');

motorS.setSpeed((int)(motorS.getMaxSpeed()*0.8));
motorS.setAcceleration(2000);
// Reset front wheels to center position
motorS.rotateTo(-500);
int angleLeft = motorS.getTachoCount();
motorS.rotateTo(500);
int angleRight = motorS.getTachoCount();
steeringScale = (Math.abs(angleLeft)+Math.abs(angleRight))/2;
motorS.rotateTo(angleLeft+steeringScale+ANGLE_STEERING_CORRECTION);
steeringScale -= STEERING_RESERVE_FROM_MAX;
motorS.resetTachoCount();
motorS.rotateTo(-steeringScale);
motorS.rotateTo(steeringScale);
motorS.rotateTo(0);

motorL.setAcceleration(4000);
motorR.setAcceleration(4000);
motorL.setSpeed(0);
motorR.setSpeed(0);
motorL.stop(true);
motorR.stop(true);
} catch (Exception e) {
this.close();
e.printStackTrace();
}
}

private void close() {
Button.LEDPattern(5);
try {
if (motorS!=null) motorS.close();
if (motorL!=null) motorL.close();
if (motorR!=null) motorR.close();
} catch (Exception e) {
e.printStackTrace();
}
lcd.clear();
lcd.drawString("Myo Truck", LCD.SCREEN_WIDTH/2, LCD.SCREEN_HEIGHT/2, GraphicsLCD.BASELINE | GraphicsLCD.HCENTER);
lcd.drawString("Closing...", LCD.SCREEN_WIDTH/2, LCD.SCREEN_HEIGHT/2+30, GraphicsLCD.BASELINE | GraphicsLCD.HCENTER);
Delay.msDelay(3000);
lcd.clear();
lcd.refresh();
Button.LEDPattern(0);
}

private void run(Hub hub, MyoDataCollector dataCollector) throws RemoteException {
while (running) {
hub.run(1000/20);
System.out.print(dataCollector);
if (sleeping==true) {
try {
Thread.sleep(200);
continue;
} catch (InterruptedException e) {
e.printStackTrace();
}
}
/*totalSteps++;
if (totalSteps>100) {
running = false;
break;
}*/
System.out.print(dataCollector);
this.setSpeed(dataCollector.getPitch());
this.setSteering(dataCollector.getRoll(), dataCollector.isLeftArm());
}
motorL.stop(true);
motorR.stop(true);
}

private void setSpeed(double pitch) {
try {
int speed = 0;
int scalePitch = MyoDataCollector.SCALE/2/SPEED_SENSITIVITY;
int scaleMotor = (int)motorL.getMaxSpeed();
if (pitch>=0) speed = (int)((-scalePitch*SPEED_SENSITIVITY+pitch)*(scaleMotor/scalePitch));
motorL.setSpeed(speed);
motorR.setSpeed(speed);
if (speed>=0) {
motorL.backward();
motorR.backward();
} else {
motorL.forward();
motorR.forward();
}
} catch (RemoteException e) {
e.printStackTrace();
}
}

private void setSteering(double roll, boolean isLeftArm) {
try {
int steering = 0;
int scale = MyoDataCollector.SCALE/2/STEERING_SENSITIVITY;
if (roll>=0) steering = (int)((-scale*STEERING_SENSITIVITY+roll)*(steeringScale/scale));
if (isLeftArm) steering = steering*-1;
motorS.rotateTo(steering);
} catch (RemoteException e) {
e.printStackTrace();
}
}

public void stop() {
this.running = false;
}

public void sleep() {
lcd.clear();
lcd.drawString("Myo Truck", LCD.SCREEN_WIDTH/2, LCD.SCREEN_HEIGHT/2, GraphicsLCD.BASELINE | GraphicsLCD.HCENTER);
lcd.drawString("Sleeping...", LCD.SCREEN_WIDTH/2, LCD.SCREEN_HEIGHT/2+30, GraphicsLCD.BASELINE | GraphicsLCD.HCENTER);
Button.LEDPattern(6);
try {
motorL.setSpeed(0);
motorR.setSpeed(0);
} catch (RemoteException e) {
e.printStackTrace();
}
this.running = true;
this.sleeping = true;
}

public void start() {
Button.LEDPattern(4);
this.running = true;
this.sleeping = false;
}

protected void finalize() throws Throwable {
try {
this.close();
} finally {
super.finalize();
}
}

public static void main(String[] args) {
try {
MyoTruck myotruck = new MyoTruck();
Hub hub = new Hub("net.havlena.myo");
Myo myo = hub.waitForMyo(10000);
if (myo == null) {
throw new RuntimeException("Unable to find a Myo!");
}
MyoDataCollector dataCollector = new MyoDataCollector(myotruck);
hub.addListener(dataCollector);
myotruck.run(hub, dataCollector);
myotruck.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}

MyoDataCollector.java

package net.havlena.myotruck;

import com.thalmic.myo.DeviceListener;
import com.thalmic.myo.FirmwareVersion;
import com.thalmic.myo.Myo;
import com.thalmic.myo.Pose;
import com.thalmic.myo.Quaternion;
import com.thalmic.myo.Vector3;
import com.thalmic.myo.enums.Arm;
import com.thalmic.myo.enums.PoseType;
import com.thalmic.myo.enums.VibrationType;
import com.thalmic.myo.enums.XDirection;

public class MyoDataCollector implements DeviceListener {
public static final int SCALE = 20;
private double rollW;
private double pitchW;
private Pose currentPose;
private Arm whichArm;
private MyoTruck myotruck;
private boolean connected = false;

public MyoDataCollector(MyoTruck myotruck) {
rollW = 0;
pitchW = 0;
currentPose = new Pose();
this.myotruck = myotruck;
}

@Override
public void onOrientationData(Myo myo, long timestamp, Quaternion rotation) {
Quaternion normalized = rotation.normalized();

double roll = Math.atan2(2.0f * (normalized.getW() * normalized.getX() + normalized.getY() * normalized.getZ()), 1.0f - 2.0f * (normalized.getX() * normalized.getX() + normalized.getY() * normalized.getY()));
double pitch = Math.asin(2.0f * (normalized.getW() * normalized.getY() - normalized.getZ() * normalized.getX()));

rollW = ((roll + Math.PI) / (Math.PI * 2.0) * SCALE);
pitchW = ((pitch + Math.PI / 2.0) / Math.PI * SCALE);
}

@Override
public void onPose(Myo myo, long timestamp, Pose pose) {
currentPose = pose;
if (currentPose.getType() == PoseType.FIST) {
myo.vibrate(VibrationType.VIBRATION_MEDIUM);
myotruck.sleep();
}
if (currentPose.getType() == PoseType.FINGERS_SPREAD) {
myo.vibrate(VibrationType.VIBRATION_SHORT);
myotruck.start();
}
if (currentPose.getType() == PoseType.THUMB_TO_PINKY) {
myo.vibrate(VibrationType.VIBRATION_SHORT);
myotruck.stop();
}
}

@Override
public void onArmSync(Myo myo, long timestamp, Arm arm, XDirection xDirection) {
whichArm = arm;
}

@Override
public void onArmUnsync(Myo myo, long timestamp) {
whichArm = null;
}

@Override
public void onAccelerometerData(Myo myo, long timestamp, Vector3 accel) {
}

@Override
public void onConnect(Myo myo, long timestamp, FirmwareVersion firmwareVersion) {
connected = true;
}

@Override
public void onDisconnect(Myo myo, long timestamp) {
connected = false;
myotruck.stop();
}

@Override
public void onPair(Myo myo, long timestamp, FirmwareVersion firmwareVersion) {
}

@Override
public void onUnpair(Myo myo, long timestamp) {
myotruck.stop();
}

@Override
public void onGyroscopeData(Myo myo, long timestamp, Vector3 gyro) {
}

@Override
public void onRssi(Myo myo, long timestamp, int rssi) {
}

@Override
public String toString() {
StringBuilder builder = new StringBuilder("\r");

String xDisplay = "STEERING: ".concat(String.format("[%s%s]", repeatCharacter('*', (int) rollW), repeatCharacter(' ', (int) (SCALE - rollW))));
String yDisplay = " SPEED: ".concat(String.format("[%s%s]", repeatCharacter('*', (int) pitchW), repeatCharacter(' ', (int) (SCALE - pitchW))));

String armString = null;
if (whichArm != null) {
armString = " ARM: ".concat(String.format("[%s]", whichArm == Arm.ARM_LEFT ? "L" : "R"));
} else {
armString = " ARM: ".concat(String.format("[?]"));
}

String poseString = null;
if (currentPose != null) {
String poseTypeString = currentPose.getType().toString();
poseString = " POSE: ".concat(String.format("[%s%" + (SCALE - poseTypeString.length()) + "s]", poseTypeString, " "));
} else {
poseString = " POSE: ".concat(String.format("[%14s]", " "));
}
builder.append(xDisplay);
builder.append(yDisplay);
builder.append(armString);
builder.append(poseString);
return builder.toString();
}

private String repeatCharacter(char character, int numOfTimes) {
StringBuilder builder = new StringBuilder();
for (int i = 0; i < numOfTimes; i++) {
builder.append(character);
}
return builder.toString();
}

public double getPitch() {
if (!connected) return -1;
return pitchW;
}

public double getRoll() {
if (!connected) return -1;
return rollW;
}

public boolean isLeftArm() {
if (whichArm != null && whichArm == Arm.ARM_RIGHT) return false;
return true;
}
}

One Response to Myo Armband controlling Lego Mindstorm EV3 with hand gestures

Emily Stevenson 20.10.2015

Very interesting project! I work on a positioning project using Android from EV3, now cannot get the coordinates from Android. I am wondering if it is possible to talk to you about it? My email is grassyfu@gmail.com.