Controlling Your Android Phone with the Wave of a Finger

March 18, 2014

My definition of fun these days involves tinkering around with different technologies and seeing how I can play them off each other. Like a crazy conductor waving his arms wildly to get an orchestra to perform together, I type away like mad until an idea in my mind finally works.

In this article, I’ll be showing you how I’ve created an Android Call Controller that mutes the ring on an incoming call with the circling of a finger. We’ll be using the following technologies:

An Android device
A Leap Motion Controller
on{X}
Node.js
Express
A bit of jQuery
Heroku

The above tech makes up the three important pieces to the puzzle:

Node server – We’ll have a Node server running that is the bridge between the Android device and a very simple web page. It will track if we want the phone to go silent or not.
on{X} – When we get an incoming call, we’ll ask the phone to poll our server to see if there’s been a request to go silent. If so, it will mute itself and send a busy message.
Web page with the Leap Motion Controller – When we’ve got our web page open and we circle our finger over the Leap Motion, we send our server a request to go silent.

What is a Leap Motion Controller?

The Leap Motion Controller is an intriguing input device that can detect your hand and finger movements. It’s a revolutionary piece of tech that is just waiting for some incredible developer to come along and work pure magic. We’ll be using the LeapJS API from the Leap Motion team to control JavaScript applications.

on{X}

One of the key advantages I’ve found with JavaScript is the ease in which you can use the language to pair up completely unrelated devices. In this demo, we’ll be pairing the Leap Motion with on{X}. on{X} is an Android app with a JavaScript API that allows you to control and/or respond to events on your Android device. If you haven’t used on{X} before, I’ve covered the basics of it in my article “Controlling Web Pages with JavaScript and on{X}”. Check that out for a quick rundown.

The Node.js Server

The index.js file contains all our server side Node application code. All the code for our webpage will be within the public folder. This is the directory structure we’re aiming towards:

Project File Structure

There isn’t too much to declare in your package.json file (shown below) for this demo. We’ll be using it to declare dependencies for our server. The main dependency for our server in this case is Express.

The engines fields are nice to include when you deploy your app onto hosting services like Heroku. They’ll point out which version of Node and npm you’re expecting to use.

{
  "name": "LeapCallController",
  "version": "0.0.1",
  "dependencies": {
    "express": "3.1.x"
  },
  "engines": {
    "node": "0.10.x",
    "npm": "1.2.x"
  }
}

To run our app on Heroku, we need a Procfile, whose contents are shown below.

web: node index.js

The Server Code

The Node server is the bridge between our web page with the Leap Motion controller and our Android device. The server code is shown below.

var http = require('http'),
    express = require('express'),
    app = express(),
    server = require('http').createServer(app),
    port = process.env.PORT || 5000,
    call = {};
    call.sound = true;

app.use(express.bodyParser());

app.get('/', function(request, response) {
  response.sendfile('public/index.html');
});

app.post('/shouldibesilent', function(request, response) {
  console.log('That phone wants to know if it should be silent...', request);
  response.json({callSound: call.sound});
});

app.post('/call', function(request, response) {
  console.log('Something is setting the call to ' + request.body.action);

  switch (request.body.action) {
    case 'mute':
      call.sound = false;
      break;
    case 'reset':
      call.sound = true;
      break;
  }

  response.json({success: true, actionReceived: request.body.action});
});

app.get(/^(.+)$/, function(req, res) {
  res.sendfile('public/' + req.params[0]);
});

server.listen(port, function() {
  console.log('Listening on ' + port);
});

We create an object named call to store information about the state of the call. call.sound is a Boolean that indicates whether or not we’ve got a request to turn the sound off (mute the phone). In our demo, we’ll only ever use call.sound, however I’ve placed it in an object so that extending the functionality of the app in future will be simple.

Communicating with Android

The following route will be used to tell our Android device what the value of call.sound is. I’ve used a JSON response as I’ve found that it seems to work best with on{X} Ajax requests. While debugging, I found it quite handy to log these requests on the server using console.log().

app.post('/shouldibesilent', function(request, response) {
  console.log('That phone wants to know if it should be silent...', request);
  response.json({callSound: call.sound});
});

Interfacing with Leap Motion

The POST route at /call is responsible for handling requests for the phone to take action. We’ll be sending a request to mute the phone and thus set call.sound to false. The code responsible for handling this is shown below.

app.post('/call', function(request, response) {
  switch (request.body.action) {
    case 'mute':
      call.sound = false;
      break;
    case 'reset':
      call.sound = true;
      break;
  }

  response.json({success: true, actionReceived: request.body.action});
});

The Client Side JavaScript Code

The index.html page in our public directory is mostly dull and uninteresting to anyone who visits at the moment. You could expand it to show a dashboard of information about incoming calls or a visualization for when you’re making different gestures to give the user a bit of feedback on whether they’re making a gesture successfully. We won’t go into that in depth in this demo.

In our demo today, we’ll be focusing on the JavaScript that provides the Leap Motion input. I’ve included jQuery solely to use the Ajax functionality, however you could do this in vanilla JavaScript to the same effect.

<script src="jquery-1.7.2.min.js"></script>

I’ve also included Underscore with LeapJS, as I’ve found some versions of the LeapJS API require it:

<script src="js/underscore.min.js"></script>
<script src="js/leap.min.js"></script>

The front end JavaScript code is shown below. To keep this tutorial simple, the JavaScript has been placed inline with the HTML.

var controller = new Leap.Controller({enableGestures: true}),
    callMuteRequestMade = false;

controller.loop(function(frame) {
  var gesture  = frame.gestures[0],
      type = gesture ? gesture.type : '';

  if (type == 'circle') {
    console.log('Circle');

    if (!callMuteRequestMade) {
      // Only ask it to mute once!
      callMuteRequestMade = true;

      $.ajax({
        url: '/call',
        type: 'POST',
        data: {
          action: 'mute'
        }
      });
    }
  }
});

controller.on('ready', function() {
  console.log('ready');
});
controller.on('connect', function() {
  console.log('connect');
});
controller.on('disconnect', function() {
  console.log('disconnect');
});
controller.on('focus', function() {
  console.log('focus');
});
controller.on('blur', function() {
  console.log('blur');
});
controller.on('deviceConnected', function() {
  console.log('deviceConnected');
});
controller.on('deviceDisconnected', function() {
  console.log('deviceDisconnected');
});

The following line sets up our Leap Motion controller and enables gestures so that we can detect the circling of a finger. You can also use gestures to detect swipes and finger taps. We’ll use the controller variable to interact with the Leap Motion from this point on:

var controller = new Leap.Controller({enableGestures: true})

The various controller.on() functions near the end of the JavaScript are there for debugging purposes. Each one lets us know when the state of our Leap Motion device changes.

We’re mainly interested in the controller.loop() function which runs repeatedly on each frame the Leap Motion detects. According to the API docs, this is about sixty times a second. Keep this in mind if you’re doing anything too resource intensive, as it will be running frequently!

In our code, each frame within controller.loop() checks for any gestures that the Leap Motion has picked up. frame.gestures contains an array of data for each hand. frame.gestures[0] means we’re only picking up the first hand. gesture.type will then let us know what gesture was picked up:

var gesture  = frame.gestures[0],
    type = gesture ? gesture.type : '';

If the gesture is a circle, we see if a request to mute the phone has already been made. Once it has been made, we set callMuteRequestMade to true. This way we don’t send hundreds of these for every frame with a circling finger. The code that accomplishes this is shown below.

if (type == 'circle') {
  console.log('Circle');

  if (!callMuteRequestMade) {
    // Only ask it to mute once!
    callMuteRequestMade = true;
...

Finally, if it is the first time a request to mute the phone has been made, we make an Ajax POST request to the /call route we set up on our server.

The on{X} Code

We’ve got our server ready to intercept calls from our web page and our Android device. We also have our web page ready to send calls to mute the phone. There is one last bit to set up – our Android device. We’ll need to create a rule in on{X} and upload it to the phone.

For the Android device, we’ll be focusing on two event handlers in the on{X} API, device.telephony.on('incomingCall') and device.telephony.on('idle'). The first is fired whenever on{X} detects an incoming call on your device. The second is fired whenever the telephony functions of the device have gone idle (e.g. the phone has stopped ringing, we’re not making any outgoing calls, etc).

The full on{X} code is shown below.

var originalCallVolume,
    originalRingerMode,
    currentPhoneNumber,
    textMessageRequested = false;

device.telephony.on('incomingCall', function(signal) {
  originalCallVolume = device.audio.ringerVolume,
  originalRingerMode = device.audio.ringerMode;
  currentPhoneNumber = signal.phoneNumber;

  device.scheduler.setTimer({
    name: 'checkingForInCallInputs', 
    time: 0,
    interval: 5 * 1000,
    exact: false
  }, function() {
    checkIfPhoneShouldBeSilent();
  });
});

device.telephony.on('idle', function() {
  device.scheduler.removeTimer('checkingForInCallInputs');
  returnToPhoneDefaults();
});

function checkIfPhoneShouldBeSilent() {
  device.ajax({
    url: 'http://yourappurlhere.com/shouldibesilent',
    type: 'POST',
    dataType: 'json',
    data: '{"call":"incoming"}',
    headers: {'Content-Type': 'application/json'}
  }, function onSuccess(body, textStatus, response) {
    var JSONResponse = JSON.parse(body);

    console.info('successfully received http response!');
    console.info(JSON.stringify(JSONResponse));

    if (JSONResponse.callSound === false) {
      device.audio.ringerVolume = 0;

      if (!textMessageRequested) {
        textMessageRequested = true;
        device.messaging.sendSms({
          to: currentPhoneNumber,
          body: 'Sorry! In the middle of a technological breakthrough. I\'ll call you back!'
        }, function(err) {
          console.log(err || 'sms was sent successfully');
        });
      }
    }
  }, function onError(textStatus, response) {
    var error = {};

    error.message = textStatus;
    error.statusCode = response.status;
    console.error('error: ',error);
  });
}

function returnToPhoneDefaults() {
  device.audio.ringerVolume = originalCallVolume;
  device.audio.ringerMode = originalRingerMode;
  textMessageRequested = false;

  device.ajax({
    url: 'http://yourappurlhere.com/call',
    type: 'POST',
    dataType: 'json',
    data: '{"action":"reset"}',
    headers: {'Content-Type': 'application/json'}
  }, function onSuccess(body, textStatus, response) {
    var JSONResponse = JSON.parse(body);

    console.info('Successfully got a response after asking to reset the call state');
    console.info(JSON.stringify(JSONResponse));
  }, function onError(textStatus, response) {
    var error = {};

    error.message = textStatus;
    error.statusCode = response.status;
    console.error('error: ',error);
  });
}

Incoming Call Detection

Whenever we’ve got an incoming call, we store the phone’s current call volume and ringer mode. This way we can set them back to these settings after the call has finished ringing so that future calls still ring. We also store the phone number of the caller so we can SMS them after we’ve muted them:

device.telephony.on('incomingCall', function(signal) {
  originalCallVolume = device.audio.ringerVolume,
  originalRingerMode = device.audio.ringerMode;
  currentPhoneNumber = signal.phoneNumber;
...

We then run device.scheduler.setTimer(), which is quite similar to the native setTimeout() function in JavaScript. Here, we check if the phone should be silent every five seconds. The other fields do the following:

name: We set this to "checkingForInCallInputs" so that we have a name to refer to when removing the timer later.
time: The time in milliseconds since Jan 1st 1970 (Unix time starts from that date) that you want the timer to start at. We set it to zero, as we’re looking to time from this moment onwards.
interval: How many millisecond intervals we want the timer to call our function. I’ve set it to once every 5 seconds.
exact: Setting this to false enables a type of power optimization for repeating timers. I’m not sure to what extent it makes a noticable difference but figured it couldn’t hurt to have this one set.

The device.scheduler.setTimer() function with these fields looks like so:

device.scheduler.setTimer({
  name: 'checkingForInCallInputs', 
  time: 0,
  interval: 5 * 1000,
  exact: false
}, function() {
  checkIfPhoneShouldBeSilent();
});

The checkIfPhoneShouldBeSilent() function is quite a long one but it is your typical Ajax request. It makes POST requests to http://yourappurlhere.com/shouldibesilent with a simple JSON string letting our server know an incoming call is being made. You need to change the URL to the URL of your own server.

Ensure you’ve got the dataType and headers set to JSON so that on{X} sends the request in the correct format:

dataType: 'json'
headers: {'Content-Type': 'application/json'}

When we’ve successfully gotten a response from the server, we parse the data using JSON.parse(body):

onSuccess(body, textStatus, response) {
  var JSONResponse = JSON.parse(body);

We then check if the JSON response from the server said it wants the phone to mute. If so, we use the on{X} API’s device.audio.ringerVolume to set the volume of the ringer to 0:

if (JSONResponse.callSound === false) {
  device.audio.ringerVolume = 0;

We don’t want to be too rude and totally ignore this person, so we optionally send them an SMS using the device.messaging.sendSms function in the on{X} API. Recall that we stored their phone number in the currentPhoneNumber variable. We also ensure we only send one SMS by setting textMessageRequested to true:

if (!textMessageRequested) {
  textMessageRequested = true;
  device.messaging.sendSms({
    to: currentPhoneNumber,
    body: 'Sorry! In the middle of a technological breakthrough. I\'ll call you back!'
  }, function(err) {
    console.log(err || 'sms was sent successfully');
  });
}

Detecting when the Phone is Idle Again

When the phone is idle again, we remove the checkingForInCallInputs timer:

device.telephony.on('idle', function() {
  device.scheduler.removeTimer('checkingForInCallInputs');

We then run the returnToPhoneDefaults() function to set the call volume, ringer mode, and textMessageRequested back to their original values:

function returnToPhoneDefaults() {
  device.audio.ringerVolume = originalCallVolume;
  device.audio.ringerMode = originalRingerMode;
  textMessageRequested = false;

We also reset our server to no longer tell our phone to be silent on the next call by making another POST request to /call with the action of reset. We don’t use the success response for anything more than debugging at the moment:

device.ajax({
  url: 'http://yourappurlhere.com/call',
  type: 'POST',
  dataType: 'json',
  data: '{"action":"reset"}',
  headers: {'Content-Type': 'application/json'}
}, function onSuccess(body, textStatus, response) {
  var JSONResponse = JSON.parse(body);

  console.info('Successfully got a response after asking to reset the call state');
  console.info(JSON.stringify(JSONResponse));
}, function onError(textStatus, response) {
  var error = {};

  error.message = textStatus;
  error.statusCode = response.status;
  console.error('error: ',error);
});

Debugging with on{X}

If at any stage you’re looking to check if something has worked, there are two methods for debugging your on{X} code:

Log your JSON responses – console.info(JSON.stringify(JSONResponse));. You can see these by going to the Rules page in on{X}, clicking your rule and selecting the logs tab.
Create notifications on your phone to show when something has worked – device.notifications.createNotification("No longer in a call, I'll stop asking.").show();

The Final Result

With your server running on the web, your web page running in a browser, and your phone connected to the web, you should now be able to mute incoming calls. As a test of our code, I’ve gotten my lovely girlfriend to call my phone like so:

Incoming Call Made to Phone

As my phone rings, I mute the incoming call by circling my finger in the air like this:

Circling my finger above the Leap Motion to mute the incoming call

Which does result in a silenced phone and an SMS sent to my girlfriend letting her know I can’t answer the call just yet:

The SMS Response Received

Conclusion

There’s a lot of fun stuff that can be done by connecting up the JavaScript APIs of various devices. With each new device that releases a JavaScript API, the possibilities grow into an ever more thrilling future of interconnectivity! Get out there, integrate a few APIs, and make something new that hasn’t been done before. If anyone interrupts your focus with a phone call, just circle your finger and get back to work.

All the code for this tutorial is available on GitHub. The reader is encouraged to pull it down and experiment with it. For example, you could adjust the on{X} code to keep the phone muted, or add support for different gestures and actions. You can also incorporate Socket.IO to improve performance over the standard Ajax requests shown here.