SmartHome User Research

For the SmartHome project, we’ve taken a step back to better understand the problem space and come at solutions based on research and evidence. As a team we spent last week interviewing people in their homes, with questions on a theme of freedom and independence. We choose early teens (12-15yrs) and post-retirement folks as a demographic that might have useful insights into this topic.

As a software engineer this has been an interesting process. There’s a temptation to jump into any new project with both feet and start hacking on code; starting with simple successes and working iteratively to add features and meet requirements. And we’ve done some of that in order to get familiar with what we expect will be our prototyping platforms - the raspberry pi and ESP8266. But the more we looked at the problem the more it became clear that we weren’t yet sure what technical questions the project would ask, let alone how to solve them. In the meantime, our team was trying to figure out a better vision for the smart home that would align with Mozilla’s values and potential solutions in a broad and confusing product space. None of us were in our comfort zone, so we made a determination to put roles aside and roll up sleeves and muck in. We’ve posted craigslist ads, we’ve had Skype calls, house visits, coffee shop rendezvous; interviewed, transcribed and now begun to process input from 15 different people.

It turns out that a curious mind, a knack for spotting patterns, analysing outcomes for the motivations and circumstances that produced them - these are the stock-in-trade of any software engineer - and they are skills that work just as well in user research and exploratory product definition as they do in software development. And perhaps more important than that, before we are engineers, we are people. We have families, jobs, aspirations, frustrations and concerns. We were young and hope to grow old. Talking to people this last week has been a great reminder that it is people that solve problems, not code.

Smart Mirror: Starting Up

With no keyboard or pointer inputs, ensuring the Smart Mirror can be restarted and booted up entirely automatically was high on my priority list. Once installed, I can’t startx or click on any icons; it needs to bring up all the backend services and the dashboard to leave it in a working state without any user intervention. That lead me down a merry path and was (for me) the trickiest part of this project.

Here’s the moving parts:

  • The kernel and OS itself, with networking and other key systems
  • The display and window manager - the subsystems that allow me to put my dashboard up on the screen
  • The mosquitto message broker
  • The gpio listeners
  • The web server
  • The browser, which should load up my dashboard URL

I went through lots of helpful posts and projects on creating linux kiosks to figure out potential approaches. While the mirror isn’t really a kiosk - a kiosk usually has keyboard/pointer/touch user input - its a reasonable match up. After a few false starts trying Firefox/Iceweasel and Chromium kiosk options, I settled on the approach outlined in this Dashing-Pi page. This eschews the LXDE desktop environment entirely and uses nodm and the matchbox window manager to boot into the browser with the minumum of unecessary fluff inbetween.

Orchestrating startup is a bit fiddly even so. First, nodm is configured to startup with/as the ‘pi’ user. The rest of the graphics/display related startup is then in a script copied to /home/pi/.xsession, which starts up the matchbox display manager, and the Uzbl browser to load the dashboard. For the backend pieces, Raspian uses the init.d system, so we install scripts in /etc/init.d/ to start up mosquitto, pm2 (which manages the node.js server(s)) and scripts to relay GPIO events as MQTT messages for the rest of the system.

That done, I can plug the thing in and in just a minute or so it brings up the dashboard on screen and responds to sensor events. The Uzbl browser is a wrapper around WebKit. It supports commands via a socket, which means that once up, I can ssh to the rPi and remotely refresh the page, navigate to other URLs and so on which has proved valuable during development as I have none of the traditional inputs (e.g ctrl+r on the keyboard) to accomplish this otherwise.

Smart Mirror: Data channels

One of the goals of this project is to get my hands dirty with integrating data and events coming from hardware (sensors attached to the raspberry pi or functions of the pi itself) and content in the browser. The raspberry pi ecosystem leans heavily on python, so while I was already using node.js for the web server, I didn’t want to get tied into a system which limited my implementation choices for publishing or consuming data. I wanted lightweight, language-agnostic messaging.

My first thought was to simply use the filesystem: writing to some files and watching for changes. That seemed fraught with potential problems and reinventing too much of the wheel. So I went shopping for messaging solutions.

After looking into ZeroQ and ZeroRPC, I settled on Mosquitto and MQTT. The Mosquitto project is a message broker (think central hub for dispatching messages to subscribers) that implements the MQTT protocol. As well as the broker (available as a raspbian-ready package) it also has command-line clients for publishing and subscribing which proved useful. There are MQTT implementations for both python and node.js (as well as a bunch of other languages.) The pub-sub paradigm turns out to be a great fit here. My “hello world” was to use the RPi.GPIO python module to watch for rising-edge button events on the GPIO pin I had jumpered a momentary button to, and publish a message over MQTT when the button is pressed. Logged into the raspberry pi, I run:

$ mosquitto_sub -t sensors/buttonup

.. to see those messages from the broker. In node.js land, the mqtt module provides the same functionality. We connect to the host/port the broker is on, subscribe to some topics and register callbacks for when messages arrive.

Finally, to bring the browser into the equation, I ended up making an adapter to subscribe to MQTT topics, and use to relay to the dashboard page. There is some (new) websocket support in mosquitto, and probably other more direct ways to accomplish this, but this works for now. As a proof of concept, I handle ‘gpio/button’ events from, and dispatch a ‘hardbuttonup’ event on the window object. A listener for this event toggles a class to flash the screen blue. This opens up a lot of opportunities as I have flexibility at every stage: the button is a stand-in for any sensor or input device that can toggle a GPIO pin high/low. The MQTT message produced can be caught by any program running on this device, or even anywhere on the network potentially. Turning this message into a DOM event enables a seamless tie-in so you can use existing frameworks to respond to the event.

Its a bit subtle, but in the video I’m clicking the button on the breadboard, and the display is flashing blue.

Smart Mirror: Getting Started on the Front-End

There are a few key requirements and moving parts to this project that I wanted to explore early on. I began at the end - the UI I wanted to show on the mirror. I initially tried out Atlasboard but quickly found it did more than I needed in some areas, and fought me to do what I wanted in others. To KISS, a ground-up implementation was going to be easier in the long run.

To get started I created a static page with a simple dashboard/widget system. It had a fixed pre-defined number of slots - a 3x3 grid of rows and columns - making layout and addressing really easiy with CSS and getElementById calls. This gave me a straight-forward way to get stuff on the screen. The widgets were a rudimentary base class implementing a init/update/render (and optionally poll to update and re-render) lifecycle.

On the server-side I made an “socket-board” express app to do reverse proxying, and keep things like API keys out of the client-side code. The name alludes to my eventual goal to hook it up to websockets for sending data between the client and data and events on device. All my feed/api requests from the browser go through the express app, and it sends on their response. To that end I created a basic config library and a request-forwarding middleware.

At this point, the dashboard runs anywhere with node.js and a modern browser.

scry-pi dashboard screenshot

Between the initial implementation and this screenshot, I’ve added a widget for the local IP address, which is particularly useful during development.

Smart Mirror Build Log: Intro

With the winding down of the Firefox OS / smartphone project, and a new interest in IoT and connected devices, I thought it would be fun and instructive to go out and build something connected. I settled on the “smart mirror” concept as a nifty thing that would be actually useful at home, and fun to make. I first saw this via hackaday featuring a Magic Mirror project. Its a doable project and should drop me in the middle of some more or less unfamiliar territory: getting hardware talking to software, lots of linux hackery, sensor inputs and gpio, and orchestrating moving parts up and down the stack. It should also be a good test platform to build on, to explore different technologies and interactions.

So, I have in mind a dashboard of sorts, displayed on a monitor/tv behind a two-way mirror. It should collect together some local and remote data of a sort that might be useful to know before you head out into the world: date/time, weather, appointments, actual (vs forecast) conditions and so on. As there wont be any keyboard/mouse/touch inputs, any direct user interactions (not sure yet what they would be if anything) would need to be hands-off - maybe voice input? some kind of presence or simply proximity detection to activate it?

I’m keeping notes as I go about this and I’ll post a series of updates with progress, dead-ends and observations. Some decisions I’ve already made - I’ll start out running this on a Raspberry Pi, the dashboard will be an HTML page in a fullscreen browser - others I’ll figure out along the way. In the past I’ve found that there is no substitute for actually making something to understand what the nature of the challenges will be, and its usually not what you expect. So, I expect to fall into all the traps, trip on all the hurdles and generally fumble my way though. Sounds like fun :)

The Story So Far

I’ve been enjoying responses to Crystal Beasley’s post about how she got into tech/development. It got me thinking about the diversity of backgrounds and paths people take to progamming and tech work; I know it never ceases to amaze me. Here’s my story.

Remove the blockages that keep IE6 around

With all the MS bashing and kill-IE6 sentiments I was wondering what are the actual impediments to upgrading? For the most part it is legacy applications in organizations that have no upgrade path. Stuff like timesheet apps, expense reporting and so on that was written eons ago that is critical to company operations, but has little or negative ROI to replace or fix.

Sure, there are cases where IE6 is in use simply because it was the browser that shipped with XP. Lets put aside the cost and difficulty of operating system upgrades, and focus on the part of the problem we can conceivably do something about. What prevents deployment of IE8 to these desktops? What compatibility problem with which applications are complicating this problem? Is there anything we can do to smooth the path to browser upgrades?

What’s needed is an audit. A big ugly audit of all the gnarly sticking points that IT organizations are faced with which cause the issue to be punted everytime it comes up. Odds are good that there are reasonable solutions to at least some of these problems if we only knew what they were. Perhaps its an upgrade or alternative with a tried and true migration path. Or just a simple code shim that can provide backwards-compatibility. I’m not discounting the ripple effect even a one-line code change can have, the risk and cost of regression, but we can offset that by connecting people in the same boat, by sharing gotchas and tried and tested solutions.

Imagine a clearing house kind of site that lists the offending applications & implementations and details the IE6 dependencies. The audit is crowd-sourced, built by the motivated individuals in every organization that deal with the problem every day but are powerless to fix it. Each problem once itemized, prioritized by the number of IE6 installs depending on it and paired with suggestions, known alternatives, actual patches and even bids or volunteers to fix it. It would be a pragmatic approach to unsticking the development of the web, as well as “naming-and-shaming” the sticking points. In the web development community we all share the burden of the continued existence of this browser, yet we are powerless to fix it. With information comes power.

Community pressure on browser vendors to support standards in their future browsers has started to yield results. But for the legacy browsers - to-date we’ve had a strategy of sitting and waiting. IE6 usage is dropping slowly but surely and for some the end is in sight - it is approaching the threshold where they can officially drop support. For many though this is still years away. What can we do to help?

AIR command line arguments

I’ve been working on the next release of the Dojo Toolbox - which is an Adobe AIR app, using the Dojo Toolkit. I’m taking a TDD kind of approach to get on a better footing for evolving this thing, and needed a quick way to run a particular set of unit tests.

I wanted to be able to do something like this:

$ adl runTests.xml testModule=toolbox.tests.SomeThing

Getting command line arguments in AIR is via the invoke event, it looks something like this:

air.InvokeEvent.INVOKE, function(evt){
window.scriptArgs = getScriptArgs(evt);

…But when I tried it, I just got a console error:

initial content not found

It turns out that command line arguments to an AIR application are expected to be filenames - like if you dropped a file onto the app’s icon. To pass through parameters and switches you first need

, like so:

$ adl runTests.xml -- testModule=toolbox.tests.all

The event your handler is passed has an arguments array property, and from there its straightforward to process what you’ve got to do the right thing. I’ll get more into how I’m doing the unit tests in another post, but as this took a little digging I thought I’d share.

A rhino prompt

I’m tinkering with another ill-conceived friday night project that may or may not see the light of day. But in the meantime, I just put together this little snippet that illustrates a lot of what’s to like about rhino:

var getInput = function() {
var br = new
return br.readLine();

var greetUser = function() {
// use out.print for the prompt
// instead of rhino's print - which is really println
java.lang.System.out.print("Your name? ");
var name = getInput();
print("Hello " + name);


Arguably (and laughably if that’s your attitude) there’s exactly nothing special here - a whole lot of lines of code to do what in a browser (not to mention any other self-respecting scripting language) is done with one:

alert("Hello: " + prompt("Your name? ", ""))

But the point is that Rhino doesn’t provide a prompt function, and it doesnt matter because you can easily make one. I’ll take flexibility and potential like this over cute predefined functions every time.

Setting up Rhino

I’ve been using the Rhino engine more and more to run command-line scripts, fiddle and try things out. But my setup has taken shape slowly, and it wasn’t much fun to be honest when I first got started.

I’m on a mac (Leopard), and here’s how I’ve got it now:

  1. Download the Rhino .jar file, you’ll find it inside the latest (binary) release.

  2. Drop it in your {user}/Library/java/Extensions folder (create it if it doesnt already exist). That way its automatically added to your classpath whenever you run java, so no need for -jar cmdline params to pull it in (and the classpath insanity that brings).
  3. I made a shell script to invoke rhino and aliased that, but actually you could simply alias the one-liner it contains:

    java jline.ConsoleRunner "$@"

    To make an alias, in my ~/.profile I’ve got the following:

    alias rhino='~/utils/'

  4. Now, in your terminal, you just type ‘
    ‘ and it puts you into an interactive shell where you can load files, write js statements and see the results instantly.

  5. But, Rhino’s shell is frankly, a crappy user experience. Its got no history, no cursor movement at all. You can just type and hit enter. If you screw something up you have to type it all over again. And the whole beauty of writing javascript like this is that you can load a .js library repeatedly as you work on it and try calling its functions. But its a PITA out of the box.

    Look again at that java cmd-line I’m using to run rhino and you’ll see its using jline. This enlightening post on Taming the Rhino finally brought happiness to my rhino shell by introducing me to jline

    You download jline and again, drop the .jar file into your Library/java/Extensions folder. Now the interactive shell is much more shell-like. You have a history on the up arrow, you can back up and move around the current line to edit, and do more scripting and less typing in general.

    To run a particular .js file and exit, you do ‘

    rhino yourfile.js
    ‘. Further cmd-line parameters populate the
    object in your script, so
    rhino script.js filename1 filename2 myoption=true
    would populate
    like this:


    FWIW I use a pattern like this to wrap my script body:

    (function(scriptArgs) {
    function main() {
    // process scriptArgs,
    // e.g. split on = to get name/value pairs
    // and populate an options object

    Back in the shell, you can load, try, load again, try again:

    $ jeltz:trunk sfoster$ rhino
    Rhino 1.7 release 1 2008 03 06
    js> load(“lib/docblock.js”);
    js: “lib/docblock.js”, line 7: uncaught JavaScript runtime exception: ReferenceError:\

    “lang” is not defined.
    at lib/docblock.js:7
    at lib/docblock.js:5
    at <stdin>:2

    js> load(“lib/langUtils.js”);
    js: “<stdin>”, line 3: Couldn’t open file “lib/langUtils.js”.
    js> load(“lib/langUtil.js”);
    js> load(“lib/docblock.js”);
    js> var p = docblock.getParser();
    js> p.parse(“/* @author sfoster /“);
    js> quit();
    jeltz:trunk sfoster$

    Try it, I think you’ll like it.