Feedback and inquiries

Intelligent Systems Group
Department of Computer Science and Engineering P.O.Box 4500
FIN-90014 University of Oulu
Finland
Tel. +358-8-553 1011
Fax. +358-8-553 2612

Physical user interfaces

One of our main tasks in developing interactive spaces is to integrate the user interfaces into the environment. Classical human computer interaction is built in a single computing device. Input and output interfaces are provided using computer peripherals. A user enters data to the system using either a mouse or a keyboard; or a keypad in the case of mobile phones. In the vast majority of the cases the system provides output through a monitor, speakers or a printer.

In interactive spaces this classical interface view is not valid anymore. Every object and surface in the environment is a potential component of the user interface. Grasping an object, moving it or just touching it with a mobile device might be considered as an input to the system managing the space. Even more, all user actions are potential commands for the system. For example, a user can say the command aloud, wave a hand, or stump the floor. The system might respond to such object handling and actions by switching on the lights, playing a video on a wall display, presenting the next slide of a slide show, printing a document, or saving a document in the user's file repository, for example.

Clearly, the user interfaces are not contained in single devices but distributed to different objects, surfaces, and devices in the environment. As the variety of user actions to command the system is larger than in traditional user interfaces, and the user interface components can be placed freely anywhere in the environment, one essential research question is how the user interfaces should be advertised to the user. That is, how can a user recognize the environmental features that belong to the user interface and how should the "command language" - actions the user can use to command the system - be communicated to the user? In addition to recognizing user interface components, the user needs to recognize the services available in the local environment. From software perspective, common interfaces are needed, as a service can be used through many different user interfaces.

Our work on physical user interfaces has so far focused on touch-based and gesture-based user interfaces.

Touch-based user interfaces

Our touch-based user interfaces are based on RFID technology: users give commands to the system by touching RFID tags in the environment with mobile phones equipped with RFID readers. Each RFID icon is placed behind an RFID icon, a graphical icon advertising the command the system performs when a user touches the tag. The command can either start a service or control an already running service.

The figure below presents some RFID icons. The top left icons might advertise that by touching that point a user can print the document on her/his terminal's display. The rest of the icons a user might find, for example, from her/his friend's office doorway. Touching these icons might initiate a call to the friend, bring an email template to the user's terminal's display (pre-filled with the friend's email address), and show the friend's calendar.

  .

Figure. Some RFID Icons.

Touching the top left icon in the second figure might configure the mobile terminal as a remote control for a wall display next to the icon. The top right icon might be placed next to the first one. Touching this second icon might drop an image from the mobile terminal's screen to the wall display. The third icon would be touched when the user wants to pick an image from the wall display. The fourth icon would save content.

Figure. More RFID Icons.

We have developed a server for connecting physical user interfaces and resources in the local environment to services in the Internet. This REACHeS server is presented below. The numbers illustrate a typical message sequence. A user starts by touching an RFID icon with a mobile phone (1). The phone reads a command from the tag and sends it to the REACHeS server (2). The server redirects the command to the service responsible for executing it (3). The service calculates a response and sends it to REACHeS (4). REACHeS sends the response to the mobile phone or to the local resources the service controls - or to both (as defined in the response). For example, a user might touch a "Next" icon, in which case a Photo Album service would instruct a local display to show the next photograph. The service experiences a direct connection to the local resources (7).

Figure. The REACHeS server.

We have implemented several prototypes containing touch-based user interfaces. A video about controlling a wall display can be found from YouTube. Part of the research on touch-based user interfaces is performed in the DIEM ICT SHOK project.

Gesture-based user interfaces

Our gesture-based user interfaces utilize mobile phones' acceleration sensors. Users hold phones in their hands when drawing gestures on the air. The gestures are recognized from the sensor data produced by the phones' acceleration sensors. The system architecture is shown in the diagram below.

Figure. Gesture recognition system architecture.

The recognition system can be taught: a user draws a gesture five times on the air and gives a name to it, e.g. "circle". The figure below presents a short interval of three acceleration signals. The gray areas identify recognized gestures "clockwise circle", "punch forward", and "swing to right".

 Any application in the phone or in the Internet accepting TCP/IP or HTTP messages can be controlled with gestures. A user can configure the commands corresponding with the taught gestures in a (gesture,command)table. This table contains a TCP/IP or HTTP message for each gesture. When the system recognizes a gesture, it sends the corresponding message. The table can also be modified dynamically. An example of the (gesture, command) table is shown below. The first four gestures in the table trigger commands to be sent to the phone's applications. That is, when the user draws the character 'p' on the air, the phone's profile will be changed. Drawing 'c' brings the calendar to the terminal's screen, and '1' and '2' trigger the corresponding speed dials. When the user draws a circle or swings the terminal to the right, commands are sent to a wall display application; to start the application, and to show the next product ad, respectively.

We have so far implemented the gesture recognition software for Nokia N95 and Nokia N900. The software for Nokia N95 can be downloaded from here. A video about the gesture UI can be found from YouTube.