TechTheDay

Here’s one ingenious technology that deserves to be added to today’s mobile devices. Forget about the unnecessary super-high resolution displays and cameras that don’t really add anything significantly useful to the user experience. Researchers at the Future Interface Group (FIGLAB) at Carnegie Mellon University have developed a system that can enable interaction with devices through sound, through the tapping on a table or platform on which a device is positioned.

Toffee: Sonic Sensors for Device Control

This sound-based device interaction technology is called Toffee, complete details of which are presented on a research paper published on the official FIGLAB website titled “Toffee: Enabling Ad Hoc, Around-Device Interaction with Acoustic Time-of-Arrival Correlation. The research team is composed of Robert Xiao, Scott E. Hudson, and Chris Harrison of the Human-Computer Interaction Institute and Greg Lew, James Marsanico, and Divya Hariharan of Carnegie Mellon University’s Electrical and Computer Engineering department.

Toffee can be described as a sensing approach that is capable of enhancing touch interaction with devices beyond touchscreens, onto ad hoc adjacent surfaces, tabletops in particular. As described on the official FIGLAB website, it is a novel application of acoustic time differences of arrival (TDOA) correlation.

How Does It Work?

Toffee works in the presence of two important factors: gravity and a hard tabletop or a similar implement. The tabletop is necessary to produce the sound and of course to have something on which the mobile device (smartphone, tablet, or laptop) can be placed. Gravity, on the other hand, is simply needed to keep the device on the tabletop.

The sound generated by a tap is detected by an apparatus (with four vibro-acoustic piezo sensors, one on every corner) positioned underneath the device to be controlled. The four vibro-acoustic sensors determine the location of the origin of the tap by individually sensing the sound waves emanating from the tap. Naturally, a tap produces sound waves that move to all directions, radiating from the tapping point. These sound waves reach the four vibro-acoustic sensors at different times, allowing the Toffee system to calculate the origin or location of the tap. Knowing the tap origin is important for Toffee since it does not register controls based on sound patterns, volume, or tone but mostly based on where the tap originates.

The research team created prototypes to test the accuracy of the tap-sensing capabilities of Toffee. Based on the results derived from the laptop prototype created, Toffee has an acceptable level of accuracy. It has a mean error of just 4.3° (based on the laptop prototype tested). This means that most taps should be properly pinpointed and the commands associated with them can be executed well enough as expected.

Applications

Basically, what Toffee does is to widen the area of interaction between humans and devices. Instead of getting constrained by the size of the touch displays or the number of buttons/keys in a device, controls can be enabled through taps around a device.

These taps can be used to adjust the volume of the media player just like what has been demonstrated on this official informational video for Toffee. Controls through the sonic sensors may also make it easier to skim through pages of documents or through pictures in photo galleries. Additionally, Toffee can also be used in gaming. Many games can become more enjoyable through tabletop tap controls.

Challenges and Limitations

Unfortunately, Toffee is still far from becoming perfect. The prototypes demonstrated are still not showing practical applications although they demonstrate the potentials of the technology. The Toffee tablet prototype was able to demonstrate a shooter game wherein the player shoots at enemies by tapping at their direction; but the gameplay looks rather inconvenient or uncomfortable. Maybe it’s just because the game demonstrated was not that well thought out. Still, what’s clear is that Toffee is an imperfect technology that still has to mature or at least be matched with a thoughtfully developed software.

Moreover, with what the informational video has shown so far, it appears that more work is needed in miniaturizing the Toffee setup. It seems to require quite a space to be integrated with a device. It would definitely be better to see a demonstration that shows Toffee seamlessly integrating within a device or at least as a dock or plate on which a device can be readily positioned.