For years, books and movies have presented audiences with fantastic, aspirational depictions of the future, promising us a world where every aspect of our lives will be made easier with cutting-edge technology. These authors and directors often envision societies brought to order by robots, designed to do everything from performing dangerous labor to making our coffee. Now, researchers may be in the process of bringing the latter to life: a team of computer scientists at Stanford University are making robots more adaptable by teaching them how to learn to use different coffee machines, even ones they have never seen before.
This isn’t the first time robots have been brought into the coffee industry: in December 2014, for example, Nestlé acquired 1,000 robots to sell espresso machines in its Japanese Nescafé stores. However, a project called RoboBarista could bring this technology closer to home. The brainchild of Ashutosh Saxena, an assistant professor of computer science at Stanford, and graduate student Jaeyong Sung, the project takes aim at a common problem in robotics: the machines must be trained for each task and be kept in a steady role to allow them to recognize and perform the work. Saxena and Sung are now teaching a robot to adapt.
In Saxena’s Robot Learning Lab, the team is testing a new deep-learning algorithm they have developed. Using the algorithm, the robot is able to operate a machine it has never seen before by consulting an instruction manual and drawing on its past experience with similar devices. But while Saxena and Sung refer to their test subject as a RoboBarista, its application isn’t limited to coffee and espresso: so far, they have trained and tested their robot with 116 different appliances, including juicers, lamps, a soda machine and bathroom sinks.
So far, one of the biggest challenges has been tackling the versatility of natural language. For example, different coffee machines might be turned on with a “knob,” “switch,” or “button,” while the coffee itself might be dispensed by pulling a “handle” or a “lever.” To adapt to this obstacle, the team has developed a deep learning neural network that helps the robot identify the action closest to the one being described.
Because the robot does this best when working with a large database, Saxena and Sung turned to the Amazon Mechanical Turk, which recruits people to conduct simple online tasks for small payments. So far, hundreds of visitors have guided the robot through various tasks by using a Web interface the team created to guide an imaginary robot arm. In addition to expanding its verbal knowledge, this also teaches the robot how to identify various controls by their shape rather than location before relating them to various labels that might be used in the instructions.
Currently, the robot uses a 3D camera and a “point cloud,” or list of coordinates of every point in an image. Once the robot translates the label in the instruction manual, it locates the control on the device it is using and consults the crowdsourced model to plan the trajectory its arm will follow to manipulate the switch. Over time, these actions can be transferred from one machine to another, allowing the robot to learn how to operate a slow cooker the same way it operates a coffee machine.
So far, the robot has performed with 60% accuracy in tests on a variety of different machines it has never seen before. Unfortunately, this process has only introduced new challenges. For example, glossy components on the machines can reflect complex light patterns that can make it difficult for the robot to identify the shapes. Because of this, the researchers have considered adding tactile feedback to help the robot operate controls correctly, as well as visual monitoring to avoid collisions. Further developments could also include trial and error processes for handling unfamiliar controls, as well as set routines to recover from failure.
Saxena and Sung will describe their work at the 2015 Robotic Science and Systems conference in Rome, which will be held July 13-17. This means it could be quite some time before you will see robots working in a mobile espresso business or operating your home coffee machine. Fortunately, if the technology were to become accessible, these robots could be an important change for the coffee industry: coffee shops currently have a 7% annual growth rate worldwide, making it one of the fastest growing industries worldwide.