At the University of Cambridge, the scientists programmed their robotic chef with a “cookbook” of nearly eight simple salad recipes.
Image Credit: Elenadesign/Shutterstock.com
After watching a video of a human illustrating one of the recipes, the robot was having the potential to determine which recipe was being prepared and make it.
Besides, the videos assisted the robot in incrementally adding to its cookbook. When the experiment reached the conclusion stage, the robot came up with a ninth recipe on its own.
Their study outcomes, reported in the journal IEEE Access, illustrate how video content could be a rich and useful source of data for automated food production, and could allow simpler and affordable deployment of robot chefs.
For nearly decades, robotic chefs have been featured in science fiction, but in fact, cooking is a difficult issue for a robot. Numerous commercial companies have constructed prototype robot chefs, even though none of these are presently commercially available, and they lag absolutely at the back of their human counterparts in terms of skill.
Human cooks have the potential to learn new recipes via observation such as watching another person cook or watching a video on YouTube. However, programming a robot to make a range of dishes is expensive and tedious.
We wanted to see whether we could train a robot chef to learn in the same incremental way that humans can—by identifying the ingredients and how they go together in the dish.
Grzegorz Sochacki, Study First Author, Department of Engineering, University of Cambridge
Sochacki, a Ph.D. candidate in Professor Fumiya Iida’s Bio-Inspired Robotics Laboratory, and his collaborators arranged eight simple salad recipes and filmed themselves making them. Further, they utilized a publicly available neural network for the robot chef to be trained.
Earlier, the neural network had been programmed to determine a variety of different objects, such as the vegetables and fruits utilized in the eight salad recipes (broccoli, apple, carrot, banana, and orange).
With the help of computer vision methods, each frame of video was examined by the robot, which was able to determine varied objects and features, like a knife and the ingredients, as well as the human demonstrator’s hands, arms, and face.
Both the videos and the recipes were transformed into vectors and the robot executed mathematical operations on the vectors. This was done to identify the similarity between a demonstration and a vector.
By properly determining the ingredients and the actions of the human chef, the robot can identify which of the recipes was being made. The robot can deduce that if the human demonstrator was holding a knife in one hand and a carrot in the other, the carrot would further get diced.
Out of the 16 videos it watched, the robot identified the proper recipe 93% of the time, even though it was able to detect just 83% of the human chef’s actions. Also, the robot was able to detect those minor changes in a recipe, thereby making a double portion or normal human error, changed and not new recipes.
Also, the robot properly identified the illustration of a new, ninth salad, added it to its cookbook, and then made it.
It’s amazing how much nuance the robot was able to detect. These recipes aren’t complex–they’re essentially chopped fruits and vegetables, but it was really effective at recognizing, for example, that two chopped apples and two chopped carrots is the same recipe as three chopped apples and three chopped carrots.
Grzegorz Sochacki, Study First Author, Department of Engineering, University of Cambridge
The videos that were utilized for training the robot chef are not like the food videos made by a few social media influencers, which are full of visual effects and fast cuts. Also, it rapidly moves back and forth between the person making the food and also the dish they are preparing.
For instance, the robot would strive to determine a carrot if the human demonstrator had their hand wrapped around it—for the robot to find the carrot, the human demonstrator had to keep the carrot up so that the robot could notice the whole vegetable.
Our robot isn’t interested in the sorts of food videos that go viral on social media—they’re simply too hard to follow. But as these robot chefs get better and faster at identifying ingredients in food videos, they might be able to use sites like YouTube to learn a whole range of recipes.
Grzegorz Sochacki, Study First Author, Department of Engineering, University of Cambridge
The study was financially supported in part by Beko plc and the Engineering and Physical Sciences Research Council (EPSRC), part of UK Research and Innovation (UKRI).
Robot ‘chef’ learns to recreate recipes from watching food videos
Robot ‘chef’ learns to recreate recipes from watching food videos. Video Credit: University of Cambridge.
Journal Reference:
Sochacki, G., et al. (2023) Recognition of Human Chef’s Intentions for Incremental Learning of Cookbook by Robotic Salad Chef. IEEE Access. https://doi.org/10.1109/ACCESS.2023.3276234.
Source: https://www.cam.ac.uk/