Virtual Cooking 2

Overview

The Virtual Cooking project has been in existence since 2019 and aims to teach robots to grasp like humans so they can assist in cooking. The work of this project focuses on adding new objects to a virtual kitchen environment and analyzing different grasp types in it, considering where the objects were picked and how much force was applied. The way of grasping in the virtual environment will also be compared to how grasping works in real life.
We use Cybergloves and the OptiTrack tracking system to record human grasping movements in virtual reality. To display the collision between the data gloves and the objects, heat maps are generated and stored in a database.
In general, we use the Unreal Engine for 3D representation and Meshroom for 3D modelling of the objects.

Tasks

The tasks of the projects are divided into two parts - Virtual Reality and Photogrammetry. Further, Virtual Reality is segregated into three groups - Comparison, Annotation and Heat Map.

Photogrammetry
											We worked on automatically creating 3D models from real life objects
											using Photogrammetry. Not only did we aim to automate the photogrammetry process but also the
											acquisition process.
											
										
 Learn More
Annotation
											The Annotation team's goal was to implement the automatic detection of
											everyday hand grasps, which could later contribute to the grasping mechanism
											improvement of kitchen robots.
									
										
 Learn More
Heat Map
											The goal of the heatmap team was to generate two different heatmaps: one to
											represent where the object was touched by the individual fingers and one to
											show the force used to hold an object. 
										
 Learn More
Comparison
										The goal of the comparison team was to conduct an experiment demonstrating the
										differences and similarities between grasping objects in VR (Virtual Reality)
										and RR (Real Reality). 
										
 Learn More

Gallery

Two different approached for collision detection: On the left inner sphere tree colliders are applied to the middle and tip joints of the fingers and on the right rays are casted from each joint to the palm, sideways and upwards from the fingers

Hand seperated into collision meshes. The model was made in Blender and the meshes in use for the detection are the tip and middle parts of each finger

Which grasps is applied depends heavily on which object is pick up. A waterbottle is grasped in a more curvy shape and a cutting board more straight

Performing the Handgrasps with kitchen items - Prismatic three fingers grasp

Performing the Handgrasps with kitchen items - Tripod variation

Performing the Handgrasps with kitchen items - Sphere finger detection

Robot arm and camera working on taking pictures for photogrammetry

3D model created with the help of MAB clicking the pictures and Meshroom to create the photogrammetry model.

This picture shows the problem of the previous implemented heatmap with the colours
mixing. It is not clear to see which finger is touching the object where.

The Forcemap is a completely new idea and shows how strong the object was touched
during the grasp. A red colour indicates a strong grasp while green indicates a lighter grasp.

The Heatmap developed from the Heatmap team with the new approach and the idea
of the dominant finger. Clearly there are no colours mixing and every finger can be identified
easily.

Cyberglove being used in combination with the pringles box to incorporate haptic feedback for testing purposes

Performing a trial run of the comparison experiment with haptic feedback

The virtual environment used for the experiment with the brown part on top mapped to an actual area within the lab for the experiment

Documentation

Presentations:

During the project related work was presented in order to provide a broad overview of the subject.

How tos:

About Us

The Virtual Cooking 2 team comprises of five Supervisors and fifteen students. Our team consists of students from the University of Bremen as well of students from the University of the Arts Bremen. There are designers and computer scientists. However, what connects us all is the interest in virtual reality and computer graphics.