Training robots to pick tomatoes

Due to a shortage of human labor, the demand for robots in areas like healthcare and agri-food is growing. These environments contain many different objects or actors, like humans, animals, plants, and other robots, which can exhibit variations in appearance and unpredictable behavior.

To deal with these situations, intelligent robotic systems are required that can assess their environment and the reason why certain actions are needed to achieve goals. Such robots need to react to unpredictable situations, caused by variations, in order to perform the task robustly.

Raw data from sensors

Currently, for the most part, information is derived from raw data gathered using sensors as the robot explores its environment. Such an approach, that relies only on sensor data, lacks the ability to interpret this information to understand how well the system is performing its task.

The latter is necessary for autonomous systems to determine whether to continue the work, re-plan, or give a sign to get help from a human. This requires knowledge about the context in which the information is obtained, and  knowledge about the environment is the core aspect of a so-called World Model (WM).

For his PhD research, Jordy Senden developed methods to create World Models for robotic systems by letting a robot build up and maintain a world model from sensor data and other models. This allows a robot to execute tasks in environments where they can be extreme variations.

Greenhouse example

For his research, Senden takes ‘The Greenhouse’ case as the running example, where the objective is to create an autonomous way of harvesting tomato trusses and prune the leaves on a tomato plant using a robot.

In such a case, there are a lot of variables and variations that could be taken into the account. Using the WM approach omits known variations while also leading to a more robust solution.

Senden shows that such an invariant world model, as well as the robotic perception and control skills that are based on this model, can be robust against expected parametric variations.

Another step to take is to develop multiple robotic skills that are able to achieve the same results, but using different techniques. Having multiple options to choose from creates redundancy to solve a task successfully.

Senden and his colleagues demonstrate this through the development of a truss detection skill that estimates the progression towards a truss while moving along the stem, based on the dynamic response of the tomato plant to an excitation force, rather than using a vision system.

These insights have led to the start of the development of a framework for knowledge-driven active perception. This framework allows for the creation of an explicit object-centered graph-based world model that models semantic relations between objects.

This world model allows for information association, between measured data and modeled knowledge, and multi-hypothesis tracking at the semantic level.

Title of PhD thesis: World Models for Robust Robotic Systems. Supervisors: René van de Molengraft, Herman Bruyninckx, and Elena Torta.