Go with the flow! Exploring the unknown with a swarm of smart marbles

A swarm of marbles that flows through your intestines in order to detect thickening. It sounds like science fiction, but it’s closer to reality than you might think. Engineer Peter Baltus of Eindhoven University of Technology has developed, in collaboration with universities and the business community, sensors that are able to inspect hard-to-reach or even inaccessible environments. His primary focus is on large-scale pipeline networks such as drinking water systems, oil pipelines or transport systems for kerosene. But his dream is that these small balls will one day explore the inside of volcanos, or maybe even the human digestive tract.

On the table in front of us lie three plastic balls the size of golf balls, only much lighter. The ridges from the 3D printer are clearly visible on the outside. One of the balls has been opened up. Inside, we see a few small circuit boards attached to each other, as well as a micro SD card and two short wires. We are in the office of Peter Baltus, professor of microelectronics at the department of Electrical Engineering at TU/e.

As soon as Baltus starts to talk, his eyes light up. His passion shines through in his energetic attitude and his enthusiastic way of speaking. These smart marbles are his ‘babies’ after all. Not that he would ever accept these credits. During the last four years, Baltus worked day and night on the so-called Phoenix project in his role as project leader, together with project partners KU Leuven, Aachen University, Antea Group, the University of Groningen, VU Amsterdam, and the University of Trento. ‘This multidisciplinary group, with universities as well as the industry, was essential to develop these smart marbles in such a short space of time.’

Financed with a €3.6 million grant from Horizon 2020 ‘Future and Emerging Technologies’ (FET), an EU program aimed at developing science fiction-like technology, the group built a device that can explore difficult-to-access environments. And that’s a real chicken-and-egg problem, because how do you design sensors for an environment you haven’t seen? ‘In order to collect good information, you need good sensors. But if you want to build good sensors, you have to know what you want to measure, and that requires knowledge of the environment you want to explore. An impossible task,’ Baltus says, laughing.

The researchers specifically looked for socially relevant applications that address primary human needs such as safety, health, energy, and food and water. Think, for instance, of the pipeline network beneath our feet; drinking water pipes, the sewage system or pipes for transporting fuel. But also think of detecting underground storage places for CO2, chemical reactors, or even blood vessels or the human digestive tract. The blue panels in the text below describe a few of the applications Baltus and his colleagues are currently working on intensively.

The advantage of being with many

After much trial and error, the research team eventually produced a collection of smart marbles. These are small balls with sensors inside, which flow through pipes filled with liquid in order to detect obstructions and leaks, or map a pipeline network. The balls are released manually into the liquid at the beginning of the pipeline system. They then float passively with the current until they reach their final destination, where they are extracted so that the researchers can read out the data. The sensors collect an enormous quantity of information along the way, which they store internally.

The marbles must be suited for measurements during a company’s production process, without having to shut it down. Baltus: ‘That’s still frequently the case, for example during the inspection of the pipeline that transports kerosene between the port of Rotterdam and Düsseldorf airport. At present, a robot carries out this inspection, but in order to do so, the pipe needs to be emptied. A costly job, because time is money. Many production processes, for instance those in the chemical industry, simply can’t be shut down at all.’ The project team anticipates this by making the marbles resistant to all kinds of liquids and changes in temperature or pressure. Even the most corrosive liquid should produce reliable measurement data.

In many cases, the pipes are too narrow to allow space for one super sensor that can do everything. The size of the ball, and of the battery in particular, is the limiting factor. That’s because every sensor needs power. But the more intelligent you make the sensors, the more power they usually need. And a large battery simply won’t fit into a small ball. That would exclude, for instance, cameras, light, microphones or drive system. The shell only fits simple sensors with low energy consumption, such as a thermometer, pressure gauge and speedometer.

Using Artificial Intelligence to create evolving sensors

That is precisely why Baltus and his project partners developed a swarm of marbles. ‘There’s not much they can do individually, but together, they collect all the data we need,’ Baltus explains. But how do you ensure that each marble is optimally adapted to its specific task? The environments that Baltus wants to explore are so extremely inaccessible, that he can’t adequately predict how the pipe will look like from inside. ‘When you are unfamiliar with an environment, you also can’t know which features a sensor is going to need in order to explore that environment.’

The researchers came up with an ingenious solution: they use Artificial Intelligence to let the marbles ‘evolve’ automatically on the computer during the design process, until they are optimally adjusted to suit their specific task. That has an additional advantage: it makes it possible to develop a generic system that can evolve into the best configuration for each application. 'The choice for Artificial Intelligence was born out of despair. If we knew how to design the marbles, we would have done it ourselves,' explains Baltus.

The exact design process of the balls works as follows: the generic balls, programmed with the scarce information and wishes of the client, traverse a company’s pipeline system once. The sensors collect all kinds of information along the way, which they store in their internal memories. The researchers collect the data from the balls and develop an initial model of the unfamiliar environment.

All the data collected by the marbles, including the first model of the environment, are subsequently copied to a virtual environment. An algorithm then starts to work on this during a series of simulations. It takes the algorithm about 500 simulation rounds to look for an optimal configuration that suits the specific implementation and wishes of the client. As soon as the new configuration doesn’t fundamentally change compared to the previous couple of configurations, the researchers know that they are close to finding the solution. The software that was designed by Artificial Intelligence is then installed into the balls, after which they are once again released in the same pipeline system. This process is repeated three to five times. After this, the balls are finally ready to be sold to a company where they can carry out inspections.

Baltus: 'I draw a parallel with evolution, because the process is so unpredictable. We don’t have a solution, so we let an algorithm do it. This algorithm mixes the most successful balls and works towards an optimum. We don’t know the results of that simulation process beforehand. Perhaps the algorithm decides to give each marble its own task, which could lead to location marbles and marbles that detect leakages. We haven’t been able to test this extensively, so we have to wait and see. The fact that our first evolution experiment shows that the principle works is actually unbelievable, and they even evolve in the right direction as well!’

Navigating with limited power

Once you have a working system with smart marbles that evolve automatically towards a certain application, you encounter one more problem: with all the data collected by the sensors, the client would like to know the exact location of the marbles at that moment. In case the sensors detected a pipe burst, you obviously need them to store the location. That is why the location sensor is extremely important, but GPS doesn’t function underground. Luckily, there are alternatives.

One of them is the use of acoustic waves, which can actually work underground. Think of an ultrasonic system for instance, comparable to the echo’s that bats use. This solution might sound promising, but there are still a number of obstacles to overcome. Baltus: 'During an initial calculation, this system used no less than 5000 times more energy than all the other sensors and systems (including memory and the microprocessor) inside the small ball combined.'

In addition, sound velocity depends on the environment; you need to calibrate speed differently in water than in mud or oil. That’s difficult when you are in an unfamiliar environment. Besides, the ball transmits its data in just one direction with an ultrasonic system. The transmitter and the receiver need to ‘see each other’ for the ball to be able to transmit the signal. And an omnidirectional transmitter that transmits the signal in every direction does exist, but it will be so heavy that the ball would sink to the bottom. After extensive testing and continued development, an ultrasonic system might be a solution for a large number of applications. But if you really want to scale down the size of the marbles, you are going to need a different system.

THE ATTRACTION OF MAGNETISM

That is why the researchers came up with an unexpected alternative for the smaller marbles: magnetism. Baltus: ‘You can transport signals via the liquid in the pipeline network with this method. Reversing magnetic fields with low frequencies, for example. The ball can transmit its own serial number this way.’ Additional benefit: the signal goes in every direction, which makes picking it up much simpler. ‘A combination of magnetism and ultrasound might be a possibility. In that case, the systems calibrate each other, which enables you to make a reverse calculation of the sound velocity. You even collect new data that way, namely, the medium in which the balls float around,’ Baltus explains.

The magnetic system, however, only functions over short distances. Luckily, that is not a problem: since the marbles travel in a large swarm, there’s always a receiver nearby. The location is determined by their relative position to each other and by the entry point in the pipeline where the marbles were inserted. This provides the researchers with a very detailed map of the pipeline system once they read out all the information stored in the marbles.

Marble in distress

Determining the location is even more important when you take into account how the marbles move through the pipelines. To reduce energy use, they passively flow with the current. Unfortunately, that gives you no control over their course. A ball could get stuck, especially in those places where the pipe is congested. And that’s exactly the kind of information that can be of great value to a client.

That is why Baltus fitted the marbles with an emergency system: ‘When a marble gets stuck, it uses its last bit of energy to send out a distress signal to the other balls nearby. In doing so, the marble also sends all the information it collected, so that no vital information is lost.’ The client can then potentially save the marble when an employee cleans the pipe.

Now, four years later, the Phoenix project has officially ended. Baltus: ‘We have shown that the concept and the evolution of the marbles works, and we’ve developed the technology that will enable us to significantly reduce the size of the marbles in the future.’ Baltus is clearly far from finished with the project. As a next step, he has started a spinoff under the name ‘Smarble', together with the Antea group. The purpose is to market the balls, a project in which Antea will take the lead. Smarble received a €100,000 euro grant from the Launchpad program to develop a business case, in addition to the HORIZON 2020 FET grant. ‘We want to produce robust and affordable swarms of marbles and put them into use. We don’t need new inventions for that, but it will take a lot of hard work to solve all kinds of practical problems,’ Baltus explains.

Besides Smarble, TU/e will continue with the research as well. Baltus: ‘We want to make the marbles even smaller, so that we can use them in healthcare, for example. Imagine that they’re small enough to flow through your bloodvessels. That would bring entirely new possibilities to the detection of obstruction. Researcher Giovanni Iacca from the university of Trento is starting up a consortium to reduce the size of the balls. He is currently looking for partners to apply for a new Horizon 2020 FET grant.’ Baltus also hopes to perform simple routines in the future. ‘Now, we only detect leaks, but in the future we also want to be able to close them right away with a kind of resin.’ Baltus plans to apply for a grant for this as well. ‘Once we’ve managed to land those subsidies, I’ll be able to start recruiting new PhD candidates and postdocs.’

RESEARCH CONTINUES

At the same time, the optimization process of the current balls will continue. ‘We want to show that our marbles can be applied in all kinds of ways. For that, we need the help of companies who are willing to let us use their processes as test locations. We will soon be testing mixing processes at Jongia and drinking water pipelines at Oasen, but are looking for many more applications for testing.’

But bachelor’s and master’s students, too, are always welcome to help Baltus further develop the balls. ‘The great thing about this work is its multidisciplinary nature. Recruiting researchers from the field of Electrical Engineering alone is not enough. For instance, I also need mechanical engineers and construction engineers, as well as experts on software, user interfaces, artificial intelligence, evolution, you name it. I hope to sit down with Innovation Space soon, to start a new program with a group of bachelor’s students from several different programs at TU/e.’

Photo credits: Bart van Overbeeke