Go with the flow! Exploring the unknown with a swarm of smart marbles
A swarm of marbles that flows through your intestines in order to detect thickening. It sounds like science fiction, but it’s closer to reality than you might think.
A swarm of marbles that flows through your intestines in order to detect thickening. It sounds like science fiction, but it’s closer to reality than you might think. Engineer Peter Baltus of Eindhoven University of Technology has developed, in collaboration with universities and the business community, sensors that are able to inspect hard-to-reach or even inaccessible environments. His primary focus is on large-scale pipeline networks such as drinking water systems, oil pipelines or transport systems for kerosene. But his dream is that these small balls will one day explore the inside of volcanos, or maybe even the human digestive tract.
On the table in front of us lie three plastic balls the size of golf balls, only much lighter. The ridges from the 3D printer are clearly visible on the outside. One of the balls has been opened up. Inside, we see a few small circuit boards attached to each other, as well as a micro SD card and two short wires. We are in the office of Peter Baltus, professor of microelectronics at the department of Electrical Engineering at TU/e.
As soon as Baltus starts to talk, his eyes light up. His passion shines through in his energetic attitude and his enthusiastic way of speaking. These smart marbles are his ‘babies’ after all. Not that he would ever accept these credits. During the last four years, Baltus worked day and night on the so-called Phoenix project in his role as project leader, together with project partners KU Leuven, Aachen University, Antea Group, the University of Groningen, VU Amsterdam, and the University of Trento. ‘This multidisciplinary group, with universities as well as the industry, was essential to develop these smart marbles in such a short space of time.’
CURRICULUM VITAE PETER BALTUS
Peter Baltus (1960) worked at Philips for 22 years, and later at NXP in Eindhoven, Nijmegen, Tokyo and Sunnyvale. In 2007, he became professor in high-frequency electronics at Eindhoven University of Technology. There, he served as director of the Centre for Wireless Technology between 2007 and 2016. Since 2017, Baltus has been Chair of the Microelectronics group at TU/e. He co-authored more than 100 papers and holds 16 US patents.
Financed with a €3.6 million grant from Horizon 2020 ‘Future and Emerging Technologies’ (FET), an EU program aimed at developing science fiction-like technology, the group built a device that can explore difficult-to-access environments. And that’s a real chicken-and-egg problem, because how do you design sensors for an environment you haven’t seen? ‘In order to collect good information, you need good sensors. But if you want to build good sensors, you have to know what you want to measure, and that requires knowledge of the environment you want to explore. An impossible task,’ Baltus says, laughing.
The researchers specifically looked for socially relevant applications that address primary human needs such as safety, health, energy, and food and water. Think, for instance, of the pipeline network beneath our feet; drinking water pipes, the sewage system or pipes for transporting fuel. But also think of detecting underground storage places for CO2, chemical reactors, or even blood vessels or the human digestive tract. The blue panels in the text below describe a few of the applications Baltus and his colleagues are currently working on intensively.
A swarm of smart marbles. One of the balls has been opened up, showing the circuit boards.
THE ADVANTAGE OF BEING WITH MANY
After much trial and error, the research team eventually produced a collection of smart marbles. These are small balls with sensors inside, which flow through pipes filled with liquid in order to detect obstructions and leaks, or map a pipeline network. The balls are released manually into the liquid at the beginning of the pipeline system. They then float passively with the current until they reach their final destination, where they are extracted so that the researchers can read out the data. The sensors collect an enormous quantity of information along the way, which they store internally.
The marbles must be suited for measurements during a company’s production process, without having to shut it down. Baltus: ‘That’s still frequently the case, for example during the inspection of the pipeline that transports kerosene between the port of Rotterdam and Düsseldorf airport. At present, a robot carries out this inspection, but in order to do so, the pipe needs to be emptied. A costly job, because time is money. Many production processes, for instance those in the chemical industry, simply can’t be shut down at all.’ The project team anticipates this by making the marbles resistant to all kinds of liquids and changes in temperature or pressure. Even the most corrosive liquid should produce reliable measurement data.
In the chemical industry, at an industrial scale, mixing liquids is often done with air bubbles. These bubbles rise to the top of a reactor vessel and properly mix the different liquids in the vessel this way. However, the measurement probes inside these types of tanks, whose purpose it is to monitor the mixing process, can seriously disrupt things.
Testing has shown that the marbles have a much less disruptive effect, primarily because they float upwards with the air bubbles. Moreover, they collect more interesting information than the probes do. The balls, however, can only do this when they are smaller than the air bubbles. This was the conclusion drawn from experiments conducted by Kay Buist en Heinrich Wörtche at TU/e’s department of Chemical Engineering and Chemistry.
Stirring and mixing equipment supplier Jongia, specialized in industrial agitators among other things, has expressed an interest after that experiment, and it now wants to test the balls in their large silos. The company will soon start with this test.
In many cases, the pipes are too narrow to allow space for one super sensor that can do everything. The size of the ball, and of the battery in particular, is the limiting factor. That’s because every sensor needs power. But the more intelligent you make the sensors, the more power they usually need. And a large battery simply won’t fit into a small ball. That would exclude, for instance, cameras, light, microphones or drive system. The shell only fits simple sensors with low energy consumption, such as a thermometer, pressure gauge and speedometer.
USING ARTIFICIAL INTELLIGENCE TO CREATE EVOLVING SENSORS
That is precisely why Baltus and his project partners developed a swarm of marbles. ‘There’s not much they can do individually, but together, they collect all the data we need,’ Baltus explains. But how do you ensure that each marble is optimally adapted to its specific task? The environments that Baltus wants to explore are so extremely inaccessible, that he can’t adequately predict how the pipe will look like from inside. ‘When you are unfamiliar with an environment, you also can’t know which features a sensor is going to need in order to explore that environment.’
PhD researcher Ahmed Hallawa of Aachen University works on (co-)evolutionary algorithms. Here, he is working on an evolutionary method for the instincts of the marbles.
The researchers came up with an ingenious solution: they use Artificial Intelligence to let the marbles ‘evolve’ automatically on the computer during the design process, until they are optimally adjusted to suit their specific task. That has an additional advantage: it makes it possible to develop a generic system that can evolve into the best configuration for each application. 'The choice for Artificial Intelligence was born out of despair. If we knew how to design the marbles, we would have done it ourselves,' explains Baltus.
The exact design process of the balls works as follows: the generic balls, programmed with the scarce information and wishes of the client, traverse a company’s pipeline system once. The sensors collect all kinds of information along the way, which they store in their internal memories. The researchers collect the data from the balls and develop an initial model of the unfamiliar environment.
All the data collected by the marbles, including the first model of the environment, are subsequently copied to a virtual environment. An algorithm then starts to work on this during a series of simulations. It takes the algorithm about 500 simulation rounds to look for an optimal configuration that suits the specific implementation and wishes of the client. As soon as the new configuration doesn’t fundamentally change compared to the previous couple of configurations, the researchers know that they are close to finding the solution. The software that was designed by Artificial Intelligence is then installed into the balls, after which they are once again released in the same pipeline system. This process is repeated three to five times. After this, the balls are finally ready to be sold to a company where they can carry out inspections.
Just before the Phoenix project ended, the researchers were able to demonstrate the evolution of the marbles. A breakthrough after four years of intensive research. How did they accomplish that? The marbles had to map out a simple pipeline system consisting of pipes and a pump on the TU/e campus. After a first round of the system, and knowing nothing more than the pipe diameter and the pump’s power level, the marbles managed to produce a map that corresponded with reality with a 78% accuracy. Then, the tricky part of the test started: the marbles had to evolve independently in the constructed virtual environment. After 500 simulation rounds, the optimization process was finished.
But the researchers didn’t know yet whether the balls had actually become smarter. That would only become apparent after the next round through the system. And sure enough, after the second round, the marbles managed to produce a map with 92% accuracy. Furthermore, they had used 20% less energy this time around. Exactly as the researchers had hoped. The process could even be reproduced. Mission accomplished! Baltus: ‘An improvement of 78% to 92% doesn’t seem much perhaps, but you have to keep in mind that in an underground network of pipes, a deviation of just 1% means for companies that they are digging a couple of meters in the wrong direction. We expect that we will be able to generate much larger improvements when we use a more extensive system, with complicated curves and lots of branches.’
Baltus: 'I draw a parallel with evolution, because the process is so unpredictable. We don’t have a solution, so we let an algorithm do it. This algorithm mixes the most successful balls and works towards an optimum. We don’t know the results of that simulation process beforehand. Perhaps the algorithm decides to give each marble its own task, which could lead to location marbles and marbles that detect leakages. We haven’t been able to test this extensively, so we have to wait and see. The fact that our first evolution experiment shows that the principle works is actually unbelievable, and they even evolve in the right direction as well!’
Gönenç Berkol, PhD researcher at the TU/e, measures a printed circuit board with integrated circuits. He designed these circuits to enable very energy-efficient and flexible signals for an ultrasonic transducer to be generated and received and analysed again.
NAVIGATING WITH LIMITED POWER
Once you have a working system with smart marbles that evolve automatically towards a certain application, you encounter one more problem: with all the data collected by the sensors, the client would like to know the exact location of the marbles at that moment. In case the sensors detected a pipe burst, you obviously need them to store the location. That is why the location sensor is extremely important, but GPS doesn’t function underground. Luckily, there are alternatives.
One of them is the use of acoustic waves, which can actually work underground. Think of an ultrasonic system for instance, comparable to the echo’s that bats use. This solution might sound promising, but there are still a number of obstacles to overcome. Baltus: 'During an initial calculation, this system used no less than 5000 times more energy than all the other sensors and systems (including memory and the microprocessor) inside the small ball combined.'
In addition, sound velocity depends on the environment; you need to calibrate speed differently in water than in mud or oil. That’s difficult when you are in an unfamiliar environment. Besides, the ball transmits its data in just one direction with an ultrasonic system. The transmitter and the receiver need to ‘see each other’ for the ball to be able to transmit the signal. And an omnidirectional transmitter that transmits the signal in every direction does exist, but it will be so heavy that the ball would sink to the bottom. After extensive testing and continued development, an ultrasonic system might be a solution for a large number of applications. But if you really want to scale down the size of the marbles, you are going to need a different system.
The smart marbles will soon be tested at drinking water company Oasen in the province of South Holland. There, the marbles have to determine the precise location of the underground network, as the registration system does not always indicate the pipelines correctly. When carrying out maintenance work on the pipes, the drinking water company wants to prevent digging a few meters off. Moreover, the excavators must not cause damage to pipes and tubes. Technicians now often make small test slots, which they dig manually, to determine exactly where the main pipes are. The smart marbles offer a fast and efficient alternative without the need for digging.
In the event of a leak, the speed and direction of the water flow changes, which a movement sensor or compass can clearly show. In addition, a gyroscope can measure the turbulence around a leak, and a microphone picks up the typical hissing noises that occur. The researchers built an user interface to ensure an easy to use system for their customers. In the picture, PhD researcher Anil Yaman of the TU/e is programming this system.
THE ATTRACTION OF MAGNETISM
That is why the researchers came up with an unexpected alternative for the smaller marbles: magnetism. Baltus: ‘You can transport signals via the liquid in the pipeline network with this method. Reversing magnetic fields with low frequencies, for example. The ball can transmit its own serial number this way.’ Additional benefit: the signal goes in every direction, which makes picking it up much simpler. ‘A combination of magnetism and ultrasound might be a possibility. In that case, the systems calibrate each other, which enables you to make a reverse calculation of the sound velocity. You even collect new data that way, namely, the medium in which the balls float around,’ Baltus explains.
TU/e PhD researcher Haoming Xin is working on a circuit board for which he designed an integrated circuit with extremely low energy consumption (tens to hundreds of nanoWatt).
The magnetic system, however, only functions over short distances. Luckily, that is not a problem: since the marbles travel in a large swarm, there’s always a receiver nearby. The location is determined by their relative position to each other and by the entry point in the pipeline where the marbles were inserted. This provides the researchers with a very detailed map of the pipeline system once they read out all the information stored in the marbles.
MARBLE IN DISTRESS
Determining the location is even more important when you take into account how the marbles move through the pipelines. To reduce energy use, they passively flow with the current. Unfortunately, that gives you no control over their course. A ball could get stuck, especially in those places where the pipe is congested. And that’s exactly the kind of information that can be of great value to a client.
That is why Baltus fitted the marbles with an emergency system: ‘When a marble gets stuck, it uses its last bit of energy to send out a distress signal to the other balls nearby. In doing so, the marble also sends all the information it collected, so that no vital information is lost.’ The client can then potentially save the marble when an employee cleans the pipe.
INSPECTING OILFIELDS IN CANADA
One of the inspirations for the Phoenix project was a request for help from an oil company in Saskatchewan, Canada. The company tries to extract oil from tar sands with a depth of several hundred meters. They need two drilling rigs to accomplish this; one of these rigs pumps down water at high pressure so that the oil flows upwards through a second one, located several hundred meters away. But, much to the frustration of the oil companies in Saskatchewan, no less than 92% of the available oil remains in the ground. This is most likely caused by branches in the porous sandstone, which leads to a pressure drop.
The company wanted to find out what takes place deep under the surface, which prompted TU/e researcher Elena Talnishnikh (who was a researcher at INCAS3 at the time) to travel to Canada. ‘The extremely cold weather, with temperatures falling to -50 degrees Celsius, renders the oil so thick that they literally store it in paper boxes. Still, the marbles neatly resurfaced. It did turn out, though, that balls larger than seven millimeters are instantly pulverized by the water pumps. That means we have to go back as soon as we figure out how to make the balls smaller,’ Baltus explains.
Now, four years later, the Phoenix project has officially ended. Baltus: ‘We have shown that the concept and the evolution of the marbles works, and we’ve developed the technology that will enable us to significantly reduce the size of the marbles in the future.’ Baltus is clearly far from finished with the project. As a next step, he has started a spinoff under the name ‘Smarble', together with the Antea group. The purpose is to market the balls, a project in which Antea will take the lead. Smarble received a €100,000 euro grant from the Launchpad program to develop a business case, in addition to the HORIZON 2020 FET grant. ‘We want to produce robust and affordable swarms of marbles and put them into use. We don’t need new inventions for that, but it will take a lot of hard work to solve all kinds of practical problems,’ Baltus explains.
Dirkjan Krijnders (on the photo) and Erik Duisterwinkel, senior advisors at Antea Group, start together with Peter Baltus the spin-off called SMARBLE. Krijnders and Duisterwinkel work on the electronics and mechanics of the large marbles (based on existing "off-the-shelf" electronic components) and the corresponding firmware.
Besides Smarble, TU/e will continue with the research as well. Baltus: ‘We want to make the marbles even smaller, so that we can use them in healthcare, for example. Imagine that they’re small enough to flow through your bloodvessels. That would bring entirely new possibilities to the detection of obstruction. Researcher Giovanni Iacca from the university of Trento is starting up a consortium to reduce the size of the balls. He is currently looking for partners to apply for a new Horizon 2020 FET grant.’ Baltus also hopes to perform simple routines in the future. ‘Now, we only detect leaks, but in the future we also want to be able to close them right away with a kind of resin.’ Baltus plans to apply for a grant for this as well. ‘Once we’ve managed to land those subsidies, I’ll be able to start recruiting new PhD candidates and postdocs.’
DO IT YOURSELF
Anyone with an interest will be able to reproduce the balls and the piping system at home with nothing more than a few simple products from an electronics and construction store. 'We make all the hardware and software we develop publicly available on the Phoenix website. We hope that smart people will get to work on developing applications that we hadn't thought of before.'' Peter Baltus' dream? That this system will soon become the go-to product for exploring inaccessible places!
At the same time, the optimization process of the current balls will continue. ‘We want to show that our marbles can be applied in all kinds of ways. For that, we need the help of companies who are willing to let us use their processes as test locations. We will soon be testing mixing processes at Jongia and drinking water pipelines at Oasen, but are looking for many more applications for testing.’
But bachelor’s and master’s students, too, are always welcome to help Baltus further develop the balls. ‘The great thing about this work is its multidisciplinary nature. Recruiting researchers from the field of Electrical Engineering alone is not enough. For instance, I also need mechanical engineers and construction engineers, as well as experts on software, user interfaces, artificial intelligence, evolution, you name it. I hope to sit down with Innovation Space soon, to start a new program with a group of bachelor’s students from several different programs at TU/e.’