How machines learn to learn

Some are picked up quickly with a truck from Hamburg, others have to wait for a feeder ship that will transport them on to a port in the Baltic Sea. So when determining the storage space, nobody knows when the storage crane will have to take the container back out of the compact block. This means many have to be moved repeatedly.

Is it possible to store them in such a way that the storage crane only has to move them once? To solve this problem you need more than just common sense. “Even when the yard is only 80 percent full, with 100 different dwell times we are left with 100¹⁹⁰⁰ possibilities for stacking the containers in the yard,” explains Jan Niklas Sikorra.

The senior consultant for the HHLA consulting company HPC Hamburg Port Consulting, which specialises in port, logistics and transport issues, thus turned his attention to entirely new options. Reinforcement learning is what Sikorra came across. In this most modern variation of artificial intelligence (AI) the AI module doesn’t blindly pursue a predetermined target. Programmed “agents” move in a virtual training environment and improve themselves using a reward function.

In short: They teach themselves the ideal solution to a task and the way to achieve it. In this case, they are looking for the needle of the ideal storage space combination in the incredibly complex container haystack. Together with other HPC experts, Sikorra is using this principle to develop an intelligent block storage management system. An AI that independently trains itself to find the best order for every container placement.

Different applications, summarised under the term artificial intelligence, are already in use at HHLA’s Altenwerder (CTA) and Burchardkai (CTB) terminals. Usually the focus is on achieving the optimal transport and logistics processes, but the creation of reliable forecasts on the service life and expected damage to equipment is another important issue.

In the terminal's management systems, AI modules close the gaps in the information available about the containers handled. The dwell time of every second container that we bring on land is unknown to us. For every tenth container there is no information regarding whether they will be transported onwards by ship, rail or truck.

Before they were assisted by AI modules, the terminal logistics staff had to rely on their experience. Today, the AI module compares 30 known data sets about the container in question with information about loads handled in the past and forecasts the probable dwell time and the mode of transport. This allows the ideal storage space to be selected with a high level of accuracy.

The system is one of the most modern port logistics processes – but it still has a handicap. It is something that Sikorra intends to eliminate: “The current storage management system is based on very complex algorithms and the AI module needs regular updating.”

The starting point for the new approach is that the module should learn independently and recognise the ideal solution. The HPC concept is based on the belief that ‘electronic brains’ could act like humans one day and learn independently through trial and error.

For many decades this remained a dream as computers lacked the processing power. But computing capacity has grown exponentially; supported by powerful frameworks and methods to analyse information and train networks, methods such as reinforcement learning are now reaching new dimensions in artificial intelligence.

Sikorra has now adapted the reinforcement learning principle so well to real terminal operations that he has been able to implement it in a model version. He got the virtual logistics agent to stack 800 containers in 100 stacks. They learned to choose storage spaces in such a way that the number of times a container has to be moved between entry and exit is minimised. Next, the consultant from Hamburg would like the learning agents to work in scenarios that correspond to the real conditions in a container terminal.

How long until it is ready? That’s hard to say due to the high demands that would be made of such a system, but the first decisive step in a quantum leap for the organisation of port processes has been taken.