Docker containers memory troubleshooting

I’ve been playing around with docker for about a week now to figure out if its the right tool for what we’re planning to do. I’ll briefly describe the setup and dive into the issue. The sandbox environment is setup in a vagrant box(referred to as the ‘host machine’ from now on) running Ubuntu 13.04(updated the default kernel). Memory assigned to the host machine is 512.

The base image is available on docker index and is about 2.5 GB in size. It contains ubuntu 12.04 lts and tons of packages installed in it. The objective is to run containers on top of Base Image and to see how many of these can be fit into. For every container which is spawned from the Base Image, mysql and postgres daemons are started automatically. This is just to make sure that there are some processes running on the containers when they come up. Yes, this was a long setup process, and I plan to go into it later. But first things first.

First Run (no swap space in host machine):

From the host machine I ran the following command/

$ docker run -d -p 22 ric03uec/shippable /start.sh

(The “start.sh” is the container initialization script. Details of it don’t matter for the scope of this post)

And it succeeded! So far so good. I had one docker container successfully running as show. Time to take it up a notch. I run the same command 4 more times and then WHAM. I get the following slammed on the screen.

$ docker run -d -p 22 ric03uec/shippable /start.sh

runtime: panic before malloc heap initialized

fatal error: runtime: cannot allocate heap metadata

I thought docker was supposed to run thousands of containers unlike the VM’s. The good folks at docker had a solution specified HERE. Turns out the memory limit was very low and hence the host machine couldn’t bring up more containers. The first instinct was to just increase the allocated memory for the host machine to about 1 or 2 GB but that doesn’t completely make sense. There had to be better way to use the existing system as it wasn’t doing anything else.  This led me to evaluate how could the swap space in the host machine be utilized and whether it could make any difference in the outcome.

VM Swap changes:

Turns out, there was no swap space available in the host machine as specified by ‘free’ command. I followed THIS post to create a 1 GB swap space from a file for the host machine. In short, the following commands (with sudo) will create a file which will be used as swap.

Check if swap is ready by using ‘free’ command.

Second Run(1GB swap):

I again started with the same command

$ docker run -d -p 22 ric03uec/shippable /start.sh

This time I could do this exactly 25 times before I saw the same error again, which means 25 running containers. This was a significant boost over what the system was doing and all this by just adding the swap space.

The system throughput, though, takes some beating when the containers are started. The following is a snapshot of vmstat when some containers were spawned in quick succession.

The ‘free’ command also reflects the same trend. When containers are spawned, the free Mem drops and after a few seconds, the OS reclaims some of the memory back by swapping out pages to the swap partition, which shows a noticeable increase in size.

At the time docker gives up and spits out the error, the swap has 10% empty space left while only 5% of spare memory is available(which i bet we’ll have hard time getting from the OS). This is shown in the following snapshot.

Manual Recovery:

Docker command will keep failing after this unless some of the containers are manually killed. This can be handled more gracefully by the docker command itself, but for the time being, you’ll have to run ‘ps -eaf | grep docker’ and sacrifice some of your beloved containers for docker to recover and be useful again.

In conclusion, this was a good exercise for me to figure out the limitations of docker. I love the tool but it still a rough at the edges and needs a thorough inspection of all fail-cases before using it in production.

 

Share Comments
comments powered by Disqus