How to fix expanding docker containers

4 min readAug 29, 2020

Do you have a docker container whose size keeps growing? Even when the container size increases slowly it means trouble when the container is expected to run for months and years. This article explains a common cause of the problem, shows how to diagnose it and how to fix it.

UnionFS

Docker containers are built using the Union File System (UnionFS). UnionFS works in layers. When anything is written, changed, or deleted from the file system, a new layer is created reflecting the change. This makes it easy to store snapshots and to share common content between docker images, but it works best with read-only docker images. Adding content, changing content, or even deleting content increases the image size by introducing new layers. Even when deleting files from a docker image, the deleted files continue to consume space. The deletion only creates another layer that marks the files as deleted

The Problem

The problem occurs if an application in the container periodically writes to a disk area that is mapped inside the container. Each write, modify, or even delete operation increases the container size.

Confirming the Problem

The following commands, performed on the docker host can confirm the problem.

docker ps --size
docker system df
docker system df -v

These commands, when executed on the docker host, show the sizes of the containers on that host. Run these commands every few days and record the result. If the container size is constantly increasing, we have confirmed the problem.

Troubleshooting the Problem

Once we know we have a problem the next step is to find the cause. For that, we need to track the size of the layers created by UnionFS and identify the directories that are changing. Docker stores its layers in the following location:

/var/lib/docker/overlay2

A simple strategy is to take a snapshot of the sizes of all the layers every few days. This can be done with a command such as:

sudo du -h /var/lib/docker/overlay2 > <timestamp>overlay2

where <timestamp> is a timestamp, such as 2020–08–23- that helps tell apart the different snapshots.

When the container size has increased you can find the cause by using diff. For example:

diff 2020–08–20-overlay2 2020–08–23-overlay2

For example, I had a postfix server in a docker container. The container size expanded a bit every time a message was processed. Using the above method I got:

$ diff 2020-08-20-overlay2 2020-08-23-overlay2
61372c61372
< 36K /var/lib/docker/overlay2/66ab82e5e9df739182d40cdf2a58128ae7d27ccd36b7a11fba2eb717390e56bc/merged/var/lib/postfix
 — -
> 32K /var/lib/docker/overlay2/66ab82e5e9df739182d40cdf2a58128ae7d27ccd36b7a11fba2eb717390e56bc/merged/var/lib/postfix
78239c78239
< 36K /var/lib/docker/overlay2/66ab82e5e9df739182d40cdf2a58128ae7d27ccd36b7a11fba2eb717390e56bc/diff/var/lib/postfix
 — -
> 32K /var/lib/docker/overlay2/66ab82e5e9df739182d40cdf2a58128ae7d27ccd36b7a11fba2eb717390e56bc/diff/var/lib/postfix

Clearly something was modified in the directory /var/lib/postfix in the docker container.

The Fix

To fix the problem the application can be modified to store its varying data in a location outside the docker container. If it is not desirable to change the application, the directory where the changing information is stored can be mapped outside the docker container.

Docker provides several methods for accomplishing this:

bind mounts
map a folder in the container to a location in the host. The location on the host is seen and managed by the user.
volumes
Same as bind mount but the location on the host is managed by docker. The user can look up that location and directly access it, but typically this is left for docker. Docker has commands to list volumes and remove volumes that are no longer used.
tmpfs mounts
map a directory in the container to memory.

Setting Up Storage Area Outside the Container

This covers tutorial covers two methods: the docker command and docker-compose. In both methods, the old container must be deleted and a new container with the external storage is created.

The docker command

Bind mount example:
$ docker run -d \ -it \ --name devtest \ --mount type=bind,source="$(pwd)"/target,target=/app \ nginx:latestThis maps the directory "$(pwd)"/target on the host to the directory /app in the container.

Volume example:
First, the volume needs to be created.
$ docker volume create my-volThe volume is created and stored in an area managed by the docker engine.
Now run the container while mounting the volume:
$ docker run -d \ --name devtest \ --mount source=myvol2,target=/app \ nginx:latest

tmpfs example:
This method is ideal for small amounts of storage with frequent changes where execution speed is important.$ docker run -d \ -it \ --name tmptest \ --mount type=tmpfs,destination=/app \ nginx:latest

The docker-compose way

I like to use docker-compose. It is self-documenting, git friendly, and reproducible.

In docker-compose there are a long-form and a short form.

Short form:
volumes: # Bind mount example: Specify an absolute path mapping - /opt/data:/var/lib/mysql

# Bind mount example: the path on the host is relative to the docker-compose file - ./cache:/tmp/cache

# Bind mount example: the path on the host is relative to the user's home directory - ~/configs:/etc/configs/:ro

# Volume example: Just specify a path in the container and let the Engine create a volume on the host - /var/lib/mysql

# Named volume - datavolume:/var/lib/mysql

# The named volumes specified above must be defined here volumes: datavolume:

Long-form:

version: "3.8"
services:
  web:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - type: volume
        source: mydata
        target: /data
        volume:
          nocopy: true
      - type: bind
        source: ./static
        target: /opt/app/static

volumes:
  mydata:

Note: When creating bind mounts, using the long syntax requires the referenced folder to be created beforehand. Using the short syntax creates the folder on the fly if it doesn’t exist. See the bind mounts documentation for more information.

Retest

Now, use the technique described above to make sure the container size is no longer expanding

References