Data in a container and the writable layer

The writable layer

An image is a stack of read-only layers. When a container runs, Docker adds a thin writable layer on top. Every file the container creates or changes (logs, uploads, databases…) lives there by default.

flowchart BT
  L1["Image layer: base OS (read-only)"] --> L2["Image layer: nginx (read-only)"]
  L2 --> W["Writable layer of container web"]
  W -.->|"docker rm web"| X["Gone with the container"]

Three drawbacks of the writable layer:

• Lost when the container is removed. docker stop and then docker start keep the data; only docker rm deletes the writable layer.
• Tied to the host running the container, so it is hard to move to another host or share with another container.
• Slower, because every write goes through the storage driver (copy-on-write). Writes to a volume go straight to the host filesystem.

Bottom line: data you need to keep belongs in a volume or bind mount, not in the writable layer. Do not rely on docker commit to "save" data into an image.

Read-only containers

A strong habit for production is to make the container filesystem read-only and give the app only the writable paths it needs:

docker run -d --name web --read-only --tmpfs /tmp --tmpfs /var/cache/nginx \
  -v web-logs:/var/log/nginx nginx:1.27

If an attacker gets in, they cannot drop tools or change binaries, and any unexpected write fails loudly, which reveals hidden state you had not planned for.

Where container data goes on the host

On Linux, Docker keeps everything under /var/lib/docker: image layers, writable layers, volumes, and container logs (/var/lib/docker/containers/<id>/<id>-json.log). A full disk there stops new containers from starting. docker system df -v breaks usage down by image, container, and volume, and docker ps -as shows which containers have grown large writable layers, usually because an app writes logs or cache to its own filesystem.

Logs belong on stdout

Containers should write logs to stdout and stderr, not to files inside the container. Docker's logging driver then collects them, docker logs can show them, and a log shipper can forward them. Writing to files inside the writable layer grows the container and loses the logs when it is removed.

What docker commit is for

docker commit web web:debug turns a container's current state into an image. It is useful for saving a broken container to investigate later, but not for building real images: nobody can see how the image was made, and it cannot be rebuilt. Use a Dockerfile for anything you will deploy.

Thinking stateless

The rule of thumb from twelve-factor apps is that containers are disposable: any container can be stopped, removed, and recreated from its image at any time. Everything that must survive goes to a volume, a database, object storage, or a configuration service. This is exactly what Kubernetes assumes later in the track.