Docker containers that keep stopping are challenging to debug. How can you determine what’s happening in the container if it won’t run?
Fortunately, all is not lost. In this guide, we’ll walk through some simple steps to keep a container running to enable more efficient debugging. We’ll also share some quick troubleshooting steps and best practices to follow when containers stop.
Docker containers are isolated filesystems that run a single foreground process. Docker uses Linux user namespaces to keep the containerized process separate from your host, even though it’s actually executed by your host’s kernel.
Because containers run just one foreground process, they stop when that process terminates. There’s no process manager or init service to keep the container running once the process exits.
A regular Linux system has init as its PID 1, but PID 1 in a container is the image’s configured entrypoint command:
$ docker run -it --rm ubuntu:latest bash
root@64ddd1f6b4b5:/# top
top - 14:27:08 up 3:51, 0 user, load average: 0.32, 0.10, 0.07
Tasks: 2 total, 1 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 1.3 sy, 0.0 ni, 96.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.3 st
MiB Mem : 7940.4 total, 5420.2 free, 1289.5 used, 1526.5 buff/cache
MiB Swap: 1710.0 total, 1710.0 free, 0.0 used. 6650.8 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 4588 3968 3456 S 0.0 0.0 0:00.03 bash
10 root 20 0 8956 5248 3072 R 0.0 0.1 0:00.00 topPID 1 in the container is bash, the command passed to docker run.
All this means that containers will stop if a bug terminates their foreground process. For instance, a containerized Node.js application will stop if buggy code crashes Node.js.
Containers may also stop for other reasons, such as running out of CPU or memory resources. These conditions are often tricky to debug because it’s not always immediately obvious what caused the stoppage.
Moreover, critical issues that prevent the containerized process from running can cause containers to get stuck in a state where they can’t be started. Your app may be crashing on launch due to improperly configured dependencies, for example, causing the container to stop immediately after you create it with docker run.
The container will restart automatically if you’ve configured a restart policy that allows it, but it will then stop immediately again if the underlying issue persists.
Take the following Dockerfile: It containerizes a simple Node.js app that tries to run an Express web server:
FROM node:latest
RUN echo 'const express = require("express");express().listen(3000, () => null);' > main.js
CMD ["main.js"]Unfortunately, containers started from this image will stop straightaway because the express npm dependency isn’t installed within the container:
$ docker run --name demo demo:latest
node:internal/modules/cjs/loader:1408
throw err;
^
Error: Cannot find module 'express'You’ll see the container’s status displayed as Exited with a non-zero exit code in the docker ps -a command’s output:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad4d29e56db7 demo:latest "docker-entrypoint.s…" 8 seconds ago Exited (1) 7 seconds agoNow, let’s look at how to troubleshoot these problems and get the container running again for debugging.
When a container stops unexpectedly, you need to solve the root cause, but you may need to get the container running first to enable effective debugging.
For instance, inspecting environment variables, files in mounted volumes, and the packages installed in the container image’s filesystem can provide vital clues to what caused the container to stop.
Unfortunately, the built-in docker start command isn’t useful if the problem causes the container to stop immediately again. Because docker start doesn’t allow you to override the container’s configured command, you can’t use it to drop into a shell or another process within an existing container.
By comparison, the debugging techniques shown below will create a new container based on the same image and keep that container running even if the image’s default command throws an error.
The methods to keep Docker containers running include:
- Overriding the container’s command to open an interactive shell session
- Using tail -f /dev/null
- Using sleep infinity
- Committing the container’s state to a new image
- Modifying the container image to wrap the default entrypoint with a custom script
Method 1. Override the container’s command to open an interactive shell session
You can pass a command to docker run to override the process that will be run in the container. You can use this in conjunction with the -i (interactive) and -t (assign a pseudo-TTY) flags to begin a shell session that lets you inspect the container’s environment:
# Connects your terminal to a Bash shell inside the container
$ docker run -it demo:latest bashOnce you’re inside the container, you can try manually launching the standard foreground process referenced in your Dockerfile. This will let you observe its output. You can also use the shell session to list environment variables and check which packages are available in the filesystem.
Not all container images will include bash, in which case this command will fail with an error. You can try to use sh instead, as this simpler alternative can be found in most popular images.
Method 2. Use tail -f /dev/null
Another way to keep a container running is to override its foreground process to run the tail command in the following mode (-f) against a permanently live stream like /dev/null. This technique allows you to run the container indefinitely in the background.
# The container will run indefinitely because it's tailing the /dev/null virtual device
$ docker run -d --name demo demo:latest tail -f /dev/null
a6231f6f9bd6f911abf54485abf11fdaf2237e33076e9fcbe8fbabe86601ddc8Once you’ve got the container started with this dummy process, you can then use docker exec to test various commands inside the container:
$ docker exec demo node main.js
node:internal/modules/cjs/loader:1408
throw err;
^
Error: Cannot find module 'express'By using docker exec to run the container’s normal command, as in this example, you can easily inspect the error that’s making that process stop. You could then try using a shell session to edit files and run commands in the container. The container will stay running until you manually stop it, ensuring you can easily test the results of your changes.
Once you’re finished, remember to apply them to your Dockerfile and rebuild your image.
Method 3. Use sleep infinity
This is a variant of the previous technique. Starting a container with its command set to sleep infinity will keep the container idle indefinitely. The sleep command starts a process that will run for a specified time – infinity, in this case, so it will never end.
$ docker run -d --name demo demo:latest sleep infinity
836e13e6724f469825c3f6aa7d4b33356e300d0215aacffbb06c7505305b1302You can then use the same techniques as discussed above to debug what’s happening in the container.
Method 4. Commit your container’s state to a new image
The docker commit command is a more creative way to keep a container running. It allows you to create a new container image from an existing container, even if that container’s already stopped. You can then modify the new container image to install debugging tools and change its default command.
It’s even possible to add extra Dockerfile instructions to the image via flags passed directly to docker commit.
The following example commits the container named demo to a new image tagged as debug-demo:latest. The new image has sleep infinity configured as its default command, so containers created from it will stay running for debugging.
$ docker commit --change='CMD ["sleep", "infinity"]' demo debug-demo:latest
sha256:7548c1e617b345fc1e30c430f81d4534493c7c992c2b12c7fefcccc91125b6a9
# Stays running indefinitely
$ docker run -d debug-demo:latestMethod 5. Modify the container image to wrap the default entrypoint with a custom script
A final option for debugging containers while keeping them running is to modify your container’s entrypoint so that it stays alive even after your main process terminates. You can achieve this with a simple wrapper script like the following:
#!/bin/bash
node main.js;
sleep infinity;You should then amend your Dockerfile’s ENTRYPOINT and CMD instructions so the script runs when the container’s started:
# <existing instructions>
COPY entrypoint.sh .
ENTRYPOINT ["bash"]
CMD ["entrypoint.sh"]In this example, we run the container’s regular foreground process (node main.js) within the custom entrypoint script. If that process terminates, the script execution moves to the next line and runs sleep infinity. This will keep the container running indefinitely, as discussed above.
Because the script becomes the container’s PID 1, not node main.js, termination of the Node process no longer impacts the container.
This approach allows you to easily view the output from the failed original process using docker logs, then jump straight into debugging without having to start another container.
The techniques we’ve shown are easy ways to start and reliably keep a Docker container running. Let’s recap with some quick tips and best practices for debugging containers that unexpectedly stop.
1. Access the container’s logs to identify why the container’s process is stopping
Container logs may reveal what happened inside the container in the moments before it stopped. For instance, if a crash made the container terminate, there’ll typically be a stack trace from your code.
You can use the docker logs command to retrieve a container’s logs. It works even if the container is currently stopped.
$ docker logs <container>2. Check the container’s exit code
Container exit codes may reveal why the container has stopped. These are visible when you run docker ps -a:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ad4d29e56db7 demo:latest "docker-entrypoint.s…" 8 seconds ago Exited (1) 7 seconds agoA non-zero exit code indicates an error occurred. To check the meaning of an exit code, you should refer to the app documentation you’re using. If you’re containerizing your own app, use a unique exit code for each type of scenario to clearly distinguish different failure reasons.
Some exit codes, such as 137, have common meanings. This code indicates the container was killed due to an out-of-memory condition, so it could restart successfully if more memory is assigned.
3. Start a shell session inside the container to investigate its environment
Starting a new container with a shell like bash or sh as its foreground process is arguably the easiest way to ensure it stays running for debugging. This guarantees the container runs while your shell session is open and lets you inspect the container’s environment from the inside.
We’ve documented this technique in more detail above.
4. Implement container monitoring to get notified when containers fail
This won’t help keep your containers running, but using a monitoring system like cadvisor or Grafana on your Docker hosts can help you detect and respond to container failures as they happen. Being alerted when containers stop or get stuck in a restart loop lets you minimize the disruption faced by users.
5. Use a process manager to keep container processes running
Finally, consider adding a process manager to your container images as an advanced option for keeping containers running. supervisord is a utility that keeps multiple sub-processes active and automatically restarts them if they fail.
Running supervisord as your container’s command makes it PID 1, so the container will stay running as long as supervisord is alive.
Running multiple processes in a container generally goes against containerization principles, so this option should be used with care. It’s best to have each container perform just one job, as this makes them more portable and scalable. However, process managers can make it easier to run debugging tools and services in your containers alongside your main foreground processes.
We also encourage you to explore the ways Spacelift offers full flexibility when it comes to customizing your workflow. You can bring your own Docker image and use it as a runner to speed up deployments that leverage third-party tools. Spacelift’s official runner image can be found here.
If you want to learn more about what you can do with Spacelift, check out this article, create a free account today, or book a demo with one of our engineers.
Docker containers stop if their foreground process terminates. This could happen due to exceeded resource constraints or a crash in your application’s code. Docker will display the container’s status as Exited with a code that can help explain what happened. The container may be restarted automatically, but could stop again if the same event occurs.
Use the techniques and best practices discussed in this guide to debug stopped containers efficiently. Utilizing logs, exit codes, shell sessions, and process managers lets you uncover what’s happening in your container so you can diagnose why it’s stopping. Once you’ve resolved the root cause, your container should start and run healthily again, ready for reliable long-term operations.
You may occasionally want to stop a container, such as to temporarily take a service offline. If that’s the case, we have a separate article on stopping, removing, and cleaning up containers. Finally, if you’re having problems with containers stopping in Kubernetes, read our guide to the CrashLoopBackOff error to debug the problem using similar techniques.
Solve your infrastructure challenges
Spacelift is a flexible orchestration solution for IaC development. It delivers enhanced collaboration, automation, and controls to simplify and accelerate the provisioning of cloud-based infrastructures.
![What are Docker Build Args & How to Use Them [Tutorial]](/_next/image?url=https%3A%2F%2Fspaceliftio.wpcomstaging.com%2Fwp-content%2Fuploads%2F2024%2F12%2F492.docker-build-args.png&w=3840&q=75)
![How to List Docker Containers [All, Running, Stopped, and More]](/_next/image?url=https%3A%2F%2Fspaceliftio.wpcomstaging.com%2Fwp-content%2Fuploads%2F2024%2F04%2F370.docker-list-container.png&w=3840&q=75)
