In this article, we will take a look into the exit code 127 in Kubernetes (K8s), explaining what it is, what the common causes of it are in K8s and Docker, how to find the exit code status in the first place, and finally how to fix it!
We will cover:
Exit codes, also known as return codes or exit statuses, are numeric values returned by a process or command to the operating system after it has finished execution. These codes are used to convey the result or status of the execution to the calling process or script.
In Unix-like operating systems, including Linux, and command-line interfaces generally, exit codes are a standard mechanism for communicating success, failure, or specific conditions back to the calling environment.
The exit code 127 is not a specific Kubernetes error code but a standard exit code used by Linux and Unix-like operating systems. However, it is commonly seen in Kubernetes environments and usually indicates that the command or binary being executed within a container could not be found.
Some standard exit codes include:
- 0: Success
- 1: General errors
- 2: Misuse of shell builtins
- 126: Command invoked cannot execute
- 127: Command not found
- 128–255: Command died due to a signal
Read also about the Exit Code 137 – OOMKilled error in Kubernetes.
Let’s look at some common causes of the exit code 127.
1. Command or binary not installed
The executable specified in the command field of a Kubernetes container is not installed in the container’s filesystem. Ensure that the necessary binaries or commands are available.
2. Incorrect path or command
The command specified in the Pod definition is incorrect or not present in the specified path. This is one of the most common causes of the error and is usually caused by an incorrect entrypoint or command in the Dockerfile or pod specification.
3. Missing dependencies
The application or script being run within the container has dependencies that are not installed. Make sure that all required dependencies are included in the container image.
4. Shell interpreters
If a script is specified as the command, ensure that the script has a valid shebang line (e.g., #!/bin/bash) and that the interpreter specified in the shebang is available in the container.
5. Shell script syntax errors
If a shell script is failing with exit code 127, check the script for syntax errors or issues that might prevent it from being executed.
6. Insufficient permissions
The user running the command within the container might not have the necessary permissions to execute the specified command. Ensure that the container is running with appropriate privileges.
7. Image compatibility
Ensure that the container image being used is compatible with the architecture and operating system of the underlying node. Mismatched images may result in commands not being found.
8. Volume mounts
If the command relies on files or binaries mounted from a volume, check that the volume mounts are correctly configured and that the required files are accessible.
9. Environment variables
Some commands might rely on specific environment variables. Ensure that the required environment variables are set correctly.
10. Kubernetes RBAC policies
If RBAC (Role-Based Access Control) is in use, ensure that the service account associated with the pod has the necessary permissions to execute the specified commands.
To troubleshoot and diagnose the issue, you can inspect the logs of the Pod using the following command:
kubectl logs <pod-name>You can also check the pod status, which provides detailed information about the pod, including its current state, recent events, and any error messages.
kubectl describe pod <pod-name>You could also add a container to your Pod specifically for debugging purposes that includes a shell (e.g., BusyBox). This allows you to enter the container and manually inspect the environment, paths, and command availability.
Example of using BusyBox for debugging:
containers:
- name: my-container
image: my-image:latest
command: ["/bin/sleep", "infinity"]
- name: debug-container
image: busybox:latest
command: ["/bin/sh"]
tty: true
stdin: trueBy carefully reviewing the logs and investigating the mentioned aspects, you should be able to identify the cause of the exit code 127 issue in your Kubernetes pod. Addressing the specific problem revealed by the logs will guide you in resolving the underlying problem and ensuring the successful execution of your commands within the pod.
Now that we know what the common causes are and how to diagnose the exit code 127 issue let’s see how to fix it.
Command or Binary Not Installed
If the required command or binary is missing, you may need to install it in your container image. Modify your Dockerfile or build process to include the installation of the necessary software.
Example (for a Dockerfile using Alpine Linux):
FROM alpine:latestRUN apk --no-cache add <package-name>Incorrect path or command
When specifying a command in your Pod definition, consider using an absolute path to the binary. This helps ensure that Kubernetes can find the binary regardless of the current working directory.
Example:
containers:
- name: my-container
image: my-image:latest
command: ["/usr/local/bin/my-command"]Missing dependencies
One cause of the command not being able to be run is that you may need to install extra packages in your container image so the command you are running can be called successfully. Ensure the command you are calling is able to be called from the terminal.
If the command requires additional setup or installation steps, consider using an init container to perform these tasks before the main container starts.
Example (using an init container to install a package):
initContainers:
- name: install-package
image: alpine:latest
command: ["apk", "--no-cache", "add", "<package-name>"]
volumeMounts:
- name: shared-data
mountPath: /data
Shell interpreters
If you are running a script, ensure that the script has a valid shebang line specifying the interpreter. Also, make sure the interpreter is available in the container.
Example shebang line in a Bash script:
#!/bin/bashVolume mounts
Check your Pod definition to ensure that the volume mounts are correctly configured. Verify that the volume names, mount paths, and subPaths (if used) are accurate.
Example:
volumes:
- name: my-volume
emptyDir: {}
containers:
- name: my-container
image: my-image:latest
volumeMounts:
- name: my-volume
mountPath: /path/in/container
You should also ensure that the volume specified in the Pod definition exists. If it’s a PersistentVolume (PV), check its status. If it’s an emptyDir or other volume type, verify that it’s being created and mounted correctly.
If you are using subPaths in your volume mounts, ensure that the specified subPaths exist in the source directory or file.
Example:
volumeMounts:
- name: my-volume
mountPath: /path/in/container
subPath: my-file.txt
Check out also how to fix CreateContainerConfigError and ImagePullBackOff error in Kubernetes.
Exit code 127 is not a specific Kubernetes error code, but rather a general error message in Unix and Linux systems that usually means ‘Command not found’. In K8s environments, it can be caused by numerous issues as listed above. Resolving it involves checking the logs and events related to the error to track down the cause.
We encourage you to also check out how Spacelift helps you manage the complexities and compliance challenges of using Kubernetes. Anything that can be run via kubectl can be run within a Spacelift stack. Find out more about how Spacelift works with Kubernetes, and get started on your journey by creating a free trial account.
Manage Kubernetes Easier and Faster
Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.
