Going to AWS re:Invent 2024?

➡️ Book a meeting with Spacelift

Kubernetes

Exit Code 127 – Fixing File or Directory Not Found Error in Kubernetes

Exit Code 127

In this article, we will take a look into the exit code 127 in Kubernetes (K8s), explaining what it is, what the common causes of it are in K8s and Docker, how to find the exit code status in the first place, and finally how to fix it!

We will cover:

  1. What is the exit code 127 error message?
  2. Common causes of the exit code 127 in Kubernetes
  3. Exit code 127 diagnosis
  4. How to fix exit code 127?

What is the exit code 127 error message?

Exit codes, also known as return codes or exit statuses, are numeric values returned by a process or command to the operating system after it has finished execution. These codes are used to convey the result or status of the execution to the calling process or script.

In Unix-like operating systems, including Linux, and command-line interfaces generally, exit codes are a standard mechanism for communicating success, failure, or specific conditions back to the calling environment.

The exit code 127 is not a specific Kubernetes error code but a standard exit code used by Linux and Unix-like operating systems. However, it is commonly seen in Kubernetes environments and usually indicates that the command or binary being executed within a container could not be found.

Some standard exit codes include:

  • 0: Success
  • 1: General errors
  • 2: Misuse of shell builtins
  • 126: Command invoked cannot execute
  • 127: Command not found
  • 128–255: Command died due to a signal

Read also about the Exit Code 137 – OOMKilled error in Kubernetes.

Common causes of the exit code 127 in Kubernetes

Let’s look at some common causes of the exit code 127.

1. Command or binary not installed

The executable specified in the command field of a Kubernetes container is not installed in the container’s filesystem. Ensure that the necessary binaries or commands are available.

2. Incorrect path or command

The command specified in the Pod definition is incorrect or not present in the specified path. This is one of the most common causes of the error and is usually caused by an incorrect entrypoint or command in the Dockerfile or pod specification.

3. Missing dependencies

The application or script being run within the container has dependencies that are not installed. Make sure that all required dependencies are included in the container image.

4. Shell interpreters

If a script is specified as the command, ensure that the script has a valid shebang line (e.g., #!/bin/bash) and that the interpreter specified in the shebang is available in the container.

5. Shell script syntax errors

If a shell script is failing with exit code 127, check the script for syntax errors or issues that might prevent it from being executed.

6. Insufficient permissions

The user running the command within the container might not have the necessary permissions to execute the specified command. Ensure that the container is running with appropriate privileges.

7. Image compatibility

Ensure that the container image being used is compatible with the architecture and operating system of the underlying node. Mismatched images may result in commands not being found.

8. Volume mounts

If the command relies on files or binaries mounted from a volume, check that the volume mounts are correctly configured and that the required files are accessible.

9. Environment variables

Some commands might rely on specific environment variables. Ensure that the required environment variables are set correctly.

10. Kubernetes RBAC policies

If RBAC (Role-Based Access Control) is in use, ensure that the service account associated with the pod has the necessary permissions to execute the specified commands.

Exit code 127 diagnosis

To troubleshoot and diagnose the issue, you can inspect the logs of the Pod using the following command:

kubectl logs <pod-name>

You can also check the pod status, which provides detailed information about the pod, including its current state, recent events, and any error messages.

kubectl describe pod <pod-name>

You could also add a container to your Pod specifically for debugging purposes that includes a shell (e.g., BusyBox). This allows you to enter the container and manually inspect the environment, paths, and command availability.

Example of using BusyBox for debugging:

containers:
  - name: my-container
    image: my-image:latest
    command: ["/bin/sleep", "infinity"]
  - name: debug-container
    image: busybox:latest
    command: ["/bin/sh"]
    tty: true
    stdin: true

By carefully reviewing the logs and investigating the mentioned aspects, you should be able to identify the cause of the exit code 127 issue in your Kubernetes pod. Addressing the specific problem revealed by the logs will guide you in resolving the underlying problem and ensuring the successful execution of your commands within the pod.

How to fix exit code 127?

Now that we know what the common causes are and how to diagnose the exit code 127 issue let’s see how to fix it.

Command or Binary Not Installed

If the required command or binary is missing, you may need to install it in your container image. Modify your Dockerfile or build process to include the installation of the necessary software.

Example (for a Dockerfile using Alpine Linux):

FROM alpine:latest
RUN apk --no-cache add <package-name>

Incorrect path or command

When specifying a command in your Pod definition, consider using an absolute path to the binary. This helps ensure that Kubernetes can find the binary regardless of the current working directory.

Example:

containers:
  - name: my-container
    image: my-image:latest
    command: ["/usr/local/bin/my-command"]

Missing dependencies

One cause of the command not being able to be run is that you may need to install extra packages in your container image so the command you are running can be called successfully. Ensure the command you are calling is able to be called from the terminal.

If the command requires additional setup or installation steps, consider using an init container to perform these tasks before the main container starts.

Example (using an init container to install a package):

initContainers:
  - name: install-package
    image: alpine:latest
    command: ["apk", "--no-cache", "add", "<package-name>"]
    volumeMounts:
    - name: shared-data
      mountPath: /data

Shell interpreters

If you are running a script, ensure that the script has a valid shebang line specifying the interpreter. Also, make sure the interpreter is available in the container.

Example shebang line in a Bash script:

#!/bin/bash

Volume mounts

Check your Pod definition to ensure that the volume mounts are correctly configured. Verify that the volume names, mount paths, and subPaths (if used) are accurate.

Example:

volumes:
  - name: my-volume
    emptyDir: {}
containers:
  - name: my-container
    image: my-image:latest
    volumeMounts:
    - name: my-volume
      mountPath: /path/in/container

You should also ensure that the volume specified in the Pod definition exists. If it’s a PersistentVolume (PV), check its status. If it’s an emptyDir or other volume type, verify that it’s being created and mounted correctly.

If you are using subPaths in your volume mounts, ensure that the specified subPaths exist in the source directory or file.

Example:

volumeMounts:
  - name: my-volume
    mountPath: /path/in/container
    subPath: my-file.txt

Check out also how to fix CreateContainerConfigError and ImagePullBackOff error in Kubernetes.

Key points

Exit code 127 is not a specific Kubernetes error code, but rather a general error message in Unix and Linux systems that usually means ‘Command not found’. In K8s environments, it can be caused by numerous issues as listed above. Resolving it involves checking the logs and events related to the error to track down the cause.

We encourage you to also check out how Spacelift helps you manage the complexities and compliance challenges of using Kubernetes. Anything that can be run via kubectl can be run within a Spacelift stack. Find out more about how Spacelift works with Kubernetes, and get started on your journey by creating a free trial account.

Manage Kubernetes Easier and Faster

Spacelift allows you to automate, audit, secure, and continuously deliver your infrastructure. It helps overcome common state management issues and adds several must-have features for infrastructure management.

Start free trial

Kubernetes Commands Cheat Sheet

Grab our ultimate cheat sheet PDF

for all the kubectl commands you need.

k8s book
Share your data and download the cheat sheet