Solving the DevOps Infrastructure Dilemma: Enabling developer velocity with control 💡

Writing .gitlab-ci.yml File with Examples [Tutorial]

22 May 2024·14 min read

Reviewed by: Flavius DinuFlavius Dinu

GitLab is one of the most popular tools for creating continuous integration and delivery (CI/CD) pipelines that automate your DevOps processes. It provides an integrated approach to CI/CD, where your pipelines sit right alongside your source repositories.

To enable GitLab CI/CD for a project, you’ll need to write a .gitlab-ci.yml file. This YAML file configures your pipelines by defining the scripts they’ll run, the conditions that will trigger them, and the job settings to apply. Because so many options are supported, getting started can feel daunting—but we’ve got you covered in this guide to the most commonly used keywords.

What we will cover:

What is the CI YAML file in GitLab?

.gitlab-ci.yml is the main YAML configuration file for GitLab CI/CD. GitLab projects that use CI/CD features have a .gitlab-ci.yml file located in their repository’s root directory. GitLab detects the file on pushes and merges, then parses it to discover the pipeline jobs to run. Created jobs are executed by an available GitLab Runner instance.

GitLab uses a conventional stage/job pipeline architecture. Stages normally execute sequentially, with the next stage only starting when all the jobs from the previous stage have succeeded. Jobs within a stage run in parallel to improve performance.

What is .gitlab-ci.yml used for?

The .gitlab-ci.yml file is used to define your project’s stages and jobs. It spans the entire DevOps lifecycle and configures all the CI/CD pipelines in your project, from code tests through to deployments. Setting it correctly is important to ensure reliability, performance, and efficiency.

Do I have to use .gitlab-ci.yml?

You’ll normally need a .gitlab-ci.yml file if you plan to use GitLab CI/CD with your project. However, it’s possible to choose a different filename instead by customizing the “CI/CD configuration file” option in your project’s settings.

GitLab also includes an Auto DevOps feature. When enabled, this automatically runs CI/CD jobs that build, test, and optionally deploy your project’s code, without requiring you to manually write a .gitlab-ci.yml file. Auto DevOps can be ideal if you’re working with a simple project where you don’t need a customized CI/CD configuration; however, it’s still a good idea to learn how to use .gitlab-ci.yml so you can customize options in the future.

.gitlab-ci.yml file keywords

All .gitlab-ci.yml files have the same fundamental structure. They are composed from keywords that configure your pipeline’s global settings—default values affecting all jobs in the pipeline—and individual jobs.

The following simple example shows a pipeline with two jobs, each in a different stage, and some global variables:

variables:
  APP_NAME: "demo"

stages:
  - test
  - build

test_job:
  stage: test
  script:
    - echo "Testing $APP_NAME"

build_job:
  stage: build
  script:
    - echo "Building $APP_NAME"

Top-level file sections that aren’t a reserved keyword are automatically interpreted as job definitions. Hence, the test_job and build_job sections define jobs, with the stage keyword identifying the stage each job belongs to.

The main keywords used to write .gitlab-ci.yml your pipelines include:

image
stages
script
before_script/after_script
tags
variables
cache
artifacts

1. image

The image keyword in .gitlab-ci.yml specifies the Docker image to use for the job. Your job will run within a container that’s started with this image. This only applies when you’re using the Docker executor for your job runner (this is the default for GitLab.com).

This keyword can be set globally or for individual jobs. The job setting overrides the global one.

image: ubuntu:latest

job1:
  script:
    - echo "Uses the ubuntu:latest image"

job2:
  image: busybox:latest
  script:
    - echo "Uses the busybox:latest image"

2. stages

In .gitlab-ci.yml, stages keyword can only be used at the top level. It defines the stages to include in your pipeline, with their execution order. Stages are run in the order that they’re written:

stages:
  - test
  - build

test_job1:
  stage: test
  # ...

test_job2:
  stage: test
  # ...

build_job:
  stage: build
  # ...

In the example above, the pipeline will begin with the test_job1 and test_job2 jobs executing in parallel within the test stage. Once they’ve both succeeded, then the build_job job will start.

Configuring stages is technically optional. Jobs that don’t use the stage keyword will be assigned to an auto-generated stage called test.

3. script

The script section in .gitlab-ci.yml file is a required keyword for your jobs. It’s where you list the commands the job will execute:

test:
  stage: test
  script:
    - echo "Running tests"
    - npm run test

The commands will be run in the order they’re written. If any command terminates with a non-zero exit code, the job is marked as failed, and any remaining commands will not run.

4. before_script and after_script

The before_script and after_script keywords allow you to run extra commands before or after a job’s main script section of the .gitlab-ci.yml file. They’re often used to distinguish commands that set up or tear down resources required by your script.

test:
  stage: test
  before_script:
    - echo "Running tests"
  script:
    - npm run test
  after_script:
    - echo "Tests complete"

before_script and after_script can also be set globally. This lets you configure default commands without having to manually copy them into each job.

5. tags

The tags keyword in .gitlab-ci.yml is the mechanism that controls which GitLab Runners can execute a job. If your job requires a specific processor architecture, operating system, or hardware tier to run, then it’s important to specify the right combination of tags to ensure those requirements are fulfilled.

build:
  tags:
    # Use a small Linux x86 runner (GitLab SaaS)
    - saas-linux-small-amd64

For a job to be picked by a runner, the runner must possess all the tags you’ve listed. When you’re using self-hosted runners, you can assign tags on the runner’s configuration page. The supported tags for GitLab.com SaaS runners are detailed in the documentation. Jobs that don’t declare any runner tags will be executed by a runner that’s configured to accept untagged jobs.

6. variables

In .gitlab-ci.yml, you can define variables globally or for specific jobs. You can then reference these variables within other parts of your CI/CD configuration, such as in your script commands — we saw this in action in the example above:

variables:
  APP_NAME: "demo"

test_job:
  stage: test
  script:
    - echo "Testing $APP_NAME"

Variables can also be defined within the GitLab interface at the project, group, and instance level. GitLab provides many predefined variables too, such as $CI_COMMIT_SHA to get the SHA of the commit the pipeline’s running for, or $CI_COMMIT_BRANCH to discover the branch name.

When a variable is defined in multiple places, interface values normally override values from your .gitlab-ci.yml file. It’s advisable to check the full variable precedence order because it’s relatively complex and can feel counterintuitive in some scenarios.

7. cache

The cache keyword allows you to cache paths within your job’s environment between different pipeline runs in .gitlab-ci.yml file. The following example caches the node_modules directory, which should allow the npm install command to complete more quickly after the first pipeline is run:

build:
  stage: build
  script:
    - npm install
  cache:
    key: node_modules
    paths:
      - node_modules

GitLab automatically stores cached paths, then restores them next time the job runs. Configuring appropriate caches is one of the main ways to improve CI/CD efficiency and optimize performance, so it’s worth looking for candidate paths with every job you write.

8. artifacts

CI/CD artifacts are files that you want to keep after a job is completed. Build outputs, test results, and compliance reports are some common types of artifacts, but there are no limitations on what you can include.

In .gitlab-ci.yml, artifacts are specified using the artifacts keyword. It accepts a list of paths that will be retained as artifacts and made available for download in GitLab’s job UI:

build:
  artifacts:
    paths:
      - out/bin/

Several different artifact options are supported, including the ability to exclude sub-paths, expire artifacts after a set time, make them public, and change the filenames they’re uploaded as.

Configuring when GitLab CI/CD jobs run

Many CI/CD pipelines can be implemented using the default GitLab architecture, where all job stages are always executed sequentially. However, sometimes you’ll likely to want more control when a job runs. There are a few keywords available in .gitlab-ci.yml to customize job behavior:

when — This is a job-level keyword that provides a simple way to set the condition for a job. It defaults to on_success, meaning the job runs once the previous stage has completed, but also accepts the values always, on_failure, manual, delayed, and never.
rules — Using rules allows you to apply complex customization to determine whether a job should run. You can use it to create if-else logical conditions, such as setting when: never for a job if the pipeline’s running against a particular branch.
workflow: rules — This functions similarly to rules, but acts at the global level. It lets you control whether the entire pipeline should run: for example, you might want to skip the pipeline if no changes have been made to your src directory since the last commit.

Furthermore, the needs keyword can be used to implement more complex execution flows for larger pipelines with many interdependent jobs. needs models your jobs as a directed acyclic graph (DAG), allowing jobs to start out-of-order once their specific dependencies have finished—even if some other jobs from previous stages are still running.

💡 You might also like:

How to create .gitlab-ci.yml files?

Now we’ve seen the capabilities available in a .gitlab-ci.yml file, how do you actually create one? There are two main ways to create a .gitlab-ci.yml file:

Manually create a .gitlab-ci.yml file, commit it to your repository, and push to GitLab — You can create a .gitlab-ci.yml file using a regular Git workflow, just like any other type of file—either locally on your workstation, or using the Web IDE available within the GitLab interface.
Use the CI/CD Pipeline Editor within the GitLab interface — GitLab includes an interactive pipeline editor that helps you author your .gitlab-ci.yml file, continually validates its syntax, and provides a visualization of your pipeline’s structure. Using the editor can help you spot errors more quickly.

Complex pipeline configurations can produce very long .gitlab-ci.yml files. You can make these more maintainable by splitting them into multiple files. Use include statements in your top-level .gitlab-ci.yml file to reference your other files, whether within your repository or at a remote location. You can also create CI/CD components that represent small units of reusable configuration, stored in a dedicated GitLab project.

Tutorial: Writing a .gitlab-ci.yml file

In this guide, we’ll create a .gitlab-ci.yml file that executes some tests and builds a Docker image for a simple app. We’re using GitLab.com — if you’re using a self-hosted GitLab server, then make sure you’ve already got a GitLab Runner instance configured with the Docker executor.

Step 1 — Prepare your project

Create a new GitLab project, then commit the following files to it. They create a minimal Express app with a fake test suite for demonstration purposes. In the following steps, we’ll write a .gitlab-ci.yml file that automatically runs the tests and then builds the app’s Docker image if the tests succeed.

.gitignore

node_modules/

package.json

{
  "name": "demo-project",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "jest"
  },
  "dependencies": {
    "express": "^4.18.3"
  },
  "devDependencies": {
    "jest": "^29.7.0"
  }
}

index.js

const express = require("express");

const app = express();

app.get("/", (req, res) => {
  res.send("Example App");
});

app.listen(3000, () => {
  console.log("Now listening");
});

demo.test.js

test("Example test", expect(true).toBe(true));

Dockerfile

FROM node:20
COPY package.json .
COPY index.js .

RUN npm install
ENTRYPOINT ["node", "index.js"]

Step 2 — Create your .gitlab-ci.yml file

Next, copy the following content to the .gitlab-ci.yml file in your repository:

stages:
  - test
  - build

test:
  stage: test
  image: node:20
  before_script:
    - npm install
  script:
    - npm run test
  cache:
    key: node_modules
    paths:
      - node_modules

build:
  stage: build
  image: docker:25.0
  services:
    - docker:25.0-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $DOCKER_IMAGE .
  after_script:
    - docker push $DOCKER_IMAGE
  variables:
    DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

This pipeline configuration demonstrates several of the keywords we explained above:

Two pipeline stages are defined, test and build.
The test job uses the before_script and script sections to install the npm dependencies, then run your project’s tests. The contents of the node_modules directory is cached between pipeline runs to improve performance.
The build job uses a combination of predefined variables and a custom variable (DOCKER_IMAGE) to create your app’s Docker image and push it to your GitLab project’s container registry.
The services field links a second container running the docker:25.0-dind image to your job’s container; this makes it possible for Docker commands to work correctly within your job’s script.

Now, you’re ready to run your pipeline!

Step 3 — Run your GitLab CI/CD pipeline

Commit all your changes and push them to your GitLab repository. Head to your project’s Build > Pipelines page — you should see a new pipeline is running, with the two stages you’ve defined:

After about a minute, you should see the pipeline status transition to Passed:

To get more information about the pipeline, click its status icon:

You can then retrieve each job’s logs by clicking the job in the pipeline graph. This allows you to check the job’s output and investigate any errors that caused the pipeline to fail.

Key points

We’ve explored the options available in .gitlab-ci.yml files and seen how to configure a pipeline for a simple project. Now you can start writing your own GitLab CI/CD pipelines to automate your software delivery process by running jobs after code is changed.

GitLab CI/CD is versatile but it’s a general-purpose platform that’s not always the best fit for every scenario. Fully automated solutions can make it easier to implement CI/CD in specific contexts, such as Spacelift for IaC management. Spacelift integrates directly with your PRs, letting you rollout IaC changes without having to handwrite any CI/CD config files.

Read more why DevOps Engineers recommend Spacelift. If you want to take your infrastructure automation to the next level, create a Spacelift account today or book a demo with one of our engineers.

The Most Flexible CI/CD Automation Tool

Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.

Start free trial

Written by

James Walker

James Walker is the founder of Heron Web, a UK-based software development studio providing bespoke solutions for SMEs. He has experience managing complete end-to-end web development workflows with DevOps, CI/CD, Docker, and Kubernetes. James is also a technical writer and has written extensively about the software development lifecycle, current industry trends, and DevOps concepts and technologies.

jhwalker.net