Jenkins is designed to streamline the software delivery process. A versatile platform that is suitable for various software engineering tasks, it is primarily used to manage CI/CD pipelines to ensure changes are validated and deployed efficiently.
In this article, we will explore the key features, benefits, and practical applications of Jenkins in modern software development and its role in DevOps practices.
What we will cover:
Jenkins is an open-source automation platform used to implement continuous integration and continuous delivery (CI/CD) processes. It automates build, test, and deployment tasks by executing jobs triggered by events such as new commits, branches, and pull requests across different types of environments. Its highly extensible plugin architecture enables seamless integration with numerous tools and technologies. As a Java-based application, Jenkins is compatible with multiple operating systems, including Windows, Linux, and macOS.
What is Jenkins used for?
Jenkins is used for building, testing, and deploying software projects and automating CI/CD pipelines. It orchestrates complex workflows across multiple development stages and integrates with various tools and technologies in the software development lifecycle. Jenkins can also monitor and report on build and deployment processes.
Is Jenkins a CI or CD tool?
Jenkins is primarily a tool for continuous integration (CI) but can also be used for continuous delivery (CD).
CI encourages development teams to integrate the source code being developed in the version control system so the automation pipeline responsible for building and running the test cases provides quick feedback to the developers. The key focus of CI is to provide this feedback as quickly as possible to allow developers to fix bugs and actively mitigate risks before deploying to production/higher environments.
CD is a practice in which automation pipelines are used to manage software releases. It involves using tools that facilitate the creation of deployment artifact dependencies and setting environment-specific variables in a repeatable and reliable manner. Continuous delivery is different from continuous deployment in that it requires manual approval before new software builds are deployed into production.
Jenkins integrates with Git repositories, where developers collaborate and commit code changes. Jenkins picks up these changes to automatically start the build process, which involves compilation, testing, and error reporting as a feedback loop to developers. This enables a quicker turnaround in the development lifecycle without affecting production.
As part of continuous delivery, you can also use various automation tasks to manage dependencies and push deployment artifacts to appropriate repositories. Examples of artifacts include jar files required by Java programs and container images for container-based deployments.
Because the tool is open-source and extensible, it has empowered the Jenkins community to develop a robust ecosystem of plugins. Numerous plugins are available for specific tasks related to version control, source code management, build, testing frameworks, deployment targets, reporting, and more. This modular approach also allows organizations to create flexible and highly customized pipelines.
It is important to grasp certain concepts to fully understand Jenkins. The list below will get you up to speed with building basic pipelines, which we will cover in the following sections.
Note: This is not an exhaustive list of features.
1. Jenkins pipeline
A Jenkins pipeline represents an end-to-end workflow built for CI/CD using multiple tools. It defines the steps required to build, test, and deliver/deploy applications automatically through various environments.
Pipelines are defined using YAML files, and in the context of Jenkins, this file is named “Jenkinsfile.” Creating CI/CD pipelines ensures standardization, enforces best practices, facilitates easy collaboration, and speeds up the delivery of new application features.
The diagram below shows the Jenkinks pipeline example:
2. Builds in Jenkins
Builds in Jenkins refer to the stage in which the application source code is compiled, tested, and packaged into a deployable artifact. A build can be triggered manually or automatically, where the Jenkins automation server polls for changes in the source code repository. The build involves various subtasks, such as compiling the code, running unit tests, static code analysis, document generation, etc.
Depending on the programming language in which the software is being developed, various plugins are available to support the build process in Jenkins. Also, depending on the type of deployment, various types of artifacts are versioned, created, and saved at predefined locations.
3. Jenkins triggers
As the name suggests, triggers are actions or events that initiate the build or deploy pipelines. Jenkins supports various types of triggers. Examples are listed below:
- SCM triggers: Jenkins polls for changes to the SCM repository. When the changes are found, Jenkins starts executing the pipeline jobs and processes the latest source code.
- Parameterized triggers: This trigger enables builds to be triggered based on user-defined inputs.
- Manual triggers: You may need to explicitly trigger the pipeline by logging into the Jenkins environment.
- Time-based triggers: Pipelines can be run based on a specific schedule. A cron job is set to start the pipeline execution after every set interval.
- Webhooks: When the pipeline runs depending on some event occurring in an external platform, we can use webhooks to trigger the pipeline execution in Jenkins.
4. Jenkins artifacts
Artifacts are files that emerge from a build process, needed for the deployment or for reporting purposes. Various runtime environments require various types of executables and binaries, which often result from the compile and build process.
Jenkins provides artifact management, which allows users to publish or archive locally on the Jenkins server or on an external platform. This improves traceability, reproducibility, and reliability in the software development process.
5. Jenkins agents
Agents are the infrastructure components leveraged by Jenkins to execute the pipeline jobs. A pipeline run needs compute resources to run the scripts and commands specified in the execution steps. Apart from the local execution on the Jenkins server, it is also possible to run the build jobs on other on-prem servers, virtual machines, or in a containerized environment.
In fact, it is highly recommended that the build jobs should not be executed on the server where Jenkins is installed. These resources/VMs are called Agents. Agents help scale Jenkins operations beyond a single team to serve the automation and build requirements at a project/organization level.
6. Jenkins pipeline stages
Stages provide a logical structure for organizing and visualizing the workflow of a pipeline in Jenkins. This allows developers to segregate a complex Jenkins pipeline into clear phases, which helps in debugging or troubleshooting issues. A stage represents a phase of the pipeline.
For example, we can have build, test, and deploy stages. Each of these stages further contains jobs that accomplish the tasks a stage is expected to do. In general, stages in Jenkins provide a logical structure to the pipelines.
Stages are shown in the “Stage view”:
7. Jenkins jobs (projects)
A job represents a predefined set of actions performed by the pipeline in a specific order. For example, the build process involves multiple jobs — build, test, push, etc. — which are the building blocks of a Jenkins pipeline. We can configure jobs to run various scripts and commands. In Jenkins, jobs accept various parameter inputs, and you can define complex workflows using them.
8. Steps in Jenkins pipelines
Steps are the smallest unit of work in Jenkins pipelines. Each command or line written in the script is a step. Jenkins supports a wide range of built-in and plugin-provided steps, as well as the ability to write custom steps in the Groovy language.
To put stages, jobs, and steps in perspective, a Jenkins pipeline has multiple phases defined by stages. Each stage contains multiple jobs, each defined by a number of steps written using a scripting language.
9. Jenkins plugins
A typical Jenkins installation contains the core platform. Plugins extend its functionality, providing additional features and integrations for various use cases.
Jenkins plugins provide a wide range of features across all the phases of the pipeline and platform management. Various third-party systems can be integrated with Jenkins via plugins. Custom ones can also be developed. Plugins leverage reusability, thus enhancing the productivity and efficiency of the development team.
10. Jenkinsfile
Jenkinsfile is a text file that describes the pipeline as code to the Jenkins server. Every project that uses Jenkins for its CI/CD has this file in the project’s root directory and is usually committed to the source code repository. It makes the Jenkins server understand the sequence of steps to be carried out in each stage. This file provides a way to define pipelines in a version-controlled and reproducible manner, allowing teams to manage their delivery process alongside their application code.
Below is an example of a Jenkinsfile for a continuous delivery pipeline that includes three stages:
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
echo 'Testing..'
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}What is the architecture of Jenkins?
The Jenkins core is developed using the Java programming language. A simple installation of a Jenkins server consists of various internal components that manage communication and process requests from external applications, such as CLIs, requests, agents, etc.
The diagram below shows all the important Jenkins components:
We will review some of the important aspects below to better understand how Jenkins works internally.
JENKINS_HOME
On the server where Jenkins is installed, all the files required for the management and configuration of plugins and pipelines, build artifacts, runtime data, etc., are stored in a specific directory named ‘JENKINS_HOME’. Depending on the OS, the exact path to this directory may vary. Configuration files stored in this directory ensure persistence across server restarts. This is also helpful in case of disaster recovery and creates a backup.
Business layer
This layer is responsible for managing core activities like job and user management, build execution, and plugin integration. The business layer handles interactions between these components and the incoming requests from external systems.
Every incoming request to trigger pipeline execution is validated for permissions, delegates actions to the agent machines, makes status updates available to appropriate systems, and handles notifications. The business layer shown in the Jenkins architecture diagram is the central system that does the groundwork and abstracts away the complexity.
Stapler web framework
In the diagram above, Jenkins implements HTTP communications via CLI, endpoints, and web interfaces. The Stapler web framework is responsible for handling incoming HTTP requests and generating dynamic web content. It performs the mapping of URLs and request bodies to appropriate Java classes, which is very important. This makes it easier to develop new plugins for Jenkins because the communication patterns are standardized by the Stapler web framework.
Remoting
The remoting component in the Jenkins architecture is an executable JAR file responsible for communication between the Jenkins core and agents. It enables decentralization of build execution by distributing build jobs to multiple agents. This also allows parallel execution, resulting in increased scalability.
Jenkins remains popular due to its flexibility, extensive plugin ecosystem, and strong community support. It offers easy integration with various tools, automates CI/CD pipelines effectively, and supports distributed builds. Being open-source and free, Jenkins provides a cost-effective solution for organizations of all sizes. Its maturity, reliability, and continuous improvements make it a trusted choice for automating software development processes.
Let’s look at Jenkins’s benefits in more detail:
| Automation platform for CI/CD | At its core, Jenkins is an automation platform that facilitates CI/CD practices and supports frequent code integration, testing, and deployment. Developers can automatically trigger the automation workflow to build, test, and deploy code changes. Jenkins offers various possibilities for performing a specific task, and this flexibility extends to the end-to-end process implementation.
|
| Scalability and distributed builds | Jenkins is designed to handle large-scale projects. The distributed build architecture allows developers to leverage multiple build agents, which enable parallel task execution, thus significantly reducing build times. This scalability is beneficial for teams working on large projects, projects with numerous dependencies, or geographically dispersed teams.
|
| Plugin ecosystem | A large plugin ecosystem contributes to Jenkins’s adoption success. Many plugins are available to accomplish various tasks while implementing automation pipelines on Jenkins. Some of the examples where plugins are frequently used are version control systems, deploying code to cloud platforms, notifications, etc.
|
| Collaboration and visibility | Jenkins enables collaboration among development teams by providing a centralized platform for monitoring and managing builds, tests, and deployments. Its web interface provides a view of real-time build status, test reports, and project progress. This is key to transparency and visibility and promotes better communication and faster issue resolution.
|
| Cost-effective and open source | Jenkins offers an open-source alternative to proprietary tools, making it accessible to organizations of all sizes. Active community support ensures continuous improvement of the platform, resulting in a cost-effective way of leveraging automation capabilities, which otherwise would be expensive from a vendor lock-in and maintenance perspective.
|
The main challenges of using Jenkins include performance and scalability issues, complex configuration and maintenance, and plugin management difficulties. Jenkins can also be resource-intensive, leading to slow builds, and it may have security vulnerabilities if not properly configured.
Here are the biggest limitations of Jenkins:
| Complex setup | While Jenkins is open-source software that helps with cost-effectiveness, its downside is the setup and maintenance operations. It is one of the oldest automation platforms around and requires considerable time and effort to set up and manage. The rollback and automatic backups are not as straightforward as they are amongst the peers, thus requiring extra care. Setting up Jenkins is quite intricate, to say the least.
|
| Plugin ecosystem | Just as being open-source can be either a benefit or a drawback, the plugin ecosystem can be a boon and a curse at the same time. On one side, leveraging third party plugins may save time, but on the other, you need to ensure plugins are up-to-date. Newer versions can also introduce breaking changes or incompatibility issues.
|
| Security vulnerabilities | Jenkins’ core platform has a history of critical vulnerabilities that introduce the potential for remote code execution, arbitrary file read access, XSS, etc. These security vulnerabilities require administrators to keep Jenkins servers up-to-date, which adds to thetakes us back to the point of complexity of setup and operations.
|
| Dated UI/UX | The Jenkins UI has received very few upgrades, and the user experience does not align with today’s standards. It offers basic navigation functionality and is not really intuitive or modern.
|
Download the Build vs. Buy Guide to Scaling Infrastructure as Code
Let’s build a simple Jenkins project with a pipeline on the Jenkins server hosted locally. In this hands-on exercise, we will touch upon various components discussed in this post.
If you don’t have a Jenkins server installed, then follow the instructions here before proceeding.
Using Jenkins involves the following steps:
- Log in to the Jenkins dashboard.
- Create a Jenkins pipeline.
- Configure pipeline options.
- Create a pipeline script.
- Trigger the pipeline run.
Step 1: Log in to the Jenkins dashboard
When you install Jenkins for the first time, you will be asked to set the login credentials for the admin user. Use these credentials to log in to the Jenkins server on https://localhost:8080.
After logging in, you will be presented with a dashboard in a Jenkins Web UI, as shown below. If you have a fresh installation, you will not see any pipelines.
Step 2: Create the Jenkins pipeline
Click on the “New Item” button from the menu. You will be presented with a screen to select the type of “Item” you would like to create. As in the image below, select “Pipeline” and give this pipeline a name. Click on “OK,” and you will be redirected to the Configure page.
Step 3: Configure pipeline options
On the general tab, you will see several configuration options related to parameterization, SCM integration, build concurrency, etc., as shown below. You should also see a separate section for setting various types of build triggers. We can leave these configuration options untouched for now.
Step 4: Create a pipeline script
Scroll down to the Pipeline section. Here, you have two options for creating pipeline scripts: Either fetch them from the SCM or write them yourself. We have not configured any SCM, so we will write our simple pipeline script in the script block that follows:
Add the script below:
pipeline {
agent any
stages {
stage('Stage 1') {
steps {
echo 'Pulling source code from SCM'
// more steps
}
}
stage('Stage 2') {
steps {
echo 'Compiling source code.'
// build commands
}
}
stage('Stage 3') {
steps {
echo 'Running tests'
// run unit test cases
}
}
}
}This is a very basic representational pipeline script written in Groovy. The outermost parentheses represent the pipeline itself. We then define the agent attribute as ‘any’ because we can currently okay to run this pipeline on any available agent.
In this case, the agent also runs on the same machine as that of the Jenkins server because we have not explicitly configured any agent. The agent attribute is followed by consecutive stage blocks, which in turn define the steps to be executed in each stage.
The pipeline above simulates three stages:
- Stage 1 – Pull the source code from SCM. Needs SCM configuration and additional steps.
- Stage 2 – Compile the source code that is pulled.
- Stage 3 – Run unit tests on the compiled code.
Click on “Save”.
Step 5: Trigger the pipeline manually
Navigate to the newly created pipeline, and you will be presented with various options, as seen below:
If you want to revisit the configurations from the previous step, click on “Configure”; otherwise, click on “Build Now” to trigger the pipeline execution. The following screen will present a graphical summary of all the pipeline runs, as shown below:
Here, we can understand that the pipeline has been run twice, and the second run has been successful. To check the logs of the second run, click on “#2” under Build History and then click on “Console Output”.
We have successfully run a very basic pipeline in Jenkins!
You are recommended to explore the Jenkins interface, configure SCM, explore plugins, and build more meaningful pipelines. To start, check out our Managing Terraform with Jenkins tutorial.
Here’s a quick recap of some best practices to follow when using Jenkins:
- Restrict access and secure the configuration files used by Jenkins, as they may contain sensitive information. It is recommended that you use a credentials plugin to manage service account credentials used for automation.
- Treat the Jenkinsfile as code and commit it to SCM. This enables versioning of the pipeline definition and helps with easy rollback, team collaboration, and traceability.
- Jenkins is all about automation. Configure triggers to automate pipeline builds.
- Configure pipelines to fail in the initial stages of the execution and implement informational logging messages. This enables developers to understand the errors quickly and thus focus on fixing the bugs for faster and more reliable software delivery.
- Regularly back up the Jenkins configuration so that in case of disaster, the recovery results in minimum information loss.
- Leverage plugins to build pipelines faster instead of building them from scratch unless it is absolutely necessary.
Spacelift is an infrastructure-as-code (IaC) management platform for OpenTofu, Terraform, Terragrunt, CloudFormation, Pulumi, Kubernetes, and Ansible resources. It provides an automated CI/CD workflow for your infrastructure changes — no need to write flaky pipeline scripts to apply your Terraform plans or Kubernetes deployments!
Using Spacelift allows you to eliminate the tedious and risky parts of infrastructure management. Compared with manually configuring these workflows in Jenkins, Spacelift is simpler, offers improved visibility and compliance, and is easier to scale across different IaC providers and cloud platforms.
Let’s walk through how it overcomes the generic CI/CD tools limitations:
- No need to control the state files: Spacelift can manage backend and state files for you. You have to declare what technology you want to control when you create a stack.
This means you no longer need to worry about it. If you still wish to manipulate the state, you can run a task as shown below.
However, manipulation of the state file is strongly discouraged and should not be done without serious reason. By manipulating the state, you endanger your whole infrastructure configuration.
- State lock is also solved by Spacelift. All runs are controlled, and two runs can’t hit the same resource simultaneously.
- Background configurations like state files are managed.
- Spacelift offers proper orchestration functionality:
- Spaces, where you can easily mirror your cloud environments with a clear split between teams/environments/accounts. Every Space can be subject to different policies and settings.
- Contexts give the overarching variable sets for multiple stacks.
- Resources help you to understand what you deployed and where there may be configuration drift.
- Runtime configurations might be helpful for checking the compatibility of a new version of an IaC tool.
- Policies are a key concept in Spacelift. This functionality helps to control the workflows with a specific set of checkpoints.
- Terraform registry is where you can store, manage, and control your Terraform modules and providers. With a generic CI/CD tool, you have to introduce another tool into your chain, whereas Spacelift has this service built in.
- Stack dependencies are useful when multiple templates are dependent on other templates. With Spacelift, you can create a clear chain of execution featuring an unlimited number of stacks. It can be helpful to differentiate and connect the creation of the infrastructure on the Preprod account and then on Prod. Or, if you have many layers, you can simply build the dependencies between Preprod Network, Preprod Database, Prod Network, and Prod Database stacks. In this way, your infrastructure stacks are connected with each other, as in real environments.
- You don’t need multi-step pipelines or multiple pipelines for different use cases. If you want to utilize the pull request functionality, Spacelift has pull request management built-in. With generic CI/CD tools, you need a separate pipeline for drift detection. With Spacelift, you can detect drift easily without additional pipelines.
- The self-service approach is built into Spacelift. I have already mentioned elements such as the Terraform registry, Spaces, and Policies, but most importantly, Spacelift offers Blueprints. This is the best way to build and share templates with development teams.
- It is very important to know what infrastructure is deployed. Spacelift allows you to do so with the Resources view. Here, you can view everything done within Spacelift. And more importantly, if you implement drift detection, it will be visible there.
Jenkins is a versatile and powerful tool for automating CI/CD processes. This tutorial gives you a comprehensive foundation for understanding and implementing Jenkins pipelines. By automating build, test, and deployment tasks, Jenkins streamlines the software development lifecycle, enhancing efficiency and reliability.
To learn more about Spacelift, create a free account today or book a demo with one of our engineers.
The Most Flexible CI/CD Automation Tool
Spacelift is an alternative to using homegrown solutions on top of a generic CI. It helps overcome common state management issues and adds several must-have capabilities for infrastructure management.
