Building and Using a Multi-node Jenkins Farm

Table of Contents

1. Introduction and Roadmap
2. Deciphering the Jenkins Architecture

• 2.1 CI/CD reminder and introduction
• 2.2 Plugin architecture
• 2.3 Deployment options
• 2.4 Business Case — Globomantics
• 2.5 Master-agent architecture

3. Step by step configuration of the Jenkins controller

• 3.1 Creation of the CloudFormation template
• 3.2 Template deployment
• 3.3 Unlock Jenkins and install essential plugins
• 3.4 Create Jenkins AMI
• 3.5 Cleanup to avoid unnecessary costs
• 3.6 Demo: Install plugins via the web interface
• 3.7 Demo: Install plugins via the CLI
• 3.8 Integration with SCM
• 3.9 Demo: SCM integration with GitHub (webhooks)
• 3.10 Demo: Jenkins publishes to an S3 bucket
• 3.11 Demo: High Availability
• 3.12 Demo: Configuring HAProxy
• 3.13 Backup and Recovery

4. Distribute the load with jobs on agents

• 4.1 Module introduction
• 4.2 Business case — Where does the load come from?
• 4.3 Demo: Prepare the agents
• 4.4 Demo: Test SSH agent authentication
• 4.5 Connect, test and run Jenkins agents

5. Monitor the Jenkins Farm

• 5.1 Email Notifications
• 5.2 Disk space monitoring
• 5.3 Integration with Prometheus
• 5.4 Visualization with Grafana

6. Securing and scaling the Jenkins Farm

• 6.1 Jenkins Security: Authentication and Authorization
• 6.2 Secure SSH connections
• 6.3 Secure agents before scaling
• 6.4 Role Management (Role-Based Access Control)
• 6.5 Future Perspectives

7. Reference code files

• 7.1 CloudFormation Template — jenkins_ami.yaml
• 7.2 CloudFormation Template with CloudWatch Logging
• 7.3 CloudWatch Agent Configuration — wizard_generated_config.json
• 7.4 Configuration metrics CloudWatch — metrics_snippet.json
• 7.5 Scripts and commands — scripts_and_config.sh
• 7.6 Allow SSH password authentication
• 7.7 Load simulation — heavy_load.sh
• 7.8 Prometheus Installation
• 7.9 Service systemd for Prometheus
• 7.10 Installing Grafana
• 7.11 Creating user gsmith for SSH
• 7.12 Jenkins Commands via SSH
• 7.13 Jenkins SSH Configuration Test
• 7.14 Services start/stop commands
• 7.15 sshd_config configuration example

1. Introduction and Roadmap

Welcome to the Building and Using a Multi-node Jenkins Farm training. This training is a true adventure into the heart of automation, scaling and efficiency. Each module represents a crucial step in mastering the art of building a Jenkins Farm that not only works, but thrives.

Course Roadmap

The progression of the course follows five main stages:

1. Uncover the secrets of Jenkins architecture — Understand the fundamentals before you build.
2. Jenkins controller node step-by-step configuration — Setting up the master infrastructure.
3. Distribute the load like a pro — Run jobs on multiple agents in parallel.
4. Monitor the Jenkins Farm — Keep the farm healthy with robust monitoring tools.
5. Secure and scale the farm — Lock down security and handle large workloads.

Prerequisites

Before starting this training, you must have:

• A working knowledge of Jenkins (if you’re just starting out, check out the course Getting Started with Jenkins by Wes Higbe).
• Experience with command line tools (CLI).
• Familiarity with the concepts of continuous integration and continuous delivery (CI/CD).
• Prior knowledge of cloud technologies (useful but not required).
• An AWS account (you can stay within the free tier limits to avoid fees — the author cannot be held responsible for any fees).

2. Deciphering the Jenkins Architecture

2.1 CI/CD reminder and introduction

This module lays the foundation for travel. It begins with a reminder of continuous integration and continuous delivery, then presents the architecture of Jenkins in order to understand how to transform it into a distributed farm.

The overall goal is to move from a simple single-node architecture, insufficient for modern needs, to a scalable distributed architecture capable of managing multiple software products and environments simultaneously.

2.2 Plugin architecture

Jenkins is a Java web application distributed as a WAR (Web Application Archive) file, executable on most servlet containers such as Tomcat or Jetty. Its architecture is modular, plugin-based, making it easy to install, update, disable, and uninstall features, either through the UI, CLI, or manual configuration.

Ways to extend Jenkins via plugins

• User Interface Controls: Elements that correspond to standard phases of the project lifecycle. The main phases are:

• Project

• Parameters

• Triggers

• Build Environment

• Build Steps

• Post-build Steps

• The machinery that makes magic work includes:

• Plugins

• Local scripts

• Local applications

• The build environment

• The temporary build workspace (temporary build workspace)

• Build history (build history)

• Added security templates for specific tools.

• Defining new view types and additional columns.

• Enrichment of the user interface with console annotators and status widgets.

• Orchestration of the build process with new build steps, post-build steps, additional listeners, actions, etc.

Core plugins

Base plugins are suggested when first starting Jenkins and installed if accepted. They allow Jenkins to deliver its main continuous integration functionality and support the project lifecycle by default.

Jenkins Project Lifecycle Phases

1. Polling — Detection of changes in the repository.
2. Pre-SCM phase — Executing a build step after detecting a code change, but before checking out from the version control system.
3. SCM checkout — Source code recovery.
4. Build steps — Compilation, testing, packaging, etc.
5. Post-build steps — Archiving artifacts, notifications, deployment, etc.

2.3 Deployment options

There are several deployment options for Jenkins:

Important precautions with Docker

Containers are designed to be ephemeral: once the container is finished, all state is lost. In the Jenkins context, this means the loss of:

• Plugins installed or updated since the container was last started.
• Changes to job configurations.
• Logs generated by normal Jenkins operations.

Solutions exist (Docker volumes on the host, streaming logs to CloudWatch on AWS), but they are beyond the scope of this training.

Choice for this training

For the purposes of this training, we will use AWS VMs while remaining within the free tier options. It is important to note that AWS policies change frequently, so be careful when running your Jenkins deployment scenarios.

2.4 Business cases — Globomantics

The fictional organization Globomantics is evolving and its needs are becoming more and more complex. The company has recently expanded its software product portfolio, requiring more sophisticated DevOps processes to manage multiple builds and environments.

As a Jenkins expert, you must:

1. Learn how to plan, configure and run distributed builds by leveraging Jenkins build farms.
2. Provide mechanisms for external tools to automate builds.
3. Integrate Jenkins with third-party systems.

In technical terms, the CI/CD perspective is moving from a single node architecture (insufficient) to a distributed architecture where a single Jenkins master node would be painfully insufficient to satisfy the new requirements. This is where Jenkins’ distributed builds come into play.

2.5 Master-agent architecture

Jenkins’ master-agent architecture is the answer to the requirements of a scaling organization. Here’s how it works:

Master node roles

• Main role: listen to build triggers and delegate work to agents.
• Secondary role: detect agent failures and redistribute work to other agents.
• Listen for repository events or other triggers.
• Retrieve source code from one or more repositories.
• Distribute the workload in the form of jobs to multiple agents.

Main advantages

The central advantage of this approach is the ability to run multiple builds in parallel.

Use case 1: Complex system with multiple components

Let’s imagine a system with many different components that can be built in parallel, whose source code potentially lives in different repositories. Each component can be mapped to a job, and jobs can be distributed to a corresponding number of agents.

Concrete example: A heavy component takes 10 minutes to build, two lighter components take 5 minutes each. You can dedicate one agent to the heavy component and another to the two light components.

Use case 2: Multiple software products and environments (Globomantics case)

• Multiple independent software products, each managed by different teams.
• Multiple environments dedicated to each product: dev, QA, stage, and prod.
• The environments dedicated to each product are independent of each other, except for the rules imposed internally by the management team.

Operating Systems Note

Each agent can potentially run on a different operating system. For example, if you are building independent components targeted for deployment on macOS, Windows, or Linux, you can dedicate agents with the appropriate operating system.

3. Step by step configuration of the Jenkins controller

3.1 Creating the CloudFormation template

The first step in configuring the Jenkins master node is to create an Amazon Machine Image (AMI). For this, we use Amazon’s infrastructure tool called CloudFormation to start an EC2 instance, install Java and Jenkins on the VM, configure logging locally and in the AWS cloud.

Once this infrastructure is in place, we will:

1. Unlock Jenkins.
2. Install the suggested basic plugins.
3. Create a reusable Jenkins AMI.

CloudFormation template structure (YAML)

The CloudFormation template is a simple configuration file in YAML format that creates two security groups and an EC2 instance that uses them.

Input parameters:

• ImageId: Specifies the base image to use. To find the image ID, simply launch an EC2 instance from the AWS console and look at the first Amazon Linux 2 entry. For example: ami-0fa40e25bf4dda1f6.
• InstanceType: Specifies the size of the EC2 instance. To stay in the free tier, the options are limited to t2.micro (default) and t2.nano.

Resources section:

• SSHSecurityGroup: Allow SSH access on port 22.
• WebSecurityGroup: Allow web access on port 8080.
• Instance: The EC2 instance itself, with a UserData script that installs Java, Jenkins, Git, Maven, creates the gsmith user, generates SSH keys, and installs the CloudWatch agent.

Here is the basic template (jenkins_ami.yaml):

AWSTemplateFormatVersion: 2010-09-09

Description: Create an AMI from an EC2 instance.
Parameters:
  ImageId:
    Description: Image ID for base EC2 instance.
    Type: AWS::EC2::Image::Id
    Default: ami-0fa40e25bf4dda1f6
  InstanceType:
    Description: Instance type to launch EC2 instances.
    Type: String
    Default: t2.micro
    AllowedValues: [ t2.micro, t2.nano ]
Resources:
  SSHSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
      - CidrIp: 0.0.0.0/0
        FromPort: 22
        IpProtocol: tcp
        ToPort: 22
  WebSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable HTTP access via user defined port
      SecurityGroupIngress:
      - CidrIp: 0.0.0.0/0
        FromPort: 8080
        IpProtocol: tcp
        ToPort: 8080
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      IamInstanceProfile: !Ref IAMInstanceProfile
      SecurityGroups:
      - !Ref SSHSecurityGroup
      - !Ref WebSecurityGroup
      UserData:
        "Fn::Base64": !Sub |
          #!/bin/bash -x
          yum update -y
          touch /home/ec2-user/install.log
          chmod a+w install.log

          echo "Installing java-21-amazon-corretto-devel: " >> /home/ec2-user/install.log
          yum install java-21-amazon-corretto-devel -y >> /home/ec2-user/install.log
          echo "END Java Installation" >> /home/ec2-user/install.log

          echo "Installing Jenkins" >> /home/ec2-user/install.log
          wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
          rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key
          yum upgrade
          yum install jenkins -y >> /home/ec2-user/install.log
          systemctl enable jenkins >> /home/ec2-user/install.log
          systemctl start jenkins >> /home/ec2-user/install.log

          yum install git -y >> /home/ec2-user/install.log
          yum install maven -y >> /home/ec2-user/install.log

          adduser gsmith
          echo 'gsmith:C@t&D0g123' | chpasswd
          mkdir /home/gsmith/.ssh && chmod 700 /home/gsmith/.ssh
          ssh-keygen -q -t rsa -N '' -f /home/gsmith/.ssh/agent_rsa 2>/dev/null <<< y >/dev/null
          touch /home/gsmith/.ssh/authorized_keys
          cat /home/gsmith/.ssh/agent_rsa.pub >> /home/gsmith/.ssh/authorized_keys
          chmod 600 /home/gsmith/.ssh/authorized_keys
          chmod 400 /home/gsmith/.ssh/agent_rsa*
          chown -R gsmith:gsmith /home/gsmith/.ssh/

          yum install amazon-cloudwatch-agent -y >> /home/ec2-user/install.log

          /opt/aws/bin/cfn-signal \
            -e $? \
            --stack ${AWS::StackName} \
            --region ${AWS::Region} \
            --resource AMICreate
          chown ec2-user:ec2-user /home/ec2-user/install.log
      KeyName: "jenkins-kp1"
      Tags:
        - Key: Name
          Value: Jenkins AMI

3.2 Deploying the template

Once the CloudFormation template has been created, here are the steps to deploy the EC2 instance:

1. In your AWS account, navigate to the CloudFormation console.
2. Click Create Stack at the top right and select With new resources.
3. Keep Choose an existing template selected.
4. Under Specify template, select Upload the template file, then click Choose file and select the template file.
5. Click Next. Enter a stack name, for example Jenkins-AMI-v1. (The name can only contain upper/lower case letters, numbers and hyphens.)
6. Under Parameters, check the default values: the AMI ID and the t2.micro size.
7. Click Next. Under Tags, specify name for the key and Jenkins-AMI-EC2-instance for the value.
8. Scroll down and click Next.
9. On the Review page, review the information entered, and then click Submit.
10. In another tab, navigate to the EC2 console and verify that the instance starts. With the initializing status, this may take a few minutes.

3.3 Unlock Jenkins and install essential plugins

Once the EC2 instance has started and the checks have passed:

1. Select the instance and copy its public URL.
2. In a separate tab, paste the URL into the browser address bar and press Enter.
3. Jenkins is running and asking for the initial administrator password.
4. In the EC2 tab, with the instance still selected, click Connect.
5. Select EC2 Instance Connect (SSH connection via browser).
6. Click Connect and wait for the command prompt to appear.
7. Run the following command to recover the initial password:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

8. Copy the password and return to the Jenkins tab. Paste the value into the Administrator password field and click Continue.
9. Click Install suggested plugins and wait for the installation to complete.
10. Create an admin user: enter a username, password, password confirmation, full name and email address, then click Save and continue. Keep this password in a secure place.
11. On the next screen, keep the default Jenkins URL and click Save and finish.
12. Click Start using Jenkins.
13. Click Log out at the top right. Jenkins is now ready to create the AMI.

3.4 Create the Jenkins AMI

Once Jenkins is configured, we can create the reusable AMI:

1. Return to the EC2 tab.
2. With the EC2 instance still selected, right-click, select Image from the context menu, and then click Create Image.
3. Specify an image name, for example Jenkins-AMI-v1.0.0, and provide a short description.
4. Keep the rest of the values ​​and click on the Create image button.
5. In the left sidebar under Images, click Friends.
6. You can see that the AMI creation is in progress — it may take several minutes.
7. Wait until the AMI has the status available.

3.5 Cleaning to avoid unnecessary costs

Once the AMI is available, we can decommission the EC2 instance and all other infrastructure resources:

1. Navigate to the CloudFormation console.
2. With the Jenkins-AMI-v1 stack selected, click the Delete button at the top right.
3. Confirm by clicking Delete stack again.
4. Check in the EC2 console that the Jenkins instance is terminating (status shutting down, then terminated).

We now have a Jenkins AMI that will allow us to start as many Jenkins instances as we want.

3.6 Demo: Install plugins via web interface

The UI method is the easiest to discover and install plugins:

1. Log back into Jenkins and navigate to Manage Jenkins → Plugin management.
2. Click the Available tab.
3. At the top right, type build monitor and press Enter.
4. Check Build Monitor View in the results and click Install without restart.
5. Wait for the installation to complete.
6. Go back to Manage Jenkins → Installed, filter for build monitor view, and verify that the plugin is installed and activated.

Use the Build Monitor View plugin:

1. On the Jenkins main page, click the + sign next to the All view tab to create a new view.
2. Select the new Build Monitor View radio button.
3. Enter a name in the View name field, for example Test Build Monitor, and click OK.
4. On the view configuration screen, scroll through the list and check the jobs to display.
5. Optionally, enter a title like Fancy Build Monitor View and a sorting criterion like Ordered by status.
6. Keep the Failure Analyzer field set to name.
7. Click Apply and OK.
8. Use the Settings menu (gear icon at top right) to adjust the text scale, number of columns, etc.

3.7 Demo: Install plugins via CLI

Managing plugins via the Jenkins CLI allows you to automate plugin management and establish Jenkins maintenance standards in your organization. This is the best approach to scaling Jenkins — managing 2, 5, or even 10 Jenkins servers is one thing, but managing 100 requires automation.

Steps to configure Jenkins CLI

1. Install the SSH server plugin:

• Navigate to Manage Jenkins → Plugins, then search for SSH server.
• Select it and click Install.
• Restart Jenkins.

2. Configure SSH keys:

• Log in again and navigate to User → Security.
• Verify that the SSH public keys section is visible.
• Generate SSH keys with ssh-keygen and copy the public key.

3. Add public key to Jenkins profile:

• Navigate to your Jenkins user profile → Configure.
• In the SSH Public Keys field, paste your public key.

4. Download the Jenkins CLI:

• Navigate to <Jenkins URL>/cli in your browser.
• Download the jenkins-cli.jar file.

5. Test the CLI:

# Afficher l'aide et vérifier que la CLI fonctionne
java -jar jenkins-cli.jar -s http://<jenkins-url>:8080/ help

6. Install a plugin via the CLI (example: role-strategy plugin):

java -jar jenkins-cli.jar -ssh -user gsmith -i ~/.ssh/aws/iam/id_rsa \
    -s http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/ \
    install-plugin role-strategy

The list of available plugins is accessible on the official Jenkins registry. For each plugin, the ID can be found by clicking on the How ​​to install button.

3.8 Integration with SCM

At this point we have covered installing Jenkins on an EC2 instance, installing Java, Maven and Git on the same EC2 instance, as well as on your local workstation. Here are the prerequisites:

On the EC2 VM:

# Sur Amazon Linux / RHEL
yum install git -y

# Sur macOS
brew install git

# Sur Windows
# Télécharger l'installateur depuis https://git-scm.com/

To enable continuous integration, Jenkins and GitHub must work together. We will cover:

1. A brief introduction to webhooks.
2. Configuring webhooks in GitHub to detect push events.
3. Configuring a Jenkins job to trigger a build upon changes in the GitHub repository.
4. Build trigger testing.

Introduction to webhooks

webhooks (also called web callbacks, HTTP push API, or Reverse API) have a simple principle: when a specific event occurs in an application, it notifies other applications that need to be informed so that they can perform processing.

In our case:

• The application where the event occurs is GitHub (push event).
• The application to be notified is Jenkins, which will perform a checkout of the latest code and a new build.

Interaction sequence (webhook flow)

Développeur → [push code] → Repository GitHub
Repository GitHub → [événement push] → Webhook déclenché
Webhook → [notification HTTP POST] → Jenkins
Jenkins → [checkout code] → Repository GitHub
Jenkins → [build] → Artefact produit

3.9 Demo: SCM integration with GitHub (webhooks)

Create the webhook in GitHub

1. Log in to GitHub, select your target repository.
2. Navigate to Settings and select Webhooks in the sidebar.
3. Click Add webhook at the top right.
4. In the Payload URL field, enter:

http://<URL publique EC2>:8080/github-webhook/

The /github-webhook/ portion tells Jenkins that the webhook should be processed by the GIT SCM plugin.

5. Keep the Content type at application/x-www-form-urlencoded.
6. Under “Which events would you like to trigger this webhook?”, select Just the push event.
7. Make sure the Active box is checked.
8. Click Add webhook.

Create and configure the Jenkins job

1. Log in to Jenkins. Click on New item.
2. Enter Sample Micro Service Build in the Enter an item name field.
3. Select Freestyle project, click OK.
4. Enter an optional description, for example: This project builds a sample microservice and is triggered via a GitHub webhook.
5. Check the GitHub project box. Paste the URL of your repository into the Project URL field.
6. In the Source Code Management section, select Git, enter the URL of your repository.
7. In the Build Triggers section, check GitHub hook trigger for GITScm polling.
8. In Build Steps, add a build step to compile the source code (for example with Maven: mvn clean package).
9. In Post-build Actions, configure archiving of the generated artifact.
10. Click Save.

3.10 Demo: Jenkins publishes to an S3 bucket

After gaining access to a repository and successfully building an artifact, it is time for Jenkins to publish to an Amazon S3 bucket, an excellent object storage space.

Why store artifacts in S3?

Looking back at the CI/CD deployment process diagram, after running automated tests in a dev environment, the artifacts that have passed the tests must be stored somewhere before being consumed by phase 3 (deployment in QA) and phase 4 (review — acceptance testing pre-release).

Jenkins can push WAR files or other artifacts to S3, then these can be distributed or retrieved to QA or review environments.

Benefits of S3

S3 serves as an artifactory similar to popular products like JFrog Artifactory.

Configuration steps

1. Create an IAM role to allow Jenkins to write to S3.
2. Update EC2 instance to use this IAM role.
3. Create an S3 bucket to host Jenkins artifacts.
4. Install the S3 Publisher plugin in Jenkins.
5. Configure an S3 profile in Jenkins.
6. Configure Jenkins job to publish to S3 in post-build stages.

3.11 Demo: High Availability

As each aspect grows over time, you will inevitably need to answer the question: what happens when my Jenkins master node goes down?

High availability architecture

                    ┌─────────────────┐
                    │   Load Balancer  │
                    │    (HAProxy)     │
                    └────────┬────────┘
                             │
              ┌──────────────┴──────────────┐
              │                             │
    ┌─────────▼────────┐         ┌──────────▼────────┐
    │  Jenkins Master 1 │         │  Jenkins Master 2  │
    │   (ACTIVE)        │         │   (PASSIVE)        │
    └─────────┬────────┘         └──────────┬─────────┘
              │                             │
              └──────────────┬──────────────┘
                             │
                    ┌────────▼────────┐
                    │  Shared Storage  │
                    │   (AWS EFS)      │
                    └─────────────────┘

The architecture includes:

• Two Jenkins master nodes sharing common storage on AWS (EFS — Elastic File System).
• These nodes are in front of a load balancer (HAProxy) which, in the event of failure, ensures a transparent failover to the healthy master node.

Infrastructure requirements

Three AWS instances in total:

• 2 for Jenkins master nodes (identical, provided via the CloudFormation template).
• 1 for the HAProxy load balancer.

Configuring credentials on each node

On each Jenkins master node, configure a private key credential:

1. Navigate to Jenkins → Credentials.
2. Under Stores scoped to Jenkins, click Jenkins Store → Global credentials.
3. Click Add credentials in the left sidebar.
4. Select SSH Username with private key for the type.
5. Keep the scope Global.
6. Set the username as jenkins.
7. Select Enter directly for the private key and provide it.
8. Enter jenkins-master-node-key for the ID — the ID can be used later to associate these credentials with a job.
9. Click Create.

Configuring Shared Storage (AWS EFS)

# Installer les utilitaires NFS
yum install nfs-utils -y

# Monter le partage EFS (ajouter dans /etc/fstab)
us-east-1a.fs-f8d1697b.efs.us-east-1.amazonaws.com:/    /var/lib/jenkins/jobs   nfs defaults    0    0

# Commande de montage manuel
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport \
  fs-f8d1697b.efs.us-east-1.amazonaws.com:/ efs

# Arrêter Jenkins avant le montage
sudo systemctl stop jenkins.service

# Vérifier le volume monté
df -h

# Changer le propriétaire du dossier jobs
sudo chown -R jenkins:jenkins /var/lib/jenkins/jobs
ls -l /var/lib/jenkins/jobs

# Redémarrer Jenkins
sudo systemctl start jenkins.service
sudo systemctl restart jenkins.service

# Vérifier le statut
systemctl status jenkins.service

3.12 Demo: Configuring HAProxy

HAProxy is the load balancer which ensures high availability between the two Jenkins master nodes.

Installing and configuring HAProxy

# Installer HAProxy
sudo yum install haproxy -y

# Éditer la configuration
sudo vi /etc/haproxy/haproxy.cfg

# Vider le fichier et ajouter la nouvelle configuration
:1,$d

HAProxy configuration (contents of /etc/haproxy/haproxy.cfg):

frontend jenkins_frontend
    bind *:80
    default_backend jenkins_backend

backend jenkins_backend
    balance roundrobin
    option httpchk GET /login
    server jenkinsmaster1 <IP_MASTER_1>:8080 check
    server jenkinsmaster2 <IP_MASTER_2>:8080 check backup

• The stanza frontend declares binding to port 80 and redirects requests to the Jenkins backend.
• The stanza backend contains the definitions of our active and passive Jenkins master nodes. With this configuration, HAProxy will send requests to jenkinsmaster1 if it is healthy. Otherwise, requests will go to jenkinsmaster2.

# Démarrer/redémarrer HAProxy
sudo systemctl start haproxy.service
sudo systemctl restart haproxy.service

# Vérifier que le service a démarré correctement
systemctl status haproxy.service

Important: Make sure port 80 is open in the security group that applies to HAProxy.

Testing High Availability

1. In your browser, navigate to the public URL of the load balancer (from the EC2 console).
2. Verify that Jenkins is accessible, with the request being served by the active node.
3. Simulate a crash by SSHing into the active node and running:

systemctl stop jenkins.service

4. Reload the page in the browser — it still works. The passive node took over transparently.

Important Note: If any jobs are running when the active node crashes, they may not be recoverable on the passive node, and you will likely need to restart the builds.

Synchronization script for passive node

# Script de synchronisation
sudo vi /opt/sync_passive_jenkins_with_master.sh

# Commandes SSH utiles pour recharger la configuration
ssh -l gsmith -i ~/.ssh/aws/ec2/slave_rsa -p 53802 ec2-3-82-125-219.compute-1.amazonaws.com help
ssh -l gsmith -i ~/.ssh/aws/ec2/slave_rsa -p 53802 ec2-3-82-125-219.compute-1.amazonaws.com reload-configuration

# Planifier le crontab
sudo vi /etc/cron.d/reload_jenkins_config

# Contenu du crontab (exécution toutes les 2 minutes)
*/2 * * * * root /bin/bash /opt/sync_passive_jenkins_with_master.sh

3.13 Backup and Recovery

There are two main approaches to backing up your Jenkins master node:

Option 1: Direct file system backup

# Sauvegarde manuelle du dossier jobs
tar cvzf master-node-jobs-2025-02-14.tar.gz /var/lib/jenkins/jobs

# Exemple de crontab pour une sauvegarde planifiée
*3 * * * * root tar cvzf master-$(date +%Y-%m-%d).tar.gz /var/lib/jenkins/jobs

You can put all your backup commands in a cron tab and schedule it at the desired frequency. Combined with AWS EFS or S3, this is a very reliable method.

Option 2: Jenkins Backup Plugin

1. Navigate to Manage Plugins and select the Available tab.
2. Filter for backup. Check the Backup plugin and install it, followed by a Jenkins restart.
3. Once in place, navigate to Manage Jenkins, scroll down and select Backup Manager.
4. Click Set up.

Recommended configuration:

• Root directory and Backup directory: adjust as needed.
• Format: leave in zip.
• Check Verbose mode and Node shutdown.
• Check Backup workspace.
• You can include and exclude portions of the workspace from your backup.
• Check Backup build history, Backup maven artifacts archives, and Backup fingerprints.

5. Click Save then click Backup Hudson configuration.

4. Distribute the load with jobs on agents

4.1 Module Introduction

In Module 1, we presented our business case and showed how it can be solved by leveraging Jenkins’ distributed master-agent architecture. In module 2, we solved a large part of the puzzle by configuring the master node. Now we are ready to take action and implement the remaining components of this distributed architecture: the agent nodes.

What we will accomplish:

1. Prepare an agent, connect it to the master node, and test the configuration.
2. Configure the agent to communicate with the master.
3. Distribute job execution to agents using labels.

4.2 Business Case — Where does the charge come from?

At Globomantics, the company has recently expanded its software product portfolio, requiring more sophisticated DevOps processes to manage multiple builds and environments. A single Jenkins master node would be painfully insufficient. This is where distributed builds come into play.

As a Jenkins expert, you must:

• Schedule, configure and run distributed builds.
• Provide mechanisms for external tools to automate builds.
• Integrate Jenkins with third-party systems.

4.3 Demo: Prepare the agents

Before configuring agents, here are the preparation tasks:

1. Start two EC2 instances — With the AMIs already available, simply right-click on the last image and select Launch.

2. Enable password and public key authentication via SSH:

# Ouvrir le fichier sshd_config
vi /etc/ssh/sshd_config

# Décommenter ou ajouter :
# PasswordAuthentication yes
# PubkeyAuthentication yes
# AuthorizedKeysFile .ssh/authorized_keys

# Sauvegarder et quitter
:wq

# Redémarrer le service SSHD
systemctl restart sshd.service
systemctl status sshd.service

3. Update Jenkins URL — After each EC2 restart, if you do not have an Elastic IP, you must update the Jenkins URL:

vi /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml
# Remplacer l'ancienne URL dans le nœud <jenkinsUrl> par la nouvelle valeur
# Redémarrer Jenkins
systemctl restart jenkins.service

4. Create username/password credentials in Jenkins:

• Navigate to Credentials → Jenkins Global → Global credentials.
• Click Add credentials in the left sidebar.
• Keep the type Username and password and the scope Global.
• Enter gsmith for the username and password used in the CloudFormation template (C@t&D0g123).
• Enter gsmith-agent for ID.

4.4 Demo: Test agent SSH authentication

# Connexion SSH avec username et mot de passe
ssh -t gsmith@54.184.34.103
# Mot de passe : C@t&D0g123

# Obtenir la clé privée depuis la VM agent (générée par le template CloudFormation)
cat /home/gsmith/.ssh/agent_rsa

# Créer le fichier avec la clé privée sur votre poste de travail
# et le nommer agent_rsa dans le dossier .ssh

# Changer les permissions de la clé
chmod 600 ~/.ssh/aws/iam/agent_rsa
ls -l ~/.ssh/aws/iam/agent_rsa

# Connexion avec la clé privée SSH
ssh -i ~/.ssh/aws/iam/agent_rsa gsmith@54.184.34.103

On the agent VM, the private key generated by the CloudFormation template is located at:

/home/gsmith/.ssh/agent_rsa

To manually generate an SSH key pair:

# Générer des clés SSH (sans passphrase)
ssh-keygen -t rsa -N ""

4.5 Connect, test and run Jenkins agents

There are several ways for agents to communicate with the master. In this case, we use SSH to launch the Jenkins agents.

Configure an agent in Jenkins

1. Navigate to Manage Jenkins → Manage Nodes.
2. Note that only the master node is present.
3. Click New node in the left sidebar.
4. Give the new node a name, for example agent1-amz.
5. Check Permanent Agent.
6. Complete the fields:

• Description: Description of the agent.
• Number of executors: number of executors available.
• Remote root directory: temporary directory on the agent node. (Note: Agents do not maintain important data — all job configurations, logs and artifacts are stored on the master node.)
• Labels: enter job1.
• Usage: select Only build jobs with label expression matching this node.

7. Launch method: select Launch agents via SSH.
8. Copy the public IP address of the EC2 instance of the agent (from the AWS console).
9. In the Host field, paste the IP.
10. From the drop-down menu, select the credentials for the user gsmith.
11. For Host Key Verification Strategy, select Manually trusted key verification strategy. Leave manual initial connection verification unchecked.
12. Keep the advanced settings as default.
13. Availability: keep on Use this agent as much as possible.
14. Click Save.

Verify that the messages displayed confirm that the authentication was successful, and that on the Nodes page, the red mark has disappeared.

Distribute jobs to agents with labels

To distribute the load according to the labels:

1. On the master node, select the job you want to run on the new agent.
2. Navigate to its configuration.
3. In the General tab, under the project name and description, find the execution restriction section.
4. Check Restrict where this project can be run.
5. In the Label Expression field, enter job1 (the label we defined for the agent).
6. Save the configuration.
7. Run the build and observe it running on the designated agent.

5. Monitor the Jenkins Farm

5.1 Email notifications

In this clip you will learn how to configure email notifications in Jenkins.

Configuring basic email notifications

1. Navigate to Manage Jenkins → Configure System.
2. Scroll down to the Email Notification section.

Minimum SMTP settings required (example with Gmail):

Create a Gmail app password

1. Go to Gmail → Manage your Google account.
2. Click Security in the left sidebar.
3. Type App Passwords in the search bar.
4. Select from the drop-down menu, provide your Gmail password.
5. Enter an application name in the App password screen.
6. Click Create and transfer the generated password to Jenkins.

Test email configuration

1. Check Test configuration by sending test email.
2. Enter the recipient’s email address.
3. Click Test configuration.
4. Verify that you received the test email from Jenkins.

Configure the job for notifications

1. Go to job configuration → Configure.
2. Scroll to Post-build Actions.
3. From the drop-down menu, select Email notification.
4. Enter the notification email address.
5. Click Save.
6. Run a build, verify success, and view the console output — the log should show the notification being sent.

5.2 Disk space monitoring

A lesson learned the hard way: if your jobs run out of disk space, they will crash or simply not run. It is imperative to implement consumption monitoring measures in Jenkins.

Install Disk Usage plugin

1. Navigate to Plugins, select the Available tab, and filter for disk usage.
2. Check the plugin and click Download and install after restart.
3. Once downloaded, check Restart Jenkins when installation is complete.

Use the Disk Usage plugin

1. On the main page, click Disk usage in the left menu.
2. The plugin displays the disk consumption of all Jenkins jobs:

• The numbers for your projects and their corresponding builds.
• Total disk consumption.
• Under All workspaces: the size consumed by repository checkouts.
• Agent workspaces are also represented.

3. By default, sizes are calculated every 6 hours, but you can customize this or trigger a recalculation manually.
4. Convenient filters are available at the top left to filter by build age.

Activate consumption view by project

1. Navigate to Home page → Manage Jenkins → Configure System.
2. Scroll down to the Disk usage section.
3. The first three parameters define the calculation frequency.
4. Find the Show disk usage trend graph on the project page parameter and check it.
5. Save the configuration.

With this option enabled, each project page will display a disk consumption trend graph.

5.3 Integration with Prometheus

Here’s a crash course on Prometheus — its architecture, its integration with Jenkins, and data visualization with Grafana.

Prometheus Architecture

Prometheus has four main components:

Metrics Collection Modes

Scraping (main mode):

• Prometheus pulls data from targets like a web crawler scrapes content from a site.
• Gives Prometheus full control over when and how data is collected.
• Ideal for monitoring always-online services.

Push model:

• Useful for jobs that start and stop quickly (e.g. batch processes).
• Less efficient, but crucial when direct scraping is not possible.
• Recommendation: favor scraping unless your system requires pushing.

Installing Prometheus

# Exécuter en tant que root ou préfixer chaque commande avec "sudo"
useradd -M -U prometheus

wget https://github.com/prometheus/prometheus/releases/download/v3.2.1/prometheus-3.2.1.linux-386.tar.gz
tar -xzvf prometheus-3.2.1.linux-386.tar.gz
mv prometheus-3.2.1.linux-386 /opt/prometheus

chown prometheus:prometheus -R /opt/prometheus

# Créer le fichier de service systemd
vi /etc/systemd/system/prometheus.service

# Charger et configurer le service pour démarrer au boot
systemctl daemon-reload
systemctl start prometheus.service
systemctl enable prometheus.service
systemctl status prometheus.service

systemd service file (/etc/systemd/system/prometheus.service):

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/prometheus \
--config.file=/opt/prometheus/prometheus.yml \
--storage.tsdb.path=/opt/prometheus/data \
--storage.tsdb.retention.time=30d

[Install]
WantedBy=multi-user.target

Integrate Prometheus with Jenkins

1. Install the Prometheus metrics plugin in Jenkins (via Manage Jenkins → Plugins).
2. The plugin exposes a metrics endpoint at http://<jenkins-url>:8080/prometheus/.
3. Configure the prometheus.yml file for Jenkins scraping:

scrape_configs:
  - job_name: 'jenkins'
    metrics_path: /prometheus
    static_configs:
      - targets: ['localhost:8080']

5.4 Visualization with Grafana

Prometheus has visualization capabilities, but for professional rendering we use Grafana.

Installing Grafana

# Créer le fichier de repo avec nano
nano /etc/yum.repos.d/grafana.repo

# Coller le contenu du repo Grafana
# Puis installer via yum
yum install grafana -y

# Sur macOS
# brew install grafana

Important: Before enabling the service, make sure that port 3000 is open in your security group. If this port is not open, the Grafana service may not start correctly.

# Recharger la configuration systemd
systemctl daemon-reload

# Démarrer Grafana
systemctl start grafana-server.service

# Configurer Grafana pour démarrer au boot
systemctl enable grafana-server.service

# Vérifier le statut
systemctl status grafana-server.service

Grafana listens on port 3000 by default.

Configure Prometheus as a data source in Grafana

1. Log in to Grafana with the credentials admin/admin.
2. Pass password change if desired (not recommended in production).
3. Delete the default dashboard.
4. Click the drop-down menu at the top left → Connections → Data sources.
5. Click Add data source.
6. Select Prometheus from the list.
7. Configure:

• Name: Jenkins Metrics
• URL: http://localhost:9090
• Access: Default
• Authentication: None for demo (in production, always configure an authentication method).

8. Scroll down and click Save and test.

Create a Jenkins dashboard in Grafana

1. Import a pre-existing Jenkins dashboard from Grafana.com (recommended Jenkins dashboard ID: 9964 or search for “Jenkins” in the catalog).
2. Configure the panels to display:

• Agent CPU and memory metrics.
• The number of current/pending builds.
• Build times.
• Job failure rates.

Load simulation script (for testing)

To test monitoring under load, here is a CPU load simulation script:

#!/bin/bash

# Nombre de processus parallèles (doit correspondre ou dépasser le nombre d'exécuteurs)
NUM_PROCESSES=$(nproc)  # Obtenir le nombre de cœurs CPU

# Durée de la charge en secondes
DURATION=60
echo "Démarrage de la simulation de charge avec $NUM_PROCESSES processus pendant $DURATION secondes..."

# Fonction pour générer une charge CPU
generate_load() {
  local end_time=$((SECONDS + DURATION))
  while [ $SECONDS -lt $end_time ]; do
    echo "scale=5000; a(1)*4" | bc -l >/dev/null 2>&1
  done
}

# Démarrer les processus de charge en arrière-plan
for i in $(seq 1 $NUM_PROCESSES); do
  generate_load &
done

# Attendre que tous les processus se terminent
wait
echo "Simulation de charge terminée."

6. Securing and scaling the Jenkins Farm

6.1 Jenkins Security: Authentication and Authorization

This topic will come up frequently in your discussions with your manager — it’s important to pay attention to it. The three main areas of security are:

1. Authentication
2. Authorization
3. Agents (securing agent nodes)

Global Security Note

Starting with Jenkins 2.214 and Jenkins LTS 2.222.1, the Enable Security check box is no longer present — security is enabled by default. If you are using an earlier version, check this box.

Authentication — Security Realm

By default, Jenkins uses its own user database as Security Realm by default.

realms are containers for grouping logically related credentials.

Authorization

The permission defines what actions logged in or anonymous users can perform.

6.2 Secure SSH connections

Here are the steps to execute commands securely over SSH.

Create the gsmith user on the system

# Créer l'utilisateur gsmith
adduser gsmith
sudo su - gsmith

# Configurer le dossier .ssh
mkdir .ssh && chmod 700 .ssh && cd .ssh

# Générer les clés SSH et définir les permissions
ssh-keygen -q -N ""  # Accepter les noms de clés par défaut
chmod 600 id_rsa && chmod 600 id_rsa.pub

# Créer authorized_keys et définir les permissions
touch authorized_keys && chmod 600 authorized_keys
chmod 400 id_rsa*

# Création automatisée (en tant que root)
adduser gsmith
sudo su - gsmith
mkdir .ssh && chmod 700 .ssh && cd .ssh
ssh-keygen -q -t rsa -N '' -f ~/.ssh/slave_rsa 2>/dev/null <<< y >/dev/null
touch authorized_keys && chmod 600 authorized_keys
chmod 400 slave_rsa*

Copy public key to Jenkins profile

1. Avoid using the AWS-provided SSH client in the browser to copy the public key — it can introduce carriage returns that disrupt the key.
2. Use your command-line SSH client to connect to the server.
3. Run cat on the public key file and copy the result.
4. In Jenkins, click the drop-down menu next to the username.
5. Scroll to Public Keys and paste the key into the SSH Public Key field.

Enable SSH in Jenkins Security

1. Navigate to Manage Jenkins → Security.
2. Scroll to the bottom and enable SSH on a static port.
3. Note the port value.
4. Don’t forget to open this port in the security group of the EC2 instance.

Test SSH configuration

# Tester que l'endpoint SSH Jenkins est accessible
curl -Lv http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/login 2>&1 | grep -i 'x-ssh-endpoint'

# Réponse attendue confirmant que la config SSH est valide :
# < X-SSH-Endpoint: http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:53801

# Définir les permissions correctes pour la clé privée
chmod 600 id_rsa

Run Jenkins commands over SSH

# Afficher l'aide
ssh -l gsmith -i id_rsa -p 53801 \
  ec2-52-89-0-29.us-west-2.compute.amazonaws.com help

# Lister tous les jobs que gsmith peut exécuter
ssh -l gsmith -i id_rsa -p 53801 \
  ec2-52-89-0-29.us-west-2.compute.amazonaws.com list-jobs

# Lancer un build
ssh -l gsmith -i id_rsa -p 53801 \
  ec2-52-89-0-29.us-west-2.compute.amazonaws.com build TestJob1
# Dans la "Console Output" du job TestJob1, vous devriez voir :
# Started from command line by gsmith

# Installer un plugin via la CLI SSH
java -jar jenkins-cli.jar -ssh -user gsmith -i ~/.ssh/aws/iam/id_rsa \
    -s http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/ \
    install-plugin role-strategy
# Vers la fin de la sortie, vous devriez voir :
# "Installing role-strategy from update center"

# Alias dans .zshrc pour installer le même plugin
alias install-role-based-plugin="java -jar jenkins-cli.jar \
  -ssh -user gsmith -i ~/.ssh/aws/iam/id_rsa \
  -s http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/ \
  install-plugin role-strategy"

Configuration example ~/.ssh/config:

Host git-codecommit.*.amazonaws.com
  User APKATPLNME26IS22PR4E
  IdentityFile ~/.ssh/aws/iam/jenkins1_iam_rsa

Host ec2-3-88-31-207.compute-1.amazonaws.com
  User ec2-user
  IdentityFile ~/.ssh/aws/iam/jenkins1_iam_rsa

6.3 Secure agents before scaling

Now that everything is configured and fully functional, let’s start hardening the installation. We don’t want anyone to be able to directly access the agents — let’s create a new security group that only allows access from the master node.

Create a new security group for agents

1. In the EC2 console, click Security Groups in the left sidebar.
2. Select the security group you want to copy.
3. Click Actions and choose Copy to new.
4. The Create Security Group dialog opens, populated with the rules of the existing security group.
5. Give the new security group a name, for example jenkins-agent-sg, and a description.
6. Select your current VPC ID.
7. Modify the Inbound rules to only allow access to agents from the master node (private IP address of the master).
8. Click Create security group.

Apply the new security group to agents

1. Return to the EC2 instances screen.
2. Right-click the agent instance.
3. Select Security → Change security groups.
4. Select the new security group jenkins-agent-sg from the drop-down menu.
5. Delete the old security group and click Save.

Change agent configuration in Jenkins

1. Take the private IP address of the agent.
2. In Jenkins master, open the agent configuration.
3. Change the value of Host from public IP address to private IP address.
4. Save the configuration.
5. Run a quick build to confirm that everything still works.

Traffic between the master and the agent no longer passes through public networks.

For agents outside your organization

If an agent is hosted outside of your organization, it is recommended to improve security by only allowing SSH key access.

6.4 Role management (Role-Based Access Control)

For improved user and group management, let’s enable Role-Based Strategy.

Activate Role-Based Strategy

1. Navigate to Jenkins → Security.
2. Scroll down until you see Role-Based Strategy under Authorization.
3. Select it from the drop-down menu.

Role types

There are three types of roles in Jenkins:

Create roles

1. Navigate to Manage Jenkins → Manage and Assign Roles.

Create a global role “developer”:

1. Type developer in the Role to add field and click Add.
2. Set permissions for developers (limited permissions recommended):

• Overall: Read
• Credentials: View
• Job: Read
• View: Configure, Create, Read
• Do not grant them the creation, configuration, deletion or manipulation of the workspace.
• Do not allow them to run jobs.

3. Click Save.

Create a test user

1. Navigate to Manage Jenkins → Manage Users → Create User.
2. Enter a username, for example dev1.
3. Enter the password, confirmation, full name and email address.
4. Click Create user.

Assign a role to a user

1. Navigate to Manage Jenkins → Manage and Assign Roles.
2. Under Assign Roles, in the Global roles section, enter dev1 in the User/group to add field.
3. dev1 appears under global roles — check the box for the developer role.
4. Click Save.
5. To test, log out and log back in as dev1 to verify that permissions are applied correctly.

6.5 Future outlook

You have gained a wealth of knowledge throughout this training, and you are now ready to take on serious challenges. There is always more to learn.

With cloud technology trends booming, the natural next step is to expand your knowledge in the wide range of technologies offered by cloud providers like AWS, Microsoft Azure, Google Cloud, Digital Ocean, and many others.

Priority areas for your professional development

1. Command line interfaces (CLI) for all major cloud providers.
2. Cloud-native CI/CD services like AWS CodeBuild and CodePipeline.
3. Scripting in Groovy, Go and other popular languages ​​applicable to DevOps and automation.
4. Integrations with scalable cloud services in the areas of automation, security, storage and high availability.

This is of course just a short list. As technologies advance, new areas emerge. The bottom line is that you have started an extraordinary journey.

7. Reference code files

This section centralizes all the code files for the practical training exercises.

7.1 CloudFormation Template — jenkins_ami.yaml

Basic CloudFormation template to create an EC2 instance with Jenkins installed, including Java, Maven, Git, gsmith user and CloudWatch agent.

AWSTemplateFormatVersion: 2010-09-09

Description: Create an AMI from an EC2 instance.
Parameters:
  ImageId:
    Description: Image ID for base EC2 instance.
    Type: AWS::EC2::Image::Id
    Default: ami-0fa40e25bf4dda1f6
  InstanceType:
    Description: Instance type to launch EC2 instances.
    Type: String
    Default: t2.micro
    AllowedValues: [ t2.micro, t2.nano ]
Resources:
  SSHSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable SSH access via port 22
      SecurityGroupIngress:
      - CidrIp: 0.0.0.0/0
        FromPort: 22
        IpProtocol: tcp
        ToPort: 22
  WebSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupDescription: Enable HTTP access via user defined port
      SecurityGroupIngress:
      - CidrIp: 0.0.0.0/0
        FromPort: 8080
        IpProtocol: tcp
        ToPort: 8080
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Ref ImageId
      InstanceType: !Ref InstanceType
      IamInstanceProfile: !Ref IAMInstanceProfile
      SecurityGroups:
      - !Ref SSHSecurityGroup
      - !Ref WebSecurityGroup
      UserData:
        "Fn::Base64": !Sub |
          #!/bin/bash -x
          yum update -y
          touch /home/ec2-user/install.log
          chmod a+w install.log
          echo "Installing java-21-amazon-corretto-devel: " >> /home/ec2-user/install.log
          yum install java-21-amazon-corretto-devel -y >> /home/ec2-user/install.log
          echo "END Java Installation" >> /home/ec2-user/install.log
          echo "=====================" >> /home/ec2-user/install.log
          echo "" >> /home/ec2-user/install.log

          echo "Installing Jenkins" >> /home/ec2-user/install.log
          wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo >> /home/ec2-user/install.log
          rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key >> /home/ec2-user/install.log
          yum upgrade
          yum install jenkins -y >> /home/ec2-user/install.log
          systemctl enable jenkins >> /home/ec2-user/install.log
          systemctl start jenkins >> /home/ec2-user/install.log
          echo "END Jenkins Installation" >> /home/ec2-user/install.log
          echo "" >> /home/ec2-user/install.log

          yum install git -y >> /home/ec2-user/install.log
          yum install maven -y >> /home/ec2-user/install.log
          echo "Installed Git & Maven" >> /home/ec2-user/install.log
          echo "" >> /home/ec2-user/install.log

          adduser gsmith
          echo 'gsmith:C@t&D0g123' | chpasswd
          mkdir /home/gsmith/.ssh && chmod 700 /home/gsmith/.ssh
          ssh-keygen -q -t rsa -N '' -f /home/gsmith/.ssh/agent_rsa 2>/dev/null <<< y >/dev/null
          touch /home/gsmith/.ssh/authorized_keys
          cat /home/gsmith/.ssh/agent_rsa.pub >> /home/gsmith/.ssh/authorized_keys
          chmod 600 /home/gsmith/.ssh/authorized_keys
          chmod 400 /home/gsmith/.ssh/agent_rsa*
          chown -R gsmith:gsmith /home/gsmith/.ssh/
          echo "Created user gsmith and all related files with proper permissions" >> /home/ec2-user/install.log

          yum install amazon-cloudwatch-agent -y >> /home/ec2-user/install.log

          /opt/aws/bin/cfn-signal \
            -e $? \
            --stack ${AWS::StackName} \
            --region ${AWS::Region} \
            --resource AMICreate
          chown ec2-user:ec2-user /home/ec2-user/install.log
      KeyName: "jenkins-kp1"
      Tags:
        - Key: Name
          Value: Jenkins AMI
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          default:
            - 01_setupCfnHup
            - 02_config-amazon-cloudwatch-agent
            - 03_restart_amazon-cloudwatch-agent
          UpdateEnvironment:
            - 02_config-amazon-cloudwatch-agent
            - 03_restart_amazon-cloudwatch-agent
        02_config-amazon-cloudwatch-agent:
          files:
            /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json:
              content: |
                {
                  "metrics": {
                    "append_dimensions": {
                      "AutoScalingGroupName": "${!aws:AutoScalingGroupName}",
                      "ImageId": "${!aws:ImageId}",
                      "InstanceId": "${!aws:InstanceId}",
                      "InstanceType": "${!aws:InstanceType}"
                    },
                    "metrics_collected": {
                      "mem": {
                        "measurement": ["mem_used_percent"]
                      },
                      "swap": {
                        "measurement": ["swap_used_percent"]
                      }
                    }
                  }
                }
        03_restart_amazon-cloudwatch-agent:
          commands:
            01_stop_service:
              command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a stop
            02_start_service:
              command: /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json -s
        01_setupCfnHup:
          files:
            /etc/cfn/cfn-hup.conf:
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}

7.2 CloudFormation Template with CloudWatch Logging

Extended template that also includes rsyslog configuration for local logging, and CloudWatch configuration for streaming logs to AWS CloudWatch.

# (Extrait de jenkins_ami_with_cloudwatch_logging.yaml)
# Ce template est identique au template de base avec les ajouts suivants
# dans la section UserData :

          yum install rsyslog -y >> /home/ec2-user/install.log
          chmod a+r /var/log/messages
          systemctl enable rsyslog
          systemctl start rsyslog

Full template for CloudWatch Agent (cwagent_amazon_linux.yaml):

AWSTemplateFormatVersion: "2010-09-09"
Description: Template to install CloudWatchAgent on amazon linux.

Parameters:
  KeyName:
    Description: Name of an existing EC2 KeyPair
    Type: AWS::EC2::KeyPair::KeyName
    Default: "jenkins-kp1"
  InstanceType:
    Description: EC2 instance type
    Type: String
    Default: m4.2xlarge
  InstanceAMI:
    Description: Managed AMI ID for EC2 Instance
    Type: AWS::EC2::Image::Id
    Default: ami-7707a10f
  SSHLocation:
    Description: IP range for SSH access
    Type: String
    Default: 0.0.0.0/0

Resources:
  EC2Instance:
    CreationPolicy:
      ResourceSignal:
        Count: 1
        Timeout: PT15M
    Type: AWS::EC2::Instance
    Metadata:
      AWS::CloudFormation::Init:
        configSets:
          default:
            - 01_setupCfnHup
            - 02_config-amazon-cloudwatch-agent
            - 03_restart_amazon-cloudwatch-agent
          UpdateEnvironment:
            - 02_config-amazon-cloudwatch-agent
            - 03_restart_amazon-cloudwatch-agent
        02_config-amazon-cloudwatch-agent:
          packages:
            yum:
              amazon-cloudwatch-agent: []
          files:
            /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_amazon-cloudwatch-agent.json:
              owner: root
              group: root
              mode: 000400
              content: !Sub |-
                {
                  "agent": {
                    "region": "${AWS::Region}",
                    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                    "debug": true
                  },
                  "logs": {
                    "logs_collected": {
                      "files": {
                        "collect_list": [
                          {
                            "file_path": "/var/log/user-data.log",
                            "log_group_name": "${LogGroup}",
                            "log_stream_name": "{instance_id}/user-data.log"
                          },
                          {
                            "file_path": "/var/log/cfn-hup.log",
                            "log_group_name": "${LogGroup}",
                            "log_stream_name": "{instance_id}/cfn-hup.log"
                          },
                          {
                            "file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
                            "log_group_name": "${LogGroup}",
                            "log_stream_name": "{instance_id}/amazon-cloudwatch-agent.log"
                          }
                        ]
                      }
                    },
                    "log_stream_name": "default_log_stream"
                  }
                }

7.3 CloudWatch Agent Configuration — wizard_generated_config.json

Configuration generated by the CloudWatch Configuration Wizard, including collection of CPU, disk, memory, and network metrics, as well as collection of system logs:

{
    "agent": {
        "metrics_collection_interval": 60,
        "run_as_user": "cwagent"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/var/log/messages",
                        "log_group_class": "STANDARD",
                        "log_group_name": "jenkins.log",
                        "log_stream_name": "{instance_id}",
                        "retention_in_days": 30
                    }
                ]
            }
        }
    },
    "metrics": {
        "aggregation_dimensions": [
            ["InstanceId"]
        ],
        "append_dimensions": {
            "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
            "ImageId": "${aws:ImageId}",
            "InstanceId": "${aws:InstanceId}",
            "InstanceType": "${aws:InstanceType}"
        },
        "metrics_collected": {
            "cpu": {
                "measurement": [
                    "cpu_usage_idle",
                    "cpu_usage_iowait",
                    "cpu_usage_user",
                    "cpu_usage_system"
                ],
                "metrics_collection_interval": 60,
                "resources": ["*"],
                "totalcpu": false
            },
            "disk": {
                "measurement": [
                    "used_percent",
                    "inodes_free"
                ],
                "metrics_collection_interval": 60,
                "resources": ["*"]
            },
            "diskio": {
                "measurement": ["io_time"],
                "metrics_collection_interval": 60,
                "resources": ["*"]
            },
            "mem": {
                "measurement": ["mem_used_percent"],
                "metrics_collection_interval": 60
            },
            "statsd": {
                "metrics_aggregation_interval": 60,
                "metrics_collection_interval": 10,
                "service_address": ":8125"
            },
            "swap": {
                "measurement": ["swap_used_percent"],
                "metrics_collection_interval": 60
            }
        }
    }
}

7.4 Configuring CloudWatch metrics — metrics_snippet.json

Minimal configuration snippet for collecting memory and swap metrics:

{
  "metrics": {
    "append_dimensions": {
      "AutoScalingGroupName": "${!aws:AutoScalingGroupName}",
      "ImageId": "${!aws:ImageId}",
      "InstanceId": "${!aws:InstanceId}",
      "InstanceType": "${!aws:InstanceType}"
    },
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent"]
      },
      "swap": {
        "measurement": ["swap_used_percent"]
      }
    }
  }
}

7.5 Scripts and commands — scripts_and_config.sh

Complete script for Jenkins HA configuration, EFS mount, HAProxy configuration and backups:

# Installer les utilitaires NFS
yum install nfs-utils -y

# Ouvrir fstab
sudo vi /etc/fstab

# Montage du partage EFS
us-east-1a.fs-f8d1697b.efs.us-east-1.amazonaws.com:/    /var/lib/jenkins/jobs   nfs defaults    0    0

# Commandes de montage manuelles
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport \
  fs-f8d1697b.efs.us-east-1.amazonaws.com:/ efs
mount -t nfs fs-f8d1697b.efs.us-east-1.amazonaws.com:/ /mnt/test

# Arrêter Jenkins
sudo systemctl stop jenkins.service

# Vérifier le volume monté
df -h

# Changer le propriétaire du dossier jobs
sudo chown -R jenkins:jenkins /var/lib/jenkins/jobs
ls -l /var/lib/jenkins/jobs

# Redémarrer ou démarrer le service Jenkins
sudo systemctl start jenkins.service
sudo systemctl restart jenkins.service

# Vérifier le statut
systemctl status jenkins.service

# Vérifier que le dossier de builds est disponible sur le partage EFS
ls -l /var/lib/jenkins/jobs/Test-HA

# Créer un script de synchronisation
sudo vi /opt/sync_passive_jenkins_with_master.sh

# Aide et rechargement de la configuration
ssh -l gsmith -i ~/.ssh/aws/ec2/slave_rsa -p 53802 \
  ec2-3-82-125-219.compute-1.amazonaws.com help
ssh -l gsmith -i ~/.ssh/aws/ec2/slave_rsa -p 53802 \
  ec2-3-82-125-219.compute-1.amazonaws.com reload-configuration

# Planifier le crontab
sudo vi /etc/cron.d/reload_jenkins_config

# Expression cron dans reload_jenkins_config (toutes les 2 minutes)
*/2 * * * * root /bin/bash /opt/sync_passive_jenkins_with_master.sh

# Installer HAProxy et le configurer
sudo yum install haproxy -y
sudo vi /etc/haproxy/haproxy.cfg

# Vider le contenu de haproxy.cfg
:1,$d

sudo systemctl start haproxy.service
sudo systemctl restart haproxy.service

# Sauvegarde manuelle du dossier jobs
tar cvzf master-node-jobs-2025-02-14.tar.gz /var/lib/jenkins/jobs

# Exemple de crontab de sauvegarde (toutes les 3 heures)
*3 * * * * root tar cvzf master-$(date +%Y-%m-%d).tar.gz /var/lib/jenkins/jobs

7.6 Allow SSH password authentication

# Ouvrir /etc/ssh/sshd_config en tant que root
vi /etc/ssh/sshd_config

# Décommenter PasswordAuthentication yes
# Décommenter PubkeyAuthentication yes
# Vérifier que AuthorizedKeysFile .ssh/authorized_keys est décommenté
# Sauvegarder et quitter : :wq

# Redémarrer le service sshd
systemctl restart sshd.service
systemctl status sshd.service

# Mettre à jour le DNS public de Jenkins (nécessaire après chaque redémarrage EC2)
# Récupérer le DNS public depuis la console EC2
vi /var/lib/jenkins/jenkins.model.JenkinsLocationConfiguration.xml
# Remplacer l'ancienne URL dans <jenkinsUrl> par la nouvelle valeur

# Redémarrer le service Jenkins
systemctl restart jenkins.service

# Connexion SSH avec username et mot de passe
ssh -t gsmith@54.184.34.103
# Mot de passe : C@t&D0g123

# Récupérer la clé privée depuis la VM agent (générée par CloudFormation)
cat /home/gsmith/.ssh/agent_rsa

# Changer les permissions de la clé SSH
chmod 600 ~/.ssh/aws/iam/agent_rsa
ls -l ~/.ssh/aws/iam/agent_rsa

# Connexion avec la clé privée SSH
ssh -i ~/.ssh/aws/iam/agent_rsa gsmith@54.184.34.103

7.7 Load simulation — heavy_load.sh

#!/bin/bash

# Nombre de processus parallèles (doit correspondre ou dépasser le nombre d'exécuteurs)
NUM_PROCESSES=$(nproc)  # Obtenir le nombre de cœurs CPU

# Durée pour exécuter la charge en secondes
DURATION=60
echo "Starting load simulation with $NUM_PROCESSES processes for $DURATION seconds..."

# Fonction pour générer une charge CPU
generate_load() {
  local end_time=$((SECONDS + DURATION))
  while [ $SECONDS -lt $end_time ]; do
    echo "scale=5000; a(1)*4" | bc -l >/dev/null 2>&1
  done
}

# Démarrer les processus de charge en arrière-plan
for i in $(seq 1 $NUM_PROCESSES); do
  generate_load &
done

# Attendre que tous les processus en arrière-plan se terminent
wait
echo "Load simulation completed."

7.8 Installing Prometheus

### INSTALLATION DE PROMETHEUS SUR LES DISTRIBUTIONS LINUX RHEL ###

# Exécuter en tant que root, ou préfixer chaque commande avec "sudo"
useradd -M -U prometheus

wget https://github.com/prometheus/prometheus/releases/download/v3.2.1/prometheus-3.2.1.linux-386.tar.gz
tar -xzvf prometheus-3.2.1.linux-386.tar.gz
mv prometheus-3.2.1.linux-386 /opt/prometheus

chown prometheus:prometheus -R /opt/prometheus

# Créer le fichier de service systemd pour le service Prometheus
vi /etc/systemd/system/prometheus.service

# Charger et configurer le nouveau service pour démarrer au boot
systemctl daemon-reload
systemctl start prometheus.service
systemctl enable prometheus.service
systemctl status prometheus.service

7.9 systemd service for Prometheus

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Restart=on-failure
ExecStart=/opt/prometheus/prometheus \
--config.file=/opt/prometheus/prometheus.yml \
--storage.tsdb.path=/opt/prometheus/data \
--storage.tsdb.retention.time=30d

[Install]
WantedBy=multi-user.target

7.10 Installing Grafana

# Créer le fichier de repo Grafana avec nano
nano /etc/yum.repos.d/grafana.repo
# (coller le contenu du repo Grafana officiel)

# Installer Grafana via yum
yum install grafana -y

# Sur macOS
# brew install grafana

# Recharger la configuration systemd
systemctl daemon-reload

# Démarrer le service Grafana
systemctl start grafana-server.service

# Configurer Grafana pour démarrer au boot
systemctl enable grafana-server.service

# Vérifier le statut
systemctl status grafana-server.service

7.11 Creating the gsmith user for SSH

# Se connecter en tant que root depuis le client SSH de la console AWS :
# Action > Connect > EC2 Instance Connect (connexion SSH via navigateur)

# Ajouter l'utilisateur
adduser gsmith
sudo su - gsmith

# Configurer le dossier .ssh
mkdir .ssh && chmod 700 .ssh && cd .ssh

# Générer les clés SSH et définir les permissions
ssh-keygen -q -N "" # Appuyer sur Entrée pour accepter les noms de clés par défaut
chmod 600 id_rsa && chmod 600 id_rsa.pub

# Créer authorized_keys et définir les permissions
touch authorized_keys && chmod 600 authorized_keys
chmod 400 id_rsa*

# Définir un mot de passe pour l'utilisateur (en tant que root)
passwd gsmith  # Fournir et confirmer le nouveau mot de passe

# Création automatisée de l'utilisateur
adduser gsmith
sudo su - gsmith
mkdir .ssh && chmod 700 .ssh && cd .ssh
ssh-keygen -q -t rsa -N '' -f ~/.ssh/slave_rsa 2>/dev/null <<< y >/dev/null
touch authorized_keys && chmod 600 authorized_keys
chmod 400 slave_rsa*
echo "Created user gsmith and all related files with proper permissions" >> /home/ec2-user/install.log

# Générer des clés SSH (méthode alternative)
ssh-keygen -t rsa -N ""

7.12 Jenkins commands via SSH

# Afficher l'aide — liste de commandes disponibles et leurs descriptions
ssh -l gsmith -i id_rsa -p 53801 \
  ec2-52-89-0-29.us-west-2.compute.amazonaws.com help

# Lister tous les jobs que gsmith peut exécuter
ssh -l gsmith -i id_rsa -p 53801 \
  ec2-52-89-0-29.us-west-2.compute.amazonaws.com list-jobs

# Lancer un build
ssh -l gsmith -i id_rsa -p 53801 \
  ec2-52-89-0-29.us-west-2.compute.amazonaws.com build TestJob1

# Dans la "Console Output" du job TestJob1, vous devriez voir :
# Started from command line by gsmith

# Installer un plugin via la CLI Jenkins
java -jar jenkins-cli.jar -ssh -user gsmith -i ~/.ssh/aws/iam/id_rsa \
    -s http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/ \
    install-plugin role-strategy

# Vers la fin de la sortie :
# "Installing role-strategy from update center"

# Alias dans .zshrc pour installer le même plugin
alias install-role-based-plugin="java -jar jenkins-cli.jar \
  -ssh -user gsmith -i ~/.ssh/aws/iam/id_rsa \
  -s http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/ \
  install-plugin role-strategy"

7.13 Jenkins SSH Configuration Test

# Vérifier que l'endpoint SSH Jenkins est accessible
curl -Lv http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:8080/login 2>&1 \
  | grep -i 'x-ssh-endpoint'

# Réponse exemple confirmant que la config SSH est valide :
# < X-SSH-Endpoint: http://ec2-52-89-0-29.us-west-2.compute.amazonaws.com:53801

# Définir les permissions correctes pour la clé privée
chmod 600 id_rsa

7.14 Services start/stop commands

# Jenkins
systemctl start jenkins.service
systemctl restart jenkins.service
systemctl status jenkins.service
systemctl stop jenkins.service

# SSHD
systemctl start sshd.service
systemctl restart sshd.service
systemctl status sshd.service
systemctl stop sshd.service

7.15 sshd_config configuration example

Host git-codecommit.*.amazonaws.com
  User APKATPLNME26IS22PR4E
  IdentityFile ~/.ssh/aws/iam/jenkins1_iam_rsa

Host ec2-3-88-31-207.compute-1.amazonaws.com
  User ec2-user
  IdentityFile ~/.ssh/aws/iam/jenkins1_iam_rsa

Search Terms

multi-node · jenkins · farm · ci/cd · git · devops · ssh · agents · configuration · prometheus · configuring · security · test · architecture · configure · grafana · installing · plugins · template · agent · authentication · cloudformation · commands · install