Using Jenkins to Automate Artifact Builds and Security

Table of Contents

1. Course presentation
2. Jenkins for DevOps Flows

• 2.1 Business case: Globomantics
• 2.2 Jenkins Capabilities
• 2.3 Jenkins Architecture
• 2.4 Select plugins
• 2.5 Schedule simple builds
• 2.6 Chain builds
• 2.7 Artifact types built with Jenkins

3. VM build automation

• 3.1 Basic installation packages
• 3.2 Virtualization
• 3.3 VMs everywhere with Vagrant
• 3.4 First Vagrant Box: Eclipse Workstation
• 3.5 Second VM: AWS EC2 with Vagrant
• 3.6 Introduction to Packer
• 3.7 Building AMIs with Packer
• 3.8 Packer integration with Jenkins
• 3.9 Introduction to HashiCorp Vault
• 3.10 Vault integration with Vagrant and Packer

4. Container Build Automation

• 4.1 Container virtualization
• 4.2 Benefits of containerization
• 4.3 Containerize an Apache Web server
• 4.4 Containerize MySQL
• 4.5 Integrate Docker with Jenkins
• 4.6 Docker Compose
• 4.7 Integrate Docker Compose with Jenkins
• 4.8 Container security with HashiCorp Vault

5. Serverless build automation

• 5.1 Serverless Features
• 5.2 Serverless business case at Globomantics
• 5.3 Serverless architecture
• 5.4 Create serverless components
• 5.5 Automate creation with Terraform
• 5.6 Integrate Terraform with Jenkins
• 5.7 Security with HashiCorp Vault (serverless)
• 5.8 Best Practices and Course Summary

1. Course presentation

This course covers automating builds of many important artifacts using Jenkins, covering the following technologies:

• Vagrant — Developer environments (local VMs)
• HashiCorp Packer — Amazon Machine Images (AMI) for the AWS Cloud
• Docker and Docker Compose — Containers and multi-container applications
• HashiCorp Terraform — Serverless environments (Infrastructure as Code)
• HashiCorp Vault — Secrets management and security

The main goal is to show how all these technologies can benefit your business and your customers. This is the first step in streamlining proven and cutting-edge technologies to provide sustainable competitive advantages in the market.

Through the journey of the fictitious company Globomantics, the course illustrates how needs evolve during a major change in business strategy. At the end of this course, you will be able to integrate all these artifacts into your CI/CD (Continuous Integration / Continuous Delivery) pipeline with Jenkins.

2. Jenkins for DevOps Flows

2.1 Business case: Globomantics

Globomantics is a mid-sized software company that has experienced excellent organic growth. Its customers are satisfied with all the products in its portfolio and come back regularly. Revenue is growing year over year, and the company has gone through Series B venture funding and then Series C, leading to several acquisitions over the past year.

However, the company now faces new challenges:

• Integrate disparate products that deliver complementary business value into a cohesive portfolio of tiered products quickly and efficiently.
• Accelerate the average velocity of developer teams to deliver new features.
• Maintain the productivity of IT teams, or even increase it.
• Add security to the mix, as more and more vulnerabilities would result from these integrations.

In summary, Globomantics needs to overhaul its CI/CD pipeline with security built in. After deliberation, the company retained:

• Jenkins as the heart of the CI/CD platform — relatively inexpensive and allowing almost unlimited customizations.
• HashiCorp for security — well-established modern secrets management solution that integrates with most secrets providers (on-premises, cloud, SaaS).

2.2 Jenkins Capabilities

Initially designed as a Continuous Integration / Continuous Delivery tool, Jenkins can be used for:

1. Integrate code produced by many developers in parallel on an integration server and check whether the code build would fail.
2. Automatically test the code before it is ready for production.
3. Deploy the code to production once testing passes and approval is obtained (manually or automatically).

Short version: Jenkins can build, test and deploy software.

Jenkins Core Features

Jenkins allows you to:

• Create units of work called jobs or projects.
• Schedule jobs at specific intervals.
• Chain jobs to run in a particular order, with the ability to prevent successive jobs from running if previous ones have failed.
• Provide a security model that determines who can do what in Jenkins.

The main motivations for automation are:

• Reduce costs and accelerate delivery of features to customers (accelerating business value).
• Deploy more software products to compete more effectively, protect or gain market share.

2.3 Jenkins Architecture

Jenkins has a flexible plugin architecture. You can easily modify its behavior by adding, removing, replacing and configuring plugins via:

• A graphical user interface (GUI)
• A command line interface (CLI)
• Various configuration options for each plugin

Main interaction flow

Développeurs --> [Push code] --> Dépôt de code
Dépôt de code --> [Trigger automatique] --> Jenkins
Jenkins --> [Notification] --> Développeurs (succès/échec)

Distributed architecture

Jenkins supports distributed builds via:

• A single master node that delegates builds to one or more agents.
• Support for parallel builds for different environments (dev, QA, staging, production).
• The master node can also run builds locally.
• It schedules jobs, communicates and monitors agents, presents results and sends notifications.

High availability: Not available out-of-the-box, but can be configured with free tools like HAProxy.

Agents can also support microservices architecture, where each Jenkins agent builds a specific API for a given environment (for example, the Cart service for the QA environment of an e-commerce system). Agents can run on different operating systems. As Jenkins is a Java web application, it can run on any platform.

2.4 Select plugins

Plugins are a topic in their own right. They allow easy customization of Jenkins features.

Top Plugin Issues

1. Endless updates and dependencies

Even small projects can require more than 25 plugins. Each installed plugin may require other plugins as dependencies. Different versions of dependencies required by different plugins may cause installation failures and/or bugs.

2. Security breaches

As attacks become more and more sophisticated, security holes are inevitable. Three options:

• Open a ticket on GitHub and wait for a fix — acceptable if the plugin is well maintained, but mitigations are still necessary. The worst scenario is when the fix never arrives because the maintainers abandoned the code.
• Fork the plugin repository and fix the code yourself — quick fix, but you inherit the work. This can easily become a weekly task to check and patch new vulnerabilities.
• Replace vulnerable plugin with an alternative — risk of unreliable maintenance program over time, or need to pay a subscription. And you have to refactor and re-test all the pipelines that depended on the replaced plugin.

3. Variable quality and maintenance

Jenkins plugins come from a wide variety of sources. Some are produced by the open source community, others by commercial vendors. Quality, update frequency, and support vary widely. Some popular plugins are very well maintained, others can be abandoned overnight.

Plugin selection criteria

Here are the key indicators to check before adopting a plugin:

Alternative approach: Shell scripts in the build step

The approach recommended in this course is to minimize the use of plugins by using shell scripts in a standard Execute Shell build step. This approach:

• Significantly reduces the number of plugins needed.
• Insulates your CI/CD installation from the vagaries of dependencies and plugin maintenance.
• Allows great flexibility and customization via script variables.
• Is easier to debug and understand.

2.5 Schedule simple builds

For simple builds, you must have:

1. All necessary plugins installed and fully operational.
2. Additional software installed on Jenkins agents.

Here is the short list of dependencies:

• System Utilities: On Linux systems this may include sed, awk, curl (often pre-installed). It is useful to update them to their latest versions and install other utilities, like KBackup for automated backups.
• Artifact build tools: Vagrant, Packer.
• Cloud specific tools: AWS CLI, HashiCorp Terraform.
• Jenkins Interface Components: Flow enhancement themes and plugins.

Once the extended nomenclature is in place, we are ready to define and test the jobs.

2.6 Chaining builds

There are several ways to link builds for better DevOps flows:

• Schedule jobs in specific sequences.
• Create dependencies between jobs (run before or after another job).
• Complex orchestrations with the join plugin: run branches in parallel, then run jobs that depend on previous parallel jobs.
• Example: Assemble common libraries, then build microservices in parallel, and finally deploy all microservices to Amazon ECS.
• Conditional logic in jobs: check conditions to decide whether the chain should continue, stop, wait, retry, send notifications, publish results, etc.
• Orchestration via external systems: triggering Jenkins via its own API (webhooks, reverse API calls, hooks).

2.7 Types of artifacts built with Jenkins

In this course, we will construct:

1. Virtual Machines — For developer desktops (via Vagrant/VirtualBox) and cloud VMs (via Vagrant/AWS and Packer/AWS).
2. Containers — Via Docker Engine integrated with Jenkins, and Docker Compose for multi-container applications.
3. Serverless components — Specific to AWS but applicable to other cloud providers, via HashiCorp Terraform.

3. Automating VM builds

3.1 Basic installation packages

Before starting the demos, you must install the following basic packages:

• Java — Required for Jenkins
• Jenkins — The central continuous integration tool
• Vagrant — To control VirtualBox and its VMs programmatically
• Ansible — For software configuration management

Installation according to operating system

Installing Jenkins on macOS (example)

# Installer Jenkins via Homebrew (version LTS)
brew install jenkins-lts

# Démarrer Jenkins
brew services start jenkins-lts

# Accéder à Jenkins
# http://localhost:8080

Important: If anything else is running on port 8080, Jenkins will not start. Make sure this port is available.

For installation, simply:

1. Unlock Jenkins by getting the token from the unlock file.
2. Install suggested plugins or select plugins manually.
3. Create the first administrator account.

3.2 Virtualization

Virtualization technology makes it possible to run multiple operating systems on the same hardware.

Two main types

Hosted Virtualization

Hardware --> Host OS --> Hypervisor --> Guest OS

• The hypervisor is just another application on the computer.
• Configuration is minimal.
• Ideal for personal use because it is flexible.
• Example: VirtualBox on macOS or Windows.

Bare Metal Virtualization

Hardware --> Hypervisor --> VMs (une application par VM)

• The hypervisor runs directly on hardware.
• Typically only one application per virtual machine.
• Better performance with less overhead between each VM.
• Meaning for use cases such as resource provisioning in data centers.
• This is what we call today Infrastructure as a Service (IaaS): AWS, Azure, GCP.

3.3 VMs everywhere with Vagrant

Brief history

Vagrant was originally created by Mitchell Hashimoto, one of the founders of HashiCorp. It still features prominently in the company’s portfolio, alongside Terraform and Packer.

What is Vagrant?

Vagrant is an open source application that allows automating VM provisioning and configuration. To use it, we define a plain text configuration file (the Vagrantfile) which describes the virtual machines and their environment, including the system packages to install. You can also integrate Vagrant with provisioning software like Chef, Puppet or Ansible.

Supported hypervisors

• Oracle VirtualBox
• VMware
• Microsoft Hyper-V
• And others via plugins (AWS, Azure, etc.)

Vagrant lifecycle

Vagrantfile --> vagrant up --> vagrant ssh --> vagrant halt --> vagrant destroy
                                                         |
                                                   vagrant status (à tout moment)

The main commands:

Important: Each of these commands must be executed from the root directory of your project, otherwise they will throw errors.

The Vagrantfile is portable between different operating systems (Windows, macOS, Linux), which makes it possible to give the same configuration to all members of a development team.

3.4 First Vagrant Box: Eclipse Workstation

Objective

Create an Ubuntu developer workstation with:

• Ubuntu (stable release)
• Java 11 (OpenJDK)
• Eclipse (IDE) configured and ready to use

Java and Eclipse installation script

#!/bin/bash 

# Définir un terminal non-interactif
export DEBIAN_FRONTEND=noninteractive

# Mettre à jour les paquets
sudo apt-get -y update

# Installer Java et définir JAVA_HOME
sudo apt-get -y install openjdk-11-jdk
sudo apt-get -y install openjdk-11-source

sudo echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/vagrant/.bashrc
sudo echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /home/vagrant/.bashrc

sudo echo 'export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64' >> /home/vagrant/.zshrc
sudo echo 'export PATH=$PATH:$JAVA_HOME/bin' >> /home/vagrant/.zshrc

# Télécharger et installer Eclipse
sudo wget -O latest-eclipse.tar.gz https://archive.eclipse.org/technology/epp/downloads/release/2022-09/RC1/eclipse-java-2022-09-RC1-linux-gtk-x86_64.tar.gz
sudo tar -zxf latest-eclipse.tar.gz -C /opt
sudo chown -R vagrant:vagrant /opt/eclipse

# Journaliser l'activité
sudo echo 'install_java_and_eclipse.sh executed' >> /tmp/vagrant.log

Note: Eclipse is downloaded with wget directly from the archive link, rather than with Snap, because Snap makes it difficult to customize the installation.

Complete Vagrantfile for Eclipse post

Vagrant.configure("2") do |config|
    config.vm.box = "codeup/Ubuntu-20.04-GUI"
    config.vm.provider "virtualbox" do |vb|
        vb.gui = true
        vb.name = "Eclipse"
        vb.memory = 2048
        vb.cpus = 2
        vb.customize ["modifyvm", :id, "--vram", 256]
        vb.customize ["modifyvm", :id, "--accelerate3d", "on"]
        vb.customize ["modifyvm", :id, "--clipboard", "bidirectional"]
        vb.customize ["modifyvm", :id, "--draganddrop", "bidirectional"]
        vb.customize ["modifyvm", :id, "--graphicscontroller", "vmsvga"]
        vb.customize ["setextradata", :id, "GUI\/LastGuestSizeHint", "1920,1080"]
    end

    config.ssh.username = 'vagrant'
    config.ssh.password = 'vagrant'
    config.ssh.insert_key = false

    # Empêcher le service de mises à jour non surveillées d'interférer avec le provisionnement
    config.vm.provision "shell", inline: "systemctl stop unattended-upgrades.service"
    
    # Corriger l'erreur dpkg stdin
    config.vm.provision "shell", inline: "ex +\"%s@DPkg@\/\/ DPkg\" -cwq /etc/apt/apt.conf.d/70debconf", privileged: true
    config.vm.provision "shell", inline: "dpkg-reconfigure debconf -f noninteractive -p critical", privileged: true

    # Créer un log sur la VM invitée
    config.vm.provision "shell", inline: "touch /tmp/vagrant.log", privileged: true

    # Rendre les scripts exécutables
    config.vm.provision "shell", path: "password_authentication_on.sh", :args => "'#{ENV['VAGRANT_PWD']}'", privileged: true
    config.vm.provision "shell", path: "install_java_and_eclipse.sh", privileged: true
    config.vm.provision "shell", path: "install_dkms.sh", privileged: true

    # Créer le raccourci bureau Eclipse
    config.vm.provision "file", source: "/Users/gsyyl/learning/vagrant/eclipse/eclipse.desktop", destination: "/home/vagrant/eclipse.desktop"
    config.vm.provision "shell", path: "create_eclipse_desktop_shortcut.sh", privileged: true
    
    # Personnaliser Eclipse
    config.vm.provision "file", source: "org_eclipse_core_runtime/.", 
        destination: "$HOME/eclipse-workspace/.metadata/.plugins/org.eclipse.core.runtime/.settings"
    config.vm.provision "file", source: "org_eclipse_jdt_launching/.", 
        destination: "$HOME/eclipse-workspace/.metadata/.plugins/org.eclipse.core.runtime"
    config.vm.provision "file", source: "startup_config/.", 
        destination: "/opt/eclipse"

    config.trigger.after :up do |t|
        t.info = "Bringing up your Vagrant guest machine, with customized Eclipse IDE!"
        t.run = {path: "adjustments.sh"}
    end
end

Password authentication enablement script

#!/bin/bash 

# Définir un terminal non-interactif
export DEBIAN_FRONTEND=noninteractive

sudo sed -i 's/\#PasswordAuthentication yes/PasswordAuthentication yes/g' /etc/ssh/sshd_config
sudo systemctl restart sshd.service

# $1 contient le mot de passe provenant de Vault
echo "ubuntu:$1" | sudo chpasswd
echo "vagrant:$1" | sudo chpasswd

# Journaliser l'activité
sudo echo 'password_authentication_on.sh executed' >> /tmp/vagrant.log

Vagrant integration with Jenkins

To integrate Vagrant with Jenkins, we create a Freestyle Job with an Execute Shell build step. The shell script reads the AWS keys from the local credentials file, exports the necessary environment variables, and calls vagrant up:

#!/bin/bash

# Variables dérivées des paramètres du job - AMI_ID et REGION
export AWS_AMI=$AMI_ID
export AWS_REGION=$REGION

# Variables d'environnement avec les credentials AWS
export AWS_ACCESS_KEY_ID=$(grep "aws_access_key_id" ~/.aws/credentials | awk '{print $3}')
export AWS_SECRET_ACCESS_KEY=$(grep "secret_access_key" ~/.aws/credentials | awk '{print $3}')
export PATH=$PATH:/usr/local/bin

# Descendre dans le répertoire cible et exécuter "vagrant up"
cd ~/learning/vagrant/apache/ansible && /usr/local/bin/vagrant up

Important points:

• Jenkins parameters (AMI_ID, REGION) are passed as environment variables to the script.
• Using an Execute Shell build step (rather than a dedicated Vagrant plugin) avoids dependency and plugin maintenance issues.
• Always manually test values ​​in your script before integrating them into Jenkins.

3.5 Second VM: AWS EC2 with Vagrant

Change default provider

You can change the default provider in Vagrant (VirtualBox) by defining the variable VAGRANT_DEFAULT_PROVIDER, or by specifying it in the Vagrantfile with config.vm.provider.

Vagrantfile for AWS EC2

class Hash
    def slice(*keep_keys)
      h = {}
      keep_keys.each { |key| h[key] = fetch(key) if has_key?(key) }
      h
    end unless Hash.method_defined?(:slice)
    def except(*less_keys)
      slice(*keys - less_keys)
    end unless Hash.method_defined?(:except)
  end

Vagrant.configure("2") do |config|
    config.vm.box = "dummy"
    config.vm.provider :aws do |aws, override|
        aws.access_key_id = ENV['AWS_ACCESS_KEY_ID']
        aws.secret_access_key = ENV['AWS_SECRET_ACCESS_KEY']
        aws.keypair_name = "ubuntu_webserver"
        aws.ami = "ami-0f136f37138f59e54"
        aws.region = "us-east-1"
        aws.instance_type = "t2.micro"
        aws.security_groups = ['launch-wizard-1']

        config.vm.synced_folder '.', '/vagrant', disabled: true

        # Surcharger l'utilisateur de la dummy box
        override.ssh.username = "ubuntu"
        override.ssh.private_key_path = "/Users/gsyyl/.ssh/aws/ubuntu_webserver.cer"
    end

    config.vm.provision "shell", inline: <<-SHELL
        sudo apt update
        sudo apt install apache2 -y
        sudo ufw allow 'Apache'
        sudo systemctl start apache2
        sudo systemctl enable apache2
        cd /var/www/html
        touch test.html
        echo "<html><body><h1>This is a Test Page!</h1></body></html>" > test.html
    SHELL
end

Key points:

• The box is dummy because for an image cloud (AMI), the box is irrelevant — the image is stored in the cloud.
• The AWS Vagrant plugin must be installed: vagrant plugin install vagrant-aws.
• AWS keys are passed via environment variables.
• Folder synchronization is disabled (disabled: true) — especially important on Windows.
• SSH credentials are overloaded to match the AWS key pair .cer file.

Generating an AWS key pair

1. Navigate to EC2 console > Key Pairs.
2. Click Create, enter ubuntu_webserver as the name.
3. Keep the default settings and click Create keypair.
4. Download the key pair and save it in a secure location (e.g. ~/.ssh/aws/).

Ansible Playbook for Apache

We can also use Ansible to provision the Apache server on the VM:

- hosts: all
  become: true
  tasks:

  - name: Installation of Apache v2 and PHP - Ubuntu
    apt:
      name:
        - apache2
        - libapache2-mod-php
      state: latest
      update_cache: yes
    when: ansible_distribution == "Ubuntu"

  - name: Copy index HTML file - entry point to site
    tags: apache,apache2,httpd
    copy:
      src: index.html
      dest: /var/www/html/index.html
      owner: root
      group: root
      mode: 0644

3.6 Introduction to Packer

Packer vs Vagrant

Both are HashiCorp tools, but designed for different purposes:

Synergy: Packer and Vagrant can work very well together, especially in DevOps environments.

When to use Packer?

Developers push application updates very quickly to DevOps environments. Packer allows you to create immutable images where the entire VM is replaced instead of being updated while it is running. This approach is preferable for production because it ensures consistency and reproducibility.

3.7 Building AMIs with Packer

HCL Syntax v2

Packer uses HashiCorp Configuration Language (HCL) version 2 to define the resources that will be deployed to the cloud VM.

Main structures of a Packer template

1. Variables Used to configure builds.

2. Data Sources Used to retrieve dynamic information, for example the ID of an AMI:

data "amazon-ami" "autogenerated_1" {
  access_key = "${var.aws_access_key}"
  secret_key  = "${var.aws_secret_key}"
  filters = {
    name                = "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-*"
    root-device-type    = "ebs"
    virtualization-type = "hvm"
  }
  most_recent = true
  owners      = ["099720109477"]
  region      = "${var.region}"
}

3. Sources (builders) Define where to build the image:

source "amazon-ebs" "autogenerated_1" {
  access_key    = "${var.aws_access_key}"
  ami_name      = "base-ubuntu-v1.0.0-${local.timestamp}"
  instance_type = "t2.micro"
  region        = "${var.region}"
  run_tags = {
    Name            = "Ubuntu-base"
    type            = "webserver-vm"
    OS              = "Ubuntu"
    author          = "George Smith"
    team            = "Infrastructure"
    sequence_number = "1"
  }
  secret_key   = "${var.aws_secret_key}"
  source_ami   = "${data.amazon-ami.autogenerated_1.id}"
  ssh_username = "ubuntu"
}

4. Blocks Build with Provisioners

build {
  sources = ["source.amazon-ebs.autogenerated_1"]

  # Provisioner shell pour le bootstrap initial
  provisioner "shell" {
    script = "./bootstrap.sh"
  }

  # Copier des fichiers sur la VM distante
  provisioner "file" {
    destination = "/home/ubuntu"
    source      = "./uploads/"
  }

  # Configurer les fichiers distants
  provisioner "shell" {
    script = "./setup_remote_files.sh"
  }

  # Activer les services
  provisioner "shell" {
    script = "./enableservices.sh"
  }

  # Provisioner Ansible pour installer JQ
  provisioner "ansible" {
    playbook_file = "install_jq_json_parser.yaml"
  }

  # Créer l'utilisateur packer avec le mot de passe Vault
  provisioner "shell" {
    environment_vars = ["PACKER_VAULT_PWD=${var.packer_pwd}"]
    script = "./create_packer_user.sh"
  }

  post-processors {
    post-processor "shell-local" {
      inline = [ "echo base-ubuntu-v1.0.0-${local.timestamp} > artifacts.log" ]
    }
  }
}

5. Post-processors Transform the final image after Packer has finished its work:

• Log work
• Send messages to cloud services (notifications)
• Zip image if local
• Convert to VirtualBox/Vagrant
• Calculate a checksum

Bootstrap.sh script

# Attendre que cloud-init soit terminé avant d'installer des paquets
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done

export DEBIAN_FRONTEND="noninteractive"

# Mises à jour OS
sudo apt-get update
sudo apt-get install dialog apt-utils -y
sudo echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
sudo apt-get install -y -q

# Utilitaires
sudo apt-get install wget -y
sudo apt-get install zip unzip -y

# Variables d'environnement
echo "export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64" >> .bashrc
echo "export PATH=$PATH:$JAVA_HOME/bin" >> .bashrc

# Java et Tomcat
sudo apt-get install openjdk-11-jdk -y
sudo apt-get install tomcat9 tomcat9-admin -y

# Activer l'authentification par mot de passe SSH
sudo sed -i -e 's/PasswordAuthentication no/PasswordAuthentication yes/' /etc/ssh/sshd_config
sudo sed -i -e 's/     lock_passwd: True/     lock_passwd: False/' /etc/cloud/cloud.cfg

echo 'ubuntu:bob123' | sudo chpasswd

Packer user creation script

#!/bin/bash

echo "Packer user password from Vault: $PACKER_VAULT_PWD"
sudo adduser packer
echo "packer:$PACKER_VAULT_PWD" | sudo chpasswd

Ansible playbook for JQ

- hosts: all
  become: true
  tasks:

  - name: Installation of JQ JSON Parser - Ubuntu
    apt:
      name:
        - jq
      state: latest
      update_cache: yes
    when: ansible_distribution == "Ubuntu"

Services activation script

# Service Tomcat
sudo systemctl start tomcat9
sudo systemctl enable tomcat9

# Redémarrer sshd pour activer l'authentification par mot de passe
sudo systemctl restart sshd

Remote file configuration script

# Déplacer les fichiers dans leur emplacement final

### Fichier des utilisateurs Tomcat ###
sudo mv /home/ubuntu/tomcat-users.xml /etc/tomcat9/tomcat-users.xml
sudo mv /home/ubuntu/studentrestapi.war /var/lib/tomcat9/webapps

### Configuration de lancement Java ###
sudo mv setenv.sh /usr/share/tomcat9/bin/setenv.sh

# Paramètres par défaut pour l'utilisateur ubuntu
sudo mv defaults.cfg /etc/cloud/cloud.cfg.d/defaults.cfg

# Ajouter les clés publiques aux known_hosts
cat ansible_automation.pub >> .ssh/authorized_keys
cat ansible_secure.pub >> .ssh/authorized_keys

3.8 Packer integration with Jenkins

Packer integration with Jenkins follows the same model as Vagrant. We create a Freestyle Job with a shell script which replaces the Vagrant call with a Packer call:

export AWS_AMI=$AMI_ID
export AWS_REGION=$REGION
export AWS_ACCESS_KEY_ID=$(grep "aws_access_key_id" ~/.aws/credentials | awk '{print $3}')
export AWS_SECRET_ACCESS_KEY=$(grep "secret_access_key" ~/.aws/credentials | awk '{print $3}')
export PATH=$PATH:/usr/local/bin

# Appel à Packer (remplace vagrant up)
cd ~/learning/packer/mongo/main/ubuntu && /usr/local/bin/packer build \
    -var "region=$REGION" \
    ubuntu-base.json.pkr.hcl

Difference with Vagrant: It is not necessary to pass the AMI_ID parameter because Packer uses a query filter in the Vagrantfile (or rather in the HCL template) to determine the source AMI.

Execution sequence:

1. Packer starts a temporary VM based on the selected AMI.
2. It provisions everything with shell and Ansible provisioners.
3. The Ansible provisioner installs JQ as a JSON parsing utility.
4. It creates the AMI.
5. It deletes the temporary VM to save cost.

After execution, you can navigate to the EC2 console under AMIs and filter on private AMIs to see the new freshly created image.

3.9 Introduction to HashiCorp Vault

What is Vault?

HashiCorp Vault is one of the most recognized and free secrets management solutions. Vault allows you to:

• Store and manage secrets according to simple policy rules.
• Integrate with many credential providers: relational databases, cloud IAM services, platforms like Okta, etc.
• Address secret sprawl — eliminate the danger of storing secrets in source code, configuration files, version control systems, documentation pages, etc.
• Periodic rotation of credentials.
• Provide an audit trail for access and use of credentials.

Bonus: Vault comes with high availability out-of-the-box. One can easily configure a Vault cluster and ensure that the installation is foolproof against infrastructure failures.

KV Store (Key-Value)

For the purposes of this course, we use Vault’s internal KV store, and more specifically version 2 of the KV store (KVv2). It is used to generate, protect and distribute user credentials for our VMs.

3.10 Vault integration with Vagrant and Packer

Essential Vault Commands

# Démarrer Vault en mode développement
vault server -dev

# Exporter l'adresse de Vault
export VAULT_ADDR='http://127.0.0.1:8200'

# Se connecter avec le root token
vault login hvs.uhD6rkakMSEx67YzeXRzkLyj

# Uploader une politique de mots de passe
vault write sys/policies/password/secure-pwd-policy policy=@secure_password_policy.hcl

# Générer un mot de passe selon la politique
export VAULT_GENERATED_PWD=$(vault read -format="json" sys/policies/password/secure-pwd-policy/generate | jq -r '.data.password')
echo $VAULT_GENERATED_PWD

# Stocker le mot de passe généré pour l'utilisateur vagrant
vault kv put -mount=secret vagrant pwd=$(echo $VAULT_GENERATED_PWD)

# Générer un autre mot de passe pour l'utilisateur packer
VAULT_GENERATED_PWD=$(vault read -format="json" sys/policies/password/secure-pwd-policy/generate | jq -r '.data.password')
vault kv put -mount=secret packer pwd=$(echo $VAULT_GENERATED_PWD)

# Lire des secrets depuis le KV store
vault kv get -mount=secret vagrant
vault kv get -mount=secret packer

# Récupérer et extraire la valeur brute du mot de passe
export VAGRANT_PWD=$(vault kv get -format="json" -mount=secret vagrant | jq -r '.data.data.pwd')
export PACKER_PWD=$(vault kv get -format="json" -mount=secret packer | jq -r '.data.data.pwd')

Strong Password Policy (HCL)

# 20 caractères de longueur
length = 20

rule "charset" {
  # Au moins une minuscule
  charset = "abcdefghijklmnopqrstuvwxyz"
  min-chars = 1
}
rule "charset" {
  # Au moins une majuscule
  charset = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
  min-chars = 1
}
rule "charset" {
  # Au moins un chiffre
  charset = "0123456789"
  min-chars = 1
}
rule "charset" {
  # Au moins un caractère spécial
  charset = "!@#$%^&*"
  min-chars = 1
}

Summary of module 3 — Good practices

• Always manually test individual components before integration.
• Use shell scripts rather than Vagrant/Packer specific Jenkins plugins.
• The general pattern: Choose a tool (Vagrant or Packer) → Build an artifact → Integrate with Jenkins via a job with an Execute Shell build step → Customize executions with script variables passed as Jenkins parameters.

4. Automating container builds

4.1 Container virtualization

Evolution of virtualization

Having guest VMs on hypervisors is good, but putting together a whole OS stack just to have one or two simple applications is too cumbersome and error prone. In the new world of rapid software development and dizzying system evolution, classic virtualization is no longer enough.

The solution: Docker

Docker creates a copy of the existing file system and packages this copy as an image. We can start several running copies of this image, called containers, whose calculation resource consumption parameters can be controlled (CPU, memory, disk, etc.).

The underlying technology: cgroups and namespaces

The basis of this technology dates back to the Linux kernel which included a mechanism called control groups (cgroups). Cgroups allow you to pass parameters to the kernel to configure the quantity of a given resource to dedicate to one or more processes.

Main parameter types in a cgroup:

4.2 Benefits of containerization

Cgroups rely on an even older mechanism in the Linux kernel: namespaces. Namespaces partition kernel resources such that one set of processes sees one set of resources, while another set of processes sees a different set of resources. The key: isolate processes from each other.

Three main benefits

1. Smaller blast radius Changes in one container do not propagate to others, limiting the impact of incidents.

2. Reduced security footprint Less attack surface per service. Each exposed service is limited to what it needs.

3. Microservices architecture Service isolation allows the emergence of the microservices architectural style, which gave birth to modern APIs (REST, GraphQL, etc.).

Orchestration (Control Plane)

Orchestration involves controlling how many containers of each type are needed at any given time, how crashing containers are replaced or restarted, etc.

Two main technologies:

• Kubernetes (by Google) — heavyweight, scales very well, ideal for 500+ containers.
• Docker Swarm — ships with Docker, promotes simplicity, ideal for less than 500 containers.

As-a-Service commercial offerings: AWS ECS, AWS Fargate, and their equivalents at Microsoft Azure and Google Cloud.

4.3 Containerize an Apache Web server

Installing Docker on macOS

# Installer Docker via Homebrew
brew install --cask docker

# Vérifier l'installation
docker version

# Installer Docker Machine
brew install docker-machine

Docker extensions for Visual Studio Code are also recommended to speed up working with Docker.

Base Dockerfile for Apache

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update 
RUN apt-get install apache2 -y
RUN apt-get install apache2-utils -y
RUN apt-get clean

COPY index.html /var/www/html/index.html

WORKDIR /usr/local/apache2/htdocs
VOLUME [ "/usr/local/apache2/htdocs" ]

EXPOSE 80
CMD ["apache2ctl", "-D", "FOREGROUND"]

Explanation of instructions:

• FROM ubuntu:20.04 — Ubuntu 20.04 base image.
• ARG DEBIAN_FRONTEND=noninteractive — Avoid interactive prompts during installations.
• RUN apt-get update && apt-get install apache2 -y — Updates and installs Apache.
• RUN apt-get clean — Cleans the apt cache to optimize the image size.
• COPY index.html /var/www/html/ — Copies a custom page to the container.
• VOLUME — Exposes the ephemeral virtual filesystem.
• EXPOSE 80 — Exposes the Apache server on port 80.
• CMD ["apache2ctl", "-D", "FOREGROUND"] — Starts Apache in foreground.

Dockerfile with ARGs (parameterized)

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

ARG PASSEDIN_VOLUME
ARG PASSEDIN_EXPOSE_PORT
ARG PASSEDIN_TARGET_ENV

RUN apt-get update 
RUN apt-get install apache2 -y
RUN apt-get install apache2-utils -y
RUN apt-get clean

COPY index.html /var/www/html/index.html

WORKDIR /usr/local/apache2/htdocs
VOLUME [ "$PASSEDIN_VOLUME" ]

EXPOSE $PASSEDIN_EXPOSE_PORT
CMD ["apache2ctl", "-D", "FOREGROUND"]

ARG allows values ​​to be passed during the docker build via the --build-arg option.

Essential Docker Commands

# Construire une image
docker build -t apache24web:latest .

# Démarrer un conteneur
docker run -d -p 80:80 apache24web:latest

# Lister les conteneurs en cours
docker ps

# Arrêter un conteneur
docker stop <container_id>

# Supprimer une image
docker image rm apache24web:latest

# Tagger et pousser vers Docker Hub
docker tag apache24web:latest moncompte/apache24web:latest
docker push moncompte/apache24web:latest

4.4 Containerize MySQL

Steps to containerize MySQL

1. Create an account on Docker Hub.
2. Assemble a Dockerfile to containerize MySQL.
3. Build a Docker image and push it to Docker Hub.
4. Start a MySQL container from the built image.

Parsing the MySQL Dockerfile (from official Ubuntu/MySQL)

Key points of the MySQL Dockerfile:

Important point: The VOLUME instruction for MySQL is critical. It allows MySQL data to persist on the host even if the container crashes or is shut down, ensuring data durability.

4.5 Integrate Docker with Jenkins

Jenkins script for Docker

#!/bin/bash

# Nettoyage du workspace avant le build (bonne pratique)
cd ~/learning/docker/apacheweb
rm -rf *

# Construire l'image Docker
docker build -t apache24web-jenkins-v1:latest \
    --build-arg PASSEDIN_VOLUME=$VOLUME \
    --build-arg PASSEDIN_EXPOSE_PORT=$EXPOSE_PORT \
    --build-arg PASSEDIN_TARGET_ENV=$TARGET_ENV \
    .

# Tagger et pousser vers Docker Hub
docker tag apache24web-jenkins-v1:latest moncompte/apache24web-jenkins-v1:latest
docker push moncompte/apache24web-jenkins-v1:latest

# Nettoyage de l'image locale
docker image rm apache24web-jenkins-v1:latest
docker image rm moncompte/apache24web-jenkins-v1:latest

Jenkins parameters for Docker job

Best practice: Always clean the workspace either before or after starting a build — preferably before, as you may want to examine artifacts after a build before removing them.

4.6 Docker Compose

What is Docker Compose?

Docker Compose is a tool that allows you to declare, execute and manage the lifecycle of multi-container applications. We can see him as the conductor of the containers. With Docker Compose, you can:

• Create and rebuild services.
• Start and stop services.
• Execute one-off commands on services.
• Stream service logs to analyzers like Splunk, Kibana, Logstash, etc.

Main Features

Variable passing mechanisms

1. .env files

Each file contains key-value pairs and represents an environment (dev, QA, staging, production). Useful for defining the same environment variables for several microservices (DRY principle).

Example .env file for dev:

PASSEDIN_MYSQL_PWD=MonMotDePasseSecurise!1
DB_DATABASE=ecommerce_db
DB_USER=app_user
DB_PASSWORD=AppPassword!2
APACHEWEB_PORT=8080
PASSEDIN_VOLUME=/usr/local/apache2/htdocs

2. Shell environment variables

Passed directly from the shell when calling docker-compose.

docker-compose.yml file

version: '3.3'
services:
  db:
    image: mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: "${PASSEDIN_MYSQL_PWD}"
      MYSQL_DATABASE: "${DB_DATABASE}"
      MYSQL_USER: "${DB_USER}"
      MYSQL_PASSWORD: "${DB_PASSWORD}"
    volumes:
      - db_data:/var/lib/mysql
    ports:
      - 3306:3306
  apacheweb:
    image: apache24web:latest
    volumes:
      - ./apacheweb:${PASSEDIN_VOLUME}
    ports:
      - ${APACHEWEB_PORT}:80
volumes:
  db_data:

Explanation:

• The db service uses the official MySQL image, with credentials passed via environment variables.
• The apacheweb service uses the previously built Apache image.
• Ports are dynamically configured via variables.
• db_data is a named volume that persists MySQL data.

4.7 Integrate Docker Compose with Jenkins

We create a Jenkins job that calls docker-compose with the appropriate configuration parameters:

#!/bin/bash

# Descendre dans le répertoire du projet
cd ~/learning/docker/dockercompose

# Démarrer les conteneurs avec Docker Compose
docker-compose up -d

# Les variables de l'environnement Jenkins sont automatiquement
# disponibles dans le docker-compose.yml via les variables d'environnement

Verification: After execution, we can verify that the two containers (Apache and MySQL) are running in Visual Studio Code (Docker extension) or via the command:

docker ps

4.8 Container Security with HashiCorp Vault

For the Docker job

Before implementing the build step, we manage the secrets in Vault:

# Exporter l'adresse de Vault
export VAULT_ADDR='http://127.0.0.1:8200'

# Démarrer Vault en mode développement (terminal séparé)
vault server -dev

# Dans un autre terminal : activer la méthode userpass
vault auth enable userpass

# Générer un mot de passe pour l'utilisateur apacheweb
vault read -format="json" sys/policies/password/secure-pwd-policy/generate | jq -r '.data.password'

# Stocker dans le KV store
vault kv put -mount=secret apacheweb pwd=<GENERATED_PASSWORD>

# Générer et stocker le mot de passe root MySQL
export MYSQL_ROOT_PWD=$(vault read -format="json" sys/policies/password/secure-pwd-policy/generate | jq -r '.data.password')
vault kv put -mount=secret mysql_root pwd=$MYSQL_ROOT_PWD

Jenkins script integrating Vault for Docker

#!/bin/bash

export VAULT_ADDR='http://127.0.0.1:8200'
echo "VAULT_TOKEN: $VAULT_TOKEN"
vault login $VAULT_TOKEN

# Récupérer le mot de passe depuis Vault
export APACHEWEB_PWD_JENKINS=$(vault kv get -format="json" -mount=secret apacheweb | jq -r '.data.data.pwd')
echo "APACHEWEB_PWD_JENKINS: $APACHEWEB_PWD_JENKINS"

cd ~/learning/docker/apacheweb
docker build -t apache24web-jenkins-v1:latest \
    --build-arg PASSEDIN_VOLUME=$VOLUME \
    --build-arg PASSEDIN_EXPOSE_PORT=$EXPOSE_PORT \
    --build-arg PASSEDIN_TARGET_ENV=$TARGET_ENV \
    --build-arg PASSED_IN_APACHEWEB_PWD=$APACHEWEB_PWD_JENKINS \
    -f Dockerfile.withargs-vault .

Dockerfile with Vault integration

FROM ubuntu:20.04

ARG DEBIAN_FRONTEND=noninteractive

ARG PASSEDIN_VOLUME
ARG PASSEDIN_EXPOSE_PORT
ARG PASSEDIN_TARGET_ENV
ARG PASSED_IN_APACHEWEB_PWD

ENV ENV_NAME=$PASSEDIN_TARGET_ENV

RUN apt-get update 
RUN apt-get install apache2 -y
RUN apt-get install apache2-utils -y
RUN apt-get clean

# Utiliser le secret passé - dans ce cas depuis HashiCorp Vault
RUN adduser apacheweb
RUN echo "apacheweb:$PASSED_IN_APACHEWEB_PWD" | chpasswd

COPY index.html /var/www/html/index.html

WORKDIR /usr/local/apache2/htdocs
VOLUME [ "$PASSEDIN_VOLUME" ]

EXPOSE $PASSEDIN_EXPOSE_PORT
CMD ["apache2ctl", "-D", "FOREGROUND"]

Important points:

• Password generated by Vault is passed as ARG to the build and used to create a user in the container.
• The VAULT_TOKEN parameter is added as a Jenkins parameter of type String.
• This approach ensures that secrets are never stored in the clear in the source code or in Jenkins.

5. Automating serverless builds

5.1 Serverless characteristics

Main Features

1. Low Barrier to Entry We do not manage servers — we only provide a block of logic in a function. All provisioning, deployment, configuration and maintenance are already taken care of. Natural result: a faster time to market.

2. Absence of servers (Hostlessness) We no longer have to manage the servers directly. Services do not expose standard performance metrics (CPU, memory, disk usage, etc.). This involves changing perspective and updating one’s skills for architecture optimization (e.g. monitoring and optimizing DynamoDB tables, DynamoDB read/write capabilities — AWS specific).

Security implications: the absence of servers implies different attack vectors. Some traditional security approaches will no longer be applicable.

3. Pay-per-use pricing model Extremely attractive to organizations launching new services or APIs, often well below the break-even point of always-on container services like AWS ECS or Fargate. As APIs mature, it’s natural to migrate to always-on Fargate containers.

4. Automatic scalability The cloud provider automatically manages scaling based on load.

Serverless vs Containers

5.2 Serverless business cases at Globomantics

New development at Globomantics: the company has slowed the pace of mergers and acquisitions in favor of lasting partnerships with selected companies. The partnerships manager announces 6 new potential partnerships per week, for an average of 8 new partnerships per month (compared to 4 per year for the last 2 years).

With this dramatic increase in joint ventures, the architecture team has approved the adoption of serverless technologies:

5.3 Serverless architecture

Lambda Use Case (AWS)

1. Server-Side Rendering Lambda Edge on AWS CloudFront processes requests closer to the end user.

2. Business APIs GET and POST functions behind AWS API Gateway to store and retrieve data in a backend (DynamoDB).

3. Pattern Lambda Fanout Asynchronous management of data changes enabling complex and decoupled processes affecting multiple business processes. Example: changing a person’s data triggers several workflows to recalculate their life insurance adjustments.

4. Event-Driven Architecture Many native cloud services can trigger Lambdas asynchronously: EventBridge, S3 events, DynamoDB streams, etc. Ideal for asynchronous tasks like sending confirmation emails on successful registrations.

5. Failure Handling Lambda can handle not only success scenarios, but also critical failure scenarios in a distributed application.

5.4 Create serverless components

The problem to be solved

Globomantics has just partnered with a company that has a very successful online store. During sales events, systems cannot handle the additional load. The corporate solution (license upgrade) would be too costly and unjustified — the e-commerce system never operates at more than 50% capacity outside of promotional periods.

The solution: Offload orders to a cloud-native serverless solution when an order placement threshold is reached (e.g. 1,000 orders per minute).

Solution architecture

[Clients] --> [API Gateway] --> [Lambda 1: Order Processor]
                                         |
                                    [SQS Queue]
                                         |
[CloudWatch Scheduler] --> [Lambda 2: Order Aggregator] --> [S3 Bucket]
                                                             (CSV file)

Components:

• AWS API Gateway — Exposes a RESTful API that calls the first Lambda.
• Lambda 1 (orders_lambda1.py) — Receives orders and places them in an SQS queue.
• AWS SQS — Command queue (Standard Queue + Dead-letter Queue).
• AWS CloudWatch Event — Acts as a cron scheduler (every 2 minutes).
• Lambda 2 (orders_lambda_aggregator.py) — Gets orders from the queue and consolidates them into a CSV file in S3.
• AWS S3 — Storing the consolidated orders CSV file.

Lambda Code 1 — Receiving and queuing commands

import boto3
import logging
import json

logger = logging.getLogger()
logger.setLevel(logging.INFO)

sqs_client = boto3.client('sqs')

def lambda_handler(event, context):
    queue_url = 'https://sqs.us-east-1.amazonaws.com/239136941756/standard_orders_queue'
    
    response = sqs_client.send_message(
        QueueUrl=queue_url,
        MessageAttributes={
            'cart_id_type': {
                'DataType': 'String',
                'StringValue': 'pre-generated'
            },
            'order_submission_type': {
                'DataType': 'String',
                'StringValue': 'online'
            },
            'retrial_count': {
                'DataType': 'Number',
                'StringValue': '4'
            }
        },
        # Extraire uniquement le body (infos de commande), ignorer les headers
        MessageBody=(json.dumps(event["body"]))
    )

    print('\nSID: {}'.format(response['MessageId']))
    print(event)
    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json',
            'Access-Control-Allow-Origin': '*'
        },
        'body': json.dumps({'success': True}),
        "isBase64Encoded": False
    }

Lambda Code 2 — Order aggregation to S3

import boto3
import logging
import json

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    sqs_client = boto3.client('sqs')
    queue_url = "https://sqs.us-east-1.amazonaws.com/239136941756/standard_orders_queue"
    
    received_response = sqs_client.receive_message(
        QueueUrl=queue_url,
        AttributeNames=['SentTimestamp'],
        MaxNumberOfMessages=1,
        MessageAttributeNames=['All'],
        VisibilityTimeout=30,
        WaitTimeSeconds=0
    )
    
    message = received_response['Messages'][0]
    receipt_handle = message['ReceiptHandle']
    logger.info("RID: {}".format(message['MessageId']))
    logger.info("Message Body: {}\n".format(message['Body']))
    
    dequeuedOrder = message['Body']
    encodedDequeuedOrder = dequeuedOrder.encode("utf-8")

    bucket_name = "lambda-demo-terraform-gs-2022-10-22"
    file_name = "offloadedOrders.csv"
    s3_path = file_name

    s3 = boto3.resource("s3")
    s3.Bucket(bucket_name).put_object(Key=s3_path, Body=encodedDequeuedOrder)
    
    return "Request Handled Successfully :)"

Example payload of an order

{
    "cartId": "78912333",
    "items": [
        {
            "name": "Lego - Batmobile Pro, 234 pieces",
            "quantity": 7,
            "cart_price": 37.99,
            "promo": "10 percent off"
        },
        {
            "name": "Lego - Batmobile Pro, 234 pieces",
            "quantity": 7,
            "cart_price": 37.99,
            "promo": "10 percent off"
        }
    ],
    "timestamp": "20191019-130532-PST"
}

5.5 Automate creation with Terraform

Terraform and API Gateway abstractions

The AWS API Gateway works with the following abstractions:

• The gateway instance itself, which contains one or more APIs.
• Each API has one or more endpoints.
• Each endpoint has one or more methods.
• An integration which in our case will point to the first Lambda function.
• A role (collection of permissions) to call the Lambda service.

Analogy: The API gateway is like the controller + everything in front of it (filters, chains) in a framework like Spring. The Lambda function is like the business service that the controller calls.

Terraform file: api_gateway.tf

############################
##### PROCESSOR LAMBDA #####
############################

resource "aws_iam_role" "iam_for_lambda" {
  name = "iam_for_lambda"
  description = "Receives offloaded orders and post them to a SQS"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

resource "aws_lambda_function" "orders_processor_lambda_gs1" {
  function_name = "orders_processor_lambda_gs1"
  description = "Receives offloaded orders and post them to a SQS"
  role = aws_iam_role.orders_lambda_role_gs1.arn

  s3_bucket = "lambda-demo-terraform-gs-2022-10-22"
  s3_key = "orders_fx.zip"

  handler = "orders_lambda1.lambda_handler"
  runtime = "python3.7"

  layers = ["${aws_lambda_layer_version.paramiko_lambda_layer.arn}"]

  environment {
    variables = {
      deployment_date = "2019-10-23"
    }
  }
}

resource "aws_iam_role" "orders_lambda_role_gs1" {
  name = "orders_lambda_role_gs1"

  assume_role_policy = <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

  tags = {
    owner = "george-smith"
  }
}

resource "aws_iam_role_policy" "orders_lambda_policy_gs1" {
  role = aws_iam_role.orders_lambda_role_gs1.id
  policy = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:GetBucketLocation"
            ],
            "Resource": "*"
        }
    ]
}
POLICY
}

Terraform file: sqs.tf

// SQS Dead-letter Queue (file de lettres mortes)
resource "aws_sqs_queue" "my_dead_letter_queue" {
  name = "my_dead_letter_queue"
  tags = {
    queue_type = "order_offloading_queue_dead"
  }
}

resource "aws_sqs_queue_policy" "my_dead_letter_queue_policy" {
  queue_url = aws_sqs_queue.my_dead_letter_queue.id
  policy = <<POLICY
{
  "Version": "2012-10-17",
  "Id": "sqspolicy",
  "Statement": [
    {
      "Sid": "First",
      "Effect": "Allow",
      "Principal": "*",
      "Action": ["sqs:SendMessage","sqs:ReceiveMessage","sqs:DeleteMessage"],
      "Resource": "${aws_sqs_queue.my_dead_letter_queue.arn}"
    }
  ]
}
POLICY
}

// SQS Standard Queue avec délai, taille max, rétention et politique de redrive
resource "aws_sqs_queue" "standard_orders_queue" {
  name = "standard_orders_queue"
  visibility_timeout_seconds = var.visibility_timeout_seconds
  delay_seconds = var.delay_seconds
  max_message_size = 2048
  message_retention_seconds = 86400 // 24 heures
  receive_wait_time_seconds = var.receive_wait_time_seconds
  redrive_policy = "{\"deadLetterTargetArn\":\"${aws_sqs_queue.my_dead_letter_queue.arn}\",\"maxReceiveCount\":4}"

  tags = {
    queue_type = "order_offloading_standard_queue"
  }
}

SQS variables

variable "visibility_timeout_seconds" {
  description = "The visibility timeout for the queue. An integer from 0 to 43200 (12 hours)"
  type        = number
  default     = 10
}

variable "delay_seconds" {
  description = "The time in seconds that the delivery of all messages will be delayed (0 to 900)"
  type        = number
  default     = 30
}

variable "receive_wait_time_seconds" {
  description = "How long should the queue delay the message"
  type        = number
  default     = 30
}

Terraform file: providers.tf

provider "aws" {
  skip_credentials_validation = true
  skip_requesting_account_id = true

  shared_config_files      = ["/Users/gsyyl/.aws/config"]
  shared_credentials_files = ["/Users/gsyyl/.aws/credentials"]
  profile                  = "default"
}

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = ">= 3.0.0"
    }
  }
}

Terraform file: Bucket S3

resource "random_string" "s3-ext" {
  length = 5
  special = false
  upper = false
}

resource "aws_s3_bucket" "offloaded-orders-s3-gs1" {
    bucket = "offloaded-orders-s3-gs1-${random_string.s3-ext.result}"
}

Outputs Terraform

output "lambda_order_processor_invoke_arn" {
  description = "The invoke ARN of the Lambda"
  value = element(
    concat(
      aws_lambda_function.orders_processor_lambda_gs1.*.invoke_arn,
      [""],
    ),
    0,
  )
}

output "orders_standard_queue_id" {
  description = "The URL for the created Amazon SQS queue"
  value = element(
    concat(
      aws_sqs_queue.standard_orders_queue.*.id,
      [""],
    ),
    0,
  )
}

Essential Terraform Commands

# Initialiser Terraform (télécharger les providers)
terraform init

# Planifier les changements (dry run)
terraform plan

# Appliquer les changements
terraform apply

# Détruire l'infrastructure
terraform destroy

# Créer le bucket S3 avec AWS CLI (commande isolée dans un job séparé)
aws s3api create-bucket \
    --bucket=lambda-demo-terraform-gs-2022-10-22 \
    --region=us-east-1

5.6 Integrate Terraform with Jenkins

Jenkins job architecture

Two Jenkins jobs are required:

1. Job 1: Create S3 bucket — Isolated in a separate job to be executed only once. The bucket will persist across serverless architecture deployments and updates.

2. Job 2: Deploy serverless environment — Can be run repeatedly to deploy and remove the architecture.

Job 1: Create the S3 bucket

Job configuration:

• Type: Freestyle project
• Build kept: 5
• Build type: Parameterized build
• Parameter: BUCKET_NAME (String) — The name of the S3 bucket

Build step script:

#!/bin/bash

# Lire les clés AWS depuis le fichier de credentials
export AWS_ACCESS_KEY_ID=$(grep "aws_access_key_id" ~/.aws/credentials | awk '{print $3}')
export AWS_SECRET_ACCESS_KEY=$(grep "secret_access_key" ~/.aws/credentials | awk '{print $3}')
export PATH=$PATH:/usr/local/bin

# Créer le bucket S3
aws s3api create-bucket \
    --bucket=$BUCKET_NAME \
    --region=us-east-1

Important: The bucket name passed via the parameter must match the value in the Terraform files that reference the bucket by name.

Job 2: Deploy the serverless environment

The job executes Terraform commands to deploy or remove the infrastructure:

#!/bin/bash

export AWS_ACCESS_KEY_ID=$(grep "aws_access_key_id" ~/.aws/credentials | awk '{print $3}')
export AWS_SECRET_ACCESS_KEY=$(grep "secret_access_key" ~/.aws/credentials | awk '{print $3}')
export PATH=$PATH:/usr/local/bin

cd ~/learning/terraform/offload_solution

# Initialiser et appliquer Terraform
terraform init
terraform apply -auto-approve

Terraform job parameters

5.7 Security with HashiCorp Vault (serverless)

Vault use cases for Lambda

The list of use cases is very extensive:

• Basic Secrets Management — AWS access keys, API tokens.
• Token generation and distribution — JWT tokens for APIs.
• Integration with special secrets engines — Databases, PKI, etc.
• Vault Deployment in the Cloud — Lambda functions can directly access Vault via the Vault API.

Store and retrieve AWS keys from Vault

# Stocker la clé d'accès AWS
vault kv put -mount=secret aws_credentials \
    access_key=$AWS_ACCESS_KEY_ID \
    secret_key=$AWS_SECRET_ACCESS_KEY

# Récupérer les clés depuis Vault dans le script Jenkins
export VAULT_ADDR='http://127.0.0.1:8200'
vault login $VAULT_TOKEN

export AWS_ACCESS_KEY_ID=$(vault kv get -format="json" -mount=secret aws_credentials | jq -r '.data.data.access_key')
export AWS_SECRET_ACCESS_KEY=$(vault kv get -format="json" -mount=secret aws_credentials | jq -r '.data.data.secret_key')

Generate tokens with Vault for API

For APIs, secret management mainly concerns tokens. Demonstration of generating tokens with Vault and passing them to the API:

# Générer un token Vault
vault token create -policy=default -ttl=1h

# Passer le token comme paramètre Jenkins
# Paramètre : VAULT_TOKEN = <token_généré>

Scenario: When the e-commerce system retrieves offloaded orders for processing, it must present a valid token when requesting it — a token that it can obtain from Vault.

Advantages:

• Secrets are never stored locally on the production workstation.
• Avoids the secret sprawl issue mentioned earlier.
• Complete audit trail of access to secrets.

5.8 Best Practices and Course Summary

What you have accomplished

During this course, you learned to:

1. Build VMs and developer environments with Vagrant.
2. Build cloud AMIs with HashiCorp Packer.
3. Containerize applications with Docker and Docker Compose.
4. Create complex serverless environments with HashiCorp Terraform.
5. Integrate all these technologies into your CI/CD pipeline with Jenkins.
6. Layer security with HashiCorp Vault on all the automation implemented.

All this in preparation for production-quality environments.

The 4 best practices

1. Always test individual components before moving forward Before pushing forward with your integration efforts, first test the individual components of your system. This saves a lot of time and a lot of headaches. This is especially true for shell scripts before integrating them into Jenkins.

2. Understand the integration mechanism in advance Always do prior research and understand the integration mechanism:

• What variables are passed?
• What files are placed on your file system?
• Are databases involved?
• Are API calls required?
• Are there any data transformations?

Knowing is half the battle.

3. Have a minimum viable test plan Many management decisions are based on “prove it to me” or “prove it to us” demonstrations. It’s also a good way to boost your own confidence. It is difficult to convince others before you yourself believe that your system can work and solve the problems in question.

4. There is always a solution No matter how complex the problem, how incompatible the system components or subsystems you need to orchestrate, believe that there is always a solution.

6. Appendix: Summary of key commands

Vagrant

vagrant up                # Démarrer la VM
vagrant ssh               # Se connecter en SSH
vagrant halt              # Arrêter la VM
vagrant destroy           # Supprimer la VM
vagrant status            # Voir l'état
vagrant plugin install vagrant-aws  # Installer le plugin AWS
vagrant list              # Lister les plugins installés

Packer

packer build ubuntu-base.json.pkr.hcl        # Construire l'image
packer build -var "region=us-east-1" ...     # Passer des variables
packer validate ubuntu-base.json.pkr.hcl     # Valider le template

Docker

docker build -t nom:tag .                    # Construire une image
docker run -d -p 8080:80 nom:tag             # Démarrer un conteneur
docker ps                                    # Lister les conteneurs
docker stop <id>                             # Arrêter un conteneur
docker image rm nom:tag                      # Supprimer une image
docker tag nom:tag repo/nom:tag              # Tagger une image
docker push repo/nom:tag                     # Pousser vers un registry
docker-compose up -d                         # Démarrer les services
docker-compose down                          # Arrêter les services

Terraform

terraform init                               # Initialiser
terraform plan                               # Prévisualiser les changements
terraform apply                              # Appliquer
terraform apply -auto-approve                # Appliquer sans confirmation
terraform destroy                            # Détruire l'infrastructure
terraform output                             # Afficher les outputs

HashiCorp Vault

vault server -dev                            # Démarrer en mode dev
vault login <token>                          # Se connecter
vault kv put -mount=secret key val=value     # Stocker un secret
vault kv get -mount=secret key               # Lire un secret
vault kv get -format="json" -mount=secret key | jq -r '.data.data.val'  # Extraire la valeur
vault write sys/policies/password/nom policy=@fichier.hcl  # Uploader une politique
vault read -format="json" sys/policies/password/nom/generate | jq -r '.data.password'  # Générer un mot de passe
vault auth enable userpass                   # Activer la méthode userpass
vault token create -policy=default -ttl=1h  # Créer un token

AWS CLI

aws s3api create-bucket --bucket=nom --region=us-east-1  # Créer un bucket
aws ec2 describe-amis --owners self          # Lister vos AMIs
aws lambda list-functions                    # Lister les Lambdas
aws sqs list-queues                          # Lister les queues SQS

7. Appendix: Additional resources and courses

Search Terms

jenkins · automate · artifact · builds · security · ci/cd · git · devops · vault · docker · terraform · packer · vagrant · script · serverless · aws · job · architecture · commands · dockerfile · hashicorp · integration · lambda · apache