Java SE Performance with JMeter

Level: Intermediate

Table of Contents

1. Course Overview
2. Introduction to Performance Testing with JMeter

• 2.1 Most common performance issues
• 2.2 Installation of JMeter
• 2.3 The demo application
• 2.4 Creating a simple JMeter script
• 2.5 The performance testing process

3. Creating the JMeter script for the application

• 3.1 Configuration of default values (HTTP Request Defaults)
• 3.2 Multiple users with CSV Data Set Config
• 3.3 JWT token extraction with JSON JMESPath Extractor
• 3.4 Transaction Controller and Request Header Manager
• 3.5 Random Variable
• 3.6 Response assertions
• 3.7 Weighted Execution Paths
• 3.8 JMeter Variables vs JMeter Properties
• 3.9 Running in non-GUI mode

4. Creating test data with JMeter

• 4.1 JDBC Configuration
• 4.2 Creating tables with JDBC Request
• 4.3 Insert users (setUp Thread Group)
• 4.4 Insertion of employees (While Controller + Loop Controller)
• 4.5 Database index

5. Scalability tests with JMeter

• 5.1 Scalability concepts
• 5.2 SSHMon Samples Collector Plugin
• 5.3 Establish scalability baseline
• 5.4 Push CPU to 100% (Scalability Test)
• 5.5 Analysis of Summary Report results

6. Detection of persistence problems

• 6.1 Glowroot: open source APM agent
• 6.2 Analysis of slow queries in Glowroot
• 6.3 Thread dumps with jps and jstack
• 6.4 Thread dump analysis
• 6.5 Adding an SQL index to correct table scans

7. Detecting memory problems

• 7.1 Memory monitoring with VisualVM
• 7.2 Calculation of Pause Time with jstat
• 7.3 Analysis of Garbage Collection logs with GCViewer
• 7.4 Class histograms with jmap and jcmd
• 7.5 Analysis of heap dumps with Eclipse MAT
• 7.6 Memory leak detection with JFR and JDK Mission Control

8. Structure of demo files
9. Command Reference
10. Useful links

1. Course Overview

This course is hands-on training on using JMeter and other open source tools to test the performance of Java applications. Most tutorials and courses on JMeter only show how to create a performance test script without going any further. This course goes beyond that.

Main topics covered

• How to build a JMeter script that models the actual usage of your application
• How to use JMeter to populate your database with thousands of records
• How to know if your application is scalable
• How to detect persistence issues (slow SQL queries, table scans, locks)
• How to detect memory problems (memory leaks, leaks, GC overhead)

Prerequisites

• Experience in Java application development
• Knowledge of basic concepts of web applications or REST APIs (HTTP, JSON)
• Basic knowledge of JMeter (recommended but not required — a JMeter 5 Getting Started course is available on Pluralsight)

Tools used in this course

2. Introduction to Performance Testing with JMeter

2.1 Most Common Performance Issues

For most applications, to achieve a good level of performance, you need to know the most common problems that can affect them:

1. Slow SQL queries: either because of a problem with the database, or because of a problem with the queries themselves (table scans, absence of indexes)
2. Chatty network calls: for example, performing many database queries instead of using a larger query to retrieve the necessary information
3. Memory leaks: objects retained in memory over time
4. Configuration issues: for example, not properly sizing a server’s thread pool
5. Concurrency issues: for example, when two or more threads block each other (deadlocks, lock contention)

Most of the time, the cause of these issues is inefficient code due to bad practices, which is easy to fix once the problem is identified.

2.2 Installing JMeter

Download and install

1. Go to jmeter.apache.org
2. In the left menu, go to the Download Releases section
3. Download the latest version binary distribution (.tgz or .zip)
4. Unzip the file and move the apache-jmeter directory to the desired location

Prerequisites: Java 8 or higher (JDK version preferred)

Starting the GUI

# Windows
bin/jmeter.bat

# Linux / Mac
bin/jmeter.sh

Installing the Plugin Manager

The operation of JMeter can be extended by plugins. The site jmeter-plugins.org lists popular plugins. Each plugin is packaged as a .jar file.

Plugin Manager installation procedure:

1. Download the jmeter-plugins-manager-X.X.jar file from the jmeter-plugins.org homepage
2. Move it to the lib/ext directory of the JMeter installation
3. Start or restart JMeter
4. A butterfly icon appears in the upper right corner — this is the Plugin Manager

Custom Thread Groups Plugin

This plugin adds Thread Groups allowing you to specify the number of users more flexibly. To install it:

1. Open the Plugin Manager (butterfly icon)
2. Go to the Available Plugins tab
3. Select Custom Thread Groups
4. Click on Apply Changes and Restart JMeter

Once installed, a right-click on the Test Plan → Add → Threads (Users) element will display the new Thread Groups, notably the Concurrency Thread Group.

2.3 The demo application

The demo application is a REST API built with Spring Boot, using H2 as a server-mode database and JSON Web Tokens (JWT) as an authentication method.

Project structure (02/demos/)

demos/
├── src/
│   └── main/
│       ├── java/com/pluralsight/api/
│       │   ├── ApiDemoApplication.java
│       │   ├── UserGenerator.java
│       │   ├── config/
│       │   │   ├── ApiExceptionHandler.java
│       │   │   └── WebSecurityConfig.java
│       │   ├── controller/
│       │   │   ├── ApiController.java
│       │   │   └── AuthController.java
│       │   ├── exception/
│       │   │   └── EmployeeNotFoundException.java
│       │   ├── filter/
│       │   │   └── JwtRequestFilter.java
│       │   ├── model/
│       │   │   ├── Employee.java
│       │   │   └── User.java
│       │   ├── repository/
│       │   │   ├── EmployeeRepository.java
│       │   │   └── UserRepository.java
│       │   ├── service/
│       │   │   └── JwtUserDetailsService.java
│       │   └── util/
│       │       └── JwtTokenUtil.java
│       └── resources/
│           └── application.properties
├── db/
├── pom.xml
├── startApp.bat / startApp.sh
└── startDB.bat / startDB.sh

pom.xml — Maven dependencies

<project>
  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.7.3</version>
  </parent>
  <groupId>com.pluralsight</groupId>
  <artifactId>api</artifactId>
  <version>0.0.1-SNAPSHOT</version>

  <properties>
    <java.version>17</java.version>
  </properties>

  <dependencies>
    <!-- Spring Boot -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-jpa</artifactId>
    </dependency>
    <dependency>
      <groupId>org.hibernate.orm</groupId>
      <artifactId>hibernate-core</artifactId>
      <version>6.1.3.Final</version>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-security</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <!-- JWT (JJWT) -->
    <dependency>
      <groupId>io.jsonwebtoken</groupId>
      <artifactId>jjwt-api</artifactId>
      <version>0.11.5</version>
    </dependency>
    <dependency>
      <groupId>io.jsonwebtoken</groupId>
      <artifactId>jjwt-impl</artifactId>
      <version>0.11.5</version>
      <scope>runtime</scope>
    </dependency>
    <dependency>
      <groupId>io.jsonwebtoken</groupId>
      <artifactId>jjwt-jackson</artifactId>
      <version>0.11.5</version>
      <scope>runtime</scope>
    </dependency>
    <!-- Commons -->
    <dependency>
      <groupId>org.apache.commons</groupId>
      <artifactId>commons-lang3</artifactId>
      <version>3.12.0</version>
    </dependency>
    <!-- H2 Database -->
    <dependency>
      <groupId>com.h2database</groupId>
      <artifactId>h2</artifactId>
      <version>2.1.214</version>
      <scope>runtime</scope>
    </dependency>
  </dependencies>
</project>

application.properties

# H2 en mode serveur TCP
spring.datasource.url=jdbc:h2:tcp://localhost/test;AUTO_SERVER=TRUE
spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=

# HikariCP — Connection Pool
spring.datasource.hikari.pool-name=MyHikariPool
spring.datasource.hikari.connection-timeout=50000
spring.datasource.hikari.max-lifetime=900000
spring.datasource.hikari.maximum-pool-size=10
spring.datasource.hikari.minimum-idle=10
spring.datasource.hikari.connection-test-query=select 1

# Optimisations HikariCP
spring.datasource.hikari.data-source-properties.cachePrepStmts=true
spring.datasource.hikari.data-source-properties.prepStmtCacheSize=250
spring.datasource.hikari.data-source-properties.prepStmtCacheSqlLimit=2048
spring.datasource.hikari.data-source-properties.useServerPrepStmts=true
spring.datasource.hikari.data-source-properties.rewriteBatchedStatements=true

# JPA / Hibernate
spring.jpa.properties.hibernate.show_sql=false
spring.jpa.properties.hibernate.format_sql=true
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.H2Dialect
spring.jpa.open-in-view=false

# Logging
logging.level.org.hibernate.SQL=INFO
logging.level.com.zaxxer.hikari.HikariConfig=INFO

server.port=8081

Model Employee.java

@Entity
@Table(name = "employees")
public class Employee {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;

    @Column(name = "first_name")
    private String firstName;

    @Column(name = "last_name")
    private String lastName;

    private BigDecimal salary;

    // constructeurs, getters, setters, equals, hashCode
}

ApiController.java — REST endpoints

@RestController
@RequestMapping("/api")
public class ApiController {
    private final EmployeeRepository employeeRepository;

    @GetMapping(value = "/employees/{id}")
    public ResponseEntity<Employee> findEmployeeById(@PathVariable("id") long id) {
        Employee employee = employeeRepository.findById(id)
                .orElseThrow(() -> new EmployeeNotFoundException("Employee not found with ID " + id));
        return ResponseEntity.ok().body(employee);
    }

    @GetMapping(value = "/employees")
    public ResponseEntity<Iterable<Employee>> getEmployees(
            @RequestParam(required = false) String start,
            @RequestParam(required = false, defaultValue = "0") int page,
            @RequestParam(required = false, defaultValue = "20") int size) {
        Sort sortByName = Sort.by("firstName");
        Pageable paging = PageRequest.of(page, size, sortByName);
        Iterable<Employee> list;

        if (StringUtils.isNotBlank(start)) {
            list = employeeRepository.findByFirstNameStartingWith(start, paging);
        } else {
            list = employeeRepository.findAll(paging);
        }
        return ResponseEntity.ok().body(list);
    }
}

EmployeeRepository.java

public interface EmployeeRepository extends JpaRepository<Employee, Long> {
    Page<Employee> findByFirstNameStartingWith(String name, Pageable pageable);
}

AuthController.java — JWT authentication

@RestController
@RequestMapping("/auth")
public class AuthController {
    @PostMapping("/login")
    public ResponseEntity<?> loginUser(@RequestBody User user) {
        Map<String, Object> responseMap = new HashMap<>();
        try {
            Authentication auth = authenticationManager.authenticate(
                    new UsernamePasswordAuthenticationToken(user.getUsername(), user.getPassword()));
            if (auth.isAuthenticated()) {
                UserDetails userDetails = userDetailsService.loadUserByUsername(user.getUsername());
                String token = jwtTokenUtil.generateToken(userDetails);
                responseMap.put("message", "Logged In");
                responseMap.put("token", token);
                return ResponseEntity.ok(responseMap);
            } else {
                responseMap.put("message", "Invalid Credentials");
                return ResponseEntity.status(401).body(responseMap);
            }
        } catch (BadCredentialsException e) {
            responseMap.put("message", "Invalid Credentials");
            return ResponseEntity.status(401).body(responseMap);
        }
    }
}

JwtTokenUtil.java — Token generation and validation

@Component
public class JwtTokenUtil implements Serializable {
    public static final long JWT_TOKEN_VALIDITY = 5 * 60 * 60; // 5 heures
    private Key key = Keys.secretKeyFor(SignatureAlgorithm.HS256);

    public String generateToken(UserDetails userDetails) {
        Map<String, Object> claims = new HashMap<>();
        return Jwts.builder()
                .setClaims(claims)
                .setSubject(userDetails.getUsername())
                .setIssuedAt(new Date(System.currentTimeMillis()))
                .setExpiration(new Date(System.currentTimeMillis() + JWT_TOKEN_VALIDITY * 1000))
                .signWith(key)
                .compact();
    }

    public Boolean validateToken(String token, UserDetails userDetails) {
        final String username = getUsernameFromToken(token);
        return (username.equals(userDetails.getUsername()) && !isTokenExpired(token));
    }
}

JwtRequestFilter.java — Security filter

@Component
public class JwtRequestFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                    HttpServletResponse response,
                                    FilterChain chain) throws ServletException, IOException {
        final String requestTokenHeader = request.getHeader("Authorization");
        if (requestTokenHeader != null && StringUtils.startsWith(requestTokenHeader, "Bearer ")) {
            String jwtToken = requestTokenHeader.substring(7);
            String username = tokenUtil.getUsernameFromToken(jwtToken);
            if (StringUtils.isNotEmpty(username) && null == SecurityContextHolder.getContext().getAuthentication()) {
                UserDetails userDetails = userDetailsService.loadUserByUsername(username);
                if (tokenUtil.validateToken(jwtToken, userDetails)) {
                    UsernamePasswordAuthenticationToken authToken = new UsernamePasswordAuthenticationToken(
                            userDetails, null, userDetails.getAuthorities());
                    authToken.setDetails(new WebAuthenticationDetailsSource().buildDetails(request));
                    SecurityContextHolder.getContext().setAuthentication(authToken);
                }
            }
        }
        chain.doFilter(request, response);
    }
}

Starting application and database

# Démarrer la base de données H2 en mode serveur TCP (Windows)
startDB.bat
# Commande équivalente :
java -Xmx1g -cp db/h2-2.1.214.jar org.h2.tools.Server -tcp -web -baseDir ./db/data

# Démarrer l'application (Windows)
startApp.bat
# Commande équivalente :
java -jar target/api-0.0.1-SNAPSHOT.jar --server.port=8081

# Compiler l'application
./mvnw clean package

Console H2

Available from: http://localhost:8082 JDBC URL: jdbc:h2:tcp://localhost/test User: sa | Password: (empty)

The application has two tables: EMPLOYEES (employee information) and USERS (application users).

Test with curl

# Obtenir un token JWT
curl -X POST -H "Content-Type: application/json" \
  -d '{"username":"user01","password":"user01"}' \
  http://localhost:8081/auth/login

# Récupérer tous les employés (avec token)
curl -H "Content-Type: application/json" \
  -H "Authorization: Bearer <TOKEN>" \
  http://localhost:8081/api/employees

# Récupérer un employé par ID
curl -H "Authorization: Bearer <TOKEN>" \
  http://localhost:8081/api/employees/1

There are initially 10 users in the database: user01 to user10. The password is the same as the username.

2.4 Creating a simple JMeter script

A basic JMeter script is created to introduce the concepts and show the general picture of performance testing.

Main elements of a JMeter Test Plan:

2.5 The performance testing process

The performance test must follow an iterative process:

Définir les objectifs
       ↓
Créer le script JMeter
       ↓
Peupler la base de données (bonne taille)
       ↓
Exécuter le test
       ↓
Analyser les résultats (logs, résultats)
       ↓
Identifier les problèmes
       ↓
Modifier l'application ou la JVM
       ↓
Ré-exécuter le test
       ↓
(répéter jusqu'à satisfaction)

Best practices

• The performance test must be carried out after the functional tests (no major bugs)
• Test environment must be equivalent to production (same hardware, software, configuration)
• Developers can also test on their own machines to catch issues early
• 3 to 10 users is often enough to detect performance issues
• As long as JMeter’s RAM and CPU usage remains relatively low (~20%), you can run the tests on the same machine as the application
• It is essential to have a good size database (tens or hundreds of thousands of records)

Tools used in this course (open source and free)

• Glowroot — Detecting persistence issues
• jcmd, jstat, jps — Dynamic JVM information
• jstack, jmap — Thread dumps and heap histograms
• VisualVM — Graphics memory profiling
• GCViewer — Analysis of GC logs
• Eclipse MAT — Analysis of heap dumps
• JFR / JMC — Java Flight Recorder and JDK Mission Control

3. Creating the JMeter script for the application

This module explains how to create a complete and professional JMeter script for the demo application, following best practices.

3.1 Configuring default values ​​(HTTP Request Defaults)

When multiple HTTP requests share the same Server Name and Port values, it is best to use the HTTP Request Defaults element which sets default values ​​for all HTTP requests.

Procedure:

1. Right-click on the Test Plan → Add → Config Element → HTTP Request Defaults
2. Move this element to the top so that it applies to all child elements
3. Set Server name or IP = localhost, Port Number = 8081
4. Remove these values from individual HTTP Request elements

3.2 Multiple Users with CSV Data Set Config

To simulate multiple users, JMeter uses a CSV file containing credentials.

Generating the CSV file with UserGenerator.java

public class UserGenerator {
    private static final String outDir = "C:\\Users\\pc\\Desktop\\";
    private static final String file = "users.csv";
    private static final int numberOfUsersToGenerate = 10;

    public static void main(String[] args) throws IOException {
        String prefix = "user";
        BCryptPasswordEncoder encoder = new BCryptPasswordEncoder();
        StringBuilder content = new StringBuilder();

        for (int i = 1; i <= numberOfUsersToGenerate; i++) {
            String user = prefix + String.format("%02d", i);
            content.append(String.format("%s,%s,%s\n",
                    user,
                    user,
                    encoder.encode(user)));
        }
        // Écriture du fichier
        File outFile = new File(outDir + file);
        FileWriter writer = new FileWriter(outFile, false);
        writer.write(content.toString());
        writer.close();
    }
}

The generated CSV file contains three columns: username,password,encryptedPassword. The username and password columns are used by JMeter for authentication.

Configuring CSV Data Set Config in JMeter

1. Right-click on the Thread Group → Add → Config Element → CSV Data Set Config
2. Set path to CSV file
3. Variables names: username,password (corresponding to the CSV columns)
4. Delimiter: ,
5. Sharing mode: All threads

3.3 Extracting the JWT token with JSON JMESPath Extractor

After the login request, you must extract the JWT token from the response to use it in subsequent requests.

Procedure:

1. Right-click on the request Login HTTP Request → Add → Post Processors → JSON JMESPath Extractor
2. Names of created variables: token
3. JMESPath expressions: token

Alternative: Use the JSON Extractor with the JSONPath expression: $.token

The token is now stored in the JMeter variable ${token}.

3.4 Transaction Controller and Request Header Manager

Transaction Controller

The Transaction Controller groups several requests into a single logical transaction, which allows the total time of all requests to be measured.

1. Right-click on the Thread Group → Add → Logic Controller → Transaction Controller
2. Check Generate parent sample to have a single line in the reports

HTTP Header Manager

To pass the JWT token in API requests, add an HTTP Header Manager:

1. Right-click on the Transaction Controller → Add → Config Element → HTTP Header Manager
2. Add a header: Authorization = Bearer ${token}

3.5 Random variables (Random Variable)

To simulate requests to random employees, use the Random Variable element:

1. Right-click on the Transaction Controller (or Test Plan) → Add → Config Element → Random Variable
2. Variable Name: employeeID
3. Minimum Value: 1
4. Maximum Value: ${employee_count} (variable defined in the Test Plan)
5. Per Thread(User): False (generator shared between all threads)

Usage in HTTP request: /api/employees/${employeeID}

User-defined variables in the Test Plan

In the properties of the Test Plan → section User Defined Variables:

3.6 Response assertions

To validate the responses, add a Response Assertion:

1. Right-click on the query → Add → Assertions → Response Assertion
2. Field to Test: Response Code
3. Pattern Matching Rules: Equals
4. Patterns to Test: 200

3.7 Weighted Execution Paths

In a realistic test, not all users perform the same actions. To model several types of actions with different probabilities (for example, 70% consultations of the list, 30% consultations by ID), we can use:

• The Throughput Controller with percentages
• Or an alternative approach based on random variables

References:

• https://ambertests.com/2017/03/21/creating-weighted-flows-in-jmeter/

3.8 JMeter Variables vs JMeter Properties

Key Differences

JMeter properties files (in bin/)

Command line options for properties

Option processing order

1. -b (custom properties files)
2. jmeter.properties
3. Log file (-j)
4. Initialization of logging
5. user.properties
6. system.properties
7. All other command line options

Property retrieval functions

• ${__property(name,defaultvalue)}: Returns the property value or the default value if not defined
• ${__P(name,defaultvalue)}: Short version of __property

3.9 Running in non-GUI mode

For real performance testing, always run JMeter in non-GUI (command line) mode to minimize resource consumption.

jmeter -n -t <chemin/script.jmx> -p <chemin/fichier.properties> -f

4. Creating test data with JMeter

For performance testing to be meaningful, the database must contain a realistic volume of data (tens to hundreds of thousands of records).

4.1 JDBC Configuration

To use JMeter with a database via JDBC, you must first configure the connection.

Procedure:

1. Download the JDBC H2 driver (included in db/h2-2.1.214.jar)
2. Place the .jar file in the lib/ directory of JMeter
3. Right-click on the Test Plan → Add → Config Element → JDBC Connection Configuration

JDBC properties for H2:

# local.properties (module 04)
db.url=jdbc:h2:tcp://localhost/test
db.user=sa
db.password=
db.users_table=users
db.employees_table=employees
db.employees_count=250000

4.2 Creating tables with JDBC Request

Procedure:

1. Add a Thread Group dedicated to creating tables
2. Right-click → Add → Sampler → JDBC Request
3. Query Type: Update Statement
4. Write DDL statements

SQL script for creating tables:

CREATE TABLE IF NOT EXISTS users (
    id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    username VARCHAR(255) NOT NULL,
    password VARCHAR(255) NOT NULL
);

CREATE TABLE IF NOT EXISTS employees (
    id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
    first_name VARCHAR(255),
    last_name VARCHAR(255),
    salary NUMERIC(20, 2)
);

CREATE INDEX IF NOT EXISTS employee_first_name_index 
ON employees (first_name);

Note: Adding the index on the first_name column is essential for performance (see section 6.5).

4.3 Inserting users (setUp Thread Group)

To insert users, use a setUp Thread Group which executes first, before other Thread Groups.

Procedure:

1. Add a CSV Data Set Config pointing to the users.csv file (generated by UserGenerator.java)
2. Right-click → Add → Logic Controller → While Controller
3. Condition: (empty) — the loop stops when the last sampler fails (end of CSV file)
4. Add a JDBC Request with the following SQL:

INSERT INTO users (username, password) VALUES ('${username}', '${encryptedPassword}')

Difference between While Controller and Loop Controller:

While Controller condition values:

• Blank: the loop stops if the last sampler in the loop fails
• LAST: the loop stops if the last sampler fails OR if the sampler just before the loop failed
• JSR223 expression: condition evaluated dynamically

4.4 Inserting employees (While Controller + Loop Controller)

To generate 250,000 employees, we combine several elements:

Structure of the Insertion Thread Group:

Insert Employees Thread Group
└── While Controller (condition: vide)
    └── Loop Controller (count: 100)
        └── JDBC Request (INSERT INTO employees)

Insert SQL query:

INSERT INTO employees (first_name, last_name, salary)
VALUES ('${firstName}', '${lastName}', ${salary})

The variables ${firstName}, ${lastName} and ${salary} can be generated with:

• A CSV Data Set Config with pre-generated data
• A Random Variable for numeric values
• JMeter functions: ${__RandomString(5,abcdefghijklmnopqrstuvwxyz)} for strings

4.5 Database indexes

Adding an index on the column used for searching is one of the most important performance fixes:

CREATE INDEX employee_first_name_index ON employees (first_name);

After adding the index, re-run the database population script so that the index is built.

5. Scalability testing with JMeter

5.1 Scalability concepts

Scalability is the ability of an application to handle increasing load. To evaluate it, we measure CPU usage by gradually adding virtual users.

Why push the CPU to 100%?

• If the CPU is at 50% with a runtime of 60 seconds, this means that the CPU was idle for 30 seconds
• By optimizing the application, the CPU could be used at 100%, thus doubling the throughput
• An application with major performance issues will not reach 100% CPU utilization

Scalability Baseline Calculation

The scalability baseline is the number of users needed to push the CPU to around 20-25%.

Why 20-25%?

• The scalability test consists of adding this number of users in stages until reaching 100%
• If 1 user brings the CPU to 20%, it will take 5 users (5 steps) to reach 100%
• This gives a 4-5 step test, easier to analyze than a 10 step test

Example:

• 1 user → CPU ~10% → baseline = 1 user (or 2 to have 20%)
• 2 users → CPU ~20-25% → baseline = 2 users
• Scalability test: increments of 2 users (2, 4, 6, 8, 10)

5.2 SSHMon Samples Collector Plugin

This plugin allows you to monitor the CPU usage of a server remotely via SSH from JMeter.

Installation:

1. In the Plugin Manager, search for SSHMon Samples Collector
2. Install the plugin

Configuration:

• Right-click → Add → Listener → SSHMon Samples Collector
• Configure SSH connection: host, port, username, password
• Add metrics to monitor (CPU, memory, etc.)

Local CPU monitoring (PowerShell):

# Utiliser Get-Counter pour mesurer l'utilisation du CPU
Get-Counter '\Processor(_Total)\% Processor Time'

Local CPU monitoring (Linux/Mac):

# Utilisation CPU en pourcentage
top -bn1 | grep "Cpu(s)" | awk '{print 100-$8}'
# ou
mpstat 1

local.properties file (module 05):

app.employee_count=250000
app.server=localhost
app.port=8081

db.url=jdbc:h2:tcp://localhost/test
db.user=sa
db.password=
db.users_table=users
db.employees_table=employees
db.employees_count=250000

ssh.host=localhost
ssh.port=22
ssh.username=pc
ssh.password=123456

5.3 Establish the scalability baseline

Concurrency Thread Group configuration for baseline:

Procedure:

1. Set the employee_count value in the Test Plan to the actual value (250,000)
2. Close all other programs to get accurate measurements
3. Verify that the database and application are started
4. Run in GUI mode first to observe CPU in real time
5. Observe the CPU value in the SSHMon Samples Collector
6. Identify the number of users corresponding to ~20-25% CPU

Example result: With 2 users, the CPU is at ~26%, sometimes up to 30%. The baseline is 2 users.

5.4 Push CPU to 100% (Scalability Test)

Configuration of the Concurrency Thread Group for the scalability test:

Running in non-GUI mode:

jmeter -n -t testScript.jmx -p local.properties -f

Interpretation of SSHMon Samples Collector:

• CPU rises gradually in steps
• At each level, we add 2 users (the baseline)
• Test passes if CPU reaches or exceeds 90-100%
• If the CPU never exceeds 60-70%, there is probably a performance issue that is preventing the application from fully utilizing the CPU

5.5 Analysis of Summary Report results

The Summary Report provides key metrics for each query:

Important: A high error rate may indicate a problem with the test, the application, the server, or a combination of these. Check JMeter logs and results before analyzing.

6. Detecting persistence issues

6.1 Glowroot: open source APM agent

Glowroot is an open source Application Performance Monitoring (APM) agent that attaches to the JVM like a Java agent and collects performance metrics, including SQL query execution times.

Starting the application with the Glowroot agent

# Windows (startAppGlowroot.bat)
java -javaagent:glowroot/glowroot.jar -jar target/api-0.0.1-SNAPSHOT.jar --server.port=8081

# Linux/Mac (startAppGlowroot.sh)
java -javaagent:glowroot/glowroot.jar -jar target/api-0.0.1-SNAPSHOT.jar --server.port=8081

The Glowroot interface is accessible at: http://localhost:4000

Test setup for Glowroot

• Use 3 to 6 users (based on baseline × 3)
• Disable all JMeter listeners (useless, consume resources)
• Test duration: 5 minutes minimum

jmeter -n -t testScript.jmx -p local.properties

6.2 Analyzing slow queries in Glowroot

In the Glowroot interface:

1. Transactions tab: Overview of endpoints with their execution time

• Default: percentage of total time
• Options: average time, throughput per minute

2. Queries tab: List of all executed SQL queries

Query metrics in Glowroot

Troubleshooting

By comparing the average query times, we can identify anomalies. For example:

• SELECT * FROM employees WHERE first_name LIKE ? → Very high average time (table scan)
• SELECT count(*) FROM employees WHERE first_name LIKE ? → High average time (table scan)
• Login requests → Normal average time

6.3 Thread dumps with jps and jstack

During an active performance test, generate thread dumps to identify lock contentions or deadlocks.

Identify application PID

jps -l

Example output:

12345 com.pluralsight.api.ApiDemoApplication
67890 org.apache.jmeter.NewDriver

Generate thread dump

# Avec jstack (redirigé vers un fichier)
jstack <PID> > 1.txt
jstack <PID> > 2.txt
jstack <PID> > 3.txt

It is recommended to take 3 to 5 thread dumps spaced a few seconds apart to identify patterns.

6.4 Analysis of thread dumps

A thread dump contains the state of all threads in the JVM at the time of capture.

Thread states:

Thread dump analysis tools:

• Manual analysis (.txt files)
• fastthread.io — Free Online Analysis
• thread dump analyzer

Problematic pattern before correction: Multiple BLOCKED threads waiting for the same monitor → contention problem on a slow SQL query.

Pattern after correction (with index): Mainly RUNNABLE threads → queries run quickly, little contention.

6.5 Adding an SQL index to correct table scans

Diagnose a scan table with EXPLAIN

In the H2 console (or any other SQL tool), use EXPLAIN to see the execution plan of a query:

EXPLAIN SELECT * FROM employees WHERE first_name LIKE 'A%' LIMIT 20;

A table scan is visible in the execution plan as TABLE SCAN. This is a sign that the query must read all rows from the table, which is very inefficient for large tables.

Adding the index

In the JMeter table creation script (JDBC Request), add at the end:

CREATE INDEX employee_first_name_index ON employees (first_name);

After adding the index, the query will use the index and the execution plan will show INDEX SCAN or INDEX LOOKUP.

Correction process

1. Delete data from Glowroot (Administration → Storage → Delete all data)
2. re-run the database population script with the CREATE INDEX statement
3. Check execution plan with EXPLAIN → should show index usage
4. Re-run the performance test
5. Check in Glowroot that query execution times have decreased

Expected result: findByFirstNameStartingWith queries have their average time divided by a large factor (often 10x or more) after the index is added.

Rule of thumb: Missing indexes are one of the most common and easiest database problems to fix.

7. Detecting memory problems

7.1 Memory monitoring with VisualVM

VisualVM is an open source tool with profiling capabilities. It connects to a local or remote JVM.

Installing the Visual GC plugin

1. Open VisualVM
2. Menu Tools → Plugins → Available Plugins
3. Search Visual GC
4. Select → Install → Next → Accept license → Install → Finish

This plugin graphically displays information about garbage collector generations.

Starting the application with 64 MB of heap

# Windows (startApp64.bat)
java -Xmx64m -jar target/api-0.0.1-SNAPSHOT.jar

# Linux/Mac (startApp64.sh)
java -Xmx64m -jar target/api-0.0.1-SNAPSHOT.jar

The recommendation is to start with a low amount of memory to see how the application behaves, then adjust accordingly to avoid wasting memory.

Configuring the test to detect memory leaks

Concurrency Thread Group :
  - Target Concurrency : 6
  - Hold Target Rate : 120 minutes (2 heures)

The goal is to show how a memory leak slowly consumes memory over time.

Interpreting Visual GC in VisualVM

• Graph shows memory spaces: Eden Space, Survivor Spaces, Old Gen (Tenured)
• A memory leak is manifested by a progressive increase in the Old Gen which does not go down after a GC
• We observe cycles: allocation in Eden → promotion to Old Gen → Old Gen which gradually increases
• After a while, the GC can no longer free enough memory → Full GC repeated and ineffective

7.2 Calculating Pause Time with jstat

Concepts of Throughput and GC Overhead

Throughput GC = time the JVM spends executing the application (not in GC)

$$\text{Throughput} = 100% - \frac{\text{Total Pause Time}}{\text{Total Time}} \times 100%$$

GC Overhead = percentage of time spent in garbage collection

$$\text{GC Overhead} = \frac{\text{Pause Time}}{1000 \text{ms}} \times 100%$$

Thresholds:

• GC Overhead ≤ 2%: normal
• GC Overhead > 2%: may indicate a problem
• GC Overhead > 50%: OutOfMemoryError: GC overhead limit exceeded is raised

Using jstat

# Identifier le PID
jps -l

# Afficher les statistiques GC toutes les secondes
jstat -gc <PID> 1s

Jstat -gc key columns:

Example of GC Overhead calculation:

Si FGCT augmente de 0.5 s sur une période de 10 s :
GC Overhead = (0.5 / 10) × 100% = 5%  → Problème potentiel

7.3 Analyzing Garbage Collection logs with GCViewer

Enabling GC logs

# Windows (startAppGCLog.bat)
java -Xmx64m \
  -Xlog:gc*:file=log.txt:tags,uptime,level:filesize=600M \
  -jar target/api-0.0.1-SNAPSHOT.jar

# Linux/Mac (startAppGCLog.sh)
java -Xmx64m \
     -Xlog:gc*:file=log.txt:tags,uptime,level:filesize=600M \
     -jar target/api-0.0.1-SNAPSHOT.jar

The -Xlog:gc* option enables Unified JVM Logging (JEP 158). It generates a log.txt file with GC events.

Format of the -Xlog option:

-Xlog:<what>:<output>:<decorators>:<output-options>

GCViewer

java -jar gcviewer-1.36.jar

Load the log.txt file in GCViewer to view:

• Heap usage graph: red line represents total heap size, blue area represents used heap
• Throughput: indicates the percentage of time that the JVM spends executing the application (vs GC)
• Pause Time: duration of GC pauses
• Full GC events: black line — each occurrence indicates a Full GC

Interpretation:

• Throughput ~99%, GC overhead ~1% → healthy
• Throughput gradually decreasing → memory leak detected
• Succession of Full GC without freeing memory → OutOfMemoryError imminent

For Java 21 and later, the application will take longer to reach this critical state, but the end result will be the same.

7.4 Class histograms with jmap and jcmd

Class histograms show how many instances of each class exist in memory and how much space they occupy.

With jcmd

# Identifier le PID
jps -l

# Histogramme des classes (h1.txt)
jcmd <PID> GC.class_histogram > h1.txt

With jmap

# Histogramme avec collecte GC avant (live objects uniquement)
jmap -histo:live <PID> > h2.txt

Difference:

• jcmd GC.class_histogram: histogram of all objects (including garbage-collectables)
• jmap -histo:live: forces a full GC before generating the histogram (only live objects)

Output format:

 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:       1234567      987654321  [B (byte array)
   2:        345678       12345678  java.lang.String
   3:         56789        9876543  java.util.HashMap$Node
...

Usage: Compare two histograms taken at different times to identify classes which accumulate instances → sign of memory leak.

7.5 Analysis of heap dumps with Eclipse MAT

Automatic capture of a heap dump (OutOfMemoryError)

# Windows (startAppHeapDump.bat)
java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -jar target/api-0.0.1-SNAPSHOT.jar

# Linux/Mac (startAppHeapDump.sh)
java -Xmx64m -XX:+HeapDumpOnOutOfMemoryError -jar target/api-0.0.1-SNAPSHOT.jar

The -XX:+HeapDumpOnOutOfMemoryError option automatically captures a heap dump (.hprof file) when an OutOfMemoryError is raised. This is the recommended method because the heap dump is captured at the right time (just before the crash).

Manual capture with jcmd

jcmd <PID> GC.heap_dump <chemin/fichier.hprof>

Analysis with Eclipse MAT

Eclipse MAT (Memory Analyzer Tool) is downloadable from eclipse.org/mat.

Key Features:

1. Leak Suspects Report: Automatic report of memory leak suspects
2. Dominator Tree: Displays the objects that hold (dominate) the largest amount of memory

• Shallow Heap: Memory occupied by the object itself
• Retained Heap: Total memory freed if the object was GC (the object + all the objects it dominates)
• % of total memory occupied

3. Outgoing References: Objects referenced by the selected object
4. Incoming References: Objects that reference the selected object
5. Path To GC Roots: Path from the object to the GC root (explains why the object cannot be collected)

Example of memory leak detected in the demo (module 07):

The WebSecurityConfig class contains a static ConcurrentHashMap which accumulates entries without ever deleting them:

// WebSecurityConfig.java (version avec memory leak)
public static final ConcurrentMap<AuditKey, String> principals = new ConcurrentHashMap();

@EventListener
public void onAuthenticationEvent(AuthenticationSuccessEvent event) {
    User user = (User) ((UsernamePasswordAuthenticationToken) event.getSource()).getPrincipal();
    principals.put(
            new AuditKey(user.getUsername(), event.getTimestamp()),
            """
            Lorem ipsum dolor sit amet... (grande chaîne de texte)
            """
    );
}

The AuditKey class:

public class AuditKey implements Serializable {
    private String username;
    private long timestamp;
    // constructeur, getters, setters
}

MAT Analysis:

• Dominator tree shows WebSecurityConfig retains large portion of memory
• The Outgoing References view shows that the key is an instance of AuditKey and the value is a String
• Path To GC Roots shows that the ConcurrentHashMap is statically referenced by WebSecurityConfig

Security warning: A heap dump contains all the data in memory (passwords, tokens, sensitive data). Strictly control where it is stored and who has access to it.

Limitations of heap dumps:

• The dump size is greater than the memory used
• Requires a lot of RAM and CPU to analyze large dumps
• It’s a snapshot at a given time — you have to capture it at the right moment

7.6 Memory leak detection with JFR and JDK Mission Control

Java Flight Recorder (JFR)

JFR is a profiling and diagnostic framework integrated into the JVM (since Java 11, open source since Java 11). It records JVM events with very low overhead.

Start JFR registration:

# Identifier le PID
jps -l

# Démarrer un enregistrement JFR avec profiling des root paths GC
jcmd <PID> JFR.start path-to-gc-roots=true settings=profile name=leak

# Capturer le fichier JFR
jcmd <PID> JFR.dump name=leak filename=leak.jfr

Extract OldObjectSample events (memory leak):

jfr print --events OldObjectSample <chemin>/leak.jfr > leak.txt

The OldObjectSample event identifies objects in Old Gen that are likely to be memory leaks.

JDK Mission Control (JMC)

Features:

• Open a .jfr file directly in JMC
• Memory view: memory usage graph
• TLAB Allocations tab: see allocations by class
• Old Object Sample tab: identify potentially leaked objects
• Path from object to GC root
• Accumulating object class

JFR vs Heap dump comparison:

Demo files (module 07):

• leak_java17.jfr — JFR registration on Java 17
• leak_java17.txt — OldObjectSample events (Java 17)
• leak_java21.jfr — JFR registration on Java 21
• leak_java21.txt — OldObjectSample events (Java 21)
• log.txt — Log GC
• h1.txt / h2.txt — Class histograms

8. Structure of demo files

Each module has a demos/ directory with demo files.

Module 02 — Introduction

02/demos/
├── src/main/java/com/pluralsight/api/  # Code source Spring Boot
├── src/main/resources/application.properties
├── db/                                  # Fichiers de la base H2
├── target/api-0.0.1-SNAPSHOT.jar       # Application compilée
├── pom.xml
├── startApp.bat / startApp.sh           # Démarrage application
├── startDB.bat / startDB.sh             # Démarrage base H2
└── testScript.jmx                       # Script JMeter simple

Module 03 — Creating the JMeter script

03/demos/
├── before/testScript-clip-03.jmx à 09.jmx  # Scripts avant modification
├── after/testScript-clip-03.jmx à 09.jmx   # Scripts après modification
└── README.txt

The file after/testScript-clip-09.jmx is the final script of the module.

Module 04 — Creation of test data

04/demos/
├── before/db-clip-02.jmx à 04.jmx      # Scripts JDBC avant modification
├── after/db-clip-01.jmx à 04.jmx       # Scripts JDBC après modification
├── local.properties                      # Propriétés (DB, user, password)
└── README.txt

The after/db-clip-04.jmx file is the final script of the module.

Module 05 — Scalability Tests

05/demos/
├── before/testScript-clip-03.jmx à 06.jmx
├── after/testScript-clip-03.jmx à 06.jmx
├── local.properties
└── README.txt

Module 06 — Persistence Issues

06/demos/
├── before/testScript-clip-03.jmx        # Script avant correction
├── after/testScript-clip-03.jmx         # Script après correction
├── before/db-clip-06.jmx                # Script DB avant index
├── after/db-clip-06.jmx                 # Script DB avec index
├── thread-dumps/Java 17/1.txt à 5.txt  # Thread dumps Java 17
├── thread-dumps/Java 21/1.txt à 5.txt  # Thread dumps Java 21
├── startAppGlowroot.bat / .sh           # Démarrage avec agent Glowroot
└── README.txt

Note on thread dumps:

• 1.txt, 2.txt, 3.txt: dumps before fixing the query (threads BLOCKED)
• 4.txt, 5.txt: dumps after correction (RUNNABLE threads)

Module 07 — Memory Problems

07/demos/
├── src/                                  # Code avec memory leak
├── pom.xml
├── h1.txt / h2.txt                       # Histogrammes de classes
├── leak_java17.txt / leak_java21.txt     # Données OldObjectSample JFR
├── log.txt                               # Log GC
├── local.properties
├── startApp64.bat / .sh                  # -Xmx64m
├── startAppGCLog.bat / .sh               # -Xmx64m -Xlog:gc*
├── startAppHeapDump.bat / .sh            # -Xmx64m -XX:+HeapDumpOnOutOfMemoryError
├── startDB.bat / startDB.sh
└── testScript.jmx

9. Command Reference

JMeter Commands

# Exécution en mode non-GUI (recommandé pour les tests)
jmeter -n -t <script.jmx> -p <fichier.properties> -f

# Avec fichier de résultats
jmeter -n -t <script.jmx> -p <fichier.properties> -l results.csv -f

# Avec propriété individuelle
jmeter -n -t <script.jmx> -Japp.users=5

JVM Commands

# Lister les processus Java
jps -l

# Statistiques GC en temps réel (toutes les secondes)
jstat -gc <PID> 1s

# Générer un thread dump
jstack <PID> > dump.txt

# Histogramme des classes (toutes instances)
jcmd <PID> GC.class_histogram > histogram.txt

# Histogramme des classes (live objects seulement)
jmap -histo:live <PID> > histogram_live.txt

# Démarrer un enregistrement JFR
jcmd <PID> JFR.start path-to-gc-roots=true settings=profile name=monEnregistrement

# Capturer le fichier JFR
jcmd <PID> JFR.dump name=monEnregistrement filename=enregistrement.jfr

# Arrêter l'enregistrement JFR
jcmd <PID> JFR.stop name=monEnregistrement

# Extraire des événements d'un fichier JFR
jfr print --events OldObjectSample enregistrement.jfr > leak.txt

# Générer un heap dump manuel
jcmd <PID> GC.heap_dump /chemin/vers/dump.hprof

Important JVM Options

Curl commands to test the API

# Login et obtenir un token
TOKEN=$(curl -s -X POST -H "Content-Type: application/json" \
  -d '{"username":"user01","password":"user01"}' \
  http://localhost:8081/auth/login | python -c "import sys,json; print(json.load(sys.stdin)['token'])")

# Récupérer les employés
curl -H "Authorization: Bearer $TOKEN" http://localhost:8081/api/employees

# Récupérer un employé par ID
curl -H "Authorization: Bearer $TOKEN" http://localhost:8081/api/employees/1

# Recherche par prénom (paramètre start)
curl -H "Authorization: Bearer $TOKEN" "http://localhost:8081/api/employees?start=A&page=0&size=20"

10. Useful links

JMeter

• JMeter Official Documentation
• HTTP Request Defaults
• CSV Data Set Config
• JSON JMESPath Extractor
• Debug Sampler
• Transaction Controller
• Random Variable
• JMeter Functions
• Test Plan and properties
• __property function
• JDBC Connection Configuration
• JDBC Request
• setUp Thread Group
• While Controller
• Loop Controller
• JMeter Plugins
• SSHMon Samples Collector
• JMeter non-GUI mode
• Weighted Flows in JMeter

Java and JVM

• Garbage Collector Java 17 Tuning Guide
• Garbage Collector Java 21 Tuning Guide
• jps command (Java 17)
• jps command (Java 21)
• jstack command (Java 17)
• jstack command (Java 21)
• Unified JVM Logging (JEP 158)
• Java Garbage Collection Logs
• nipafx — Java Unified Logging

Tools

• Glowroot (open source APM)
• Glowroot GitHub
• VisualVM
• Eclipse MAT
• GCViewer

Additional Resources

• How to get CPU usage (Linux)
• Get-Counter Documentation (PowerShell)
• Solution warning bootstrap classpath (Glowroot)

Search Terms

java · se · performance · jmeter · backend · architecture · full-stack · web · test · application · scalability · glowroot · memory · thread · analysis · data · plugin · starting · commands · configuring · controller · csv · dumps · heap