Overview.
"TestNG Load Generator" allows you to run
TestNG-based tests as load tests.
You need advanced
Java
programming skills to utilize this load generator.
Also, it would be best if you have at least a basic understanding of the
Selenium
framework since the actual logic of the load testing scenarios is implemented via calling
RemoteWebDriver
directly.
All integrations with the Perforator platform are open-sourced.
Please take a look at our GitLab repository
"Perforator SDK Java"
if you are interested in the internal implementation of the load generator.
Configuration and execution of this load generator is done via a dedicated
Maven
plugin, while actual logic implementation is done via implementing
tests using TestNG framework.
The example below can give you a sense of what a real-world configuration might look like.
<plugin>
<groupId>io.perforator.sdk</groupId>
<artifactId>perforator-maven-plugin</artifactId>
<version>1.6.5</version>
<configuration>
<apiClientId>YOUR_API_CLIENT_ID</apiClientId>
<apiClientSecret>YOUR_API_CLIENT_SECRET</apiClientSecret>
<projectKey>YOUR_PROJECT_KEY</projectKey>
<suiteXmlFile>src/test/resources/load_test.xml</suiteXmlFile>
<concurrency>50</concurrency>
<duration>5m</duration>
<rampUp>1m</rampUp>
<rampDown>30s</rampDown>
</configuration>
</plugin>public class ExampleTest {
@Test
public void verify() throws Exception {
RemoteWebDriver driver = PerSuiteRemoteWebDriverManager.getRemoteWebDriver();
Perforator.transactionally("Open landing page and await to be loaded", () -> {
driver.navigate().to("https://verifications.perforator.io/");
awaitToBeVisible(driver, "#async-container");
});
Perforator.transactionally("Verify page 1", () -> {
click(driver, "ul.navbar-nav li:nth-child(1) a");
awaitToBeVisible(driver, "#async-container");
});
Perforator.transactionally("Verify page 2", () -> {
click(driver, "ul.navbar-nav li:nth-child(2) a");
awaitToBeVisible(driver, "#async-container");
});
}
}Jump Start.
Execute `mvn --version` in your terminal to validate both.
examples-testng-release-1.6.5 % ./cloud-full-run.sh
[INFO] --- perforator-maven-plugin:1.6.5:testng (default-cli) @ perforator-sdk-examples-testng ---
[INFO] - Created browser cloud bcc0b2fd-65b6-4ffc-bcb0-1d4b86dd711a
[INFO] - Browser cloud is not ready yet. Status = QUEUED, RequestedBrowsers = 50, StartingBrowsers = 0, ReadyBrowsers = 0
[INFO] - Browser cloud is not ready yet. Status = PROVISIONING, RequestedBrowsers = 50, StartingBrowsers = 0, ReadyBrowsers = 0
[INFO] - Browser cloud is not ready yet. Status = PROVISIONING, RequestedBrowsers = 50, StartingBrowsers = 14, ReadyBrowsers = 36
[INFO] - Browser cloud is ready: selenium hub = https://selenium.perforator.io/---/---/wd/hub
-------------------------------------------------------------------------------------------------------------------------------
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |
| Load generation is about to start. |
| Please open the link below in the browser to see statistics in real-time. |
| https://app.perforator.io/statistics/73df630c-86b4-40e0-8ab4-a4ac2e9ec373/28cd1cae-8fb8-4417-bace-ee2f15fb1924 |
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |
-------------------------------------------------------------------------------------------------------------------------------
[INFO] - suites: active = 1, failed = 0, successful = 0; transactions: active = 2, failed = 0, successful = 0
[INFO] - suites: active = 13, failed = 0, successful = 16; transactions: active = 36, failed = 0, successful = 165
[INFO] - suites: active = 30, failed = 0, successful = 109; transactions: active = 84, failed = 0, successful = 930
[INFO] - suites: active = 50, failed = 0, successful = 314; transactions: active = 128, failed = 0, successful = 2613
[INFO] - suites: active = 50, failed = 0, successful = 1355; transactions: active = 142, failed = 0, successful = 10955
[INFO] - suites: active = 50, failed = 0, successful = 2352; transactions: active = 132, failed = 0, successful = 18905
[INFO] - suites: active = 9, failed = 0, successful = 2403; transactions: active = 24, failed = 0, successful = 19269
[INFO] - Terminating browser cloud -> bcc0b2fd-65b6-4ffc-bcb0-1d4b86dd711a
[INFO] - Browser cloud has been successfully terminated -> bcc0b2fd-65b6-4ffc-bcb0-1d4b86dd711a
[INFO] - suites: active = 0, failed = 0, successful = 2412; transactions: active = 0, failed = 0, successful = 19296
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:49 min
[INFO] Finished at: 2022-02-15T23:49:22-06:00
[INFO] ------------------------------------------------------------------------Maven Build.
Prebuilt example
comes with ready-to-use pom.xml where all dependencies and plugins are already configured.
It might be a case when you would like to integrate TestNG Load Generator into an existing
maven project or prefer to configure the build from the ground up, so please take a look at the instructions below.
1. All compiled and well-tested artifacts of the Perforator SDK are uploaded to
GitLab based maven repo.
Please update your pom.xml by adding
pluginRepositories
and
repositories pointed to our maven repo, so maven can automatically download all required artifacts.
<project>
...
<pluginRepositories>
...
<pluginRepository>
<id>perforator-gitlab-maven</id>
<url>https://gitlab.com/api/v4/projects/32457860/packages/maven</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</pluginRepository>
...
</pluginRepositories>
...
<repositories>
...
<repository>
<id>perforator-gitlab-maven</id>
<url>https://gitlab.com/api/v4/projects/32457860/packages/maven</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
<releases>
<enabled>true</enabled>
</releases>
</repository>
...
</repositories>
...
</project>2. Add perforator-sdk-load-generator-testng artifact to your dependencies list, so all classes required by the load generator are available during compilation and execution phases.
<project>
...
<dependencies>
...
<dependency>
<groupId>io.perforator.sdk</groupId>
<artifactId>perforator-sdk-load-generator-testng</artifactId>
<version>1.6.5</version>
</dependency>
...
</dependencies>
...
</project>3. Add perforator-maven-plugin to the build plugins section.
<project>
...
<build>
...
<plugins>
...
<plugin>
<groupId>io.perforator.sdk</groupId>
<artifactId>perforator-maven-plugin</artifactId>
<version>1.6.5</version>
<configuration>
...
</configuration>
</plugin>
...
</plugins>
...
</build>
...
</project>Configuration.
TestNGLoadGenerator configuration and execution are done via a dedicated maven plugin perforator-maven-plugin.
There are four required settings to run this load generator:
-
apiClientId
- client ID for API authentication.
Please navigate to the
API Settings
page to get your API Client ID.
-
apiClientSecret
- client Secret for API authentication.
Please navigate to the
API Settings
page to get your API Client Secret.
-
projectKey
- identifier of the project to create new execution records.
Please refer to the
"Projects"
chapter to learn more about projects and get your project key.
-
suiteXmlFile
- location of the TestNG suite XML file to execute.
Additionally, the load generator supports many more optional settings allowing you to fine-tune load test execution.
Feel free to review all available options at the Settings tab.
There are two major sets of configuration options:
- Settings with the "Scope: loadGenerator" define the overall behavior of the load generator.
- Settings with the "Scope: suite" control behavior of the individual suite.
It might be a case when you would like to apply different load testing scenarios and control such scenarios independently.
Perforator maven plugin allows you to configure multiple suites where every suite is executed independently.
Please take a look at the three configurations below,
where the first two options define single suite execution,
while the third one allows you to run multiple suites independently.
<!-- Single suite configuration -->
<plugin>
<groupId>io.perforator.sdk</groupId>
<artifactId>perforator-maven-plugin</artifactId>
<version>1.6.5</version>
<configuration>
<apiClientId>YOUR_API_CLIENT_ID</apiClientId>
<apiClientSecret>YOUR_API_CLIENT_SECRET</apiClientSecret>
<projectKey>YOUR_PROJECT_KEY</projectKey>
<suiteXmlFile>src/test/resources/load_test.xml</suiteXmlFile>
<concurrency>50</concurrency>
<duration>30m</duration>
</configuration>
</plugin><!-- Single suite configuration -->
<plugin>
<groupId>io.perforator.sdk</groupId>
<artifactId>perforator-maven-plugin</artifactId>
<version>1.6.5</version>
<configuration>
<apiClientId>YOUR_API_CLIENT_ID</apiClientId>
<apiClientSecret>YOUR_API_CLIENT_SECRET</apiClientSecret>
<projectKey>YOUR_PROJECT_KEY</projectKey>
<suites>
<suite>
<suiteXmlFile>src/test/resources/load_test.xml</suiteXmlFile>
<concurrency>50</concurrency>
<duration>30m</duration>
</suite>
</suites>
</configuration>
</plugin><!-- Multiple suites configuration -->
<plugin>
<groupId>io.perforator.sdk</groupId>
<artifactId>perforator-maven-plugin</artifactId>
<version>1.6.5</version>
<configuration>
<apiClientId>YOUR_API_CLIENT_ID</apiClientId>
<apiClientSecret>YOUR_API_CLIENT_SECRET</apiClientSecret>
<projectKey>YOUR_PROJECT_KEY</projectKey>
<suites>
<suite>
<suiteXmlFile>src/test/resources/load_test_suite_1.xml</suiteXmlFile>
<concurrency>50</concurrency>
<duration>30m</duration>
</suite>
<suite>
<suiteXmlFile>src/test/resources/load_test_suite_2.xml</suiteXmlFile>
<concurrency>30</concurrency>
<duration>25m</duration>
</suite>
</suites>
</configuration>
</plugin>Overrides.
Load generator uses the following order while resolving specific setting values:
1. Command line arguments.
2. Environment variables.
3. pom.xml setting.
This approach allows you to prepare a configuration file once and then execute it with different options.
A real-world example is when you store your project in a version control system like GIT, and you don't want to expose security keys.
So you can configure a new environment variable dedicated to the specific property, and a load generator automatically resolves it.
All Settings mention command-line argument and environment variable where an override is available.
For example, apiClientId has the following overrides:
- Command line arg: -DloadGenerator.apiClientId=...
- Environment variable: LOADGENERATOR_APICLIENTID=...
TestNG Tips.
The main goal of all Perforator-based load tests is to send commands and actions to remote browsers,
so such browsers generate an actual load on your website.
You can send all necessary actions to remote browsers using the
RemoteWebDriver instance.
Prebuilt example
comes with the dedicated TestNG listener PerSuiteRemoteWebDriverManager
responsible for starting a new browser session at the beginning of suite instance processing and automatically terminating it at the end.
It might be a case when the default behavior is not desirable,
or you would like to tight control when to start and stop a browser session programmatically.
So you can call Perforator.startRemoteWebDriver()
directly.
Please don't forget to terminate a browser session by calling
RemoteWebDriver.quit()
when you are done with your logic.
Otherwise, the browser cloud will become fully occupied with active browsers, and you will not be able to create a new one.
Note:
Starting a new browser session takes around 2 seconds under the average load
and five or even more seconds when all active browsers in the cloud drain the CPU heavily.
So, it is not suggested to start/stop browsers too frequently.
Otherwise, such behavior can have a significant impact on load test results.
package com.example;
import io.perforator.sdk.loadgenerator.core.Perforator;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.ISuite;
import org.testng.ISuiteListener;
public class PerSuiteRemoteWebDriverManager implements ISuiteListener {
private static final ThreadLocal<RemoteWebDriver> REMOTE_WEB_DRIVER = new InheritableThreadLocal<>();
@Override
public void onStart(ISuite suite) {
REMOTE_WEB_DRIVER.set(
Perforator.startRemoteWebDriver()
);
}
@Override
public void onFinish(ISuite suite) {
RemoteWebDriver driver = REMOTE_WEB_DRIVER.get();
if(driver != null) {
try {
driver.quit();
} finally {
REMOTE_WEB_DRIVER.remove();
}
}
}
public static RemoteWebDriver getRemoteWebDriver() {
return REMOTE_WEB_DRIVER.get();
}
}Transactions.
TestNG load generator automatically reports suite instance processing as Top-Level transaction,
where a transaction name is generated as "suite - ${suite.name}".
Suite name is automatically extracted from
suiteXmlFile.
For example, if your TestNG suite has a suite definition as
<suite name="Example Load Testing Suite">,
then a top-level transaction should be generated as "suite - Example Load Testing Suite".
The load generator additionally generates nested transactions for each test and method execution.
For example, you can expect the following transactions to be automatically generated:
- "test - Example Test".
- "method - com.example.ExampleTest.verify".
Additionally, the prebuilt example automatically starts a new remote browser at the beginning of suite instance processing
and terminates it at the end.
Such actions automatically report nested transactions with the names below:
- "Open browser session".
- "Close browser session".
You can also report manual transactions if you would like to measure performance of the specific code blocks.
Such manually reported transactions have the following characteristics:
- Parent Transaction ID is automatically picked from the top-level transaction.
- Parent Transaction Name is automatically picked from the top-level transaction.
- Transaction ID is automatically generated in UUID format.
- Transaction Name is the name of transaction you manually supply.
- Transaction Type is a constant value of "Nested".
- Session ID is automatically picked from the browser session.
Note:
Load generator submits transactions to the analytical system only when you execute your load test using browsers in the cloud.
For example, no transactions are reported if
webDriverMode = local,
which is a case for local.sh and local.cmd scripts.
Please take a look at the examples below on how to report transactions manually:
package com.example;
import io.perforator.sdk.loadgenerator.core.Perforator;
import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import org.testng.annotations.Test;
public class ExampleTest {
@Test
public void verify() throws Exception {
RemoteWebDriver driver = PerSuiteRemoteWebDriverManager.getRemoteWebDriver();
// Report a transaction for Runnable block
Perforator.transactionally("Your transaction name #1", () -> {
driver.navigate().to("https://verifications.perforator.io/");
awaitToBeVisible(driver, "#async-container");
});
// Report a transaction for Supplier block
WebElement container = Perforator.transactionally("Your transaction name #2", () -> {
driver.navigate().to("https://verifications.perforator.io/");
return awaitToBeVisible(driver, "#async-container");
});
}
private static WebElement awaitToBeVisible(RemoteWebDriver driver, String cssSelector) {
return new WebDriverWait(driver,30).until(
ExpectedConditions.visibilityOfElementLocated(
By.cssSelector(cssSelector)
)
);
}
}Logging.
All "TestNG" examples come with the prebuilt log4j2.xml configuration file, so you can easily fine-tune the logging of your load test.
Please take a look at the default logging config below.
<?xml version="1.0" encoding="UTF-8"?>
<Configuration
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" strict="true"
xmlns="http://logging.apache.org/log4j/2.0/config"
xsi:schemaLocation="http://logging.apache.org/log4j/2.0/config
https://raw.githubusercontent.com/apache/logging-log4j2/master/log4j-core/src/main/resources/Log4j-config.xsd"
>
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="[%d][%X{X-PERFORATOR-CONTEXT}][%c{1}][%4p] - %m%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="Console"/>
</Root>
<Logger name="org.openqa.selenium" level="error" />
</Loggers>
</Configuration>- logWorkerID - it adds worker ID to the logging context of the current thread.
- logSuiteInstanceID - it adds suite instance ID to the logging context of the current thread.
- logRemoteWebDriverSessionID - it adds browser session ID to the logging context of the current thread.
- logTransactionID - it adds currently active transaction ID to the logging context of the current thread.
Note: It is not suggested to change the logging level to DEBUG or TRACE when you run your load test at full speed because it will log too much information, and process output might become too overloaded, so please use it at your own risk.
Running.
There is no enforced workflow on how to prepare and execute a load test, but here is a suggested one:
1. Make necessary changes in the perforator-maven-plugin configuration.
2. Make necessary changes in your TestNG-based tests.
3. Verify changes using local browsers.
4. Dry run your changes using a small number of remote browsers and ensure that all browser actions are not affected by network delays.
5. Execute your load test at full speed
Please take a look at the scripts below prepared for the suggested workflow.
- local.sh - executes your TestNG suite(s) using Chrome browsers started locally. This script is helpful to verify that everything works as intended visually.
- local-headless.sh - it has the same purpose as local.sh, but valuable for CI/CD environments where a graphical environment is unavailable.
- local-debug.sh - it executes the logic using maven in debug mode, so you can attach external debugger from your IDE.
- cloud-dry-run.sh - it executes your TestNG suite(s) using only 10 browsers started in the cloud. The main idea of this script is to avoid extra charges when you need to ensure that your logic is executed correctly using cloud-based browsers. For example, it is not optimal to run your full-powered test using thousands of browsers and then, in a minute, realize that you have a simple mistake in css selector.
- cloud-full-run.sh - it executes your load test at full speed as defined in maven plugin configuration.
- local.cmd - executes your TestNG suite(s) using Chrome browsers started locally. This script is helpful to verify that everything works as intended visually.
- local-headless.cmd - it has the same purpose as local.sh, but valuable for CI/CD environments where a graphical environment is unavailable.
- local-debug.cmd - it executes the logic using maven in debug mode, so you can attach external debugger from your IDE.
- cloud-dry-run.cmd - it executes your TestNG suite(s) using only 10 browsers started in the cloud. The main idea of this script is to avoid extra charges when you need to ensure that your logic is executed correctly using cloud-based browsers. For example, it is not optimal to run your full-powered test using thousands of browsers and then, in a minute, realize that you have a simple mistake in css selector.
- cloud-full-run.cmd - it executes your load test at full speed as defined in maven plugin configuration.
Feel free to execute maven commands directly in the terminal for more granular control.
For example: `mvn clean test-compile perforator:testng -Dsuite.webDriverMode=cloud -Dsuite.concurrency=10`
Execute `mvn --version` in your terminal to validate both.
Debugging.
All modern IDEs like IntelliJ IDEA, Eclipse, or Netbeans allow you to attach a debugger to the external process.
You can execute local-debug.sh(on Linux/macOS) or local-debug.cmd(on Windows) prebuilt scripts in your terminal, and it will start the maven process in debug mode so that you can attach an external debugger from your IDE.
Please use the following settings while attaching an external debugger to the maven process:
- transport: dt_socket
- host: localhost
- port: 8000
Note: It is not recommended to attach a debugger when running your suite(s) using browsers in the cloud, i.e., webDriverMode = cloud.
Cloud-based browsers are automatically killed if there is no activity within 60 seconds, so the debugger's extended blockage of the suite processor thread will automatically lead to remote browser termination.
examples-testng-release-1.6.5 % ./local-debug.sh
Preparing to execute Maven in debug mode
Listening for transport dt_socket at address: 8000
[INFO] --- perforator-maven-plugin:1.6.5:testng (default-cli) @ perforator-sdk-examples-testng ---
[2022-02-10 04:53:44][INFO] - Using chromedriver 98.0.4758.80 (resolved driver for Chrome 98)
[2022-02-10 04:53:44][INFO] - Exporting webdriver.chrome.driver as /Users/max/.cache/selenium/chromedriver/mac64/98.0.4758.80/chromedriver
[2022-02-10 04:53:48][INFO] - suites: active = 1, failed = 0, successful = 0; transactions: active = 1, failed = 0, successful = 1
[2022-02-10 04:53:53][INFO] - suites: active = 1, failed = 0, successful = 0; transactions: active = 1, failed = 0, successful = 1Limitations.
While browsers generating the load on your website are running in the cloud, the load generator itself is executed locally and is only responsible for orchestrating remote browsers.
Such behavior comes with certain requirements applied on local hardware and internet connection speed.
Here are minimal requirements to orchestrate every 500 remote browsers:
- 2 CPU cores
- 2 GB RAM
- 20 Mbps Internet bandwidth
For example, you need the following resources locally to run a load test with 2000 remote browsers:
- CPU cores: 2000 ÷ 500 × 2 Cores = 8 Cores
- RAM: 2000 ÷ 500 × 2GB = 8 GB
- Internet: 2000 ÷ 500 × 20Mbps = 80 Mbps
Warning: You might see very confusing load test statistics and high errors rate if you don't meet minimal hardware and internet bandwidth requirements.