There has been a lot of excitement surrounding large language models (LLMs) and especially ChatGPT made available by OpenAI. Heralded as a paradigm in making powerful AI capabilities widely accessible, the GPT-3.5 and GPT-4 models behind the ChatGPT interface continue to find uses in business, data science, linguistics, and writing that did not exist in the short time before their release.
In this article, we will illustrate how the LLM behind ChatGPT can be leveraged to simplify certain steps in creating JMeter test plans. We will review several examples and discuss the benefits and limitations of using large language models as AI assistants in the specific context of load testing.
Getting Started
Our first step enlisting the assistance of AI is to create an account. From the ChatGPT app, you can sign up for a free account which gives you access to the GPT-3.5 model – or you can subscribe for a premium and gain access to the more capable GPT-4 model. For our purposes, the free version will suffice.
Introducing ChatGPT for JMeter Test Design
When interacting with ChatGPT, the initial input from the user must be supplied in the form of a prompt. To get started, let’s consider the following prompt:
Create a JMeter test that probes the ultimate load of the website "http://www.my-enterprise-app.com/".
The output for each instance of ChatGPT will vary slightly, however we can consistently expect the response in the format which follows. For this general prompt, in this case we received a list of instructions on how to create a JMeter test:
If this is not the first time using JMeter, this list of instructions is only marginally useful. This begins to highlight the strengths and weaknesses of the current generation of AI assistant tools. It would be far more beneficial to us if we were much more specific in our requests.
We can do this using the following prompt:
How can I set up a JMeter thread group that targets "http://www.my-enterprise-app.com/" and scales to one million threads over the course of one hour?
Now things are getting more interesting. While still confronted with a number of generic steps, ChatGPT begins to tell us some specifics. To scale to the level of 1M threads, it recommends using the Ultimate Thread Group plugin:
In addition to this, the output also lists some additional instructions for adding HTTP Request samplers to our thread group, and setting the target URL to “www.my-enterprise-app.com”:
However, there are still many manual tasks that remain, and blanks that need to be filled in before we have a working test plan on our hands. For the experienced test designer, we will want to be even more specific in our prompts to ChatGPT to maximize our efficiency using this tool.
ChatGPT helps with Focused Tasks
Suppose that we wanted to stress test a new user registration page within our target test application. The form on the page accepts username, email, and password fields. We have a large list of these data fields in a CSV file. We can consider asking ChatGPT using the following prompt:
Create a JMeter script that generates “username”, “email”, and “password” form input for an HTTP Request sampler from CSV source data.
The output is much more specific for our use case. We have file structure, field mappings, and specific instructions on how to integrate our data with various test plan elements:
There are still many tasks that we as the test designer need to complete ourselves. Despite this, ChatGPT has served us in identifying how to configure the test plan elements and outlaid a step-by-step plan on how to achieve what we have asked. Though our use case has limitations, it is still an impressive use of this newly available technology.
Using ChatGPT to Code JMeter Scripts
Undeniably ChatGPT excels at conversational output. It is concise, easy to understand, and even polite. However, large language models are not limited to the syntax of written grammar. They also possess a surprisingly good capacity for generating code. Let’s take the next example a step further and enlist ChatGPT to write a short Groovy script as part of a slightly more complicated JMeter test.
Let’s use the following prompt:
Write a Groovy script for a JMeter Post processor element that extracts a “success” value from a JSON response.
With enthusiasm, ChatGPT quickly provides the following:
It suggests using the JSON Extractor element, which is appropriate for our use case. Furthermore, it understands the context of “a ‘success’ value from a JSON response” usually comes as part of a fully-qualified JSON structure. Without providing the actual format, it inferred that structure as a likely expected response. Using this, it provides the following Groovy script as requested:
import org.apache.jmeter.threads.JMeterVariables
// Get the extracted "success" values from the JSON Extractor
def successValues = vars.getObject("successValue_matchNr")
// Initialize a counter for successful responses
def successfulResponses = 0
// Loop through the extracted values and count successful responses
for (int i = 1; i <= successValues.toInteger(); i++) {
def successValue = vars.get("successValue_" + i)
if (successValue == "true") {
successfulResponses++
}
}
// Set the total successful responses count to a JMeter variable
vars.put("successfulResponses", successfulResponses.toString())
Finally, it also summarizes how the script works:
Most large language models including GPT-3.5 have understanding of most programming languages. It is possible to ask ChatGPT to automate other coding tasks. This includes writing the JMX test plan file itself. If we wanted to same time and automate some boilerplate tasks in generating the skeleton test plan for any of the above examples, we could follow up entering this prompt:
Write the raw JMX file for a simple JMeter test.
The following result is provided by ChatGPT. From the xml code block, we can save the output as a *.JMX file:
Opening the output directly in JMeter is possible without any further editing of this code:
Conclusion
The recent availability of large language models backed by AI assistant interfaces like ChatGPT represent a new tool to help developers automate their workflows. Though often romanticized in the media as a form of sentient intelligence, and feared by many who don’t understand them – in their present iteration they are merely tools. Like any tool, we can expect the best result by understanding how to properly use them.
Did you know that RedLine13 offers a full-featured time-limited free trial? Sign up now, and start testing today!