JMeter is one of the most popular load testing frameworks. However, to use JMeter at scale requires supporting infrastructure. In this post, we will be comparing and contrasting do-it-yourself JMeter testing versus managed JMeter load testing.
With the rise of cloud computing architectures, it has become increasingly the standard to test using cloud-based resources. This means writing scripts to provision servers, coordinate load generators, and handle data collection. Load testers using JMeter have two general options. They can opt to design and build the entire stack themselves as a “DIY” approach. Alternatively, testers can use ready-made solutions such as RedLine13 to abstract away everything except for the actual load test.
JMeter itself is a command-line tool, and though there is a GUI mode it is not optimal to generate load in this way. This necessitates coordinating often multiple instances across multiple load generating servers, which requires significant infrastructure programming. There is also the concern about keeping a DIY stack up to date with the latest software releases. With some time and effort invested this is entirely possible with the DIY approach.
RedLine13 manages this, keeping testing software, operating systems, and dependencies updated with the latest releases and security fixes. This can be a significant advantage for small teams who might want to focus on writing actual tests versus investing in a custom test infrastructure. With RedLine13, you can access your tests from anywhere and at any time through an easy-to-use web portal. Support exists for multiple test types “out-of-the-box”, including JMeter, Gatling, custom scripting, and more.
Integrating with AWS
There are multiple cloud providers supplying commodity compute instances. Utilizing these resources has become the de facto standard in cloud-based load testing. When tasked with a DIY approach, it is up to the test administrator to provision and configure these instances, run tests, collect results, and ultimately dispose of these resources. This can add a significant amount of work to a load testing project. It also adds complexity which can require some degree of the testing of the test architecture itself. Though it confers the advantage of flexibility, taking this approach can increase time and cost.
RedLine13 can help testers achieve this by providing a vetted platform which tightly integrates with AWS. Running a JMeter test on RedLine13 is as simple as uploading the test file, selecting the number and specifications of the EC2 instances to be provisioned, and clicking “Start Test”.
Hidden Costs of Load Testing
With cloud-based load testing, there are finite resource costs which are known. Primarily these are server costs for running load generators, and associated networking resources. However, with a DIY setup one has to be cognizant of additional hidden costs. A significant investment in infrastructure is required to coordinate load generators and collect results. Similarly, maintenance of that infrastructure also requires time and attention in keeping that software updated with the latest releases and security patches. There are also hidden costs in testing that infrastructure itself to ensure that the results obtained are reliable.
Comparing this to RedLine13, most of these hidden costs no longer apply. You have access to a vetted platform that does not require you as the tester to address these issues. This allows you to focus on what matters most — running your load tests.
When developing your own testing architecture, inevitably one would seek to automate common and repetitive tasks. However, this comes only after identifying these repetitive tasks, and then requires a development investment to create these shortcuts.
RedLine13 has already identified many of these convenience features, which are available on the platform for your use today:
- Duplicate a load test with a single click
- Automatically download logs and JTL files from load generators
- Automatically merge JTL files across multiple load generators
- Generate JMeter dashboard reports with a single click
- Organize, search, and sort from all previously run load tests
Though a DIY architecture can offer ultimate flexibility, this comes at the cost of supporting it yourself. Redline13 offers a few distinct advantages over a purely DIY load testing architecture. One of these is through a knowledgeable technical support team. Apart from your actual load test, all components of the architecture responsible for running your test are backed by a responsive technical support team.
RedLine13 offers email support with all plans and usually same-day responses within normal business hours. When compared to a DIY approach, this can offload a significant burden from your own technical team.
Cost of RedLine13 vs. DIY
It is possible to build your entire test infrastructure from scratch. Doing so would allow fine-grained and complete control over every aspect. However, in many cases the test architecture does not need to be so specialized as to justify this. More commonly it is advantageous to utilize a ready-built load testing platform that performs all of the required tasks. This is the dichotomy of “build” vs. “rent”.
RedLine13 provides a robustly featured platform for a low monthly subscription. On the pricing and features page, you can compare three different plans (Free, Basic, and Premium). Each plan provides increasingly more features that otherwise you would have to build for yourself.
HBO Latin America tried DIY
Many companies, big and small, have tried a do-it-yourself approach. HBO Latin America is one of those companies that tried to build their own framework to save money. Read more here.
Ready to compare running your test on RedLine13? Sign-up for a fully featured free trial and start testing today!