Skip to content
  • ZipCode Api
  • Blog
  • About RedLine13
RedLine13
Primary Navigation Menu
Menu
  • Start Testing
  • Demo
  • Pricing
  • Docs
    • Knowledge Base
    • AWS IAM Setup Instructions
    • Running a RedLine13 Load Test with Advanced Options
    • Scalability
    • Writing Open Load Tests in Your Language
    • Jenkins Plugin Setup
    • AWS Approval for Large Tests
    • Pro Features
  • JMeter
  • Partners

Why Doesn’t Continuous Load Testing Work?

By: RedLine13

How many businesses or organizations actually do continuous load testing?

There are typically two purposes for load tests: event-based and continuous load tests.

The first is the one we normally think of: load tests that are run due to some external event. It could be a new marketing campaign that is going to drive additional volume. It could be a new version of software and wanting to make sure that it will handle the volume. Or it could be Black Friday and the increased volume.

The second one is not so common: continuous load test. This could be run every day or week or even longer timeframe. Or potentially shorter if it makes sense. The purpose of this test is to ensure that there are good and consistent results. There are innumerable internal and external factors that could change and lead to unacceptable performance. By running tests on a regular basis, we are hopefully finding out issues or seeing trends before our users are.

As for tools, you can load test your project continuously with Jenkins. Jenkins is one of the most popular open source CI/CD tools used to build, deploy and test your project continuously. RedLine13 supports integration with Jenkins which means you can build, deploy and load test your project continuously with Jenkins. 

But what does it cost?

Most people would agree that this all makes sense. The concern often turns to the cost to do this. How could cost be an issue for RedLine13 users when we say it’s Almost Free? The AWS cost per test is miniscule. There is a RedLine13 subscription cost, but if you have an Annual Premium subscription for $2,500 and run just one test a day, it is $6.85 per test. And if you only need an Annual Basic, it’s $1.37 per test. So what does that mean? 

It means that it is cheap but they aren’t the only costs. Drew Batshaw, Co-Founder and CTO of Waggl and RedLine13 customer, explained when they run event based tests:  “Whenever we make changes that we think might impact load or are needing to increase our load.”

Waggl is efficient and has also automated the work with AWS scripts and CloudFormation templates that they use for building and tearing down the environment they point the load testing at.

They see the benefits in continuous load testing but it would be expensive for them. Drew pointed something out:  “The biggest cost is not Redline13 or the AWS costs but the cost to run the environment that gets load testing. For us (and I imagine most people), that is an environment that is as expensive to run as our real production environment.  It is not unusual for us to spend thousands of dollars a month on that if we are doing a lot of load testing (and that is not running that environment all the time).” 

Is there a solution for continuous load testing? What do you do?
In the meantime, you can try your own test, event-based or continuous, with RedLine13 here.

2019-11-27
Previous Post: API Load Testing with cURL and RedLine13
Next Post: How many vCPUs needed for Load Testing?

Recent Posts

  • Use Cases for the JMeter Dashboard Report
  • AWS Costs for Large JMeter Load Tests Run by Real Customers
  • AWS Multiple Account Setup for Load Testing – Securing your Application Environment
  • Merge Results for Data Analysis
  • Debug your JMeter Test with Output Files

Related

  • SAML SSO
  • E-Learning Companies and Load Testing
  • AWS Multiple Account Setup for Load Testing – Securing your Application Environment
  • AWS Costs for Large JMeter Load Tests Run by Real Customers
  • Why BlazeMeter Customers Chose RedLine13
  • Is BlazeMeter Scared Of RedLine13?
  • RedLine13 Customer HBO Latin America Speaks at Customer Advisory Board Meeting
  • ThinkLogic and High Volume Tests on RedLine13
  • Why ACT Moved from BlazeMeter – Highlights from the RedLine13 Customer Advisory Board Meeting
  • University of California Wanted a LoadRunner Alternative

© RedLine13, LLC | Privacy Policy | Contract
Contact Us: info@redline13.com