Skip to content
  • ZipCode Api
  • Blog
  • About RedLine13
RedLine13
RedLine13
Primary Navigation Menu
Menu
  • Start Testing
  • Demo
  • Pricing
  • JMeter
  • Partners
  • Docs
    • Documentation Home
    • AWS Set Up for load testing
    • AWS Approval for Large Tests
    • PHP, NodeJS, Python Load Tests
    • Scalability
    • Jenkins Plugin Setup
    • Premium Features
    • Knowledge Base

Merging Very Large JTL Files

By: David Koziel

Merging Very Large JTL Files

RedLine13 has made changes to handle the cases where it is necessary for merging very large JTL files, which are output by JMeter.  RedLine13 has for years been able to handle large JTL files needed to support high volume load tests with no problem.  Now RedLine13 can handle JTL files of 70GB and greater.  We’ve included details on why we made the change for those that want to understand more.

A customer was running tests which generated JTL files in excess of 70GB across multiple servers.  To give you some idea of the size of their test, it ran for over 2 hours, generated over 4.8 million requests at an average rate of more than 55,000 requests per second.  The JTL files that were generated were as large as 70GB.  While our system handled the process of collecting and merging the JTL files, there were some external factors which caused issues.  Here we will explain what the nature of the problem was, how we solved it, and ultimately what we learned to make RedLine13 even better.

Issues that we overcame:

The primary issues were as follows:

  1. We were unable to write the merged JTL file back to S3 because of the massive file size.  This is due to the limitations of the S3 “putObject” method which only permits files up to about 5GB to be uploaded.
    • This issue was addressed by using the “upload” method of getting files into S3.  Using this method raises the upload limit to the maximum S3 object size of 5TB.
  2. Generating the JMeter Dashboard report took more than 2 hours to process.  However, our watchdog monitoring is configured to terminate worker instances taking longer than 2 hours to complete.
    • This issue was addressed by dynamically optimizing this timeout value based on how large the generated JTL file is.

The ultimate result:

When the customer reported the issue, we diagnosed the problem and made the changes.  As a result, RedLine13 now supports scaling tests even bigger.  Nobody had previously tried using our service for such a massive scale test.  However, with a few small improvements to our architecture we have proven we can support it.

Did you know that RedLine13 offers a free trial account? Sign up and start testing today!

2021-10-22
Previous Post: SAML Configuration with Google and RedLine13
Next Post: GitHub Actions for JMeter Test Plans

Recent Posts

  • JMeter XML Format Post Processor
  • Order of Elements in JMeter
  • The JMeter Synthesis Report
  • Using the JMeter Plugins Manager
  • JMeter Rotating JTL Listener

Related

  • JMeter XML Format Post Processor
  • Order of Elements in JMeter
  • The JMeter Synthesis Report
  • Using the JMeter Plugins Manager
  • JMeter Rotating JTL Listener
  • Using Test Fragments in JMeter Tests
  • Step-by-Step Guide to Testing with JMeter
  • Functional Testing vs Performance Testing
  • A Gentle Introduction to Load Testing
  • Using the JMeter Counter Element

© RedLine13, LLC | Privacy Policy | Contract
Contact Us: info@redline13.com

Designed using Responsive Brix. Powered by WordPress.