Writing Custom Load Tests

Custom load tests allow the greatest degree of freedom and flexibility on our platform.  Some things that are shared between any of our supported language-specific load tests are:

  • The ability to run any code you want to execute
  • RedLine13 API calls can report test execution and timing

Each language is documented independently and can be found on the corresponding page using the links below:

Choosing your Language

There are several factors which will drive your decision on which supported language to use for your custom load test (e.g., PHP, NodeJS, or Python).  One of the primary reasons you may choose to write a custom load test in the first place is to have ultimate control and flexibility over your test.  You can achieve this with any of the supported languages.  However, no matter which one you select the next factor would be your relative familiarity with each of these languages.  In this post we will focus on writing a custom NodeJS test, but the concepts on our platform will generally carry over into the other supported languages.

Submitting Performance Tests

  • There are two ways to submit a custom load test
    • Submit the Load Test File (e.g., [ANY_NAME].js) in the language specified.
    • Submit a compressed file (which can be .tar, tar.gz, or .tgz) with the custom file specifically named.  For NodeJS tests this is “CustomTest.js“.
  • Custom Load Tests support CSV files, compressed archives, and extra files as attachments:
    • CSV files can be split across servers using the “split” option.
    • Compressed files can be marked to be expanded using the “expand” option.
    • All files are placed in the root of the load test.

Package Support in Load Tests

Each Language supports a packaging mechanism to install more packages.  For NodeJS, NPM is built into the test, so you just need to include your own package.json and we will run npm install.

Versioning

Using the Node Version Manger (NVM), our Node Version Manager Plugin supports all version of NodeJS.

Test Harness for local testing

A Test Harness is available for all languages, including examples for the NodeJS Harness.  If you are writing a custom test in a different language (such as PHP or Python), you will want to select the appropriate test harness as described above.

NodeJS Test

To write a custom load test in Node.js, you need to create a Node.js file named CustomTest.js if packaged in compressed file, otherwise it can be any name.

You can include the helper class loadTestingSession for testing HTTP requests.  This class helps to manage cookies and track performance data.  The following code snippet illustrates how to add this:

var LoadTestingSession = require("./loadTestingSession.js");

You will need to define a function constructor with the arguments  (redlineApi, testNum, rand, config).  Here is an example constructor definition:

/**
 * Build a custom test by creating a constructor
 * @param redlineAPI - redline metric calls
 * @param testNum - when simulating many users, this is the user number
 * @param rand - a test identifier, same across all users within test
 * @param config - hashmap for your config variables passed through
 */
function MyCustomTest(redlineApi, testNum, rand, config)
{
    ...
}

An explanation of the corresponding constructor arguments above are as follows:

  • redlineApi – This variable will let you access Redline API methods to report back data.
  • testNum – This is the test number that is being run.
  • rand – This is a random string associated with the test that typically is unused.
  • config – This is the parsed result of ini.parse. config.load_resources will be set to "1" if you enabled the associated setting when starting the test.

The test does not start when the constructor is called.  However, here we can setup variables for this instance of the test and read configuration.  Here is an example which accomplishes this:

function MyCustomTest(redlineApi, testNum, rand, config)
{
  // We need Redline API to report results, errors, requests.
  this.redlineApi = redlineApi;

  // Keep track of test information.
  this.testNum = testNum;
  this.rand = rand;

  // This is the test configuration file (loadtest.ini)
  this.config = config;

  // Hardcode endpoint, requires plugin to customize.
  this.url = this.config.url;
  if ( !this.url )
    this.url = "https://httpbin.org/status/200";

  // This is just example, you might need build a redline plugin to add these parameters.
  this.minDelayMs = this.config.min | 500;
  this.maxDelayMs = this.config.max | 10000;
}

As such, the test starts when runTest is called on the class.  This method accepts a callback function that expects a single parameter that is true if the test failed and false otherwise.  The callback can be called using callback.call(that, failed);.  Here is an example of that:

/** 
 * Run test 
 * @param redlineCallback as tests are asynch callback is used to tell runner that test is complete (success or failure) 
 */
MyCustomTest.prototype.runTest = function( redlineCallback )
{
    ...
}

Inside your test, it’s all your code and work.  You can use the redlineApi class to tell RedLine13 the performance data to track:

MyCustomTest.prototype.runTest = function( redlineCallback )
{
  var that = this;

  // Set delay
  var delay = 1;
  if (this.delayRangeMs != 0)
    delay = Math.floor((Math.random()*this.delayRangeMs)+this.minDelayMs);
  // delay = 1;

  // Make request after timeout
  console.log("Making request to " + that.url + " in " + delay + "ms.");
  setTimeout(function() {
    try
    {
      // Helper function make HTTP Request.,  source @ https://github.com/redline13/harness-custom-test-nodejs/blob/master/CustomTestSimple.js#L78
      that.loadPage(that.url, function( failed ) {
        // Iteration complete
        that.remainingIterations--;

        // Another iteration?
        if (!failed && that.remainingIterations > 0)
          that.runTest(redlineCallback);
        else
          // Callback
          redlineCallback(failed, 'on-load-page' );
      });
    } catch (e) {
      // Callback
      console.log(e);
      that.redlineApi.recordError(""+e + (e.stack ? "\n" + e.stack : ""));
      redlineCallback(true, 'on-exception' );
    }
  }, delay);
};

This file should export a class using module.exports = YourCustomTestClassName;.

// Required Export for Test to be loaded and executed.
module.exports = MyCustomTest;

An instance of this class will be started for each user specified in the “Number of User” form field. To save CPU usage on the load agents, each Node.js process will run multiple tests (i.e., will instantiate multiple instances of your class).  This is important if you are using any global variables as they could be shared, best to keep things at the scope of the instance.

Packaging your test

A test can be just uploaded for quick testing, however you should create a .tar or .tgz file that contains CustomTest.js and optionally package.json in the root directory.  This is the file that should be uploaded to start the test.  The package.json file will be used prior to the test with npm install package.json to install any extra libraries you may need.  You can also include other helper classes in the tar file.

Recording methods available

The RedLine13 API contains the following methods:

  • redlineApi.recordPageTime(ts, time) – This records and aggregates the total page load time for a request, with ts being the UNIX timestamp when the request started and time being the time to complete the request. This function is used in the UI to report the overall average response time. The goToUrl method of LoadTestingSession calls this for you.
  • redlineApi.recordURLPageLoad(url, ts, time, err, kb, rc, user) – This records and aggregates the total page load time for a request to a specific URL, with ts being the UNIX timestamp when the request started and time being the time to complete the request.  You can also pass in err to record an error and kb to record the number of kilobytes downloaded.  The response code can be specified using rc, and the user identifier can be passed as user.  This function is used in the UI to report the per page average response time but does not affect the overall response time that is calculated.  The goToUrl and loadResources methods ofLoadTestingSession calls this for you.
  • redlineApi.recordDownloadSize(kb) – This records the number of kilobytes downloaded for a request. The goToUrl and loadResources methods of LoadTestingSession calls this for you.
  • redlineApi.recordError(error) – This records an error that will be displayed in the UI.
  • redlineApi.recordProgress(testNum, percent) – This records the progress (between 0 and 100) of a single test. This allows you to enable more accurate progress than the default, which is based only on the number completed vs. the test size.

LoadTestingSession Helper Class

You may wish to use the included LoadTestingSession(testNum, redlineApi, loadResources) class.  Simply call the constructor and then call loadPage, passing in a callback function.  The callback will receive a single parameter of true if the test failed and false otherwise.

Tips

  • To make better use of NodeJs and invoke test in Async use a setTimeout call, as shown in the example above.
  • You can use the random string that is passed in as the second parameter to your CustomTest constructor to append to URLs.  This can help you to easily identify load testing requests in your server logs.
  • If you need additional resources for your test, you can download a file (e.g., a .tgz file) with the resources in the CustomTest constructor.  You should include code to ensure that only one test per server downloads the files.  If you know the number of tests per server, you can use the test number to decide which test should download the files.  Alternatively, you can attempt to create a lock file, with the first process successfully creating the lock file doing the file download.

Complete Examples

You can find an example CustomTestSimple.js, which performs a Simple URL test with echo.