Background
VAN IN is the largest educational company in Belgium. They develop innovative learning solutions for students and teachers from kindergarten through higher education. VAN IN is part of the European group Sanoma Learning, a division of Sanoma.
The main platform in Primary Education, Bingel, guides pupils in their learning to reach their true potential, and gives the support teachers need to shine in front of the classroom. It is developed by teams in Belgium, Poland and The Netherlands, and available in different countries in Europe. Usage has grown tremendously due to COVID.
Tom Acke is a Software tester at VAN IN and is one of those responsible for load testing the Bingel application.
Objective
The primary objectives VAN IN sought was running load tests to verify that the platform can handle the load for every new release. This usually occurs weekly. Their load test consists of one large test script, which takes about 3-4 hours.
In addition to that, VAN IN wanted to automate their processes.
Solution
VAN IN chose to write their load tests in Apache JMeter and run them on RedLine13 for each new release. Here are the challenges and details of the solution.
Automation
VAN IN has automated their processes for developers to run load tests. Whenever a new release candidate reaches the pre-production environment, a bitbucket pipeline is triggered by the developer responsible for that release candidate. This pipeline will extract 20,000 random users from the database which will then be fed to the JMeter script in the form of a csv file. Whenever a load test is started, a message will be sent in the general communication platform to inform all users that the pre-production environment will be under increased load.
Datadog Monitoring
VAN IN uses Datadog for monitoring of their application. Datadog is a monitoring and security platform for cloud applications. It brings together end-to-end traces, metrics, and logs to make applications, infrastructure, and third-party services observable. These capabilities help businesses secure their systems, avoid downtime, and ensure customers are getting the best user experience.
There are dedicated dashboards available in Datadog where all desired metrics are combined. In the past it was always difficult for VAN IN to bundle all metrics in order to compare the metrics of every load test run. Their current dashboard contains the following metrics: DB connections, DB system cpu, application server CPU / memory / instance scaling and application server logging.
At the end of every load test run, Datadog metrics are compared against previous runs and their benchmark. Additionally the RedLine13 metrics (average response times / errors ) are verified to make sure no anomalies occurred during the tests.
One example of how RedLine13 helped was when a load test uncovered a query to their database that wasn’t indexed correctly. The test failed. Using database monitoring helped them uncover the cause before it went into production.
Server Sizes
In the beginning, VAN IN ran multiple tests with different server sizes before they selected the right server size. They experimented a lot as there is no clear guide on how to determine these server sizes. They read about test optimization, for instance, disabling Jmeter samplers, and that eventually led to a stable test server. They mainly ran into memory limitations of the test servers, which is why they ended up running their tests on a memory optimized server: r5.12xlarge. There is a post which walks through in detail how RedLine13 customers can select the right server.
Pass/Fail
Another challenge that VAN IN worked through was that so many groups were involved in interpreting the results that it made it difficult for all to agree on results (e.g., What is an acceptable number of errors?). They have about 20 developers of which any can be responsible for the next release and thus also following up the load tests on our pre-production environment. They used to have an average number of errors which occurred but due to changes within their application, these can also increase over time. Currently their runs have about 2,000 – 4,000 errors on 15,444,964 executed requests. They still have technical tasks running in their teams to reduce this number as close to zero as possible as this will make it easier to flag a test run as ‘failed’.
Results
VAN IN has used RedLine13 since 2015 and has run over 2,000 tests on RedLine13.
In summary from the customer:
In addition to our performance testing framework, RedLine13 provides an easy way to manage our tests, monitor progress and analyze results. You can even modify and re-run any previous test in just a few short clicks. RedLine13 has proved to be a very reliable tool and is an essential part of our performance testing framework.