Aaron Mirsky of NICE Actimize recently discussed their performance testing evolution. Aaron is a Senior Software Architect at NICE Actimize which provides trading surveillance solutions that address such concerns as payment fraud and cybercrime for their customers worldwide.
NICE Actimize helps financial institutions keep a step ahead of financial crime in an environment where monetary transactions can be highly time-sensitive. This made the performance of NICE Actimize software very important. They went from using a stopwatch to Selenium to Apache JMeter in their efforts to accurately performance test their applications. One key step involved identifying user interface requests and emulating the log of associated RESTful APIs within a JMeter test plan. They further wanted to automate this test at scale, which involved simulating a high number of users and terabytes of data by adding RedLine13 to their environment to run JMeter tests in the cloud. In addition, they integrated Splunk to achieve highly granular analysis of their test and target application.
Performance test current and future applications at scale.
NICE Actimize helps financial institutions assess the risks they face, stay compliant with ever-changing regulations, and keep a step ahead of financial crime. Today over four billion transactions are monitored every single day – protecting an estimated six trillion dollars in assets. They have over 750 clients representing major banks and institutions using their software either via the NICE Actimize cloud services or installed on-prem in their internal networks.
All those clients have thousands to millions of their own customers regularly making financial transactions. If a transaction is flagged by the NICE Actimize detection software, a Work Item is generated for investigation via a web-based UI called ActOne. The transaction may be frozen until a client’s investigation and customer service teams can verify the legitimacy of the exchange. For certain environments, a decision must be made by the investigator within minutes. The investigator needs many data inputs from a variety of sources in order to facilitate that decision. Therefore, a lot of effort is made to visually optimize and organize all the information the investigator needs – and what goes with that is the performance impacts of retrieving and displaying all that data.
A while ago there were not many effective tools available, and NICE Actimize initially started their performance testing the web UI with a person holding a stopwatch running through a manual test plan. To achieve automation, NICE Actimize used Selenium to create functional user interface tests for their web applications. Selenium lends well to team members of all skill levels with its ease of use. Setting up a Selenium test can be as simple as recording a sequence of events in the web browser, and then replaying those events.
While performance tests involving a single user have their utility in testing the user experience, it is not a tool that is optimal for load testing at scale. The primary reason for this is how the Selenium architecture works. Running a Selenium-based test involves the performance overhead of loading an entire web browser and all of its associated dependencies. While a single machine can handle one or more instances, the performance requirements rapidly increase when testing dozens to hundreds of users. With even many small scale load tests frequently testing in the hundreds to thousands of users, the computational resources to run Selenium load tests rapidly burgeons.
It was also realized that a front-end performance test via the browser can indeed give a picture of the wait-time a real user can experience. For R&D however, it hides the source of performance bottlenecks. Latency can happen in different parts (browser, transit, application, DB), but NICE Actimize only had a global number. The software was transitioning from back-end generated HTML to front-end frameworks with much more vibrant and complex displays. Developers needed to know if performance bottlenecks lay in the backend processing or front-end display. To do this, accurate measures of backend interactions were needed at load so to differentiate from overall time captured in the UI.
Performance Testing Cost
Originally the performance testing was done within the company IT with Virtual Machines. Aside from the cost of maintaining the computing resources, a performance test needed to run on data sets for potentially 100s of millions of rows of data. These data sets were preserved as data dumps so the test scenario could be reset. But the process was slow. A full database restore process and preparation took over 24 hours.
Features could not be performance tested until stabilized in by quality testing and and full regression performance testing only happened when the main source branch had all features in and stable. This means often performance testing happened late and close to the release. If a performance issue was found, each re-test for debugging or checking fixes often necessitated that data reset – meaning repeated 24-hour waits with a release deadline already alarmingly close.
New features bring new bugs, which includes performance. A big feature can impact other seemingly unrelated functionality in unexpected ways. In addition, smaller isolated bug fixes and patch foldings are happening during the development process. When performance testing happens near the end of a cycle, it can at times be hard to figure out what code change is the potential culprit. But looking back in history from the end is to look at all code changes that happened since the beginning.
A More Efficient Approach to Testing
The big change was to automate the performance testing process and do it in the cloud. The test data was migrated to AWS AMIs and the schema data reset shrunk from 24 hours to four minutes. ActOne supports containerization and so they deployed test instances over EKS. The entire process of build, prepare data and deployments was managed by a Jenkins pipeline. Now at the click of a button, the process from clean build to a ready test environment took under an hour.
The next phase was to automate the actual performance test and collect the results.
The solution was to adopt JMeter. At a high level, this involved identifying user interface requests and emulating the log of associated RESTful APIs within a JMeter test plan. NICE Actimize went through their applications and isolated their UI requests from backend requests to their RESTful APIs. They learned that if you need to run your test at any significant scale, this is an efficient way to accomplish this. The result is that an actual user session is little more than a series of ordered requests and responses. By isolating these requests and scaffolding them into a framework that is suitable to emulate them, it is possible to run a similar scenario outside the browser. This is the premise of a JMeter test, and it is an effective way to take an equivalent of a Selenium test to scale.
They then combined these requests with logic inside their JMeter test plan to emulate user workflows on their website. Without the overhead of the full browser, it was possible to simulate far more users on a single load generator.
With the backend capture with JMeter reflecting specific user scenarios, NICE Actimize had the performance numbers of the backend versus overall which can then determine the front-end. Regression tests done on both helped quickly identify if performance degraded in the browser or in the RESTful APIs.
Running a load test in JMeter is usually a straightforward endeavor, especially once your test plan has been built and validated. However, to achieve additional scale with your test in JMeter alone becomes exponentially more difficult. Running your test on multiple machines simultaneously is the most obvious solution. To achieve this would require the infrastructure to manage those instances and synchronize test execution. CTO Efim Dimenstein had used RedLine13 previously and recommended it to the team. Once their test was proved on a single instance of JMeter, they would increase the number of load generators within RedLine13 to scale the test.
You can read more about how to scale on RedLine13 in this related blog post.
As the process is now automated, instead of an email from the performance tester, it made sense for the data collection and visualization to be automatic as well. NICE Actimize , was able to integrate a Splunk agent into those instances to achieve advanced instrumentation for their tests. Splunk is a framework which facilitates collection of data metrics from network-based applications. Splunk pulls data from builds, Jenkins data, and other sources to give them a visual view. NICE Actimize built a tool to pull the JMeter results to build a simplified JSON and feed it into Splunk.
NICE Actimize included the Splunk agent in their containerized AMI to instrument their target test application. This allows them to visualize performance metrics which complement their test results from RedLine13.
RedLine13 offers a full-featured free trial. You can sign up here.