{"id":6934,"date":"2019-10-04T16:50:48","date_gmt":"2019-10-04T20:50:48","guid":{"rendered":"https:\/\/www.redline13.com\/blog\/?p=6934"},"modified":"2019-10-04T16:50:48","modified_gmt":"2019-10-04T20:50:48","slug":"advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike","status":"publish","type":"post","link":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/","title":{"rendered":"Advanced Load Testing Troubleshooting &#8211; the Case of the Last Minute Spike"},"content":{"rendered":"<p>Anyone who has done load testing knows it\u2019s a lot more than creating a script and running it. There are going to be times when it requires what we call \u2018Advanced Load Testing Troubleshooting\u2019.<\/p>\n<p>Here is a real case of a RedLine13 customer who was trying to figure out what was causing an unusual load testing graph result. The issue was the graph Percentiles per Page Response Time graph. But sometimes the problem we see is not the real problem. Let&#8217;s not get ahead of ourselves. So let the troubleshooting begin.<\/p>\n<h3>The (Perceived) Problem<\/h3>\n<p>For all the tests the company was running, they saw the behavior that at the end of the test at exactly the last 1 minute with a sudden spike in the graph.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"971\" height=\"361\" class=\"wp-image-6973\" src=\"http:\/\/54.161.25.214\/wp-content\/uploads\/2019\/10\/word-image-26.png\" srcset=\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-26.png 971w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-26-300x112.png 300w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-26-768x286.png 768w\" sizes=\"auto, (max-width: 971px) 100vw, 971px\" \/><\/p>\n<p>Their understanding of the above graph behavior is that, since they stop the test suddenly at the end of the test, there may be some requests being sent but did not receive the response due to that spike is happening. Their questions:<\/p>\n<ol>\n<li>Would this sudden spike cause any effect on the overall tests response times (they consider 90 Percentile values)?<\/li>\n<li>Is there a way they could get response times table values by eliminating the last 5 minutes?<\/li>\n<\/ol>\n<p>Our initial response was that they could do this manually:<\/p>\n<ul>\n<li>Download the merged .JTL results<\/li>\n<li>Strip out last 5 minutes<\/li>\n<li>Reimport into JMeter directly<\/li>\n<\/ul>\n<p>But we investigated a little more. We saw how their reference test had a spike in the response at the very end of their test. We saw it in the JMeter JTL file and our results.<\/p>\n<p>Despite the fact that the customer said they \u201cstop the test suddenly at the end of the test&#8221; it is odd that this would cause the result time at the end to spike, since JMeter JTL is written at each response. If the test ends abruptly we would expect the actual result not to be written.<\/p>\n<p>So we dug into their JTL focusing on just one endpoint for no particular reason except it helps dig deeper.<\/p>\n<p>From the graph we show (and the standard JMeter generated graphs) you would think it is end of test.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"592\" height=\"402\" class=\"wp-image-6975\" src=\"http:\/\/54.161.25.214\/wp-content\/uploads\/2019\/10\/word-image-28.png\" srcset=\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-28.png 592w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-28-300x204.png 300w\" sizes=\"auto, (max-width: 592px) 100vw, 592px\" \/><\/p>\n<p>But when we hit toggle average and looked at the sum line, something looked odd.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-6937\" src=\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-2.png\" alt=\"Load Testing Troubleshooting\" width=\"1008\" height=\"620\" \/><\/p>\n<p>Their total time\/throughput is higher on rampup and rampdown. So, using their JTL we stripped out just the results for this endpoint and generated a graph &#8211; but instead of 1 minute aggregation we used 10ms.<\/p>\n<h3>Load Testing Troubleshooting Diagnosis<\/h3>\n<p>With the graph below we could see they had micro-performance issues through out the test. Since their test is large enough, they get averaged out at per minute and even per 10s.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-6938\" src=\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3.png\" alt=\"Troubleshooting Diagnosis\" width=\"2468\" height=\"1522\" srcset=\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3.png 2468w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3-300x185.png 300w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3-1024x631.png 1024w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3-768x474.png 768w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3-1536x947.png 1536w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3-2048x1263.png 2048w, https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/word-image-3-1568x967.png 1568w\" sizes=\"auto, (max-width: 2468px) 100vw, 2468px\" \/><\/p>\n<p>Our conclusion was that their 99% percentile is accurate and it is not a last minute issue. But these tiny issues during the test were the problem. With this information, the customer was able to identify where their real problems were. It was not a last minute issue.<\/p>\n<p>Another case of advanced load testing troubleshooting solved.<\/p>\n<p>This difficulty in being able to see the problem was caused by the granularity of the report. Here&#8217;s another post that talks about <a href=\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-granularity\/\">Advanced Load Testing &#8211; Granularity<\/a>.<\/p>\n<div class=\"entry-content\">\n<div class=\"entry-the-content\">\n<p>You can try your own test, whether it is simple or advanced, on <a href=\"https:\/\/www.redline13.com\/Service\">RedLine13<\/a>\u00a0for free.<\/p>\n<\/div>\n<\/div>\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Anyone who has done load testing knows it\u2019s a lot more than creating a script and running it. There are going to be times when it requires what we call \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer who was trying to figure out what was causing an unusual load testing graph result. The issue was the graph Percentiles per Page Response Time graph. But sometimes the problem we see is<a class=\"more-link\" href=\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\">Read More &rarr;<\/a><\/p>\n","protected":false},"author":1,"featured_media":6939,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,1],"tags":[165,321,520],"class_list":{"0":"entry","1":"post","2":"publish","3":"author-user","4":"post-6934","6":"format-standard","7":"has-post-thumbnail","8":"category-blog","9":"category-uncategorized","10":"post_tag-customer","11":"post_tag-load-testing-graph","12":"post_tag-troubleshooting"},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.12 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Advanced Load Testing Troubleshooting - RedLine13<\/title>\n<meta name=\"description\" content=\"Load testing often requires \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer with an unusual load testing graph result.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Advanced Load Testing Troubleshooting - RedLine13\" \/>\n<meta property=\"og:description\" content=\"Load testing often requires \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer with an unusual load testing graph result.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\" \/>\n<meta property=\"og:site_name\" content=\"RedLine13\" \/>\n<meta property=\"article:published_time\" content=\"2019-10-04T20:50:48+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/Advanced-Troubleshooting.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1008\" \/>\n\t<meta property=\"og:image:height\" content=\"620\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"RedLine13\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"RedLine13\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\"},\"author\":{\"name\":\"RedLine13\",\"@id\":\"https:\/\/www.redline13.com\/blog\/#\/schema\/person\/4acbcdcb8a9c72ec5a274e69c0ebea28\"},\"headline\":\"Advanced Load Testing Troubleshooting &#8211; the Case of the Last Minute Spike\",\"datePublished\":\"2019-10-04T20:50:48+00:00\",\"dateModified\":\"2019-10-04T20:50:48+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\"},\"wordCount\":553,\"publisher\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/#organization\"},\"keywords\":[\"customer\",\"Load Testing Graph\",\"Troubleshooting\"],\"articleSection\":[\"Blog\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\",\"url\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\",\"name\":\"Advanced Load Testing Troubleshooting - RedLine13\",\"isPartOf\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/#website\"},\"datePublished\":\"2019-10-04T20:50:48+00:00\",\"dateModified\":\"2019-10-04T20:50:48+00:00\",\"description\":\"Load testing often requires \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer with an unusual load testing graph result.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.redline13.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Advanced Load Testing Troubleshooting &#8211; the Case of the Last Minute Spike\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.redline13.com\/blog\/#website\",\"url\":\"https:\/\/www.redline13.com\/blog\/\",\"name\":\"RedLine13\",\"description\":\"(Almost) Free Load Testing in the Cloud\",\"publisher\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.redline13.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.redline13.com\/blog\/#organization\",\"name\":\"RedLine13\",\"url\":\"https:\/\/www.redline13.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.redline13.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2013\/06\/cropped-rl13-header-logo.jpg\",\"contentUrl\":\"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2013\/06\/cropped-rl13-header-logo.jpg\",\"width\":300,\"height\":68,\"caption\":\"RedLine13\"},\"image\":{\"@id\":\"https:\/\/www.redline13.com\/blog\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.redline13.com\/blog\/#\/schema\/person\/4acbcdcb8a9c72ec5a274e69c0ebea28\",\"name\":\"RedLine13\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.redline13.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b4c9a289323b21a01c3e940f150eb9b8c542587f1abfd8f0e1cc1ffc5e475514?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b4c9a289323b21a01c3e940f150eb9b8c542587f1abfd8f0e1cc1ffc5e475514?s=96&d=mm&r=g\",\"caption\":\"RedLine13\"},\"sameAs\":[\"http:\/\/127.0.0.1\"],\"url\":\"https:\/\/www.redline13.com\/blog\/author\/user\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Advanced Load Testing Troubleshooting - RedLine13","description":"Load testing often requires \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer with an unusual load testing graph result.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/","og_locale":"en_US","og_type":"article","og_title":"Advanced Load Testing Troubleshooting - RedLine13","og_description":"Load testing often requires \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer with an unusual load testing graph result.","og_url":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/","og_site_name":"RedLine13","article_published_time":"2019-10-04T20:50:48+00:00","og_image":[{"width":1008,"height":620,"url":"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2019\/10\/Advanced-Troubleshooting.png","type":"image\/png"}],"author":"RedLine13","twitter_card":"summary_large_image","twitter_misc":{"Written by":"RedLine13","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/#article","isPartOf":{"@id":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/"},"author":{"name":"RedLine13","@id":"https:\/\/www.redline13.com\/blog\/#\/schema\/person\/4acbcdcb8a9c72ec5a274e69c0ebea28"},"headline":"Advanced Load Testing Troubleshooting &#8211; the Case of the Last Minute Spike","datePublished":"2019-10-04T20:50:48+00:00","dateModified":"2019-10-04T20:50:48+00:00","mainEntityOfPage":{"@id":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/"},"wordCount":553,"publisher":{"@id":"https:\/\/www.redline13.com\/blog\/#organization"},"keywords":["customer","Load Testing Graph","Troubleshooting"],"articleSection":["Blog"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/","url":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/","name":"Advanced Load Testing Troubleshooting - RedLine13","isPartOf":{"@id":"https:\/\/www.redline13.com\/blog\/#website"},"datePublished":"2019-10-04T20:50:48+00:00","dateModified":"2019-10-04T20:50:48+00:00","description":"Load testing often requires \u2018Advanced Load Testing Troubleshooting\u2019. Here is a real case of a RedLine13 customer with an unusual load testing graph result.","breadcrumb":{"@id":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.redline13.com\/blog\/2019\/10\/advanced-load-testing-troubleshooting-the-case-of-the-last-minute-spike\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.redline13.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Advanced Load Testing Troubleshooting &#8211; the Case of the Last Minute Spike"}]},{"@type":"WebSite","@id":"https:\/\/www.redline13.com\/blog\/#website","url":"https:\/\/www.redline13.com\/blog\/","name":"RedLine13","description":"(Almost) Free Load Testing in the Cloud","publisher":{"@id":"https:\/\/www.redline13.com\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.redline13.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.redline13.com\/blog\/#organization","name":"RedLine13","url":"https:\/\/www.redline13.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.redline13.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2013\/06\/cropped-rl13-header-logo.jpg","contentUrl":"https:\/\/www.redline13.com\/blog\/wp-content\/uploads\/2013\/06\/cropped-rl13-header-logo.jpg","width":300,"height":68,"caption":"RedLine13"},"image":{"@id":"https:\/\/www.redline13.com\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/www.redline13.com\/blog\/#\/schema\/person\/4acbcdcb8a9c72ec5a274e69c0ebea28","name":"RedLine13","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.redline13.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b4c9a289323b21a01c3e940f150eb9b8c542587f1abfd8f0e1cc1ffc5e475514?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b4c9a289323b21a01c3e940f150eb9b8c542587f1abfd8f0e1cc1ffc5e475514?s=96&d=mm&r=g","caption":"RedLine13"},"sameAs":["http:\/\/127.0.0.1"],"url":"https:\/\/www.redline13.com\/blog\/author\/user\/"}]}},"_links":{"self":[{"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/posts\/6934","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/comments?post=6934"}],"version-history":[{"count":0,"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/posts\/6934\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/media\/6939"}],"wp:attachment":[{"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/media?parent=6934"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/categories?post=6934"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.redline13.com\/blog\/wp-json\/wp\/v2\/tags?post=6934"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}