{"id":8333,"date":"2017-02-24T17:25:23","date_gmt":"2017-02-24T17:25:23","guid":{"rendered":"http:\/\/www.abstracta.us\/?p=8333"},"modified":"2025-05-05T21:20:55","modified_gmt":"2025-05-05T21:20:55","slug":"best-time-start-performance-testing","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/","title":{"rendered":"When is the Best Time to Start Performance Testing?"},"content":{"rendered":"<p><!-- Go to www.addthis.com\/dashboard to customize your tools --><script src=\"\/\/s7.addthis.com\/js\/300\/addthis_widget.js#pubid=ra-58d80a50fc4f926d\" type=\"text\/javascript\"><\/script><\/p>\n<h1><span style=\"font-weight: 400; color: #333333;\">Should we test performance throughout development or at the end? &#8211; Agile vs. Waterfall approaches<\/span><\/h1>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">When taking into account the performance of existing systems or ones built from scratch, teams have to determine at what point in the development process they are going to benefit most from running performance tests. I\u2019ve spoken about this topic at a couple of conferences, including the CMG imPACt Conference in La Jolla alongside Sof\u00eda Palamarchuk, and I thought, \u201cWhy not summarize the talk in a post?\u201d So, the purpose of this post is to answer the question: <\/span><\/span><\/p>\n<p><span style=\"color: #333333;\"><b>Should we start performance testing at the beginning, alongside development (taking the agile approach) or at the end (the waterfall approach)?<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">First, a quick overview of what performance testing implies:<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">A computer\u2019s performance is characterized by the amount of useful work accomplished by a computer system compared to the time spent and resources used. Remember, performance doesn\u2019t only refer to speed. For example, a system that is very fast that uses 100% of the CPU is not performant. Therefore, it is important that we check both the user experience (the perceived response time, or the velocity) and how stressed the servers are. Furthermore, if we only pay attention to response times, we could only be seeing symptoms when what we really want to find are the underlying root causes in order to identify those bottlenecks and then the ways in which we could improve.<\/span><\/p>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">Now, back to the question at hand: Should we take the waterfall or agile performance testing approach? <\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">To clarify what we mean by the two, the <strong>agile approach<\/strong> is when we start performance testing at the beginning of the development process and continue with it along with the whole evolution of the application. <\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">The <strong>waterfall approach<\/strong> is when we leave all the performance testing activities for the end of development, as acceptance testing, checking that the system performs as needed. <\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Let\u2019s take a look at what they entail and then the advantages and disadvantages of each.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"The_Waterfall_Approach\"><\/span>The Waterfall Approach<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">In this approach, we usually wait until the end of development to start testing. The performance tests take the form of acceptance testing and if the criteria are met, the system is ready to go into production. This involves simulating the expected load scenario. <\/span><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Pros\"><\/span><strong><span style=\"color: #3056a2;\">Pros<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><span style=\"font-weight: 400; color: #333333;\">It is easier to plan and assign resources for performance testing since you are only doing it for a designated period of time.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Typically, you try to use a test environment as similar to that of production as possible, which is beneficial because it is more realistic. <\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">It allows for the focus on specific characteristics to test (like X amount of functionalities under a specific context). <\/span><\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Cons\"><\/span><strong><span style=\"color: #3056a2;\">Cons<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">While it\u2019s great that we are testing in a similar environment to that of production, at the same time it can be difficult to obtain the infrastructure <\/span><span style=\"font-weight: 400;\">for an exclusive usage for the tests (you need to isolate the SUT in order to have trustworthy result<\/span><span style=\"font-weight: 400;\">s.<\/span><\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">The cost of making architectural changes toward the end of development (if testing discovers that it\u2019s necessary) is high.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">There is a high risk that comes with waiting to assure performance at the very end because you never know how much work you will have in front of you to go back and fix things in order to reach your performance goals. <\/span><\/li>\n<\/ul>\n<h2><span class=\"ez-toc-section\" id=\"The_Agile_Approach\"><\/span>The Agile Approach<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">The agile approach to performance testing entails starting testing from the very beginning with unit tests. It is key to have a continuous integration environment in place and what happens is that instead of merely carrying out <\/span><b>performance testing<\/b><span style=\"font-weight: 400;\">, we carry out <\/span><b>performance engineering<\/b><span style=\"font-weight: 400;\">. <\/span><\/span><\/p>\n<h3><span class=\"ez-toc-section\" id=\"Pros-2\"><\/span><strong><span style=\"color: #3056a2;\">Pros<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><span style=\"font-weight: 400; color: #333333;\">Minimize risk.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Obtain early, constant feedback.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Learn about best practices as you go and over time, continuously improve. When you start testing early, if you do something wrong, you have time to catch your mistake and avoid making it again. It\u2019s great for reducing the likelihood of spreading bad practices all over the system.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Facilitate continuous integration. <\/span><\/li>\n<\/ul>\n<h3><span class=\"ez-toc-section\" id=\"Cons-2\"><\/span><strong><span style=\"color: #3056a2;\">Cons<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h3>\n<ul>\n<li><span style=\"font-weight: 400; color: #333333;\">It requires more automation effort in writing and maintaining scripts. <\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Problems may arise if you automate too little or too much at certain levels. For example, it\u2019s best to automate as many unit performance tests as possible, have some at the API level, and only automate the most critical test cases at the GUI level. This is in accordance with Michael Cohn\u2019s automation pyramid idea but applied to performance. Be aware that you will have to ponder what a performance unit test is in your case. <\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Sometimes teams fail to recognize that it\u2019s a fallacy that if you test components separately, the system will work properly. That\u2019s not necessarily true. You need to test the components individually and then, test them working together in order to achieve the best results. <\/span><\/li>\n<\/ul>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">When choosing between these two approaches, it is first important to take stock of the <\/span><b>people, technology, and processes<\/b><span style=\"font-weight: 400;\"> with which you have to work. It is important to have testers with both the proper soft and hard skills for performance testing. <\/span><\/span><span style=\"font-weight: 400;\"><span style=\"color: #333333;\">You also have to take into consideration which tools to use when load testing (ie, JMeter, BlazeMeter, Gatling, etc.) and monitoring both on the server side (ie, New Relic, NMON, perfmon, etc.) as well as the client side (with tools like <a href=\"https:\/\/bit.ly\/2zi111K\" target=\"_blank\" rel=\"noopener noreferrer\">Apptim<\/a>, Page Speed, and Yslow,<\/span><\/span><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">).<\/span><span style=\"font-weight: 400;\"> Processes include test design, test automation, test execution, and measurement. When coming up with an execution plan, we recommend testing against a baseline and then use an iterative, incremental approach (I\u2019ll write a follow-up post on what that means soon!).<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">So, which approach is right for you? Well, it depends what your desired outcome is.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"When_to_Go_with_the_Waterfall_Approach\"><\/span>When to Go with the Waterfall Approach<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">You may want to run a load simulation at the end when:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400; color: #333333;\">You need to verify that your existing system supports a certain load.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Your customers require evidence that your system meets a certain standard for performance (for example, if your client is a bank and it wants to make sure that its online banking system can support 100,000 daily users).<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">If you suspect that certain tuning is required for the specific context where the application will run.<\/span><\/li>\n<\/ul>\n<h2><span class=\"ez-toc-section\" id=\"When_to_Go_with_the_Agile_Approach\"><\/span>When to Go with the Agile Approach<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">You may need to take this approach that entails performance engineering throughout development when:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400; color: #333333;\">If you want to reduce risk and costs.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">You want to increase the team\u2019s collective knowledge of performance engineering and monitoring since they learn about it during the whole process.<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">When your goal is to follow a continuous integration scheme.<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400; color: #333333;\">Can we definitively claim that one approach is better than the other in every instance?<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">No.<\/span><\/p>\n<p><span style=\"color: #333333;\"><b>We need both approaches in different stages of our development cycle. <\/b><span style=\"font-weight: 400;\">We should start early by doing performance engineering and we also need to simulate load for acceptance testing.<\/span><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">And, in reality, the two approaches are not all that different. Both require the same use of people, technology, and processes, but they slightly vary depending on how far along in development you may be.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"Choosing_Between_the_Two_in_Real_Life\"><\/span>Choosing Between the Two in Real Life<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">At Abstracta, many of our clients come to us asking to take the waterfall approach, with the intention of running load simulations and acceptance tests before go-live of a new version of their system or after making a change to their architecture, etc. One instance in which we\u2019ve done this was for a financial institution that had recently merged with another, and it needed to assure that after having doubled the number of accounts in its internet banking system, the performance wouldn\u2019t suffer. <\/span><\/p>\n<p><span style=\"color: #333333;\"><span style=\"font-weight: 400;\">Other clients have taken the performance engineering route, such as the e-commerce giant, Shutterfly, who runs performance tests continuously. This enables them to have a continuous integration environment, releasing updates frequently in order to enhance the user experience without allowing performance degradations. Read more about Shutterfly\u2019s continuous performance testing scheme <a href=\"http:\/\/abstracta.us\/blog\/performance-testing\/shutterfly-continuous-performance-testing\/\"><span style=\"font-weight: 400;\">here<\/span><\/a><\/span><span style=\"font-weight: 400;\">.<\/span><\/span><\/p>\n<p><b><span style=\"color: #333333;\">If you are looking for more support, check out our other<\/span> <\/b><a href=\"http:\/\/www.abstracta.us\/category\/performance-testing\/\"><b>performance articles<\/b><\/a><span style=\"color: #333333;\"> <b>or <\/b><\/span><a href=\"http:\/\/www.abstracta.us\/contact-us\/\"><b>contact<\/b><\/a><span style=\"color: #333333;\"><b> one of our consultants today to start performance testing.<\/b><\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">What are your experiences with the two approaches?<\/span><\/p>\n<hr \/>\n<h2><span class=\"ez-toc-section\" id=\"Recommended_for_You\"><\/span>Recommended for You<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"color: #00b674;\"><a href=\"http:\/\/abstracta.us\/blog\/performance-testing\/why-performance-testing-is-necessary\/\"><span style=\"font-weight: 400;\">Why Performance Testing is Necessary<\/span><\/a><br \/>\n<\/span><\/p>\n<p><a href=\"http:\/\/abstracta.us\/blog\/performance-testing\/3-key-performance-testing-metrics-every-tester-should-know\/\">3 Metrics Every Performance Tester Should Know<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Should we test performance throughout development or at the end? &#8211; Agile vs. Waterfall approaches When taking into account the performance of existing systems or ones built from scratch, teams have to determine at what point in the development process they are going to benefit&#8230;<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32],"tags":[68],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>When is the Best Time to Start Performance Testing? | Abstracta<\/title>\n<meta name=\"description\" content=\"Should we start performance testing at the beginning, alongside development (taking the agile approach) or at the end (the waterfall approach)?\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"When is the Best Time to Start Performance Testing? | Abstracta\" \/>\n<meta property=\"og:description\" content=\"Should we start performance testing at the beginning, alongside development (taking the agile approach) or at the end (the waterfall approach)?\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-24T17:25:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:20:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2017\/02\/start-final.png\" \/>\n\t<meta property=\"og:image:width\" content=\"420\" \/>\n\t<meta property=\"og:image:height\" content=\"236\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@fltoledo\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/\",\"name\":\"When is the Best Time to Start Performance Testing? | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"datePublished\":\"2017-02-24T17:25:23+00:00\",\"dateModified\":\"2025-05-05T21:20:55+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/7421e539de0357d3adb0c69ed469a1c2\"},\"description\":\"Should we start performance testing at the beginning, alongside development (taking the agile approach) or at the end (the waterfall approach)?\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/performance-testing\/\",\"name\":\"Performance Testing\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/performance-testing\/best-time-start-performance-testing\/\",\"name\":\"When is the Best Time to Start Performance Testing?\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/7421e539de0357d3adb0c69ed469a1c2\",\"name\":\"Federico Toledo, Chief Quality Officer at Abstracta\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6de7ec6536c4028b5c02ad4ec1b9af0d?s=96&d=blank&r=g\",\"caption\":\"Federico Toledo, Chief Quality Officer at Abstracta\"},\"description\":\"Co-founder and COO of Abstracta\",\"sameAs\":[\"https:\/\/twitter.com\/fltoledo\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/8333"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=8333"}],"version-history":[{"count":21,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/8333\/revisions"}],"predecessor-version":[{"id":12628,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/8333\/revisions\/12628"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=8333"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=8333"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=8333"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}