{"id":15442,"date":"2022-11-09T18:09:12","date_gmt":"2022-11-09T18:09:12","guid":{"rendered":"http:\/\/abstracta.us\/blog\/?p=15442"},"modified":"2025-05-05T21:19:47","modified_gmt":"2025-05-05T21:19:47","slug":"mobile-application-testing-strategy-performance-validations-in-ci-cd","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/","title":{"rendered":"Mobile Application Testing Strategy: How to Integrate Mobile Performance Validations in a CI\/CD Pipeline"},"content":{"rendered":"\n

When it comes to creating quality software, it is essential to understand why it is important to integrate mobile performance validations in a CI\/CD pipeline, and specifically how to do it, as part of your Mobile Application Testing Strategy. Apptim has a CLI tool that allows you to run automated performance validations of your app, integrate to CI\/CD, and set pass\/fail criteria for the most important KPIs<\/a> that affect user experience. Find out more in this article, with an interview with Apptim\u2019s CEO, Sofia Palamarchuk. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The importance of Continuous Integration and Continuous Delivery in today’s world cannot be overstated.<\/strong> By doing this, IT professionals are able to create better software every second, which is crucial in the era in which software quality is critical.\u00a0<\/p>\n\n\n\n

It also deeply impacts the growth, sustainability, and scalability of companies<\/strong>. As you can see, it is key to take it in mind for your Mobile Application Testing Strategy.<\/strong><\/p>\n\n\n\n

According to \u201cThe Cost of Poor Software Quality in the US\u201d CISQ<\/a> report 2020, the total cost of poor software quality in the US was $2.08 trillion (T) in 2020.<\/strong><\/p>\n\n\n\n

Just as Abstracta\u2019s Chief Technology Officer Roger Abelenda<\/strong> said<\/a>, \u201cAn efficient continuous integration pipeline is key to creating quality software and remaining competitive in today\u2019s technological landscape. Why? Because it allows focusing on the changes delivered.<\/strong> Thus, it provides value to users sooner, trying ideas faster, avoiding regression problems, and streamlining software delivery\u201d.<\/p>\n\n\n\n

Long story short: when it comes to software testing, understanding how to integrate mobile performance validations into a CI\/CD pipeline is really relevant. And Apptim is a truly useful tool for achieving this goal.\u00a0<\/strong><\/p>\n\n\n\n

How Can Apptim Help With Your Mobile Application Testing Strategy?<\/strong><\/h4>\n\n\n\n
\"Mat\u00edas<\/figure>\n\n\n\n

Apptim has been our first spin-off at Abstracta! It is a mobile app performance testing solution used by more than 250 companies worldwide. One is Playtika, based in Israel and one of the world’s largest mobile gaming companies.\u00a0<\/strong><\/p>\n\n\n\n

As explained by Matias Reina<\/strong>, CEO of Abstracta, “Apptim measures device-side performance as opposed to server-side performance as most tools<\/strong>. It is true that some of these metrics are affected by the server, but that is not their focus”.<\/p>\n\n\n\n

In order to get a better understanding of this point, Matias exemplified that Apptim measures from the moment a user clicks on the search for products until all the products are actually displayed. “It involves two different processing methods, one on the server and one on the mobile phone.”\u00a0<\/strong><\/p>\n\n\n\n

Mat\u00edas went deeper: “The problem comes when you need to simulate a high demand, with large amounts of load and thousands of users, and you use tools like JMeter instead of Apptim. In these cases, the analysis is usually split in two: scripts\/automation focused on the server (with Jmeter) and scripts\/automation focused on the client (with Apptim).\u201d<\/p>\n\n\n\n

And he continued: “On the other hand, client-side processing depends a lot on the type of device, so it is good to be able to measure on different types of devices to ensure that all target users have a good experience.<\/strong>“<\/p>\n\n\n\n

With all this in mind, it is clear that Apptim can be a great ally when it comes to mobile performance testing, as part of your Mobile Application Testing Strategy. But how to automate these validations on mobile? And how to truly take advantage of all the benefits offered by Apptim? We talked about this and more with Sofia Palamarchuk, CEO at Apptim.<\/strong><\/p>\n\n\n\n

\"Sof\u00eda<\/figure>\n\n\n\n

– How can people automate mobile performance validations?<\/em><\/strong><\/h4>\n\n\n\n

Sof\u00eda Palamarchuk: <\/em><\/strong>You first need to find a way to run automated profiling of your app. The easiest way to get started is by reusing automated functional tests that have been created for functional validation purposes. These tests run at the UI level on a package version of the app that can be installed in a real device or emulator and simulate a real user journey. While this test runs, you can capture performance data of what\u2019s happening \u201cunder the hood\u201d.<\/p>\n\n\n\n

Most mobile teams already have some type of automated UI tests running in their CI pipeline. If your app is in beta or new to the market, you may be thinking about adding them in the near future. This is the best time to think about how to include mobile performance validations in your app release process, always focusing on your Mobile Application Testing Strategy. <\/p>\n\n\n\n

For example, here\u2019s what happens in an Appium<\/a> test that runs a typical user journey in an e-commerce app: a user searches for a product, selects the product from a list, adds it to a cart, navigates to the cart, and completes the checkout. This functional test might be checking to make sure that the correct product was added to the cart, the product quantity is correct, or that the checkout is working properly. At the same time, we validate what the response time is for a simple action like clicking the \u201cadd to cart\u201d button<\/a> as well as the memory usage if the action is performed several times. Will it cause an OutOfMemory error? <\/p>\n\n\n\n

If your team doesn\u2019t have any automated functional tests, we strongly recommend you automate a small and valuable use case to start measuring performance over time. For example, measure the app\u2019s startup time or test the login user experience. <\/p>\n\n\n\n

– Should people run these validations using emulators or real devices?<\/em><\/strong><\/h4>\n\n\n\n

Sof\u00eda Palamarchuk: <\/strong>It depends. If what you\u2019re most interested in is comparing or benchmarking different performance metrics of your app (like v2.6 versus v2.5), you should have test environments that are as similar as possible<\/a>. In particular, the devices used to test should be the same.

As part of your Mobile Application Testing Strategy, you\u2019ll want to minimize the noise in the data that comes with using different environments and look at differences in the measured performance on each version. For this purpose, emulators can be of great help because you can specify the hardware and OS version of the emulated device and use the same emulator for benchmarking. It\u2019s also a cost-effective alternative to using real devices if you run frequent benchmarks.\u00a0<\/p>\n\n\n\n

On the other hand, if you\u2019re looking to evaluate the real user experience, you need to be as close as possible to real-world conditions. This means testing on real hardware. In addition to looking for noticeable performance differences from one app version to the other, you\u2019ll want to make sure the app\u2019s performance is acceptable on specific devices.

You can do this by defining thresholds per device. For example, memory usage cannot be more than 300MB on a specific device. Or, you can get notified if the FPS is lower than 10 on any screen (and probably fail the build pipeline).<\/p>\n\n\n\n

– What pass and fail criteria should professionals use<\/em><\/strong> for their Mobile Application Testing Strategy? <\/em><\/strong><\/h4>\n\n\n\n

Sof\u00eda Palamarchuk: <\/strong>This is one of the most common questions we get asked and, arguably, the most difficult to answer. Google and Apple provide some best practices for pass\/fail criteria. For example, an app rendering at 60 FPS provides the best experience to the end user. Does this mean you have a performance issue if your app renders at 30 FPS? Well, it depends on what type of app you have. A mobile game or an app that has heavy graphics will have higher FPS requirements.

Transactional apps may not need high levels of FPS because knowing how fast certain transactions are completed is more important. Measuring<\/a> the end-user response time<\/a> of the login page or an action like adding an item to a cart is a good way to measure transaction speed.<\/p>\n\n\n\n

Our recommendation is to define pass\/fail criteria<\/a> with the whole development team as non-functional requirements. This can be the number of crashes or errors, the average percentage of CPU usage (like under 50%), or the app startup time (like under 3 seconds). The end goal is to have more confidence in the quality of every build. If you\u2019re meeting your pass\/fail targets every time, you\u2019ll have more certainty regarding the end-user experience.\u00a0<\/p>\n\n\n\n

– How can people use Apptim in CI\/CD?<\/em><\/strong><\/h4>\n\n\n\n

Sof\u00eda Palamarchuk: <\/strong>We have a CLI tool that allows mobile developers to run automated performance validations of their app, integrate to CI\/CD, and set pass\/fail criteria for the most important KPIs<\/a> that affect user experience. Those interested in a demo can reach out to our team at hello@apptim.com.<\/p>\n\n\n\n

Would you like to contact <\/strong>Apptim<\/strong><\/a>? Over 10,000 teams have used Apptim for mobile performance testing! <\/strong>Get in touch here<\/strong><\/a>.<\/strong><\/p>\n\n\n\n

Are you looking for a partner to perform software testing? Abstracta is one of the most reliable companies in software quality engineering. Get to know our solutions, and contact us to talk about how we can help you grow your business.<\/strong><\/p>\n\n\n\n

Follow us on <\/strong>Linkedin<\/strong><\/a> & <\/strong>Twitter<\/strong><\/a> to be part of our community!<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

When it comes to creating quality software, it is essential to understand why it is important to integrate mobile performance validations in a CI\/CD pipeline, and specifically how to do it, as part of your Mobile Application Testing Strategy. Apptim has a CLI tool that…<\/p>\n","protected":false},"author":61,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[79],"tags":[376,565],"yoast_head":"\nMobile Application Testing Strategy: Performance Validations in CI\/CD<\/title>\n<meta name=\"description\" content=\"Understand the importance of integrating performance validations in a CI\/CD pipeline, as part of your Mobile Application Testing Strategy.\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Mobile Application Testing Strategy: Performance Validations in CI\/CD\" \/>\n<meta property=\"og:description\" content=\"Understand the importance of integrating performance validations in a CI\/CD pipeline, as part of your Mobile Application Testing Strategy.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-09T18:09:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:19:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2022\/11\/How-to-Integrate-Mobile-Performance-Validations-in-a-CI-CD-Pipeline-01.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1250\" \/>\n\t<meta property=\"og:image:height\" content=\"705\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AbstractaUS\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/abstracta.us\/wp-content\/uploads\/2023\/05\/image-42-1024x682.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/\",\"name\":\"Mobile Application Testing Strategy: Performance Validations in CI\/CD\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/#primaryimage\"},\"datePublished\":\"2022-11-09T18:09:12+00:00\",\"dateModified\":\"2025-05-05T21:19:47+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/1bfcc322c93b05aad83d4c8c2b573a0c\"},\"description\":\"Understand the importance of integrating performance validations in a CI\/CD pipeline, as part of your Mobile Application Testing Strategy.\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/\",\"name\":\"Mobile Testing\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/\",\"url\":\"https:\/\/abstracta.us\/blog\/mobile-testing\/mobile-application-testing-strategy-performance-validations-in-ci-cd\/\",\"name\":\"Mobile Application Testing Strategy: How to Integrate Mobile Performance Validations in a CI\/CD Pipeline\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/1bfcc322c93b05aad83d4c8c2b573a0c\",\"name\":\"Natalie Rodgers, Marketing Team Lead at Abstracta\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/9a23da822367e20ddb98b59d5273eb3e?s=96&d=blank&r=g\",\"caption\":\"Natalie Rodgers, Marketing Team Lead at Abstracta\"},\"description\":\"Marketing Team Lead & AI SEO Specialist at Abstracta\",\"sameAs\":[\"https:\/\/www.linkedin.com\/in\/natalierodgersok\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/15442"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/61"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=15442"}],"version-history":[{"count":2,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/15442\/revisions"}],"predecessor-version":[{"id":15792,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/15442\/revisions\/15792"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=15442"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=15442"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=15442"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}