Can Auto Playwright Boost Testing with AI?

Learn all about Auto Playwright, an AI-enhanced evolution of Playwright for test automation. Get insights, use cases, and guides from Abstracta.

As software development teams continue to advance their automation strategies, new tools and approaches aim to reduce manual scripting and streamline test maintenance. Among these, AI-enhanced solutions have gained attention for their potential to interpret plain-language instructions and accelerate test creation.

One of the most popular tools in the test automation field is Playwright, which provides a solid foundation for robust and reliable testing, including the ability to create playwright api test cases efficiently.

It’s an open-source framework that facilitates the creation of automated tests for web applications and allows running them across various browsers.

With all the technological changes in the IT industry regarding Artificial Intelligence, Auto Playwright emerged. This tool is an evolution of Playwright, incorporating AI to take test automation to a new level by simplifying the way test code is generated and executed.

Does Auto Playwright improve test development efficiency? We answer this in detail in this article by conducting tests with real use cases and analyzing both its capabilities and limitations.

Explore our case studies and explore our AI powered test automation solutions!

Introducing Auto Playwright Tests

Auto Playwright leverages OpenAI‘s capabilities to transform plain text instructions into a series of function calls and function executions.

It acts as an intelligent intermediary between your plain text commands and the code needed to automate tests, making the process more intuitive and accessible.

This visionary initiative was inspired by the ZeroStep project, which offers a similar API but with a more robust implementation through its private platform. 

How Does Auto Playwright Work? 

First and foremost, you need to write tests in plain text. 

These instructions must describe the actions or checks you want to perform on your web application, like “click the login button” or “get the page title”. This makes it a valuable tool in web development workflows, where maintaining application integrity through automation is key to product quality.

For example, in a user registration process, you could automate the complete flow from opening the registration page to submitting the form and confirming successful account creation, using plain text prompts.

Then, you need to translate them using OpenAI’s runFunctions feature, which interprets these instructions.

Once it translates the instructions, Auto Playwright determines and executes the functions that allow performing the described actions. For instance, it could be clicking, filling out forms, navigating different pages, or searching and verifying content. From simple clicks to form submissions, it can mimic various user interactions required to validate front-end behavior accurately.

A common use case includes validating a search text input, where users type queries to filter or retrieve specific data within the application.

Here you can find all the actions Auto Playwright handles.

As these functions are executed, Auto Playwright provides information or results. The result is generated code that automates the actions described in natural language, streamlining test creation for developers and testers alike. This may include confirmation of successfully performed actions, data retrieved from the page, or the results of checks and assertions made.

Next, we’ll explore how to get started with the tool, from installation to running your first automated tests. We’ll also analyze the value its use would bring to your projects.

Don’t miss this webinar, with tips for low-code test automation 

Step by Step: Setting Up Auto Playwright from Scratch

Like any project based on Node, we need to meet certain prerequisites.

First, we need Node installed on our computer. Additionally, we require an Integrated Development Environment (IDE). In our case, we’ll use VSC. Once installed, we can start setting up Auto Playwright.

Next, we share a step-by-step guide to help you on this path

1. Create an API Key on OpenAI

OpenAI is an Artificial Intelligence research organization leading the development of advanced technologies like GPT-3 and DALL-E, focusing on promoting AI’s ethical and beneficial use.

To get an OpenAI API key, you must register on the OpenAI platform, create an account, and, in some cases, accept the terms of service.

Steps to Get an API Key:

• Visit https://openai.com.
• Log in or create a new account.
• Select “API.”
• Hover over the icons on the left to open the menu and select “API Keys.”
• Click on “Create new secret key.”
• Name the key and click on “Create secret key.”
• Generate the key.

One detail about this tool is that you must have available usage on your API Key to work with it.

Although it’s a free platform and you don’t need to subscribe to any plan, there is a cost associated with your API Key. Here you can find more pricing information.

2. Install the Playwright Framework from the Console 

For this, you should run: npm init playwright@latest

3. Install Auto-playwright

At this step, you ought to run npm install auto-playwright -D in the console.

At this point, we’ll see all the dependencies  including the Auto Playwright dependency in our package.json:

4. Set Variables 

In this last step, you need to set the environment variables in the .env file following the tool’s documentation.

Remember, this API key is the one generated in step N° 1: OPENAI_API_KEY=’sk-…”

These steps are essential to get the tool up and running, but in more advanced setups, additional authentication layers may be required.

Debugging and Flexibility in Test Execution

While writing effective tests is vital, understanding how they behave during execution is equally important. Auto Playwright supports this need with enhanced debugging features.

A standout tool is the playwright inspector, which provides a visual UI that displays each step in the test as it happens. This helps you catch subtle issues, like timing problems or misidentified elements, before they become bugs in production.

During a test run, Auto Playwright also prints logs and intermediate steps to the console, helping teams understand how plain-language prompts are converted into executable steps. It’s especially useful for analyzing failures, especially in flows with user inputs, like a registration form, or sequences that depend on specific user-provided data.

Moreover, when validating functionality like search behavior, you may want to confirm that the search box equal value matches your expected input. Most teams start their tests by declaring a const browser instance and isolating the session, which helps maintain consistent executions. In some cases, particularly in shared testing environments, teams also use an api token to tag or identify executions across systems.

These strategies not only simplify debugging but also strengthen test coverage and resilience, especially in test suites that evolve frequently.

Using Auto Playwright for Testing

The real strength of Auto Playwright lies in its ability to execute test scenarios using AI. 

How to leverage Auto Playwright to run different scenarios?

First, we’ll create our tests using the “auto” function.

This function requires plain text prompts and arguments containing the test object.

Next, we share an example (page):

auto("<your prompt>", { page, test });

Additionally, we can optionally declare the API model to use through the “options” parameter:

const options = {

  // If true, debugging information is printed in the console.

  debug: boolean,

  // The OpenAI model 

  model: "gpt-4-1106-preview",

  // The OpenAI API key

  openaiApiKey: 'sk-...',

};

This is how it would look in each prompt: 

auto("<tu prompt>", { page, test }, options);

Possible Case

Consider the Following Scenario:

We need to test the functionality of adding a product to a web application’s shopping cart as part of an end-to-end testing scenario. With Auto Playwright, we can do it with simple, human-like instructions.

To test the functionality, we’ll create two tests.

In the first test, we’ll use several prompts to simulate an action to execute in each of them. For the second one, we’ll create a single prompt, which will contain all the actions sequentially.

Our Tests Would Look Like This:

import { test, expect } from "@playwright/test";
import { auto } from "auto-playwright";


test('auto Playwright test 1: Shopping cart functionality - Several prompts', async ({ page }) => {
 await page.goto('http://opencart.abstracta.us/');
 await auto('Go to Phones & PDAs from menu bar', { page, test });
 await auto('Add item 'Palm Treo Pro' to cart', { page, test });
 await auto('Go to Shopping Cart", { page, test });
 const itemsList = await auto('list all items in the cart', { page, test });
 expect(itemsList[0]).toContain('Palm Treo Pro');


});
test("auto Playwright test 2: Add a product to the cart with a single prompt", async ({ page }) => {


 await page.goto('http://opencart.abstracta.us/');
 const result = await auto(`
 1. Go to Phones & PDAs from the menu bar.
 2. Add item 'Palm Treo Pro' to the cart.
 3. Go to the Shopping Cart
 4. Verify in the list of items in the cart are the item 'Palm Treo Pro'.`, { page, test });


expect(result).toBeTruthy();
});

A successful test could verify the appearance of a success message confirming the product has been added to the cart.

Test Execution:

To run our tests in the console, we’ll write npx playwright test. As they execute, we’ll see the information Auto Playwright provides.

Results and Conclusions

Integrating AI into your QA workflows aligns with modern testing practices focused on efficiency, scalability, and reduced technical debt.

Although Auto Playwright stands out as an innovative project in test automation, enabling rapid creation of automation flows with minimal setup, and helping teams test more efficiently, its current level of sophistication and versatility are not enough for test automation.

Despite its significant contribution to accelerating test development and facilitating script creation, it needs to overcome certain limitations to be considered a comprehensive solution in the test automation field.

Below, we detail its advantages and disadvantages, considering both its capabilities and limitations.

Advantages

• Simplifies the process and reduces the amount of code needed, making test writing easier.
• Speeds up development by automating repetitive and tedious tasks, significantly reducing manual effort and increasing team productivity.

Disadvantages

• The AI processing speed in Auto Playwright is significantly slower compared to traditional test automation methods.
• Using Auto Playwright can be challenging in cases where there is not just one element but several elements with the same text, as it requires specificity in automation commands. This becomes especially relevant given the dynamic nature of modern web UIs, which often render content conditionally or in real time.
• There is a high cost associated with the OpenAI API calls linked to Auto Playwright, which can be prohibitive for some organizations.
• Occasional failures in API calls can result in inconsistent test results, undermining their reliability. These results often manifest as flaky tests, which can be difficult to debug and may decrease confidence in test automation outcomes. These issues can lead to the need for manual intervention during test analysis and slow down CI/CD pipelines.

In summary, Auto Playwright has promising potential to improve test development efficiency, but it’s crucial to consider its limitations and evaluate whether it meets expectations in terms of performance, cost, and reliability compared to more established solutions in the market.

Based on its limitations and the need for more advanced solutions for demanding test environments, we do not recommend implementing Auto Playwright as the main test automation tool in a development project.

While it does not yet fully support self-healing tests, this capability could be a valuable future enhancement to reduce the maintenance of automated scripts.

From our specialists’ perspective, it’s crucial to explore more robust and efficient options that better suit the needs and demands of the testing process.

Ultimately, we would recommend its use only for less frequent and detailed tests, without much complexity, to complement its implementation with other more robust, stable, and efficient testing approaches.

Looking ahead, we will continue to monitor Auto Playwright updates to assess how its technological advances can mitigate these limitations over time.

Need Support With Test Automation?

With over 16 years of experience and a global presence, Abstracta is a leading technology solutions company with offices in the United States, Chile, Colombia, and Uruguay. We specialize in software development, AI-driven innovations & copilots, and end-to-end software testing services.

We automate software testing tasks with AI and cutting-edge tools and offer effective testing solutions by implementing automated tests that enhance software quality, impact, and ROI. We continuously incorporate top-tier tools and methodologies to stay aligned with evolving industry needs and deliver reliable, scalable results.

This commitment allows us to design strategies tailored to each context, maximizing efficiency while adapting to the complexity of modern software systems.

We believe that actively bonding ties propels us further and helps us enhance our clients’ software. That’s why we’ve built robust partnerships with industry leaders Microsoft, Datadog, Tricentis,  Perforce BlazeMeter, and Saucelabs to provide the latest in cutting-edge technology. 

Explore our tailored test automation services and contact us to get started!

Follow us on Linkedin & X to be part of our community!

Recommended Articles

How Canada Can Lead Health Tech with AI-Powered Systems

What is Throughput in Performance Testing? Your Ultimate Guide

Open Banking: The API Opportunity for Fintech and Banks