Can Auto Playwright Boost Testing with AI?

Learn all about Auto Playwright, an AI-enhanced evolution of Playwright for test automation. Get insights, use cases, and guides from Abstracta.

In the fast-paced world of software development and Artificial Intelligence (AI), it is crucial to use tools that keep up with technological innovations and simplify and optimize processes.

One of the most popular tools in the test automation field is Playwright, which provides a solid foundation for robust and reliable testing.

It’s an open-source framework that facilitates the creation of automated tests for web applications and allows running them across various browsers.

With all the technological changes in the IT industry regarding Artificial Intelligence, Auto Playwright emerged. This tool is an evolution of Playwright, incorporating AI to take test automation to a new level.

Does Auto Playwright improve test development efficiency? We answer this in detail in this article by conducting tests with real use cases and analyzing both its capabilities and limitations.

Explore our case studies and explore our AI powered test automation solutions!

Introducing Auto Playwright Tests

Auto Playwright leverages OpenAI‘s capabilities to transform plain text instructions into a series of function executions.

It acts as an intelligent intermediary between your plain text commands and the code needed to automate tests, making the process more intuitive and accessible.

This visionary initiative was inspired by the ZeroStep project, which offers a similar API but with a more robust implementation through its private platform. 

How Does Auto Playwright Work? 

First and foremost, you need to write tests in plain text. 

These instructions must describe the actions or checks you want to perform on your web application, like “click the login button” or “get the page title.” 

Then, you need to translate them using OpenAI’s runFunctions feature, which interprets these instructions. 

Once it translates the instructions, Auto Playwright determines and executes the functions that allow performing the described actions.

For instance, it could be clicking, filling out forms, navigating different pages, or searching and verifying content.

Here you can find all the actions Auto Playwright handles.

As these functions are executed, Auto Playwright provides information or results. This may include confirmation of successfully performed actions, data retrieved from the page, or the results of checks and assertions made.

Next, we’ll explore how to get started with the tool, from installation to running your first automated tests. We’ll also analyze the value its use would bring to your projects.

Don’t miss this webinar, with tips for low-code test automation 

Step by Step: Setting Up Auto Playwright from Scratch

Like any project based on Node, we need to meet certain prerequisites.

First, we need Node installed on our computer. Additionally, we require an Integrated Development Environment (IDE). In our case, we’ll use VSC. Once installed, we can start setting up Auto Playwright.

Next, we share a step-by-step guide to help you on this path

1. Create an API Key on OpenAI

OpenAI is an Artificial Intelligence research organization leading the development of advanced technologies like GPT-3 and DALL-E, focusing on promoting AI’s ethical and beneficial use.

To get an OpenAI API key, you must register on the OpenAI platform, create an account, and, in some cases, accept the terms of service.

Steps to get an API key:

• Visit https://openai.com.
• Log in or create a new account.
• Select “API.”
• Hover over the icons on the left to open the menu and select “API Keys.”
• Click on “Create new secret key.”
• Name the key and click on “Create secret key.”
• Generate the key.

One detail about this tool is that you must have available usage on your API Key to work with it.

Although it’s a free platform and you don’t need to subscribe to any plan, there is a cost associated with your API Key. Here you can find more pricing information.

2. Install the Playwright framework from the console 

For this, you should run: npm init playwright@latest

3. Install Auto-playwright

At this step, you ought to run npm install auto-playwright -D in the console.

At this point, we’ll see all the dependencies  including the Auto Playwright dependency in our package.json:

4. Set variables 

In this last step, you need to set the environment variables in the .env file following the tool’s documentation.

Remember, this API key is the one generated in step N° 1: OPENAI_API_KEY=’sk-…”

Using Auto Playwright for Testing

The real strength of Auto Playwright lies in its ability to execute test scenarios using AI. 

How to leverage Auto Playwright to run different scenarios?

First, we’ll create our tests using the “auto” function.

This function requires plain text prompts and arguments containing the test object.

Next, we share an example (page):

auto("<your prompt>", { page, test });

Additionally, we can optionally declare the API model to use through the “options” parameter:

const options = {

  // If true, debugging information is printed in the console.

  debug: boolean,

  // The OpenAI model 

  model: "gpt-4-1106-preview",

  // The OpenAI API key

  openaiApiKey: 'sk-...',

};

This is how it would look in each prompt: 

auto("<tu prompt>", { page, test }, options);

Possible Case

Consider the following scenario:

We need to test the functionality of adding a product to a web application’s shopping cart. With Auto Playwright, we can do it with simple, human-like instructions.

To test the functionality, we’ll create two tests. 

In the first test, we’ll use several prompts to simulate an action to execute in each of them. For the second one, we’ll create a single prompt, which will contain all the actions sequentially.

Our tests would look like this:

import { test, expect } from "@playwright/test";
import { auto } from "auto-playwright";


test('auto Playwright test 1: Shopping cart functionality - Several prompts', async ({ page }) => {
 await page.goto('http://opencart.abstracta.us/');
 await auto('Go to Phones & PDAs from menu bar', { page, test });
 await auto('Add item 'Palm Treo Pro' to cart', { page, test });
 await auto('Go to Shopping Cart", { page, test });
 const itemsList = await auto('list all items in the cart', { page, test });
 expect(itemsList[0]).toContain('Palm Treo Pro');


});
test("auto Playwright test 2: Add a product to the cart with a single prompt", async ({ page }) => {


 await page.goto('http://opencart.abstracta.us/');
 const result = await auto(`
 1. Go to Phones & PDAs from the menu bar.
 2. Add item 'Palm Treo Pro' to the cart.
 3. Go to the Shopping Cart
 4. Verify in the list of items in the cart are the item 'Palm Treo Pro'.`, { page, test });


expect(result).toBeTruthy();
});

Test Execution:

To run our tests in the console, we’ll write npx playwright test. As they execute, we’ll see the information Auto Playwright provides.

Results and Conclusions

Although Auto Playwright stands out as an innovative project in test automation, its current level of sophistication and versatility are not enough for test automation.

Despite its significant contribution to accelerating test development and facilitating script creation, it needs to overcome certain limitations to be considered a comprehensive solution in the test automation field.

Below, we detail its advantages and disadvantages, considering both its capabilities and limitations.

Advantages

• Simplifies the process and reduces the amount of code needed, making test writing easier.

• Speeds up development by automating repetitive and tedious tasks, increasing team productivity.

Disadvantages

• The AI processing speed in Auto Playwright is significantly slower compared to traditional test automation methods.

• Using Auto Playwright can be challenging in cases where there is not just one element but several elements with the same text, as it requires specificity in automation commands.

• There is a high cost associated with the openAI API calls linked to Auto Playwright, which can be prohibitive for some organizations.

• Occasional failures in API calls can result in inconsistent test results, undermining their reliability.

In summary, Auto Playwright has promising potential to improve test development efficiency, but it’s crucial to consider its limitations and evaluate whether it meets expectations in terms of performance, cost, and reliability compared to more established solutions in the market.

Based on its limitations and the need for more advanced solutions for demanding test environments, we do not recommend implementing Auto Playwright as the main test automation tool in a development project.

From our specialists’ perspective, it’s crucial to explore more robust and efficient options that better suit the needs and demands of the testing process.

Ultimately, we would recommend its use only for less frequent and detailed tests, without much complexity, to complement its implementation with other more robust, stable, and efficient testing approaches.

Looking ahead, we will continue to monitor Auto Playwright updates in order to assess how its technological advances can mitigate these limitations over time.

Need Support With Test Automation?

Boasting over 15 years of expertise in software testing, custom software development, and AI-driven solutions, along with a global footprint, we assist a wide range of clients from startups to Fortune 500 companies. 

We automate testing tasks with AI and cutting-edge tools and offer effective testing solutions by implementing automated tests that enhance software quality, impact, and ROI. 

Explore our tailored test automation services and contact us to discuss how we can help you grow your business.

Follow us on Linkedin & X to be part of our community!

Recommended Articles

Clutch Awards: Abstracta, Global Leader for Spring 2024

Generative AI for Dummies: A Comprehensive Guide for Beginners and Beyond

Automated Functional Testing Guide