Test Automation Circles v2.0 Beta

Home > Blog > Test Automation Circles v2.0 Beta

A Framework for E2E Test Automation in the AI Era

Introduction

Test Automation Circles is a framework that visualizes the key considerations for implementing E2E test automation as concentric circles. By working from the center outward and considering each layer simultaneously, teams can achieve test automation that is "less failure-prone and more sustainable."

Version 2.0 Beta introduces the perspective of human-AI collaboration across all layers, reflecting the rapid advancement of AI and LLM technologies.

A Note on This Beta Version

This is a beta release. The landscape of AI-powered testing is evolving rapidly, and we anticipate this framework will continue to evolve based on real-world feedback and technological developments. We welcome your thoughts and experiences.

The Fundamental Question

Before diving into the framework, we must acknowledge an emerging tension in our field:

Today, we talk about humans collaborating with AI—using AI as a powerful assistant for test automation. AI helps us explore, generate test cases, and analyze results faster than ever before.

But tomorrow? The balance may shift. As AI capabilities grow:

• AI may take the lead in test automation, with humans in a supporting role
• Humans could become the bottleneck in an AI-driven testing pipeline
• The fundamental question emerges: If AI drives automation, who is quality assurance really for?

These questions don't have clear answers yet. This framework is designed with current human-AI collaboration in mind, but we must remain aware that the paradigm itself may transform.

Why / How / What

Even in the AI era, the Circle principle remains relevant. However, the degree of change varies significantly across these dimensions.

Core: Why?

Why do you test?

The most important question when implementing test automation is "Why are we doing this test?" Without a clear objective, running automated tests can become an end in itself, or the team may lose direction when scaling the test suite.

Items to Decide

• Objective
• Expected Results

In the AI era, this layer remains unchanged. Quality assurance, risk mitigation, faster feedback loops—these fundamental purposes persist. The importance of stakeholder alignment also remains constant.

However, this is precisely where we should pause and reflect: As AI takes on more of the execution, does the "why" need to be reconsidered? Is quality assurance for the end users, for the development team, for the business, or increasingly, for the AI systems themselves?

Concept: How?

How do you test?

Based on the established objectives, determine the concept of "what kind of testing to conduct." Version 2.0 adds items to address the AI era.

1. Test Strategy

Key Considerations for AI Utilization Policy

• Which phases will use AI? (Design support? Execution? Analysis?)
• Autonomous agents vs. Copilot-style assistance
• Cost and speed trade-offs
• Scope of AI decision-making vs. human approval requirements
• Team AI literacy development plan
• When to convert AI-Driven tests into scripted Automated tests

2. Testability Design [New]

A new addition in v2.0. Consider the Testability of the system under test from the design phase. In the AI era, not only human testability but also AI testability becomes crucial.

Why AI Testability Matters

In AI browser automation represented by Playwright MCP, the page representation method has shifted from DOM/selectors to Accessibility Tree (ARIA Snapshot). This makes the following critical:

• Semantic Clarity: Appropriate ARIA roles, aria-label, semantic HTML structure
• Contextual Explainability: Can the LLM understand the "intent" of elements?
• Deterministic Predictability: Does the same operation return the same result?

Key Insight: Teams that have seriously invested in accessibility naturally have higher AI Testability. Improving accessibility for AI also benefits human users who rely on assistive technologies.

3. Test Design

4. Test Scope [Updated]

In the AI era, test scope is no longer a binary choice between "automated" and "manual." We now have three distinct domains, each with its own characteristics and appropriate use cases.

The Three Domains of Testing

AI-Driven Test: A New Category

This is particularly relevant in AI-driven development and Spec-driven development workflows:

Development → Build → [AI-Driven Test] → Stable? → [Convert to Automated Test]
                            ↓
                     Issue found → Fix

In this flow:

1. Write a spec (or convey intent in natural language)
2. AI generates code
3. Build completes
4. AI launches browser via MCP and immediately verifies behavior ← This is new
5. Issues found are fixed immediately
6. Stable flows are converted to traditional automated tests

Strategic Decisions for Test Scope

5. Data Design

Automated tests are often executed repeatedly, so idempotency (same operation yields same result regardless of repetition) must be considered. In the AI era, data strategies must also account for LLM non-determinism (possibility of slightly different results from the same instruction).

6. Execution Plan

In addition to pipeline design, include AI agent integration methods and token cost management in the plan.

Architecture: What?

What do you use for automated testing?

Once Core and Concept are determined, decide on the architecture and build the actual test automation system.

Items to Decide

• Tool selection (MCP compatibility as a new criterion)
• Framework (language, test framework)
• Environment setup
• CI/CD integration

Technology Evolution in the AI Era

Monitoring and Control: Real.

Realize automated testing

Test automation operations involve creating/modifying test scripts, repeated execution, result analysis, and reporting. Automated testing doesn't end when it's created.

Items to Decide

• Test automation solution
• Execution method
• Result analysis
• Reporting
• AI-Driven Test logging [New]

AI-Driven Test Logging: A Critical Addition

When AI executes tests, logging takes on a fundamentally different character compared to traditional automated tests.

Why Traditional Logging Is Insufficient

What to Log for AI-Driven Tests

Why This Matters

1. Trust: "AI tested it" is meaningless without knowing what was actually tested
2. Auditability: Evidence trail for quality assurance
3. Debugging: Trace what happened when issues occur
4. Improvement: Evaluate whether AI decisions were appropriate; refine prompts and systems
5. Accountability: Explanation to stakeholders

Key Point: Without proper logging, AI-Driven testing becomes a black box. The value of rapid feedback is lost if you cannot understand, reproduce, or learn from what the AI did.

Base: Base.

Foundation for building automated testing

This is the foundation for achieving test automation. No matter how excellent the strategy or tools, continuation is difficult without a solid foundation.

Items to Decide

• Resources
• Teams
• Skill Set
• Culture

Skill Changes Required in the AI Era

Important: "Tools becoming easier doesn't mean skills are unnecessary." The type of skills required changes. Without understanding Playwright, you cannot judge whether MCP output is valid.

Looking Ahead: Questions Without Answers

As we implement this framework, several questions remain open:

1. Who decides quality? When AI can execute more tests faster than humans can review, how do we define "enough" quality assurance?
2. Human as bottleneck: If AI can generate, execute, and analyze tests faster than humans can provide feedback, do we need to redesign the entire workflow?
3. Trust calibration: How much should we trust AI-driven test results? What verification mechanisms do humans need?
4. The purpose of QA: If AI handles most automation, does QA become about validating AI behavior rather than software behavior?
5. The logging paradox: If we log everything AI does for auditability, who reviews those logs? Does this create a new bottleneck?

These are not problems to solve today, but awareness to carry as we evolve our practices.

Summary

Test Automation Circles v2.0 Beta maintains the principles of "working from center to outside" and "considering each layer simultaneously" even in the AI era.

The key changes in this version:

1. Test Strategy: Added AI utilization policy and the three-domain test scope model
2. Testability Design [New]: Both Human Testability and AI Testability
3. Test Scope [Updated]: Three domains—Automated, AI-Driven, and Human testing
4. Monitoring and Control [Updated]: Critical importance of AI-Driven test logging

The traditional warning that "starting with tools leads to failure" remains important in the AI era—perhaps even more so. Without making "AI-aware" decisions at the Strategy stage, simply adding tools later won't produce results.

And as AI capabilities continue to advance, we must remain open to the possibility that the human-AI relationship we design today may need fundamental revision tomorrow.

Test Automation Circles v2.0 Beta

Arrangility Sdn. Bhd.

https://www.arrangility.com/blog/test-automation-circles

We welcome feedback on this beta version. Share your thoughts and experiences as you apply this framework to your own context.