Home > Blog > Test Automation Circles v2.0 Beta
Author: Tomotaka ASAGI
Published: Jan 4, 2026
A Framework for E2E Test Automation in the AI Era
Introduction
Test Automation Circles is a framework that visualizes the key considerations for implementing E2E test automation as concentric circles. By working from the center outward and considering each layer simultaneously, teams can achieve test automation that is "less failure-prone and more sustainable."
Version 2.0 Beta introduces the perspective of human-AI collaboration across all layers, reflecting the rapid advancement of AI and LLM technologies.
A Note on This Beta Version
This is a beta release. The landscape of AI-powered testing is evolving rapidly, and we anticipate this framework will continue to evolve based on real-world feedback and technological developments. We welcome your thoughts and experiences.
The Fundamental Question
Before diving into the framework, we must acknowledge an emerging tension in our field:
Today, we talk about humans collaborating with AI—using AI as a powerful assistant for test automation. AI helps us explore, generate test cases, and analyze results faster than ever before.
But tomorrow? The balance may shift. As AI capabilities grow:
- AI may take the lead in test automation, with humans in a supporting role
- Humans could become the bottleneck in an AI-driven testing pipeline
- The fundamental question emerges: If AI drives automation, who is quality assurance really for?
These questions don't have clear answers yet. This framework is designed with current human-AI collaboration in mind, but we must remain aware that the paradigm itself may transform.
Why / How / What
Even in the AI era, the Circle principle remains relevant. However, the degree of change varies significantly across these dimensions.
Perspective | Degree of Change | Description |
Why (Core) | Unchanged | Why we test—the essence remains the same |
How (Concept) | Significantly Changed | Human-AI collaboration strategy, Testability design |
What (Architecture) | Dramatically Changed | Tools, ARIA Snapshot, MCP, etc. |
Core: Why?
Why do you test?
The most important question when implementing test automation is "Why are we doing this test?" Without a clear objective, running automated tests can become an end in itself, or the team may lose direction when scaling the test suite.
Items to Decide
- Objective
- Expected Results
In the AI era, this layer remains unchanged. Quality assurance, risk mitigation, faster feedback loops—these fundamental purposes persist. The importance of stakeholder alignment also remains constant.
However, this is precisely where we should pause and reflect: As AI takes on more of the execution, does the "why" need to be reconsidered? Is quality assurance for the end users, for the development team, for the business, or increasingly, for the AI systems themselves?
Concept: How?
How do you test?
Based on the established objectives, determine the concept of "what kind of testing to conduct." Version 2.0 adds items to address the AI era.
1. Test Strategy
Traditional Perspective | AI Era Additions |
Risk-based strategy | AI utilization policy (which phases?) |
Test level allocation | Human-AI responsibility boundaries |
Priority setting | Autonomous vs. Copilot approach |
— | AI testing maturity goals |
— | Test scope allocation across three domains |
Key Considerations for AI Utilization Policy
- Which phases will use AI? (Design support? Execution? Analysis?)
- Autonomous agents vs. Copilot-style assistance
- Cost and speed trade-offs
- Scope of AI decision-making vs. human approval requirements
- Team AI literacy development plan
- When to convert AI-Driven tests into scripted Automated tests
2. Testability Design [New]
A new addition in v2.0. Consider the Testability of the system under test from the design phase. In the AI era, not only human testability but also AI testability becomes crucial.
Why AI Testability Matters
In AI browser automation represented by Playwright MCP, the page representation method has shifted from DOM/selectors to Accessibility Tree (ARIA Snapshot). This makes the following critical:
- Semantic Clarity: Appropriate ARIA roles, aria-label, semantic HTML structure
- Contextual Explainability: Can the LLM understand the "intent" of elements?
- Deterministic Predictability: Does the same operation return the same result?
Key Insight: Teams that have seriously invested in accessibility naturally have higher AI Testability. Improving accessibility for AI also benefits human users who rely on assistive technologies.
3. Test Design
Traditional Perspective | AI Era Additions |
Test case design | Prompt design (instructions to AI) |
Scenario design | AI agent behavior design |
Selector strategy | Accessibility structure utilization |
4. Test Scope [Updated]
In the AI era, test scope is no longer a binary choice between "automated" and "manual." We now have three distinct domains, each with its own characteristics and appropriate use cases.
The Three Domains of Testing
AI-Driven Test: A New Category
This is particularly relevant in AI-driven development and Spec-driven development workflows:
Development → Build → [AI-Driven Test] → Stable? → [Convert to Automated Test]
↓
Issue found → Fix
In this flow:
- Write a spec (or convey intent in natural language)
- AI generates code
- Build completes
- AI launches browser via MCP and immediately verifies behavior ← This is new
- Issues found are fixed immediately
- Stable flows are converted to traditional automated tests
Strategic Decisions for Test Scope
Decision | Considerations |
Which tests belong where? | AI-Driven for exploration and quick verification; Automated for regression |
When to formalize? | Not everything needs to become a scripted test case |
Investment allocation | Script creation effort vs. AI token costs |
Conversion criteria | When does an AI-Driven test become worth scripting? |
5. Data Design
Automated tests are often executed repeatedly, so idempotency (same operation yields same result regardless of repetition) must be considered. In the AI era, data strategies must also account for LLM non-determinism (possibility of slightly different results from the same instruction).
6. Execution Plan
In addition to pipeline design, include AI agent integration methods and token cost management in the plan.
Architecture: What?
What do you use for automated testing?
Once Core and Concept are determined, decide on the architecture and build the actual test automation system.
Items to Decide
- Tool selection (MCP compatibility as a new criterion)
- Framework (language, test framework)
- Environment setup
- CI/CD integration
Technology Evolution in the AI Era
Monitoring and Control: Real.
Realize automated testing
Test automation operations involve creating/modifying test scripts, repeated execution, result analysis, and reporting. Automated testing doesn't end when it's created.
Items to Decide
- Test automation solution
- Execution method
- Result analysis
- Reporting
- AI-Driven Test logging [New]
AI-Driven Test Logging: A Critical Addition
When AI executes tests, logging takes on a fundamentally different character compared to traditional automated tests.
Why Traditional Logging Is Insufficient
Aspect | Automated Test | AI-Driven Test |
Execution | Determined by script, predictable | AI decides, potentially non-deterministic |
Reproducibility | Same script = Same behavior | Same prompt ≠ Guaranteed same behavior |
Log purpose | Record results | Record decision-making process is essential |
What to Log for AI-Driven Tests
Item | Content | Why It Matters |
Prompt/Instruction | What was the AI instructed to do? | Reproduction and improvement |
ARIA Snapshot | The page structure the AI "saw" | Basis for AI's decisions |
AI Decision Rationale | Why the AI chose that action | Debugging, auditing |
Executed Actions | Click, input, navigation history | Traceability |
Detected Issues | Anomalies, errors, deviations from expectations | Quality records |
Context | Timestamp, environment, model version | Reproducibility |
Why This Matters
- Trust: "AI tested it" is meaningless without knowing what was actually tested
- Auditability: Evidence trail for quality assurance
- Debugging: Trace what happened when issues occur
- Improvement: Evaluate whether AI decisions were appropriate; refine prompts and systems
- Accountability: Explanation to stakeholders
Key Point: Without proper logging, AI-Driven testing becomes a black box. The value of rapid feedback is lost if you cannot understand, reproduce, or learn from what the AI did.
Base: Base.
Foundation for building automated testing
This is the foundation for achieving test automation. No matter how excellent the strategy or tools, continuation is difficult without a solid foundation.
Items to Decide
- Resources
- Teams
- Skill Set
- Culture
Skill Changes Required in the AI Era
Traditional Skills | Additional Skills Required |
Selector strategy | Accessibility knowledge |
Wait handling | LLM characteristics understanding |
Browser behavior understanding | Prompt design |
Programming | AI output validity evaluation |
— | Log analysis for AI behavior |
Important: "Tools becoming easier doesn't mean skills are unnecessary." The type of skills required changes. Without understanding Playwright, you cannot judge whether MCP output is valid.
Looking Ahead: Questions Without Answers
As we implement this framework, several questions remain open:
- Who decides quality? When AI can execute more tests faster than humans can review, how do we define "enough" quality assurance?
- Human as bottleneck: If AI can generate, execute, and analyze tests faster than humans can provide feedback, do we need to redesign the entire workflow?
- Trust calibration: How much should we trust AI-driven test results? What verification mechanisms do humans need?
- The purpose of QA: If AI handles most automation, does QA become about validating AI behavior rather than software behavior?
- The logging paradox: If we log everything AI does for auditability, who reviews those logs? Does this create a new bottleneck?
These are not problems to solve today, but awareness to carry as we evolve our practices.
Summary
Test Automation Circles v2.0 Beta maintains the principles of "working from center to outside" and "considering each layer simultaneously" even in the AI era.
The key changes in this version:
- Test Strategy: Added AI utilization policy and the three-domain test scope model
- Testability Design [New]: Both Human Testability and AI Testability
- Test Scope [Updated]: Three domains—Automated, AI-Driven, and Human testing
- Monitoring and Control [Updated]: Critical importance of AI-Driven test logging
The traditional warning that "starting with tools leads to failure" remains important in the AI era—perhaps even more so. Without making "AI-aware" decisions at the Strategy stage, simply adding tools later won't produce results.
And as AI capabilities continue to advance, we must remain open to the possibility that the human-AI relationship we design today may need fundamental revision tomorrow.
Test Automation Circles v2.0 Beta
Arrangility Sdn. Bhd.
https://www.arrangility.com/blog/test-automation-circles
We welcome feedback on this beta version. Share your thoughts and experiences as you apply this framework to your own context.