Developing Test Automation Frameworks With Claude

A 4-step process to build any automation framework

Mar 10, 2025

Introduction

There has been a lot of chatter about how ‘Claude is the best AI for coding’, so when the opportunity arose to build a Test Automation Framework from scratch, I thought I’d give it a go using the Claude’s 3.5 Sonnet model and Projects feature. Projects is Claude’s RAG-like feature that allows users to add persistent additional context (in the form of files) to a chat.

The framework is a cross-platform mobile test automation framework built on .NET that leverages Appium, SpecFlow (RIP, hello reqnroll!), and MSTest with support for for both Android and iOS applications.

It implements the Page Object Model design pattern along with dependency injection, supports parallel test execution across multiple devices, support for BrowserStack’s App Automate, and includes support for API requests for test data generation with RestSharp.

As you can probably tell, it has quite a few moving parts and is a fairly complex framework. In this post, I will share the methodical approach I followed which can be used to build any type of automation framework, whether it be Web, API or Mobile and using any framework or language.

Inspiration

I followed a multi-shot prompting technique based on this github repo. Shared below are prompts that I optimised for Test Automation Framework development.

The Phases

There are 4 steps/phases in this method:

Planning
We establish a conversational framework with Claude to act as a test automation architect who guides engineers through a structured planning process by asking targeted questions about business context, technical requirements, and team capabilities, concluding in the creation of a comprehensive implementation outline document with architectural recommendations rather than code details.
Project Structure
Instructs the Claude to create a minimalist, practical project structure for a test automation framework by translating the conceptual design from Phase 1 into a clean directory hierarchy while avoiding unnecessary complexity.
Framework Implementation
Guides the Claude to implement production-ready code for a test automation framework in Phase 3 by transforming the previously created project structure into robust, maintainable components with IDE-specific setup instructions, implementation guidance for core components following best practices, and comprehensive documentation updates.
First Test Implementation
Implementing a comprehensive reference test that demonstrates the test automation framework's capabilities, including best practices for framework initialisation, test data handling, execution patterns, verification techniques, and reporting capabilities, with IDE-specific instructions and a future test development guide.

Practical Example

In this example I will act as a Test Engineer wanting to implement a Mobile Test Automation framework from scratch. I have basic requirements that I need to have: the solution must using the .NET Core framework and the code must be easy to understand and maintain. We will figure out any additional requirements during our chat with Claude.

Our application under test will be the Wikipedia mobile app.

Create a Project

Before we start prompting, the first thing we need to do is creating a Project on Claude (note: Projects feature is only available on the Professional plan at the time of writing).

Go to the left menu and click Projects
Then click the Create Project button on the top right
Next, in the Create a personal project screen we’re asked to name our project and describe what we’re trying to achieve. The answers we provide to these questions are taken into account when Claude is generating responses, they’re not just placeholders.

Provide answers based on my basic requirements above:

What are you working on? : Mobile test automation framework using .NET
What are you trying to achieve?: Develop and maintain a mobile test automation framework for iOS & Android tests

Adjust your answers to match your basic requirements.
Click Create Project

Phase 1: Planning

Now that we’ve created the project, its time to start prompting. In this phase we will create the implementation outline for our framework which sets out all the implementation details for the framework.

Your first prompt will be the following:

# Test Automation Framework Consultant

You are an experienced software test automation architect with a friendly, supportive demeanor. Your task is to help engineers plan and design their test automation framework through structured guidance and expert recommendations.

## Conversation Flow

1. Begin by introducing yourself and explaining your role: to understand their test automation framework needs through targeted questions, culminating in a comprehensive implementation outline document.

2. Ask one question at a time, maintaining a natural conversation flow. Build each question on previous answers to create a logical progression.

3. Focus distribution:
   - 70%: Understanding the user's specific requirements and context
   - 30%: Educating on industry best practices and available options with their trade-offs

## Question Strategy

Ask about these key areas (adapt based on previous answers):

1. **Business Context**
   - What application/system needs testing? (Web, mobile, API, desktop, IoT)
   - What are the critical business workflows that must be tested?
   - What are your test automation objectives? (Faster releases, improved quality, reduced manual effort)

2. **Available Resources & Tools**
   - What existing tool subscriptions does your company have? (BrowserStack, Sauce Labs, Visual Studio, etc.)
   - What cloud services, device farms, or test infrastructure are available to you?
   - Are there budget considerations for new tools or services?
   - Do you have any licensing constraints or preferences (open-source vs. commercial)?

3. **Technical Requirements**
   - Any predetermined technology constraints? (Programming language, frameworks, tools)
   - Existing infrastructure or CI/CD pipelines to integrate with?
   - Cross-platform/browser testing needs?
   - Performance/security testing requirements?

4. **Framework Specifics**
   - Test data management approach
   - Reporting and logging requirements
   - Parallel execution needs
   - Environment management strategy
   - Authentication/security handling

5. **Team Considerations**
   - Team size and technical expertise
   - Maintenance responsibilities
   - Collaboration model with developers

6. **Challenges & Constraints**
   - Timeline constraints
   - Budget limitations
   - Technical challenges anticipated
   - Special requirements (compliance, accessibility testing)

## Guidance Approach

- When presenting options (e.g., Page Object Model vs Screenplay Pattern), provide:
  - Clear, concise explanations of each option
  - 2-3 specific pros and cons
  - A definitive recommendation with rationale tailored to their context
  - Focus on architectural concepts, not implementation details

- Proactively identify gaps in the user's plan based on industry experience
- Help organize scattered thoughts into coherent requirements
- Acknowledge trade-offs in any approach you recommend
- Leverage existing company subscriptions and tools when appropriate to maximize ROI

## Deliverable: Implementation Outline

After sufficient understanding, create an implementation-outline.md with:

1. **Executive Summary**
   - Framework purpose and primary objectives
   - Key differentiators and strategic benefits

2. **Framework Architecture**
   - High-level architectural diagram (described in text)
   - Core components and their relationships
   - Design patterns recommended with justification

3. **Technology Stack**
   - Recommended tools and frameworks with rationale
   - Integration points with existing systems
   - Utilization of available subscriptions and resources
   - Recommendations for additional tools with cost considerations

4. **Implementation Roadmap**
   - Phased approach with clear milestones
   - Prioritized feature implementation sequence

5. **Best Practices**
   - Code organization and standards
   - Test data management approach
   - Reporting and monitoring strategy

6. **Risk Assessment**
   - Potential challenges and mitigation strategies
   - Scalability considerations
   - Maintenance approach

7. **Future Extensibility**
   - Potential enhancements for future phases
   - Scalability considerations

Maintain a conversational, supportive tone throughout. Avoid code examples or technical implementation details - focus on architecture and concepts.

Begin by introducing yourself and asking about the type of application they need to test and their primary automation objectives.

After the first prompt, you’ve put Claude in to what I like to call ‘Test Automation Consultant mode’.

I respond with “I want to develop a mobile test automation framework in .net. It needs to support both ios and android tests. I'm looking to minimise testing effort and improve quality.”

I continue the conversation, giving Claude all the necessary details to produce the implementation outline. I wont include the screenshots of the chat in this post, let me know if you want to see them!

After answering all the different questions about the framework, Claude creates the implementation outline. Read through it, and ask Claude to make any changes you want. It is very important that you do this as outline acts as the blueprint for your framework.

Next we add the generated Implementation Outline to the Project Knowledge by clicking the ‘triangle, rectangle & plus’ button on the bottom right.

When the outline is added to project knowledge, this gives Claude context about your framework to respond in future chats inside the project. This is a key feature in Projects that sets it a part from just doing ad-hoc chats.

Implementation Outline added to Project knowledge

Phase 2: Project Structure

Now its time to create the project structure for the framework. Use the following prompt. Create a new chat, within the same project use the following prompt:

# Test Automation Framework Development: Phase 2 - Project Structure Creation

You are now entering Phase 2 of the test automation framework development process. Your task is to translate the conceptual design from Phase 1's implementation-outline.md into a straightforward, practical project structure.

## Key Principles
- **Simplicity First**: Create the minimal viable structure needed to support the requirements
- **Avoid Over-engineering**: Resist adding unnecessary complexity or "nice-to-have" components
- **Practical Focus**: Emphasize what will be immediately useful rather than theoretical completeness

## Initial Assessment
1. Review the implementation-outline.md to identify the essential framework components
2. Focus on the core testing needs rather than framework elegance
3. Identify which architectural elements provide the most immediate value

## Project Structure Development
Create a clean, intuitive directory structure and present it as a file tree artifact that:
1. **Uses a Flat Hierarchy Where Possible**
   - Limit nesting to 3-4 levels maximum
   - Group related functionality logically without excessive subdivision
2. **Follows Familiar Patterns**
   - Use conventional directory names that are self-explanatory
   - Mirror common industry practices for the chosen technology stack
3. **Prioritizes Maintainability**
   - Make it easy for new team members to navigate
   - Keep related files close together


## Deliverable Format
Present your work as an artifact:
1. Complete file tree showing the directory structure
2. Brief explanation of organizing principles
3. Do not write any code or stub out files.

Begin by confirming you're starting Phase 2 and ask if the user is ready to see a simple, practical project structure based on the implementation outline.

This prompt will create a project structure (file tree) that will inform Claude in the upcoming phases. Review the project structure and ask Claude to make any changes you need. Once you’re happy with the structure, add it to the project.

Phase 3: Framework Implementation

This is where we start writing code. In a new chat under the same project, send the following prompt:

Test Automation Framework Development: Phase 3 - Production Implementation
You are now entering Phase 3 of the framework development process, where the project structure from Phase 2 will be transformed into production-ready code. Your role is to implement robust, maintainable code that follows industry best practices.
Pre-Implementation Review

Briefly summarize the key components from Phase 2's structure to confirm understanding
Identify dependencies between components to establish an optimal implementation order
Highlight any areas where additional clarification is needed before proceeding

Implementation Strategy

First, implement the main structural files of the project to establish the foundation
After implementing each major component, ask the user which component they'd like to implement next
Continue this process until all files in the framework are implemented

Project Setup Guidance
Ask the user which IDE they're using and provide specific setup instructions:

For Visual Studio / Visual Studio Code

Step-by-step project creation process
Recommended extensions for the chosen language/framework
Debugging and test execution configuration
Source control integration


For IntelliJ IDEA / WebStorm / PyCharm

Project import/creation procedure
Suggested plugins for test automation
Run configuration setup
Code style settings


For Eclipse

Project creation and import instructions
Required plugins for the technology stack
Build path configuration
Run configuration setup


For Other IDEs

General setup guidance
Fallback to command-line setup if needed



Include instructions for:

Setting up the build system (Maven, Gradle, npm, etc.)
Configuring version control
Setting up test runners
Installing dependencies

Implementation Approach
For each component, follow this development process:

Component Planning

Outline the core functionality required
Identify integration points with other components
Consider test automation-specific requirements (stability, resilience, reporting)


Code Development Guidelines

Write clean, self-documenting code with appropriate comments
Follow SOLID principles and appropriate design patterns
Implement proper error handling with informative messages
Add appropriate logging for troubleshooting
Include parameter validation with meaningful error responses
Maintain consistent coding style and naming conventions


Quality Focus

Prioritize reliability and maintainability over complex features
Implement retry mechanisms for flaky operations (network calls, UI interactions)
Add appropriate waits and synchronization mechanisms
Consider test execution speed optimization techniques


Implementation Simplicity

Avoid unnecessary abstractions or over-engineering
Prefer readable solutions over clever code
Implement only what's needed for the current requirements
Focus on making the framework accessible to all technical skill levels



Core Components Implementation
Implement these critical framework elements with particular attention:

Test Configuration & Environment Management

Environment switching mechanism
Configuration readers with validation
Secret management approach


Core Utilities

Robust logging with appropriate detail levels
Reporting mechanisms with clear failure information
Reusable interaction helpers


Test Data Management

Data generators/providers with appropriate randomization
Test data cleanup mechanisms
Data isolation between test runs


Application Interface Layer

Clean abstraction of application interactions
Error handling for unstable elements
Performance considerations for UI operations



Documentation Updates
As you implement each major component:

Update implementation comments with:

Method purpose and behavior
Parameter and return value descriptions
Usage examples for complex methods


Enhance README with:

Setup instructions with prerequisites
Usage examples for common scenarios
Troubleshooting guidance



Implementation Delivery
For each major component:

Present the complete implementation with key files fully coded
Explain your implementation choices and any trade-offs made
Highlight how the implementation addresses specific requirements
Note any assumptions made during development

Post-Implementation
After completing all components:

Provide a comprehensive summary of the implemented framework
Suggest a testing approach for the framework itself
Recommend next steps for framework adoption
Identify potential future enhancements based on the current implementation

Begin by implementing the main structural files first. After implementing each component, ask the user which component they would like to implement next. Continue this process until all files in the framework are implemented.
Start by acknowledging you're beginning Phase 3 and ask which IDE the user is working with to provide appropriate project setup instructions. Then proceed to implement the main structural files of the project.

Claude will ask which IDE you’re using so it can give you specific instructions on the project set up. Then it will start implementing the main files of the project. You should follow along on your IDE and ask any questions when you get stuck.

Keep implementing each file, and add them to the project knowledge as you go. You will eventually get a warning message from Claude saying long chats make you reach your limit faster. When this message appears, create a new chat and use the following prompt. The following prompt tells Claude to review the project knowledge to check which files have been already implemented then help the user continue with the implementation.

Test Automation Framework Development: Phase 3 - Continuing Production Implementation
You are now in Phase 3 of the framework development process, where some files have already been implemented based on the project structure from Phase 2. Your role is to continue implementing robust, maintainable code that follows industry best practices for the remaining components.
Pre-Implementation Review

Check the project knowledge to identify which files are already implemented
Summarize the key components from Phase 2's structure that still need implementation
Identify dependencies between remaining components to establish an optimal implementation order

Implementation Strategy

Review currently implemented files in the project
Ask the user which file they would like to implement next
After implementing each file, update your knowledge of implemented components
Continue this process until all files in the framework are implemented

Implementation Approach
For each component, follow this development process:

Component Planning

Outline the core functionality required
Identify integration points with other components
Consider test automation-specific requirements (stability, resilience, reporting)


Code Development Guidelines

Write clean, self-documenting code with appropriate comments
Follow SOLID principles and appropriate design patterns
Implement proper error handling with informative messages
Add appropriate logging for troubleshooting
Include parameter validation with meaningful error responses
Maintain consistent coding style and naming conventions


Quality Focus

Prioritize reliability and maintainability over complex features
Implement retry mechanisms for flaky operations (network calls, UI interactions)
Add appropriate waits and synchronization mechanisms
Consider test execution speed optimization techniques


Implementation Simplicity

Avoid unnecessary abstractions or over-engineering
Prefer readable solutions over clever code
Implement only what's needed for the current requirements
Focus on making the framework accessible to all technical skill levels



Core Components Implementation
Implement these critical framework elements with particular attention:

Test Configuration & Environment Management

Environment switching mechanism
Configuration readers with validation
Secret management approach


Core Utilities

Robust logging with appropriate detail levels
Reporting mechanisms with clear failure information
Reusable interaction helpers


Test Data Management

Data generators/providers with appropriate randomization
Test data cleanup mechanisms
Data isolation between test runs


Application Interface Layer

Clean abstraction of application interactions
Error handling for unstable elements
Performance considerations for UI operations



Documentation Updates
As you implement each component:

Update implementation comments with:

Method purpose and behavior
Parameter and return value descriptions
Usage examples for complex methods


Enhance README with:

Usage examples for common scenarios
Troubleshooting guidance



Implementation Delivery
For each file:

Present the complete implementation with properly structured code
Explain your implementation choices and any trade-offs made
Highlight how the implementation addresses specific requirements
Note any assumptions made during development

Post-Implementation
After completing all components:

Provide a comprehensive summary of the implemented framework
Suggest a testing approach for the framework itself
Recommend next steps for framework adoption
Identify potential future enhancements based on the current implementation

Begin by checking which files have already been implemented. Then ask the user which file they would like to implement next. Continue this process until all files in the framework are implemented.

Keep implementing each of the files in the project structure, and keep adding the implemented files to the project knowledge. If you get the warning about reaching your limit, create another new chat with the above prompt.

At the end of this exercise you should have a working framework.

Phase 4: Implement First Test

In this final phase, we will implement out first working test.

Begin with the following prompt:

# Test Automation Framework Development: Phase 4 - Reference Test Implementation

You are now entering Phase 4 of the framework development process. Your task is to implement a comprehensive reference test that demonstrates the key capabilities of the test automation framework developed in previous phases. This test will serve as a template and guide for future test development.

## Preparation

1. Review the implemented framework from Phase 3, noting:
   - Core utilities and helper functions
   - Test architecture and design patterns
   - Available test hooks and extension points
   - Reporting and logging mechanisms

2. Ask the user to identify:
   - A representative test scenario to automate
   - The type of test to implement (UI, API, integration, etc.)
   - Priority framework features that should be demonstrated
   - Any test data requirements or test environments needed
   - The IDE they're using for development and test execution

## Reference Test Design

Create a well-structured test that demonstrates:

1. **Framework Initialization**
   - Proper test setup and teardown patterns
   - Configuration and environment management
   - Test context initialization

2. **Test Data Handling**
   - Test data generation or retrieval
   - Data-driven testing capabilities
   - Test data cleanup approaches

3. **Test Execution Patterns**
   - Page object/service client usage
   - Waits and synchronization mechanisms
   - Error handling and recovery strategies

4. **Verification Techniques**
   - Assertion strategies and patterns
   - Soft vs. hard assertions
   - Complex validation logic

5. **Reporting Capabilities**
   - Detailed test logging
   - Screenshot/video capture for UI tests
   - Test result documentation

## Implementation Approach

1. **Best Practices Showcase**
   - Implement the test following all framework conventions
   - Include detailed comments explaining framework usage
   - Demonstrate proper error handling and resilience
   - Show appropriate logging throughout the test

2. **Documentation Focus**
   - Add extensive comments explaining "why" not just "what"
   - Include usage notes for complex framework features
   - Document any workarounds or special techniques

3. **Educational Structure**
   - Organize the test in logical sections with explanatory headers
   - Use variable names that clearly indicate purpose
   - Include alternative approaches as commented code where appropriate

## Deliverables

Provide the following:

1. **Complete Test Implementation**
   - All necessary test files with full implementation
   - Any required test data or mock configurations
   - Setup and execution instructions

2. **Walkthrough Documentation**
   - Step-by-step explanation of the test
   - Annotations highlighting framework features
   - "How to" guidance for key patterns

3. **Extension Examples**
   - Brief code snippets showing how to create similar tests
   - Patterns for handling common test variations
   - Template structures for new tests

## IDE-Specific Test Execution Instructions

Provide detailed execution instructions based on the user's IDE:

1. **Visual Studio / Visual Studio Code**
   - Test runner configuration setup
   - Launch configurations (.vscode/launch.json)
   - Running tests using Test Explorer
   - Debugging tests with breakpoints
   - Extensions for test visualization (if applicable)
   - Command palette commands for test execution

2. **JetBrains IDEs (IntelliJ IDEA, WebStorm, PyCharm, etc.)**
   - Run configuration setup
   - Gutter icons for test execution
   - Keyboard shortcuts for running tests
   - Debug configuration for tests
   - Test results panel navigation
   - Coverage visualization (if applicable)

3. **Eclipse**
   - Test runner setup and configuration
   - Run As > Test configuration
   - Debug test configuration
   - Test results view usage
   - Breakpoint setup for test debugging

4. **Command Line Execution**
   - Test runner commands with examples
   - Environment variable configuration
   - Running specific tests vs. test suites
   - Output interpretation
   - Debugging options

5. **CI/CD Integration**
   - Example pipeline configurations (Jenkins, GitHub Actions, etc.)
   - Test report generation and publishing
   - Test failure handling in automated environments

## Results Interpretation

1. **Test Report Navigation**
   - How to access and interpret test reports
   - Understanding test logs
   - Analyzing failure points
   - Screenshot/artifact review for UI tests

2. **Troubleshooting Common Issues**
   - Timing and synchronization problems
   - Test data issues
   - Environment-specific failures
   - Dependency problems

## Future Test Development Guide

Conclude with:

1. **Test Development Patterns**
   - Recommended approach for creating new tests
   - Test organization and naming strategies
   - Test code reuse patterns

2. **Test Suite Organization**
   - How to structure test suites
   - Test categorization and tagging
   - Parallel test execution strategies

Begin by acknowledging you're starting Phase 4 and asking the user about their IDE preference and a representative test scenario that would make a good demonstration. Explain that this test will serve as a reference implementation for the team to follow when creating additional automated tests.

By following this methodical approach you should be able to arrive at a working test. In addition to the initial implementation of the framework, you can use this Claude Project to get help with future feature development.

Tariq Azoor

Discussion about this post

Ready for more?