Writing Effective Test Cases

Test cases are the backbone of your QA coverage. In Certyn, each test case is a set of natural-language instructions that an AI agent follows in a real browser. The better the instructions, the more reliable the results.

Anatomy of a Test Case

Every test case has:

• Name — short, descriptive title ("User can complete checkout with credit card")
• Description — what this test validates and why it matters
• Instructions — step-by-step procedure the agent follows
• Tags — categories for organizing and selecting tests (e.g., smoke, checkout, regression)

Writing Good Instructions

Be specific about actions

Instead of "test the login", write explicit steps:

"Navigate to /login. Enter 'test@example.com' in the email field. Enter 'password123' in the password field. Click 'Sign in'. Verify the dashboard loads and shows the user's name."

State expected outcomes

Every test should end with a clear pass/fail condition:

"After submitting the form, verify: (1) a success toast appears with 'Profile updated', (2) the saved values persist after page refresh."

Include context when needed

If the test depends on preconditions, state them:

"Prerequisites: user must have at least one item in their cart. Navigate to /cart and proceed to checkout..."

Keep instructions sequential

Number your steps or write them as a natural flow. Avoid branching logic — one test, one path:

"1. Navigate to the products page. 2. Add the first product to the cart. 3. Open the cart. 4. Increase quantity to 3. 5. Verify the total updates correctly."

Test Case Lifecycle

Creation

Test cases can be created:

• Manually — write instructions in the dashboard or CLI
• By the exploratory agent — when it discovers testable behaviors during exploration
• By promoting observations — when a finding should become a recurring test

Execution

When a process runs, each included test case gets its own execution. The Test Executor agent:

1. Reads the instructions
2. Executes each step in a browser
3. Reports the outcome: passed, failed, or blocked

Review and maintenance

Over time, test cases may need updates:

• Needs Review — flagged for human review before execution
• Quarantined — fails intermittently, excluded until stabilized
• Obsolete — superseded by a newer test case

Tagging Strategy

Tags determine which tests run in which process. A consistent strategy saves time:

Tag	Purpose	Process
smoke	Critical path, must always pass	Smoke Suite
regression	Full coverage	Regression Suite
checkout	Checkout-specific tests	Checkout Flow
auth	Authentication and authorization	Auth Suite
mobile	Mobile-specific behaviors	Mobile Regression

A single test case can have multiple tags. A checkout login test might be tagged smoke, checkout, and auth.

Test Outcomes

The agent reports one of three outcomes:

• Passed — all steps completed, all assertions met
• Failed — the agent detected an issue (wrong content, broken flow, error state)
• Blocked — the agent couldn't complete the test (environment down, missing data, navigation error)

Failed tests indicate real bugs. Blocked tests indicate environment or setup issues.

Tips

• Write instructions as if explaining to a new team member who has never seen the app
• One test case, one scenario — don't combine "login and checkout and settings" in one test
• Use tags generously — they're the primary way to organize and select tests for processes
• Review agent-generated test cases — they're a good starting point but may need refinement
• Keep test counts manageable — 50 well-written tests beat 200 vague ones
• Use the CLI to quickly manage tests: certyn tests list, certyn tests overview, certyn tests flaky