Browser Automation with BugBrain

The BugBrain AI agent uses Playwright (an open-source browser automation library) to control a real Chrome browser. Combined with AI decision-making, it can handle complex, dynamic web applications that deterministic scripts cannot.

What the Agent Controls

The agent has full control over a Playwright-powered browser:

┌──────────────────────────────────┐
│  BugBrain AI Agent (Claude)      │
│  - Sees: screenshot, page DOM    │
│  - Decides: what action to take  │
└──────────┬───────────────────────┘
           │
     ┌─────▼─────────┐
     │ Playwright    │
     │ - Navigate    │
     │ - Click       │
     │ - Type        │
     │ - Wait        │
     │ - Assert      │
     └─────┬─────────┘
           │
     ┌─────▼──────────────────┐
     │ Chrome / Chromium      │
     │ (the actual browser)   │
     └────────────────────────┘
           │
     ┌─────▼──────────────────┐
     │ Your Web Application   │
     │ (under test)           │
     └────────────────────────┘

Browser Configuration

Each test execution configures the browser with:

Device Emulation

Simulate specific devices without needing physical hardware:

Create device presets in Settings → Device Presets to reuse configurations across tests.

Network & Locale

Fine-tune execution environment:

{
  "viewport": {
    "width": 1920,
    "height": 1080
  },
  "device_scale_factor": 1,
  "is_touch_enabled": false,
  "user_agent": "Mozilla/5.0...",
  "locale": "en-US",
  "timezone": "America/New_York",
  "proxy": "http://proxy.internal:8080"  // Optional
}

Browser Options

Control browser behavior:

• Headless mode (default) — No visible browser window, ~30% faster
• Headed mode — Show browser window for debugging
• Extra Chromium args — Disable GPU, increase memory, etc.

How Playwright Works

Playwright automates a real Chrome/Chromium browser via the Chrome DevTools Protocol (CDP). This means:

✅ Handles modern JavaScript — React, Vue, Angular, dynamic content ✅ Real browser environment — JavaScript runs in the actual browser context ✅ No synthetic events — Not simulating clicks; actually clicking ✅ Cookie/session support — Maintains cookies across navigation ✅ Network interception — Can capture/inspect all HTTP requests

Actions the Agent Can Perform

// Navigation
await page.goto('https://example.com');
await page.goBack();
await page.reload();
 
// Interaction
await page.click('#submit-button');
await page.fill('#email', 'user@example.com');
await page.select('#country', 'USA');
await page.check('#terms-checkbox');
await page.press('#search', 'Enter');
 
// Waiting
await page.waitForSelector('#success-message');
await page.waitForNavigation();
await page.waitForFunction(() => window.data !== null);
 
// Assertions
expect(page).toContainText('Welcome');
expect(page).toHaveURL(/checkout/);
expect(element).toBeVisible();
 
// Data Extraction
const title = await page.textContent('#page-title');
const options = await page.$$eval('select option', els =>
  els.map(el => el.textContent)
);
 
// Screenshots
await page.screenshot({ path: 'page.png' });

Handling Complex Scenarios

JavaScript-Heavy Applications

Playwright automatically waits for JavaScript to settle:

// React app that renders dynamically
// ✓ Playwright waits for React to finish rendering
// ✓ Agent sees the final rendered state
// ✓ Can then interact with rendered elements

Asynchronous Operations

The agent is smart about waiting:

// User clicks "Search"
// → API request is made
// → Results load asynchronously
// → Playwright waits for results to appear
// → Agent verifies results are visible

Modal Dialogs & Overlays

Agent can interact with modals:

// Pop-up appears
// Agent waits for modal to be visible
// Agent fills form inside modal
// Agent clicks "Submit"
// Modal closes, page continues

Form Submission Handling

Different form types are handled seamlessly:

// Traditional form submission
form.submit() → page navigates → page loads
 
// AJAX form submission
form.submit() → API request → page updates in-place
 
// Multi-step form
Step 1 submission → Step 2 appears → continue
 
// File uploads
await page.setInputFiles('input[type="file"]', 'path/to/file')

Execution Parameters

You can configure how the agent executes tests:

Max Iterations

Default: 50 iterations
Range: 1–100

Each iteration is one “think-act-observe” cycle. More iterations = more actions, but also more cost and time.

When to adjust:

• ↑ Increase for complex multi-step flows with lots of waiting
• ↓ Decrease for simple tests (login, button click) to save cost

Execution Timeout

Default: 300 seconds (5 minutes)
Range: 10–600 seconds

Maximum duration for a single test. If exceeded, test fails with “timeout” error.

When to adjust:

• ↑ Increase if your app is slow or tests lots of pages
• ↓ Decrease for fast tests or to catch hangs early

Wait Configuration

{
  "element_wait_timeout_ms": 30000,    // 30s to find element
  "page_load_timeout_ms": 30000,       // 30s for page to load
  "action_delay_ms": 100,              // 100ms between actions
  "retry_on_failure": true,            // Retry failed actions
  "max_retries": 3                     // Up to 3 retries per action
}

Session Caching (Optimization)

After a successful login, Playwright’s storageState (cookies + localStorage) is saved. On the next run with the same persona:

Run 1:
  1. Open login page
  2. Fill email
  3. Fill password
  4. Click Sign In
  5. Wait for redirect
  [Total: ~8-10 seconds]

Run 2 (cached session):
  1. Restore cookies + localStorage
  2. Navigate to app
  3. Verify authenticated
  [Total: ~1-2 seconds]

Cache TTL: 8 hours (configurable)

Sessions are invalidated automatically when you update persona credentials. You can also manually clear sessions:

DELETE /api/v1/sessions/{persona_id}

Failure Recovery

When an action fails, the agent attempts recovery:

Step: "Click the 'Sign In' button"

Attempt 1:
  Click [data-testid='signin-button'] → element not found

Attempt 2 (retry with different selector):
  Click button:contains('Sign In') → element not found

Attempt 3 (wait and try again):
  Wait 2 seconds
  Click [data-testid='signin-button'] → success ✓

Result: Action succeeded after retry

The agent tries up to 3 retries per action (configurable) with exponential backoff.

Performance Tuning

For Speed

• Use headless mode (default)
• Decrease max iterations for simple tests
• Use session caching to skip redundant logins

For Reliability

• Increase wait timeouts for slow applications
• Use explicit waits in your test steps
• Add retry logic for flaky elements
• Use stable selectors (data attributes work best)

For Cost Optimization

• Cache sessions — Eliminates redundant logins
• Optimize iterations — Fewer steps = faster execution
• Use shorter timeouts to fail fast
• Batch related tests for better resource utilization

Monitoring Execution

While tests run, monitor:

• Real-time progress — Current step, time elapsed
• Screenshots — Visual verification of each step
• Console logs — JavaScript errors and warnings
• Network requests — API calls made by the app
• Telemetry — Event traces for debugging

View all this in the Execution Detail page after the test finishes.

Limits & Constraints

Troubleshooting

Tests Run Very Slowly

• Check if app has performance issues (use DevTools)
• Increase wait timeouts for your test
• Check for slow API endpoints
• Review your test steps for unnecessary waits

Agent Gets Confused / Wrong Clicks

• Use more specific selectors ([data-testid] over .button)
• Add explicit assertions between steps
• Break complex flows into smaller tests
• Provide clearer, more specific step descriptions

Session Caching Not Working

• Verify persona type is correct (basic, OTP, TOTP)
• Check if credentials are being stored properly
• Try manually clearing sessions and re-authenticating
• Ensure persona is correctly linked to the project

Next Steps

• Failure Analysis — Understand why tests fail
• Authentication Testing — Set up login testing
• Test Execution — Run and monitor tests