673 lines
19 KiB
Markdown
673 lines
19 KiB
Markdown
# Browser Automation Examples
|
|
|
|
Complete workflows demonstrating the `use_browser` tool capabilities.
|
|
|
|
## Table of Contents
|
|
|
|
1. [E-Commerce Workflows](#e-commerce-workflows)
|
|
2. [Form Automation](#form-automation)
|
|
3. [Data Extraction](#data-extraction)
|
|
4. [Multi-Tab Operations](#multi-tab-operations)
|
|
5. [Dynamic Content Handling](#dynamic-content-handling)
|
|
6. [Authentication Workflows](#authentication-workflows)
|
|
|
|
---
|
|
|
|
## E-Commerce Workflows
|
|
|
|
### Complete Booking Flow
|
|
|
|
Navigate multi-step booking process with validation:
|
|
|
|
```json
|
|
// Step 1: Search
|
|
{action: "navigate", payload: "https://booking.example.com"}
|
|
{action: "await_element", selector: "input[name=destination]"}
|
|
{action: "type", selector: "input[name=destination]", payload: "San Francisco"}
|
|
{action: "type", selector: "input[name=checkin]", payload: "2025-12-01"}
|
|
{action: "click", selector: "button.search"}
|
|
|
|
// Step 2: Select hotel
|
|
{action: "await_element", selector: ".hotel-results"}
|
|
{action: "click", selector: ".hotel-card:first-child .select"}
|
|
|
|
// Step 3: Choose room
|
|
{action: "await_element", selector: ".room-options"}
|
|
{action: "click", selector: ".room[data-type=deluxe] .book"}
|
|
|
|
// Step 4: Guest info
|
|
{action: "await_element", selector: "form.guest-info"}
|
|
{action: "type", selector: "input[name=firstName]", payload: "Jane"}
|
|
{action: "type", selector: "input[name=lastName]", payload: "Smith"}
|
|
{action: "type", selector: "input[name=email]", payload: "jane@example.com"}
|
|
|
|
// Step 5: Review
|
|
{action: "click", selector: "button.review"}
|
|
{action: "await_element", selector: ".summary"}
|
|
|
|
// Validate
|
|
{action: "extract", payload: "text", selector: ".hotel-name"}
|
|
{action: "extract", payload: "text", selector: ".total-price"}
|
|
```
|
|
|
|
### Price Comparison Across Sites
|
|
|
|
Open multiple stores in tabs and compare:
|
|
|
|
```json
|
|
// Store 1
|
|
{action: "navigate", payload: "https://store1.com/product/12345"}
|
|
{action: "await_element", selector: ".price"}
|
|
|
|
// Open Store 2
|
|
{action: "new_tab"}
|
|
{action: "navigate", tab_index: 1, payload: "https://store2.com/product/12345"}
|
|
{action: "await_element", tab_index: 1, selector: ".price"}
|
|
|
|
// Open Store 3
|
|
{action: "new_tab"}
|
|
{action: "navigate", tab_index: 2, payload: "https://store3.com/product/12345"}
|
|
{action: "await_element", tab_index: 2, selector: ".price"}
|
|
|
|
// Extract all prices
|
|
{action: "extract", tab_index: 0, payload: "text", selector: ".price"}
|
|
{action: "extract", tab_index: 1, payload: "text", selector: ".price"}
|
|
{action: "extract", tab_index: 2, payload: "text", selector: ".price"}
|
|
|
|
// Get product info
|
|
{action: "extract", tab_index: 0, payload: "text", selector: ".product-name"}
|
|
{action: "extract", tab_index: 0, payload: "text", selector: ".stock-status"}
|
|
```
|
|
|
|
### Product Data Extraction
|
|
|
|
Scrape structured product information:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://shop.example.com/product/123"}
|
|
{action: "await_element", selector: ".product-details"}
|
|
|
|
// Extract all product data with one eval
|
|
{action: "eval", payload: `
|
|
({
|
|
name: document.querySelector('h1.product-name').textContent.trim(),
|
|
price: document.querySelector('.price').textContent.trim(),
|
|
image: document.querySelector('.product-image img').src,
|
|
description: document.querySelector('.description').textContent.trim(),
|
|
stock: document.querySelector('.stock-status').textContent.trim(),
|
|
rating: document.querySelector('.rating').textContent.trim(),
|
|
reviews: Array.from(document.querySelectorAll('.review')).map(r => ({
|
|
author: r.querySelector('.author').textContent,
|
|
rating: r.querySelector('.stars').textContent,
|
|
text: r.querySelector('.review-text').textContent
|
|
}))
|
|
})
|
|
`}
|
|
```
|
|
|
|
### Batch Product Extraction
|
|
|
|
Get multiple products from category page:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://shop.example.com/category/electronics"}
|
|
{action: "await_element", selector: ".product-grid"}
|
|
|
|
// Extract all products as array
|
|
{action: "eval", payload: `
|
|
Array.from(document.querySelectorAll('.product-card')).map(card => ({
|
|
name: card.querySelector('.product-name').textContent.trim(),
|
|
price: card.querySelector('.price').textContent.trim(),
|
|
image: card.querySelector('img').src,
|
|
url: card.querySelector('a').href,
|
|
inStock: !card.querySelector('.out-of-stock')
|
|
}))
|
|
`}
|
|
```
|
|
|
|
---
|
|
|
|
## Form Automation
|
|
|
|
### Multi-Step Registration Form
|
|
|
|
Handle progressive form with validation at each step:
|
|
|
|
```json
|
|
// Step 1: Personal info
|
|
{action: "navigate", payload: "https://example.com/register"}
|
|
{action: "await_element", selector: "input[name=firstName]"}
|
|
|
|
{action: "type", selector: "input[name=firstName]", payload: "John"}
|
|
{action: "type", selector: "input[name=lastName]", payload: "Doe"}
|
|
{action: "type", selector: "input[name=email]", payload: "john@example.com"}
|
|
{action: "click", selector: "button.next"}
|
|
|
|
// Wait for step 2
|
|
{action: "await_element", selector: "input[name=address]"}
|
|
|
|
// Step 2: Address
|
|
{action: "type", selector: "input[name=address]", payload: "123 Main St"}
|
|
{action: "type", selector: "input[name=city]", payload: "Springfield"}
|
|
{action: "select", selector: "select[name=state]", payload: "IL"}
|
|
{action: "type", selector: "input[name=zip]", payload: "62701"}
|
|
{action: "click", selector: "button.next"}
|
|
|
|
// Wait for step 3
|
|
{action: "await_element", selector: "input[name=cardNumber]"}
|
|
|
|
// Step 3: Payment
|
|
{action: "type", selector: "input[name=cardNumber]", payload: "4111111111111111"}
|
|
{action: "select", selector: "select[name=expMonth]", payload: "12"}
|
|
{action: "select", selector: "select[name=expYear]", payload: "2028"}
|
|
{action: "type", selector: "input[name=cvv]", payload: "123"}
|
|
|
|
// Review before submit
|
|
{action: "click", selector: "button.review"}
|
|
{action: "await_element", selector: ".summary"}
|
|
|
|
// Extract confirmation
|
|
{action: "extract", payload: "markdown", selector: ".summary"}
|
|
```
|
|
|
|
### Search with Multiple Filters
|
|
|
|
Use dropdowns, checkboxes, and text inputs:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://library.example.com/search"}
|
|
{action: "await_element", selector: "form.search"}
|
|
|
|
// Category dropdown
|
|
{action: "select", selector: "select[name=category]", payload: "books"}
|
|
|
|
// Price range
|
|
{action: "type", selector: "input[name=priceMin]", payload: "10"}
|
|
{action: "type", selector: "input[name=priceMax]", payload: "50"}
|
|
|
|
// Checkboxes via JavaScript
|
|
{action: "eval", payload: "document.querySelector('input[name=inStock]').checked = true"}
|
|
{action: "eval", payload: "document.querySelector('input[name=freeShipping]').checked = true"}
|
|
|
|
// Search term and submit
|
|
{action: "type", selector: "input[name=query]", payload: "chrome devtools\n"}
|
|
|
|
// Wait for results
|
|
{action: "await_element", selector: ".results"}
|
|
|
|
// Count and extract
|
|
{action: "eval", payload: "document.querySelectorAll('.result').length"}
|
|
{action: "extract", payload: "text", selector: ".result-count"}
|
|
```
|
|
|
|
### File Upload
|
|
|
|
Handle file input using JavaScript:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://example.com/upload"}
|
|
{action: "await_element", selector: "input[type=file]"}
|
|
|
|
// Read file and set via JavaScript (for testing)
|
|
{action: "eval", payload: `
|
|
const fileInput = document.querySelector('input[type=file]');
|
|
const dataTransfer = new DataTransfer();
|
|
const file = new File(['test content'], 'test.txt', { type: 'text/plain' });
|
|
dataTransfer.items.add(file);
|
|
fileInput.files = dataTransfer.files;
|
|
`}
|
|
|
|
// Submit
|
|
{action: "click", selector: "button.upload"}
|
|
{action: "await_text", payload: "Upload complete"}
|
|
```
|
|
|
|
---
|
|
|
|
## Data Extraction
|
|
|
|
### Article Scraping
|
|
|
|
Extract blog post with metadata:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://blog.example.com/article"}
|
|
{action: "await_element", selector: "article"}
|
|
|
|
// Extract complete article structure
|
|
{action: "eval", payload: `
|
|
({
|
|
title: document.querySelector('article h1').textContent.trim(),
|
|
author: document.querySelector('.author-name').textContent.trim(),
|
|
date: document.querySelector('time').getAttribute('datetime'),
|
|
tags: Array.from(document.querySelectorAll('.tag')).map(t => t.textContent.trim()),
|
|
content: document.querySelector('article .content').textContent.trim(),
|
|
images: Array.from(document.querySelectorAll('article img')).map(img => ({
|
|
src: img.src,
|
|
alt: img.alt
|
|
})),
|
|
links: Array.from(document.querySelectorAll('article a')).map(a => ({
|
|
text: a.textContent.trim(),
|
|
href: a.href
|
|
}))
|
|
})
|
|
`}
|
|
```
|
|
|
|
### Table Data Extraction
|
|
|
|
Convert HTML table to structured JSON:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://example.com/data/table"}
|
|
{action: "await_element", selector: "table"}
|
|
|
|
// Extract table with headers
|
|
{action: "eval", payload: `
|
|
(() => {
|
|
const headers = Array.from(document.querySelectorAll('table thead th'))
|
|
.map(th => th.textContent.trim());
|
|
const rows = Array.from(document.querySelectorAll('table tbody tr'))
|
|
.map(tr => {
|
|
const cells = Array.from(tr.cells).map(td => td.textContent.trim());
|
|
return Object.fromEntries(headers.map((h, i) => [h, cells[i]]));
|
|
});
|
|
return rows;
|
|
})()
|
|
`}
|
|
```
|
|
|
|
### Paginated Results
|
|
|
|
Extract data across multiple pages:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://example.com/results?page=1"}
|
|
{action: "await_element", selector: ".results"}
|
|
|
|
// Page 1
|
|
{action: "eval", payload: "Array.from(document.querySelectorAll('.result')).map(r => r.textContent.trim())"}
|
|
|
|
// Navigate to page 2
|
|
{action: "click", selector: "a.next-page"}
|
|
{action: "await_element", selector: ".results"}
|
|
{action: "await_text", payload: "Page 2"}
|
|
|
|
// Page 2
|
|
{action: "eval", payload: "Array.from(document.querySelectorAll('.result')).map(r => r.textContent.trim())"}
|
|
|
|
// Continue pattern for additional pages...
|
|
```
|
|
|
|
---
|
|
|
|
## Multi-Tab Operations
|
|
|
|
### Email Receipt Extraction
|
|
|
|
Find specific email and extract data:
|
|
|
|
```json
|
|
// List available tabs
|
|
{action: "list_tabs"}
|
|
|
|
// Switch to email tab (assume index 2 from list)
|
|
{action: "click", tab_index: 2, selector: "a[title*='Receipt']"}
|
|
{action: "await_element", tab_index: 2, selector: ".email-body"}
|
|
|
|
// Extract receipt details
|
|
{action: "extract", tab_index: 2, payload: "text", selector: ".order-number"}
|
|
{action: "extract", tab_index: 2, payload: "text", selector: ".total-amount"}
|
|
{action: "extract", tab_index: 2, payload: "markdown", selector: ".items-list"}
|
|
```
|
|
|
|
### Cross-Site Data Correlation
|
|
|
|
Extract from one site, verify on another:
|
|
|
|
```json
|
|
// Get company phone from website
|
|
{action: "navigate", payload: "https://company.com/contact"}
|
|
{action: "await_element", selector: ".contact-info"}
|
|
{action: "extract", payload: "text", selector: ".phone-number"}
|
|
|
|
// Store result: "+1-555-0123"
|
|
|
|
// Open verification site in new tab
|
|
{action: "new_tab"}
|
|
{action: "navigate", tab_index: 1, payload: "https://phonevalidator.com"}
|
|
{action: "await_element", tab_index: 1, selector: "input[name=phone]"}
|
|
|
|
// Fill and search
|
|
{action: "type", tab_index: 1, selector: "input[name=phone]", payload: "+1-555-0123\n"}
|
|
{action: "await_element", tab_index: 1, selector: ".results"}
|
|
|
|
// Extract validation result
|
|
{action: "extract", tab_index: 1, payload: "text", selector: ".verification-status"}
|
|
```
|
|
|
|
### Parallel Data Collection
|
|
|
|
Collect data from multiple sources simultaneously:
|
|
|
|
```json
|
|
// Tab 0: Weather
|
|
{action: "navigate", tab_index: 0, payload: "https://weather.com/city"}
|
|
{action: "await_element", tab_index: 0, selector: ".temperature"}
|
|
|
|
// Tab 1: News
|
|
{action: "new_tab"}
|
|
{action: "navigate", tab_index: 1, payload: "https://news.com"}
|
|
{action: "await_element", tab_index: 1, selector: ".headlines"}
|
|
|
|
// Tab 2: Stock prices
|
|
{action: "new_tab"}
|
|
{action: "navigate", tab_index: 2, payload: "https://stocks.com"}
|
|
{action: "await_element", tab_index: 2, selector: ".market-summary"}
|
|
|
|
// Extract all data
|
|
{action: "extract", tab_index: 0, payload: "text", selector: ".temperature"}
|
|
{action: "extract", tab_index: 1, payload: "text", selector: ".headline:first-child"}
|
|
{action: "extract", tab_index: 2, payload: "text", selector: ".market-summary"}
|
|
```
|
|
|
|
---
|
|
|
|
## Dynamic Content Handling
|
|
|
|
### Infinite Scroll Loading
|
|
|
|
Load all content with scroll-triggered pagination:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://example.com/feed"}
|
|
{action: "await_element", selector: ".feed-item"}
|
|
|
|
// Count initial items
|
|
{action: "eval", payload: "document.querySelectorAll('.feed-item').length"}
|
|
|
|
// Scroll and wait multiple times
|
|
{action: "eval", payload: "window.scrollTo(0, document.body.scrollHeight)"}
|
|
{action: "eval", payload: "new Promise(r => setTimeout(r, 2000))"}
|
|
|
|
{action: "eval", payload: "window.scrollTo(0, document.body.scrollHeight)"}
|
|
{action: "eval", payload: "new Promise(r => setTimeout(r, 2000))"}
|
|
|
|
{action: "eval", payload: "window.scrollTo(0, document.body.scrollHeight)"}
|
|
{action: "eval", payload: "new Promise(r => setTimeout(r, 2000))"}
|
|
|
|
// Extract all loaded items
|
|
{action: "eval", payload: `
|
|
Array.from(document.querySelectorAll('.feed-item')).map(item => ({
|
|
title: item.querySelector('.title').textContent.trim(),
|
|
date: item.querySelector('.date').textContent.trim(),
|
|
url: item.querySelector('a').href
|
|
}))
|
|
`}
|
|
```
|
|
|
|
### Wait for AJAX Response
|
|
|
|
Wait for loading indicator to disappear:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://app.com/dashboard"}
|
|
{action: "await_element", selector: ".content"}
|
|
|
|
// Trigger AJAX request
|
|
{action: "click", selector: "button.load-data"}
|
|
|
|
// Wait for spinner to appear then disappear
|
|
{action: "eval", payload: `
|
|
new Promise(resolve => {
|
|
const checkGone = () => {
|
|
const spinner = document.querySelector('.spinner');
|
|
if (!spinner || spinner.style.display === 'none') {
|
|
resolve(true);
|
|
} else {
|
|
setTimeout(checkGone, 100);
|
|
}
|
|
};
|
|
checkGone();
|
|
})
|
|
`}
|
|
|
|
// Now safe to extract
|
|
{action: "extract", payload: "text", selector: ".data-table"}
|
|
```
|
|
|
|
### Modal Dialog Handling
|
|
|
|
Open modal, interact, wait for close:
|
|
|
|
```json
|
|
{action: "click", selector: "button.open-settings"}
|
|
{action: "await_element", selector: ".modal.visible"}
|
|
|
|
// Interact with modal
|
|
{action: "type", selector: ".modal input[name=username]", payload: "newuser"}
|
|
{action: "select", selector: ".modal select[name=theme]", payload: "dark"}
|
|
{action: "click", selector: ".modal button.save"}
|
|
|
|
// Wait for modal to close
|
|
{action: "eval", payload: `
|
|
new Promise(resolve => {
|
|
const check = () => {
|
|
const modal = document.querySelector('.modal.visible');
|
|
if (!modal) {
|
|
resolve(true);
|
|
} else {
|
|
setTimeout(check, 100);
|
|
}
|
|
};
|
|
check();
|
|
})
|
|
`}
|
|
|
|
// Verify settings saved
|
|
{action: "await_text", payload: "Settings saved"}
|
|
```
|
|
|
|
### Wait for Button Enabled
|
|
|
|
Wait for form validation before submission:
|
|
|
|
```json
|
|
{action: "type", selector: "input[name=email]", payload: "user@example.com"}
|
|
{action: "type", selector: "input[name=password]", payload: "securepass123"}
|
|
|
|
// Wait for submit button to become enabled
|
|
{action: "eval", payload: `
|
|
new Promise(resolve => {
|
|
const check = () => {
|
|
const btn = document.querySelector('button[type=submit]');
|
|
if (btn && !btn.disabled && !btn.classList.contains('disabled')) {
|
|
resolve(true);
|
|
} else {
|
|
setTimeout(check, 100);
|
|
}
|
|
};
|
|
check();
|
|
})
|
|
`}
|
|
|
|
// Now safe to click
|
|
{action: "click", selector: "button[type=submit]"}
|
|
```
|
|
|
|
---
|
|
|
|
## Authentication Workflows
|
|
|
|
### Standard Login
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://app.example.com/login"}
|
|
{action: "await_element", selector: "form.login"}
|
|
|
|
// Fill credentials
|
|
{action: "type", selector: "input[name=email]", payload: "user@example.com"}
|
|
{action: "type", selector: "input[name=password]", payload: "password123\n"}
|
|
|
|
// Wait for redirect
|
|
{action: "await_text", payload: "Dashboard"}
|
|
|
|
// Verify logged in
|
|
{action: "extract", payload: "text", selector: ".user-name"}
|
|
```
|
|
|
|
### OAuth Flow
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://app.example.com/connect"}
|
|
{action: "await_element", selector: "button.oauth-login"}
|
|
|
|
// Trigger OAuth
|
|
{action: "click", selector: "button.oauth-login"}
|
|
|
|
// Wait for OAuth provider page
|
|
{action: "await_text", payload: "Authorize"}
|
|
|
|
// Fill OAuth credentials
|
|
{action: "await_element", selector: "input[name=username]"}
|
|
{action: "type", selector: "input[name=username]", payload: "oauthuser"}
|
|
{action: "type", selector: "input[name=password]", payload: "oauthpass\n"}
|
|
|
|
// Wait for redirect back
|
|
{action: "await_text", payload: "Connected successfully"}
|
|
```
|
|
|
|
### Session Persistence Check
|
|
|
|
```json
|
|
// Load page
|
|
{action: "navigate", payload: "https://app.example.com/dashboard"}
|
|
{action: "await_element", selector: "body"}
|
|
|
|
// Check if logged in via cookie/localStorage
|
|
{action: "eval", payload: "document.cookie.includes('session_id')"}
|
|
{action: "eval", payload: "localStorage.getItem('auth_token') !== null"}
|
|
|
|
// Verify user data loaded
|
|
{action: "extract", payload: "text", selector: ".user-profile"}
|
|
```
|
|
|
|
---
|
|
|
|
## Advanced Patterns
|
|
|
|
### Conditional Workflow
|
|
|
|
Branch based on page content:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://example.com/status"}
|
|
{action: "await_element", selector: "body"}
|
|
|
|
// Check status
|
|
{action: "extract", payload: "text", selector: ".status-message"}
|
|
|
|
// If result contains "Available":
|
|
{action: "click", selector: "button.purchase"}
|
|
{action: "await_text", payload: "Added to cart"}
|
|
|
|
// If result contains "Out of stock":
|
|
{action: "click", selector: "button.notify-me"}
|
|
{action: "type", selector: "input[name=email]", payload: "notify@example.com\n"}
|
|
```
|
|
|
|
### Error Recovery
|
|
|
|
Handle and retry failed operations:
|
|
|
|
```json
|
|
{action: "navigate", payload: "https://app.example.com/data"}
|
|
{action: "await_element", selector: ".content"}
|
|
|
|
// Attempt operation
|
|
{action: "click", selector: "button.load"}
|
|
|
|
// Check for error
|
|
{action: "eval", payload: "!!document.querySelector('.error-message')"}
|
|
|
|
// If error present, retry
|
|
{action: "click", selector: "button.retry"}
|
|
{action: "await_element", selector: ".data-loaded"}
|
|
```
|
|
|
|
### Screenshot Comparison
|
|
|
|
Capture before and after states:
|
|
|
|
```json
|
|
// Initial state
|
|
{action: "navigate", payload: "https://example.com"}
|
|
{action: "await_element", selector: ".content"}
|
|
{action: "screenshot", payload: "/tmp/before.png"}
|
|
|
|
// Make changes
|
|
{action: "click", selector: "button.dark-mode"}
|
|
{action: "await_element", selector: "body.dark"}
|
|
|
|
// Capture new state
|
|
{action: "screenshot", payload: "/tmp/after.png"}
|
|
|
|
// Or screenshot specific element
|
|
{action: "screenshot", payload: "/tmp/header.png", selector: "header"}
|
|
```
|
|
|
|
---
|
|
|
|
## Tips for Complex Workflows
|
|
|
|
### Build Incrementally
|
|
|
|
Start simple, add complexity:
|
|
|
|
1. Navigate and verify page loads
|
|
2. Extract one element
|
|
3. Add interaction
|
|
4. Add waiting logic
|
|
5. Add error handling
|
|
6. Add validation
|
|
|
|
### Use JavaScript for Complex Logic
|
|
|
|
When multiple operations needed, use `eval`:
|
|
|
|
```json
|
|
{action: "eval", payload: `
|
|
(async () => {
|
|
// Complex multi-step logic
|
|
const results = [];
|
|
const items = document.querySelectorAll('.item');
|
|
|
|
for (const item of items) {
|
|
if (item.classList.contains('active')) {
|
|
results.push({
|
|
id: item.dataset.id,
|
|
text: item.textContent.trim()
|
|
});
|
|
}
|
|
}
|
|
|
|
return results;
|
|
})()
|
|
`}
|
|
```
|
|
|
|
### Validate Selectors
|
|
|
|
Always test selectors return expected elements:
|
|
|
|
```json
|
|
// Check element exists
|
|
{action: "eval", payload: "!!document.querySelector('button.submit')"}
|
|
|
|
// Check element visible
|
|
{action: "eval", payload: "window.getComputedStyle(document.querySelector('button.submit')).display !== 'none'"}
|
|
|
|
// Check element count
|
|
{action: "eval", payload: "document.querySelectorAll('.item').length"}
|
|
```
|