Clicks, scrolling, and XHR in Python

Use the browser when a website generates cards using JavaScript or requires interaction. Browser and HTTP calls follow the same address policies, while a browser page owns its session. Self-hosted agents need browser support enabled.

def run(ctx):
    page = ctx.browser().open(ctx.param('catalog_url'), ['*/api/products*'])
    try:
        page.click('button.show-products')
        page.type('input[name="search"]', 'cable')
        page.press('input[name="search"]', 'Enter')
        page.wait_for('.product-card')
        seen = set()
        for step in range(10):
            if ctx.cancelled():
                ctx.abort('Stopped by the user')
            dom = ctx.dom(page.content(), ctx.param('catalog_url'))
            added = 0
            for card in dom.qa('.product-card'):
                sku = card.attr('data-sku')
                if not sku or sku in seen:
                    continue
                seen.add(sku)
                added += 1
                ctx.emit({'sku': sku, 'name': card.text()}, sku)
            if added == 0:
                break
            page.scroll(1, 500)
            ctx.progress(step + 1, 10, 'Scrolling catalog')
    finally:
        page.close()

Selectors and the ten-step bound are examples. Finding an existing card after wait_for does not guarantee the next batch has arrived. The example stops when it finds no new stable keys.

Start response capture before an action. The second argument of open captures initial requests; capture_responses(patterns) enables later capture and clears the previous buffer. An empty list disables capture. wait_for_response(pattern, timeout_ms) waits within a bound; responses() reads captured results. Do not log private response bodies or authentication headers.

Always close the page in finally, including on errors and cancellation. screenshot() returns an artifact identifier rather than PNG bytes inside the Python process. See BrowserPage for method arguments and default timeouts.