Pagination and categories in Python

Every crawl needs a maximum page count, a stop condition, and stable product keys. HTTP 200 does not prove a new page was loaded.

Page number in a URL

def run(ctx):
    base = ctx.param('catalog_url', '')
    if not base:
        ctx.abort('Enter a catalog URL')
    max_pages = min(100, max(1, int(ctx.param('max_pages', 10))))
    seen = set()
    for number in range(1, max_pages + 1):
        if ctx.cancelled():
            ctx.abort('Collection stopped')
        response = ctx.http().get(ctx.url().with_param(base, 'page', number))
        if not response.ok():
            ctx.fail('Page ' + str(number) + ': HTTP ' + str(response.status))
        cards = response.dom().qa('.product-card')
        if not cards:
            break
        added = 0
        for card in cards:
            sku = card.attr('data-sku')
            if not sku or sku in seen:
                continue
            seen.add(sku)
            added += 1
            name = card.q('.title')
            ctx.emit({'sku': sku, 'name': name.text() if name is not None else ''}, sku)
        ctx.progress(number, max_pages, 'Catalog pages')
        if added == 0:
            break

Python range excludes its right boundary, hence max_pages + 1. The local set removes duplicates only within this run; do not emit a set as data. The key passed to emit links records across runs. Selectors are examples and require testing against your source.

Categories and cursors

Check that category_urls is a list. Bound both category and page counts. Choose whether one SKU across categories should be one record or several: include the category in the key only when separate records are intended.

For next buttons or infinite scrolling, use the browser example. Wait for changed cards or the expected XHR. Stop when the source repeats an already seen result. A cursor API also needs a set of seen cursors so a repeated cursor cannot loop forever.

Update persistent page state after processing that page. Retry safety should come from stable keys, not an assumption that a previously sent batch was fully committed.