JavaScript-powered websites: clicks and scrolling
Start with a regular HTTP request. If it lacks the products, open the page in the browser. A browser uses more resources, so keep the page open only for the required actions.
Click and verify new data
After a click, wait for loading and compare keys already collected. The selectors below are examples; choose them for your website. Bound the number of repeats so a broken button cannot create an endless loop.
async function run(ctx) {
const url = ctx.param('catalog_url', '');
if (!url) ctx.abort('Enter a catalog URL');
const page = await ctx.browser().open(url);
const seen = new Set();
try {
for (let step = 0; step < 5; step++) {
if (await ctx.cancelled()) ctx.abort('Collection stopped');
await page.waitFor('.product-card');
const dom = ctx.dom(await page.content(), url);
let newRecords = 0;
for (const card of dom.qa('.product-card')) {
const sku = card.attr('data-sku');
if (!sku || seen.has(sku)) continue;
seen.add(sku);
newRecords++;
ctx.emit({ sku, name: card.q('.title')?.text() ?? '' }, sku);
}
ctx.progress(step + 1, 5, 'Catalog');
if (step === 4 || newRecords === 0 || !dom.q('button.load-more')) break;
await page.click('button.load-more');
}
} finally {
await page.close();
}
}
Use await page.scroll(steps, delayMs) or await page.scrollTo(selector) for scrolling. type(), press(), and select() handle form fields. Inspect changed page content after an action rather than relying only on a successful click.
Network responses
If the page requests JSON after a click, enable capture for selected URLs with captureResponses(patterns), then use responses() or waitForResponse(pattern, timeoutMs). Capture cannot recover requests made before it was enabled. For initial page loading, pass the patterns as the second argument to ctx.browser().open(url, patterns).
Never put tokens, cookies, or authorization headers from captured responses into output records. Select only needed product fields. The browser service limits pattern count and response size.