Your first JavaScript collection scenario

This example collects catalog cards and saves an SKU, name, price, and URL. Use it as a starting point: replace the URL and CSS selectors with ones for your website.

Preparation

Create a site and then a JavaScript scenario. Define site fields sku, name, price, and url. If the catalog URL changes, add a catalog_url scenario parameter. ctx.param() reads it when the run starts.

Code

async function run(ctx) {
  const url = ctx.param('catalog_url', '');
  if (!url) ctx.abort('Enter a catalog URL');
  ctx.log('Loading catalog');

  const response = await ctx.http().get(url);
  if (!response.ok()) ctx.fail(`Catalog returned HTTP ${response.status}`);
  const cards = response.dom().qa('.product-card');
  if (!cards.length) ctx.fail('Product cards were not found');

  let saved = 0;
  for (const card of cards) {
    if (await ctx.cancelled()) ctx.abort('Collection stopped');
    const sku = card.attr('data-sku');
    const link = card.q('a')?.abs('href');
    if (!sku || !link) continue;
    ctx.emit({
      sku,
      name: card.q('.title')?.text() ?? '',
      price: ctx.text().money(card.q('.price')?.text() ?? null),
      url: link,
    }, sku);
    saved++;
  }

  ctx.metric('products_saved', saved);
  ctx.progress(1, 1, 'Catalog processed');
  ctx.log('Done', { saved });
}

ctx.http().get() is asynchronous, so it needs await. HTML parsing and price cleanup happen locally. ctx.emit() submits a plain object and uses sku as a stable key. If the website has different products with the same SKU, choose a key that is truly unique for your task.

Validate and extend

Start with a trial run. Compare several records with the source website: names, prices, links, and omitted products. For a multi-page catalog, add pagination. If products appear only after a click, use browser actions. After checking the result, publish the version and start a regular run separately.

Scenario lifecycle · Context reference