Documentation

Cloudflare sign-in page: keep one collection session (TypeScript)

Follow the open-form, sign-in and catalog-reading sequence on the ScrapingCourse page with an additional access check.

Choose TypeScript, paste the complete source from this article and set the run parameters below. The source for this collection is ScrapingCourse. This walkthrough applies to the specified demonstration page.

Example settings

{
  "path": "/login/cf-antibot",
  "mode": "login",
  "expected": 12
}

path selects the page, mode selects an algorithm branch, and expected sets the control count for this example. It is not a plan limit. When the source dataset changes, check the expected count against it.

How the algorithm works

Select /login/cf-antibot and the login mode. The remaining sign-in algorithm is shared.
Open the page with the browser SDK and the auto_captcha setting. Site settings govern additional page checks; the source does not implement a separate challenge-solving procedure.
Read the public demonstration credential hint when the form is available. The code does not store fixed sign-in values for this page.
Fill both inputs and submit the normal form in the already open session.
Wait for a nonempty product name after sign-in. The appearance of actual data starts card extraction.
Emit products with stable keys, check the expected count and close the browser.

Products are processed by collect(html). Read the name first, skip empty names, and use the card link as the key, falling back to the name when necessary. seen prevents the same key being emitted twice. This matters when a later HTML read includes products already collected.

Details that matter

This variant differs from ordinary sign-in by its page route. The field names, card collector and completeness check remain the same.

TypeScript details

In TypeScript, ctx: Akmelo.Context supplies SDK method hints, Promise<void> describes the result of run, and Set<string> holds string keys. These types do not change execution order: network and browser calls still require await. collect(html: string): number returns the current page card count.

Expected output

The configured demonstration dataset expects 12 unique records. The final check compares the actual unique key count with expected.

Field	Contents
`name`	Product name with leading and trailing whitespace removed.
`price`	Price converted from text to a number.
`image`	Image URL read from the card src attribute.

Complete scenario source

The source is shared by this collection. This example runs its login branch using the parameters above. Copy the complete block into the TypeScript editor; no fixed sign-in credentials or service keys need to be added.

async function run(ctx: Akmelo.Context): Promise<void> {
  const origin = 'https://www.scrapingcourse.com';
  const path = String(ctx.param('path', '/pagination'));
  const mode = String(ctx.param('mode', 'http'));
  const auto = ctx.param<boolean>('auto_captcha', true) !== false;
  const seen = new Set<string>();
  const collect = (html: string): number => {
    const dom = ctx.dom(html);
    for (const card of dom.qa('.product-item, li.product')) {
      const name = card.q('.product-name, .woocommerce-loop-product__title')?.text().trim() ?? '';
      const key = card.q('a')?.attr('href') ?? name;
      if (!name || seen.has(key)) continue;
      const price = ctx.text().money(card.q('.product-price, .price')?.text() ?? '');
      const image = card.q('img')?.attr('src') ?? '';
      seen.add(key);
      ctx.emit({name, price, image}, key);
    }
    return dom.qa('.product-item, li.product').length;
  };
  if (mode === 'http') {
    let url: string | null = origin + path;
    const visited = new Set<string>();
    while (url && visited.size < 50) {
      if (visited.has(url)) throw new Error('Pagination returned an already processed page');
      visited.add(url);
      const response = await ctx.http().withoutCache().get(url, auto);
      if (!response.ok()) throw new Error('The source could not be opened: ' + response.status);
      const html = response.body;
      collect(html);
      const next = ctx.dom(html).attr('a[rel="next"], a.next.page-numbers', 'href');
      url = next ? ctx.url().absolutize(url, next) : null;
    }
    if (url) throw new Error('The pagination safety limit was reached');
  } else {
    const page = await ctx.browser().open(origin + path, ['*/ajax/products*'], {}, auto);
    try {
      if (mode === 'login') {
        const dom = ctx.dom(await page.content());
        const hint = dom.qa('p').map(node => node.text()).find(text => text.includes('Demo Credentials:')) ?? '';
        const password = hint.match(/Password:\s*([^\s|]+)/i)?.[1];
        let email = hint.match(/Email:\s*([^\s|]+@[^\s|]+)/i)?.[1];
        const encoded = dom.attr('[data-cfemail]', 'data-cfemail');
        if (!email && encoded && encoded.length > 2) {
          const key = parseInt(encoded.slice(0, 2), 16);
          email = '';
          for (let i = 2; i < encoded.length; i += 2) email += String.fromCharCode(parseInt(encoded.slice(i, i + 2), 16) ^ key);
        }
        if (!email || !password) throw new Error('The page does not contain demo credentials');
        await page.type('#email', email);
        await page.type('#password', password);
        await page.click('button[type="submit"]');
      }
      if (mode === 'challenge') {
        const text = (ctx.dom(await page.content()).text('body') ?? '').replace(/\s+/g, ' ');
        if (!/You (?:successfully )?bypassed|successfully bypassed|Congratulations/i.test(text)) throw new Error('The expected page content was not received');
        ctx.emit({name: path, passed: true}, path);
      } else {
        await page.waitFor('.product-name:not(:empty)', 15000);
        if (mode === 'button') {
          let complete = false;
          for (let i = 0; i < 30; i++) {
            const before = ctx.dom(await page.content());
            collect(await page.content());
            const button = before.q('#load-more-btn');
            if (!button || /display:\s*none/.test(button.attr('style') ?? '')) { complete = true; break; }
            const offset = button.attr('data-offset');
            const count = before.qa('.product-item').length;
            await page.click('#load-more-btn');
            const response = await page.waitForResponse('*offset=' + offset, 15000) as Akmelo.Row | null;
            if (!response || (Number(response.status) < 200 || Number(response.status) >= 300) || response.truncated || response.error) throw new Error('The next batch could not be loaded');
            if (String(response.body ?? '').trim() === '') { complete = true; break; }
            await page.waitFor('.product-item:nth-child(' + (count + 1) + ')');
          }
          if (!complete) throw new Error('The batch loading safety limit was reached');
        }
        if (mode === 'scroll') {
          let stable = 0;
          let previous = 0;
          for (let i = 0; i < 40 && stable < 3; i++) {
            await page.scroll(3, 400);
            const count = collect(await page.content());
            stable = count === previous ? stable + 1 : 0;
            previous = count;
          }
          if (stable < 3) throw new Error('The feed is still loading: increase the scroll limit');
        }
        collect(await page.content());
      }
    } finally { await page.close(); }
  }
  const expected = Number(ctx.param('expected', 1));
  if (mode !== 'challenge' && seen.size !== expected) throw new Error('Incomplete result: ' + seen.size + ' instead of ' + expected);
}