Documentation

Cloudflare page: check the returned content (PHP)

Check access to the expected content on the ScrapingCourse demonstration page. The result is one verification record rather than a product catalog.

Choose PHP, paste the complete source from this article and set the run parameters below. The source for this collection is ScrapingCourse. This walkthrough applies to the specified demonstration page.

Example settings

{
  "path": "/cloudflare-challenge",
  "mode": "challenge",
  "expected": 1
}

path selects the page, mode selects an algorithm branch, and expected sets the control count for this example. It is not a plan limit. When the source dataset changes, check the expected count against it.

How the algorithm works

Select /cloudflare-challenge and the challenge mode, which is separate from card collection.
Open the page with the browser SDK and the current auto_captcha value. The scenario checks the content returned by the page.
Read the body text with DOM tools and normalize repeated whitespace and line breaks to one space.
Match the demonstration page success text using a case-insensitive expression with bypassed variants or Congratulations.
Only after a match, emit name containing the page path and passed: true. Use that path as the record key.
Close the browser in finally. Product-card extraction and product-count comparison do not run in challenge mode.

Details that matter

Success depends on the control text of this particular page. expected: 1 describes one output record; this branch does not count products.

PHP details

PHP calls network and browser methods sequentially with ->. courseCollect receives $seen and returns its updated array, so assign each result back. Close the page in finally; $ctx->fail() stops the run with a readable message.

Expected output

This branch emits one record with name and passed. name is /cloudflare-challenge; passed is true only after matching the control text. The final seen count check is skipped for challenge mode.

Complete scenario source

The source is shared by this collection. This example runs its challenge branch using the parameters above. Copy the complete block into the PHP editor; no fixed sign-in credentials or service keys need to be added.

<?php

function run(\Sdk\Context $ctx): void
{
    $origin = 'https://www.scrapingcourse.com';
    $path = (string) $ctx->param('path', '/pagination');
    $mode = (string) $ctx->param('mode', 'http');
    $auto = $ctx->param('auto_captcha', true) !== false;
    $seen = [];
    if ($mode === 'http') {
        $url = $origin.$path;
        $visited = [];
        while ($url !== null && count($visited) < 50) {
            if (isset($visited[$url])) {
                $ctx->fail('Pagination returned an already processed page');
            }
            $visited[$url] = true;
            $response = $ctx->http()->withoutCache()->get($url, $auto);
            if (! $response->ok()) {
                $ctx->fail('The source could not be opened: '.$response->status);
            }
            $html = $response->body;
            $seen = courseCollect($ctx, $html, $seen);
            $next = $ctx->dom($html)->attr('a[rel="next"], a.next.page-numbers', 'href');
            $url = $next ? $ctx->url()->absolutize($url, $next) : null;
        }
        if ($url !== null) {
            $ctx->fail('The pagination safety limit was reached');
        }
    } else {
        $page = $ctx->browser()->open($origin.$path, ['*/ajax/products*'], [], $auto);
        try {
            if ($mode === 'login') {
                $dom = $ctx->dom($page->content());
                $hint = '';
                foreach ($dom->qa('p') as $p) {
                    if (str_contains($p->text(), 'Demo Credentials:')) {
                        $hint = $p->text();
                    }
                }
                preg_match('/Password:\s*([^\s|]+)/i', $hint, $password);
                preg_match('/Email:\s*([^\s|]+@[^\s|]+)/i', $hint, $emailMatch);
                $email = $emailMatch[1] ?? '';
                if ($email === '' || ! isset($password[1])) {
                    $ctx->fail('The page does not contain demo credentials');
                }
                $page->type('#email', $email)->type('#password', $password[1])->click('button[type="submit"]');
            }
            if ($mode === 'challenge') {
                $text = preg_replace('/\s+/', ' ', $ctx->dom($page->content())->text('body') ?? '');
                if (! preg_match('/You (?:successfully )?bypassed|successfully bypassed|Congratulations/i', $text)) {
                    $ctx->fail('The expected page content was not received');
                }
                $ctx->emit(['name' => $path, 'passed' => true], $path);
            } else {
                $page->waitFor('.product-name:not(:empty)', 15000);
                if ($mode === 'button') {
                    $complete = false;
                    for ($i = 0; $i < 30; $i++) {
                        $html = $page->content();
                        $before = $ctx->dom($html);
                        $seen = courseCollect($ctx, $html, $seen);
                        $button = $before->q('#load-more-btn');
                        if ($button === null || preg_match('/display:\s*none/', $button->attr('style') ?? '')) {
                            $complete = true;
                            break;
                        }
                        $offset = $button->attr('data-offset');
                        $count = count($before->qa('.product-item'));
                        $page->click('#load-more-btn');
                        $response = $page->waitForResponse('*offset='.$offset, 15000);
                        if ($response === null || ($response['status'] < 200 || $response['status'] >= 300) || $response['truncated'] || $response['error']) {
                            $ctx->fail('The next batch could not be loaded');
                        }
                        if (trim($response['body'] ?? '') === '') {
                            $complete = true;
                            break;
                        }
                        $page->waitFor('.product-item:nth-child('.($count + 1).')');
                    }
                    if (! $complete) {
                        $ctx->fail('The batch loading safety limit was reached');
                    }
                }
                if ($mode === 'scroll') {
                    $stable = 0;
                    $previous = 0;
                    for ($i = 0; $i < 40 && $stable < 3; $i++) {
                        $page->scroll(3, 400);
                        $html = $page->content();
                        $seen = courseCollect($ctx, $html, $seen);
                        $count = count($ctx->dom($html)->qa('.product-item'));
                        $stable = $count === $previous ? $stable + 1 : 0;
                        $previous = $count;
                    }
                    if ($stable < 3) {
                        $ctx->fail('The feed is still loading: increase the scroll limit');
                    }
                }
                $seen = courseCollect($ctx, $page->content(), $seen);
            }
        } finally {
            $page->close();
        }
    }
    $expected = (int) $ctx->param('expected', 1);
    if ($mode !== 'challenge' && count($seen) !== $expected) {
        $ctx->fail('Incomplete result: '.count($seen).' instead of '.$expected);
    }
}

function courseCollect(\Sdk\Context $ctx, string $html, array $seen): array
{
    $dom = $ctx->dom($html);
    foreach ($dom->qa('.product-item, li.product') as $card) {
        $name = trim($card->q('.product-name, .woocommerce-loop-product__title')?->text() ?? '');
        $key = $card->q('a')?->attr('href') ?? $name;
        if ($name === '' || isset($seen[$key])) {
            continue;
        }
        $seen[$key] = true;
        $ctx->emit(['name' => $name, 'price' => $ctx->text()->money($card->q('.product-price, .price')?->text() ?? ''),
            'image' => $card->q('img')?->attr('src') ?? ''], $key);
    }

    return $seen;
}