PHP SDK: Concurrency, logs, and progress

Process independent requests in small batches. Concurrency still respects a site's requests-per-second limit and your plan limits. If a source allows one request per second, a larger batch must not become dozens of simultaneous requests.

Example: process a list of product pages

Pass a list of URLs in product_urls, with at most eight in this example. Each page should have .sku and h1. Split a large catalog into batches.

<?php
function run(\Sdk\Context $ctx): void
{
    $urls = $ctx->param('product_urls', []);
    if (!is_array($urls) || count($urls) === 0 || count($urls) > 8) {
        $ctx->abort('Pass 1 to 8 addresses in product_urls');
    }
    $total = count($urls);
    $ctx->progress(0, $total, 'Loading product pages');
    $responses = $ctx->http()->pool($urls);
    $done = 0;
    foreach ($responses as $response) {
        if (!$response->ok()) {
            $ctx->abort('Product page error: HTTP ' . $response->status);
        }
        $dom = $response->dom();
        $sku = $dom->text('.sku');
        if ($sku === null || $sku === '') {
            $ctx->abort('No SKU found on a product page');
        }
        $ctx->emit(['sku' => $sku, 'name' => $dom->text('h1')], $sku);
        $done++;
        $ctx->progress($done, $total, 'Processing product pages');
        $ctx->log('Product page processed', ['done' => $done, 'total' => $total]);
    }
    $ctx->flush();
    $ctx->log('All product pages sent for saving');
}

pool() takes 1–8 addresses and preserves response order. At most two HTTP requests run concurrently under the shared site limits. Browser actions have their own limits and do not automatically become concurrent.

Named progress stages

$ctx->progress($done, $total, $label) shows a count and stage name. Choose labels such as Finding categories, Reading stock, and Checking products. If the total is unknown, pass null. Do not guess a percentage.

You can pass an explicit percentage as the fourth argument: $ctx->progress(15, 30, 'Products', 65.0). This is useful when product pages account for only part of the algorithm. Assign sensible weights to stages so overall progress does not move backward.

100% of a stage means the work you declared is complete. The run may still be saving records and preparing the result. Use the run status to determine final completion.

What to log

Use $ctx->log('Message', ['field' => 'value'], 'info'). Use warning for skipped or unexpected items. Include the category, page number, product count, and reason for stopping. Do not log entire source responses, passwords, or thousands of identical lines.

Store state needed after resuming through $ctx->state()->get() and set(). Mark a page processed only after passing its records on. Use stable keys when reading it again. A local variable is not a saved cursor.

Runtime limits

Scripts have no direct file system, shell, or unrestricted network access. Use SDK methods. For files in external S3, use $ctx->storage('code'), where code is an allowed site connection. The editor suggests available functions and methods and checks them when you save.