Python SDK: HttpClient: website requests

Get a client with ctx.http(). Requests use the site settings, proxy, limits, and request log. Network calls are synchronous.

Examples are fragments inside def run(ctx):. All SDK methods are synchronous; do not add await.

Methods

• with_headers()
• without_cache()
• get()
• post()
• request()
• pool()
• pages()
• try_json()

with_headers

ctx.http().with_headers(headers) -> HttpClient

Result: HttpClient. An SDK object; use its methods instead of emitting it directly.

Return a copy of the client with additional HTTP headers. The original client is unchanged.

result = ctx.http().with_headers({'Accept': 'application/json'})

without_cache

ctx.http().without_cache() -> HttpClient

No arguments.

Result: HttpClient. An SDK object; use its methods instead of emitting it directly.

Return a client copy that fetches fresh responses without reading the cache.

result = ctx.http().without_cache()

get

ctx.http().get(url, auto_captcha=True) -> Response

Result: Response. An SDK object; use its methods instead of emitting it directly.

Load a page or JSON from a URL with GET.

result = ctx.http().get(ctx.param('catalog_url', 'https://example.com/catalog'))

post

ctx.http().post(url, body=None, auto_captcha=True) -> Response

Result: Response. An SDK object; use its methods instead of emitting it directly.

Send a POST request with a body. To send exact JSON, serialize it explicitly and set Content-Type.

import json
client = ctx.http().with_headers({'Content-Type': 'application/json'})
result = client.post(ctx.param('api_url'), json.dumps({'page': 1}))

request

ctx.http().request(method, url, body=None, auto_captcha=True) -> Response

Result: Response. An SDK object; use its methods instead of emitting it directly.

Send a request with an explicit HTTP method. get or post is usually enough.

result = ctx.http().request('GET', ctx.param('catalog_url'))

pool

ctx.http().pool(urls, auto_captcha=True) -> list[Response]

Result: list[Response].

Load a small batch of independent URLs in parallel.

result = ctx.http().pool([ctx.param('first_url'), ctx.param('second_url')])

Concurrency and response size. The regular profile accepts 1 to 8 URLs and runs at most two requests at a time. A self-hosted agent with the High throughput profile accepts up to 6000 URLs. Actual concurrency is bounded by the site's concurrency setting and the network service's shared capacity. The rps rate, burst, robots.txt and other site policies still apply.

Responses preserve input order. The total SDK response, including metadata, is limited to 8,000,000 bytes. Use smaller batches for large HTML pages and process batches in sequence. A network exception or an exceeded size limit fails the pool; bodies are never silently truncated. An HTTP status such as 404 or 500 is not a network exception by itself: inspect status and response.ok() on each response. 6000 HTTP requests do not mean 6000 browsers or scripts. See High HTTP throughput on your agent.

pages

ctx.http().pages(template, start, end, auto_captcha=True) -> Iterator[tuple[int, Response]]

Result: Iterator[tuple[int, Response]].

Visit a known range of pages in order. Use {page} in the template for the page number.

for number, response in ctx.http().pages(ctx.param('page_template'), 1, 3):
    ctx.log('Page loaded', {'page': number, 'status': response.status})

try_json

ctx.http().try_json(urls, auto_captcha=True) -> Any | None

Result: Any | None.

Try several JSON addresses in order and use the first suitable response.

result = ctx.http().try_json([ctx.param('first_url'), ctx.param('second_url')])

Errors and execution context

All calls are synchronous. Invalid arguments or malformed source data can raise ValueError/TypeError; a platform refusal or unavailable service raises RuntimeError. Missing elements/values return None or an empty collection only where explicitly documented. Do not convert every exception into an empty successful result. External-service methods are unavailable in export transformations; see transformation limits and environment.

Automatic CAPTCHA handling

auto_captcha is optional and defaults to True. Pass False to disable automatic solving. With automatic mode disabled, the request does not start paid CAPTCHA tasks. Platform and site settings, supported challenge types and quotas still apply. This argument is also available in get, post, request, pool, pages and try_json.

response = ctx.http().get(url, auto_captcha=False)

CAPTCHA on a self-hosted agent

Automatic mode and opt-out also work on your agent. A browser, connectivity to Akmelo and an eligible plan are required. No additional CAPTCHA keys need to be configured. Checks and costs belong to the agent's workspace; local execution does not make CAPTCHA handling free.

CAPTCHA usage reference

Complete CAPTCHA SDK and REST reference covers all hint fields, automatic defaults, explicit calls, session opt-out, result verification, agents, limits and failures. User guide and current prices explain setup and billing.

Object selection uses kind: "yandex_image"; automatic detection normally needs no hints. image_selector identifies only the task image, instructions_selector the selection order, and success_selector a result that appears after success. No target-domain list is needed. Unsupported variants fail instead of returning fabricated data. Do not repeatedly retry the same rejected task.