How to Set Up httpx for Async Parsing with Mobile Proxies: A Step-by-Step Guide
Introduction
In this step-by-step guide, you'll learn how to create a high-load asynchronous parser in Python using the httpx library and mobile proxies. We'll set up a connection pool, configure proper proxy rotation, manage load limits, retry mechanisms, error handling, and performance metrics. By the end, you'll have a functional project that can be adapted for any competitive public data gathering tasks, all within the rules of the law and the target website.
What you'll end up with: a working template for an httpx parser, mobile proxy configuration, safe limits, retries, logging, result export, and a set of techniques to enhance performance and resilience.
Who is this guide for: developers and analysts with basic Python knowledge. Beginners will find each detail explained in simple terms, while advanced users will discover performance tuning and complex rotation scenarios.
What you should know beforehand: basics of Python, understanding of asynchronicity (asyncio), and fundamental HTTP principles. If anything is unclear, the Basic Concepts section will help you quickly catch up.
How much time it will take: 2-4 hours for initial setup and testing, followed by 1-2 hours for fine-tuning to suit your case.
Below you will find the table of contents with internal links:
- Go to Environment Preparation
- Step 1: Creating Your Project and Environment
- Step 2: Connecting Mobile Proxy in httpx
- Step 3: Connection Pool and Rotation
- Step 4: Performance and Limits
- Step 5: Error Handling and Resilience
- Step 6: Logging, Metrics, and Test Run
- Result Verification
⚠️ Attention: Before starting, make sure to comply with the laws and the rules of the target resource. Review the robots.txt file and the website's user agreement, always request only allowed public data, and avoid creating excessive loads.
Preparation
Necessary tools and access:
- Python 3.10 or newer.
- Package manager pip and virtual environment venv.
- Code editor: VS Code, PyCharm, or any other preferred editor.
- An account with a mobile proxy provider. In this guide, we mention mobileproxy.space as a popular example of a provider that supports rotation, sticky sessions, and detailed logging. You can choose any similar service.
- A defined list of target pages and load limits (RPS, delays), along with contact with the resource owner as necessary.
System requirements:
- Operating System: Linux, macOS, or Windows. For high-load tasks, Linux is preferable.
- CPU: 2-4 cores or higher.
- RAM: at least 4 GB; for larger tasks, 8-16 GB.
- Stable internet connection. Delays to the proxy affect overall speed.
What to download and install:
- Install Python 3.10+ from the official Python supplier's website for your OS.
- Check that pip is available by running the command in the terminal: pip --version.
- Prepare the project directory. Create a folder, for example, mp_async_scraper.
- Inside, create a virtual environment: python -m venv .venv.
- Activate the environment. On Windows: .venv\Scripts\activate. On Linux/macOS: source .venv/bin/activate.
- Install dependencies: httpx[http2], httpcore, anyio, loguru, pyyaml, orjson. Command: pip install 'httpx[http2]' httpcore anyio loguru pyyaml orjson.
- Optional for Linux: pip install uvloop for speeding up the event loop (use only under asyncio in a UNIX environment).
Creating backups:
- Keep configs and URL list files separately. Make copies before editing them.
- Save logs in a separate logs folder and archive them so that you don't lose information during debugging.
Tip: Get into the habit of recording the version of dependencies in a requirements.txt file. This will help reproduce the environment on servers or for colleagues.
Basic Concepts
Key terms in simple language:
- httpx — a modern HTTP library for Python with support for asynchronous mode, HTTP2, connection pools, and fine configuration of timeouts and proxies.
- Asynchronous parsing — a way to run multiple network requests in parallel within a single process using non-blocking input/output operations. This increases throughput and reduces delays.
- Mobile proxy — a proxy server that routes traffic through mobile networks. Such traffic appears to websites as traffic from regular mobile devices. Providers like mobileproxy.space offer IP rotation, sticky sessions, and detailed statistics.
- Connection pool — a mechanism for reusing TCP connections. This reduces the overhead of establishing connections and improves performance.
- Proxy rotation — changing the outgoing IP or proxy itself to distribute load and maintain proper operation under request frequency limits from resources.
- Limits — upper limits for request frequency or simultaneous connections. These can be set on both the client side and the side of the proxy provider or the target resource.
- Retries — attempts to resend requests after failures, featuring configurable delay and stop strategies.
Key principles of operation:
- Asynchronicity yields benefits when performing a large number of network operations. The more parallel independent requests, the more noticeable the profit.
- Proxies should be used carefully: do not exceed limits, evenly distribute load, and respect the site's rules.
- Long-lived connections (keep-alive) and HTTP2 reduce overhead and increase throughput.
What is important to understand before starting:
- High load comes with responsibility. Monitor failure rates, load levels, and response times.
- Some websites dynamically change their rules and limits. Build flexibility into the configuration.
⚠️ Attention: It is not permissible to exploit vulnerabilities, bypass technical access restrictions, or violate the terms of use of websites. This guide teaches safe and correct client and proxy settings.
Step 1: Create Your Project and Environment
Goal of This Step
Prepare the project structure, virtual environment, dependencies, and basic configuration and code files.
Detailed Instructions
- Create a project folder named mp_async_scraper.
- Open a terminal in the project root.
- Create a virtual environment: python -m venv .venv.
- Activate the environment. Windows: .venv\Scripts\activate. Linux/macOS: source .venv/bin/activate.
- Install packages: pip install 'httpx[http2]' httpcore anyio loguru pyyaml orjson.
- Create a file named config.yaml. Insert the example below and edit as necessary.
- Create a file named proxies.txt and add one line for each proxy in the format user:pass@host:port. This data will be provided by your provider, e.g., mobileproxy.space.
- Create a file named scraper.py and insert the basic asynchronous client code.
Example config.yaml
target_urls: ['https://your-target.example/path1', 'https://your-target.example/path2'] concurrency: 50 per_host_semaphore: 10 request_timeout: 15.0 connect_timeout: 10.0 read_timeout: 15.0 write_timeout: 10.0 pool_max_connections: 200 pool_max_keepalive: 50 retry_attempts: 4 retry_backoff_base: 0.5 retry_backoff_cap: 8.0 jitter: 0.3 respect_retry_after: true http2: true proxy_rotation_after_requests: 50 proxy_rotation_after_errors: 3 headers: { 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chrome/123.0 Safari/537.36', 'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' } output_file: 'data.jsonl' log_level: 'INFO'Example proxies.txt
mp_user1:mp_pass1@mp_host1:mp_port1 mp_user2:mp_pass2@mp_host2:mp_port2Basic skeleton scraper.py
import asyncio, httpx, yaml, orjson from loguru import logger from pathlib import Path def load_config(path: str) -> dict: return yaml.safe_load(Path(path).read_text(encoding='utf-8')) async def fetch(client: httpx.AsyncClient, url: str) -> dict: r = await client.get(url) r.raise_for_status() return {'url': url, 'status': r.status_code, 'length': len(r.content)} async def main(): cfg = load_config('config.yaml') limits = httpx.Limits(max_connections=cfg['pool_max_connections'], max_keepalive_connections=cfg['pool_max_keepalive']) timeout = httpx.Timeout(connect=cfg['connect_timeout'], read=cfg['read_timeout'], write=cfg['write_timeout'], pool=cfg['request_timeout']) async with httpx.AsyncClient(limits=limits, timeout=timeout, http2=cfg.get('http2', True), headers=cfg.get('headers', {}), trust_env=False) as client: tasks = [fetch(client, u) for u in cfg['target_urls']] results = await asyncio.gather(*tasks, return_exceptions=True) for it in results: logger.info(str(it)) if __name__ == '__main__': asyncio.run(main())Important Points
- trust_env=false ensures that system proxies do not intercept traffic.
- http2=true often speeds up multiple requests to the same host.
Tip: Keep the config in git and use a separate file for local parameters, such as config.local.yaml, which should not be committed.
Expected Result
You should have a ready project structure with files and the ability to perform a basic test request without a proxy.
Possible Problems and Solutions
- httpx import error — check the activation of the virtual environment and the version of Python.
- Write permissions — ensure you have permission for the project folder.
✅ Check: Run python scraper.py. In the logs, you will see the statuses of requests to the target URLs from config.yaml.
Step 2: Connecting Mobile Proxy in httpx (with code)
Goal of This Step
Set up request routing through mobile proxies and ensure that the client is correctly authenticated with the provider.
Detailed Instructions
- Open the proxies.txt file and ensure that each line is in the format user:pass@host:port. This data is provided in the provider's personal account, such as mobileproxy.space.
- Create a module named proxy_pool.py for managing proxies and providing the current proxy for each request.
- Enhance scraper.py so that each request goes through the specified proxy. For simplicity, start with one proxy.
- Verify that proxy authentication is successful by checking the logs in the provider's personal account or through the provider's diagnostics.
Code: Simple Proxy Client
# file: proxy_pool.py from pathlib import Path class ProxyPool: def __init__(self, path: str): raw = Path(path).read_text(encoding='utf-8').strip().splitlines() self.proxies = [line.strip() for line in raw if line.strip() and not line.strip().startswith('#')] self.index = 0 def next_proxy(self) -> str: if not self.proxies: raise RuntimeError('No proxies loaded') p = self.proxies[self.index % len(self.proxies)] self.index += 1 return f'http://{p}' def as_httpx_proxies(self, proxy_url: str) -> dict: return {'http://': proxy_url, 'https://': proxy_url}# patch scraper.py to use proxy import asyncio, httpx, yaml, orjson from loguru import logger from pathlib import Path from proxy_pool import ProxyPool def load_config(path: str) -> dict: return yaml.safe_load(Path(path).read_text(encoding='utf-8')) async def fetch(client: httpx.AsyncClient, url: str) -> dict: r = await client.get(url) r.raise_for_status() return {'url': url, 'status': r.status_code, 'length': len(r.content)} async def main(): cfg = load_config('config.yaml') pool = ProxyPool('proxies.txt') proxy_url = pool.next_proxy() limits = httpx.Limits(max_connections=cfg['pool_max_connections'], max_keepalive_connections=cfg['pool_max_keepalive']) timeout = httpx.Timeout(connect=cfg['connect_timeout'], read=cfg['read_timeout'], write=cfg['write_timeout'], pool=cfg['request_timeout']) proxies = pool.as_httpx_proxies(proxy_url) async with httpx.AsyncClient(limits=limits, timeout=timeout, http2=cfg.get('http2', True), headers=cfg.get('headers', {}), trust_env=False, proxies=proxies, verify=True) as client: tasks = [fetch(client, u) for u in cfg['target_urls']] results = await asyncio.gather(*tasks, return_exceptions=True) for it in results: logger.info(str(it)) if __name__ == '__main__': asyncio.run(main())Important Points
- httpx accepts proxies in the http or https format. Here, we use http. If the provider specifies support only for http proxies, this is sufficient for the client's https requests.
- Make sure that the username and password contain only valid characters. If there are special characters, they should be URL-encoded.
Tip: If the provider, such as mobileproxy.space, supports sticky session, tie one proxy to a series of requests to one domain for stability and context retention.
Expected Result
Requests go through the mobile proxy, which can be seen in the provider's personal account statistics or by indirect signs (change of external IP in the provider's diagnostics).
Possible Problems and Solutions
- Authentication fails — check the correctness of user:pass and the host address with the port, as well as any restrictions in the personal account.
- SSL errors — keep verify=true and check system certificates. Install certifi if necessary.
✅ Check: Restart python scraper.py and check the proxy provider's logs, such as mobileproxy.space, to verify that traffic is visible from your client.
Step 3: Connection Pool and Rotation
Goal of This Step
Configure a resilient httpx connection pool, implement proxy rotation according to the rules, and support sticky sessions and safe switching after N requests and errors.
Detailed Instructions
- Add counters for each proxy usage and rotation logic in proxy_pool.py; use the proxy_rotation_after_requests and proxy_rotation_after_errors parameters from the config.
- Implement acquire method that returns the current proxy and increases the usage count, and report method that logs errors and successes.
- In scraper.py, create multiple clients or a single client with dynamic proxy switching for each request. In httpx, dynamic switching is easier through creating a proxy client or through request-level transport; we will go with proxy clients combined with pooling.
- Limit the simultaneous number of requests to one proxy to avoid load spikes.
Code: Extended Proxy Pool
# file: proxy_pool.py import time, itertools from pathlib import Path class ProxyEntry: def __init__(self, raw: str): self.raw = raw self.proxy_url = f'http://{raw}' self.ok = 0 self.fail = 0 self.usages = 0 self.last_rotate = time.monotonic() class ProxyPool: def __init__(self, path: str, rotate_after_requests: int = 50, rotate_after_errors: int = 3): lines = [l.strip() for l in Path(path).read_text(encoding='utf-8').splitlines() if l.strip() and not l.strip().startswith('#')] self.entries = [ProxyEntry(l) for l in lines] if not self.entries: raise RuntimeError('No proxies loaded') self.rotate_after_requests = rotate_after_requests self.rotate_after_errors = rotate_after_errors self._cycle = itertools.cycle(self.entries) def acquire(self) -> ProxyEntry: e = next(self._cycle) e.usages += 1 return e def should_rotate(self, e: ProxyEntry) -> bool: if e.usages >= self.rotate_after_requests: return True if e.fail >= self.rotate_after_errors: return True return False def report_success(self, e: ProxyEntry): e.ok += 1 def report_failure(self, e: ProxyEntry): e.fail += 1 def reset_entry(self, e: ProxyEntry): e.usages = 0 e.fail = 0 e.last_rotate = time.monotonic()Code: Proxy Clients and Parallelism Limiting
# patch scraper.py import asyncio, httpx, yaml, orjson from loguru import logger from pathlib import Path from proxy_pool import ProxyPool, ProxyEntry def load_config(path: str) -> dict: return yaml.safe_load(Path(path).read_text(encoding='utf-8')) async def fetch_with_proxy(client: httpx.AsyncClient, pool: ProxyPool, entry: ProxyEntry, url: str) -> dict: try: r = await client.get(url) r.raise_for_status() pool.report_success(entry) return {'url': url, 'status': r.status_code, 'length': len(r.content)} except Exception as ex: pool.report_failure(entry) raise ex async def worker(url_q: asyncio.Queue, pool: ProxyPool, client_factory, per_host_sem: asyncio.Semaphore, results: list): while True: try: url = url_q.get_nowait() except asyncio.QueueEmpty: break entry = pool.acquire() if pool.should_rotate(entry): pool.reset_entry(entry) entry = pool.acquire() proxies = {'http://': entry.proxy_url, 'https://': entry.proxy_url} async with client_factory(proxies) as client: async with per_host_sem: try: res = await fetch_with_proxy(client, pool, entry, url) results.append(res) except Exception as e: results.append({'url': url, 'error': str(e)}) async def main(): cfg = load_config('config.yaml') pool = ProxyPool('proxies.txt', cfg['proxy_rotation_after_requests'], cfg['proxy_rotation_after_errors']) limits = httpx.Limits(max_connections=cfg['pool_max_connections'], max_keepalive_connections=cfg['pool_max_keepalive']) timeout = httpx.Timeout(connect=cfg['connect_timeout'], read=cfg['read_timeout'], write=cfg['write_timeout'], pool=cfg['request_timeout']) def client_factory(proxies: dict): return httpx.AsyncClient(limits=limits, timeout=timeout, http2=cfg.get('http2', True), headers=cfg.get('headers', {}), trust_env=False, proxies=proxies, verify=True) url_q = asyncio.Queue() for u in cfg['target_urls']: url_q.put_nowait(u) per_host_sem = asyncio.Semaphore(cfg['per_host_semaphore']) results = [] workers = [asyncio.create_task(worker(url_q, pool, client_factory, per_host_sem, results)) for _ in range(cfg['concurrency'])] await asyncio.gather(*workers) for it in results: logger.info(str(it)) if __name__ == '__main__': asyncio.run(main())Important Points
- Creating clients for each proxy simplifies substitution and isolation of the connection pool by proxy.
- per_host_semaphore controls simultaneous requests, reducing spikes and overload errors.
Tip: Keep statistics for each proxy and periodically dump it into logging to see which proxies frequently return errors.
Expected Result
Stable requests with even proxy rotation in sequence, reducing errors during peaks.
Possible Problems and Solutions
- Excessive overhead in client creation — you can cache clients per proxy and reuse them. We will explore this in an extended example below.
- Insufficiently even rotation — increase rotate_after_requests for longer sessions.
✅ Check: Run the parser with 10-20 URLs and several proxies. In the logs, you should see requests being spread across different proxies. Errors should not be piling up.
Step 4: Performance and Limits
Goal of This Step
Refine the architecture for high load: use a cache of proxy clients, add a global rate limit and per-channel rate limiter, configure timeouts, and implement exponential backoff retries.
Detailed Instructions
- Transition to a model of persistent clients: one httpx.AsyncClient for each proxy, reused by multiple workers.
- Add a global concurrency semaphore and a per_host_semaphore, as well as a simple rate limiter through asyncio.Semaphore + sleep.
- Implement exponential backoff with jitter for retrying temporary failures and 429 responses.
- Carefully configure timeouts: keep connect lower, read slightly higher, and balance the pool.
Code: Proxy Clients with Cache and Retries
# file: client_pool.py import asyncio, random, httpx class ClientPool: def __init__(self, httpx_limits: httpx.Limits, httpx_timeout: httpx.Timeout, headers: dict, http2: bool, verify: bool = True): self._limits = httpx_limits self._timeout = httpx_timeout self._headers = headers self._http2 = http2 self._verify = verify self._clients = {} async def get(self, proxy_url: str) -> httpx.AsyncClient: c = self._clients.get(proxy_url) if c is None: proxies = {'http://': proxy_url, 'https://': proxy_url} c = httpx.AsyncClient(limits=self._limits, timeout=self._timeout, http2=self._http2, headers=self._headers, trust_env=False, proxies=proxies, verify=self._verify) self._clients[proxy_url] = c return c async def aclose(self): await asyncio.gather(*(c.aclose() for c in self._clients.values())) async def retry_request(self, client: httpx.AsyncClient, method: str, url: str, attempts: int, base: float, cap: float, jitter: float, respect_retry_after: bool): last = None for i in range(attempts): try: r = await client.request(method, url) if r.status_code in (429, 503): ra = r.headers.get('retry-after') if respect_retry_after and ra: try: delay = float(ra) except: delay = min(cap, base * (2 ** i)) delay += random.random() * jitter await asyncio.sleep(delay) continue r.raise_for_status() return r except (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteError, httpx.RemoteProtocolError, httpx.PoolTimeout) as e: last = e delay = min(cap, base * (2 ** i)) delay += random.random() * jitter await asyncio.sleep(delay) except httpx.HTTPStatusError as e: last = e if 500 <= e.response.status_code < 600: delay = min(cap, base * (2 ** i)); delay += random.random() * jitter; await asyncio.sleep(delay); continue break raise last if last else RuntimeError('Unknown error')# patch scraper.py to use client cache import asyncio, httpx, yaml, orjson, random from loguru import logger from pathlib import Path from proxy_pool import ProxyPool, ProxyEntry from client_pool import ClientPool def load_config(path: str) -> dict: return yaml.safe_load(Path(path).read_text(encoding='utf-8')) async def fetch(pool: ProxyPool, client_pool: ClientPool, url: str, attempts: int, base: float, cap: float, jitter: float, respect_retry_after: bool, per_host_sem: asyncio.Semaphore, global_sem: asyncio.Semaphore, results: list): entry = pool.acquire() if pool.should_rotate(entry): pool.reset_entry(entry); entry = pool.acquire() client = await client_pool.get(entry.proxy_url) async with global_sem: async with per_host_sem: try: r = await client_pool.retry_request(client, 'GET', url, attempts, base, cap, jitter, respect_retry_after) pool.report_success(entry) results.append({'url': url, 'status': r.status_code, 'length': len(r.content)}) except Exception as e: pool.report_failure(entry) results.append({'url': url, 'error': str(e)}) async def main(): cfg = load_config('config.yaml') pool = ProxyPool('proxies.txt', cfg['proxy_rotation_after_requests'], cfg['proxy_rotation_after_errors']) limits = httpx.Limits(max_connections=cfg['pool_max_connections'], max_keepalive_connections=cfg['pool_max_keepalive']) timeout = httpx.Timeout(connect=cfg['connect_timeout'], read=cfg['read_timeout'], write=cfg['write_timeout'], pool=cfg['request_timeout']) client_pool = ClientPool(limits, timeout, cfg.get('headers', {}), cfg.get('http2', True), verify=True) global_sem = asyncio.Semaphore(cfg['concurrency']) per_host_sem = asyncio.Semaphore(cfg['per_host_semaphore']) urls = cfg['target_urls'] results = [] tasks = [asyncio.create_task(fetch(pool, client_pool, u, cfg['retry_attempts'], cfg['retry_backoff_base'], cfg['retry_backoff_cap'], cfg['jitter'], cfg.get('respect_retry_after', True), per_host_sem, global_sem, results)) for u in urls] await asyncio.gather(*tasks) await client_pool.aclose() for it in results: logger.info(str(it)) if __name__ == '__main__': asyncio.run(main())Important Points
- One client per proxy eliminates unnecessary handshakes and speeds up operations.
- Exponential backoff with jitter softens spikes and helps withstand brief network issues or a 429 response.
Tip: Start with lower limits on concurrency and increase gradually while monitoring failure metrics and average delays.
Expected Result
Stable operation under load, predictable delays, and a controlled error level. Performance increases thanks to client reuse and proper limits.
Possible Problems and Solutions
- High latency during DNS or TLS — keep clients alive, use HTTP2, and monitor max_keepalive_connections.
- Provider restrictions — decrease concurrency or add more proxies.
✅ Check: Conduct a test on 100-200 URLs. Assess average latency and retry rates. Error spikes should be smoothed.
Step 5: Error Handling and Resilience
Goal of This Step
Add detailed error classification, collect failures for reprocessing, support Retry-After, and implement basic protection against degradation of a single proxy by breaking the chain.
Detailed Instructions
- In client_pool.retry_request, handling for 429 and 5xx responses with Retry-After and backoff is already in place. Check that attempts and cap are adequate.
- Add failure logging to a file for re-processing.
- Implement a primitive circuit breaker: after a high number of errors on a proxy, pause before reusing it.
- Add error status and codes to logs for easier diagnostics.
Code: Fail File and Circuit Breaker
# patch proxy_pool.py import time, itertools from pathlib import Path class ProxyEntry: def __init__(self, raw: str): self.raw = raw self.proxy_url = f'http://{raw}' self.ok = 0 self.fail = 0 self.usages = 0 self.last_rotate = time.monotonic() self.suspended_until = 0.0 class ProxyPool: def __init__(self, path: str, rotate_after_requests: int = 50, rotate_after_errors: int = 3, suspend_seconds: float = 15.0): lines = [l.strip() for l in Path(path).read_text(encoding='utf-8').splitlines() if l.strip() and not l.strip().startswith('#')] self.entries = [ProxyEntry(l) for l in lines] if not self.entries: raise RuntimeError('No proxies loaded') self.rotate_after_requests = rotate_after_requests self.rotate_after_errors = rotate_after_errors self.suspend_seconds = suspend_seconds self._it = itertools.cycle(self.entries) def acquire(self) -> ProxyEntry: for _ in range(len(self.entries)): e = next(self._it) now = time.monotonic() if now < e.suspended_until: continue e.usages += 1 return e raise RuntimeError('All proxies are suspended') def should_rotate(self, e: ProxyEntry) -> bool: if e.usages >= self.rotate_after_requests: return True if e.fail >= self.rotate_after_errors: return True return False def report_success(self, e: ProxyEntry): e.ok += 1 e.fail = max(0, e.fail - 1) def report_failure(self, e: ProxyEntry): e.fail += 1 if e.fail >= self.rotate_after_errors: e.suspended_until = time.monotonic() + self.suspend_seconds def reset_entry(self, e: ProxyEntry): e.usages = 0 e.fail = 0 e.last_rotate = time.monotonic() e.suspended_until = 0.0# patch scraper.py to write fails import asyncio, httpx, yaml, orjson, random from loguru import logger from pathlib import Path from proxy_pool import ProxyPool, ProxyEntry from client_pool import ClientPool def write_fails(path: str, items: list): if not items: return with open(path, 'a', encoding='utf-8') as f: for it in items: if 'error' in it: f.write(orjson.dumps(it).decode('utf-8') + '\n') async def fetch(pool: ProxyPool, client_pool: ClientPool, url: str, attempts: int, base: float, cap: float, jitter: float, respect_retry_after: bool, per_host_sem: asyncio.Semaphore, global_sem: asyncio.Semaphore, results: list): try: entry = pool.acquire() except RuntimeError as e: results.append({'url': url, 'error': 'no_proxy_available'}) return if pool.should_rotate(entry): pool.reset_entry(entry); entry = pool.acquire() client = await client_pool.get(entry.proxy_url) async with global_sem: async with per_host_sem: try: r = await client_pool.retry_request(client, 'GET', url, attempts, base, cap, jitter, respect_retry_after) pool.report_success(entry) results.append({'url': url, 'status': r.status_code, 'length': len(r.content)}) except Exception as e: pool.report_failure(entry) results.append({'url': url, 'error': str(e), 'proxy': entry.proxy_url}) async def main(): cfg = yaml.safe_load(Path('config.yaml').read_text(encoding='utf-8')) pool = ProxyPool('proxies.txt', cfg['proxy_rotation_after_requests'], cfg['proxy_rotation_after_errors']) limits = httpx.Limits(max_connections=cfg['pool_max_connections'], max_keepalive_connections=cfg['pool_max_keepalive']) timeout = httpx.Timeout(connect=cfg['connect_timeout'], read=cfg['read_timeout'], write=cfg['write_timeout'], pool=cfg['request_timeout']) client_pool = ClientPool(limits, timeout, cfg.get('headers', {}), cfg.get('http2', True), verify=True) global_sem = asyncio.Semaphore(cfg['concurrency']) per_host_sem = asyncio.Semaphore(cfg['per_host_semaphore']) urls = cfg['target_urls'] results = [] tasks = [asyncio.create_task(fetch(pool, client_pool, u, cfg['retry_attempts'], cfg['retry_backoff_base'], cfg['retry_backoff_cap'], cfg['jitter'], cfg.get('respect_retry_after', True), per_host_sem, global_sem, results)) for u in urls] await asyncio.gather(*tasks) await client_pool.aclose() fails = [it for it in results if 'error' in it] write_fails('fails.jsonl', fails) for it in results: logger.info(str(it)) if __name__ == '__main__': asyncio.run(main())Important Points
- Suspending a problematic proxy for a short time protects against a flood of retries to a non-working endpoint.
- The fails.jsonl file enables selective retries for problematic URLs later.
Tip: Log unique error causes separately to understand what issues occur most frequently: timeouts, limit breaches, or content errors.
Expected Result
The system maintains resilience even during partial proxy failures, avoiding total collapse, and correctly accumulates tasks for retrying.
Possible Problems and Solutions
- All proxies go into suspended — decrease concurrency or add more proxies, or increase pauses.
- Frequent 429 errors — slow down, enable respect Retry-After, distribute requests more evenly.
✅ Check: Simulate errors by increasing concurrency. Ensure there isn't a constant storm of errors: proxies should be temporarily suspended, redistributing the load.
Step 6: Logging, Metrics, and Test Run
Goal of This Step
Add comprehensive logging and metrics, conduct a test run, and verify the correctness of results.
Detailed Instructions
- Configure loguru to write to a file and rotate logs.
- Add simple metrics: counters for successful and unsuccessful requests, and average delays.
- Conduct a test run on a limited list of URLs and record the results.
Code: Logging and Metrics
# patch scraper.py for logging and metrics import asyncio, httpx, yaml, orjson, time from loguru import logger from pathlib import Path from proxy_pool import ProxyPool from client_pool import ClientPool def load_config(path: str) -> dict: return yaml.safe_load(Path(path).read_text(encoding='utf-8')) async def run_batch(urls: list, cfg: dict): pool = ProxyPool('proxies.txt', cfg['proxy_rotation_after_requests'], cfg['proxy_rotation_after_errors']) limits = httpx.Limits(max_connections=cfg['pool_max_connections'], max_keepalive_connections=cfg['pool_max_keepalive']) timeout = httpx.Timeout(connect=cfg['connect_timeout'], read=cfg['read_timeout'], write=cfg['write_timeout'], pool=cfg['request_timeout']) client_pool = ClientPool(limits, timeout, cfg.get('headers', {}), cfg.get('http2', True), verify=True) global_sem = asyncio.Semaphore(cfg['concurrency']) per_host_sem = asyncio.Semaphore(cfg['per_host_semaphore']) attempts = cfg['retry_attempts']; base = cfg['retry_backoff_base']; cap = cfg['retry_backoff_cap']; jitter = cfg['jitter']; respect_retry_after = cfg.get('respect_retry_after', True) results, timings = [], [] async def one(url: str): t0 = time.perf_counter() entry = None try: entry = pool.acquire() if pool.should_rotate(entry): pool.reset_entry(entry); entry = pool.acquire() client = await client_pool.get(entry.proxy_url) async with global_sem: async with per_host_sem: r = await client_pool.retry_request(client, 'GET', url, attempts, base, cap, jitter, respect_retry_after) pool.report_success(entry) dt = time.perf_counter() - t0 timings.append(dt) results.append({'url': url, 'status': r.status_code, 'length': len(r.content), 'dt': dt}) except Exception as e: if entry: pool.report_failure(entry) dt = time.perf_counter() - t0 timings.append(dt) results.append({'url': url, 'error': str(e), 'dt': dt}) tasks = [asyncio.create_task(one(u)) for u in urls] await asyncio.gather(*tasks) await client_pool.aclose() return results, timings def summarize(results: list, timings: list): ok = sum(1 for r in results if 'status' in r) fail = sum(1 for r in results if 'error' in r) avg = sum(timings) / len(timings) if timings else 0.0 p_ok = (ok / len(results) * 100.0) if results else 0.0 return {'ok': ok, 'fail': fail, 'avg_sec': round(avg, 3), 'success_pct': round(p_ok, 1)} async def main(): cfg = load_config('config.yaml') logger.remove(); logger.add('logs/run.log', level=cfg.get('log_level', 'INFO'), rotation='10 MB', retention='7 days') logger.info('Start test run') results, timings = await run_batch(cfg['target_urls'], cfg) summary = summarize(results, timings) Path(cfg['output_file']).write_text('\n'.join(orjson.dumps(r).decode('utf-8') for r in results), encoding='utf-8') logger.info(f'Summary: {summary}') if __name__ == '__main__': asyncio.run(main())Important Points
- Log rotation saves disk space.
- JSONL format is convenient for subsequent processing and failure analysis.
Tip: Keep a summary of metrics for the run separately for comparing configuration experiments later.
Tip: Enable a separate log for errors. This will simplify debugging and throttling.
Expected Result
You will have a data file data.jsonl, logs with details of the run, and a statistical summary including success rate and average delay.
Possible Problems and Solutions
- Log file grows quickly — decrease the logging level or increase the rotation threshold.
- The average response time is high — check the limits, add proxies, optimize headers, and timeouts.
✅ Check: Open logs/run.log and confirm statuses, errors, and the final summary are recorded. Ensure data.jsonl contains objects line by line.
Result Verification
Checklist
- Project contains config.yaml, proxies.txt, scraper.py, proxy_pool.py, client_pool.py.
- The parser runs and processes the URL list.
- Requests go through mobile proxies; this is visible in the provider’s personal account, such as mobileproxy.space, based on connection statistics.
- Logs are generated, metrics are compiled, and results are written to data.jsonl.
- In case of failures, retries and suspensions of problematic proxies are triggered.
How to Test
- Run a small set of URLs: 20-50 addresses.
- Check the success rate and average delay.
- Gradually increase concurrency and measure the gains.
Success Metrics
- Success rate of 95% or higher under a stable network.
- Average delay decreases with HTTP2 enabled and correct pool configuration.
- Errors are evenly distributed; there is no single problematic proxy that breaks the whole run.
Common Issues and Solutions
- Problem: constant timeouts. Cause: too high concurrency or weak network. Solution: decrease concurrency, check proxy response times, increase read_timeout.
- Problem: 429 Too Many Requests. Cause: exceeding limits set by the target site. Solution: slow down, enable respect for Retry-After, increase the number of proxies.
- Problem: SSL error. Cause: lack of root certificates or MITM antivirus software. Solution: install certifi, check trusted certificates, disable SSL interception in antivirus software.
- Problem: proxy authentication fails. Cause: incorrect credentials. Solution: double-check user:pass, URL-encode any special characters.
- Problem: clients are created too frequently. Cause: lack of client caching. Solution: use ClientPool, reuse clients per proxy.
- Problem: all traffic goes through a single proxy. Cause: no rotation. Solution: enable ProxyPool with cyclic issuance and thresholds for rotate_after_requests and rotate_after_errors.
- Problem: large volume of logs. Cause: logging level is too detailed. Solution: lower the level to INFO, enable rotation, and set retention.
Additional Capabilities
- Automatic IP updates via the provider's API. Many, including mobileproxy.space, allow you to change the IP on request. Insert a method in ProxyPool that calls for IP change on a schedule. Monitor the limits for such operations.
- Sticky sessions by domains. Tie one proxy to one domain to lose cookies and context less often.
- Separate header profiles. Change User-Agent based on the domain and content type to meet the characteristics of real devices.
- DNS preloading and independent resolvers. Reduce delays through DNS caching and stable resolution.
- Retry queue. You can separate fails.jsonl into a dedicated worker that processes failures at reduced limits.
- Parallel processes. Scale by processes or machines, distributing the URL lists into batches and collecting results centrally.
Tip: As load grows to tens of thousands of requests per minute, switch to a separate queuing service and coordinator, while running parser in a cluster, controlling limits.
Tip: Do not store proxy secrets in plain text in the repository. Use environment variables or secret storage.
FAQ
- How many mobile proxies do I need? Start with 3-5 for a small volume and increase the number while monitoring for 429 and average delay. Load should be evenly distributed.
- Are sticky sessions necessary? Yes, if you're gathering a lot of data from one domain and it’s important to maintain cookies and context. This increases stability.
- What timeouts should I choose? connect 5-10 seconds, read 10-20, write 10-15, pool 10-20. Settings depend on the network and response from the target service.
- Is there benefit to HTTP2? Yes, especially with multiple requests to one host. It results in lower overhead and better compression.
- How does rotation affect performance? Frequent rotation creates additional instability. Maintain balance: more requests to one proxy if connection quality and limits are good.
- How to store results? JSONL is convenient for streaming. You can also write to a database with buffering and transaction confirmation.
- How to safely increase load? Gradually increase concurrency by 10-20%, measuring success rate and average delay each time.
- Can I use one proxy for everything? Not recommended. Distributing the load across multiple proxies reduces overload risks and increases resilience.
- What to do with failures? Re-run fails.jsonl at lower limits, analyze the reasons, improve backoff and rotation.
- How to check that traffic is going through the proxy? Check statistics in the provider's personal account, such as mobileproxy.space, where you can see connections and bytes of traffic.
Conclusion
We configured an asynchronous httpx client with mobile proxies, added a connection pool, cached clients to proxies, rotation, limits, and resilient retries. You now have a template ready to scale for real tasks. Next, optimize the configuration, balance the number of proxies and limits, enhance rotation logic and metrics. If necessary, leverage the additional capabilities of providers like mobileproxy.space, including sticky sessions and IP change via API. Remember to always follow the rules of target websites and maintain responsible load handling.
⚠️ Attention: Always verify compliance with legal requirements and the rules of target resources. Do not request closed data and do not exceed permissible load limits.
Tip: Create a separate document with operational regulations: target metrics, hourly limits, rotation rules, and contacts for responsible parties. This will simplify maintenance and scaling.