How to Set Up Proxies in Scrapy

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}

# spider.py
def start_requests(self):
    yield scrapy.Request(
        "https://httpbin.org/ip",
        meta={"proxy": "http://user:pass@host:port"},
    )

Scrapy's built-in HttpProxyMiddleware routes any request whose meta['proxy'] is set, so a rotating gateway is as simple as assigning a new value per request. For large crawls pair it with a rotation middleware such as scrapy-rotating-proxies or a residential backconnect endpoint that rotates the exit IP server-side. Keep CONCURRENT_REQUESTS and AUTOTHROTTLE tuned to your proxy plan's session limits to avoid bans.

Scrapy works cleanly with rotating backconnect endpoints — point it at a single gateway host and the provider rotates the exit IP for you.

Our #1 pick is a sponsored partner. Remaining providers are ranked editorially by rating and value.