How to Integrate ProxyTee with Scrapy Framework
Plug ProxyTee's 30M+ rotating residential ports into Scrapy downloader middlewares to crawl millions of e-commerce, directory, and SERP pages at flat monthly port rates.
Why Pair Scrapy Framework with ProxyTee?
Scrapy is Python's premier industrial-strength asynchronous crawling framework. When crawling millions of product pages or search listings, per-GB residential proxies quickly become the single largest line item in your cloud budget. Integrating ProxyTee's Unlimited Residential Ports into your Scrapy `DownloaderMiddleware` lets your spiders run at full concurrency 24/7 with predictable flat-rate billing.
- 30,000,000+ Real Household IPs: Route Scrapy Framework traffic through genuine residential consumer ISPs across 100+ countries.
- Zero Per-Gigabyte Fees: Run 24/7 Scrapy Framework workflows on flat-rate concurrent ports starting at $29/month.
- Native HTTP, HTTPS & SOCKS5: Switch protocols or adjust sticky rotation windows (3–60 minutes) directly inside the ProxyTee dashboard.
Built for 24/7 Scrapy Spiders
Crawl terabytes of HTML and JSON feeds every month on flat-rate ports ($13.99–$29/port).
Multi-Port Load Balancing
Distribute Scrapy `CONCURRENT_REQUESTS` across 5, 25, 50, or 100 ProxyTee ports effortlessly.
30M+ Residential IP Pool
Avoid subnet bans on Amazon, Walmart, Booking.com, Yelp, and Google.
Serverless & Scrapyd Compatible
Deploy on Scrapyd, AWS Lambda, or Kubernetes with IP Whitelist authentication.
Step-by-Step Scrapy Framework Proxy Setup
Follow these four steps to connect your ProxyTee residential port to Scrapy Framework.
1. Configure Your ProxyTee Port Pool
In the ProxyTee Dashboard, set your ports to 3-minute auto-rotation and either whitelist your Scrapy server IP or copy your `username:password` credentials.
2. Add a Custom Proxy Middleware in `middlewares.py`
Create a lightweight `ProxyTeeMiddleware` that assigns `request.meta['proxy']` from your pool of ProxyTee ports.
3. Enable the Middleware in `settings.py`
Register `ProxyTeeMiddleware` in `DOWNLOADER_MIDDLEWARES` before Scrapy's built-in `HttpProxyMiddleware`.
4. Launch Your Spider (`scrapy crawl`)
Run your Scrapy spiders continuously with automatic retry handling and zero per-GB bandwidth charges.
Scrapy Downloader Middleware (`middlewares.py`) for ProxyTee
import base64
import random
class ProxyTeeDownloaderMiddleware:
PROXY_PORTS = [
"http://gate.proxytee.com:8080",
"http://gate.proxytee.com:8081",
"http://gate.proxytee.com:8082",
]
PROXY_USER_PASS = b"your_proxytee_username:your_proxytee_password"
def process_request(self, request, spider):
request.meta["proxy"] = random.choice(self.PROXY_PORTS)
encoded = base64.b64encode(self.PROXY_USER_PASS).decode("utf-8")
request.headers["Proxy-Authorization"] = f"Basic {encoded}"Related Scrapy Framework & Automation Guides
Get Started Web Scraping with Scrapy
Web scraping is a core technique for developers who need to extract data from websites at scale. Whether you're building price monitoring tools, collecting real-time analytics, or...
Top Python Web Scraping Libraries for 2026
Web scraping is an essential technique for extracting data from the internet, and Python is a popular choice for this task, thanks to its wide range of powerful libraries. This...
Strategies to Cut Your Web Scraping Costs
Cut your web scraping costs without sacrificing performance is entirely possible with the right tools and strategy. In today's data-driven world, web scraping powers everything...
Flexible pricing plans. Any scale.
Whether you're an individual, a small team, or a growing enterprise, choose the number of concurrent ports that fits your workload.
1 Port
Ideal for personal use or small projects with minimal traffic.
- Unlimited bandwidth
- 30 million IPs pool
- 3–60 minutes rotation
- HTTP & SOCKS5
- API Access
- User:pass & IP whitelist
5 Ports
Perfect for small teams or light testing across multiple devices.
- Unlimited bandwidth
- 30 million IPs pool
- 3–60 minutes rotation
- HTTP & SOCKS5
- API Access
- User:pass & IP whitelist
10 Ports
Great for medium-sized teams or projects with moderate traffic.
- Unlimited bandwidth
- 30 million IPs pool
- 3–60 minutes rotation
- HTTP & SOCKS5
- API Access
- User:pass & IP whitelist
25 Ports
Designed for growing teams needing more bandwidth.
- Unlimited bandwidth
- 30 million IPs pool
- 3–60 minutes rotation
- HTTP & SOCKS5
- API Access
- User:pass & IP whitelist
50 Ports
Best for businesses requiring stable connections and higher capacity.
- Unlimited bandwidth
- 30 million IPs pool
- 3–60 minutes rotation
- HTTP & SOCKS5
- API Access
- User:pass & IP whitelist
100 Ports
Suited for large-scale operations across multiple devices.
- Unlimited bandwidth
- 30 million IPs pool
- 3–60 minutes rotation
- HTTP & SOCKS5
- API Access
- User:pass & IP whitelist
Ready to Scale Up?
Credits Plans — Flexible pricing for high-volume users.
| Credits Package | Base Validity: Non-expire | Pro Validity: 30 days | Pro Max Validity: 15 days |
|---|---|---|---|
$1,000 Credits Save up to 30% | $900 | $800 | $700 |
$2,000 Credits Save up to 35% | $1,700 | $1,500 | $1,300 |
$5,000 Credits Save up to 40% | $4,000 | $3,500 | $3,000 |
$10,000 Credits Save up to 45% | $7,500 | $6,500 | $5,500 |
| Use credits to purchase or renew any proxy plan on demand. | Select Base | Select Pro | Select Pro Max |
Frequently Asked Questions About Scrapy Framework Proxies
Subscribe to a multi-port plan (5, 10, 25, 50, or 100 ports) and use random.choice() or round-robin selection in your Scrapy downloader middleware, while each port also auto-rotates its underlying residential IP every 3–60 minutes.