Web scraping on a VPS with rotating proxies

Most scrapers start life on a laptop. That's fine until you want the data every morning, the laptop is asleep, and the site you're collecting from has started answering your home address with a CAPTCHA.

Moving the scraper to a small VPS fixes the first problem: it runs on a schedule, all the time, on a fast connection. Putting rotating proxies in front of it fixes the second. This post sets up both, with Scrapy on Ubuntu, and ends with a crawler that runs itself once a day.

Why a VPS, and why proxies on top#

A VPS gives a scraper what a laptop can't: it's always on, it has a stable connection, and it sits in a data centre close to most of the sites you'll fetch from. A small server is plenty. Scrapy is light: one vCPU and 2 GB of RAM will run a crawl of tens of thousands of pages without breaking a sweat.

What a VPS doesn't give you is more than one address. Every request your crawler makes comes from the server's single IPv4, and sites notice when one address fetches a few thousand pages. The usual answers are rate limits, CAPTCHAs and blocks, sometimes for a whole range rather than just you.

A rotating proxy service sits between your crawler and the site. You send every request to one gateway, and the gateway sends it on from an address picked from a large pool. To the site, your crawl looks like modest traffic from many places instead of a lot from one. Your own server's address stays out of it, so it isn't the one that ends up on a blocklist.

One distinction, because it matters for our Acceptable Use Policy: this is about using a proxy service from your server. Running an open proxy on a VPSNine server, one anyone can relay through, isn't allowed.

Scrape politely first#

Proxies spread your requests out. They don't make a badly behaved crawler acceptable, and a crawl that hammers a small site is a problem whichever address it comes from. Before you write any code:

• Read robots.txt and the site's terms. Scrapy obeys robots.txt by default. Leave it on.
• Go slowly. A few requests a second to one site is a lot for a small business's server. Scrapy's AutoThrottle backs off when the site slows down.
• Collect what you need. Public prices, listings and articles are the usual targets. Personal data brings legal duties with it (GDPR in Europe, for example) whatever its source.
• Say who you are. Set a user agent with a way to contact you. Site owners are far more patient with a crawler they can email.

Install Scrapy on the server#

These steps use Ubuntu 24.04; Debian 12 is the same. Connect over SSH (see Connect with SSH if this is your first time), then create a user for the scraper so it doesn't run as root:

sudo apt update
sudo apt install -y python3-venv
sudo adduser --disabled-password --gecos "" scraper
sudo -iu scraper

As the scraper user, create a project with its own Python environment:

python3 -m venv ~/venv
~/venv/bin/pip install scrapy
~/venv/bin/scrapy startproject quotes
cd quotes

Recent Ubuntu and Debian releases refuse a system-wide pip install, and the virtual environment is the clean way around that.

For practice we'll scrape quotes.toscrape.com, a site built for exactly this. Save this as quotes/spiders/quotes.py:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("a.tag::text").getall(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Run it once without proxies to check it works:

~/venv/bin/scrapy crawl quotes -O quotes.json

You should get a file with 100 quotes.

Route requests through the proxy gateway#

A rotating proxy service gives you a gateway host, a port, a username and a password. Test them from the server with curl before touching Scrapy:

curl https://api.ipify.org; echo
curl -x "http://USERNAME:PASSWORD@GATEWAY:PORT" https://api.ipify.org; echo
curl -x "http://USERNAME:PASSWORD@GATEWAY:PORT" https://api.ipify.org; echo

The first line prints your server's own address. The other two should print addresses from the proxy pool, and with a rotating gateway they'll usually differ from each other.

Keep the credentials out of your code. Exit back to your own account (exit) and put them in a file only root can read:

sudo tee /etc/quotes-scraper.env > /dev/null <<'EOF'
PROXY_GATEWAY=GATEWAY:PORT
PROXY_USER=USERNAME
PROXY_PASS=PASSWORD
EOF
sudo chmod 600 /etc/quotes-scraper.env

Scrapy already knows how to use a proxy: set request.meta["proxy"] to a URL with credentials in it and its built-in proxy middleware does the rest. So all we need is a small middleware that fills that in for every request. Add this to quotes/middlewares.py:

import os
import secrets
from urllib.parse import quote


class RotatingProxyMiddleware:
    """Sends every request through the proxy gateway, each on its own session."""

    def __init__(self):
        self.gateway = os.environ["PROXY_GATEWAY"]
        self.user = os.environ["PROXY_USER"]
        self.password = quote(os.environ["PROXY_PASS"], safe="")

    def process_request(self, request, spider):
        # A request can ask for a sticky session; otherwise it gets a new one.
        session = request.meta.get("proxy_session") or secrets.token_hex(8)
        user = quote(f"{self.user}-session-{session}", safe="")
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.gateway}"

Then switch it on and slow the crawl down in quotes/settings.py:

DOWNLOADER_MIDDLEWARES = {
    "quotes.middlewares.RotatingProxyMiddleware": 350,
}

ROBOTSTXT_OBEY = True
USER_AGENT = "quotes-bot (+mailto:you@example.com)"
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True

The number 350 places our middleware before Scrapy's own proxy middleware (750), so the proxy URL is set by the time that one reads it.

What the session ID is for#

The -session-... part of the username is the bit most guides skip, and it's why a "rotating" setup sometimes doesn't seem to rotate.

A rotating gateway picks a new exit address per connection, not per request. For HTTPS sites, your client opens a tunnel through the proxy and sends requests inside it, and the whole tunnel leaves from one address. Scrapy, like most HTTP clients, keeps connections open and reuses them, so without anything else a long run of requests can come from the same address.

Most gateways let you name a session in the username. The same session name keeps the same exit; a new name gets a new one. Our middleware gives every request a fresh random name, so every request gets its own connection and its own exit. That costs a new TLS handshake per request, which is fine at polite crawl speeds.

Sometimes you want the opposite. Logging in, or paging through search results the site ties to one visitor, works better from one address. Set the session yourself for those requests:

yield scrapy.Request(url, meta={"proxy_session": "login1"})

Check your provider's documentation for the exact username format; -session-<id> is common but not universal. Gateways also cap how long a sticky session holds its address, so long jobs should move to a new session name now and then.

Run it every day with a systemd timer#

As the scraper user, make a folder for the output:

sudo -iu scraper mkdir -p /home/scraper/data

Create the service, /etc/systemd/system/quotes-scraper.service:

[Unit]
Description=Quotes scraper
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=scraper
WorkingDirectory=/home/scraper/quotes
EnvironmentFile=/etc/quotes-scraper.env
ExecStart=/home/scraper/venv/bin/scrapy crawl quotes -O /home/scraper/data/quotes.json

And the timer that starts it, /etc/systemd/system/quotes-scraper.timer:

[Unit]
Description=Run the quotes scraper daily

[Timer]
OnCalendar=*-*-* 06:00:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target

systemd reads the environment file before it drops to the scraper user, so the credentials can stay root-only. Start the timer and run the job once by hand to check it:

sudo systemctl daemon-reload
sudo systemctl enable --now quotes-scraper.timer
sudo systemctl start quotes-scraper.service
journalctl -u quotes-scraper.service -n 30

The log ends with Scrapy's stats: requests made, responses by status code, items scraped. systemctl list-timers shows when the next run is due. Persistent=true means a run missed while the server was rebooting happens as soon as it's back.

Choosing the proxies#

Which kind you need depends on the sites you're collecting from.

• Rotating datacenter proxies are the place to start. They're fast and cheap per gigabyte, and most sites are happy with polite traffic from them.
• Residential proxies exit through home connections. They cost more, and they're worth it only for sites that turn away data-centre addresses outright.
• Unmetered plans, priced by concurrent connections instead of by gigabyte, make sense once your crawls move a lot of data.

Text pages are small, so a crawl of a few thousand pages a day uses surprisingly little proxy traffic. Skip images and other files you don't need and it uses even less.

We use rotating proxies from Node4 in our own examples. In fairness, we should say Node4 is run by Tekari, the same company that runs VPSNine. Any provider with a rotating gateway works with the middleware above.

How big a server you need#

Scrapy itself needs very little. The smallest plan runs crawls like this one comfortably, and the limits you'll hit first are the site's patience and your proxy allowance, not CPU.

That changes with headless browsers. If the site builds its pages with JavaScript and you need Playwright or Selenium to render them, each browser takes a few hundred megabytes of RAM, and several in parallel want 4 GB or more. Choosing a plan covers the sizes.

Either way, the shape stays the same: a small server that never sleeps, a crawler that takes its time, and a pool of addresses so no single one does all the work.