# Web scraping on a VPS with rotating proxies

*2026-10-18 · VPSNine*

> Run a Scrapy crawler on a Linux VPS on a schedule, send its requests through rotating proxies, and keep it polite enough that nobody needs to block it.

Source: https://vpsnine.com/blog/web-scraping-on-a-vps-with-rotating-proxies

Most scrapers start life on a laptop. That's fine until you want the data every morning, the laptop is asleep, and the site you're collecting from has started answering your home address with a CAPTCHA.

Moving the scraper to a small VPS fixes the first problem: it runs on a schedule, all the time, on a fast connection. Putting rotating proxies in front of it fixes the second. This post sets up both, with Scrapy on Ubuntu, and ends with a crawler that runs itself once a day.

## Why a VPS, and why proxies on top

A VPS gives a scraper what a laptop can't: it's always on, it has a stable connection, and it sits in a data centre close to most of the sites you'll fetch from. A small server is plenty. Scrapy is light: one vCPU and 2 GB of RAM will run a crawl of tens of thousands of pages without breaking a sweat.

What a VPS doesn't give you is more than one address. Every request your crawler makes comes from the server's single IPv4, and sites notice when one address fetches a few thousand pages. The usual answers are rate limits, CAPTCHAs and blocks, sometimes for a whole range rather than just you.

A rotating proxy service sits between your crawler and the site. You send every request to one gateway, and the gateway sends it on from an address picked from a large pool. To the site, your crawl looks like modest traffic from many places instead of a lot from one. Your own server's address stays out of it, so it isn't the one that ends up on a blocklist.

One distinction, because it matters for our [Acceptable Use Policy](https://vpsnine.com/aup): this is about *using* a proxy service from your server. Running an open proxy on a VPSNine server, one anyone can relay through, isn't allowed.

## Scrape politely first

Proxies spread your requests out. They don't make a badly behaved crawler acceptable, and a crawl that hammers a small site is a problem whichever address it comes from. Before you write any code:

- **Read robots.txt and the site's terms.** Scrapy obeys robots.txt by default. Leave it on.
- **Go slowly.** A few requests a second to one site is a lot for a small business's server. Scrapy's AutoThrottle backs off when the site slows down.
- **Collect what you need.** Public prices, listings and articles are the usual targets. Personal data brings legal duties with it (GDPR in Europe, for example) whatever its source.
- **Say who you are.** Set a user agent with a way to contact you. Site owners are far more patient with a crawler they can email.

## Install Scrapy on the server

These steps use Ubuntu 24.04; Debian 12 is the same. Connect over SSH (see [Connect with SSH](https://vpsnine.com/help/connect-with-ssh) if this is your first time), then create a user for the scraper so it doesn't run as root:

```bash
sudo apt update
sudo apt install -y python3-venv
sudo adduser --disabled-password --gecos "" scraper
sudo -iu scraper
```

As the `scraper` user, create a project with its own Python environment:

```bash
python3 -m venv ~/venv
~/venv/bin/pip install scrapy
~/venv/bin/scrapy startproject quotes
cd quotes
```

Recent Ubuntu and Debian releases refuse a system-wide `pip install`, and the virtual environment is the clean way around that.

For practice we'll scrape [quotes.toscrape.com](https://quotes.toscrape.com), a site built for exactly this. Save this as `quotes/spiders/quotes.py`:

```python
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("a.tag::text").getall(),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)
```

Run it once without proxies to check it works:

```bash
~/venv/bin/scrapy crawl quotes -O quotes.json
```

You should get a file with 100 quotes.

## Route requests through the proxy gateway

A rotating proxy service gives you a gateway host, a port, a username and a password. Test them from the server with curl before touching Scrapy:

```bash
curl https://api.ipify.org; echo
curl -x "http://USERNAME:PASSWORD@GATEWAY:PORT" https://api.ipify.org; echo
curl -x "http://USERNAME:PASSWORD@GATEWAY:PORT" https://api.ipify.org; echo
```

The first line prints your server's own address. The other two should print addresses from the proxy pool, and with a rotating gateway they'll usually differ from each other.

Keep the credentials out of your code. Exit back to your own account (`exit`) and put them in a file only root can read:

```bash
sudo tee /etc/quotes-scraper.env > /dev/null <<'EOF'
PROXY_GATEWAY=GATEWAY:PORT
PROXY_USER=USERNAME
PROXY_PASS=PASSWORD
EOF
sudo chmod 600 /etc/quotes-scraper.env
```

Scrapy already knows how to use a proxy: set `request.meta["proxy"]` to a URL with credentials in it and its built-in proxy middleware does the rest. So all we need is a small middleware that fills that in for every request. Add this to `quotes/middlewares.py`:

```python
import os
import secrets
from urllib.parse import quote


class RotatingProxyMiddleware:
    """Sends every request through the proxy gateway, each on its own session."""

    def __init__(self):
        self.gateway = os.environ["PROXY_GATEWAY"]
        self.user = os.environ["PROXY_USER"]
        self.password = quote(os.environ["PROXY_PASS"], safe="")

    def process_request(self, request, spider):
        # A request can ask for a sticky session; otherwise it gets a new one.
        session = request.meta.get("proxy_session") or secrets.token_hex(8)
        user = quote(f"{self.user}-session-{session}", safe="")
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.gateway}"
```

Then switch it on and slow the crawl down in `quotes/settings.py`:

```python
DOWNLOADER_MIDDLEWARES = {
    "quotes.middlewares.RotatingProxyMiddleware": 350,
}

ROBOTSTXT_OBEY = True
USER_AGENT = "quotes-bot (+mailto:you@example.com)"
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True
```

The number 350 places our middleware before Scrapy's own proxy middleware (750), so the proxy URL is set by the time that one reads it.

### What the session ID is for

The `-session-...` part of the username is the bit most guides skip, and it's why a "rotating" setup sometimes doesn't seem to rotate.

A rotating gateway picks a new exit address per connection, not per request. For HTTPS sites, your client opens a tunnel through the proxy and sends requests inside it, and the whole tunnel leaves from one address. Scrapy, like most HTTP clients, keeps connections open and reuses them, so without anything else a long run of requests can come from the same address.

Most gateways let you name a session in the username. The same session name keeps the same exit; a new name gets a new one. Our middleware gives every request a fresh random name, so every request gets its own connection and its own exit. That costs a new TLS handshake per request, which is fine at polite crawl speeds.

Sometimes you want the opposite. Logging in, or paging through search results the site ties to one visitor, works better from one address. Set the session yourself for those requests:

```python
yield scrapy.Request(url, meta={"proxy_session": "login1"})
```

Check your provider's documentation for the exact username format; `-session-<id>` is common but not universal. Gateways also cap how long a sticky session holds its address, so long jobs should move to a new session name now and then.

## Run it every day with a systemd timer

As the `scraper` user, make a folder for the output:

```bash
sudo -iu scraper mkdir -p /home/scraper/data
```

Create the service, `/etc/systemd/system/quotes-scraper.service`:

```ini
[Unit]
Description=Quotes scraper
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=scraper
WorkingDirectory=/home/scraper/quotes
EnvironmentFile=/etc/quotes-scraper.env
ExecStart=/home/scraper/venv/bin/scrapy crawl quotes -O /home/scraper/data/quotes.json
```

And the timer that starts it, `/etc/systemd/system/quotes-scraper.timer`:

```ini
[Unit]
Description=Run the quotes scraper daily

[Timer]
OnCalendar=*-*-* 06:00:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target
```

systemd reads the environment file before it drops to the `scraper` user, so the credentials can stay root-only. Start the timer and run the job once by hand to check it:

```bash
sudo systemctl daemon-reload
sudo systemctl enable --now quotes-scraper.timer
sudo systemctl start quotes-scraper.service
journalctl -u quotes-scraper.service -n 30
```

The log ends with Scrapy's stats: requests made, responses by status code, items scraped. `systemctl list-timers` shows when the next run is due. `Persistent=true` means a run missed while the server was rebooting happens as soon as it's back.

## Choosing the proxies

Which kind you need depends on the sites you're collecting from.

- **Rotating datacenter proxies** are the place to start. They're fast and cheap per gigabyte, and most sites are happy with polite traffic from them.
- **Residential proxies** exit through home connections. They cost more, and they're worth it only for sites that turn away data-centre addresses outright.
- **Unmetered plans**, priced by concurrent connections instead of by gigabyte, make sense once your crawls move a lot of data.

Text pages are small, so a crawl of a few thousand pages a day uses surprisingly little proxy traffic. Skip images and other files you don't need and it uses even less.

We use [rotating proxies](https://node4.io/rotating-proxies) from Node4 in our own examples. In fairness, we should say Node4 is run by Tekari, the same company that runs VPSNine. Any provider with a rotating gateway works with the middleware above.

## How big a server you need

Scrapy itself needs very little. The smallest [plan](https://vpsnine.com/plans) runs crawls like this one comfortably, and the limits you'll hit first are the site's patience and your proxy allowance, not CPU.

That changes with headless browsers. If the site builds its pages with JavaScript and you need Playwright or Selenium to render them, each browser takes a few hundred megabytes of RAM, and several in parallel want 4 GB or more. [Choosing a plan](https://vpsnine.com/help/choosing-a-plan) covers the sizes.

Either way, the shape stays the same: a small server that never sleeps, a crawler that takes its time, and a pool of addresses so no single one does all the work.