Mirroring on demand - ParabolaWiki

# Mirroring on demand

Note: see Creating a mirror instead for an alternative in which packages are synchronized periodically, which is the method normally used by public mirrors.

Since 70-100Gb could be a bit too much for mirroring Parabola repos, you can make your web server fetch, cache and serve packages on users' demand.

The basic procedure would be like this:

- Maintain repo databases updated. This means don't cache them at all or cache them for as much as a day.

- Make your web server proxy package and package signatures (or anything but *.db) and cache them in a way that the second query will be served locally.

Feel free to add instructions for your favourite web servers here.

## Using Nginx

Being a reverse proxy, Nginx can be configured to proxy Parabola repos while filling a local cache. Here's an explained snippet:

    # You can redirect queries to any number of mirrors to distribute the load
    # as long as they serve the 'repo.parabolagnulinux.org' subdomain
    upstream parabola {
        # Our repo IP
        server 80.87.131.252;
        # Other mirror IPs!
    }

    # The mirror
    # It serves two subdomains:
    # * The actual mirror, and
    # * A bogus main repo subdomain so this server can act as another mirror
    #   upstream ;)
    server {
        listen 80;
        server_name mirror.example.org repo.parabolagnulinux.org;

        root /srv/http/mirror.example.org;
        autoindex on;

        # Databases are never cached
        # Queries are proxied directly to upstream
        # We trick upstream into serving the main repo subdomain
        location ~ \.db$ {
            expires 1h;
            proxy_pass http://parabola;
            proxy_set_header Host repo.parabolagnulinux.org;
        }

        # Get and cache everything else
        # The error_page directive means 'go fetch anything that can't be found here'
        location / {
            expires 7d;
            error_page 403 404 = @get;
        }

        # Bogus location that redirects queries
        # Pass it to the parabola upstream
        # We trick upstream into serving the main repo subdomain
        # Store the files in this format
        # Give them 644 permissions
        location @get {
            proxy_pass http://parabola;
            proxy_set_header Host repo.parabolagnulinux.org;
            proxy_store /srv/http/mirror.example.org$request_uri;
            proxy_store_access user:rw group:r all:r;
        }
    }

## Cleaning up

You can cleanup old packages using the db-cleanup utility from our dbscripts. Clone it from https://projects.parabolagnulinux.org/dbscripts.git, edit the config file to suit your needs and run db-cleanup every once in a while (we do it weekly).

Category:

- Hacking