While pypip.in is available under the MIT license on GitHub, it’s not explained how to really use it properly.
You can gather how to set-up the Python source of the project and get the Twisted process running, this is totally reliant on using the img.shields.io.
I decided to write this article explaining how to install your own copy of the shields nodejs code, pypipin itself and even cover off supervisord and Varnish too.
shields & nodejs
nodejs
First of all you’ll need to get the latest source code copy of nodejs from the nodejs download page.
Extract it.
tar -xvzf node-<VERSION>.tar.gz
cd node-<VERSION>
You’ll need to install the build tools, if you don’t have them already.
sudo apt-get install build-essential
And then make and install node.
make && sudo make install
Redis
Redis is used to temporarily store PyPI responses.
sudo apt-get install redis-server
shields
You’ll need to install git for the next step.
sudo apt-get install git-core
And get a copy of the shields repository.
git clone git@github.com:badges/shields.git
Once it’s checked out, you’ll need to install it’s requirements.
cd shields && npm install
You can confirm this works by running the shields server.
node server 8080
pypipin
Clone the shields repository, the same way you did for shields above.
git clone git@github.com:badges/pypipins.git
To do this properly, you’ll need to make sure you have virtualenv for Python.
sudo apt-get install python-dev
wget -O - https://bootstrap.pypa.io/get-pip.py | sudo python
sudo pip install virtualenv
Next, you’ll need to create a virtual environment.
virtualenv /path/to/where/you/want/it/
Then you can active it.
. /path/to/your/virtualenv/bin/activate
And install the dev requirements from the pypipins directory.
pip install -r /path/to/pypipins/clone/requirements-dev.txt
You’ll need to edit shields.py, commenting out the img.shields.io host and uncommon the local one.
/path/to/pypipins/clone/shields/shields.py
# SHIELD_URL = "https://img.shields.io/badge/%s-%s-%s.%s"
SHIELD_URL = "http://localhost:9000/badge/%s-%s-%s.%s" # pypip.in uses a local version of img.shields.io
Once this is done, you can test the pypipins server.
/path/to/your/virtualenv/bin/python /path/to/pypipins/clone/shields/shields.py
supervisor(d)
Always install supervisor from apt, rather than from pip.
sudo apt-get install supervisor
Then cd to where conf.d config files are stored for supervisor.
cd /etc/supervisor/conf.d/
In here, you’ll need to create a configuration for the shields nodejs server and also for pypipins server.
shields.conf
[program:shields]
directory=/path/to/shields/
command=node server 9000
stdout_logfile=/var/log/supervisor/shields.log
stderr_logfile=/var/log/supervisor/shields.error.log
user=www-data
group=www-data
autostart=true
autorestart=true
pypipin.conf
[program:pypipin]
command=/path/to/virtualenv/bin/python /path/to/pypipin/clone/shields/shields.py
stdout_logfile=/var/log/supervisor/pypipin.log
stderr_logfile=/var/log/supervisor/pypipin.error.log
user=www-data
group=www-data
autostart=true
autorestart=true
Once this is done, you’ll need to load them in to supervisor itself.
sudo supervisorctl
reread
add shields
add pypipin
Now supervisor will automatically start both processes and keep them alive.
Varnish
The final step is to put Varnish in front of the system to cache images for you. The shields server has the ability to use redis for caching but, I’d rather do this with a proper HTTP cache rather than use redis.
sudo apt-get install varnish
Tell Varnish to run on port 80.
/etc/default/varnish
DAEMON_OPTS="-a :80 \
-T localhost:6082 \
-f /etc/varnish/default.vcl \
-S /etc/varnish/secret \
-s malloc,256m"
This will run the Varnish HTTP server on port 80 and keep it’s admin interface hidden from the world, binding it to port 6082 on the lo interface.
The final step is to tell Varnish about the pypipins server.
/etc/varnish/default.vcl
backend default {
.host = "127.0.0.1";
.port = "8888";
}
sub vcl_recv {
if (req.request != "GET") {
return(pipe);
}
if (req.request == "GET") {
remove req.http.cookie;
remove req.http.authenticate;
remove req.http.Etag;
remove req.http.If-None-Match;
return(lookup);
}
return(pass);
}
sub vcl_fetch {
if (beresp.status >= 300) {
return(hit_for_pass);
}
set beresp.ttl = 1h;
set beresp.grace = 6h;
unset beresp.http.Set-Cookie;
unset beresp.http.Etag;
unset beresp.http.Cache-Control;
set beresp.http.Cache-Control = "no-cache";
return (deliver);
}
sub vcl_deliver {
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT";
set resp.http.X-Cache-Hits = obj.hits;
} else {
set resp.http.X-Cache = "MISS";
}
}
All done, restart Varnish.
sudo /etc/init.d/varnish restart
You’ll be able to go to http://yourserver.tld/download/<PACKAGE>/badge.svg and everything should be working as expected.
Notes on PyPy
I personally use PyPy for running the pypipins server because, it’s a long running process and PyPy speeds it up wonderfully.
If you’re using Debian 7, the latest version of PyPy as of writing is 2.3.1 and requires libffi6, if you’re using one of the prebuilt binaries. libffi6 is only available in Jessie which is currently in testing.
You can either use an older version of PyPy or, backport libffi6 from Jessie.
/etc/apt/sources.list
deb ftp://ftp.debian.org/debian/ jessie main
/etc/apt/preferences.d/jessie
Package: *
Pin: release a=wheezy
Pin-Priority: 900
Package: libffi*
Pin: release a=jessie
Pin-Priority: 910
This will keep all packages pinned to wheezy except libffi+wildcard, which will be pulled from Jessie.
You can then simply install libffi6 from Jessie.
sudo apt-get update
sudo apt-get -u install libffi6/jessie
SSL
If you want to use SSL with your shields, you’ll need to install nginx in front of Varnish.
So instead of running Varnish on port 80, as shown above. Put it on a different port, install and use nginx as you would for any other website and simply proxy all requests back to Varnish.
