Some time back in April 2013 I was bored and looking for a new project to keep my attention, if only for a short period of time.

My colleague @codeinthehole had an idea but no time to implement it, this idea was to have shields like those of travis-ci (shown below) but displaying package download counts.

Status of blackhole on Travis CI

Tech stack

From the very start I decided to use Tornado framework, although this may change in the future.

The original plan was to generate the images using Pillow (PIL) and then simply cache them to disk. I decided it would make far more sense to do this using Varnish and not have to worry about it working as expected.

Manually generating the images

The images were originally generated from a base template using Pillow, but sadly Python’s image manipulation is not very good, especially it’s text manipulation and the shields could …

Today I can happily announce that the Google Analytics tracking code has been removed from this website.

Goodybye Google Analytics & hello Open Web Analytics

I’ve been planning on doing it from a while because I do not like Google Analytics, I don’t like being tracked and I actively stopped trying to use Google services for my own reasons.

The company I work for uses Piwik for some of our clients, I am not a fan of Piwik or how it works and does things. I did some research and found some service providers but their free options were limited or I felt they weren’t a good fit, eventually I stumbled upon Open Web Analytics and decided that it not only suited my purposes, but it meant servers I control would hold the analytical data, rather than some third party.

Hello DuckDuckGo

After launching the new version of …

I love my prompt, always have and always will. I spend so much of my life in a terminal, usually with half a dozen mini terminals open in each tab. As such I like to tweak it and get it as perfect as possible for my life, needs and even mood.

In the past I’ve had quite a large PS1 that covers multiple lines and gives a lot of information, after having that PS1 in one form or another for some time I decided it was time for a change, to a smaller PS1 that takes up a lot less space.

So here it is, the first image is my standard PS1 when in a git repository, the red @ means a file hasn’t been added to Git, a blue @ means a tracked file has been modified but not stage and finally a green @ means a file is staged …

Over the last week I’ve been doing a huge amount of refactoring of Blackhole as well as writing dozens of additional tests. To make Blackhole more testable I needed to make a big change to how the program is launched and controlled.

setup.py scripts vs. entry_points

Whenever I’ve written Python programs that require some kind of command line script I’ve always used distutils’ scripts, this can be seen in blackhole’s setup.py on GitHub or in the three line example below.

scripts=[
    'blackhole/bin/blackhole',
],

In doing so, it allowed me to be lazy and write a lot of prodecural code in the main “binary” which made it pretty much impossible to test. You can also see that on GitHub in the main “binary”.

I’ve noticed that most people who write Python packages that have some kind of command line entry point use distutils …

Pelican is a Python-powered static blog generator that processes ReStructuredText and Markdown articles and pages and converts them to HTML. I use Pelican to power this blog.

There is a YouTube RST directive built in to Pelican core but it really shouldn’t exist there.

I submitted a pull request for Pelican core to enable Vimeo videos in articles but that request was declined because it didn’t belong in the core. So I decided I would write it as a plugin instead and while I was doing it, also wrote a plugin for YouTube so that it could be removed from the core.

There is a decent amount of detail in the Pelican documentation on how to write plugins, I’m not going to cover the whole process but I thought I would cover a little of what I did.

Adding an RST directive

Really all we’re doing …

Blackhole has always been able to handle unencrypted SMTP and for a long time it’s been able to handle encrypted SMTP via TLSv1.

One thing Blackhole hasn’t been able to do until the 1.7.0 release is handle STARTTLS.

In the past the STARTTLS command would cause Blackhole to return the standard 250 OK response but would continue to operate on unencrypted SMTP.

I wanted to fix this and do it properly, but this meant learning how to do so with Tornado, which itself proved to be tricky. I ended up deciding to go to my local coding spot - the pub and hash it out.

connection_stream

The first thing I had to do was refactor the code that created the instance of tornado.iostream.IOStream and tornado.iostream.SSLIOStream so that it didn’t actually do the ssl wrapping.

def connection_stream(connection):
    """
    Detect which socket the connection …

As part of my effort to make Blackhole as useful and usable as possible, I needed to be able to support SSL/TLS enabled connections.

Tornado itself has two built-in IOStreams that help us do the job; the first is the standard IOStream and the second is the SSLIOStream.

With this in mind we simply need to spawn two sockets, by default these listen on port 25 for standard SMTP and port 465 for SSL/TLS encrypted SMTP. With these two sockets bound we’re then very simply able to listen for incoming connections on either socket and use socket.socket.getsockname() to figure out if the connection is to the encrypted or unencrypted socket.

Code

def connection_stream(connection):
    """
    Detect which socket the connection is being made on,
    create and iostream for the connection, wrapping it
    in SSL if connected over the SSL socket.

    The parameter 'connection' is an instance …

I recently wrote an article on using haproxy, SSL and SPDY with nginx backend servers.

This article is a little extra on top of that to explain how to enable statistics for haproxy so you can monitor the backend statuses etc.

Example stats page

Moar stats!

Enabling stats

listen stats :8000
    mode http
    stats enable
    stats hide-version
    stats realm haproxy\ stats
    stats uri /
    stats auth admin:admin

Place the above content in the haproxy configuration file (/etc/haproxy/haproxy.cfg).

Be sure to replace admin:admin with your a proper username and password, username first, password after the colon.

Restart haproxy, and then browse to http://yousite.com:8000.

I have been a frequent audience member of DJUGL for a few years now and spend most of the time asking questions, playing devils advocate and generally being my annoying self.

I have repeatedly said I would do a talk but never got round to it until Jon basically forced me to get round to it.

My talk was on blackhole/blackhole.io and covered several topics including PyPy, SimpleMTA and moved on to talking about spamming and starting work on my honey pot suite called Nectar.

You can find the slides on Speaker Deck, sadly I ran out of time when creating them and although I was promised time to finish them at work, I got busy. So I replaced content with “Taylor Replacements(tm)”.

The event, attendees and other speakers are listen on the Lanyrd event page.

Several people took photos of the event, I don’t remember …

I wrote an article last week explaining that I had changed my blog and built my own nginx packages with SPDY built in.

I decided I would take things a little further and poke around with haproxy some more. The initial plan was to compile the latest dev source of haproxy with SSL termination enabled.

In doing so I realised I would lose SPDY support, which upset me a little. After some digging I found that the 1.5-dev branch of haproxy supports npn and thus can handle SPDY.

I tweaked my builds a little more and managed to get haproxy running as an SSL terminating load balancer, with SPDY connections being sent off to my nginx servers with SPDY enabled and all other non-SPDY connections were passed on to an nginx virtual host with SPDY disabled.

Requirements

I have released my haproxy build as a debian file below …

I decided to rebuild syslog.tv as pure HTML using RST and Pelican and rebrand it as kura.io.

In doing so I decided I would go all out and use SPDY and ngx_pagespeed (mod_pagespeed) for fun to see exactly what I could do.

Sadly no version of nginx has been officially released with SPDY or ngx_pagespeed enabled, you can compile nginx from source to enable SPDY so I thought I would go ahead and do it, releasing some Debian packages in the process.

After compiling nginx from the source package available at the Ubuntu PPA, I decided I would go further and compile in ngx_pagespeed.

Installing

I have released the 4 required debian packages below (please note they are only available for amd64);

FILE GPG MD5 SHA1
nginx_1.4.1_all.deb owGMl1NwLw… 42f790a1f6… f4495055e9…
nginx-common_1.4.1_all.deb owGMt1NwJU… ca4ec5688d… 633bfc2eaa…
nginx-full_1.4.1_amd64.deb owG09ndUk1… 4776dc6c7f… 0e4e124acc…
nginx-doc_1 …

I spend a LOT of time with tunnels open to multiple machines, connecting directly to PostgreSQL, RabbitMQ and many other services all via SSH.

I have written several helper functions and this is the final version that I created in a small competition with @codeinthehole.

Installation

Simply add the contents to ~/.bashrc

Usage

Usage is pretty simply, just called portforward from the command line, pressing <TAB> as you type in a server name from your ~/.ssh/config file and the same with the port.

portforward sy<TAB>

Will become:

portforward syslog.tv

And finally

portforward syslog.tv 15672