I love my prompt, always have and always will. I spend so much of my life in a terminal, usually with half a dozen mini terminals open in each tab. As such I like to tweak it and get it as perfect as possible for my life, needs and even mood.

In the past I’ve had quite a large PS1 that covers multiple lines and gives a lot of information, after having that PS1 in one form or another for some time I decided it was time for a change, to a smaller PS1 that takes up a lot less space.

So here it is, the first image is my standard PS1 when in a git repository, the red @ means a file hasn’t been added to Git, a blue @ means a tracked file has been modified but not stage and finally a green @ means a file is staged …

Over the last week I’ve been doing a huge amount of refactoring of Blackhole as well as writing dozens of additional tests. To make Blackhole more testable I needed to make a big change to how the program is launched and controlled.

scripts=[
    'blackhole/bin/blackhole',
],

Pelican is a Python-powered static blog generator that processes ReStructuredText and Markdown articles and pages and converts them to HTML. I use Pelican to power this blog.

There is a YouTube RST directive built in to Pelican core but it really shouldn’t exist there.

I submitted a pull request for Pelican core to enable Vimeo videos in articles but that request was declined because it didn’t belong in the core. So I decided I would write it as a plugin instead and while I was doing it, also wrote a plugin for YouTube so that it could be removed from the core.

There is a decent amount of detail in the Pelican documentation on how to write plugins, I’m not going to cover the whole process but I thought I would cover a little of what I did.

Blackhole has always been able to handle unencrypted SMTP and for a long time it’s been able to handle encrypted SMTP via TLSv1.

One thing Blackhole hasn’t been able to do until the 1.7.0 release is handle STARTTLS.

In the past the STARTTLS command would cause Blackhole to return the standard 250 OK response but would continue to operate on unencrypted SMTP.

I wanted to fix this and do it properly, but this meant learning how to do so with Tornado, which itself proved to be tricky. I ended up deciding to go to my local coding spot - the pub and hash it out.

def connection_stream(connection):
    """
    Detect which socket the connection …

As part of my effort to make Blackhole as useful and usable as possible, I needed to be able to support SSL/TLS enabled connections.

Tornado itself has two built-in IOStreams that help us do the job; the first is the standard IOStream and the second is the SSLIOStream.

With this in mind we simply need to spawn two sockets, by default these listen on port 25 for standard SMTP and port 465 for SSL/TLS encrypted SMTP. With these two sockets bound we’re then very simply able to listen for incoming connections on either socket and use socket.socket.getsockname() to figure out if the connection is to the encrypted or unencrypted socket.

def connection_stream(connection):
    """
    Detect which socket the connection is being made on,
    create and iostream for the connection, wrapping it
    in SSL if connected over the SSL socket.

    The parameter 'connection' is an instance …

I recently wrote an article on using haproxy, SSL and SPDY with nginx backend servers.

This article is a little extra on top of that to explain how to enable statistics for haproxy so you can monitor the backend statuses etc.

Example stats page

listen stats :8000
    mode http
    stats enable
    stats hide-version
    stats realm haproxy\ stats
    stats uri /
    stats auth admin:admin

I have been a frequent audience member of DJUGL for a few years now and spend most of the time asking questions, playing devils advocate and generally being my annoying self.

I have repeatedly said I would do a talk but never got round to it until Jon basically forced me to get round to it.

My talk was on blackhole/blackhole.io and covered several topics including PyPy, SimpleMTA and moved on to talking about spamming and starting work on my honey pot suite called Nectar.

You can find the slides on Speaker Deck, sadly I ran out of time when creating them and although I was promised time to finish them at work, I got busy. So I replaced content with “Taylor Replacements(tm)”.

The event, attendees and other speakers are listen on the Lanyrd event page.

Several people took photos of the event, I don’t remember …

I wrote an article last week explaining that I had changed my blog and built my own nginx packages with SPDY built in.

I decided I would take things a little further and poke around with haproxy some more. The initial plan was to compile the latest dev source of haproxy with SSL termination enabled.

In doing so I realised I would lose SPDY support, which upset me a little. After some digging I found that the 1.5-dev branch of haproxy supports npn and thus can handle SPDY.

I tweaked my builds a little more and managed to get haproxy running as an SSL terminating load balancer, with SPDY connections being sent off to my nginx servers with SPDY enabled and all other non-SPDY connections were passed on to an nginx virtual host with SPDY disabled.

I decided to rebuild syslog.tv as pure HTML using RST and Pelican and rebrand it as kura.gg.

In doing so I decided I would go all out and use SPDY and ngx_pagespeed (mod_pagespeed) for fun to see exactly what I could do.

Sadly no version of nginx has been officially released with SPDY or ngx_pagespeed enabled, you can compile nginx from source to enable SPDY so I thought I would go ahead and do it, releasing some Debian packages in the process.

After compiling nginx from the source package available at the Ubuntu PPA, I decided I would go further and compile in ngx_pagespeed.

I spend a LOT of time with tunnels open to multiple machines, connecting directly to PostgreSQL, RabbitMQ and many other services all via SSH.

I have written several helper functions and this is the final version that I created in a small competition with @codeinthehole.

Gist removed. Sorry.

portforward sy<TAB>

portforward syslog.tv

portforward syslog.tv 15672

I wrote this little Python program a while ago and now people are starting to email me about it, asking for it to be part of the DenyHosts Debian packages so I figured I’d write a quick article on it.

If you’re like my developers, you’ll find yourself getting banned from servers all the time and have to come speak to someone like me (your sys engineer/admin), or maybe you are an admin and are sick of people banning themselves and want and easy way to unban them.

So give this a try:

https://github.com/kura/denyhosts-unban

From the GitHub page you can download either the tarballs, zipballs or a Debian .deb package. Install it using the instructions in the README and you’re good to go.

sudo denyhosts-unban 10.0.0.1 …

If like me you run in to issue when using OpenJDK, my issues come from it’s memory problems when you’re allocating and using large amounts of memory - mostly for Solr where we’re concerned but obviously I’d switch for other high memory usage instances too.

So without further ado, lets get the installation going.

You’ll need Debian’s “add-apt-repository”, on servers this doesn’t usually come by default so we’ll need to install it.

sudo apt-get install python-software-properties

Next we need to add Java’s PPA.

sudo add-apt-repository ppa:sun-java-community-team/sun-java6

Once this is done we’ll need to update our apt caches and install Java 6.

sudo apt-get install sun-java6-jdk

Now that this is installed we should get the Java version, remember it for future.

java -version

You’ll get something like this

java version "1.6.0_20" OpenJDK Runtime Environment (IcedTea6 1.9 …

You’ll need mkpasswd, on Debian 6 and Ubuntu 12.04 you can install this using:

sudo apt-get install whois

It is pretty weird that it comes with the whois package though…

And then run

mkpasswd -m sha-512

I have built and released an open-source email server in the past for testing send rates and speeds, this project was called SimpleMTA and is available here.

Recently I have rebuilt this project for an internal project at work using the Tornado framework. Sadly this project as a whole cannot be released but a version of this code will be released in the near future.

Until that is released I have launched a new service called blackhole.io

I’ve recently been toying with my Raspberry Pi mirror including moving it out on to Amazon’s S3. I’ve written an article on how to back up to S3, but that isn’t enough when it comes to serving data from S3.

I needed the ability to RSYNC data from the official Raspberry Pi servers on to mine and then in to S3 and for that I used s3fs and FUSE.

sudo apt-get install fuse-utils

24 hours of SSH attacks against a single server, visualised on a world map using Python.

When a country stays lit up for more than 1 tick of the clock in the left hand corner it means that multiple attacks are happening from different IP addresses. An attacker is banned after;

• 1 failed root login,
• 3 failed user logins (including invalid users) and
• 3 failed system logins.

I have several servers powering syslog including it’s Raspberry Pi mirror, load balancer and email servers. All of my servers are hosted using Linode in their London data centre and have Linode’s back-up system doing both daily and weekly snapshots.

For the app and database servers I do server-side backups storing each website and it’s database in it’s own folder within /backup in case I require a quick back-up to fix something, rather than the server has died.

This is all well and good but I like having an off-site backup too and for that I use S3…

The unattended-upgrades package used on Debian is based on the one from Ubuntu. It is generally pretty safe in my opinion but I only ever enable it for security upgrades.

apt-get install unattended-upgrades apticron

APT::Periodic::Enable "1";
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
Unattended-Upgrade::Mail "**YOUR_EMAIL_HERE**";

// Automatically upgrade packages from these (origin, archive) pairs
Unattended-Upgrade::Allowed-Origins {
    "${distro_id} stable";
    "${distro_id} ${distro_codename}-security";
};

// Automatically reboot *WITHOUT CONFIRMATION* if a
 // the file /var/run/reboot-required is found after the upgrade
 Unattended-Upgrade::Automatic-Reboot "false";

*I would generally not advise using this unless you have skill at debugging why OOM has spawned and also debugging kernel panics after they happen, from logs.*

It is possible to configure your kernel to panic when OOM is spawned, which in itself is not useful but, coupled with a kernel option for auto-rebooting a system when the kernel panics it can be a very useful tool.

Think before implementing this and use at your own risk, I take zero responsibility for you using this.

sudo sysctl vm.panic_on_oom=1
sudo sysctl kernel.panic=X # X is the amount of seconds to wait before rebooting

*DO NOT FORGET TO CHANGE X*

This will inject the changes in to a system that is currently running but will be forgotten on reboot so use the lines below to save permanently.

sudo echo "vm.panic_on_oom=1" >> /etc/sysctl.conf
sudo echo "kernel.panic …

Pound is a great little load balancer, it’s fast, opensource and supports SSL termination, which is great!

sudo apt-get install pound

ListenHTTP
End

ListenHTTP
    Address 0.0.0.0 # all interfaces
    Port 80
    HeadRemove "X-Forwarded-For"
End

A simple yet effective method for protecting your mail server from spam is to use greylisting. In simple terms, when an email is received the server will temporarily reject it with a 450 response code claiming that the server is busy, the sending server should then attempt to try to deliver at a later point in time, if enough time has passed the recipient server will then accept the incoming mail and whitelist the send address for a period of time.

This is effective because most spam servers are configured not to retry the send whereas real mail servers generally will retry. This sadly does not protect against spam coming from comprised mail servers or accounts like on Hotmail.com.

sudo apt-get install postgrey