This article is a continuation of retrying dynamically configured upstreams that gives an example of how you can configure OpenResty to update your upstream backend servers dynamically with DNS.

Breaking down init_worker_by_lua_block

init_worker_by_lua_block can be used to make an nginx worker do some fun stuff. In this instance we’re going to use it in conjunction with ngx.timer.every and the resty.dns.resolver.

Here is the full example of my init_worker_by_lua_block.

init_worker_by_lua_block {
    _backend_servers = {}

    local function update_dns()
        -- Set up the resolver
        local resolver = require "resty.dns.resolver"
        local r, err = resolver:new{
            nameservers = {"1.1.1.1", {"1.0.0.1", 53} },  -- Cloudflare
            retrans = 5,  -- 5 retransmissions on receive timeout
            timeout = 1000,  -- 1 sec
        }
        if not r then
            ngx.log(ngx.ERR, "failed to instantiate resolver: ", err)
            return
        end

        -- Pull DNS records
        -- Use a hardcoded domain to make this example easier
        local answers, err, tries = r:query("kura.gg …

Preamble

OpenResty is a modified version of nginx with LuaJIT compiled in and many nginx options that can be controlled or modified via Lua. It is very commonly used in content delivery networks for it’s configurability.

As such, we use OpenResty and one of the features we use is the ability to dynamically modify upstream backends. To achieve this we use some logic within OpenResty to update upstreams based on DNS records. This means we can pull upstreams in and out of service via DNS records and have OpenResty update it’s upstream proxy passing configuration without needing to push configs out to hundreds of servers and reload daemons.

The logic behind how we update the upstream backends is beyond the scope of this post, so let’s just say we have a table of upstream servers.

local upstream_servers = {
    "10.0.0.1:443",
    "10.0.0.2:443 …

I have built and released nginx 1.5.10 with SPDY 3.1. Like the nginx 1.5.9 release , this release comes with ngx_pagespeed 1.7.30.3-beta and is available on apt.kura.io or as downloads below.

MD5

9fe2e5273cc195161268f7d85261c4e2  nginx_1.5.10_all.deb
edc55aa4866036eade02cd49957a9126  nginx-common_1.5.10_all.deb
0361cdb3d00ac6e65c5e9d6167ba0d36  nginx-doc_1.5.10_all.deb
833264c08fc6212f55ae37c26bd5cbc5  nginx-light_1.5.10_amd64.deb
c5c1ffa0dd93673ac4a859a11d1b3b50  nginx-full_1.5.10_amd64.deb
245d7628f143a6116ceb30c707264737  nginx-extras_1.5.10_amd64.deb
dc404a346db86006672b5a6f8b016402  nginx-naxsi_1.5.10_amd64.deb
ad3b7cf166752c2a8017bba8f6810496  nginx-naxsi-ui_1.5.10_all.deb
cdb47100b4fef09bb8a8e414cd48554e  nginx-light-dbg_1.5.10_amd64.deb
1ee2067aef2e1fcbc559dfdf9b8269ad  nginx-full-dbg_1.5.10_amd64.deb
9f528d80802dd6a78d85b8558e65f650  nginx-extras-dbg_1.5.10_amd64.deb
324dbf6afdff615d7c2bbe367f73bd1f  nginx-naxsi-dbg_1.5.10_amd64 …

I have built and released nginx 1.5.9 with ngx_pagespeed module 1.7.30.3-beta and published them on apt.kura.io.

MD5

e3595519df9865941f0bd5c2c28bba18  nginx_1.5.9_all.deb
b2f3e4dcded2ce419175be8f4329b81e  nginx-common_1.5.9_all.deb
379ea6ad2805f3584609617deb5d1db9  nginx-doc_1.5.9_all.deb
140247350a651b24bde7278ce1f18148  nginx-light_1.5.9_amd64.deb
0a28a4965f00a5e739187aa81a16af3f  nginx-full_1.5.9_amd64.deb
e54ab6b670cfe7d56502ef08b6e2656f  nginx-extras_1.5.9_amd64.deb
9bb9b524c523a69f9f263eefbe5f5783  nginx-naxsi_1.5.9_amd64.deb
7b8e254515d3b6f90b8e55c720d314b8  nginx-naxsi-ui_1.5.9_all.deb
389b8b53360695d5fbbff183a3e94c4a  nginx-light-dbg_1.5.9_amd64.deb
b8b44d1519cb59761984bc06159aee85  nginx-full-dbg_1.5.9_amd64.deb
c8139d0abb8a04b22507342a80e6f5ea  nginx-naxsi-dbg_1.5.9_amd64.deb
76466c2c427e7263b3629b413ebd49a6  nginx-extras-dbg_1.5.9_amd64.deb

SHA1

cc1e71b89c30de80083e55acb4b4cebc0f5f1fd7  nginx_1.5.9_all.deb
07d2bb7bdb038f5491b8707a56dafbf0a10a1b74  nginx-common_1.5.9_all.deb
ba0921fc31ae539fba175ab4392b3bc691593047  nginx-doc_1.5.9_all …

I have previously released haproxy1.5-dev19 with SSL & SPDY support enabled and nginx 1.4.1 with SPDY support and pagespeed, although I do not remember which version of pagespeed.

I have received a few messages asking me for the latest version of haproxy, nginx and pagespeed so I decided to finally build and release them.

Requirements

haproxy requires openssl-1.0.1d or higher.

On a standard Debian 7 install you should have openssl-1.0.1e-2, you can find which version you have by running

dpkg -l openssl

This should return something similar to

ii  openssl        1.0.1e-2        amd64        Secure Socket Layer (SSL) binary and related cryptographic tools

Build notes

Builds were done on Debian 7 AMD64, I will not be providing 32bit versions as this is mainly for my own usage and amusement.

haproxy

This haproxy build is compiled against openssl, providing the npn module, allowing for haproxy …

I wrote an article last week explaining that I had changed my blog and built my own nginx packages with SPDY built in.

I decided I would take things a little further and poke around with haproxy some more. The initial plan was to compile the latest dev source of haproxy with SSL termination enabled.

In doing so I realised I would lose SPDY support, which upset me a little. After some digging I found that the 1.5-dev branch of haproxy supports npn and thus can handle SPDY.

I tweaked my builds a little more and managed to get haproxy running as an SSL terminating load balancer, with SPDY connections being sent off to my nginx servers with SPDY enabled and all other non-SPDY connections were passed on to an nginx virtual host with SPDY disabled.

Requirements

I have released my haproxy build as a debian file below …

I decided to rebuild syslog.tv as pure HTML using RST and Pelican and rebrand it as kura.io.

In doing so I decided I would go all out and use SPDY and ngx_pagespeed (mod_pagespeed) for fun to see exactly what I could do.

Sadly no version of nginx has been officially released with SPDY or ngx_pagespeed enabled, you can compile nginx from source to enable SPDY so I thought I would go ahead and do it, releasing some Debian packages in the process.

After compiling nginx from the source package available at the Ubuntu PPA, I decided I would go further and compile in ngx_pagespeed.

Installing

I have released the 4 required debian packages below (please note they are only available for amd64);

FILE GPG MD5 SHA1
nginx_1.4.1_all.deb owGMl1NwLw… 42f790a1f6… f4495055e9…
nginx-common_1.4.1_all.deb owGMt1NwJU… ca4ec5688d… 633bfc2eaa…
nginx-full_1.4.1_amd64.deb owG09ndUk1… 4776dc6c7f… 0e4e124acc…
nginx-doc_1 …

The point

The whole point of this is to get as much load off of Apache as possible to keep the server running nice and smoothly.

Configuration

The configuration below will mean that nginx will serve basically everything;

  • static files
  • uploaded files and
  • cached content

simply replace the VARIABLES below and everything should be good to go, if copy-pasting from below isn’t working properly you can download a full copy from here.

server {
  listen 80;
  server_name **DOMAIN_HERE**;
  access_log /var/log/nginx/access.**DOMAIN_HERE**.log;

  gzip on;
  gzip_disable msie6; # disable gzip for IE6
  gzip_static on;
  gzip_comp_level 9; # highest level of compression
  gzip_proxied any;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;

  proxy_redirect off;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_pass_header Set-Cookie;root **/PATH/TO/WORDPRESS**;

  # default location, used for the basic proxying
  location / {
    # if we're requesting a file and …

Recently I started using Pound as a load balancer to a cluster of nginx servers and found my access logs were filled with the IP address of the load balancer. I did some digging and found the correct way to “fix” this.

First thing you need to do is make sure you remove X-Forwarded-For from Pound

ListenHTTP
    # ... snip ...
    # ... snip ...
    HeadRemove "X-Forwarded-For"
End

Once this is done, reload Pound.

Next you need nginx compiled with realip module - http://wiki.nginx.org/NginxHttpRealIpModule

On Ubuntu/Debian servers this module comes by default, otherwise you may have to compile it in yourself using the following option:

--with-http_realip_module

Once this is all done modify your nginx vhosts and add the following 2 lines

set_real_ip_from [IP];
real_ip_header X-Forwarded-For;

Where [IP] is the IP address of your load balancer.

To configure this to work with Apache you need the mod_rpaf module.

I was recently tasked with adding Google tracking cookies to our nginx logging for a couple of sites. It was so it could be pushed through a log processor.

It turned out too be a little trickier than it would have been with Apache, but the process itself is still quite simple.

Open up the server definition you wish to add it to and add a custom log format like below

log_format g-a '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '"__utma=$cookie___utma;__utmb=$cookie___utmb;__utmc=$cookie___utmc;__utmv=$cookie___utmv;__utmz=$cookie_umtz"';

This log format can then be added to your access log like below:

access_log /var/log/nginx/access.example.com.log g-a;

Reload nginx

sudo /etc/init.d/nginx reload

If all goes well, you should see Google Analytics appearing in your access logs like below

1.1.1.1 - - [05/Jun/2011:20:35:50 +0100] "GET / HTTP …

Lately I’ve been doing a lot of work with Varnish, this includes testing it within a load balanced environment, putting it behind nginx, putting it in front of Solr, the list goes on.

This blog post will hopefully give you an insight in to a simple way of combining nginx, Varnish and Apache to create a powerful Wordpress environment that can really take a hammering.

I’m going to assume you already have Apache and nginx working together, if not I suggest you read my other articles on these subjects to learn how to combine them.

Installing Varnish

sudo apt-get install varnish

Configuring Apache

I suggest binding Apache to port 81, this is easy to change, open the following file in your favourite editor.

/etc/apache2/ports.conf

Change the Listen and NameVirtualHost lines to:

Listen 81
NameVirtualHost *:81

This will mean you need to go and change all …

Configuration changes

I made some modifications to my nginx configuration this weekend to improve performance and clear up some bugs.

upstream backend {
    server 127.0.0.1:81 fail_timeout=120s;
}

server {
    listen 80;
    server_name syslog.tv;

    access_log /var/log/nginx/access.syslog.tv.log;

    gzip on;
    gzip_disable msie6;
    gzip_static on;
    gzip_comp_level 9;
    gzip_proxied any;
    gzip_types text/plain text/css application/x-javascript text/xml
    application/xml application/xml+rss text/javascript;

   location / {
        root /var/www/syslog.tv;

        set $wordpress_logged_in "";
        set $comment_author_email "";
        set $comment_author "";

        if ($http_cookie ~* "wordpress_logged_in_[^=]*=([^%]+)%7C") {
             set $wordpress_logged_in wordpress_logged_in_$1;
        }

        if ($http_cookie ~* "comment_author_email_[^=]*=([^;]+)(;|$)") {
            set $comment_author_email comment_author_email_$1;
        }

        if ($http_cookie ~* "comment_author_[^=]*=([^;]+)(;|$)") {
            set $comment_author comment_author_$1;
        }

        set $my_cache_key "$scheme://$host$uri$is_args$args$wordpress_logged_in$comment_author_email$comment_author";

        client_max_body_size 8m;

        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass_header Set-Cookie;
        proxy_cache cache;
        proxy_cache_key $my_cache_key;
        proxy_cache_valid 200 302 60m;
        proxy_cache_valid 404 1m;
        proxy_pass http://backend;
    }

    location ~* .(jpg|png|gif|jpeg|js|css …

The title of this post is a bit stupid, but I honestly couldn’t think of any other way to write it…

When compiling nginx by hand, by default make install will push the binaries out to /usr/local/nginx and it doesn’t come with a start/stop script, understandably because it doesn’t know which OS it is going to be installed on etc etc.

Recently I was tasked with building nginx to an old Red Hat Enterprise Live 4 server with no yum installation, no nginx package in up2date and not being able to find an RPM that’s link wasn’t dead.

I’ve always felt that, being a Debian user, people think of me as only being able to use apt-get and if I’m feeling especially adventurous dpkg - to install applications. Some people know that I build .deb files in my spare time, but …

This is a very quick blog to show you how to show a users IP address in your Apache access logs when the site in question is being reverse proxied to Apache through nginx.

You need the rpaf module for Apache, on Debian and Ubuntu this is simple to install

sudo apt-get install libapache2-mod-rpaf
sudo a2enmod rpaf
sudo /etc/init.d/apache2 restart

This set of commands will do the following;

  1. Update apt package list
  2. Install libapache2-mod-rpaf
  3. Enable mod-rpaf
  4. Gracefully restart Apache (doesn’t kill connections)

Once installed you simple need to be sure to pass the correct headers through, so open up one of your nginx site configuration files and add the following within the server definition.

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

So you should have something that looks like this, but without the “… snip …”

server {
    # ...snip...
    location / {
        # ...snip...
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # ...snip...
    }
}

Today I ran in to something I’d never seen before when configuring nginx.

I ran the nginx config test, as I usually do before I restart it.

nginx -t

But, the response I got was interesting

2010/03/18 21:16:09 [emerg] 12299#0: could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32
2010/03/18 21:16:09 [emerg] 12299#0: the configuration file /etc/nginx/nginx.conf test failed

I found that one of the domain names I was using was over 32 characters in length, nginx’s default max length.

Thankfully the fix was simple.

http {
    # ...snip...
    server_names_hash_bucket_size 64;
    # ...snip...
}

This is yet another follow up to post to several previous posts about using nginx as a reverse proxy with caching. It is actually a direct addition to my post from a week or so ago which outlined how to actually using nginx’s proxy caching feature which can be read here — /2010/02/07/nginx-proxy_cache-and-explained-benchmarked/.

Even more changes?

Yes, even more changes, these are basic changes that are there to improve the caching capabilities and also implement load balancing.

Cache changes

The first set of changes are in the main nginx configuration file

/etc/nginx/nginx.conf

These changes basically just change the proxy_cache key

proxy_cache_path /var/www/nginx_cache levels=1:2 keys_zone=cache:8m max_size=1000m inactive=600m;
proxy_temp_path /tmp/nginx;
proxy_cache_key "$scheme://$host$request_uri";

I’ve decided to put the temporary caches file in to an nginx specific directory, just to separate them from other cache files …

The beginning

Where to begin? nginx would be a good start I suppose. It’s far easier and makes much for sense for you to actually read about nginx from it’s own website - http://nginx.org/en/ - but just to give a simple explanation too; `nginx is king of static content HTTP servers.`

Anyone that has dealt with Apache on medium to high traffic websites will know that Apache is bit of a `wheezy, old geezer` when it comes to content serving using it’s mpm-worker (threaded). Very often high traffic will cause server load to go through the roof but for serving dynamic content, there really is no better HTTP server than Apache, so this leaves us in a bit of a predicament; a high powered website with dynamic content and lots of static files like JS, CSS and imagery, what do we do?!

In this example `dynamic …

The problem

So lets get to the problem first. I have several lightly to medium loaded sites running on some virtual servers, they servers themselves are highly configured to run beautifully on our host environments, very, very RAM intensive but low disk I/O and low CPU usage.

As mentioned, the sites are relatively low loaded, they’ll generally hang around at between 3,000-5,000 unique hits a day and are run through Apache using PHP, various PHP modules and MySQL, a simple generic LAMP environment, yet customised to suit it’s surroundings and host.

The sites themselves run fine on that setup, no issues at all on normal days, but on set days of the week these sites can double in unique hits or even more than double, with KeepAlive enabled and a KeepAliveTimeout set low Apache has problems handling this kind of load (I should point out …

This is a quick follow up to a previous post about getting this blog running on nginx with a reverse proxy to Apache 2.

It seems the issue stems from 3 mods I had installed and enabled

  1. mod-spamhaus
  2. mod-evasive and
  3. mod-security

The 3, when running together are a fantastic way to strengthen any web server from attack, be it DOS, injection, XLS etc. I’ve sworn by all 3 of them for years now and I thought I had them cracked for security:performance ratio, when it comes to reverse proxying requests from nginx to Apache 2 where WordPress is concerned, apparently I was very wrong.

The issue wasn’t so bad when the cache was full, but seeing as my cache is only alive for an hour that leaves an open point for the cache to be recreated when a user views the page. This in itself isn’t …

This is a rather old article, for more up-to-date information please see;

  1. /2010/02/07/nginx-proxy_cache-and-explained-benchmarked/
  2. /2010/02/14/more-nginx-proxy_cache-optimizations-and-nginx-load-balancing/

I’ve started collecting a few blogs on my servers now and figured from this one on I would consolidate it in to one workable, usable location. Removing my need to update multiple plugins, themes and WordPress itself, over and over.

This time round I thought I’d do it properly, and properly in my book is as complicated and “awesome” as it can possibly be, without growing legs and running off to stomp a city.

Love

I’ve fallen in-love with nginx (http://nginx.org/) over the last 6 months or so, I’d been an avid user of LighTTPD for a very long time before but started to look in to nginx mid year as a replacement. I learned that at my new job they used nginx for …