Posted on 17 Mar 2015
As part of my Raspberry Pi cluster (aka “bramble” as I found out recently) I have an nginx load balancer. Here are the steps I took to create it.
oestrich/nginx-pi
FROM oestrich/arch-pi
MAINTAINER Eric Oestrich "eric@oestrich.org"
RUN pacman -Syu --noconfirm \
&& pacman -S --noconfirm \
nginx \
&& pacman -Sc --noconfirm
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log
VOLUME ["/var/cache/nginx"]
EXPOSE 80 443
CMD ["nginx", "-g", "daemon off;"]
This starts with the base nginx container, which I took from here and adapted for arch-pi.
nginx Dockerfile
FROM oestrich/nginx-pi
MAINTAINER Eric Oestrich "eric@oestrich.org"
ADD ssl /etc/nginx/ssl
ADD sites /etc/nginx/sites
ADD nginx.conf /etc/nginx/nginx.conf
Next I have a container that will have all of the information required built into it. This is very specific to my cluster and will only be published on the private registry I have.
There are two folders and a file that gets added in. /etc/nginx/ssl contains all of the ssl certificates that will be required. /etc/nginx/sites has all of the virtual hosts that this nginx container will be serving. Lastly, /etc/nginx/nginx.conf is the base nginx configuration.
nginx.conf
user root;
worker_processes 2;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
gzip on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;
ssl_session_cache shared:SSL:10m;
# If you have a lot of virtual hosts, this is required
server_names_hash_bucket_size 64;
include /etc/nginx/sites/*;
}
This is a pretty simple nginx.conf file. I added in my ssl configuration and that’s about it.
sites/example.com
upstream website {
server docker01:5000;
}
server {
listen 80;
server_name example.com;
rewrite ^(.*) https://$host$1 permanent;
}
server {
listen 443;
server_name example.com;
ssl on;
ssl_certificate /etc/nginx/ssl/example_com.crt;
ssl_certificate_key /etc/nginx/ssl/example_com.key;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://website;
}
keepalive_timeout 10;
server_tokens off;
add_header Strict-Transport-Security "max-age=31536000";
}
This virtual host configuration file sets up a redirect for regular http to https, and has all server traffic go over https. One very important thing to remember when using nginx with proxy passing, make sure you set the X-Forwarded-Proto or rails will think you are accessing it via http and not https. It took a good amount of time to figure that out. HSTS is also set for a year to make sure clients always return via https.
Once all of this is set up you build the container and push it to a private repository. Pull the nginx container to the docker host and install this service file.
nginx.service
Description=nginx
Requires=docker.service
After=docker.service
[Service]
ExecStart=/usr/bin/docker run --rm -p 443:443 -p 80:80 --name nginx registry.example.com/nginx
Restart=always
[Install]
WantedBy=multi-user.target
This service file keeps the nginx container running. It will always restart the service to make sure nginx is running. The --rm flag is important so that after the container is stopped it will remove the named container. This allows for the container to start again with the same name when systemd starts the service next.
That’s it. You should now have an nginx docker container that will serve traffic.
Posted on 24 Feb 2015
I’ve done this a few times before, but everytime I forget how to do it and refinding the information is generally a time consuming process. Below is the bind configuration I have for a local DNS server.
/etc/bind/named.conf.local
zone "example.com" {
type master;
file "/etc/bind/db.example.com";
};
Make sure the domain inside of the first line zone "example.com" { is spell correctly. I spent a good deal of time trying to figure out why a domain was not being loaded and it was just a simple spelling error.
/etc/bind/db.starbas.es
;
; BIND data file for example.com
;
$TTL 604800
@ IN SOA ns.example.com. root.example.com. (
2015022001 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
IN A 192.168.1.2
;
@ IN NS ns.example.com.
@ IN A 192.168.1.2
ns IN A 192.168.1.2
host IN A 192.168.1.4
alias IN CNAME host.example.com.
Make sure the serial number increments each time the file changes. I use the date plus a number “01” that increments. I use two digits because I tend to edit it a lot when first figuring things out.
The line with “SOA” should have your nameserver and the an email address that has a period instead of the “@” sign. The other numbers I took from the example file.
Posted on 11 Feb 2015
For a while I’ve been interested in doing some kind of cluster with Raspberry Pis. When the second version came out, I decided to look into docker based deployment. I found some really cool stuff in CoreOS, but I doubt we’ll get that any time soon. For now I will make due with a plain Arch install running docker.
Docker was easy to install on Arch, but I ran into a problem fairly quickly. I couldn’t find any images that were built for the ARM platform on the docker hub. This seemed like a good chance to try out building a docker image from scratch.
oestrich/arch-pi Dockerfile
The arch-pi image is the latest raspberry pi Arch tar.gz that is extracted into the same folder as the Dockerfile. I use arch-chroot on that folder to be able to remove a few packages first. This helps shrink the image considerably. I removed most of the firmware and kernels. Since docker uses the host kernel, this is OK to do.
oestrich/base-pi Dockerfile
FROM oestrich/arch-pi
MAINTAINER Eric Oestrich <eric@oestrich.org>
RUN pacman -Syu --noconfirm
RUN pacman -S --noconfirm \
base-devel \
libffi \
libyaml \
openssl \
zlib \
git
RUN git clone https://github.com/sstephenson/ruby-build.git
RUN cd ruby-build && ./install.sh
RUN ruby-build 2.2.0 /opt/ruby
ENV PATH $PATH:/opt/ruby/bin
This docker image installs everything needed to get ruby installed. Note that this will take a good hour or so to build. I did discover that the Raspberry Pi was quicker than the old t1.micro instance on AWS, go Raspberry Pi.
oestrich/base-pi-web Dockerfile
FROM oestrich/base-pi
MAINTAINER Eric Oestrich <eric@oestrich.org>
RUN pacman -Syu --noconfirm
RUN pacman -S --noconfirm \
libxml2 \
libxslt \
imagemagick \
openssl \
postgresql \
python2
RUN gem install bundler
RUN gem install foreman
RUN gem install libv8 --version 3.16.14.7
RUN gem install nokogiri --version 1.6.5
This docker image gets the base environment set for a rails app. Whatever system libraries are required and some base gems like bundler and foreman. I also installed libv8 and nokogiri because together they take about 40 minutes to install. I used the --version flag to install the exact version that bundler will install. This lets bundler simple use the installed gem, saving 40 minutes each time the Gemfile changes.
project/web Dockerfile
FROM oestrich/base-pi-web
MAINTAINER Eric Oestrich <eric@oestrich.org>
RUN mkdir -p /apps/project
ADD Gemfile* /apps/project/
WORKDIR /apps/project
RUN bundle -j4 --without development test
ADD . /apps/project
ADD .env.production /apps/project/.env
RUN . /apps/project/.env && bundle exec rake assets:precompile
CMD ["foreman", "start", "web"]
This last docker image installs a rails app. It copies over the Gemfile and Gemfile.lock into the container first to install gems. This way you only rebundle if your Gemfile changes, saving time on each deploy. I also want to point out the -j4 option on the bundle. The Raspberry Pi 2 is quad core, so let’s use all the power available. Assets are compiled and then ends with a CMD of foreman start web.
An alternative to CMD is to use ENTRYPOINT:
ENTRYPOINT ["foreman", "start"]
This will let you use different foreman apps without having to build a separate image for each one.
docker run -t -i -p 5000:5000 project/web worker
Posted on 28 Jan 2015
This is a small script I wrote to backup a postgres database to S3 via cron. It runs nightly and saves 30 days of databases. The README has a good bit of information about what is required to set it up. I will copy some of it here though.
.pgpass
A .pgpass file is required for this script as a password cannot be passed in. The format is listed below. It should be only accessible to the user running the script and will be located in the home folder. See the postgress wiki for more information.
hostname:port:database:username:password
Cron
This is the command I used for cron, it sets up the environment for rbenv.
0 0 * * * /bin/bash -c 'PATH=/opt/rbenv/shims:/opt/rbenv/bin:$PATH RBENV_ROOT=/opt/rbenv ruby /home/deploy/pg-to-s3/backup.rb'
backup.rb
#!/usr/bin/env ruby
require 'time'
require 'aws-sdk'
require 'fileutils'
# .pgpass file required, it is in the following format
# hostname:port:database:username:password
pg_user = ENV["POSTGRES_USERNAME"] || "postgres"
pg_host = ENV["POSTGRES_HOST"] || "localhost"
pg_port = ENV["POSTGRES_PORT"] || "5432"
pg_database = ENV["POSTGRES_DATABASE"]
bucket_name = ENV["BACKUP_BUCKET_NAME"]
project_name = ENV["PROJECT_NAME"]
# backup pg
time = Time.now.strftime("%Y-%m-%d")
filename = "backup.#{Time.now.to_i}.#{time}.sql.dump"
`pg_dump -Fc --username=#{pg_user} --no-password --host #{pg_host} --port #{pg_port} #{pg_database} > #{filename}`
# verify file exists and file size is > 0 bytes
unless File.exists?(filename) && File.new(filename).size > 0
raise "Database was not backed up"
end
s3 = AWS.s3
bucket = s3.buckets[bucket_name]
object = bucket.objects["#{project_name}/#{filename}"]
object.write(Pathname.new(filename), {
:acl => :private,
})
if object.exists?
FileUtils.rm(filename)
end
DAYS_30 = 30 * 24 * 60 * 60
objects = bucket.objects.select do |object|
time = Time.at(object.key.split("/").last.split(".")[1].to_i)
time < Time.now - DAYS_30
end
objects.each do |object|
object.delete
end
Posted on 15 Jan 2015
I reconfigured SSL for nginx this morning and wanted to post the config that got me an “A” on SSL Labs. I will keep this post updated as I change it over time. Hopefully this will be helpful to someone else who is setting nginx up with SSL.
/etc/nginx/config.d/ssl.conf
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;
ssl_session_cache shared:SSL:10m;
/etc/nginx/sites-enabled/example.com
server {
listen 443;
ssl on;
ssl_certificate /path/to/crt/example_com.crt;
ssl_certificate_key /path/to/key/example_com.key;
# ...
}
For redirecting all traffic on non-SSL to SSL:
server {
listen 80 default deferred;
server_name example.com;
rewrite ^(.*) https://$host$1 permanent;
}
For creating the crt file, the server’s crt should be first followed by a chain to the root CA. I got this flipped a few times and it took a bit to realize they were in the wrong order. It should be noted that nginx -t will let you know of this problem.
