Posted on 15 Nov 2015
For a project at work we use Elasticsearch. It was hosted inside of Kubernetes so we didn’t have a nice way to access the production instance from our browsers. I came up with this small proxy rack app that fowards the request through the Rails app. I have it authenticated via devise so only admins can view the actual instance.
The Rack App
class ElasticsearchForwarder
def self.call(env)
new.call(env)
end
def call(env)
request = Rack::Request.new(env)
case request.request_method
when "GET", "POST", "OPTIONS", "HEAD", "PUT", "DELETE"
response = client.send(request.request_method.downcase, request.path_info) do |f|
f.params = request.params
f.body = request.body.read
end
[response.status, headers(response.headers), [response.body]]
else
[405, {}, []]
end
end
private
def headers(headers)
cors_headers.merge(headers)
end
def cors_headers
{
"Access-Control-Allow-Origin" => "*",
"Access-Control-Allow-Methods" => "OPTIONS, HEAD, GET, POST, PUT, DELETE",
"Access-Control-Allow-Headers" => "X-Requested-With, Content-Type, Content-Length",
"Access-Control-Max-Age" => "1728000",
}
end
def client
@client ||= Faraday.new(AppContainer.elasticsearch_url) do |conn|
conn.adapter :net_http
end
end
end
Make sure to replace AppContainer.elasticsearch_url with your own elasticsearch instance information. Otherwise it is a pretty simple rack app that just uses Faraday to forward your request on to elasticsearch. I set up CORS headers to allow anything, but I found with using the devise authentication it required that you were on the same domain anyways.
Mounting the App
authenticate :user, lambda { |user| user.admin? } do
mount ElasticsearchForwarder => "/elasticsearch"
end
Here we mount the app inside of a devise authenticate block, making sure the user is an admin and logged in.
Future Improvements
I would like to make this less reliant on the devise cookie. We had to host our own ElasticHQ in order for devise to let requests through. It’s not that bad, but would have been cleaner to not require hosting our own.
Posted on 13 Oct 2015
We’ve so far seen how to put postgres inside of Kubernetes, here is how I was able to put a Rails app inside of Kubernetes. This is my deploy script that builds the docker container, and then pushes it out into the cluster. I’ll show snippets first then the entire script at the end.
Build Docker Image
build_dir=`mktemp -d`
git archive master | tar -x -C $build_dir
echo $build_dir
cd $build_dir
docker build -t $docker_image .
docker tag -f $docker_image $docker_tag
gcloud docker push $docker_tag
This creates a temp directory and creates a clean copy of the repo to build. I don’t want to build in the working directory because it might contain files you don’t want to build into the image. This makes sure it only ever contains what the master branch has.
It also pushes the image up tagged as the git sha. This lets us make sure we’re always deploying the correct code, seen when we update the replication controllers.
Optionally migrate
docker tag -f $docker_image "$docker_image:migration"
gcloud docker push "$docker_image:migration"
kubectl create -f config/kubernetes/migration-pod.yml
echo Waiting for migration pod to start...
sleep 10
while [ true ]; do
result=`kubectl get pods migration | grep ExitCode`
if [[ $result == *"ExitCode:0"* ]]; then
break
else
sleep 1
fi
done
kubectl delete pod migration
This section is done with a switch -m. It pushes to a special tag named migration and then creates a pod that will run rake db:migrate. It pushes to the special tag so that the migration pod can specify that branch and always have the latest code. Below is the yaml that is used to create the migration pod.
config/kubernetes/migration-pod.yml
apiVersion: v1
kind: Pod
metadata:
name: migration
labels:
name: migration
spec:
restartPolicy: Never
volumes:
- name: environment-variables
secret:
secretName: app-env
containers:
- name: migration
image: gcr.io/project/app:migration
imagePullPolicy: Always
args: ["bundle", "exec", "rake", "db:migrate"]
volumeMounts:
- name: environment-variables
readOnly: true
mountPath: /etc/env
Update Replication Controllers
kubectl rolling-update app-web --image=$docker_tag --update-period="10s"
kubectl rolling-update app-worker --image=$docker_tag --update-period="10s"
This updates the replication controller’s image. We set the update period to 10 seconds to speed the process up. The default time is 60 seconds. Kubernetes will take out one pod at a time and update to the new docker image. Once it completes the rolling update all pods will be running the newly deployed code.
Update to loading kubernetes secrets
I had to update how I was loading secrets from last time. I replaced foreman.sh with env.sh and changed the docker CMD.
env.sh
#!/bin/bash
ruby env.rb .env /etc/env && source .env && $*
$* is worth pointing out. Bash uses it as a variable for all arguments. In the case of the docker file below it will expand out to:
ruby env.rb .env /etc/env && source .env && foreman start web
Dockerfile
ENTRYPOINT ["./env.sh"]
CMD ["foreman", "start", "web"]
Full Script
#!/bin/bash
set -e
# Get options
migrate=''
deploy=''
while getopts 'md' flag; do
case "${flag}" in
m) migrate='true' ;;
d) deploy='true' ;;
*) error "Unexpected option ${flag}" ;;
esac
done
sha=`git rev-parse HEAD`
docker_image="gcr.io/project/app"
docker_tag="$docker_image:$sha"
# Create an clean directory to build
build_dir=`mktemp -d`
current_dir=`pwd`
git archive master | tar -x -C $build_dir
echo $build_dir
cd $build_dir
# Build docker image and push
docker build -t $docker_image .
docker tag -f $docker_image $docker_tag
gcloud docker push $docker_tag
# Migrate if required
if [ $migrate ]; then
docker tag -f $docker_image "$docker_image:migration"
gcloud docker push "$docker_image:migration"
kubectl create -f config/kubernetes/migration-pod.yml
echo Waiting for migration pod to start...
sleep 10
while [ true ]; do
result=`kubectl get pods migration | grep ExitCode`
if [[ $result == *"ExitCode:0"* ]]; then
break
else
sleep 1
fi
done
kubectl delete pod migration
fi
if [ $deploy ]; then
# Update images on kubernetes
kubectl rolling-update app-web --image=$docker_tag --update-period="10s"
kubectl rolling-update app-worker --image=$docker_tag --update-period="10s"
fi
# clean up build folder
cd $current_dir
rm -r $build_dir
Posted on 29 Sep 2015
For work I’m starting on a new API and I decided to use Collection+JSON after using it at REST Fest. I enjoyed making a ruby client for it, so I figured I would see how it works further than a weekend.
I like to use ActiveModel::Serializers and this DSL has made generating Collection+JSON extremely simple. We are using ActiveModel::Serializers master branch on github. Not the gem currently on Rubygems so you may need to tweak some things to get it working on the older version.
Using The DSL
app/controllers/orders_controller.rb
class OrdersController < ApplicationController
def index
render :json => orders, :serializer => OrdersSerializer, :meta => {
:href => orders_url(params.slice(:search, :page, :per)),
:up => root_url,
:search => params[:search] || "",
}
end
def show
render :json => [order], :serializer => OrdersSerializer, :meta => {
:href => order_url(order.first),
:up => orders_url,
:search => "",
}
end
end
Here is the controller. Of note is show displaying an array of orders, since there is no single element in Collection+JSON. This was a lot nicer than I expected, it flowed very well.
app/serializers/order_serializer.rb
class OrderSerializer < ActiveModel::Serialier
include CollectionJson
attributes :name, :ordered_at
href do
meta(:href)
end
end
Here is the OrderSerializer. It includes CollectionJson which adds the DSL. The important feature is href. I found that passing in the href from the controller is best. I pull the href out from the meta hash. I’m not super sure this is the right way to go, but it’s worked well so far.
app/serializers/orders_serializer.rb
class OrdersSerializer < ActiveModel::Serializer
include CollectionJson
link "up" do
meta(:up)
end
query "http://example.com/rels/order-search" do
href orders_url
data :search, :value => meta(:search), :prompt => "Order Name"
end
template do
data "name", :value => "", :prompt => "Order Name"
end
href do
meta(:href)
end
private
def include_template?
true
end
end
Here we see some more of the DSL. Generating links, queries, and the template. You can see more about these in the code below.
require 'collection_json'
ActiveModel::Serializer.config.adapter = :collection_json
class ActiveModel::Serializer::ArraySerializer
include Rails.application.routes.url_helpers
delegate :default_url_options, :to => "ActionController::Base"
end
class ActiveModel::Serializer
include Rails.application.routes.url_helpers
delegate :default_url_options, :to => "ActionController::Base"
end
Here we set the adapter to collection_json and include routes so the serializers can be hypermedia driven. I’m not super stoked about how many times we need to include routes, but I couldn’t figure out a way around it.
The Details
Here are the two files that make the above happen:
app/serializers/collection_json.rb
module CollectionJson
extend ActiveSupport::Concern
module RoutesHelpers
include Rails.application.routes.url_helpers
delegate :default_url_options, :to => "ActionController::Base"
end
module CollectionObject
def encoding(encoding)
@encoding = encoding
end
def data(name, options = {})
@data << options.merge({
:name => name,
})
end
def compile(metadata)
reset_query
@meta = metadata
instance_eval(&@block)
generate_json
end
private
def meta(key)
@meta && @meta[key]
end
end
class Query
include CollectionObject
include RoutesHelpers
def initialize(rel, &block)
@rel = rel
@block = block
end
def href(href)
@href = href
end
private
def generate_json
json = {
:rel => @rel,
:href => CGI.unescape(@href),
:data => @data,
}
json[:encoding] = @encoding if @encoding
json
end
def reset_query
@data = []
@encoding = nil
@href = nil
end
end
class Template
include CollectionObject
include RoutesHelpers
def initialize(&block)
@block = block
end
private
def generate_json
json = { :data => @data }
json[:encoding] = @encoding if @encoding
json
end
def reset_query
@data = []
@encoding = nil
end
end
class_methods do
include RoutesHelpers
def link(rel, &block)
@links ||= []
@links << {
:rel => rel,
:href => block,
}
end
def links
@links
end
def query(rel, &block)
@queries ||= []
@queries << Query.new(rel, &block)
end
def queries
@queries
end
def template(&block)
return @template unless block_given?
@template = Template.new(&block)
end
def href(&block)
define_method(:href) do
instance_eval(&block)
end
end
end
def links
return unless self.class.links.present?
self.class.links.map do |link|
link.merge({
:href => instance_eval(&link[:href]),
})
end.select do |link|
link[:href].present?
end
end
def queries
return unless self.class.queries.present?
self.class.queries.map do |query|
query.compile(@meta)
end
end
def template
if respond_to?(:include_template?, true) && include_template?
self.class.template.compile(@meta) if self.class.template.present?
end
end
def meta(key)
@meta && @meta[key]
end
end
lib/collection_json.rb
module ActiveModel
class Serializer
module Adapter
class CollectionJson < Base
def initialize(serializer, options = {})
super
@hash = {}
end
def serializable_hash(options = {})
@hash[:version] = "1.0"
@hash[:href] = serializer.href
@hash[:links] = serializer.links if serializer.links.present?
@hash[:queries] = serializer.queries if serializer.queries.present?
@hash[:template] = serializer.template if serializer.template.present?
add_items
{
collection: @hash,
}
end
def as_json(options = nil)
serializable_hash(options)
end
private
def add_items
if serializer.respond_to?(:each)
serializer.map do |serializer|
add_item(serializer)
end
else
add_item(serializer) if serializer.object
end
end
def add_item(serializer)
@hash[:items] ||= []
item = {}
item[:href] = serializer.href
item[:links] = serializer.links if serializer.links.present?
serializer.attributes.each do |key, value|
next unless value.present?
item[:data] ||= []
data = {
name: key.to_s,
value: value,
}
item[:data] << data
end if serializer.attributes.present?
@hash[:items] << item
end
end
end
end
end
Posted on 15 Sep 2015
When hosting a Rails application, I really like to push as much configuration into the environment as possible. This is why I like dotenv and used it pretty extensively in the bramble. With the bramble I stuck the .env file inside of the container. Moving to Kubernetes I didn’t want to continue with this.
Kubernetes has something called a secret that stores key/values and you mount them as a volume in the pod. This works great except I want to continue using dotenv so I don’t write the entire application specifically for Kubernetes.
To manage this I wrote a simple script that converts a secrets folder into an environment file. I run this little script before starting foreman.
env.rb
env = {}
Dir["#{ARGV[1]}/*"].each do |file|
key = file.split("/").last
key = key.gsub("-", "_").upcase
env[key] = File.read(file).strip
end
File.open(ARGV[0], "w") do |file|
env.each do |key, value|
file.puts(%{export #{key}="#{value}"})
end
end
ruby env.rb .env /etc/etc
The file looks at the folder passed in as the second argument. Secret key values have to be hyphenated so we convert hyphens to underscores and uppercase everything. Then write the values to the first file in a format that dotenv accepts.
Secrets
To create the secret you upload the following YAML to kubernetes. Once the secret is created you mount them in a pod.
secrets.yml
apiVersion: "v1"
kind: "Secret"
metadata:
name: app-env
data:
rails-env: cHJvZHVjdGlvbg==
rack-env: cHJvZHVjdGlvbg==
rails-serve-static-files: dHJ1ZQ==
kubectl create -f secrets.yml
This file creates the secret app-env. Values are base 64 encoded.
app-controller.yml
apiVersion: v1
kind: ReplicationController
metadata:
name: app
labels:
name: app
spec:
replicas: 3
selector:
name: app
template:
metadata:
labels:
name: app
spec:
volumes:
- name: environment-variables
secret:
secretName: app-env
containers:
- name: teashop
image: smartlogic/app
imagePullPolicy: Always
ports:
- containerPort: 5000
volumeMounts:
- name: environment-variables
readOnly: true
mountPath: /etc/env
This replication controller mounts the secret into /etc/env.
Dockerfile
To use the new script in your Dockerfile change the entrypoint to:
ENTRYPOINT ["./foreman.sh"]
With foreman.sh as:
#!/bin/bash
ruby env.rb .env /etc/env && foreman start ${BASH_ARGV[0]}
Improvements
This isn’t the nicest set up but it is a good stop gap until we can figure something else out. Some places to consider going forward are Keywhiz and etcd. Either of those should be a much better configuration management.
Posted on 31 Aug 2015
I have been playing around with Kubernetes in the last week or so and have really started liking it. One of the main things I wanted to get running was Postgres inside of the cluster. This is particularly challenging because postgres requires a volume that sticks around with the docker container. If it doesn’t you will lose your data.
In researching this I found out kubernetes has a persistence manager that lets you mount Google Compute disks. This also works for AWS EBS volumes if your cluster is hosted over there.
Create the disk
gcloud compute disks create pg-data-disk --size 50GB
Make sure this will be in the same zone as the cluster.
Next attach the disk to an already running instance, possibly a new instance. We only need to temporarily attach the disk so we can format it.
gcloud compute instances attach-disk pg-disk-formatter --disk pg-data-disk
After the disk is attached, ssh into the instance and run the following commands. This will mount and then format the disk with ext4. Then we unmount the drive.
sudo /usr/share/google/safe_format_and_mount -m "mkfs.ext4 -F" /dev/sdb /media/pg-data/
sudo umount /media/pg-data/
gcloud compute instances detach-disk pg-disk-formatter --disk pg-data-disk
Set up Kubernetes
With all of this complete we can start on the kubernetes side. Create the following four files.
postgres-persistence.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: pg-data-disk
labels:
name: pg-data-disk
spec:
capacity:
storage: 50Gi
accessModes:
- ReadWriteOnce
gcePersistentDisk:
pdName: "pg-data-disk"
fsType: "ext4"
This creates a persistent volume that pods can mount. Set it up with the same information that you used to create the disk.
postgres-claim.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pg-data-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50Gi
This creates a claim on the persistent volume that pods will use to attach the volume. It should have the same information as above.
postgres-pod.yml
apiVersion: v1
kind: Pod
metadata:
name: postgres
labels:
name: postgres
spec:
containers:
- name: postgres
image: postgres
env:
- name: DB_PASS
value: password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- containerPort: 5432
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: pg-data
volumes:
- name: pg-data
persistentVolumeClaim:
claimName: pg-data-claim
There are a few important features of this file. We use the persistent claim in the volumes section and mount it in the volumeMounts section. Postgres doesn’t have it’s data dir at the top level of a mount so we move it lower with the environment variable PGDATA. You can also change the password with DB_PASS.
postgres-service.yml
apiVersion: v1
kind: Service
metadata:
name: postgres
labels:
name: postgres
spec:
ports:
- port: 5432
selector:
name: postgres
This creates a service to easily access the postgres pod from other pods in the cluster. It creates a DNS entry pods can use to connect.
Create resources on Kubernetes
kubectl create -f postgres-persistence.yml
kubectl create -f postgres-claim.yml
kubectl create -f postgres-pod.yml
kubectl create -f postgres-service.yml
Run all of these commands and Postgres will be up and running inside your cluster.
Improvements
This works great accept for creating new databases. I had to attach to the running docker container to create new databases. This isn’t too bad, but could be a lot better.
It would also probably be better to create a replication controller instead of just a regular pod. That way if the pod died it would come back online by itself.
Overall kubernetes has been super fun to work with, especially the version hosted by Google.
