Having based my Raspberry Pi cluster on Arch Linux, I need to update the bramble fairly regularly. I have a weekly event on Sundays that reminds me to keep things up to date.
My updating process:
- Update all docker hosts
- Update all file system hosts
- Update the rest
Normally this is a very uneventful process as nothing breaks. Just in case something does though, I always try to have a "canary" raspberry pi that gets updated first and doesn't have anything important on it. The last time I updated something did go wrong and I was very glad of this process.
Something went wrong
When I updated my canary pi nothing went wrong. Everything looked good. I continued on to update pis that had slightly more important things on it. Eventually I got to a pi that had a docker container on it.
After restarting to complete the update (the kernel was updated), I checked to see that all of the containers started. When they didn't I noticed I couldn't run any containers with a failure of "socket operation on non-socket".
I found this docker issue on github and was able to downgrade to the previous docker I had installed. Everything was working again and I didn't take down anything important.
Docker has since updated to 1.7.1 (not yet released, very soon) and fixed the issue.
Always be careful when running Arch as a server OS and make sure to have some sort of early warning system with updates.