For years some friends and I had a bunch of VMs hosted in Kimsufi, a subcompany of OVH with very competitive prices for compute in France and Canada. This was working well, but we realized that a lot of what we were running there was able to run on free services in the cloud and the rest on a cluster of Raspberry Pis. That's how this pet project started.

Here is some of the things we were running:

• Various private git repos
• Jenkins
• Nexus
• Rocket.Chat
• OpenVPN Server
• Nextcloud (previously Owncloud)
• Webpages and blogs
• Various backend Apps running in containers

The Plan

The idea was to move as much as possible self hosted services to free versions in the cloud and the rest of the things run them in RPis. Just to be clear, we knew that we were be able to run this on any cloud provider just moving the containers, but we had laying around a bunch of RPis in our houses and we thought “why not try to use that compute?", at the end of the day they consume very little power and the cost of having them is very small. But they should be able to fail and keep the services up and running. So we decided to orchestrate the containers between our houses on RPis.

For the Git repos, Jenkins and Nexus we decided to go all the way with GitLab, we've been running more and more stuff in there since they offer private git repos, good CI pipelines and Docker registries for free. Let break down stuff by their functions:

CI Pipelines

Their CI pipelines are one of the best I've seen, they have multiple executors, in our case everything runs in a Docker container on agents or runners donated by Digital Ocean, if you prefer to run your builds on your own agents, you can register one fairly easy following these steps

Nexus & Docker Registry

Another cool feature of GitLab is the ability to have a private Docker registry in your Git repository and since we've been using Docker as our artifact object, this was a perfect match.

So far we are good with storing our artifacts as Docker images in the Gitlab registry, but they also have releases which we could use to store binaries if necessary, following with this the industry standard of storing releases in the repository.

Rocket.Chat

With Rocket.Chat we decided to leave it and go with other solutions like Telegram and Slack. There was nothing wrong with Rocket.Chat, I actually think is one of the best and it even runs well in RPis, but we were using it only as a chat platform, we weren't using much their integrations and so far we've been talking more and more over Telegram and less on Rocket.chat. Maybe if in the future we could come back to it.

OpenVPN

The OpenVPN server is running in one of our Raspberry Pis on this container: https://github.com/kylemanna/docker-openvpn

Nextcloud

The official Nextcloud server supports ARM architectures, so we just deployed their docker image

Webapps and backend Apps

For this we needed to recompile them to run on ARM as well, to do it we added the following steps to our builds: https://blog.hypriot.com/post/setup-simple-ci-pipeline-for-arm-images/

After that we ended up with a .gitlab-ci.yml like:

image: docker:latest

variables:
  DOCKER_DRIVER: overlay2

services:
- docker:dind

before_script:
- docker info

build:
  stage: build
  script:
    - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com
    - docker build . -t registry.gitlab.com/r1ckr/my-superapp/armv7:v1
    - docker push registry.gitlab.com/r1ckr/my-superapp/armv7:v1

And the only difference between other Dockerfiles and the new one for ARM is the FROM image line:

FROM hypriot/rpi-alpine
...

Now we just needed to decide what container orchestration mechanism to use, for this project we decided to go with Docker Swarm, since at the time of this writing the setup is incredibly easy compared to others and the compute consumption is really small, which is very important when running on Pis.

So we went ahead and created a Docker Swarm cluster between our Raspberry Pis and installed Portainer for the UI. After that was just a matter of setting up the compose files for our services and deploy all our apps.

There was some game servers running on windows machines that were a bit complicated to run on raspberry pis, so we moved them to Azure.

Summary

Support for ARM in Docker is bigger every day, and our apps have been running in Raspberry Pis for quite a while now. There are other challenges to overcome like their WiFi stability (not great), so we have to run them connected to ethernet.

Kubernetes is increasing its support for ARM too, but so far is better to have RPIs as a worker nodes and not to run the whole cluster in them, but they are getting pretty close to that. Maybe the next post will be replacing Swarm by K8s.

Thanks for reading this post and we hope you liked it, any questions drop them in the comments.