I just got a PicoCluster for Raspberry Pi 4s and I wanted to set it up to run BOINC for COVID 19 research with Rosetta@home with a docker swarm. Since there was a lot to get right be able to do it, I thought some other people might get some use for it too.

PicoCluster out of the box:

To be able to run it you need at least have a distro that is 64 bit, so Raspian as of this moment is not a good choice. There’s some beta distros for Hypriot OS, DietPi and some other things that might work, but I had a hard time getting those set up to my liking. I ended up using Ubuntu Server for Raspberry Pi (Ubuntu 20.04 LTS Server) since it has 64 bit Kernel and 64 bit userspace and also supports cloud-init that helps a lot when setting up headless servers.

In this post I will describe the setup to do it from creating the SD cards to booting up, using the Boinc docker image and attaching to the swarm manager. This installation instructions assume that you have ethernet connections to your Pi’s.

1. Getting the image and customize
2. Installing Docker and setting up the swarm manager
3. Adding workers to the Docker Swarm
4. Setting up labels for the workers
5. Setting up Boinc for the swarm

1. Getting the image and customize

1. First go to the Ubuntu homepage and download the server image for Raspberry Pi 4.
2. Flash the image to an SDCard with your favorite flasher. I usually use ApplePi-Baker because it can also make backups of SDCard, but another popular one is Etcher.
3. Mount the newly flashed card and find the user-data file. This is what cloud-init uses to customize the installation when you first boot up. Replace the file with the following:

#cloud-config

hostname: <yourhostname>

# Change the default user to be pi
system_info:
  default_user:
    name: pi

chpasswd:
  expire: false
  list:
  - pi:<your password>

# Enable password authentication with the SSH daemon
ssh_pwauth: true

## On first boot, use ssh-import-id to give the specific users SSH access to
## the default user
ssh_import_id:
- gh:<your github name>

This will make your newly flashed sdcard to change the default user from Ubuntu to pi and to change the password to and fetch your ssh key from github so that you wont have to type in a password to login. It will also set up the hostname to <your hostname>.

4. Save and eject the image.

2. Installing Docker and setting up the Swarm manager

1. Insert the image into your pi and power up.
2. Find the IP of your pi. You can use your router and see what IP it gets or find it by using a network scanner (I use LanScan).
3. Log in with ssh:

$> ssh pi@your_pi_ip

4. Now it’s time to install a fresh docker:

$> curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add 
$> sudo add-apt-repository "deb [arch=arm64] https://download.docker.com/linux/ubuntu bionic stable"
$> sudo apt-get install docker-ce docker-ce-cli containerd.io
$> sudo usermod -aG docker pi

5. Start the swarm manager on that Pi

$> docker swarm init

This will print out a join token that you will need for the workers.

6. Set up the boinc network for the swarm

$> docker network create -d overlay --attachable boinc

7. Here you will also install Portainer so that you get a nice web interface to handle the swarm:

$> curl -L https://downloads.portainer.io/portainer-agent-stack.yml -o portainer-agent-stack.yml
$> docker stack deploy --compose-file=portainer-agent-stack.yml portainer

You can handle the Docker swarm on http://yourmanagerip:9000.

3. Adding workers to the Docker swarm

1. Follow the instructions to flash a SDCard and install Docker, but don’t do stage 5 in the previous instruction (don’t start the swarm manager). Do this for every worker node you have.
2. Add the workers to the swarm:

$> docker swarm join --token <your token>

3. Create the mount point directory for the work units that Boinc will use later:

$> sudo mkdir /opt/appdata; sudo mkdir /opt/appdata/boinc

4. Setting up labels for the workers

1. This is to avoid choking the manager. Before I figured this out, I couldn’t access Portainer, nor could I ssh in to the manager.
2. You can go in to portainer on http://yourManagerPiIP:9000 and check the swarm and add labels with the web interface or add them with (be logged in on the manager):

   

$> docker node update --label-add boinc=true <workername>

5. Setting up Boinc for the swarm

1. Now it’s time to set up the Boinc docker image (Here I have set a limit to 3.5 Gb, since I have 4 Gb Raspberry Pi’s. Change it if you have lower memory):

docker service create \
  --mode global
  --name boinc \
  --network=boinc \
  -p 31416:31416 \
  --constraint 'node.labels.boinc==true' \
  --mount type=bind,source=/opt/appdata/boinc,destination=/var/lib/boinc \
  -e BOINC_GUI_RPC_PASSWORD=<your desired password> \
  -e BOINC_CMD_LINE_OPTIONS="--allow_remote_gui_rpc" \
  --limit-memory=3.5G \
  boinc/client:arm64v8

2. Now the Boinc client is up and running on one node. Since it’s set to global mode it will spin up on any new node you add with the label boinc with the value true.
3. Log in to Rosetta@home and get your account key on your Account->Account Keys and get the key.
4. Now we want to connect all the workers to the Rosetta@home project (you can do it after you have spun up all the nodes, or after each one):

$> docker run --rm --network boinc boinc/client:arm32v7 boinccmd_swarm --passwd <your desired password> --project_attach https://boinc.bakerlab.org/rosetta/ <your account key>

5. After this it should be up and running. You can either run a Boinc Manager on you main machine and log on to each individual worker and see it running, check out the logs in the Service menu and check the log of each worker or run this on the manager node:

docker run --rm --network boinc boinc/client:arm64v8 boinccmd_swarm --passwd Striker --get_simple_gui_info

Stay safe and have a nice day!