Installing kubernetes and weave on a blade server cluster (unfinished)
- 07 Mar 2017: Post was created (diff)
Goal: Get 28 blade servers up and running in a kubernetes cluster
NOTE
This article is not finished, and will change.
Planning
Things (I think) I have to do
- Set up Weave Net that creates a virtual network that connects Docker containers across multiple hosts and enables their automatic discovery
- Install Kubernetes on one of the blades
- Install Kubernetes on an additional blade and add it to the cluster
- Verify the child blade configuration and create a pre-configured ISO that can be easily deployed to the rest of the blades
- Figure out what to use the cluster for
Setting up Weave Net
The reason this is the first step, before even touching the Kubernetes installation, is
- we don’t have any spare public IPv4 addresses for the blades
- we have an ocean of free IPv6 addresses, but Kubernetes does not play well with IPv6-only hosts
- when different people are configuring their own containers, we want the networks to be isolated
- we want to centralize the network configuration without involving additional hardware
Installing Weave
This is the machine I’m starting off with
# cat /etc/issue.net
Ubuntu 16.04.2 LTS
Verify that the version of docker is > 1.6
# docker --version
Docker version 17.03.0-ce, build 3a232c8
Install weave
# curl -L git.io/weave -o /usr/local/bin/weave
# chmod a+x /usr/local/bin/weave
I want to disable the automatic version check
# export CHECKPOINT_DISABLE=1
Time to sit down and read for a bit: Understanding how Weave Net Works
Launch weave on the main host
# weave launch
After launch, it should have pulled all of its docker images
# weave version
weave script 1.9.3
weave router 1.9.3
weave proxy 1.9.3
weave plugin 1.9.3
Ensure that containers launched via the Docker command line are automatically attached to the Weave network
# eval $(weave env)
Start a container and make sure no errors are being thrown upon creation. (Read Using Weave Net for troubleshooting errors)
# docker run --name test-1 -it weaveworks/ubuntu
Success!
Blade server #1
This machine is only reachable through IPv6 and therefore needs some extra tender love and care. Add an IPv6 DNS server address (Google DNS)
# echo "nameserver 2001:4860:4860:0:0:0:0:8888" >> /etc/resolv.conf
Then see if it works
# dig +short stigok.com AAAA
2a02:ed06::211
Now another test
# dig +short git.io AAAA
Did not give us anything. Which means I have to find another way to download the weave binary on this host. However related, [Where is the git.io link redirecting to?], it did not solve this issue, as github.com itself doesn’t have any AAAA records either. I’m using SCP to transfer the binary directly. Note that it has to be put in a directory in your $PATH. Since this box is Fedora, /usr/local/bin is not by default. However, /usr/local/sbin is.
# scp root@weave-host:/usr/local/bin/weave /usr/local/sbin/weave
Make sure docker is running
# systemctl status docker
If not, start it
# systemctl start docker
The docker image repository is not IPv6 enabled, so grab the images from the other host. To enable my non-root user to control the Docker daemon, I add myself to the docker group. On Fedora, this group is not added by default, so first I create it.
# groupadd docker
# gpasswd --add user docker
List images
# ssh user@weave-host docker images --format '{{.Repository}}'
weaveworks/plugin
weaveworks/weave
weaveworks/weaveexec
weaveworks/weavedb
weaveworks/ubuntu
Ok, so now lets fetch all of those and load them into this host
# ssh user@weave-host docker save weaveworks/plugin weaveworks/weave weaveworks/weaveexec weaveworks/weavedb weaveworks/ubuntu | docker load
7cbcbac42c44: Loading layer [==================================================>] 5.05 MB/5.05 MB
43f5e1d45225: Loading layer [==================================================>] 2.048 kB/2.048 kB
cdd5598a1676: Loading layer [==================================================>] 18.97 MB/18.97 MB
b7828b5e45cc: Loading layer [==================================================>] 12.94 MB/12.94 MB
7f2483959486: Loading layer [==================================================>] 9.478 MB/9.478 MB
5ca8af2897f9: Loading layer [==================================================>] 4.581 MB/4.581 MB
a78e0bf11eb8: Loading layer [==================================================>] 3.206 MB/3.206 MB
782b8b79808c: Loading layer [==================================================>] 4.512 MB/4.512 MB
fbbaf58e6c6a: Loading layer [==================================================>] 34.9 MB/34.9 MB
86d0e9420b70: Loading layer [==================================================>] 26.48 MB/26.48 MB
ead8a0e246bb: Loading layer [==================================================>] 2.048 kB/2.048 kB
fe4aaeadcb92: Loading layer [==================================================>] 277 kB/277 kB
45698d068987: Loading layer [==================================================>] 9.681 MB/9.681 MB
Loaded image: weaveworks/plugin:1.9.3
Loaded image: weaveworks/weave:1.9.3
Loaded image: weaveworks/weaveexec:1.9.3
d8330d2726bf: Loading layer [==================================================>] 2.048 kB/2.048 kB
Loaded image: weaveworks/weavedb:latest
9436069b92a3: Loading layer [==================================================>] 127.6 MB/127.6 MB
19429b698a22: Loading layer [==================================================>] 14.85 kB/14.85 kB
82b57dbc5385: Loading layer [==================================================>] 11.78 kB/11.78 kB
737f40e80b7f: Loading layer [==================================================>] 4.608 kB/4.608 kB
5f70bf18a086: Loading layer [==================================================>] 1.024 kB/1.024 kB
a7cf6da36ea3: Loading layer [==================================================>] 3.968 MB/3.968 MB
Loaded image: weaveworks/ubuntu:latest
Verify that the images are now locally available
# docker images --format '{{.Repository}}'
weaveworks/plugin
weaveworks/weave
weaveworks/weaveexec
weaveworks/weavedb
weaveworks/ubuntu
Launch weave and connect it to the weave host
# weave launch weave-host
Allow communication on the non-weave network, to avoid blocking communication between the weave nodes
iptables -I INPUT -s 10.8.1.0/24 -j ACCEPT
This article is not finished. Come back later
If you have any comments or feedback, please send me an e-mail. (stig at stigok dotcom).
Did you find any typos, incorrect information, or have something to add? Then please propose a change to this post.