Ansible Kubernetes 練習

我個人是把 k8s 當作一個方便開發用的平台，有點當作是一個Esxi平台
不必吃工作機的效能直接使用 k8s 的效能，必要的時候透過 port-forward 來進行測試
我沒有太深入 k8s 相關的技巧，不過這對我來說已經十分方便

主要是參考這篇開發 Ansible Playbooks 部署 Kubernetes v1.11.x HA 叢集 Kyle Bai大大的進行安裝

不過大大是使用 Ubuntu ，而我打算安裝在 CentOS 上面，所以遇到了一些狀況和一些智障的小細節

Hardware/Software

cpu/ram
- master: 2C4G ↑
- node: 1C2G ↑
OS:

linux (CentOS7/RHEL7)
- kernel 3.10 ↑

docker version 1.9 ↑

Install CentOS

Account
- account: k8s
- password: 12345678
Root
- account: root
- password: 12345678
VirtualBox: 6.0.14 r133895 (Qt5.6.3)

Hostname	IP	RAM	CPU	OS
k8s-m1	192.168.0.1	4G RAM	4 Core	CentOS 7.7
k8s-n1	192.168.0.2	2G RAM	2 Core	CentOS 7.7
k8s-n2	192.168.0.3	2G RAM	2 Core	CentOS 7.7

Installation

事前準備

所有節點的網路之間可以互相溝通
部署節點對其他節點不需要 SSH 密碼即可登入
所有節點都擁有 Sudoer 權限，並且不需要輸入密碼
所有節點需要安裝 Python
所有節點需要設定 /etc/host 解析到所有主機
部署節點需要安裝 Ansible

Hostname

1
2
3


hostnamectl set-homename k8s-m1
hostnamectl set-homename k8s-g1
hostnamectl set-homename k8s-g2

Hosts

1
2
3
4
5
6


vim /etc/hosts

# 加入
192.168.0.1   k8s-m1
192.168.0.2   k8s-g1
192.168.0.3   k8s-g2

SELinux

允許 containers 連到 host

1
2


setenforce 0
sed -i 's@SELINUX=enforcing@SELINUX=disabled@' /etc/sysconfig/selinux

Linux Module

Containter_runtime = containerd

Forward

1

echo 1 > /proc/sys/net/ipv4/ip_forward

br_netfilter

1
2
3
4
5


sudo modprobe br_netfilter
echo "br_netfilter" > /etc/modules-load.d/br_netfilter.conf

echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.conf

Enabled

1
2


sysctl -p
lsmod | grep br_netfilter

NTP

確保時間一致且可更新

1

sudo yum install ntp -y

1
2


sudo systemctl start ntpd
sudo systemctl enable ntpd

SSH PubeKey

讓派送機可以無密碼連線至節點機

不建議，但也能設定密碼派送

1
2
3
4
5


ssh-keygen -t rsa -P ""

ssh-copy-id 192.168.0.1
ssh-copy-id 192.168.0.2
ssh-copy-id 192.168.0.3

Firewall

Disable the firewall or open the service port on the master node

iptables相關功能或模組的啟用與停用，後續kubernetes會接手管理這部分的網路功能

master 關閉防火牆或開啟服務埠

1
2
3
4
5


systemctl disable firewalld && systemctl stop firewalld

or

iptables -A INPUT -i enp0s3 -p tcp --dport 8443 -j ACCEPT

Install Ansible

Install Package

1
2


sudo yum install -y epel-release
sudo yum install -y ansible python-netaddr cowsay git

ansible version

1
2
3
4
5
6
7
8


ansible --version
...
ansible 2.9.1
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

Git Download

1
2


git clone https://github.com/kairen/kube-ansible.git
cd kube-ansible

Setting

首先編輯檔案 inventory/hosts.ini

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[etcds]
k8s-m1 ansible_user=k8s ansible_become_pass=12345678

[masters]
k8s-m1 ansible_user=k8s ansible_become_pass=12345678

[nodes]
k8s-g1 ansible_user=k8s ansible_become_pass=12345678

[kube-cluster:children]
masters
nodes

Check Cluster Status

1
2
3
4
5
6


$ ansible -i inventory/hosts.ini all -m ping
k8s-g1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
...

k8s Cluster 安裝設定

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


---

kube_version: 1.14.3

# Container runtime,
# Supported: docker, nvidia-docker, containerd.
container_runtime: containerd

# Container network,
# Supported: calico, flannel.
cni_enable: true
container_network: calico

# Kubernetes HA extra variables.
vip_interface: ""
vip_address: 172.16.35.9

# Kubernetes extra addons
enable_ingress: true
enable_dashboard: true
enable_logging: true
enable_monitoring: true
enable_metric_server: true

grafana_user: "admin"
grafana_password: "12345678"

執行安裝

1

ansible-playbook -i inventory/hosts.ini cluster.yml

Addons 佈署

確認以上安裝沒有問題後，進行安裝 Kubernetes extra addons

然後透過 kubernetes-dashboard 確認服務是否成功佈署

https://192.168.0.1:8443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

1
2
3
4


$ ansible-playbook -i inventory/hosts.ini addons.yml
...
PLAY RECAP ******************************************************************
k8s-m1                     : ok=27   changed=22   unreachable=0    failed=0

Cody's 探索日誌