📚[Sprint1] 쿠버네티스 무게감 있게 설치하기

*해당 글은 인프런 『쿠버네티스 어나더 클래스 (지상편) - Sprint1』 강의를 기반으로 복습&정리 차 작성되었습니다.

 

📌실습 환경 구성

이번에는 쿠버네티스 실습환경을 구성해보고자 합니다.
개인적으로 공부할때는 VMware관련 제품을 사용하다보니 VMware Workstation을 통해서 진행하려고 했으나, 첫 실습이기 때문에 실습내용에 더 집중하고자 VirtualBox로 진행했습니다.

출처: https://inf.run/k7mF

🔍Vagrant를 활용하여  VM 생성

* OS: Rocky Linux
* 실습 스크립트를 활용하여 k8s 마스터 노드 생성

# Vagrant 폴더 생성
D:\> mkdir k8s

# Vagrant 스크립트 다운로드
D:\k8s> curl -O https://raw.githubusercontent.com/k8s-1pro/install/main/ground/k8s-1.27/vagrant-2.3.4/Vagrantfile

# Rocky Linux Repo 세팅
D:\k8s> curl -O https://raw.githubusercontent.com/k8s-1pro/install/main/ground/k8s-1.27/vagrant-2.3.4/rockylinux-repo.json
D:\k8s> vagrant box add rockylinux-repo.json

# Vagrant Disk 설정 Plugin 설치 
D:\k8s> vagrant plugin install vagrant-vbguest vagrant-disksize

# Vagrant 실행 (VM생성)
D:\k8s> vagrant up

스크립트를 이용하면 단 몇 분만에 마스터 노드 생성이 가능해집니다.. 여기서 그치면 안되겠죠..? 
 
어떻게 스크립트가 구성되어 마스터 노드가 생성되었는지 아래와 같은 순서에 따라 그 과정을 천천히 살펴보면 이해하는 데 도움이 됩니다. 
 

📌스크립트 맛보고 뜯어보기

🔍OS 구성

Vagrant.configure("2") do |config|
    
  config.vm.box = "rockylinux/8"		→ OS 종류 선택
  config.disksize.size = "50GB"			→ 디스크 크기 설정
  config.vbguest.installer_options = { allow_kernel_upgrade: true }
  config.vbguest.auto_update = false
  config.vm.provision :shell, privileged: true, inline: $install_default

  config.vm.define "master-node" do |master|			→ VM Name 설정
    master.vm.hostname = "k8s-master"			→ hostname 설정
    master.vm.network "private_network", ip: "192.168.56.30"	→ ip 설정(private 대역)
	master.vm.provider :virtualbox do |vb|
      vb.memory = 6144		→ 메모리 설정
      vb.cpus = 4			→ CPU 설정
	  vb.customize ["modifyvm", :id, "--firmware", "efi"]
	end
    master.vm.provision :shell, privileged: true, inline: $install_master
  end

end

$install_default = <<-SHELL		→ node별 기본 설치 언어 설정

네트워크는 Private-Network로 내부 통신을 하고, NAT를 통해 외부 인터넷망과 통신하여 Kubernetes설치 시 필요한 패키지들을 받을 수 있도록 하였습니다. 
 

🔍 컨테이너 런타임 및 쿠버네티스 설치

기본적인 타임존 설정과 패키지 업데이트를 진행 한 후 본격적으로 쿠버네티스를 설치하기 위한 작업을 시작합니다.

# 방화벽 off 및 비활성화
[root@k8s-master ~]# systemctl stop firewalld 
[root@k8s-master ~]# systemctl disable firewalld

# swap 비활성화
[root@k8s-master ~]# swapoff -a && sed -i '/ swap / s/^/#/' /etc/fstab 

# swap 사용률 0 확인
[root@k8s-master ~]# free
              total        used        free      shared  buff/cache   available
Mem:        6063280     1599260      130720       19168     4333300     4161612
Swap:             0           0           0


# fstab에 swap 부분 주석처리 확인
[root@k8s-master ~]# cat /etc/fstab | grep swap
#/swapfile none swap defaults 0 0

 
IPv4를 포워딩하여 iptables가 브리지된 트래픽을 보게 하기
(리눅스 노드의 iptables가 브리지된 트래픽을 올바르게 보기 위한 요구 사항으로 해당 세팅 값을 확인)

 

# 설정 세팅 확인
[root@k8s-master ~]# cat /etc/modules-load.d/k8s.conf
overlay
br_netfilter

[root@k8s-master ~]# cat /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1

# 모듈 적재 확인
[root@k8s-master ~]# lsmod | grep overlay
overlay               139264  35

[root@k8s-master ~]# lsmod | grep br_netfilter
br_netfilter           24576  0
bridge                290816  1 br_netfilter

컨테이너 런타임 설치

# docker repo 설정
[root@k8s-master ~]# yum install -y yum-utils
[root@k8s-master ~]# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# containerd 설치
[root@k8s-master ~]# yum install -y containerd.io-1.6.21-3.1.el8
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl enable --now containerd

# containerd CRI 활성화
[root@k8s-master ~]# sed -i 's/^disabled_plugins/#disabled_plugins/' /etc/containerd/config.toml

# ["cri"] 주석처리 확인
[root@k8s-master ~]# cat /etc/containerd/config.toml
#   Copyright 2018-2022 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#disabled_plugins = ["cri"]	
...

[root@k8s-master ~]# systemctl restart containerd

kubeadm 설치

# kubernetes.repo 설정
cat /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl

# repo 확인
[root@k8s-master yum.repos.d]# yum repolist enabled
repo id                                    repo name
appstream                                  Rocky Linux 8 - AppStream
baseos                                     Rocky Linux 8 - BaseOS
docker-ce-stable                           Docker CE Stable - x86_64
extras                                     Rocky Linux 8 - Extras
kubernetes                                 Kubernetes

# SELinux 설정 확인
[root@k8s-master ~]# cat /etc/selinux/config
SELINUX=permissive	→ permissive 확인

[root@k8s-master ~]# sestatus
Current mode:                   permissive		→ permissive 확인
Mode from config file:          permissive		→ permissive 확인

# kubelet, kubeadm, kubectl 패키지 설치
[root@k8s-master ~]# yum install -y kubelet-1.27.1-0.x86_64 kubeadm-1.27.1-0.x86_64 kubectl-1.27.1-0.x86_64 --disableexcludes=kubernetes
systemctl enable --now kubelet

 

🔍 마스터 노드 세팅

kubeadm으로 클러스터 생성

# pod 네트워크 세팅
[root@k8s-master ~]# kubeadm init --pod-network-cidr=20.96.0.0/12 --apiserver-advertise-address 192.168.56.30


# master node 상태확인
[root@k8s-master ~]# kubectl get node
NAME         STATUS   ROLES           AGE   VERSION
k8s-master   Ready    control-plane   85m   v1.27.1


# pod network cidr 설정 확인
[root@k8s-master ~]# kubectl cluster-info dump | grep -m 1 cluster-cidr
                            "--cluster-cidr=20.96.0.0/12",

# apiserver advertise address 적용 확인
[root@k8s-master ~]# kubectl cluster-info
Kubernetes control plane is running at https://192.168.56.30:6443

# kubernetes component pod 확인 (Running 확인)
[root@k8s-master ~]# kubectl get pods -n kube-system
NAME                                 READY   STATUS    RESTARTS   AGE
coredns-5d78c9869d-l8kpn             1/1     Running   0          85m
coredns-5d78c9869d-qftxg             1/1     Running   0          85m
etcd-k8s-master                      1/1     Running   0          86m
kube-apiserver-k8s-master            1/1     Running   0          86m
kube-controller-manager-k8s-master   1/1     Running   0          86m
kube-proxy-tht2n                     1/1     Running   0          85m
kube-scheduler-k8s-master            1/1     Running   0          86m
metrics-server-7db4fb59f9-r8zwv      1/1     Running   0          85m

# kubectl 사용 설정
[root@k8s-master ~]# mkdir -p $HOME/.kube
[root@k8s-master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]# chown $(id -u):$(id -g) $HOME/.kube/config

# 인증서 설정 확인
[root@k8s-master ~]# cat ~/.kube/config
...
    server: https://192.168.56.30:6443	→ IP 설정 확인
...

# Pod Network 설치 (calico)
[root@k8s-master ~]# kubectl create -f https://raw.githubusercontent.com/k8s-1pro/install/main/ground/k8s-1.27/calico-3.25.1/calico.yaml
[root@k8s-master ~]# kubectl create -f https://raw.githubusercontent.com/k8s-1pro/install/main/ground/k8s-1.27/calico-3.25.1/calico-custom.yaml

# Calico Pod 상태 확인 (Running 확인)
[root@k8s-master ~]# kubectl get -n calico-system pod
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-789dc4c76b-ls888   1/1     Running   0          97m
calico-node-qck7p                          1/1     Running   0          97m
calico-typha-86d895795d-wghjr              1/1     Running   0          97m
csi-node-driver-69qsk                      2/2     Running   0          97m

[root@k8s-master ~]# kubectl get -n calico-apiserver pod
NAME                                READY   STATUS    RESTARTS   AGE
calico-apiserver-5c57b4569f-dd6qk   1/1     Running   0          96m
calico-apiserver-5c57b4569f-s5h6p   1/1     Running   0          96m


# Calico에 pod network cidr 적용 확인
[root@k8s-master ~]# kubectl get installations.operator.tigera.io default -o yaml  | grep cidr
      cidr: 20.96.0.0/12
        cidr: 20.96.0.0/12
        
# Master에 Pod를 생성 할수 있도록 설정
[root@k8s-master ~]# kubectl taint nodes k8s-master node-role.kubernetes.io/control-plane-

# Taints 해제 확인
[root@k8s-master ~]# kubectl describe nodes | grep Taints
Taints:             <none>

 
쿠버네티스 편의 기능 설치

# kubectl 자동완성 기능 설정
echo "source <(kubectl completion bash)" >> ~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc

# kubectl 자동완성 기능 설정 확인
[root@k8s-master ~]# cat ~/.bashrc
# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
source <(kubectl completion bash)
alias k=kubectl
complete -o default -F __start_kubectl k

# Dashboard 설치
kubectl create -f https://raw.githubusercontent.com/k8s-1pro/install/main/ground/k8s-1.27/dashboard-2.7.0/dashboard.yaml

# Dashboard Running 확인
[root@k8s-master ~]# kubectl get pod -n kubernetes-dashboard
NAME                                         READY   STATUS    RESTARTS   AGE
dashboard-metrics-scraper-5cb4f4bb9c-rpwk4   1/1     Running   0          111m
kubernetes-dashboard-6bc7c98694-5d9bz        1/1     Running   0          111m

# Metrics Server 설치
kubectl create -f https://raw.githubusercontent.com/k8s-1pro/install/main/ground/k8s-1.27/metrics-server-0.6.3/metrics-server.yaml

# Metrics Server 설치 확인
[root@k8s-master ~]# kubectl get pod -n kube-system  | grep metrics
metrics-server-7db4fb59f9-r8zwv      1/1     Running   0          112m

🔍pod 상태 확인

[root@k8s-master ~]# k get pods -A
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
calico-apiserver       calico-apiserver-6d7f77dc86-24nkn            1/1     Running   0          19m
calico-apiserver       calico-apiserver-6d7f77dc86-9mjt6            1/1     Running   0          19m
calico-system          calico-kube-controllers-789dc4c76b-zf2z6     1/1     Running   0          21m
calico-system          calico-node-bsgs5                            1/1     Running   0          21m
calico-system          calico-typha-956d8d9f5-kwjq5                 1/1     Running   0          21m
calico-system          csi-node-driver-rwf22                        2/2     Running   0          21m
kube-system            coredns-5d78c9869d-j4dk6                     1/1     Running   0          21m
kube-system            coredns-5d78c9869d-zwwhn                     1/1     Running   0          21m
kube-system            etcd-k8s-master                              1/1     Running   0          21m
kube-system            kube-apiserver-k8s-master                    1/1     Running   0          21m
kube-system            kube-controller-manager-k8s-master           1/1     Running   0          21m
kube-system            kube-proxy-gx62q                             1/1     Running   0          21m
kube-system            kube-scheduler-k8s-master                    1/1     Running   0          21m
kube-system            metrics-server-7db4fb59f9-jl8mz              1/1     Running   0          21m
kubernetes-dashboard   dashboard-metrics-scraper-5cb4f4bb9c-2cr78   1/1     Running   0          21m
kubernetes-dashboard   kubernetes-dashboard-6bc7c98694-zd7xz        1/1     Running   0          21m
tigera-operator        tigera-operator-549d4f9bdb-svv47             1/1     Running   0          21m

🔍 Kubernetes Dashboard 접속 및 상태 확인

https://192.168.56.30:30000/#/login