[Kubernetes] 로드밸런싱을 위한 Control Plane 중설 및 HAProxy 장애 처리

2025.01.11 - [DevOps/Kubernetes] - [kubernetes]로드밸런싱을 위한 control-plane 증설

[kubernetes]로드밸런싱을 위한 control-plane 증설

2025.01.25 - [DevOps/Kubernetes] - [Kubernetes] 로드밸런싱을 위해 Control-plane 증설 이후 작업

[Kubernetes] 로드밸런싱을 위해 Control-plane 증설 이후 작업

 

이전에 발행한 글들에서 haproxy 부분에서의 미흡한 부분들을 보완하였습니다. 

 

현상

# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
     Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-03-14 16:07:27 KST; 14s ago
       Docs: man:haproxy(1)
             file:/usr/share/doc/haproxy/configuration.txt.gz
   Main PID: 1547134 (haproxy)
     Status: "Ready."
      Tasks: 13 (limit: 9441)
     Memory: 41.9M (peak: 43.4M)
        CPU: 221ms
     CGroup: /system.slice/haproxy.service
             ├─1547134 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
             └─1547137 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Mar 14 16:07:27 kube-master-200 systemd[1]: Starting haproxy.service - HAProxy Load Balancer...
Mar 14 16:07:27 kube-master-200 haproxy[1547134]: [NOTICE]   (1547134) : New worker (1547137) forked
Mar 14 16:07:27 kube-master-200 haproxy[1547134]: [NOTICE]   (1547134) : Loading success.
Mar 14 16:07:27 kube-master-200 systemd[1]: Started haproxy.service - HAProxy Load Balancer.
Mar 14 16:07:27 kube-master-200 haproxy[1547137]: [WARNING]  (1547137) : Server kubernetes-backend/kube-master-210 is DOWN, reason: Layer4 connection problem, info: "Connection refused", >
Mar 14 16:07:27 kube-master-200 haproxy[1547137]: Server kubernetes-backend/kube-master-210 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 >
Mar 14 16:07:27 kube-master-200 haproxy[1547137]: Server kubernetes-backend/kube-master-210 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 2 >
Mar 14 16:07:28 kube-master-200 haproxy[1547137]: [WARNING]  (1547137) : Server kubernetes-backend/kube-master-220 is DOWN, reason: Layer4 connection problem, info: "Connection refused", >
Mar 14 16:07:28 kube-master-200 haproxy[1547137]: Server kubernetes-backend/kube-master-220 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 >
Mar 14 16:07:28 kube-master-200 haproxy[1547137]: Server kubernetes-backend/kube-master-220 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1

# nc -zv 192.168.207.200 6443
Connection to 192.168.207.200 6443 port [tcp/*] succeeded!
# nc -zv 192.168.207.210 6443
nc: connect to 192.168.207.210 port 6443 (tcp) failed: Connection refused
# nc -zv 192.168.207.220 6443
nc: connect to 192.168.207.220 port 6443 (tcp) failed: Connection refused

 

3개의 control-plane에 모두 ha-proxy를 설치하였으나 설정이 미흡하여 제대로 프록시 설정이 이루어지지 않고 있었습니다. 

 

haproxy 설정 파일 수정

3개의 모든 control-plane 아래 설정을 추가해준다. 

# vi /etc/haproxy/haproxy.cfg

frontend kubernetes-frontend
    bind *:16443
    default_backend kubernetes-backend

backend kubernetes-backend
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 3s fall 3 rise 2
    server kube-master-200 192.168.207.200:6443 check
    server kube-master-210 192.168.207.210:6443 check
    server kube-master-220 192.168.207.220:6443 check

 

🔹 추가 설명

• balance roundrobin → 각 Master 노드로 트래픽을 고르게 분배
• option httpchk GET /healthz → /healthz 엔드포인트를 이용한 상태 체크
• http-check expect status 200 → 응답이 200 OK일 때만 정상 노드로 판단
• inter 3s fall 3 rise 2 → 3초마다 체크하며, 3회 실패 시 비정상 처리, 2회 정상 응답 시 복구

 

haproxy 장애 감지(Backend 서버 확인)

echo "show servers state" | socat stdio /run/haproxy/admin.sock
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord srv_use_ssl srv_check_port srv_check_addr srv_agent_addr srv_agent_port
3 kubernetes-backend 1 kube-master-200 192.168.207.200 0 0 1 1 411 17 2 0 6 0 0 0 - 6443 - 0 0 - - 0
3 kubernetes-backend 2 kube-master-210 192.168.207.210 0 0 1 1 410 8 2 0 6 0 0 0 - 6443 - 0 0 - - 0
3 kubernetes-backend 3 kube-master-220 192.168.207.220 0 0 1 1 409 8 2 0 6 0 0 0 - 6443 - 0 0 - - 0

 

6443 포트 확인

nc -zv 192.168.207.200 6443
nc -zv 192.168.207.210 6443
nc -zv 192.168.207.220 6443

• 제대로 연결된 경우

# nc -zv 192.168.207.200 6443
Connection to 192.168.207.200 6443 port [tcp/*] succeeded!

• 제대로 연결되지 않은 경우

# nc -zv 192.168.207.220 6443
nc: connect to 192.168.207.220 port 6443 (tcp) failed: Connection refused

 

kube-apiserver 데몬 설정

• 모든 control-plane에 실행

kube-apiserver 바이너리 다운로드

export KUBE_VERSION=v1.32.2
curl -LO https://dl.k8s.io/release/${KUBE_VERSION}/bin/linux/amd64/kube-apiserver
chmod +x kube-apiserver
sudo mv kube-apiserver /usr/local/bin/

 

Systemd Service File 생성

• --advertise-address=192.168.207.200 의 아이피는 각 control-plane 서버 아이피를 넣어준다. 
  • 210 : --advertise-address=192.168.207.210
  • 220 : --advertise-address=192.168.207.220

# vi /etc/systemd/system/kube-apiserver.service

[Unit]
Description=Kubernetes API Server
Documentation=https://kubernetes.io/docs/
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address=192.168.207.200 \
  --bind-address=0.0.0.0 \
  --secure-port=6443 \
  --service-cluster-ip-range=10.96.0.0/12 \
  --etcd-servers=https://192.168.207.200:2379,https://192.168.207.210:2379,https://192.168.207.220:2379 \
  --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt \
  --etcd-certfile=/etc/kubernetes/pki/etcd/server.crt \
  --etcd-keyfile=/etc/kubernetes/pki/etcd/server.key \
  --authorization-mode=Node,RBAC \
  --allow-privileged=true \
  --enable-admission-plugins=NodeRestriction \
  --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt \
  --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key \
  --tls-cert-file=/etc/kubernetes/pki/apiserver.crt \
  --tls-private-key-file=/etc/kubernetes/pki/apiserver.key \
  --client-ca-file=/etc/kubernetes/pki/ca.crt \
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \
  --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \
  --service-account-key-file=/etc/kubernetes/pki/sa.pub \
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \
  --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt \
  --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key \
  --requestheader-allowed-names=front-proxy-client \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --enable-bootstrap-token-auth=true

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

 

서비스 활성화 및 실행

# sudo systemctl daemon-reload
# sudo systemctl enable kube-apiserver
# sudo systemctl start kube-apiserver
# sudo systemctl status kube-apiserver

 

kube-apiserver 상태 확인

# kubectl get componentstatuses
# kubectl get nodes

 

6443 포트 확인

• 안정적으로 연결된 것 확인

# nc -zv 192.168.207.200 6443
Connection to 192.168.207.200 6443 port [tcp/*] succeeded!
# nc -zv 192.168.207.210 6443
Connection to 192.168.207.210 6443 port [tcp/*] succeeded!
# nc -zv 192.168.207.220 6443
Connection to 192.168.207.220 6443 port [tcp/*] succeeded!