# Kubernetes

# K8s Top Commands

#####    
Список nodes

```ruby
kubectl get nodes
# kubectl get nodes -o wide
```

##### Создать namespace

```ruby
kubectl create ns my-ns
```

##### Сохранить манифест

```ruby
kubectl get ns my-ns -o yaml > ns.yaml
```


##### Применить манифест

```ruby
kubectl apply -f ns.yaml
```

##### Подробное info

```ruby
kubectl describe ns my-ns
```

##### Добавить label

```ruby
kubectl label ns my-ns kubernetes=rulezz
```

##### Добавить аннотацию

```ruby
kubectl annotate ns my-ns annotations=likelabels
```

# Установка K8s

<span style="color:rgb(0,0,0);"> </span>

### Авто развертывание k8s кластера

##### Terraform создание VMs

<p class="callout info">Ресурсы облака **Yandex.Cloud** на **Ubuntu 24.04 server** (2-CPU/4gb-RAM/30gb-Disk)</p>

<details id="bkmrk-main.tf-terraform-%7B-"><summary>main.tf</summary>

```ruby
terraform {
  required_providers {
    yandex = {
      source = "yandex-cloud/yandex"
    }
    local = {
      source = "hashicorp/local"
    }
  }
}

provider "yandex" {
  service_account_key_file = "${file("~/.ssh/yc.service.key")}"
  folder_id                = "b1gtv3uhmd1feqdajugp"
  zone                     = "ru-central1-a"
}

# Актуальный образ Ubuntu 24.04 LTS
data "yandex_compute_image" "last_ubuntu" {
  family = "ubuntu-24-04-lts"
}

# Существующая подсеть, в которую разместятся все ВМ
data "yandex_vpc_subnet" "default_a" {
  name = "default-ru-central1-a"
}

locals {
  task_name    = "kube"
  worker_names = ["node01", "node02", "node03"]
  ssh_pub_key  = file("~/.ssh/yc.pub")
  ssh_priv_key = "~/.ssh/yc"
}

# Control-plane нода (inventory-имя — kube-master)
resource "yandex_compute_instance" "cplane01" {
  name        = "${local.task_name}-cplane01"
  platform_id = "standard-v3"

  resources {
    cores  = 2
    memory = 4
  }

  boot_disk {
    initialize_params {
      image_id = data.yandex_compute_image.last_ubuntu.id
      size     = 30
    }
  }

  labels = {
    task_name  = local.task_name
    user_email = "xeonmp22_at_gmail.com"
    role       = "control-plane"
  }

  network_interface {
    subnet_id = data.yandex_vpc_subnet.default_a.subnet_id
    nat       = true
  }

  metadata = {
    ssh-keys = "ubuntu:${local.ssh_pub_key}"
  }
}

# 3 worker-ноды (node01..node03), создаются циклом по local.worker_names
resource "yandex_compute_instance" "workers" {
  for_each    = toset(local.worker_names)
  name        = "${local.task_name}-${each.key}"
  platform_id = "standard-v3"

  resources {
    cores  = 2
    memory = 4
  }

  boot_disk {
    initialize_params {
      image_id = data.yandex_compute_image.last_ubuntu.id
      size     = 30
    }
  }

  labels = {
    task_name  = local.task_name
    user_email = "xeonmp22_at_gmail.com"
    role       = "worker"
  }

  network_interface {
    subnet_id = data.yandex_vpc_subnet.default_a.subnet_id
    nat       = true
  }

  metadata = {
    ssh-keys = "ubuntu:${local.ssh_pub_key}"
  }
}

# Генерирует inventory.ini с реальными IP всех 4 ВМ (шаблон — inventory.tpl)
resource "local_file" "ansible_inventory" {
  filename = "${path.module}/inventory.ini"
  content = templatefile("${path.module}/inventory.tpl", {
    master_ip          = yandex_compute_instance.cplane01.network_interface.0.nat_ip_address
    master_internal_ip = yandex_compute_instance.cplane01.network_interface.0.ip_address
    workers            = yandex_compute_instance.workers
    ssh_priv_key       = local.ssh_priv_key
  })
}

# Ждёт готовности SSH на control-plane перед запуском Ansible
resource "null_resource" "wait_ssh_master" {
  depends_on = [yandex_compute_instance.cplane01]

  connection {
    type        = "ssh"
    user        = "ubuntu"
    host        = yandex_compute_instance.cplane01.network_interface.0.nat_ip_address
    private_key = file(local.ssh_priv_key)
    timeout     = "7m"
  }

  provisioner "remote-exec" {
    inline = ["echo check connection"]
  }
}

# Ждёт готовности SSH на каждой worker-ноде перед запуском Ansible
resource "null_resource" "wait_ssh_workers" {
  for_each   = yandex_compute_instance.workers
  depends_on = [yandex_compute_instance.workers]

  connection {
    type        = "ssh"
    user        = "ubuntu"
    host        = each.value.network_interface.0.nat_ip_address
    private_key = file(local.ssh_priv_key)
    timeout     = "7m"
  }

  provisioner "remote-exec" {
    inline = ["echo check connection"]
  }
}

# Устанавливает Ansible-коллекции
resource "null_resource" "ansible_collections" {
  provisioner "local-exec" {
    command = "ansible-galaxy collection install community.general ansible.posix"
  }
}

# Единый запуск playbook.yml сразу на все 4 ноды (master + workers)
resource "null_resource" "provision_cluster" {
  depends_on = [
    null_resource.ansible_collections,
    local_file.ansible_inventory,
    null_resource.wait_ssh_master,
    null_resource.wait_ssh_workers,
  ]

  provisioner "local-exec" {
    command = "ansible-playbook -i ${local_file.ansible_inventory.filename} playbook.yml"
  }
}

output "cplane01_internal_ip" {
  value = yandex_compute_instance.cplane01.network_interface.0.ip_address
}

output "cplane01_external_ip" {
  value = yandex_compute_instance.cplane01.network_interface.0.nat_ip_address
}

output "workers_ips" {
  value = {
    for k, v in yandex_compute_instance.workers : k => {
      internal = v.network_interface.0.ip_address
      external = v.network_interface.0.nat_ip_address
    }
  }
}
```

</details><details id="bkmrk-inventory.tpl-%5Bkube-"><summary>inventory.tpl</summary>

```ruby
[kube-master]
kube-master ansible_host=${master_ip} internal_ip=${master_internal_ip}

[kube-node]
%{ for name, inst in workers ~}
${name} ansible_host=${inst.network_interface.0.nat_ip_address}
%{ endfor ~}

[k8s_cluster:children]
kube-master
kube-node

[k8s_cluster:vars]
ansible_user=ubuntu
ansible_ssh_private_key_file=${ssh_priv_key}
ansible_ssh_common_args='-o StrictHostKeyChecking=no'
```

</details>##### Ansible Пост-Настройка VMs

<details id="bkmrk-playbook.yml-------n"><summary>playbook.yml</summary>

```ruby
---

- name: Common OS / kubeadm / kubelet / kubectl prerequisites (все ноды)
  hosts: k8s_cluster
  become: true
  vars:
    k8s_minor_version: "1.37"
  tasks:
    - name: Common prerequisites
      ansible.builtin.import_tasks: common-prereqs.yml
  handlers:
    - name: Restart containerd
      ansible.builtin.systemd:
        name: containerd
        state: restarted

- name: Initialize control-plane and get join command
  hosts: kube-master
  become: true
  vars:
    pod_network_cidr: "10.244.0.0/16"
  tasks:
    - name: Check whether the cluster is already initialized
      ansible.builtin.stat:
        path: /etc/kubernetes/admin.conf
      register: admin_conf

    - name: kubeadm init
      ansible.builtin.command: >
        kubeadm init
        --apiserver-advertise-address={{ internal_ip }}
        --pod-network-cidr={{ pod_network_cidr }}
      when: not admin_conf.stat.exists

    - name: Create .kube directory for ubuntu user
      ansible.builtin.file:
        path: /home/ubuntu/.kube
        state: directory
        owner: ubuntu
        group: ubuntu
        mode: "0700"

    - name: Copy admin.conf into ubuntu kubeconfig
      ansible.builtin.copy:
        src: /etc/kubernetes/admin.conf
        dest: /home/ubuntu/.kube/config
        remote_src: true
        owner: ubuntu
        group: ubuntu
        mode: "0600"

    - name: Apply Flannel CNI manifest
      become_user: ubuntu
      environment:
        KUBECONFIG: /home/ubuntu/.kube/config
      ansible.builtin.command: >
        kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

    - name: Wait for control-plane node to become Ready
      become_user: ubuntu
      environment:
        KUBECONFIG: /home/ubuntu/.kube/config
      ansible.builtin.shell: >
        kubectl get nodes --no-headers | awk '{print $2}' | grep -q '^Ready$'
      register: node_ready
      retries: 20
      delay: 10
      until: node_ready.rc == 0
      changed_when: false

    - name: Getting join command
      ansible.builtin.shell: kubeadm token create --print-join-command
      register: k8s_token

- name: Setting kubernetes worker nodes
  hosts: kube-node
  become: true
  gather_facts: false
  tasks:
    - name: Check whether this node already joined the cluster
      ansible.builtin.stat:
        path: /etc/kubernetes/kubelet.conf
      register: kubelet_conf

    - name: Joining cluster
      ansible.builtin.shell: "{{ hostvars['kube-master']['k8s_token'].stdout }}"
      when: not kubelet_conf.stat.exists
```

</details><details id="bkmrk-common-prereqs.yml--"><summary>common-prereqs.yml</summary>

```ruby
- name: Disable swap
  ansible.builtin.command: swapoff -a
  changed_when: false

- name: Remove swap entry from /etc/fstab
  ansible.builtin.replace:
    path: /etc/fstab
    regexp: '^([^#].*\sswap\s.*)$'
    replace: '# \1'

- name: Load kernel modules (overlay, br_netfilter)
  community.general.modprobe:
    name: "{{ item }}"
    state: present
  loop:
    - overlay
    - br_netfilter

- name: Persist kernel modules across reboots
  ansible.builtin.copy:
    dest: /etc/modules-load.d/k8s.conf
    content: |
      overlay
      br_netfilter

- name: Set sysctl params required by Kubernetes networking
  ansible.posix.sysctl:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/k8s.conf
    reload: true
  loop:
    - { name: "net.bridge.bridge-nf-call-iptables", value: "1" }
    - { name: "net.bridge.bridge-nf-call-ip6tables", value: "1" }
    - { name: "net.ipv4.ip_forward", value: "1" }

- name: Install containerd
  ansible.builtin.apt:
    name: containerd
    update_cache: true
    state: present

- name: Ensure /etc/containerd exists
  ansible.builtin.file:
    path: /etc/containerd
    state: directory

- name: Generate default containerd config (if not present)
  ansible.builtin.shell: containerd config default > /etc/containerd/config.toml
  args:
    creates: /etc/containerd/config.toml

- name: Enable SystemdCgroup in containerd config
  ansible.builtin.replace:
    path: /etc/containerd/config.toml
    regexp: "SystemdCgroup = false"
    replace: "SystemdCgroup = true"
  notify: Restart containerd

- name: Ensure containerd is started and enabled
  ansible.builtin.systemd:
    name: containerd
    state: started
    enabled: true

- name: Install packages required to add the Kubernetes apt repo
  ansible.builtin.apt:
    name:
      - apt-transport-https
      - ca-certificates
      - curl
      - gpg
    state: present

- name: Create /etc/apt/keyrings directory
  ansible.builtin.file:
    path: /etc/apt/keyrings
    state: directory
    mode: "0755"

- name: Download Kubernetes apt signing key
  ansible.builtin.get_url:
    url: "https://pkgs.k8s.io/core:/stable:/v{{ k8s_minor_version }}/deb/Release.key"
    dest: /tmp/k8s-release.key
    mode: "0644"

- name: Dearmor key into apt keyring
  ansible.builtin.shell: gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg /tmp/k8s-release.key
  args:
    creates: /etc/apt/keyrings/kubernetes-apt-keyring.gpg

- name: Add Kubernetes apt repository
  ansible.builtin.copy:
    dest: /etc/apt/sources.list.d/kubernetes.list
    content: "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v{{ k8s_minor_version }}/deb/ /\n"

- name: Install kubelet, kubeadm, kubectl
  ansible.builtin.apt:
    name:
      - kubelet
      - kubeadm
      - kubectl
    update_cache: true
    state: present

- name: Hold kubelet/kubeadm/kubectl at installed version
  ansible.builtin.dpkg_selections:
    name: "{{ item }}"
    selection: hold
  loop:
    - kubelet
    - kubeadm
    - kubectl
```

</details>### <span style="color:rgb(0,0,0);">Ручная установка k8s кластера</span>

- **[Install kubeadm #1](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)**
- **[Install kubeadm #2](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)**
- **[Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/)**

<table id="bkmrk-%D0%A4%D0%B8%D0%B7.-%D1%81%D0%B5%D1%80%D0%B2%D0%B5%D1%80-%D0%92%D0%9C-%D0%A0%D0%BE%D0%BB%D1%8C-"><tbody><tr><td>**Hypervisor**</td><td>**VM/Node**</td><td>**Role**</td><td>**IP**</td></tr><tr><td>**srv-01**</td><td>vm-01</td><td>control-plane</td><td>192.168.1.10</td></tr><tr><td>**srv-01**</td><td>vm-02</td><td>worker</td><td>192.168.1.11</td></tr><tr><td>**srv-02**</td><td>vm-03</td><td>worker</td><td>192.168.1.12</td></tr><tr><td>**srv-02**</td><td>vm-04</td><td>worker</td><td>192.168.1.13</td></tr></tbody></table>

<p class="callout warning">Требуется полная сетевая связность между хостами-гипервизорами.</p>

#### <span style="color:rgb(185,106,217);">VM-01/02/03/04</span>

##### <span style="color:rgb(53,152,219);">Подготовка ОС</span>

**Уникальные hostname / MAC / product\_uuid**

```ruby
sudo hostnamectl set-hostname vm-01       # своё имя на каждой ноде
ip link                                   # проверка MAC
sudo cat /sys/class/dmi/id/product_uuid   # проверка uuid
```

Прописать `/etc/hosts` со всеми 4 IP+именами (или настроить DNS).

**Отключение swap**

```ruby
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
```

**Включение Модулей ядра и sysctl**

```ruby
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
```

```ruby
sudo modprobe overlay
sudo modprobe br_netfilter
```

```ruby
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
```

```ruby
sudo sysctl --system
```

<p class="callout info">**Диапазоны открытых портов:Control-plane:** 6443, 2379-2380, 10250, 10259, 10257  
**Workers:** 10250, 30000-32767  
</p>

##### <span style="color:rgb(53,152,219);">Установка containerd</span>

```ruby
sudo apt-get update
sudo apt-get install -y containerd

sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd
sudo systemctl enable containerd
```

##### <span style="color:rgb(53,152,219);">Установка kubeadm, kubelet, kubectl</span>

```ruby
sudo apt-get install -y apt-transport-https ca-certificates curl gpg

curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.34/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.34/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
```

<p class="callout info">Замените `v1.34` на актуальную минорную версию (**[Kubernetes Releases](https://kubernetes.io/releases/)**)</p>

#### <span style="text-decoration:underline;"><span style="color:rgb(185,106,217);text-decoration:underline;">VM-01</span></span>

##### <span style="color:rgb(53,152,219);">Инициализация control-plane</span>

```ruby
sudo kubeadm init \
  --apiserver-advertise-address=192.168.1.10 \
  --pod-network-cidr=10.244.0.0/16
```

<p class="callout warning">Сохраните `token`из команды `kubeadm join ...`Повторная генерация: `kubeadm token create --print-join-command`</p>

##### <span style="color:rgb(53,152,219);">Настройка kubectl</span>

```ruby
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
```

##### <span style="color:rgb(53,152,219);">Сетевой плагин CNI</span>

```ruby
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml
```

`kubectl get nodes` — control-plane должна перейти в `Ready`

#### <span style="color:rgb(185,106,217);">VM-02/03/04</span>

##### <span style="color:rgb(53,152,219);">Подключение workers</span>

```ruby
sudo kubeadm join 192.168.1.10:6443 --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>
```

### <span style="color:rgb(0,0,0);">TEST k8s кластера</span>

<p class="callout info"><span style="color:rgb(0,0,0);">На <span style="color:rgb(185,106,217);">**VM-01** <span style="color:rgb(0,0,0);">controller-manager</span></span></span></p>

```ruby
kubectl get nodes -o wide
kubectl get pods -A
```

<p class="callout success">**Успешный результат**  
All NODEs — `Ready`  
System PODs (coredns, kube-proxy, CNI) — `Running`</p>

Для управления с другой машины скопируйте `~/.kube/config` с `vm-01`

<table id="bkmrk-" style="width:41.3095%;"></table>

# Схемы k8s

#### Общая схема Архитектуры k8s

[![k8s.v2.jpg](https://wiki-docs.ru/uploads/images/gallery/2026-10/scaled-1680-/k8s-v2.jpg)](https://wiki-docs.ru/uploads/images/gallery/2026-10/k8s-v2.jpg)