Kubernetes 部署

Summary: Author: 张亚飞 | 阅读时间: 37 minute read | Published: 2018-12-22
Filed under Categories: MarkDownTags: Tag,

Reference:


Kubernetes 集群安装

本文准备部署一个 一主两从 的 三节点 Kubernetes 集群,整体节点规划如下表所示:

主机名 IP 角色
v.us.8 66.42.110.223 k8s主节点
v.cs.8 66.42.108.136 k8s从节点
a.us.1 47.244.154.194 k8s从节点
a.us.0 47.89.12.166 k8s从节点
m.us.0 103.37.147.250 k8s从节点

准备工作

关闭系统的 Swap 交换分区,方法如下:

sudo swapoff -a
free –h

调整系统参数 vm.swappiness:

echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p

同时需要修改 /etc/fstab 注释 Swap 相关的条目

sed -i '/swap/s/^/#/' /etc/fstab

Docker1.13 版本开始调整了默认的防火墙规则,禁用了 iptables filter 表中 FOWARD 链,这样会引起 Kubernetes 集群中跨 NodePod 无法通信,在各个 Docker 节点执行下面的命令:

sudo iptables -P FORWARD ACCEPT

其它备注

sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X && sudo iptables -L
sudo ipvsadm -C

开启 IP 转发 不然会导致容器无法访问网络

echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf
echo net.bridge.bridge-nf-call-iptables=1 >> /etc/sysctl.conf
echo net.bridge.bridge-nf-call-ip6tables=1 >> /etc/sysctl.conf
sysctl -p

安装 Docker

...此处省略

安装 Kubernetes-v1.13.2

安装基础支持软件:

  1. 添加 Kubernetes 官方源
  • Ubuntu
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

vi /etc/apt/sources.list.d/kubernetes.list

deb http://apt.kubernetes.io/ kubernetes-xenial main

注意: 墙内按以下方式安装:

增加 kubernetes aliyun 镜像源

sudo apt-get update && sudo apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg |sudo apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

安装 kubeletkubeadmkubectl 套件

sudo apt update
sudo apt install kubelet kubectl kubeadm

配置安装源,修改文件 /etc/yum.repos.d/kubernetes.repo 内容如下

* [kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

禁用 selinux

setenforce 0

安装 kubeletkubeadm

yum update
yum install -y kubelet kubeadm

4 启用 kubelet

systemctl enable kubelet && systemctl start kubelet

kubeadm 安装完以后,就可以使用它来快速安装部署 Kubernetes 集群了.


重新安装部署

首先删除已安装的残留文件:

sudo kubeadm reset
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X && sudo iptables -L
sudo rm -rf /var/lib/rook

使用 kubeadm 安装 Kubernetes 集群

在做好了准备工作之后,下面介绍如何使用 kubeadm 安装 Kubernetes 集群,我们将首先安装 master 节点,然后将 slave 节点一个个加入到集群中去.

  • 首先查看需要安装的镜像:
Wed Jan 23 13:48:50 coam@v.us.8:~$ kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.13.2
k8s.gcr.io/kube-controller-manager:v1.13.2
k8s.gcr.io/kube-scheduler:v1.13.2
k8s.gcr.io/kube-proxy:v1.13.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.6
  • 使用 kubeadm 初始化 master 节点

指定 IP 地址及 kubernetes 版本 1.13.2:

sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --apiserver-advertise-address=172.31.141.97

使用初始化配置初始化 Master 节点

参考 kubernetes: k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1

$ cat kubeadm-cluster-init.yml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.2
clusterName: kubernetes
imageRepository: k8s.gcr.io
certificatesDir: /etc/kubernetes/pki
controlPlaneEndpoint: ""
controllerManager: {}
apiServer:
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

使用配置文件初始化:

sudo kubeadm init --config kubeadm-cluster-init.yml

查看其它默认配置

kubeadm config print init-defaults
kubeadm config print join-defaults

查看运行配置:

kubeadm config view

附启动过程:

$ sudo kubeadm init --config kubeadm-cluster-init.yml
* [init] Using Kubernetes version: v1.13.2
* [preflight] Running pre-flight checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.1. Latest validated version: 18.06
* [preflight] Pulling images required for setting up a Kubernetes cluster
* [preflight] This might take a minute or two, depending on the speed of your internet connection
* [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
* [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
* [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
* [kubelet-start] Activating the kubelet service
* [certs] Using certificateDir folder "/etc/kubernetes/pki"
* [certs] Generating "front-proxy-ca" certificate and key
* [certs] Generating "front-proxy-client" certificate and key
* [certs] Generating "etcd/ca" certificate and key
* [certs] Generating "etcd/server" certificate and key
* [certs] etcd/server serving cert is signed for DNS names [v.us.8 localhost] and IPs [66.42.110.223 127.0.0.1 ::1]
* [certs] Generating "etcd/peer" certificate and key
* [certs] etcd/peer serving cert is signed for DNS names [v.us.8 localhost] and IPs [66.42.110.223 127.0.0.1 ::1]
* [certs] Generating "etcd/healthcheck-client" certificate and key
* [certs] Generating "apiserver-etcd-client" certificate and key
* [certs] Generating "ca" certificate and key
* [certs] Generating "apiserver" certificate and key
* [certs] apiserver serving cert is signed for DNS names [v.us.8 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 66.42.110.223]
* [certs] Generating "apiserver-kubelet-client" certificate and key
* [certs] Generating "sa" key and public key
* [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
* [kubeconfig] Writing "admin.conf" kubeconfig file
* [kubeconfig] Writing "kubelet.conf" kubeconfig file
* [kubeconfig] Writing "controller-manager.conf" kubeconfig file
* [kubeconfig] Writing "scheduler.conf" kubeconfig file
* [control-plane] Using manifest folder "/etc/kubernetes/manifests"
* [control-plane] Creating static Pod manifest for "kube-apiserver"
* [control-plane] Creating static Pod manifest for "kube-controller-manager"
* [control-plane] Creating static Pod manifest for "kube-scheduler"
* [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
* [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
* [apiclient] All control plane components are healthy after 20.502192 seconds
* [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
* [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
* [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "v.us.8" as an annotation
* [mark-control-plane] Marking the node v.us.8 as control-plane by adding the label "node-role.kubernetes.io/master=''"
* [mark-control-plane] Marking the node v.us.8 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
* [bootstrap-token] Using token: vv296e.4j26mg3pqnvnxhcj
* [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
* [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
* [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
* [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
* [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
* [addons] Applied essential addon: CoreDNS
* [addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 66.42.110.223:6443 --token vv296e.4j26mg3pqnvnxhcj --discovery-token-ca-cert-hash sha256:3c480fd2249f99a9727e50e58f189c6510cab952ca49c551fc238792b3224e00

按提示要求拷贝用户配置文件

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
  • 查看主节点状态
$ kubectl get nodes,po,svc --all-namespaces
NAME          STATUS     ROLES    AGE   VERSION
node/v.us.8   NotReady   master   88s   v1.13.2

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-86c58d9df4-9gprm         0/1     Pending   0          69s
kube-system   pod/coredns-86c58d9df4-h729x         0/1     Pending   0          69s
kube-system   pod/etcd-v.us.8                      1/1     Running   0          13s
kube-system   pod/kube-apiserver-v.us.8            1/1     Running   0          23s
kube-system   pod/kube-controller-manager-v.us.8   1/1     Running   0          21s
kube-system   pod/kube-proxy-lcx69                 1/1     Running   0          69s
kube-system   pod/kube-scheduler-v.us.8            1/1     Running   0          7s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP         88s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   83s

这里显示节点都是 NotReady 原因可以用 kubectl describe node <节点状态> 看到:

$ kubectl describe node v.us.8
Name:               v.us.8
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/hostname=v.us.8
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 25 Jan 2019 20:47:35 +0800
Taints:             node-role.kubernetes.io/master:NoSchedule
                    node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 25 Jan 2019 20:49:05 +0800   Fri, 25 Jan 2019 20:47:30 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 25 Jan 2019 20:49:05 +0800   Fri, 25 Jan 2019 20:47:30 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 25 Jan 2019 20:49:05 +0800   Fri, 25 Jan 2019 20:47:30 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Fri, 25 Jan 2019 20:49:05 +0800   Fri, 25 Jan 2019 20:47:30 +0800   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  66.42.110.223
  Hostname:    v.us.8
Capacity:
 cpu:                2
 ephemeral-storage:  61883672Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             4039292Ki
 pods:               110
Allocatable:
 cpu:                2
 ephemeral-storage:  57031992021
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             3936892Ki
 pods:               110
System Info:
 Machine ID:                 3a7c1ff61e60446a83df16b875b1f675
 System UUID:                3a7c1ff6-1e60-446a-83df-16b875b1f675
 Boot ID:                    8fa74a63-7bdf-47e6-8629-5a343f64ec5e
 Kernel Version:             4.18.0-10-generic
 OS Image:                   Ubuntu 18.10
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.9.1
 Kubelet Version:            v1.13.2
 Kube-Proxy Version:         v1.13.2
PodCIDR:                     10.244.0.0/24
Non-terminated Pods:         (5 in total)
  Namespace                  Name                              CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                              ------------  ----------  ---------------  -------------  ---
  kube-system                etcd-v.us.8                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         24s
  kube-system                kube-apiserver-v.us.8             250m (12%)    0 (0%)      0 (0%)           0 (0%)         34s
  kube-system                kube-controller-manager-v.us.8    200m (10%)    0 (0%)      0 (0%)           0 (0%)         32s
  kube-system                kube-proxy-lcx69                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         80s
  kube-system                kube-scheduler-v.us.8             100m (5%)     0 (0%)      0 (0%)           0 (0%)         18s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                550m (27%)  0 (0%)
  memory             0 (0%)      0 (0%)
  ephemeral-storage  0 (0%)      0 (0%)
Events:
  Type    Reason                   Age                  From                Message
  ----    ------                   ----                 ----                -------
  Normal  Starting                 107s                 kubelet, v.us.8     Starting kubelet.
  Normal  NodeHasSufficientMemory  107s (x8 over 107s)  kubelet, v.us.8     Node v.us.8 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure    107s (x8 over 107s)  kubelet, v.us.8     Node v.us.8 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID     107s (x7 over 107s)  kubelet, v.us.8     Node v.us.8 status is now: NodeHasSufficientPID
  Normal  NodeAllocatableEnforced  107s                 kubelet, v.us.8     Updated Node Allocatable limit across pods
  Normal  Starting                 79s                  kube-proxy, v.us.8  Starting kube-proxy.

查看 coredns 容器状态,可以在 Event 看到因为网络插件未就绪,所以容器没有启动成功

$ kubectl describe po $(kubectl get po -n=kube-system | grep coredns | tail -n 1 | awk '{print $1}') -n=kube-system
Name:               coredns-86c58d9df4-lnt47
Namespace:          kube-system
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             k8s-app=kube-dns
                    pod-template-hash=86c58d9df4
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      ReplicaSet/coredns-86c58d9df4
Containers:
  coredns:
    Image:       k8s.gcr.io/coredns:1.2.6
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from coredns-token-kqgfm (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  coredns-token-kqgfm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  coredns-token-kqgfm
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly
                 node-role.kubernetes.io/master:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  7m12s (x2 over 7m12s)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.

安装扩展插件

安装 CNI 网络插件

  • 安装 Weave 网络插件:
wget -O k8s-plugins-weave-daemonset-k8s-1.8.yaml https://github.com/weaveworks/weave/releases/download/v2.5.1/weave-daemonset-k8s-1.8.yaml
$ kubectl apply -f k8s-plugins-weave-daemonset-k8s-1.8.yaml
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
  • 再次查看主节点状态:
$ kubectl get nodes,po,svc --all-namespaces
NAME          STATUS   ROLES    AGE    VERSION
node/v.us.8   Ready    master   3m2s   v1.13.2

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-86c58d9df4-9gprm         1/1     Running   0          2m43s
kube-system   pod/coredns-86c58d9df4-h729x         1/1     Running   0          2m43s
kube-system   pod/etcd-v.us.8                      1/1     Running   0          107s
kube-system   pod/kube-apiserver-v.us.8            1/1     Running   0          117s
kube-system   pod/kube-controller-manager-v.us.8   1/1     Running   0          115s
kube-system   pod/kube-proxy-lcx69                 1/1     Running   0          2m43s
kube-system   pod/kube-scheduler-v.us.8            1/1     Running   0          101s
kube-system   pod/weave-net-zgpfc                  2/2     Running   0          27s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP         3m2s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   2m57s

安装 Rook 存储插件

安装存储插件

安装 Rook + Ceph

git clone https://github.com/rook/rook.git

首先删除已安装的残留文件:

rm -rf /var/lib/rook
  1. 第一步: Rook
$ kubectl create -f rook/cluster/examples/kubernetes/ceph/operator.yaml
namespace/rook-ceph-system created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
role.rbac.authorization.k8s.io/rook-ceph-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
serviceaccount/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
deployment.apps/rook-ceph-operator created
  1. 第二步: Ceph
$ kubectl create -f rook/cluster/examples/kubernetes/ceph/cluster.yaml
namespace/rook-ceph created
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-mgr-system created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
cephcluster.ceph.rook.io/rook-ceph created

查看存储插件情况

$ kubectl describe pods -n rook-ceph-system
Name:               rook-ceph-operator-76cf7f88f-6gpsn
Namespace:          rook-ceph-system
Priority:           0
PriorityClassName:  <none>
Node:               <none>
Labels:             app=rook-ceph-operator
                    pod-template-hash=76cf7f88f
Annotations:        <none>
Status:             Pending
IP:
Controlled By:      ReplicaSet/rook-ceph-operator-76cf7f88f
Containers:
  rook-ceph-operator:
    Image:      rook/ceph:master
    Port:       <none>
    Host Port:  <none>
    Args:
      ceph
      operator
    Environment:
      ROOK_ALLOW_MULTIPLE_FILESYSTEMS:    false
      ROOK_LOG_LEVEL:                     INFO
      ROOK_MON_HEALTHCHECK_INTERVAL:      45s
      ROOK_MON_OUT_TIMEOUT:               300s
      ROOK_DISCOVER_DEVICES_INTERVAL:     60m
      ROOK_HOSTPATH_REQUIRES_PRIVILEGED:  false
      ROOK_ENABLE_SELINUX_RELABELING:     true
      ROOK_ENABLE_FSGROUP:                true
      NODE_NAME:                           (v1:spec.nodeName)
      POD_NAME:                           rook-ceph-operator-76cf7f88f-6gpsn (v1:metadata.name)
      POD_NAMESPACE:                      rook-ceph-system (v1:metadata.namespace)
    Mounts:
      /etc/ceph from default-config-dir (rw)
      /var/lib/rook from rook-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-slt7h (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  rook-config:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-ceph-system-token-slt7h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-system-token-slt7h
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  99s   default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  • 有时候一个 pod 创建之后一直是 pending,没有日志,也没有 pull 镜像,describe 的时候发现里面有一句话: 1 node(s) had taints that the pod didn't tolerate.
1 node(s) had taints that the pod didn't tolerate.

直译意思是节点有了污点无法容忍,执行 kubectl get no -o yaml | grep taint -A 5 之后发现该节点是不可调度的.这是因为 kubernetes 出于安全考虑默认情况下无法在 master 节点上部署 pod,于是用下面方法解决:

因为我只有1个节点,要“计算资源”的话,就只能将master节点的taint去掉,否则普通的Pod默认不会调度上来.

kubectl taint nodes --all node-role.kubernetes.io/master-

参考 Kubernetes使用中发现的问题和错误

列出所有 namespace 中的所有 pod

$ kubectl get pods --all-namespaces
NAMESPACE          NAME                                 READY   STATUS      RESTARTS   AGE
kube-system        coredns-86c58d9df4-9gprm             1/1     Running     0          59m
kube-system        coredns-86c58d9df4-h729x             1/1     Running     0          59m
kube-system        etcd-v.us.8                          1/1     Running     0          58m
kube-system        kube-apiserver-v.us.8                1/1     Running     0          58m
kube-system        kube-controller-manager-v.us.8       1/1     Running     0          58m
kube-system        kube-proxy-lcx69                     1/1     Running     0          59m
kube-system        kube-scheduler-v.us.8                1/1     Running     0          58m
kube-system        weave-net-zgpfc                      2/2     Running     0          56m
rook-ceph-system   rook-ceph-agent-8pdms                1/1     Running     0          26m
rook-ceph-system   rook-ceph-operator-76cf7f88f-6gpsn   1/1     Running     0          54m
rook-ceph-system   rook-discover-tn9n5                  1/1     Running     0          26m
rook-ceph          rook-ceph-mgr-a-8649f78d9b-r42xx     1/1     Running     0          24m
rook-ceph          rook-ceph-mon-a-78c8b48c79-vrlsx     1/1     Running     0          25m
rook-ceph          rook-ceph-mon-b-84f8979fd9-gfhpj     1/1     Running     0          25m
rook-ceph          rook-ceph-mon-c-5b49565bf7-fc6lm     1/1     Running     0          25m
rook-ceph          rook-ceph-osd-0-85f7c8c89-98vm5      1/1     Running     0          24m
rook-ceph          rook-ceph-osd-prepare-v.us.8-sc7gw   0/2     Completed   0          24m

再次查看存储插件情况:

$ kubectl describe pods -n rook-ceph-system
Name:               rook-ceph-agent-pmvfs
Namespace:          rook-ceph-system
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:50:27 +0800
Labels:             app=rook-ceph-agent
                    controller-revision-hash=547ff695d9
                    pod-template-generation=1
Annotations:        <none>
Status:             Running
IP:                 66.42.110.223
Controlled By:      DaemonSet/rook-ceph-agent
Containers:
  rook-ceph-agent:
    Container ID:  docker://1fb8f624697ff7a5cba8971b06e9971450e7fe3232e17d2df1507d9fae4b86ad
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      agent
    State:          Running
      Started:      Sat, 26 Jan 2019 10:50:28 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:                   rook-ceph-system (v1:metadata.namespace)
      NODE_NAME:                        (v1:spec.nodeName)
      AGENT_MOUNT_SECURITY_MODE:       Any
      ROOK_ENABLE_SELINUX_RELABELING:  true
      ROOK_ENABLE_FSGROUP:             true
    Mounts:
      /dev from dev (rw)
      /flexmnt from flexvolume (rw)
      /lib/modules from libmodules (rw)
      /sys from sys (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-q8t87 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  flexvolume:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
    HostPathType:
  dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:
  libmodules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  rook-ceph-system-token-q8t87:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-system-token-q8t87
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  105s  default-scheduler  Successfully assigned rook-ceph-system/rook-ceph-agent-pmvfs to v.us.8
  Normal  Pulled     104s  kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    104s  kubelet, v.us.8    Created container
  Normal  Started    104s  kubelet, v.us.8    Started container

Name:               rook-ceph-operator-76cf7f88f-bxfhj
Namespace:          rook-ceph-system
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:50:26 +0800
Labels:             app=rook-ceph-operator
                    pod-template-hash=76cf7f88f
Annotations:        <none>
Status:             Running
IP:                 10.32.0.8
Controlled By:      ReplicaSet/rook-ceph-operator-76cf7f88f
Containers:
  rook-ceph-operator:
    Container ID:  docker://c21dd15e4b9719b71c002e796d231f8fcc2df10aa0f85a128595cf419e90641e
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      operator
    State:          Running
      Started:      Sat, 26 Jan 2019 10:50:27 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_ALLOW_MULTIPLE_FILESYSTEMS:    false
      ROOK_LOG_LEVEL:                     INFO
      ROOK_MON_HEALTHCHECK_INTERVAL:      45s
      ROOK_MON_OUT_TIMEOUT:               300s
      ROOK_DISCOVER_DEVICES_INTERVAL:     60m
      ROOK_HOSTPATH_REQUIRES_PRIVILEGED:  false
      ROOK_ENABLE_SELINUX_RELABELING:     true
      ROOK_ENABLE_FSGROUP:                true
      NODE_NAME:                           (v1:spec.nodeName)
      POD_NAME:                           rook-ceph-operator-76cf7f88f-bxfhj (v1:metadata.name)
      POD_NAMESPACE:                      rook-ceph-system (v1:metadata.namespace)
    Mounts:
      /etc/ceph from default-config-dir (rw)
      /var/lib/rook from rook-config (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-q8t87 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rook-config:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-ceph-system-token-q8t87:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-system-token-q8t87
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  2m18s (x2 over 2m18s)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled         106s                   default-scheduler  Successfully assigned rook-ceph-system/rook-ceph-operator-76cf7f88f-bxfhj to v.us.8
  Normal   Pulled            105s                   kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal   Created           105s                   kubelet, v.us.8    Created container
  Normal   Started           105s                   kubelet, v.us.8    Started container

Name:               rook-discover-g7psj
Namespace:          rook-ceph-system
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:50:27 +0800
Labels:             app=rook-discover
                    controller-revision-hash=6585f46f49
                    pod-template-generation=1
Annotations:        <none>
Status:             Running
IP:                 10.32.0.9
Controlled By:      DaemonSet/rook-discover
Containers:
  rook-discover:
    Container ID:  docker://a61b20fe2d4453843bdb19b5ae9a41ce3d0f1392cf1500e8d73510890cfded79
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      discover
      --discover-interval
      60m
    State:          Running
      Started:      Sat, 26 Jan 2019 10:50:28 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAMESPACE:  rook-ceph-system (v1:metadata.namespace)
      NODE_NAME:       (v1:spec.nodeName)
    Mounts:
      /dev from dev (rw)
      /run/udev from udev (ro)
      /sys from sys (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-q8t87 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:
  sys:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:
  udev:
    Type:          HostPath (bare host directory volume)
    Path:          /run/udev
    HostPathType:
  rook-ceph-system-token-q8t87:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-system-token-q8t87
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  105s  default-scheduler  Successfully assigned rook-ceph-system/rook-discover-g7psj to v.us.8
  Normal  Pulled     104s  kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    104s  kubelet, v.us.8    Created container
  Normal  Started    104s  kubelet, v.us.8    Started container

查看成功安装的 pods 信息:

$ kubectl describe pods -n rook-ceph
Name:               rook-ceph-mgr-a-8649f78d9b-wwxrc
Namespace:          rook-ceph
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:51:24 +0800
Labels:             app=rook-ceph-mgr
                    ceph_daemon_id=a
                    instance=a
                    mgr=a
                    pod-template-hash=8649f78d9b
                    rook_cluster=rook-ceph
Annotations:        prometheus.io/port: 9283
                    prometheus.io/scrape: true
Status:             Running
IP:                 10.32.0.13
Controlled By:      ReplicaSet/rook-ceph-mgr-a-8649f78d9b
Init Containers:
  config-init:
    Container ID:  docker://dcc20ca41bc1d165f230aa91212c050f03f5ed0de1a4f9fc06766dcd9eee0e85
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      mgr-init
      --config-dir=/var/lib/rook
      --mgr-name=a
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:25 +0800
      Finished:     Sat, 26 Jan 2019 10:51:27 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_MGR_KEYRING:             <set to the key 'keyring' in secret 'rook-ceph-mgr-a'>  Optional: false
      ROOK_PRIVATE_IP:               (v1:status.podIP)
      ROOK_PUBLIC_IP:                (v1:status.podIP)
      ROOK_MGR_MODULE_SERVER_ADDR:   (v1:status.podIP)
      ROOK_CEPH_VERSION_NAME:       mimic
      ROOK_MON_ENDPOINTS:           <set to the key 'data' of config map 'rook-ceph-mon-endpoints'>  Optional: false
      ROOK_MON_SECRET:              <set to the key 'mon-secret' in secret 'rook-ceph-mon'>          Optional: false
      ROOK_ADMIN_SECRET:            <set to the key 'admin-secret' in secret 'rook-ceph-mon'>        Optional: false
      ROOK_CEPH_CONFIG_OVERRIDE:    /etc/rook/config/override.conf
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /etc/rook/config from rook-config-override (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-mgr-token-kdq4t (ro)
Containers:
  mgr:
    Container ID:  docker://6c25bf1aa3f3956157e6f8550156fd8bcb44fb804f2e63d4923d6556aa2d7242
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Ports:         6800/TCP, 9283/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      ceph-mgr
    Args:
      --foreground
      --id
      a
    State:          Running
      Started:      Sat, 26 Jan 2019 10:51:29 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:           rook-ceph-mgr-a-8649f78d9b-wwxrc (v1:metadata.name)
      POD_NAMESPACE:      rook-ceph (v1:metadata.namespace)
      NODE_NAME:           (v1:spec.nodeName)
      ROOK_CLUSTER_NAME:  rook-ceph
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-mgr-token-kdq4t (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rook-data:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  ceph-default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-config-override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-config-override
    Optional:  false
  rook-ceph-mgr-token-kdq4t:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-mgr-token-kdq4t
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  84s   default-scheduler  Successfully assigned rook-ceph/rook-ceph-mgr-a-8649f78d9b-wwxrc to v.us.8
  Normal  Pulled     83s   kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    83s   kubelet, v.us.8    Created container
  Normal  Started    83s   kubelet, v.us.8    Started container
  Normal  Pulled     80s   kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    80s   kubelet, v.us.8    Created container
  Normal  Started    79s   kubelet, v.us.8    Started container

Name:               rook-ceph-mon-a-f5cb646cb-chkkh
Namespace:          rook-ceph
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:50:43 +0800
Labels:             app=rook-ceph-mon
                    ceph_daemon_id=a
                    mon=a
                    mon_cluster=rook-ceph
                    pod-template-hash=f5cb646cb
                    rook_cluster=rook-ceph
Annotations:        <none>
Status:             Running
IP:                 10.32.0.11
Controlled By:      ReplicaSet/rook-ceph-mon-a-f5cb646cb
Init Containers:
  config-init:
    Container ID:  docker://86d583a911c172840f3a3d6e3d409236acca2c00ff662fd2ee8b3a15b39d91d1
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      mon-init
      --config-dir=/var/lib/rook
      --name=a
      --port=6789
      --fsid=27650da9-f702-4375-a03f-b2becc7dd19b
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:50:44 +0800
      Finished:     Sat, 26 Jan 2019 10:50:44 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_PRIVATE_IP:             (v1:status.podIP)
      ROOK_PUBLIC_IP:             10.108.69.163
      ROOK_CLUSTER_NAME:          rook-ceph
      ROOK_MON_ENDPOINTS:         <set to the key 'data' of config map 'rook-ceph-mon-endpoints'>  Optional: false
      ROOK_MON_SECRET:            <set to the key 'mon-secret' in secret 'rook-ceph-mon'>          Optional: false
      ROOK_ADMIN_SECRET:          <set to the key 'admin-secret' in secret 'rook-ceph-mon'>        Optional: false
      ROOK_CEPH_CONFIG_OVERRIDE:  /etc/rook/config/override.conf
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /etc/rook/config from rook-config-override (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
  monmap-init:
    Container ID:  docker://17ae2ea125a21bc2fdf4ddbe9f32ec329cb42f815d6793c0eacd94d7a95f640a
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/monmaptool
    Args:
      /var/lib/rook/mon-a/monmap
      --create
      --clobber
      --fsid
      27650da9-f702-4375-a03f-b2becc7dd19b
      --add
      a
      10.108.69.163:6789
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:50:45 +0800
      Finished:     Sat, 26 Jan 2019 10:50:51 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
  mon-fs-init:
    Container ID:  docker://61548e99e2e563ad85dce3289c2a1dfbf006135d2aaf9c36cd56273f0af1da8d
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      ceph-mon
    Args:
      --mkfs
      --monmap
      /var/lib/rook/mon-a/monmap
      --name
      mon.a
      --mon-data
      /var/lib/rook/mon-a/data
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:50:52 +0800
      Finished:     Sat, 26 Jan 2019 10:50:52 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Containers:
  mon:
    Container ID:  docker://e81184e54049b9a71e30c61fa86b6a007569a996ef431b2ca3a5094deaa5b949
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          6789/TCP
    Host Port:     0/TCP
    Command:
      ceph-mon
    Args:
      --foreground
      --public-addr
      10.108.69.163:6789
      --name
      mon.a
      --mon-data
      /var/lib/rook/mon-a/data
    State:          Running
      Started:      Sat, 26 Jan 2019 10:50:53 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:       rook-ceph-mon-a-f5cb646cb-chkkh (v1:metadata.name)
      POD_NAMESPACE:  rook-ceph (v1:metadata.namespace)
      NODE_NAME:       (v1:spec.nodeName)
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rook-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook
    HostPathType:
  ceph-default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-config-override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-config-override
    Optional:  false
  default-token-99qjp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99qjp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=v.us.8
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2m5s  default-scheduler  Successfully assigned rook-ceph/rook-ceph-mon-a-f5cb646cb-chkkh to v.us.8
  Normal  Pulled     2m4s  kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    2m4s  kubelet, v.us.8    Created container
  Normal  Started    2m4s  kubelet, v.us.8    Started container
  Normal  Pulled     2m3s  kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    2m3s  kubelet, v.us.8    Created container
  Normal  Started    2m3s  kubelet, v.us.8    Started container
  Normal  Pulled     117s  kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    117s  kubelet, v.us.8    Created container
  Normal  Started    116s  kubelet, v.us.8    Started container
  Normal  Pulled     116s  kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    116s  kubelet, v.us.8    Created container
  Normal  Started    115s  kubelet, v.us.8    Started container

Name:               rook-ceph-mon-b-74d8f8c4d8-pd9jg
Namespace:          rook-ceph
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:50:59 +0800
Labels:             app=rook-ceph-mon
                    ceph_daemon_id=b
                    mon=b
                    mon_cluster=rook-ceph
                    pod-template-hash=74d8f8c4d8
                    rook_cluster=rook-ceph
Annotations:        <none>
Status:             Running
IP:                 10.32.0.12
Controlled By:      ReplicaSet/rook-ceph-mon-b-74d8f8c4d8
Init Containers:
  config-init:
    Container ID:  docker://c44807cc893e966182b92243f2778e32548c51cb59f004fe9582fb1b94964f5e
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      mon-init
      --config-dir=/var/lib/rook
      --name=b
      --port=6789
      --fsid=27650da9-f702-4375-a03f-b2becc7dd19b
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:00 +0800
      Finished:     Sat, 26 Jan 2019 10:51:00 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_PRIVATE_IP:             (v1:status.podIP)
      ROOK_PUBLIC_IP:             10.108.82.238
      ROOK_CLUSTER_NAME:          rook-ceph
      ROOK_MON_ENDPOINTS:         <set to the key 'data' of config map 'rook-ceph-mon-endpoints'>  Optional: false
      ROOK_MON_SECRET:            <set to the key 'mon-secret' in secret 'rook-ceph-mon'>          Optional: false
      ROOK_ADMIN_SECRET:          <set to the key 'admin-secret' in secret 'rook-ceph-mon'>        Optional: false
      ROOK_CEPH_CONFIG_OVERRIDE:  /etc/rook/config/override.conf
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /etc/rook/config from rook-config-override (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
  monmap-init:
    Container ID:  docker://84f49bc588a3bf9d7920efbdec2f7c5800cc744e7d28e7d8413a330fa1f00fcd
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/monmaptool
    Args:
      /var/lib/rook/mon-b/monmap
      --create
      --clobber
      --fsid
      27650da9-f702-4375-a03f-b2becc7dd19b
      --add
      a
      10.108.69.163:6789
      --add
      b
      10.108.82.238:6789
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:01 +0800
      Finished:     Sat, 26 Jan 2019 10:51:06 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
  mon-fs-init:
    Container ID:  docker://0b8ed2af1e040ddf426ebaf283f98825e4a601b2ac36af84afb687096e57470c
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      ceph-mon
    Args:
      --mkfs
      --monmap
      /var/lib/rook/mon-b/monmap
      --name
      mon.b
      --mon-data
      /var/lib/rook/mon-b/data
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:07 +0800
      Finished:     Sat, 26 Jan 2019 10:51:07 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Containers:
  mon:
    Container ID:  docker://94a4a7123c16357735c5700574404d574becd6540072eae3da5d158db81fb7eb
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          6789/TCP
    Host Port:     0/TCP
    Command:
      ceph-mon
    Args:
      --foreground
      --public-addr
      10.108.82.238:6789
      --name
      mon.b
      --mon-data
      /var/lib/rook/mon-b/data
    State:          Running
      Started:      Sat, 26 Jan 2019 10:51:08 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:       rook-ceph-mon-b-74d8f8c4d8-pd9jg (v1:metadata.name)
      POD_NAMESPACE:  rook-ceph (v1:metadata.namespace)
      NODE_NAME:       (v1:spec.nodeName)
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rook-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook
    HostPathType:
  ceph-default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-config-override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-config-override
    Optional:  false
  default-token-99qjp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99qjp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=v.us.8
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  109s  default-scheduler  Successfully assigned rook-ceph/rook-ceph-mon-b-74d8f8c4d8-pd9jg to v.us.8
  Normal  Pulled     108s  kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    108s  kubelet, v.us.8    Created container
  Normal  Started    108s  kubelet, v.us.8    Started container
  Normal  Pulled     108s  kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    107s  kubelet, v.us.8    Created container
  Normal  Started    107s  kubelet, v.us.8    Started container
  Normal  Pulled     101s  kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    101s  kubelet, v.us.8    Created container
  Normal  Started    101s  kubelet, v.us.8    Started container
  Normal  Pulled     100s  kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    100s  kubelet, v.us.8    Created container
  Normal  Started    100s  kubelet, v.us.8    Started container

Name:               rook-ceph-mon-c-89f845f9b-sp5gb
Namespace:          rook-ceph
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:51:13 +0800
Labels:             app=rook-ceph-mon
                    ceph_daemon_id=c
                    mon=c
                    mon_cluster=rook-ceph
                    pod-template-hash=89f845f9b
                    rook_cluster=rook-ceph
Annotations:        <none>
Status:             Running
IP:                 10.32.0.10
Controlled By:      ReplicaSet/rook-ceph-mon-c-89f845f9b
Init Containers:
  config-init:
    Container ID:  docker://56b3a64cf0bcf14218914ff3aac1099e2cb3321eb90a9bb035e0081e75bdac81
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      mon-init
      --config-dir=/var/lib/rook
      --name=c
      --port=6789
      --fsid=27650da9-f702-4375-a03f-b2becc7dd19b
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:14 +0800
      Finished:     Sat, 26 Jan 2019 10:51:14 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_PRIVATE_IP:             (v1:status.podIP)
      ROOK_PUBLIC_IP:             10.101.235.61
      ROOK_CLUSTER_NAME:          rook-ceph
      ROOK_MON_ENDPOINTS:         <set to the key 'data' of config map 'rook-ceph-mon-endpoints'>  Optional: false
      ROOK_MON_SECRET:            <set to the key 'mon-secret' in secret 'rook-ceph-mon'>          Optional: false
      ROOK_ADMIN_SECRET:          <set to the key 'admin-secret' in secret 'rook-ceph-mon'>        Optional: false
      ROOK_CEPH_CONFIG_OVERRIDE:  /etc/rook/config/override.conf
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /etc/rook/config from rook-config-override (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
  monmap-init:
    Container ID:  docker://c108f45e90d305349c8a950ff3c55300a9c5b16a31a905e31683b20022411a46
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/monmaptool
    Args:
      /var/lib/rook/mon-c/monmap
      --create
      --clobber
      --fsid
      27650da9-f702-4375-a03f-b2becc7dd19b
      --add
      a
      10.108.69.163:6789
      --add
      b
      10.108.82.238:6789
      --add
      c
      10.101.235.61:6789
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:15 +0800
      Finished:     Sat, 26 Jan 2019 10:51:15 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
  mon-fs-init:
    Container ID:  docker://9cdec22e6c1d5d0984fab8e0f3f496c94fd7e55f41ebf034996133176b9eb361
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      ceph-mon
    Args:
      --mkfs
      --monmap
      /var/lib/rook/mon-c/monmap
      --name
      mon.c
      --mon-data
      /var/lib/rook/mon-c/data
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:16 +0800
      Finished:     Sat, 26 Jan 2019 10:51:16 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Containers:
  mon:
    Container ID:  docker://b49a0dba61a9c21a85ee3d8426b4f17139a6fc14016a18f2a94e08128e741ef4
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          6789/TCP
    Host Port:     0/TCP
    Command:
      ceph-mon
    Args:
      --foreground
      --public-addr
      10.101.235.61:6789
      --name
      mon.c
      --mon-data
      /var/lib/rook/mon-c/data
    State:          Running
      Started:      Sat, 26 Jan 2019 10:51:17 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:       rook-ceph-mon-c-89f845f9b-sp5gb (v1:metadata.name)
      POD_NAMESPACE:  rook-ceph (v1:metadata.namespace)
      NODE_NAME:       (v1:spec.nodeName)
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rook-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook
    HostPathType:
  ceph-default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-config-override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-config-override
    Optional:  false
  default-token-99qjp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-99qjp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=v.us.8
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  95s   default-scheduler  Successfully assigned rook-ceph/rook-ceph-mon-c-89f845f9b-sp5gb to v.us.8
  Normal  Pulled     94s   kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    94s   kubelet, v.us.8    Created container
  Normal  Started    94s   kubelet, v.us.8    Started container
  Normal  Pulled     93s   kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    93s   kubelet, v.us.8    Created container
  Normal  Started    93s   kubelet, v.us.8    Started container
  Normal  Pulled     92s   kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    92s   kubelet, v.us.8    Created container
  Normal  Started    92s   kubelet, v.us.8    Started container
  Normal  Pulled     91s   kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    91s   kubelet, v.us.8    Created container
  Normal  Started    91s   kubelet, v.us.8    Started container

Name:               rook-ceph-osd-0-56d79fc479-rrwqf
Namespace:          rook-ceph
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:51:45 +0800
Labels:             app=rook-ceph-osd
                    ceph-osd-id=0
                    pod-template-hash=56d79fc479
                    rook_cluster=rook-ceph
Annotations:        <none>
Status:             Running
IP:                 10.32.0.15
Controlled By:      ReplicaSet/rook-ceph-osd-0-56d79fc479
Init Containers:
  config-init:
    Container ID:  docker://6e77fa1b850e3645315882f7af790b8f0142fa881e0b6724b782315bda974f16
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      osd
      init
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:46 +0800
      Finished:     Sat, 26 Jan 2019 10:51:46 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_NODE_NAME:             v.us.8
      ROOK_CLUSTER_ID:            13f6284d-2115-11e9-a969-560001d945cd
      ROOK_PRIVATE_IP:             (v1:status.podIP)
      ROOK_PUBLIC_IP:              (v1:status.podIP)
      ROOK_CLUSTER_NAME:          rook-ceph
      ROOK_MON_ENDPOINTS:         <set to the key 'data' of config map 'rook-ceph-mon-endpoints'>  Optional: false
      ROOK_MON_SECRET:            <set to the key 'mon-secret' in secret 'rook-ceph-mon'>          Optional: false
      ROOK_ADMIN_SECRET:          <set to the key 'admin-secret' in secret 'rook-ceph-mon'>        Optional: false
      ROOK_CONFIG_DIR:            /var/lib/rook
      ROOK_CEPH_CONFIG_OVERRIDE:  /etc/rook/config/override.conf
      ROOK_FSID:                  <set to the key 'fsid' in secret 'rook-ceph-mon'>  Optional: false
      ROOK_OSD_DATABASE_SIZE:     1024
      ROOK_OSD_JOURNAL_SIZE:      1024
      ROOK_OSDS_PER_DEVICE:       1
      TINI_SUBREAPER:
      ROOK_OSD_ID:                0
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /etc/rook/config from rook-config-override (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
  copy-bins:
    Container ID:  docker://dfffa73630bc958bdb8f53d8c170753384e5cf864f43aa79b43f087e011c41d9
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      osd
      copybins
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:47 +0800
      Finished:     Sat, 26 Jan 2019 10:51:47 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_PATH:  /rook
    Mounts:
      /rook from rook-binaries (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
Containers:
  osd:
    Container ID:  docker://80754f875fb92c04b73d63b61a10742a3c61937cd2aa5cce78e6f479e1fa97a2
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      ceph-osd
    Args:
      --foreground
      --id
      0
      --conf
      /var/lib/rook/osd0/rook-ceph.config
      --osd-data
      /var/lib/rook/osd0
      --keyring
      /var/lib/rook/osd0/keyring
      --cluster
      rook-ceph
      --osd-uuid
      eb7bf21b-20eb-42e8-b818-6f057f041428
      --osd-journal=/var/lib/rook/osd0/journal
    State:          Running
      Started:      Sat, 26 Jan 2019 10:51:48 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      ROOK_NODE_NAME:       v.us.8
      ROOK_PRIVATE_IP:       (v1:status.podIP)
      ROOK_PUBLIC_IP:        (v1:status.podIP)
      TINI_SUBREAPER:
      POD_NAME:             rook-ceph-osd-0-56d79fc479-rrwqf (v1:metadata.name)
      POD_NAMESPACE:        rook-ceph (v1:metadata.namespace)
      NODE_NAME:             (v1:spec.nodeName)
      ROOK_OSD_UUID:        eb7bf21b-20eb-42e8-b818-6f057f041428
      ROOK_OSD_ID:          0
      ROOK_OSD_STORE_TYPE:  filestore
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /rook from rook-binaries (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rook-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook
    HostPathType:
  ceph-default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-config-override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-config-override
    Optional:  false
  rook-binaries:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-ceph-osd-token-rbdvz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-osd-token-rbdvz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=v.us.8
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  63s   default-scheduler  Successfully assigned rook-ceph/rook-ceph-osd-0-56d79fc479-rrwqf to v.us.8
  Normal  Pulled     62s   kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    62s   kubelet, v.us.8    Created container
  Normal  Started    62s   kubelet, v.us.8    Started container
  Normal  Pulled     61s   kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    61s   kubelet, v.us.8    Created container
  Normal  Started    61s   kubelet, v.us.8    Started container
  Normal  Pulled     60s   kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    60s   kubelet, v.us.8    Created container
  Normal  Started    60s   kubelet, v.us.8    Started container

Name:               rook-ceph-osd-prepare-v.us.8-5zhnn
Namespace:          rook-ceph
Priority:           0
PriorityClassName:  <none>
Node:               v.us.8/66.42.110.223
Start Time:         Sat, 26 Jan 2019 10:51:40 +0800
Labels:             app=rook-ceph-osd-prepare
                    controller-uid=4e44413e-2115-11e9-a969-560001d945cd
                    job-name=rook-ceph-osd-prepare-v.us.8
                    rook_cluster=rook-ceph
Annotations:        <none>
Status:             Succeeded
IP:                 10.32.0.14
Controlled By:      Job/rook-ceph-osd-prepare-v.us.8
Containers:
  copy-bins:
    Container ID:  docker://4bbcc98a2106f395691c26fbdd1a24f93f64c0632760e54a4b4c69888db48daa
    Image:         rook/ceph:master
    Image ID:      docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
    Port:          <none>
    Host Port:     <none>
    Args:
      ceph
      osd
      copybins
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:41 +0800
      Finished:     Sat, 26 Jan 2019 10:51:41 +0800
    Ready:          False
    Restart Count:  0
    Environment:
      ROOK_PATH:  /rook
    Mounts:
      /rook from rook-binaries (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
  provision:
    Container ID:  docker://9a57e04709c6994ecf3b0cc234066e5413e2fe797b67d918ccf43ddafa2d4bc5
    Image:         ceph/ceph:v13
    Image ID:      docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
    Port:          <none>
    Host Port:     <none>
    Command:
      /rook/tini
    Args:
      --
      /rook/rook
      ceph
      osd
      provision
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Jan 2019 10:51:41 +0800
      Finished:     Sat, 26 Jan 2019 10:51:45 +0800
    Ready:          False
    Restart Count:  0
    Environment:
      ROOK_NODE_NAME:             v.us.8
      ROOK_CLUSTER_ID:            13f6284d-2115-11e9-a969-560001d945cd
      ROOK_PRIVATE_IP:             (v1:status.podIP)
      ROOK_PUBLIC_IP:              (v1:status.podIP)
      ROOK_CLUSTER_NAME:          rook-ceph
      ROOK_MON_ENDPOINTS:         <set to the key 'data' of config map 'rook-ceph-mon-endpoints'>  Optional: false
      ROOK_MON_SECRET:            <set to the key 'mon-secret' in secret 'rook-ceph-mon'>          Optional: false
      ROOK_ADMIN_SECRET:          <set to the key 'admin-secret' in secret 'rook-ceph-mon'>        Optional: false
      ROOK_CONFIG_DIR:            /var/lib/rook
      ROOK_CEPH_CONFIG_OVERRIDE:  /etc/rook/config/override.conf
      ROOK_FSID:                  <set to the key 'fsid' in secret 'rook-ceph-mon'>  Optional: false
      ROOK_OSD_DATABASE_SIZE:     1024
      ROOK_OSD_JOURNAL_SIZE:      1024
      ROOK_OSDS_PER_DEVICE:       1
    Mounts:
      /etc/ceph from ceph-default-config-dir (rw)
      /rook from rook-binaries (rw)
      /var/lib/rook from rook-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  rook-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/rook
    HostPathType:
  ceph-default-config-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-config-override:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-config-override
    Optional:  false
  rook-binaries:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  rook-ceph-osd-token-rbdvz:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rook-ceph-osd-token-rbdvz
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=v.us.8
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  68s   default-scheduler  Successfully assigned rook-ceph/rook-ceph-osd-prepare-v.us.8-5zhnn to v.us.8
  Normal  Pulled     67s   kubelet, v.us.8    Container image "rook/ceph:master" already present on machine
  Normal  Created    67s   kubelet, v.us.8    Created container
  Normal  Started    67s   kubelet, v.us.8    Started container
  Normal  Pulled     67s   kubelet, v.us.8    Container image "ceph/ceph:v13" already present on machine
  Normal  Created    67s   kubelet, v.us.8    Created container
  Normal  Started    67s   kubelet, v.us.8    Started container

如果看到一直是 Init:CrashLoopBackOff 状态:

$ kubectl get pods --all-namespaces
NAMESPACE          NAME                                 READY   STATUS                  RESTARTS   AGE
kube-system        coredns-86c58d9df4-4prl5             1/1     Running                 0          16m
kube-system        coredns-86c58d9df4-zfqqm             1/1     Running                 0          16m
kube-system        etcd-v.us.8                          1/1     Running                 0          15m
kube-system        kube-apiserver-v.us.8                1/1     Running                 0          15m
kube-system        kube-controller-manager-v.us.8       1/1     Running                 0          15m
kube-system        kube-proxy-jt8xr                     1/1     Running                 0          16m
kube-system        kube-scheduler-v.us.8                1/1     Running                 0          15m
kube-system        weave-net-rn2rs                      2/2     Running                 0          13m
rook-ceph-system   rook-ceph-agent-9hzv2                1/1     Running                 0          12m
rook-ceph-system   rook-ceph-operator-76cf7f88f-ssr5q   1/1     Running                 0          13m
rook-ceph-system   rook-discover-jtndt                  1/1     Running                 0          12m
rook-ceph          rook-ceph-mon-a-6b87f9967-dvrk5      0/1     Init:CrashLoopBackOff   7          12m
rook-ceph          rook-ceph-mon-d-5646c6458f-rs97z     0/1     Init:CrashLoopBackOff   7          10m
rook-ceph          rook-ceph-mon-f-86dd4765db-fzmzd     0/1     Init:CrashLoopBackOff   6          9m20s

显示部署信息,发现如下错误:

$ kubectl describe pods -n rook-ceph
      Message:      The keyring does not match the existing keyring in /var/lib/rook/mon-a/data/keyring. You may need to delete the contents of dataDirHostPath on the host from a previous deployment.

由于是重新部署,所以安装前需要清理残留文件后稍等片刻即可自动恢复:

rm -rf /var/lib/rook/mon-*
  • 创建 PV
$ cat kubernetes-plugins-rook-storage.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  replicated:
    size: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
provisioner: ceph.rook.io/block
parameters:
  blockPool: replicapool
  # Specify the namespace of the rook cluster from which to create volumes.
  # If not specified, it will use `rook` as the default namespace of the cluster.
  # This is also the namespace where the cluster will be
  clusterNamespace: rook-ceph
  # Specify the filesystem type of the volume. If not specified, it will use `ext4`.
  #fstype: xfs
  # (Optional) Specify an existing Ceph user that will be used for mounting storage with this StorageClass.
  #mountUser: user1
  # (Optional) Specify an existing Kubernetes secret name containing just one key holding the Ceph user secret.
  # The secret must exist in each namespace(s) where the storage will be consumed.
  #mountSecret: ceph-user1-secret

使用配置安装

$ kubectl apply -f kubernetes-plugins-rook-storage.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block-oss created

参考 storageclass.yaml

  • 此时查看 k8s 集群部署详情:
$ kubectl get nodes,po,svc --all-namespaces
NAME          STATUS   ROLES    AGE     VERSION
node/v.us.8   Ready    master   7m37s   v1.13.2

NAMESPACE          NAME                                     READY   STATUS      RESTARTS   AGE
kube-system        pod/coredns-86c58d9df4-8gzz2             1/1     Running     0          7m18s
kube-system        pod/coredns-86c58d9df4-lr6n5             1/1     Running     0          7m18s
kube-system        pod/etcd-v.us.8                          1/1     Running     0          6m35s
kube-system        pod/kube-apiserver-v.us.8                1/1     Running     0          6m22s
kube-system        pod/kube-controller-manager-v.us.8       1/1     Running     0          6m39s
kube-system        pod/kube-proxy-4gmbk                     1/1     Running     0          7m18s
kube-system        pod/kube-scheduler-v.us.8                1/1     Running     0          6m17s
kube-system        pod/weave-net-jb2tt                      2/2     Running     0          6m28s
rook-ceph-system   pod/rook-ceph-agent-pmvfs                1/1     Running     0          5m22s
rook-ceph-system   pod/rook-ceph-operator-76cf7f88f-bxfhj   1/1     Running     0          5m55s
rook-ceph-system   pod/rook-discover-g7psj                  1/1     Running     0          5m22s
rook-ceph          pod/rook-ceph-mgr-a-8649f78d9b-wwxrc     1/1     Running     0          4m25s
rook-ceph          pod/rook-ceph-mon-a-f5cb646cb-chkkh      1/1     Running     0          5m6s
rook-ceph          pod/rook-ceph-mon-b-74d8f8c4d8-pd9jg     1/1     Running     0          4m50s
rook-ceph          pod/rook-ceph-mon-c-89f845f9b-sp5gb      1/1     Running     0          4m36s
rook-ceph          pod/rook-ceph-osd-0-56d79fc479-rrwqf     1/1     Running     0          4m4s
rook-ceph          pod/rook-ceph-osd-prepare-v.us.8-5zhnn   0/2     Completed   0          4m9s

NAMESPACE     NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes                ClusterIP   10.96.0.1        <none>        443/TCP         7m37s
kube-system   service/kube-dns                  ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP   7m33s
rook-ceph     service/rook-ceph-mgr             ClusterIP   10.105.220.216   <none>        9283/TCP        4m11s
rook-ceph     service/rook-ceph-mgr-dashboard   ClusterIP   10.99.44.224     <none>        8443/TCP        4m11s
rook-ceph     service/rook-ceph-mon-a           ClusterIP   10.108.69.163    <none>        6789/TCP        5m6s
rook-ceph     service/rook-ceph-mon-b           ClusterIP   10.108.82.238    <none>        6789/TCP        4m50s
rook-ceph     service/rook-ceph-mon-c           ClusterIP   10.101.235.61    <none>        6789/TCP        4m36s
  • 如果安装失败,重新初始化集群环境
kubeadm reset
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X && sudo iptables -L

从节点 v.cs.8 部署

  • 安装基础软件
sudo apt install kubelet kubeadm kubectl
  • 初始化加入集群
Tue Jan 15 22:32:09 coam@v.cs.8:~/docker-us$ sudo kubeadm join 172.31.141.97:6443 --token cd7m0v.c8w082mh057x499u --discovery-token-ca-cert-hash sha256:db45ba01d83b7ed325295f8a9554c405238b5005bba22053eb85ca9a9638ec0e
* [preflight] Running pre-flight checks
	[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
* [discovery] Trying to connect to API Server "172.31.141.97:6443"
* [discovery] Created cluster-info discovery client, requesting info from "https://172.31.141.97:6443"
* [discovery] Requesting info from "https://172.31.141.97:6443" again to validate TLS against the pinned public key
* [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.31.141.97:6443"
* [discovery] Successfully established connection with API Server "172.31.141.97:6443"
* [join] Reading configuration from the cluster...
* [join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
* [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
* [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
* [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
* [kubelet-start] Activating the kubelet service
* [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
* [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "v.cs.8" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

配置开机自启动

所有节点 设置 kubelet 开机自启动

systemctl enable kubelet.service

在主节点 v.us.8 查看主节点状态

$ kubectl get nodes,po,svc --all-namespaces
NAME          STATUS   ROLES    AGE     VERSION
node/v.cs.8   Ready    <none>   3m26s   v1.13.2
node/v.us.8   Ready    master   6h3m    v1.13.2

NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-86c58d9df4-pn8lb         1/1     Running   0          6h3m
kube-system   pod/coredns-86c58d9df4-s6pbk         1/1     Running   0          6h3m
kube-system   pod/etcd-v.us.8                      1/1     Running   0          6h2m
kube-system   pod/kube-apiserver-v.us.8            1/1     Running   0          6h2m
kube-system   pod/kube-controller-manager-v.us.8   1/1     Running   0          6h2m
kube-system   pod/kube-flannel-ds-amd64-6l7s7      1/1     Running   0          3m26s
kube-system   pod/kube-flannel-ds-amd64-b29hm      1/1     Running   0          3m49s
kube-system   pod/kube-proxy-6fv94                 1/1     Running   0          6h3m
kube-system   pod/kube-proxy-r9nzh                 1/1     Running   0          3m26s
kube-system   pod/kube-scheduler-v.us.8            1/1     Running   0          6h2m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP         6h3m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP   6h3m

安装 Dashboard

如果点登陆没反应 记得注意官方文档这句话,也就是登录到 Dashboard 必须满足 HTTPS 访问,或者 HTTP + localhost 访问,不然点击登陆按钮不会有任何反应

  • 安装 Dashboard 控制面板
wget -O k8s-dashboard.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
$ kubectl apply -f k8s-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created

查看服务是否启动

$ kubectl get pods --all-namespaces
NAMESPACE          NAME                                   READY   STATUS      RESTARTS   AGE
kube-system        kubernetes-dashboard-57df4db6b-h4dxg   1/1     Running     0          37s

创建访问 Dashboard 的账户

这里推荐使用第二种方式

1. 用 Dashboard 自带的角色添加权限

kubernetes-dashboardServiceAccount 绑定权限

$ cat k8s-dashboard-admin.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
 name: kubernetes-dashboard
 labels:
   k8s-app: kubernetes-dashboard
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: cluster-admin
subjects:
 - kind: ServiceAccount
   name: kubernetes-dashboard
   namespace: kube-system

执行安装

kubectl apply -f k8s-dashboard-admin.yaml

此账户获取 Token 的方式

kubectl describe secrets $(kubectl get secrets --namespace kube-system | grep dashboard-token | awk '{print $1}') --namespace kube-system | grep token: | awk '{print $2}'

2. 新建一个管理员

安全一点的方式是新建一个账户 赋予权限

$ cat k8s-dashboard-coam-role.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: coam-admin
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
  kind: ClusterRole
  name: cluster-coam-admin
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: coam-admin
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: coam-admin
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile

执行安装:

$ kubectl apply -f k8s-dashboard-coam-role.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin created
serviceaccount/admin created

此账户获取 Token 的方式

kubectl describe secrets $(kubectl get secrets --namespace kube-system | grep admin-token | awk '{print $1}') --namespace kube-system | grep token: | awk '{print $2}'
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi10b2tlbi05N3hzbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjI4ZGM1ZjdmLTIxMTgtMTFlOS1hOTY5LTU2MDAwMWQ5NDVjZCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTphZG1pbiJ9.hqxnX8Wwuj4CLzHletpLXbpe-D460EMZq9MZ3m0vfXuuFU41-Q8lsCfRFgzYyAV_aKNT96ps2jEPtJn9UXkqyb5mg8MuCAkFpLrHieGcPm2skKVGlIqBpIeweAQ80YCRAVGB1pDVsiXU_2pbbdCdvAKgsB0io1QTfo5EbxJn28qR4Aj6ke4YQx-yKPBK1i5h3IA4oS9Q0JiTxsbgXEnoSaZrqJ9Q9VSvcWVrj6oxb3Q9pwSeko_LRr0GDRXgINiZjxqz32Eshj6p4pG3JG3MOWvWDXO7WK2OkwnxwVUnfiAPx1qYYbgVAAx2zIOpiLrv5K5MLYZ8kzAnbPoXdm0SVw

访问面板

1. 通过 Proxy 方式访问:

第一种通过 kubectl proxy 暴露API (这种方法只能本地访问)

开启代理

kubectl proxy &

然后访问 http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login

为了让 kubernetes-dashboard 能通过公网 ip 访问,应携带以下参数:

kubectl proxy --address 0.0.0.0 --accept-hosts '.*'

参考 How allow access to Kubernetes-Dashboard from master real routed IP address?

通过以下链接访问

http://66.42.110.223:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

2. 通过新建服务对外暴露端口:

通过 NodeIP + NodePort 访问,此方法可以任意访问.但是存在证书问题,忽略即可

重新修改 k8s-dashboard.yaml 拉到底 找到 Service 区域 spec 改为 NodePort

修改 k8s-dashboard.yaml 文件

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
    - port: 8443
      targetPort: 8443
      nodePort: 30443
  selector:
    k8s-app: kubernetes-dashboard

重新部署,使其生效

$ kubectl apply -f k8s-dashboard.yaml
secret/kubernetes-dashboard-certs unchanged
secret/kubernetes-dashboard-csrf created
serviceaccount/kubernetes-dashboard unchanged
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal configured
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged
deployment.apps/kubernetes-dashboard unchanged
service/kubernetes-dashboard configured

访问 https://NodeIP:30443 如果提示 NET::ERR_CERT_INVALID 点击继续访问

使用 Token 登录面板

打开 Dashboard 之后 点击 Token 登录 输入上面获取到的 Token 登录即可

Dashboard 管理

  • 查看 dashboardpod 是否正常启动及外网暴露端口:
$ kubectl get nodes,po,svc --all-namespaces -o wide
NAME          STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
node/a.us.1   Ready    <none>   50m   v1.13.2   172.31.141.97   <none>        Ubuntu 18.04.1 LTS      4.15.0-43-generic           docker://18.6.1
node/v.cs.8   Ready    <none>   50m   v1.13.2   66.42.108.136   <none>        CentOS Linux 7 (Core)   3.10.0-957.1.3.el7.x86_64   docker://18.9.1
node/v.us.8   Ready    master   23h   v1.13.2   66.42.110.223   <none>        Ubuntu 18.10            4.18.0-10-generic           docker://18.9.1

NAMESPACE          NAME                                       READY   STATUS      RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
kube-system        pod/coredns-86c58d9df4-8gzz2               1/1     Running     0          23h   10.32.0.6       v.us.8   <none>           <none>
kube-system        pod/coredns-86c58d9df4-lr6n5               1/1     Running     0          23h   10.32.0.7       v.us.8   <none>           <none>
kube-system        pod/etcd-v.us.8                            1/1     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
kube-system        pod/kube-apiserver-v.us.8                  1/1     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
kube-system        pod/kube-controller-manager-v.us.8         1/1     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
kube-system        pod/kube-proxy-4gmbk                       1/1     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
kube-system        pod/kube-proxy-7mds5                       1/1     Running     0          50m   66.42.108.136   v.cs.8   <none>           <none>
kube-system        pod/kube-proxy-d6q5c                       1/1     Running     0          50m   172.31.141.97   a.us.1   <none>           <none>
kube-system        pod/kube-scheduler-v.us.8                  1/1     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
kube-system        pod/kubernetes-dashboard-57df4db6b-xnvnt   1/1     Running     0          23h   10.32.0.14      v.us.8   <none>           <none>
kube-system        pod/weave-net-6hmbd                        2/2     Running     0          50m   172.31.141.97   a.us.1   <none>           <none>
kube-system        pod/weave-net-7dt22                        2/2     Running     1          50m   66.42.108.136   v.cs.8   <none>           <none>
kube-system        pod/weave-net-jb2tt                        2/2     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
rook-ceph-system   pod/rook-ceph-agent-89vz7                  1/1     Running     0          50m   172.31.141.97   a.us.1   <none>           <none>
rook-ceph-system   pod/rook-ceph-agent-dmv66                  1/1     Running     0          50m   66.42.108.136   v.cs.8   <none>           <none>
rook-ceph-system   pod/rook-ceph-agent-pmvfs                  1/1     Running     0          23h   66.42.110.223   v.us.8   <none>           <none>
rook-ceph-system   pod/rook-ceph-operator-76cf7f88f-bxfhj     1/1     Running     0          23h   10.32.0.8       v.us.8   <none>           <none>
rook-ceph-system   pod/rook-discover-4wvz8                    1/1     Running     0          50m   10.36.0.1       v.cs.8   <none>           <none>
rook-ceph-system   pod/rook-discover-g7psj                    1/1     Running     0          23h   10.32.0.9       v.us.8   <none>           <none>
rook-ceph-system   pod/rook-discover-zwqn6                    1/1     Running     0          50m   10.44.0.1       a.us.1   <none>           <none>
rook-ceph          pod/rook-ceph-mgr-a-8649f78d9b-wwxrc       1/1     Running     0          23h   10.32.0.13      v.us.8   <none>           <none>
rook-ceph          pod/rook-ceph-mon-a-f5cb646cb-chkkh        1/1     Running     0          23h   10.32.0.11      v.us.8   <none>           <none>
rook-ceph          pod/rook-ceph-mon-b-74d8f8c4d8-pd9jg       1/1     Running     0          23h   10.32.0.12      v.us.8   <none>           <none>
rook-ceph          pod/rook-ceph-mon-c-89f845f9b-sp5gb        1/1     Running     0          23h   10.32.0.10      v.us.8   <none>           <none>
rook-ceph          pod/rook-ceph-osd-0-56d79fc479-rrwqf       1/1     Running     0          23h   10.32.0.15      v.us.8   <none>           <none>
rook-ceph          pod/rook-ceph-osd-prepare-v.us.8-5zhnn     0/2     Completed   0          23h   10.32.0.14      v.us.8   <none>           <none>

NAMESPACE     NAME                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE   SELECTOR
default       service/kubernetes                ClusterIP   10.96.0.1        <none>        443/TCP          23h   <none>
kube-system   service/kube-dns                  ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP    23h   k8s-app=kube-dns
kube-system   service/kubernetes-dashboard      NodePort    10.100.112.7     <none>        8443:30443/TCP   23h   k8s-app=kubernetes-dashboard
rook-ceph     service/rook-ceph-mgr             ClusterIP   10.105.220.216   <none>        9283/TCP         23h   app=rook-ceph-mgr,rook_cluster=rook-ceph
rook-ceph     service/rook-ceph-mgr-dashboard   ClusterIP   10.99.44.224     <none>        8443/TCP         23h   app=rook-ceph-mgr,rook_cluster=rook-ceph
rook-ceph     service/rook-ceph-mon-a           ClusterIP   10.108.69.163    <none>        6789/TCP         23h   app=rook-ceph-mon,ceph_daemon_id=a,mon=a,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph     service/rook-ceph-mon-b           ClusterIP   10.108.82.238    <none>        6789/TCP         23h   app=rook-ceph-mon,ceph_daemon_id=b,mon=b,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph     service/rook-ceph-mon-c           ClusterIP   10.101.235.61    <none>        6789/TCP         23h   app=rook-ceph-mon,ceph_daemon_id=c,mon=c,mon_cluster=rook-ceph,rook_cluster=rook-ceph

部署镜像容器

通过 YAML 配置文件创建一个 nginx 部署:

$ wget -O kubernetes-deployment-nginx.yaml https://k8s.io/examples/application/deployment.yaml
$ kubectl apply -f kubernetes-deployment-nginx.yaml
deployment.apps/nginx-deployment created

配置文件示例

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

查看部署状态:

$ kubectl describe deployment nginx-deployment
Name:                   nginx-deployment
Namespace:              default
CreationTimestamp:      Sun, 27 Jan 2019 10:34:19 +0800
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},"spec":{"replica...
Selector:               app=nginx
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=nginx
  Containers:
   nginx:
    Image:        nginx:1.7.9
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   nginx-deployment-76bf4969df (2/2 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  22s   deployment-controller  Scaled up replica set nginx-deployment-76bf4969df to 2

查看这次部署创建的 pods:

$ kubectl get pods -l app=nginx
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-76bf4969df-7q4c8   1/1     Running   0          79s
nginx-deployment-76bf4969df-q6dxd   1/1     Running   0          79s

删除部署

kubectl delete deployment nginx-deployment

完全删除安装(会删除所有包括Docker)

kubeadm reset -f
yum remove docker* kubeadm kubectl kubelet -y
ip link delete docker0
ip link delete cni0
ip link delete weave
ip link delete flannel.1
rpm -e $(rpm -qa | grep docker)

  • 如果你的集群安装过程中遇到了其他问题,我们可以使用下面的命令来进行重置:
$ kubeadm reset
$ ifconfig cni0 down && ip link delete cni0
$ ifconfig flannel.1 down && ip link delete flannel.1
$ rm -rf /var/lib/cni/
  • 查看节点状态

curl https://47.244.154.194:6443 -k

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
}
  • 查看集群状态
Sat Jan 19 17:34:35 coam@v.us.8:~$ kubectl cluster-info
Kubernetes master is running at https://66.42.110.223:6443
KubeDNS is running at https://66.42.110.223:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
  • 查看节点状态
$ kubectl get nodes,po,svc --all-namespaces
NAME          STATUS   ROLES    AGE   VERSION
node/v.cs.8   Ready    <none>   51m   v1.13.2
node/v.us.8   Ready    master   83m   v1.13.2

NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-86c58d9df4-bmhwl               1/1     Running   0          83m
kube-system   pod/coredns-86c58d9df4-slksc               1/1     Running   0          83m
kube-system   pod/etcd-v.us.8                            1/1     Running   0          82m
kube-system   pod/kube-apiserver-v.us.8                  1/1     Running   0          82m
kube-system   pod/kube-controller-manager-v.us.8         1/1     Running   0          82m
kube-system   pod/kube-flannel-ds-amd64-8c5f5            1/1     Running   0          81m
kube-system   pod/kube-flannel-ds-amd64-n9j64            1/1     Running   0          51m
kube-system   pod/kube-proxy-7p2pq                       1/1     Running   0          51m
kube-system   pod/kube-proxy-nm6mc                       1/1     Running   0          83m
kube-system   pod/kube-scheduler-v.us.8                  1/1     Running   0          82m
kube-system   pod/kubernetes-dashboard-57df4db6b-pjlzw   1/1     Running   0          40m

NAMESPACE     NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       service/kubernetes             ClusterIP   10.96.0.1       <none>        443/TCP         83m
kube-system   service/kube-dns               ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP   83m
kube-system   service/kubernetes-dashboard   ClusterIP   10.98.123.100   <none>        443/TCP         40m

  • kubectl

#* 启用主控机同时作为工作节点

#kubectl taint nodes --all node-role.kubernetes.io/master

获取集群配置信息

kubectl -n kube-system get cm kubeadm-config -oyaml

获取集群节点信息

kubectl get nodes

获取组件的健康状态

kubectl get cs

安装网络插件

wget -O k8s-plugins-kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sudo kubectl apply -f k8s-plugins-kube-flannel.yml

执行如下的命令,获取当前系统上所有在运行的 pod 的状态,指定名称空间为 kube-system,为系统级的 pod,命令如下

Thu Jan 17 22:51:51 coam@a.us.1:~$ sudo kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-86c58d9df4-d6tb2         1/1     Running   0          4h57m
coredns-86c58d9df4-drnjl         1/1     Running   0          4h57m
etcd-a.us.1                      1/1     Running   1          4h57m
kube-apiserver-a.us.1            1/1     Running   1          4h56m
kube-controller-manager-a.us.1   1/1     Running   1          4h57m
kube-flannel-ds-amd64-5rlxt      1/1     Running   0          91s
kube-flannel-ds-amd64-nbvpf      1/1     Running   0          91s
kube-proxy-2jwhp                 1/1     Running   1          4h57m
kube-proxy-l7shz                 1/1     Running   0          8m6s
kube-scheduler-a.us.1            1/1     Running   1          4h56m

需要查看 pod 的详细信息,则添加选项 -o wide

Thu Jan 17 22:56:18 coam@a.us.1:~$ kubectl get pods -n kube-system -o wide
NAME                             READY   STATUS    RESTARTS   AGE    IP              NODE     NOMINATED NODE   READINESS GATES
coredns-86c58d9df4-d6tb2         1/1     Running   0          5h3m   10.244.0.4      a.us.1   <none>           <none>
coredns-86c58d9df4-drnjl         1/1     Running   0          5h3m   10.244.1.2      a.us.0   <none>           <none>
etcd-a.us.1                      1/1     Running   1          5h2m   172.31.141.97   a.us.1   <none>           <none>
kube-apiserver-a.us.1            1/1     Running   1          5h2m   172.31.141.97   a.us.1   <none>           <none>
kube-controller-manager-a.us.1   1/1     Running   1          5h2m   172.31.141.97   a.us.1   <none>           <none>
kube-flannel-ds-amd64-5rlxt      1/1     Running   0          7m2s   172.31.141.98   a.us.0   <none>           <none>
kube-flannel-ds-amd64-nbvpf      1/1     Running   0          7m2s   172.31.141.97   a.us.1   <none>           <none>
kube-proxy-2jwhp                 1/1     Running   1          5h3m   172.31.141.97   a.us.1   <none>           <none>
kube-proxy-l7shz                 1/1     Running   0          13m    172.31.141.98   a.us.0   <none>           <none>
kube-scheduler-a.us.1            1/1     Running   1          5h2m   172.31.141.97   a.us.1   <none>           <none>

执行如下命令,获取当前系统的名称空间

Thu Jan 17 22:53:18 coam@a.us.1:~$ kubectl get ns
NAME          STATUS   AGE
default       Active   4h59m
kube-public   Active   4h59m
kube-system   Active   4h59m

拆卸集群

首先处理各节点:

kubectl drain v.cs.8 --delete-local-data --force --ignore-daemonsets
kubectl delete node v.cs.8

一旦节点移除之后,则可以执行如下命令来重置集群:

kubeadm reset

kubernetes 更新

  • 检查是否有更新
kubeadm upgrade plan
  • 更新到指定版本
kubeadm upgrade apply v1.13.0

  • kubeadm

注意: 在阿里云的专有网络上通过参数指定外网ip --apiserver-advertise-address=47.244.154.194 无法启动,需要添加参数 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=47.244.154.194 方式启动:

IP证书问题参考 Invalid x509 certificate for kubernetes master

sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=47.244.154.194

几种集群启动参数备注

kubeadm reset
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=149.28.93.7

# 外网ip - 启动失败
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=47.244.154.194 --apiserver-cert-extra-sans=47.244.154.194

# 内网ip - 启动成功
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.31.141.97

# 通用ip - 启动成功
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0

忽略交换分区错误: –ignore-preflight-errors=Swap

  • 列出 token
kubeadm token list
  • 创建新的 token
kubeadm token create --print-join-command
  • 加入 Kubernetes 集群网络
sudo kubeadm join 47.244.154.194:6443 --token cq3dja.33tdj0lzoqncu0h0 --discovery-token-ca-cert-hash sha256:16131f78ab6af4de0c228ef2a9b27bab5e49f76a22532f4249759ae432e77dd2

从节点同样需要到互联网上拉取 flannel 的镜像,并启动,因此需要一点时间,用 docker images 查看是否成功拉取 flannel 镜像到本地,等 flannel 拉取成功启动后,在 master 上可以看到 node 的状态为 Ready.


kubectl -n kube-system get cm kubeadm-config -oyaml
kubectl get pods --all-namespaces
kubectl get pods -n kube-system -o wide
/etc/cni/net.d/
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=66.42.110.223
sudo kubeadm join 66.42.110.223:6443 --token j3ft16.l3qw4c7jga8ft873 --discovery-token-ca-cert-hash sha256:d892e13c2d7705c6503145a667b6bfa6efd9667295898a1c337ab8cf406cc394

Comments

Cor-Ethan, the beverage → www.iirii.com