Kubernetes 部署
Summary: Author: 张亚飞 | Read Time: 37 minute read | Published: 2018-12-22
Filed under
—
Categories:
MarkDown
—
Tags:
Tag,
Reference:
Kubernetes 集群安装
本文准备部署一个 一主两从 的 三节点 Kubernetes 集群,整体节点规划如下表所示:
| 主机名 | IP | 角色 |
|---|---|---|
| v.us.8 | 66.42.110.223 | k8s主节点 |
| v.cs.8 | 66.42.108.136 | k8s从节点 |
| a.us.1 | 47.244.154.194 | k8s从节点 |
| 47.89.12.166 | k8s从节点 | |
| 103.37.147.250 | k8s从节点 |
准备工作
关闭系统的 Swap 交换分区,方法如下:
sudo swapoff -a
free –h
调整系统参数 vm.swappiness:
echo "vm.swappiness=0" >> /etc/sysctl.conf && sysctl -p
同时需要修改 /etc/fstab 注释 Swap 相关的条目
sed -i '/swap/s/^/#/' /etc/fstab
Docker 从 1.13 版本开始调整了默认的防火墙规则,禁用了 iptables filter 表中 FOWARD 链,这样会引起 Kubernetes 集群中跨 Node 的 Pod 无法通信,在各个 Docker 节点执行下面的命令:
sudo iptables -P FORWARD ACCEPT
其它备注
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X && sudo iptables -L
sudo ipvsadm -C
开启 IP 转发 不然会导致容器无法访问网络
echo net.ipv4.ip_forward=1 >> /etc/sysctl.conf
echo net.bridge.bridge-nf-call-iptables=1 >> /etc/sysctl.conf
echo net.bridge.bridge-nf-call-ip6tables=1 >> /etc/sysctl.conf
sysctl -p
安装 Docker
...此处省略
安装 Kubernetes-v1.13.2
安装基础支持软件:
- 添加
Kubernetes官方源
- Ubuntu
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
vi /etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
注意: 墙内按以下方式安装:
增加 kubernetes aliyun 镜像源
sudo apt-get update && sudo apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg |sudo apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
安装
kubelet、kubeadm及kubectl套件
sudo apt update
sudo apt install kubelet kubectl kubeadm
CentOS7
配置安装源,修改文件 /etc/yum.repos.d/kubernetes.repo 内容如下
* [kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
禁用 selinux
setenforce 0
安装 kubelet 和 kubeadm
yum update
yum install -y kubelet kubeadm
4 启用 kubelet
systemctl enable kubelet && systemctl start kubelet
kubeadm安装完以后,就可以使用它来快速安装部署Kubernetes集群了.
重新安装部署
首先删除已安装的残留文件:
sudo kubeadm reset
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X && sudo iptables -L
sudo rm -rf /var/lib/rook
使用 kubeadm 安装 Kubernetes 集群
在做好了准备工作之后,下面介绍如何使用
kubeadm安装Kubernetes集群,我们将首先安装master节点,然后将slave节点一个个加入到集群中去.
- 首先查看需要安装的镜像:
Wed Jan 23 13:48:50 coam@v.us.8:~$ kubeadm config images list
k8s.gcr.io/kube-apiserver:v1.13.2
k8s.gcr.io/kube-controller-manager:v1.13.2
k8s.gcr.io/kube-scheduler:v1.13.2
k8s.gcr.io/kube-proxy:v1.13.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.2.24
k8s.gcr.io/coredns:1.2.6
- 使用
kubeadm初始化master节点
指定
IP地址及kubernetes版本1.13.2:
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --apiserver-advertise-address=172.31.141.97
使用初始化配置初始化 Master 节点
参考 kubernetes: k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1
$ cat kubeadm-cluster-init.yml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.13.2
clusterName: kubernetes
imageRepository: k8s.gcr.io
certificatesDir: /etc/kubernetes/pki
controlPlaneEndpoint: ""
controllerManager: {}
apiServer:
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
使用配置文件初始化:
sudo kubeadm init --config kubeadm-cluster-init.yml
查看其它默认配置
kubeadm config print init-defaults
kubeadm config print join-defaults
查看运行配置:
kubeadm config view
附启动过程:
$ sudo kubeadm init --config kubeadm-cluster-init.yml
* [init] Using Kubernetes version: v1.13.2
* [preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.1. Latest validated version: 18.06
* [preflight] Pulling images required for setting up a Kubernetes cluster
* [preflight] This might take a minute or two, depending on the speed of your internet connection
* [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
* [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
* [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
* [kubelet-start] Activating the kubelet service
* [certs] Using certificateDir folder "/etc/kubernetes/pki"
* [certs] Generating "front-proxy-ca" certificate and key
* [certs] Generating "front-proxy-client" certificate and key
* [certs] Generating "etcd/ca" certificate and key
* [certs] Generating "etcd/server" certificate and key
* [certs] etcd/server serving cert is signed for DNS names [v.us.8 localhost] and IPs [66.42.110.223 127.0.0.1 ::1]
* [certs] Generating "etcd/peer" certificate and key
* [certs] etcd/peer serving cert is signed for DNS names [v.us.8 localhost] and IPs [66.42.110.223 127.0.0.1 ::1]
* [certs] Generating "etcd/healthcheck-client" certificate and key
* [certs] Generating "apiserver-etcd-client" certificate and key
* [certs] Generating "ca" certificate and key
* [certs] Generating "apiserver" certificate and key
* [certs] apiserver serving cert is signed for DNS names [v.us.8 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 66.42.110.223]
* [certs] Generating "apiserver-kubelet-client" certificate and key
* [certs] Generating "sa" key and public key
* [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
* [kubeconfig] Writing "admin.conf" kubeconfig file
* [kubeconfig] Writing "kubelet.conf" kubeconfig file
* [kubeconfig] Writing "controller-manager.conf" kubeconfig file
* [kubeconfig] Writing "scheduler.conf" kubeconfig file
* [control-plane] Using manifest folder "/etc/kubernetes/manifests"
* [control-plane] Creating static Pod manifest for "kube-apiserver"
* [control-plane] Creating static Pod manifest for "kube-controller-manager"
* [control-plane] Creating static Pod manifest for "kube-scheduler"
* [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
* [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
* [apiclient] All control plane components are healthy after 20.502192 seconds
* [uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
* [kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
* [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "v.us.8" as an annotation
* [mark-control-plane] Marking the node v.us.8 as control-plane by adding the label "node-role.kubernetes.io/master=''"
* [mark-control-plane] Marking the node v.us.8 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
* [bootstrap-token] Using token: vv296e.4j26mg3pqnvnxhcj
* [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
* [bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
* [bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
* [bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
* [bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
* [addons] Applied essential addon: CoreDNS
* [addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 66.42.110.223:6443 --token vv296e.4j26mg3pqnvnxhcj --discovery-token-ca-cert-hash sha256:3c480fd2249f99a9727e50e58f189c6510cab952ca49c551fc238792b3224e00
按提示要求拷贝用户配置文件
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
- 查看主节点状态
$ kubectl get nodes,po,svc --all-namespaces
NAME STATUS ROLES AGE VERSION
node/v.us.8 NotReady master 88s v1.13.2
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-86c58d9df4-9gprm 0/1 Pending 0 69s
kube-system pod/coredns-86c58d9df4-h729x 0/1 Pending 0 69s
kube-system pod/etcd-v.us.8 1/1 Running 0 13s
kube-system pod/kube-apiserver-v.us.8 1/1 Running 0 23s
kube-system pod/kube-controller-manager-v.us.8 1/1 Running 0 21s
kube-system pod/kube-proxy-lcx69 1/1 Running 0 69s
kube-system pod/kube-scheduler-v.us.8 1/1 Running 0 7s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 88s
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 83s
这里显示节点都是
NotReady原因可以用kubectl describe node <节点状态>看到:
$ kubectl describe node v.us.8
Name: v.us.8
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/hostname=v.us.8
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Fri, 25 Jan 2019 20:47:35 +0800
Taints: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoSchedule
Unschedulable: false
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Fri, 25 Jan 2019 20:49:05 +0800 Fri, 25 Jan 2019 20:47:30 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 25 Jan 2019 20:49:05 +0800 Fri, 25 Jan 2019 20:47:30 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 25 Jan 2019 20:49:05 +0800 Fri, 25 Jan 2019 20:47:30 +0800 KubeletHasSufficientPID kubelet has sufficient PID available
Ready False Fri, 25 Jan 2019 20:49:05 +0800 Fri, 25 Jan 2019 20:47:30 +0800 KubeletNotReady runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
InternalIP: 66.42.110.223
Hostname: v.us.8
Capacity:
cpu: 2
ephemeral-storage: 61883672Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4039292Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 57031992021
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 3936892Ki
pods: 110
System Info:
Machine ID: 3a7c1ff61e60446a83df16b875b1f675
System UUID: 3a7c1ff6-1e60-446a-83df-16b875b1f675
Boot ID: 8fa74a63-7bdf-47e6-8629-5a343f64ec5e
Kernel Version: 4.18.0-10-generic
OS Image: Ubuntu 18.10
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://18.9.1
Kubelet Version: v1.13.2
Kube-Proxy Version: v1.13.2
PodCIDR: 10.244.0.0/24
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system etcd-v.us.8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 24s
kube-system kube-apiserver-v.us.8 250m (12%) 0 (0%) 0 (0%) 0 (0%) 34s
kube-system kube-controller-manager-v.us.8 200m (10%) 0 (0%) 0 (0%) 0 (0%) 32s
kube-system kube-proxy-lcx69 0 (0%) 0 (0%) 0 (0%) 0 (0%) 80s
kube-system kube-scheduler-v.us.8 100m (5%) 0 (0%) 0 (0%) 0 (0%) 18s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 550m (27%) 0 (0%)
memory 0 (0%) 0 (0%)
ephemeral-storage 0 (0%) 0 (0%)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 107s kubelet, v.us.8 Starting kubelet.
Normal NodeHasSufficientMemory 107s (x8 over 107s) kubelet, v.us.8 Node v.us.8 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 107s (x8 over 107s) kubelet, v.us.8 Node v.us.8 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 107s (x7 over 107s) kubelet, v.us.8 Node v.us.8 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 107s kubelet, v.us.8 Updated Node Allocatable limit across pods
Normal Starting 79s kube-proxy, v.us.8 Starting kube-proxy.
查看
coredns容器状态,可以在Event看到因为网络插件未就绪,所以容器没有启动成功
$ kubectl describe po $(kubectl get po -n=kube-system | grep coredns | tail -n 1 | awk '{print $1}') -n=kube-system
Name: coredns-86c58d9df4-lnt47
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: k8s-app=kube-dns
pod-template-hash=86c58d9df4
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/coredns-86c58d9df4
Containers:
coredns:
Image: k8s.gcr.io/coredns:1.2.6
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-kqgfm (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-kqgfm:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-kqgfm
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 7m12s (x2 over 7m12s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
安装扩展插件
安装 CNI 网络插件
- 安装
Weave网络插件:
wget -O k8s-plugins-weave-daemonset-k8s-1.8.yaml https://github.com/weaveworks/weave/releases/download/v2.5.1/weave-daemonset-k8s-1.8.yaml
$ kubectl apply -f k8s-plugins-weave-daemonset-k8s-1.8.yaml
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.extensions/weave-net created
- 再次查看主节点状态:
$ kubectl get nodes,po,svc --all-namespaces
NAME STATUS ROLES AGE VERSION
node/v.us.8 Ready master 3m2s v1.13.2
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-86c58d9df4-9gprm 1/1 Running 0 2m43s
kube-system pod/coredns-86c58d9df4-h729x 1/1 Running 0 2m43s
kube-system pod/etcd-v.us.8 1/1 Running 0 107s
kube-system pod/kube-apiserver-v.us.8 1/1 Running 0 117s
kube-system pod/kube-controller-manager-v.us.8 1/1 Running 0 115s
kube-system pod/kube-proxy-lcx69 1/1 Running 0 2m43s
kube-system pod/kube-scheduler-v.us.8 1/1 Running 0 101s
kube-system pod/weave-net-zgpfc 2/2 Running 0 27s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3m2s
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 2m57s
安装 Rook 存储插件
安装存储插件
安装 Rook + Ceph
git clone https://github.com/rook/rook.git
首先删除已安装的残留文件:
rm -rf /var/lib/rook
- 第一步:
Rook
$ kubectl create -f rook/cluster/examples/kubernetes/ceph/operator.yaml
namespace/rook-ceph-system created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
role.rbac.authorization.k8s.io/rook-ceph-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
serviceaccount/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
deployment.apps/rook-ceph-operator created
- 第二步:
Ceph
$ kubectl create -f rook/cluster/examples/kubernetes/ceph/cluster.yaml
namespace/rook-ceph created
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-mgr-system created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
cephcluster.ceph.rook.io/rook-ceph created
查看存储插件情况
$ kubectl describe pods -n rook-ceph-system
Name: rook-ceph-operator-76cf7f88f-6gpsn
Namespace: rook-ceph-system
Priority: 0
PriorityClassName: <none>
Node: <none>
Labels: app=rook-ceph-operator
pod-template-hash=76cf7f88f
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/rook-ceph-operator-76cf7f88f
Containers:
rook-ceph-operator:
Image: rook/ceph:master
Port: <none>
Host Port: <none>
Args:
ceph
operator
Environment:
ROOK_ALLOW_MULTIPLE_FILESYSTEMS: false
ROOK_LOG_LEVEL: INFO
ROOK_MON_HEALTHCHECK_INTERVAL: 45s
ROOK_MON_OUT_TIMEOUT: 300s
ROOK_DISCOVER_DEVICES_INTERVAL: 60m
ROOK_HOSTPATH_REQUIRES_PRIVILEGED: false
ROOK_ENABLE_SELINUX_RELABELING: true
ROOK_ENABLE_FSGROUP: true
NODE_NAME: (v1:spec.nodeName)
POD_NAME: rook-ceph-operator-76cf7f88f-6gpsn (v1:metadata.name)
POD_NAMESPACE: rook-ceph-system (v1:metadata.namespace)
Mounts:
/etc/ceph from default-config-dir (rw)
/var/lib/rook from rook-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-slt7h (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
rook-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-ceph-system-token-slt7h:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-system-token-slt7h
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 99s default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
- 有时候一个
pod创建之后一直是pending,没有日志,也没有pull镜像,describe的时候发现里面有一句话:1 node(s) had taints that the pod didn't tolerate.
1 node(s) had taints that the pod didn't tolerate.
直译意思是节点有了污点无法容忍,执行
kubectl get no -o yaml | grep taint -A 5之后发现该节点是不可调度的.这是因为kubernetes出于安全考虑默认情况下无法在master节点上部署pod,于是用下面方法解决:
因为我只有1个节点,要“计算资源”的话,就只能将master节点的taint去掉,否则普通的Pod默认不会调度上来.
kubectl taint nodes --all node-role.kubernetes.io/master-
列出所有
namespace中的所有pod
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-86c58d9df4-9gprm 1/1 Running 0 59m
kube-system coredns-86c58d9df4-h729x 1/1 Running 0 59m
kube-system etcd-v.us.8 1/1 Running 0 58m
kube-system kube-apiserver-v.us.8 1/1 Running 0 58m
kube-system kube-controller-manager-v.us.8 1/1 Running 0 58m
kube-system kube-proxy-lcx69 1/1 Running 0 59m
kube-system kube-scheduler-v.us.8 1/1 Running 0 58m
kube-system weave-net-zgpfc 2/2 Running 0 56m
rook-ceph-system rook-ceph-agent-8pdms 1/1 Running 0 26m
rook-ceph-system rook-ceph-operator-76cf7f88f-6gpsn 1/1 Running 0 54m
rook-ceph-system rook-discover-tn9n5 1/1 Running 0 26m
rook-ceph rook-ceph-mgr-a-8649f78d9b-r42xx 1/1 Running 0 24m
rook-ceph rook-ceph-mon-a-78c8b48c79-vrlsx 1/1 Running 0 25m
rook-ceph rook-ceph-mon-b-84f8979fd9-gfhpj 1/1 Running 0 25m
rook-ceph rook-ceph-mon-c-5b49565bf7-fc6lm 1/1 Running 0 25m
rook-ceph rook-ceph-osd-0-85f7c8c89-98vm5 1/1 Running 0 24m
rook-ceph rook-ceph-osd-prepare-v.us.8-sc7gw 0/2 Completed 0 24m
再次查看存储插件情况:
$ kubectl describe pods -n rook-ceph-system
Name: rook-ceph-agent-pmvfs
Namespace: rook-ceph-system
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:50:27 +0800
Labels: app=rook-ceph-agent
controller-revision-hash=547ff695d9
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 66.42.110.223
Controlled By: DaemonSet/rook-ceph-agent
Containers:
rook-ceph-agent:
Container ID: docker://1fb8f624697ff7a5cba8971b06e9971450e7fe3232e17d2df1507d9fae4b86ad
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
agent
State: Running
Started: Sat, 26 Jan 2019 10:50:28 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAMESPACE: rook-ceph-system (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
AGENT_MOUNT_SECURITY_MODE: Any
ROOK_ENABLE_SELINUX_RELABELING: true
ROOK_ENABLE_FSGROUP: true
Mounts:
/dev from dev (rw)
/flexmnt from flexvolume (rw)
/lib/modules from libmodules (rw)
/sys from sys (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-q8t87 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
flexvolume:
Type: HostPath (bare host directory volume)
Path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
HostPathType:
dev:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType:
sys:
Type: HostPath (bare host directory volume)
Path: /sys
HostPathType:
libmodules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
rook-ceph-system-token-q8t87:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-system-token-q8t87
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 105s default-scheduler Successfully assigned rook-ceph-system/rook-ceph-agent-pmvfs to v.us.8
Normal Pulled 104s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 104s kubelet, v.us.8 Created container
Normal Started 104s kubelet, v.us.8 Started container
Name: rook-ceph-operator-76cf7f88f-bxfhj
Namespace: rook-ceph-system
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:50:26 +0800
Labels: app=rook-ceph-operator
pod-template-hash=76cf7f88f
Annotations: <none>
Status: Running
IP: 10.32.0.8
Controlled By: ReplicaSet/rook-ceph-operator-76cf7f88f
Containers:
rook-ceph-operator:
Container ID: docker://c21dd15e4b9719b71c002e796d231f8fcc2df10aa0f85a128595cf419e90641e
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
operator
State: Running
Started: Sat, 26 Jan 2019 10:50:27 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_ALLOW_MULTIPLE_FILESYSTEMS: false
ROOK_LOG_LEVEL: INFO
ROOK_MON_HEALTHCHECK_INTERVAL: 45s
ROOK_MON_OUT_TIMEOUT: 300s
ROOK_DISCOVER_DEVICES_INTERVAL: 60m
ROOK_HOSTPATH_REQUIRES_PRIVILEGED: false
ROOK_ENABLE_SELINUX_RELABELING: true
ROOK_ENABLE_FSGROUP: true
NODE_NAME: (v1:spec.nodeName)
POD_NAME: rook-ceph-operator-76cf7f88f-bxfhj (v1:metadata.name)
POD_NAMESPACE: rook-ceph-system (v1:metadata.namespace)
Mounts:
/etc/ceph from default-config-dir (rw)
/var/lib/rook from rook-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-q8t87 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rook-config:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-ceph-system-token-q8t87:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-system-token-q8t87
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m18s (x2 over 2m18s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
Normal Scheduled 106s default-scheduler Successfully assigned rook-ceph-system/rook-ceph-operator-76cf7f88f-bxfhj to v.us.8
Normal Pulled 105s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 105s kubelet, v.us.8 Created container
Normal Started 105s kubelet, v.us.8 Started container
Name: rook-discover-g7psj
Namespace: rook-ceph-system
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:50:27 +0800
Labels: app=rook-discover
controller-revision-hash=6585f46f49
pod-template-generation=1
Annotations: <none>
Status: Running
IP: 10.32.0.9
Controlled By: DaemonSet/rook-discover
Containers:
rook-discover:
Container ID: docker://a61b20fe2d4453843bdb19b5ae9a41ce3d0f1392cf1500e8d73510890cfded79
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
discover
--discover-interval
60m
State: Running
Started: Sat, 26 Jan 2019 10:50:28 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAMESPACE: rook-ceph-system (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
Mounts:
/dev from dev (rw)
/run/udev from udev (ro)
/sys from sys (ro)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-system-token-q8t87 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
dev:
Type: HostPath (bare host directory volume)
Path: /dev
HostPathType:
sys:
Type: HostPath (bare host directory volume)
Path: /sys
HostPathType:
udev:
Type: HostPath (bare host directory volume)
Path: /run/udev
HostPathType:
rook-ceph-system-token-q8t87:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-system-token-q8t87
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 105s default-scheduler Successfully assigned rook-ceph-system/rook-discover-g7psj to v.us.8
Normal Pulled 104s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 104s kubelet, v.us.8 Created container
Normal Started 104s kubelet, v.us.8 Started container
查看成功安装的
pods信息:
$ kubectl describe pods -n rook-ceph
Name: rook-ceph-mgr-a-8649f78d9b-wwxrc
Namespace: rook-ceph
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:51:24 +0800
Labels: app=rook-ceph-mgr
ceph_daemon_id=a
instance=a
mgr=a
pod-template-hash=8649f78d9b
rook_cluster=rook-ceph
Annotations: prometheus.io/port: 9283
prometheus.io/scrape: true
Status: Running
IP: 10.32.0.13
Controlled By: ReplicaSet/rook-ceph-mgr-a-8649f78d9b
Init Containers:
config-init:
Container ID: docker://dcc20ca41bc1d165f230aa91212c050f03f5ed0de1a4f9fc06766dcd9eee0e85
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
mgr-init
--config-dir=/var/lib/rook
--mgr-name=a
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:25 +0800
Finished: Sat, 26 Jan 2019 10:51:27 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_MGR_KEYRING: <set to the key 'keyring' in secret 'rook-ceph-mgr-a'> Optional: false
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: (v1:status.podIP)
ROOK_MGR_MODULE_SERVER_ADDR: (v1:status.podIP)
ROOK_CEPH_VERSION_NAME: mimic
ROOK_MON_ENDPOINTS: <set to the key 'data' of config map 'rook-ceph-mon-endpoints'> Optional: false
ROOK_MON_SECRET: <set to the key 'mon-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_ADMIN_SECRET: <set to the key 'admin-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_CEPH_CONFIG_OVERRIDE: /etc/rook/config/override.conf
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/etc/rook/config from rook-config-override (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-mgr-token-kdq4t (ro)
Containers:
mgr:
Container ID: docker://6c25bf1aa3f3956157e6f8550156fd8bcb44fb804f2e63d4923d6556aa2d7242
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Ports: 6800/TCP, 9283/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
ceph-mgr
Args:
--foreground
--id
a
State: Running
Started: Sat, 26 Jan 2019 10:51:29 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAME: rook-ceph-mgr-a-8649f78d9b-wwxrc (v1:metadata.name)
POD_NAMESPACE: rook-ceph (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
ROOK_CLUSTER_NAME: rook-ceph
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-mgr-token-kdq4t (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rook-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
ceph-default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-config-override:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rook-config-override
Optional: false
rook-ceph-mgr-token-kdq4t:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-mgr-token-kdq4t
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 84s default-scheduler Successfully assigned rook-ceph/rook-ceph-mgr-a-8649f78d9b-wwxrc to v.us.8
Normal Pulled 83s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 83s kubelet, v.us.8 Created container
Normal Started 83s kubelet, v.us.8 Started container
Normal Pulled 80s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 80s kubelet, v.us.8 Created container
Normal Started 79s kubelet, v.us.8 Started container
Name: rook-ceph-mon-a-f5cb646cb-chkkh
Namespace: rook-ceph
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:50:43 +0800
Labels: app=rook-ceph-mon
ceph_daemon_id=a
mon=a
mon_cluster=rook-ceph
pod-template-hash=f5cb646cb
rook_cluster=rook-ceph
Annotations: <none>
Status: Running
IP: 10.32.0.11
Controlled By: ReplicaSet/rook-ceph-mon-a-f5cb646cb
Init Containers:
config-init:
Container ID: docker://86d583a911c172840f3a3d6e3d409236acca2c00ff662fd2ee8b3a15b39d91d1
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
mon-init
--config-dir=/var/lib/rook
--name=a
--port=6789
--fsid=27650da9-f702-4375-a03f-b2becc7dd19b
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:50:44 +0800
Finished: Sat, 26 Jan 2019 10:50:44 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: 10.108.69.163
ROOK_CLUSTER_NAME: rook-ceph
ROOK_MON_ENDPOINTS: <set to the key 'data' of config map 'rook-ceph-mon-endpoints'> Optional: false
ROOK_MON_SECRET: <set to the key 'mon-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_ADMIN_SECRET: <set to the key 'admin-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_CEPH_CONFIG_OVERRIDE: /etc/rook/config/override.conf
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/etc/rook/config from rook-config-override (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
monmap-init:
Container ID: docker://17ae2ea125a21bc2fdf4ddbe9f32ec329cb42f815d6793c0eacd94d7a95f640a
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
/usr/bin/monmaptool
Args:
/var/lib/rook/mon-a/monmap
--create
--clobber
--fsid
27650da9-f702-4375-a03f-b2becc7dd19b
--add
a
10.108.69.163:6789
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:50:45 +0800
Finished: Sat, 26 Jan 2019 10:50:51 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
mon-fs-init:
Container ID: docker://61548e99e2e563ad85dce3289c2a1dfbf006135d2aaf9c36cd56273f0af1da8d
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
ceph-mon
Args:
--mkfs
--monmap
/var/lib/rook/mon-a/monmap
--name
mon.a
--mon-data
/var/lib/rook/mon-a/data
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:50:52 +0800
Finished: Sat, 26 Jan 2019 10:50:52 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Containers:
mon:
Container ID: docker://e81184e54049b9a71e30c61fa86b6a007569a996ef431b2ca3a5094deaa5b949
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: 6789/TCP
Host Port: 0/TCP
Command:
ceph-mon
Args:
--foreground
--public-addr
10.108.69.163:6789
--name
mon.a
--mon-data
/var/lib/rook/mon-a/data
State: Running
Started: Sat, 26 Jan 2019 10:50:53 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAME: rook-ceph-mon-a-f5cb646cb-chkkh (v1:metadata.name)
POD_NAMESPACE: rook-ceph (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rook-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/rook
HostPathType:
ceph-default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-config-override:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rook-config-override
Optional: false
default-token-99qjp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-99qjp
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=v.us.8
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m5s default-scheduler Successfully assigned rook-ceph/rook-ceph-mon-a-f5cb646cb-chkkh to v.us.8
Normal Pulled 2m4s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 2m4s kubelet, v.us.8 Created container
Normal Started 2m4s kubelet, v.us.8 Started container
Normal Pulled 2m3s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 2m3s kubelet, v.us.8 Created container
Normal Started 2m3s kubelet, v.us.8 Started container
Normal Pulled 117s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 117s kubelet, v.us.8 Created container
Normal Started 116s kubelet, v.us.8 Started container
Normal Pulled 116s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 116s kubelet, v.us.8 Created container
Normal Started 115s kubelet, v.us.8 Started container
Name: rook-ceph-mon-b-74d8f8c4d8-pd9jg
Namespace: rook-ceph
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:50:59 +0800
Labels: app=rook-ceph-mon
ceph_daemon_id=b
mon=b
mon_cluster=rook-ceph
pod-template-hash=74d8f8c4d8
rook_cluster=rook-ceph
Annotations: <none>
Status: Running
IP: 10.32.0.12
Controlled By: ReplicaSet/rook-ceph-mon-b-74d8f8c4d8
Init Containers:
config-init:
Container ID: docker://c44807cc893e966182b92243f2778e32548c51cb59f004fe9582fb1b94964f5e
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
mon-init
--config-dir=/var/lib/rook
--name=b
--port=6789
--fsid=27650da9-f702-4375-a03f-b2becc7dd19b
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:00 +0800
Finished: Sat, 26 Jan 2019 10:51:00 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: 10.108.82.238
ROOK_CLUSTER_NAME: rook-ceph
ROOK_MON_ENDPOINTS: <set to the key 'data' of config map 'rook-ceph-mon-endpoints'> Optional: false
ROOK_MON_SECRET: <set to the key 'mon-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_ADMIN_SECRET: <set to the key 'admin-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_CEPH_CONFIG_OVERRIDE: /etc/rook/config/override.conf
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/etc/rook/config from rook-config-override (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
monmap-init:
Container ID: docker://84f49bc588a3bf9d7920efbdec2f7c5800cc744e7d28e7d8413a330fa1f00fcd
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
/usr/bin/monmaptool
Args:
/var/lib/rook/mon-b/monmap
--create
--clobber
--fsid
27650da9-f702-4375-a03f-b2becc7dd19b
--add
a
10.108.69.163:6789
--add
b
10.108.82.238:6789
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:01 +0800
Finished: Sat, 26 Jan 2019 10:51:06 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
mon-fs-init:
Container ID: docker://0b8ed2af1e040ddf426ebaf283f98825e4a601b2ac36af84afb687096e57470c
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
ceph-mon
Args:
--mkfs
--monmap
/var/lib/rook/mon-b/monmap
--name
mon.b
--mon-data
/var/lib/rook/mon-b/data
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:07 +0800
Finished: Sat, 26 Jan 2019 10:51:07 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Containers:
mon:
Container ID: docker://94a4a7123c16357735c5700574404d574becd6540072eae3da5d158db81fb7eb
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: 6789/TCP
Host Port: 0/TCP
Command:
ceph-mon
Args:
--foreground
--public-addr
10.108.82.238:6789
--name
mon.b
--mon-data
/var/lib/rook/mon-b/data
State: Running
Started: Sat, 26 Jan 2019 10:51:08 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAME: rook-ceph-mon-b-74d8f8c4d8-pd9jg (v1:metadata.name)
POD_NAMESPACE: rook-ceph (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rook-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/rook
HostPathType:
ceph-default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-config-override:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rook-config-override
Optional: false
default-token-99qjp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-99qjp
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=v.us.8
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 109s default-scheduler Successfully assigned rook-ceph/rook-ceph-mon-b-74d8f8c4d8-pd9jg to v.us.8
Normal Pulled 108s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 108s kubelet, v.us.8 Created container
Normal Started 108s kubelet, v.us.8 Started container
Normal Pulled 108s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 107s kubelet, v.us.8 Created container
Normal Started 107s kubelet, v.us.8 Started container
Normal Pulled 101s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 101s kubelet, v.us.8 Created container
Normal Started 101s kubelet, v.us.8 Started container
Normal Pulled 100s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 100s kubelet, v.us.8 Created container
Normal Started 100s kubelet, v.us.8 Started container
Name: rook-ceph-mon-c-89f845f9b-sp5gb
Namespace: rook-ceph
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:51:13 +0800
Labels: app=rook-ceph-mon
ceph_daemon_id=c
mon=c
mon_cluster=rook-ceph
pod-template-hash=89f845f9b
rook_cluster=rook-ceph
Annotations: <none>
Status: Running
IP: 10.32.0.10
Controlled By: ReplicaSet/rook-ceph-mon-c-89f845f9b
Init Containers:
config-init:
Container ID: docker://56b3a64cf0bcf14218914ff3aac1099e2cb3321eb90a9bb035e0081e75bdac81
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
mon-init
--config-dir=/var/lib/rook
--name=c
--port=6789
--fsid=27650da9-f702-4375-a03f-b2becc7dd19b
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:14 +0800
Finished: Sat, 26 Jan 2019 10:51:14 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: 10.101.235.61
ROOK_CLUSTER_NAME: rook-ceph
ROOK_MON_ENDPOINTS: <set to the key 'data' of config map 'rook-ceph-mon-endpoints'> Optional: false
ROOK_MON_SECRET: <set to the key 'mon-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_ADMIN_SECRET: <set to the key 'admin-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_CEPH_CONFIG_OVERRIDE: /etc/rook/config/override.conf
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/etc/rook/config from rook-config-override (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
monmap-init:
Container ID: docker://c108f45e90d305349c8a950ff3c55300a9c5b16a31a905e31683b20022411a46
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
/usr/bin/monmaptool
Args:
/var/lib/rook/mon-c/monmap
--create
--clobber
--fsid
27650da9-f702-4375-a03f-b2becc7dd19b
--add
a
10.108.69.163:6789
--add
b
10.108.82.238:6789
--add
c
10.101.235.61:6789
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:15 +0800
Finished: Sat, 26 Jan 2019 10:51:15 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
mon-fs-init:
Container ID: docker://9cdec22e6c1d5d0984fab8e0f3f496c94fd7e55f41ebf034996133176b9eb361
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
ceph-mon
Args:
--mkfs
--monmap
/var/lib/rook/mon-c/monmap
--name
mon.c
--mon-data
/var/lib/rook/mon-c/data
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:16 +0800
Finished: Sat, 26 Jan 2019 10:51:16 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Containers:
mon:
Container ID: docker://b49a0dba61a9c21a85ee3d8426b4f17139a6fc14016a18f2a94e08128e741ef4
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: 6789/TCP
Host Port: 0/TCP
Command:
ceph-mon
Args:
--foreground
--public-addr
10.101.235.61:6789
--name
mon.c
--mon-data
/var/lib/rook/mon-c/data
State: Running
Started: Sat, 26 Jan 2019 10:51:17 +0800
Ready: True
Restart Count: 0
Environment:
POD_NAME: rook-ceph-mon-c-89f845f9b-sp5gb (v1:metadata.name)
POD_NAMESPACE: rook-ceph (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-99qjp (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rook-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/rook
HostPathType:
ceph-default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-config-override:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rook-config-override
Optional: false
default-token-99qjp:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-99qjp
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=v.us.8
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 95s default-scheduler Successfully assigned rook-ceph/rook-ceph-mon-c-89f845f9b-sp5gb to v.us.8
Normal Pulled 94s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 94s kubelet, v.us.8 Created container
Normal Started 94s kubelet, v.us.8 Started container
Normal Pulled 93s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 93s kubelet, v.us.8 Created container
Normal Started 93s kubelet, v.us.8 Started container
Normal Pulled 92s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 92s kubelet, v.us.8 Created container
Normal Started 92s kubelet, v.us.8 Started container
Normal Pulled 91s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 91s kubelet, v.us.8 Created container
Normal Started 91s kubelet, v.us.8 Started container
Name: rook-ceph-osd-0-56d79fc479-rrwqf
Namespace: rook-ceph
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:51:45 +0800
Labels: app=rook-ceph-osd
ceph-osd-id=0
pod-template-hash=56d79fc479
rook_cluster=rook-ceph
Annotations: <none>
Status: Running
IP: 10.32.0.15
Controlled By: ReplicaSet/rook-ceph-osd-0-56d79fc479
Init Containers:
config-init:
Container ID: docker://6e77fa1b850e3645315882f7af790b8f0142fa881e0b6724b782315bda974f16
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
osd
init
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:46 +0800
Finished: Sat, 26 Jan 2019 10:51:46 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_NODE_NAME: v.us.8
ROOK_CLUSTER_ID: 13f6284d-2115-11e9-a969-560001d945cd
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: (v1:status.podIP)
ROOK_CLUSTER_NAME: rook-ceph
ROOK_MON_ENDPOINTS: <set to the key 'data' of config map 'rook-ceph-mon-endpoints'> Optional: false
ROOK_MON_SECRET: <set to the key 'mon-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_ADMIN_SECRET: <set to the key 'admin-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_CONFIG_DIR: /var/lib/rook
ROOK_CEPH_CONFIG_OVERRIDE: /etc/rook/config/override.conf
ROOK_FSID: <set to the key 'fsid' in secret 'rook-ceph-mon'> Optional: false
ROOK_OSD_DATABASE_SIZE: 1024
ROOK_OSD_JOURNAL_SIZE: 1024
ROOK_OSDS_PER_DEVICE: 1
TINI_SUBREAPER:
ROOK_OSD_ID: 0
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/etc/rook/config from rook-config-override (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
copy-bins:
Container ID: docker://dfffa73630bc958bdb8f53d8c170753384e5cf864f43aa79b43f087e011c41d9
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
osd
copybins
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:47 +0800
Finished: Sat, 26 Jan 2019 10:51:47 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_PATH: /rook
Mounts:
/rook from rook-binaries (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
Containers:
osd:
Container ID: docker://80754f875fb92c04b73d63b61a10742a3c61937cd2aa5cce78e6f479e1fa97a2
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
ceph-osd
Args:
--foreground
--id
0
--conf
/var/lib/rook/osd0/rook-ceph.config
--osd-data
/var/lib/rook/osd0
--keyring
/var/lib/rook/osd0/keyring
--cluster
rook-ceph
--osd-uuid
eb7bf21b-20eb-42e8-b818-6f057f041428
--osd-journal=/var/lib/rook/osd0/journal
State: Running
Started: Sat, 26 Jan 2019 10:51:48 +0800
Ready: True
Restart Count: 0
Environment:
ROOK_NODE_NAME: v.us.8
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: (v1:status.podIP)
TINI_SUBREAPER:
POD_NAME: rook-ceph-osd-0-56d79fc479-rrwqf (v1:metadata.name)
POD_NAMESPACE: rook-ceph (v1:metadata.namespace)
NODE_NAME: (v1:spec.nodeName)
ROOK_OSD_UUID: eb7bf21b-20eb-42e8-b818-6f057f041428
ROOK_OSD_ID: 0
ROOK_OSD_STORE_TYPE: filestore
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/rook from rook-binaries (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
rook-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/rook
HostPathType:
ceph-default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-config-override:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rook-config-override
Optional: false
rook-binaries:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-ceph-osd-token-rbdvz:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-osd-token-rbdvz
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=v.us.8
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 63s default-scheduler Successfully assigned rook-ceph/rook-ceph-osd-0-56d79fc479-rrwqf to v.us.8
Normal Pulled 62s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 62s kubelet, v.us.8 Created container
Normal Started 62s kubelet, v.us.8 Started container
Normal Pulled 61s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 61s kubelet, v.us.8 Created container
Normal Started 61s kubelet, v.us.8 Started container
Normal Pulled 60s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 60s kubelet, v.us.8 Created container
Normal Started 60s kubelet, v.us.8 Started container
Name: rook-ceph-osd-prepare-v.us.8-5zhnn
Namespace: rook-ceph
Priority: 0
PriorityClassName: <none>
Node: v.us.8/66.42.110.223
Start Time: Sat, 26 Jan 2019 10:51:40 +0800
Labels: app=rook-ceph-osd-prepare
controller-uid=4e44413e-2115-11e9-a969-560001d945cd
job-name=rook-ceph-osd-prepare-v.us.8
rook_cluster=rook-ceph
Annotations: <none>
Status: Succeeded
IP: 10.32.0.14
Controlled By: Job/rook-ceph-osd-prepare-v.us.8
Containers:
copy-bins:
Container ID: docker://4bbcc98a2106f395691c26fbdd1a24f93f64c0632760e54a4b4c69888db48daa
Image: rook/ceph:master
Image ID: docker-pullable://rook/ceph@sha256:f2f01be4f6b569b44adaaed1eceec9d1a91175520492584965213099eba46f27
Port: <none>
Host Port: <none>
Args:
ceph
osd
copybins
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:41 +0800
Finished: Sat, 26 Jan 2019 10:51:41 +0800
Ready: False
Restart Count: 0
Environment:
ROOK_PATH: /rook
Mounts:
/rook from rook-binaries (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
provision:
Container ID: docker://9a57e04709c6994ecf3b0cc234066e5413e2fe797b67d918ccf43ddafa2d4bc5
Image: ceph/ceph:v13
Image ID: docker-pullable://ceph/ceph@sha256:a53f2367450bc4cc5ac5a6783f1e105b32053277d97b52e55209a7fd30ef5a9d
Port: <none>
Host Port: <none>
Command:
/rook/tini
Args:
--
/rook/rook
ceph
osd
provision
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sat, 26 Jan 2019 10:51:41 +0800
Finished: Sat, 26 Jan 2019 10:51:45 +0800
Ready: False
Restart Count: 0
Environment:
ROOK_NODE_NAME: v.us.8
ROOK_CLUSTER_ID: 13f6284d-2115-11e9-a969-560001d945cd
ROOK_PRIVATE_IP: (v1:status.podIP)
ROOK_PUBLIC_IP: (v1:status.podIP)
ROOK_CLUSTER_NAME: rook-ceph
ROOK_MON_ENDPOINTS: <set to the key 'data' of config map 'rook-ceph-mon-endpoints'> Optional: false
ROOK_MON_SECRET: <set to the key 'mon-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_ADMIN_SECRET: <set to the key 'admin-secret' in secret 'rook-ceph-mon'> Optional: false
ROOK_CONFIG_DIR: /var/lib/rook
ROOK_CEPH_CONFIG_OVERRIDE: /etc/rook/config/override.conf
ROOK_FSID: <set to the key 'fsid' in secret 'rook-ceph-mon'> Optional: false
ROOK_OSD_DATABASE_SIZE: 1024
ROOK_OSD_JOURNAL_SIZE: 1024
ROOK_OSDS_PER_DEVICE: 1
Mounts:
/etc/ceph from ceph-default-config-dir (rw)
/rook from rook-binaries (rw)
/var/lib/rook from rook-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from rook-ceph-osd-token-rbdvz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
rook-data:
Type: HostPath (bare host directory volume)
Path: /var/lib/rook
HostPathType:
ceph-default-config-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-config-override:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: rook-config-override
Optional: false
rook-binaries:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
rook-ceph-osd-token-rbdvz:
Type: Secret (a volume populated by a Secret)
SecretName: rook-ceph-osd-token-rbdvz
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/hostname=v.us.8
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 68s default-scheduler Successfully assigned rook-ceph/rook-ceph-osd-prepare-v.us.8-5zhnn to v.us.8
Normal Pulled 67s kubelet, v.us.8 Container image "rook/ceph:master" already present on machine
Normal Created 67s kubelet, v.us.8 Created container
Normal Started 67s kubelet, v.us.8 Started container
Normal Pulled 67s kubelet, v.us.8 Container image "ceph/ceph:v13" already present on machine
Normal Created 67s kubelet, v.us.8 Created container
Normal Started 67s kubelet, v.us.8 Started container
如果看到一直是
Init:CrashLoopBackOff状态:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-86c58d9df4-4prl5 1/1 Running 0 16m
kube-system coredns-86c58d9df4-zfqqm 1/1 Running 0 16m
kube-system etcd-v.us.8 1/1 Running 0 15m
kube-system kube-apiserver-v.us.8 1/1 Running 0 15m
kube-system kube-controller-manager-v.us.8 1/1 Running 0 15m
kube-system kube-proxy-jt8xr 1/1 Running 0 16m
kube-system kube-scheduler-v.us.8 1/1 Running 0 15m
kube-system weave-net-rn2rs 2/2 Running 0 13m
rook-ceph-system rook-ceph-agent-9hzv2 1/1 Running 0 12m
rook-ceph-system rook-ceph-operator-76cf7f88f-ssr5q 1/1 Running 0 13m
rook-ceph-system rook-discover-jtndt 1/1 Running 0 12m
rook-ceph rook-ceph-mon-a-6b87f9967-dvrk5 0/1 Init:CrashLoopBackOff 7 12m
rook-ceph rook-ceph-mon-d-5646c6458f-rs97z 0/1 Init:CrashLoopBackOff 7 10m
rook-ceph rook-ceph-mon-f-86dd4765db-fzmzd 0/1 Init:CrashLoopBackOff 6 9m20s
显示部署信息,发现如下错误:
$ kubectl describe pods -n rook-ceph
Message: The keyring does not match the existing keyring in /var/lib/rook/mon-a/data/keyring. You may need to delete the contents of dataDirHostPath on the host from a previous deployment.
由于是重新部署,所以安装前需要清理残留文件后稍等片刻即可自动恢复:
rm -rf /var/lib/rook/mon-*
- 创建
PV
$ cat kubernetes-plugins-rook-storage.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph
spec:
replicated:
size: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: ceph.rook.io/block
parameters:
blockPool: replicapool
# Specify the namespace of the rook cluster from which to create volumes.
# If not specified, it will use `rook` as the default namespace of the cluster.
# This is also the namespace where the cluster will be
clusterNamespace: rook-ceph
# Specify the filesystem type of the volume. If not specified, it will use `ext4`.
#fstype: xfs
# (Optional) Specify an existing Ceph user that will be used for mounting storage with this StorageClass.
#mountUser: user1
# (Optional) Specify an existing Kubernetes secret name containing just one key holding the Ceph user secret.
# The secret must exist in each namespace(s) where the storage will be consumed.
#mountSecret: ceph-user1-secret
使用配置安装
$ kubectl apply -f kubernetes-plugins-rook-storage.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block-oss created
- 此时查看
k8s集群部署详情:
$ kubectl get nodes,po,svc --all-namespaces
NAME STATUS ROLES AGE VERSION
node/v.us.8 Ready master 7m37s v1.13.2
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-86c58d9df4-8gzz2 1/1 Running 0 7m18s
kube-system pod/coredns-86c58d9df4-lr6n5 1/1 Running 0 7m18s
kube-system pod/etcd-v.us.8 1/1 Running 0 6m35s
kube-system pod/kube-apiserver-v.us.8 1/1 Running 0 6m22s
kube-system pod/kube-controller-manager-v.us.8 1/1 Running 0 6m39s
kube-system pod/kube-proxy-4gmbk 1/1 Running 0 7m18s
kube-system pod/kube-scheduler-v.us.8 1/1 Running 0 6m17s
kube-system pod/weave-net-jb2tt 2/2 Running 0 6m28s
rook-ceph-system pod/rook-ceph-agent-pmvfs 1/1 Running 0 5m22s
rook-ceph-system pod/rook-ceph-operator-76cf7f88f-bxfhj 1/1 Running 0 5m55s
rook-ceph-system pod/rook-discover-g7psj 1/1 Running 0 5m22s
rook-ceph pod/rook-ceph-mgr-a-8649f78d9b-wwxrc 1/1 Running 0 4m25s
rook-ceph pod/rook-ceph-mon-a-f5cb646cb-chkkh 1/1 Running 0 5m6s
rook-ceph pod/rook-ceph-mon-b-74d8f8c4d8-pd9jg 1/1 Running 0 4m50s
rook-ceph pod/rook-ceph-mon-c-89f845f9b-sp5gb 1/1 Running 0 4m36s
rook-ceph pod/rook-ceph-osd-0-56d79fc479-rrwqf 1/1 Running 0 4m4s
rook-ceph pod/rook-ceph-osd-prepare-v.us.8-5zhnn 0/2 Completed 0 4m9s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7m37s
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 7m33s
rook-ceph service/rook-ceph-mgr ClusterIP 10.105.220.216 <none> 9283/TCP 4m11s
rook-ceph service/rook-ceph-mgr-dashboard ClusterIP 10.99.44.224 <none> 8443/TCP 4m11s
rook-ceph service/rook-ceph-mon-a ClusterIP 10.108.69.163 <none> 6789/TCP 5m6s
rook-ceph service/rook-ceph-mon-b ClusterIP 10.108.82.238 <none> 6789/TCP 4m50s
rook-ceph service/rook-ceph-mon-c ClusterIP 10.101.235.61 <none> 6789/TCP 4m36s
- 如果安装失败,重新初始化集群环境
kubeadm reset
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X && sudo iptables -L
从节点 v.cs.8 部署
- 安装基础软件
sudo apt install kubelet kubeadm kubectl
- 初始化加入集群
Tue Jan 15 22:32:09 coam@v.cs.8:~/docker-us$ sudo kubeadm join 172.31.141.97:6443 --token cd7m0v.c8w082mh057x499u --discovery-token-ca-cert-hash sha256:db45ba01d83b7ed325295f8a9554c405238b5005bba22053eb85ca9a9638ec0e
* [preflight] Running pre-flight checks
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
* [discovery] Trying to connect to API Server "172.31.141.97:6443"
* [discovery] Created cluster-info discovery client, requesting info from "https://172.31.141.97:6443"
* [discovery] Requesting info from "https://172.31.141.97:6443" again to validate TLS against the pinned public key
* [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "172.31.141.97:6443"
* [discovery] Successfully established connection with API Server "172.31.141.97:6443"
* [join] Reading configuration from the cluster...
* [join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
* [kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
* [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
* [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
* [kubelet-start] Activating the kubelet service
* [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
* [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "v.cs.8" as an annotation
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.
配置开机自启动
所有节点 设置
kubelet开机自启动
systemctl enable kubelet.service
在主节点 v.us.8 查看主节点状态
$ kubectl get nodes,po,svc --all-namespaces
NAME STATUS ROLES AGE VERSION
node/v.cs.8 Ready <none> 3m26s v1.13.2
node/v.us.8 Ready master 6h3m v1.13.2
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-86c58d9df4-pn8lb 1/1 Running 0 6h3m
kube-system pod/coredns-86c58d9df4-s6pbk 1/1 Running 0 6h3m
kube-system pod/etcd-v.us.8 1/1 Running 0 6h2m
kube-system pod/kube-apiserver-v.us.8 1/1 Running 0 6h2m
kube-system pod/kube-controller-manager-v.us.8 1/1 Running 0 6h2m
kube-system pod/kube-flannel-ds-amd64-6l7s7 1/1 Running 0 3m26s
kube-system pod/kube-flannel-ds-amd64-b29hm 1/1 Running 0 3m49s
kube-system pod/kube-proxy-6fv94 1/1 Running 0 6h3m
kube-system pod/kube-proxy-r9nzh 1/1 Running 0 3m26s
kube-system pod/kube-scheduler-v.us.8 1/1 Running 0 6h2m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6h3m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 6h3m
安装 Dashboard
如果点登陆没反应 记得注意官方文档这句话,也就是登录到
Dashboard必须满足HTTPS访问,或者HTTP+localhost访问,不然点击登陆按钮不会有任何反应
- 安装
Dashboard控制面板
wget -O k8s-dashboard.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
$ kubectl apply -f k8s-dashboard.yaml
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
查看服务是否启动
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kubernetes-dashboard-57df4db6b-h4dxg 1/1 Running 0 37s
创建访问 Dashboard 的账户
这里推荐使用第二种方式
1. 用 Dashboard 自带的角色添加权限
给
kubernetes-dashboard的ServiceAccount绑定权限
$ cat k8s-dashboard-admin.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
执行安装
kubectl apply -f k8s-dashboard-admin.yaml
此账户获取
Token的方式
kubectl describe secrets $(kubectl get secrets --namespace kube-system | grep dashboard-token | awk '{print $1}') --namespace kube-system | grep token: | awk '{print $2}'
2. 新建一个管理员
安全一点的方式是新建一个账户 赋予权限
$ cat k8s-dashboard-coam-role.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: coam-admin
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
roleRef:
kind: ClusterRole
name: cluster-coam-admin
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: coam-admin
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: coam-admin
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
执行安装:
$ kubectl apply -f k8s-dashboard-coam-role.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin created
serviceaccount/admin created
此账户获取
Token的方式
kubectl describe secrets $(kubectl get secrets --namespace kube-system | grep admin-token | awk '{print $1}') --namespace kube-system | grep token: | awk '{print $2}'
eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi10b2tlbi05N3hzbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJhZG1pbiIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjI4ZGM1ZjdmLTIxMTgtMTFlOS1hOTY5LTU2MDAwMWQ5NDVjZCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTphZG1pbiJ9.hqxnX8Wwuj4CLzHletpLXbpe-D460EMZq9MZ3m0vfXuuFU41-Q8lsCfRFgzYyAV_aKNT96ps2jEPtJn9UXkqyb5mg8MuCAkFpLrHieGcPm2skKVGlIqBpIeweAQ80YCRAVGB1pDVsiXU_2pbbdCdvAKgsB0io1QTfo5EbxJn28qR4Aj6ke4YQx-yKPBK1i5h3IA4oS9Q0JiTxsbgXEnoSaZrqJ9Q9VSvcWVrj6oxb3Q9pwSeko_LRr0GDRXgINiZjxqz32Eshj6p4pG3JG3MOWvWDXO7WK2OkwnxwVUnfiAPx1qYYbgVAAx2zIOpiLrv5K5MLYZ8kzAnbPoXdm0SVw
访问面板
1. 通过 Proxy 方式访问:
第一种通过 kubectl proxy 暴露API (这种方法只能本地访问)
开启代理
kubectl proxy &
为了让
kubernetes-dashboard能通过公网 ip 访问,应携带以下参数:
kubectl proxy --address 0.0.0.0 --accept-hosts '.*'
参考 How allow access to Kubernetes-Dashboard from master real routed IP address?
通过以下链接访问
http://66.42.110.223:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
2. 通过新建服务对外暴露端口:
通过 NodeIP + NodePort 访问,此方法可以任意访问.但是存在证书问题,忽略即可
重新修改
k8s-dashboard.yaml拉到底 找到Service区域spec改为NodePort
修改 k8s-dashboard.yaml 文件
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
spec:
type: NodePort
ports:
- port: 8443
targetPort: 8443
nodePort: 30443
selector:
k8s-app: kubernetes-dashboard
重新部署,使其生效
$ kubectl apply -f k8s-dashboard.yaml
secret/kubernetes-dashboard-certs unchanged
secret/kubernetes-dashboard-csrf created
serviceaccount/kubernetes-dashboard unchanged
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal configured
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal unchanged
deployment.apps/kubernetes-dashboard unchanged
service/kubernetes-dashboard configured
访问 https://NodeIP:30443 如果提示
NET::ERR_CERT_INVALID点击继续访问
使用 Token 登录面板
打开 Dashboard 之后 点击 Token 登录 输入上面获取到的 Token 登录即可
Dashboard 管理
- 查看
dashboard的pod是否正常启动及外网暴露端口:
$ kubectl get nodes,po,svc --all-namespaces -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node/a.us.1 Ready <none> 50m v1.13.2 172.31.141.97 <none> Ubuntu 18.04.1 LTS 4.15.0-43-generic docker://18.6.1
node/v.cs.8 Ready <none> 50m v1.13.2 66.42.108.136 <none> CentOS Linux 7 (Core) 3.10.0-957.1.3.el7.x86_64 docker://18.9.1
node/v.us.8 Ready master 23h v1.13.2 66.42.110.223 <none> Ubuntu 18.10 4.18.0-10-generic docker://18.9.1
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system pod/coredns-86c58d9df4-8gzz2 1/1 Running 0 23h 10.32.0.6 v.us.8 <none> <none>
kube-system pod/coredns-86c58d9df4-lr6n5 1/1 Running 0 23h 10.32.0.7 v.us.8 <none> <none>
kube-system pod/etcd-v.us.8 1/1 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
kube-system pod/kube-apiserver-v.us.8 1/1 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
kube-system pod/kube-controller-manager-v.us.8 1/1 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
kube-system pod/kube-proxy-4gmbk 1/1 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
kube-system pod/kube-proxy-7mds5 1/1 Running 0 50m 66.42.108.136 v.cs.8 <none> <none>
kube-system pod/kube-proxy-d6q5c 1/1 Running 0 50m 172.31.141.97 a.us.1 <none> <none>
kube-system pod/kube-scheduler-v.us.8 1/1 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
kube-system pod/kubernetes-dashboard-57df4db6b-xnvnt 1/1 Running 0 23h 10.32.0.14 v.us.8 <none> <none>
kube-system pod/weave-net-6hmbd 2/2 Running 0 50m 172.31.141.97 a.us.1 <none> <none>
kube-system pod/weave-net-7dt22 2/2 Running 1 50m 66.42.108.136 v.cs.8 <none> <none>
kube-system pod/weave-net-jb2tt 2/2 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
rook-ceph-system pod/rook-ceph-agent-89vz7 1/1 Running 0 50m 172.31.141.97 a.us.1 <none> <none>
rook-ceph-system pod/rook-ceph-agent-dmv66 1/1 Running 0 50m 66.42.108.136 v.cs.8 <none> <none>
rook-ceph-system pod/rook-ceph-agent-pmvfs 1/1 Running 0 23h 66.42.110.223 v.us.8 <none> <none>
rook-ceph-system pod/rook-ceph-operator-76cf7f88f-bxfhj 1/1 Running 0 23h 10.32.0.8 v.us.8 <none> <none>
rook-ceph-system pod/rook-discover-4wvz8 1/1 Running 0 50m 10.36.0.1 v.cs.8 <none> <none>
rook-ceph-system pod/rook-discover-g7psj 1/1 Running 0 23h 10.32.0.9 v.us.8 <none> <none>
rook-ceph-system pod/rook-discover-zwqn6 1/1 Running 0 50m 10.44.0.1 a.us.1 <none> <none>
rook-ceph pod/rook-ceph-mgr-a-8649f78d9b-wwxrc 1/1 Running 0 23h 10.32.0.13 v.us.8 <none> <none>
rook-ceph pod/rook-ceph-mon-a-f5cb646cb-chkkh 1/1 Running 0 23h 10.32.0.11 v.us.8 <none> <none>
rook-ceph pod/rook-ceph-mon-b-74d8f8c4d8-pd9jg 1/1 Running 0 23h 10.32.0.12 v.us.8 <none> <none>
rook-ceph pod/rook-ceph-mon-c-89f845f9b-sp5gb 1/1 Running 0 23h 10.32.0.10 v.us.8 <none> <none>
rook-ceph pod/rook-ceph-osd-0-56d79fc479-rrwqf 1/1 Running 0 23h 10.32.0.15 v.us.8 <none> <none>
rook-ceph pod/rook-ceph-osd-prepare-v.us.8-5zhnn 0/2 Completed 0 23h 10.32.0.14 v.us.8 <none> <none>
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 23h <none>
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 23h k8s-app=kube-dns
kube-system service/kubernetes-dashboard NodePort 10.100.112.7 <none> 8443:30443/TCP 23h k8s-app=kubernetes-dashboard
rook-ceph service/rook-ceph-mgr ClusterIP 10.105.220.216 <none> 9283/TCP 23h app=rook-ceph-mgr,rook_cluster=rook-ceph
rook-ceph service/rook-ceph-mgr-dashboard ClusterIP 10.99.44.224 <none> 8443/TCP 23h app=rook-ceph-mgr,rook_cluster=rook-ceph
rook-ceph service/rook-ceph-mon-a ClusterIP 10.108.69.163 <none> 6789/TCP 23h app=rook-ceph-mon,ceph_daemon_id=a,mon=a,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph service/rook-ceph-mon-b ClusterIP 10.108.82.238 <none> 6789/TCP 23h app=rook-ceph-mon,ceph_daemon_id=b,mon=b,mon_cluster=rook-ceph,rook_cluster=rook-ceph
rook-ceph service/rook-ceph-mon-c ClusterIP 10.101.235.61 <none> 6789/TCP 23h app=rook-ceph-mon,ceph_daemon_id=c,mon=c,mon_cluster=rook-ceph,rook_cluster=rook-ceph
部署镜像容器
通过 YAML 配置文件创建一个 nginx 部署:
$ wget -O kubernetes-deployment-nginx.yaml https://k8s.io/examples/application/deployment.yaml
$ kubectl apply -f kubernetes-deployment-nginx.yaml
deployment.apps/nginx-deployment created
配置文件示例
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
查看部署状态:
$ kubectl describe deployment nginx-deployment
Name: nginx-deployment
Namespace: default
CreationTimestamp: Sun, 27 Jan 2019 10:34:19 +0800
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},"spec":{"replica...
Selector: app=nginx
Replicas: 2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Containers:
nginx:
Image: nginx:1.7.9
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-76bf4969df (2/2 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 22s deployment-controller Scaled up replica set nginx-deployment-76bf4969df to 2
查看这次部署创建的 pods:
$ kubectl get pods -l app=nginx
NAME READY STATUS RESTARTS AGE
nginx-deployment-76bf4969df-7q4c8 1/1 Running 0 79s
nginx-deployment-76bf4969df-q6dxd 1/1 Running 0 79s
删除部署
kubectl delete deployment nginx-deployment
完全删除安装(会删除所有包括Docker)
kubeadm reset -f
yum remove docker* kubeadm kubectl kubelet -y
ip link delete docker0
ip link delete cni0
ip link delete weave
ip link delete flannel.1
rpm -e $(rpm -qa | grep docker)
- 如果你的集群安装过程中遇到了其他问题,我们可以使用下面的命令来进行重置:
$ kubeadm reset
$ ifconfig cni0 down && ip link delete cni0
$ ifconfig flannel.1 down && ip link delete flannel.1
$ rm -rf /var/lib/cni/
- 查看节点状态
curl https://47.244.154.194:6443 -k
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {
},
"code": 403
}
- 查看集群状态
Sat Jan 19 17:34:35 coam@v.us.8:~$ kubectl cluster-info
Kubernetes master is running at https://66.42.110.223:6443
KubeDNS is running at https://66.42.110.223:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
- 查看节点状态
$ kubectl get nodes,po,svc --all-namespaces
NAME STATUS ROLES AGE VERSION
node/v.cs.8 Ready <none> 51m v1.13.2
node/v.us.8 Ready master 83m v1.13.2
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-86c58d9df4-bmhwl 1/1 Running 0 83m
kube-system pod/coredns-86c58d9df4-slksc 1/1 Running 0 83m
kube-system pod/etcd-v.us.8 1/1 Running 0 82m
kube-system pod/kube-apiserver-v.us.8 1/1 Running 0 82m
kube-system pod/kube-controller-manager-v.us.8 1/1 Running 0 82m
kube-system pod/kube-flannel-ds-amd64-8c5f5 1/1 Running 0 81m
kube-system pod/kube-flannel-ds-amd64-n9j64 1/1 Running 0 51m
kube-system pod/kube-proxy-7p2pq 1/1 Running 0 51m
kube-system pod/kube-proxy-nm6mc 1/1 Running 0 83m
kube-system pod/kube-scheduler-v.us.8 1/1 Running 0 82m
kube-system pod/kubernetes-dashboard-57df4db6b-pjlzw 1/1 Running 0 40m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 83m
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP 83m
kube-system service/kubernetes-dashboard ClusterIP 10.98.123.100 <none> 443/TCP 40m
- kubectl
#* 启用主控机同时作为工作节点
#kubectl taint nodes --all node-role.kubernetes.io/master
获取集群配置信息
kubectl -n kube-system get cm kubeadm-config -oyaml
获取集群节点信息
kubectl get nodes
获取组件的健康状态
kubectl get cs
安装网络插件
wget -O k8s-plugins-kube-flannel.yml https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sudo kubectl apply -f k8s-plugins-kube-flannel.yml
执行如下的命令,获取当前系统上所有在运行的
pod的状态,指定名称空间为kube-system,为系统级的pod,命令如下
Thu Jan 17 22:51:51 coam@a.us.1:~$ sudo kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-86c58d9df4-d6tb2 1/1 Running 0 4h57m
coredns-86c58d9df4-drnjl 1/1 Running 0 4h57m
etcd-a.us.1 1/1 Running 1 4h57m
kube-apiserver-a.us.1 1/1 Running 1 4h56m
kube-controller-manager-a.us.1 1/1 Running 1 4h57m
kube-flannel-ds-amd64-5rlxt 1/1 Running 0 91s
kube-flannel-ds-amd64-nbvpf 1/1 Running 0 91s
kube-proxy-2jwhp 1/1 Running 1 4h57m
kube-proxy-l7shz 1/1 Running 0 8m6s
kube-scheduler-a.us.1 1/1 Running 1 4h56m
需要查看
pod的详细信息,则添加选项-o wide
Thu Jan 17 22:56:18 coam@a.us.1:~$ kubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-86c58d9df4-d6tb2 1/1 Running 0 5h3m 10.244.0.4 a.us.1 <none> <none>
coredns-86c58d9df4-drnjl 1/1 Running 0 5h3m 10.244.1.2 a.us.0 <none> <none>
etcd-a.us.1 1/1 Running 1 5h2m 172.31.141.97 a.us.1 <none> <none>
kube-apiserver-a.us.1 1/1 Running 1 5h2m 172.31.141.97 a.us.1 <none> <none>
kube-controller-manager-a.us.1 1/1 Running 1 5h2m 172.31.141.97 a.us.1 <none> <none>
kube-flannel-ds-amd64-5rlxt 1/1 Running 0 7m2s 172.31.141.98 a.us.0 <none> <none>
kube-flannel-ds-amd64-nbvpf 1/1 Running 0 7m2s 172.31.141.97 a.us.1 <none> <none>
kube-proxy-2jwhp 1/1 Running 1 5h3m 172.31.141.97 a.us.1 <none> <none>
kube-proxy-l7shz 1/1 Running 0 13m 172.31.141.98 a.us.0 <none> <none>
kube-scheduler-a.us.1 1/1 Running 1 5h2m 172.31.141.97 a.us.1 <none> <none>
执行如下命令,获取当前系统的名称空间
Thu Jan 17 22:53:18 coam@a.us.1:~$ kubectl get ns
NAME STATUS AGE
default Active 4h59m
kube-public Active 4h59m
kube-system Active 4h59m
拆卸集群
首先处理各节点:
kubectl drain v.cs.8 --delete-local-data --force --ignore-daemonsets
kubectl delete node v.cs.8
一旦节点移除之后,则可以执行如下命令来重置集群:
kubeadm reset
kubernetes 更新
- 检查是否有更新
kubeadm upgrade plan
- 更新到指定版本
kubeadm upgrade apply v1.13.0
- kubeadm
注意: 在阿里云的专有网络上通过参数指定外网ip
--apiserver-advertise-address=47.244.154.194无法启动,需要添加参数--apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=47.244.154.194方式启动:
IP证书问题参考 Invalid x509 certificate for kubernetes master
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0 --apiserver-cert-extra-sans=47.244.154.194
几种集群启动参数备注
kubeadm reset
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=149.28.93.7
# 外网ip - 启动失败
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=47.244.154.194 --apiserver-cert-extra-sans=47.244.154.194
# 内网ip - 启动成功
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=172.31.141.97
# 通用ip - 启动成功
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=0.0.0.0
忽略交换分区错误:
–ignore-preflight-errors=Swap
- 列出
token
kubeadm token list
- 创建新的
token
kubeadm token create --print-join-command
- 加入
Kubernetes集群网络
sudo kubeadm join 47.244.154.194:6443 --token cq3dja.33tdj0lzoqncu0h0 --discovery-token-ca-cert-hash sha256:16131f78ab6af4de0c228ef2a9b27bab5e49f76a22532f4249759ae432e77dd2
从节点同样需要到互联网上拉取
flannel的镜像,并启动,因此需要一点时间,用docker images查看是否成功拉取flannel镜像到本地,等flannel拉取成功启动后,在master上可以看到node的状态为Ready.
kubectl -n kube-system get cm kubeadm-config -oyaml
kubectl get pods --all-namespaces
kubectl get pods -n kube-system -o wide
/etc/cni/net.d/
sudo kubeadm init --kubernetes-version=v1.13.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=66.42.110.223
sudo kubeadm join 66.42.110.223:6443 --token j3ft16.l3qw4c7jga8ft873 --discovery-token-ca-cert-hash sha256:d892e13c2d7705c6503145a667b6bfa6efd9667295898a1c337ab8cf406cc394
Comments