docker日志部分参考：
1. kubelet启动日志解读
1.1 获取kubelet的完整启动日志的方法
-
- 先停止kubelet服务和docker服务（systemctl stop kubelet && systemctl stop docker）
-
- 启动docker服务（systemctl start docker）
-
- 清空/var/log/messages文件
-
- 启动kubelet服务（systemctl start kubelet）
-
- 查看/var/log/messages文件中的日志，即为整个kubelet服务启动过程的日志
1.2 通过日志分析，本人对kubelet理解如下：
1.2.1 kubelet通过调用docker的api去创建容器，包括apiserver、etcd、scheduler、controller-manager、kube-proxy、coredns、flannel容器
1.2.2 kubelet服务启动成功后，调用apiserver去注册自己的状态（类似于kubeedge启动后会建立与cloud端的长连接，并发送心跳数据）
1.2.3 kubelet自身就是每个个master/node节点上的agent，负责调用本机的cri、cni、volume接口去管理容器
1.2.4 kubelet启动pod中容器的顺序是先启动pause容器、再启动pod中设置的init容器（init容器启动成功后会被删除），再启动设置的container容器
1.2.5 kubelet本身会启动一个10250的端口，即每个节点都一个10250端口，（猜测是和apiserver交互的端口）
1.2.6 对于Node、Service、Pod、ConfigMap、Secret、write event等的管理，k8s管理的所有入口都是apiserver，kubelet通过调用apiserver，apiserver在查询etcd、或调用scheduler、controller-manager来。如果需要创建pod，再调度集群服务器的kubelet服务来创建容器
1.2.7 kubelet的kuberuntime_manager.go负责管理CRI（容器运行时接口，我们默认的容器运行时为docker）
1.3 日志文件分段分析如下：
Oct 31 17:39:25 master1 systemd: Started kubelet: The Kubernetes Node Agent.
以上日志分析systemd：启动kubelet服务
Oct 31 17:39:26 master1 kubelet: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 17:39:26 master1 kubelet: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 17:39:26 master1 systemd: Started Kubernetes systemd probe.
Oct 31 17:39:26 master1 kubelet: I1031 17:39:26.389046 12269 server.go:410] Version: v0.0.0-master+c97fe5036ef3d
以上日志分析server.go：打印版本v0.0.0-master+c97fe5036ef3d，由于是自己编译的版本，因此版本号显示不对
Oct 31 17:39:26 master1 kubelet: I1031 17:39:26.400798 12269 plugins.go:100] No cloud provider specified.
Oct 31 17:39:26 master1 kubelet: I1031 17:39:26.401128 12269 server.go:773] Client rotation is on, will bootstrap in background
Oct 31 17:39:26 master1 kubelet: I1031 17:39:26.454740 12269 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
以上日志分析certificate_store.go：从"/var/lib/kubelet/pki/kubelet-client-current.pem"文件加载cert/key pair
Oct 31 17:39:26 master1 dockerd: 截掉部分调用dockerAPI日志
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.045098 12269 server.go:636] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.048091 12269 container_manager_linux.go:265] container manager verified user specified cgroup-root exists: []
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.048278 12269 container_manager_linux.go:270] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
以上日志分析container_manager_linux.go：创建Container Manager对象
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.049983 12269 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.050058 12269 container_manager_linux.go:305] Creating device plugin manager: true
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.050226 12269 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider: &{kubelet.sock /var/lib/kubelet/device-plugins/ map[] {0 0} <nil> {{} [0 0 0]} 0x2469f50 0x8b5e318 0x246abd0 map[] map[] map[] map[] map[] 0xc0008c7b90 [0] 0x8b5e318}
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.050746 12269 state_mem.go:36] [cpumanager] initializing new in-memory state store
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.051630 12269 state_mem.go:84] [cpumanager] updated default cpuset: ""
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.051774 12269 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.051889 12269 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider: &{{0 0} 0x8b5e318 10000000000 0xc0007f34a0 <nil> <nil> <nil> <nil> map[memory:{{104857600 0} {<nil>} BinarySI}]}
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.052368 12269 kubelet.go:287] Adding pod path: /etc/kubernetes/manifests
以上日志分析kubelet.go：设置Pod的配置文件路径/etc/kubernetes/manifests
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.053005 12269 kubelet.go:312] Watching apiserver
Oct 31 17:39:27 master1 kubelet: E1031 17:39:27.091790 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:27 master1 kubelet: E1031 17:39:27.096124 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:27 master1 kubelet: E1031 17:39:27.096651 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.098023 12269 client.go:75] Connecting to docker on unix:///var/run/docker.sock
以上日志分析client.go：连接到本地的docker服务unix:///var/run/docker.sock
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.098141 12269 client.go:104] Start docker client with request timeout=2m0s
Oct 31 17:39:27 master1 kubelet: W1031 17:39:27.189438 12269 docker_service.go:563] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.189813 12269 docker_service.go:240] Hairpin mode set to "hairpin-veth"
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.353785 12269 docker_service.go:255] Docker cri networking managed by cni
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.432729 12269 docker_service.go:260] Docker Info: &{ID:7OSQ:74NZ:NHFX:KTG7:KLRN:PMMQ:N34I:QA7C:WEVU:3BEU:SRF5:AZYT Containers:21 ContainersRunning:1 ContainersPaused:0 ContainersStopped:20 Images:9 Driver:overlay2 DriverStatus:[[Backing Filesystem xfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:[] Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:[] Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:28 OomKillDisable:true NGoroutines:42 SystemTime:2019-10-31T17:39:27.360404599+08:00 LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:3.10.0-1062.4.1.el7.x86_64 OperatingSystem:CentOS Linux 7 (Core) OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc00066a3f0 NCPU:2 MemTotal:1927499776 GenericResources:[] DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:master1 Labels:[] ExperimentalBuild:false ServerVersion:19.03.4 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr:192.168.156.243 LocalNodeState:error ControlAvailable:false Error:error while loading TLS certificate in /var/lib/docker/swarm/certificates/swarm-node.crt: certificate (1 - ejzm4a44o0jmox195n0m04ygf) not valid after Wed, 30 Jan 2019 07:50:00 UTC, and it is currently Thu, 31 Oct 2019 17:39:11 CST: x509: certificate has expired or is not yet valid RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:b34a5c8af56e510852c35414db4c1f4fa6172339 Expected:b34a5c8af56e510852c35414db4c1f4fa6172339} RuncCommit:{ID:3e425f80a8c931f88e6d94a8c831b9d5aa481657 Expected:3e425f80a8c931f88e6d94a8c831b9d5aa481657} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=seccomp,profile=default] ProductLicense: Warnings:[]}
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.433067 12269 docker_service.go:273] Setting cgroupDriver to cgroupfs
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.547318 12269 remote_runtime.go:59] parsed scheme: ""
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.547722 12269 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.548004 12269 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.548165 12269 clientconn.go:577] ClientConn switching balancer to "pick_first"
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.548799 12269 remote_image.go:50] parsed scheme: ""
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.548920 12269 remote_image.go:50] scheme "" not registered, fallback to default scheme
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.549184 12269 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
Oct 31 17:39:27 master1 kubelet: I1031 17:39:27.549310 12269 clientconn.go:577] ClientConn switching balancer to "pick_first"
Oct 31 17:39:28 master1 kubelet: E1031 17:39:28.095946 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:28 master1 kubelet: E1031 17:39:28.103635 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:28 master1 kubelet: E1031 17:39:28.104284 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:29 master1 kubelet: E1031 17:39:29.100668 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:29 master1 kubelet: E1031 17:39:29.108034 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:29 master1 kubelet: E1031 17:39:29.110804 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.107064 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.111348 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.115157 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
以上日志分析kubelet.go：等待apiserver启动起来
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.559823 12269 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Oct 31 17:39:30 master1 kubelet: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.611570 12269 kuberuntime_manager.go:207] Container runtime docker initialized, version: 19.03.4, apiVersion: 1.40.0
以上日志分析kuberuntime_manager.go:容器运行时（Container runtime）docker初始化完成，docker server的版本为19.03.4，API的版本为1.40.0
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.692676 12269 server.go:1065] Started kubelet
以上日志分析`kubelet服务启动完成‘
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.697173 12269 kubelet.go:1302] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.701757 12269 event.go:246] Unable to write event: 'Post https://192.168.156.211:6443/api/v1/namespaces/default/events: dial tcp 192.168.156.211:6443: connect: connection refused' (may retry after sleeping)
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.704927 12269 server.go:145] Starting to listen on 0.0.0.0:10250
以上日志分析`server.go‘：启动了一个本地监听10250（猜测是节点的kubelet服务和apiserver交互的端口）
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.754005 12269 server.go:354] Adding debug handlers to kubelet server.
以上日志分析`server.go‘：在kubelet服务中加入debug handlers
Oct 31 17:39:30 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.733626 12269 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
以上日志分析`fs_resource_analyzer.go‘： 启动FS ResourceAnalyzer
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.773754 12269 status_manager.go:156] Starting to sync pod status with apiserver
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.773967 12269 kubelet.go:1822] Starting kubelet main sync loop.
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.774129 12269 kubelet.go:1839] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
以上日志分析`同步pod的状态‘： 由于container runtime状态检查未完成，因此后续再做同步pod的状态
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.775574 12269 volume_manager.go:249] Starting Kubelet Volume Manager
以上日志分析`volume_manager.go‘： 启动Kubelet Volume管理
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.826644 12269 controller.go:135] failed to ensure node lease exists, will retry in 200ms, error: Get https://192.168.156.211:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master1?timeout=10s: dial tcp 192.168.156.211:6443: connect: connection refused
以上日志分析`检查节点租约时候存在‘： 通过调用apiserver检查
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.835859 12269 desired_state_of_world_populator.go:131] Desired state populator starts to run
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.845722 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.156.211:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
以上日志分析`调用apiserver查询CSIDriver‘
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.846569 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://192.168.156.211:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
以上日志分析`调用apiserver查询RuntimeClass‘
Oct 31 17:39:30 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.897594 12269 kubelet.go:1839] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Oct 31 17:39:30 master1 kubelet: E1031 17:39:30.908028 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.908374 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.967261 12269 clientconn.go:104] parsed scheme: "unix"
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.967340 12269 clientconn.go:104] scheme "unix" not registered, fallback to default scheme
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.967518 12269 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}
Oct 31 17:39:30 master1 kubelet: I1031 17:39:30.967659 12269 clientconn.go:577] ClientConn switching balancer to "pick_first"
Oct 31 17:39:30 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.010499 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.041155 12269 controller.go:135] failed to ensure node lease exists, will retry in 400ms, error: Get https://192.168.156.211:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master1?timeout=10s: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 kubelet: I1031 17:39:31.110733 12269 kubelet.go:1839] skipping pod synchronization - container runtime status check may not have completed yet
Oct 31 17:39:31 master1 kubelet: W1031 17:39:31.145375 12269 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "coredns-5644d7b6d9-rv29w_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "abd2a726d0b830c86e542fce986089a60ae9d09343fa689bfbb54962bdd9831d"
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.146359 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.146952 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.165877 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.166316 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.260751 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 kubelet: I1031 17:39:31.270058 12269 kubelet_node_status.go:72] Attempting to register node master1
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.276640 12269 kubelet_node_status.go:94] Unable to register node "master1" with API server: Post https://192.168.156.211:6443/api/v1/nodes: dial tcp 192.168.156.211:6443: connect: connection refused
以上日志分析`kubelet_node_status.go‘：尝试注册节点xxx，通过调用API服务的接口来判断节点是否启动成功。由于这里apiserver还没启动起来，因此后面一直会重试调用apiserver
Oct 31 17:39:31 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.362853 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 kubelet: W1031 17:39:31.401205 12269 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "coredns-5644d7b6d9-bcx8t_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "9d6499c3c52b9a2c49dea34fb9ba590d0e572a96c468e25bd0650c29cfe4b804"
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.473840 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 kubelet: I1031 17:39:31.485764 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.505997 12269 controller.go:135] failed to ensure node lease exists, will retry in 800ms, error: Get https://192.168.156.211:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master1?timeout=10s: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 kubelet: I1031 17:39:31.535997 12269 kubelet.go:1839] skipping pod synchronization - container runtime status check may not have completed yet
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.600367 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.714286 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:31 master1 kubelet: I1031 17:39:31.795291 12269 kubelet_node_status.go:72] Attempting to register node master1
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.802374 12269 kubelet_node_status.go:94] Unable to register node "master1" with API server: Post https://192.168.156.211:6443/api/v1/nodes: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.819034 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.855750 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://192.168.156.211:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.867369 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.156.211:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:31 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:31 master1 kubelet: E1031 17:39:31.925689 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:31 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器及image信息
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.026392 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.048236 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.128841 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.148721 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.156632 12269 cpu_manager.go:161] [cpumanager] starting with none policy
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.156667 12269 cpu_manager.go:162] [cpumanager] reconciling every 10s
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.156793 12269 policy_none.go:42] [cpumanager] none policy: Start
以上日志分析`cpumanager‘：负责管理cpu资源的分配，说明cpu资源的管理是由kubelet负责的（kubelet把cpu限制写入到docker容器的配置中）
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.170594 12269 plugin_manager.go:116] Starting Kubelet Plugin Manager
以上日志分析`plugin_manager.go‘：负责插件的管理，如果对kubelet有扩展，可以考虑Kubelet Plugin Manager相关了内容
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.171536 12269 eviction_manager.go:246] eviction manager: failed to get summary stats: failed to get node info: node "master1" not found
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.172147 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.172652 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.204399 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.229742 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.307992 12269 controller.go:135] failed to ensure node lease exists, will retry in 1.6s, error: Get https://192.168.156.211:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master1?timeout=10s: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.311985 12269 kubelet_node_status.go:72] Attempting to register node master1
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.315049 12269 kubelet_node_status.go:94] Unable to register node "master1" with API server: Post https://192.168.156.211:6443/api/v1/nodes: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.331623 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: W1031 17:39:32.336969 12269 pod_container_deletor.go:75] Container "abd2a726d0b830c86e542fce986089a60ae9d09343fa689bfbb54962bdd9831d" not found in pod's containers
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.338967 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.348264 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/8c026a5b6d81c05befbedbc789d3ae9a-k8s-certs") pod "kube-controller-manager-master1" (UID: "8c026a5b6d81c05befbedbc789d3ae9a")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.348719 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/8c026a5b6d81c05befbedbc789d3ae9a-kubeconfig") pod "kube-controller-manager-master1" (UID: "8c026a5b6d81c05befbedbc789d3ae9a")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.348874 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/8c026a5b6d81c05befbedbc789d3ae9a-ca-certs") pod "kube-controller-manager-master1" (UID: "8c026a5b6d81c05befbedbc789d3ae9a")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.349008 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/8c026a5b6d81c05befbedbc789d3ae9a-etc-pki") pod "kube-controller-manager-master1" (UID: "8c026a5b6d81c05befbedbc789d3ae9a")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.349210 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvolume-dir" (UniqueName: "kubernetes.io/host-path/8c026a5b6d81c05befbedbc789d3ae9a-flexvolume-dir") pod "kube-controller-manager-master1" (UID: "8c026a5b6d81c05befbedbc789d3ae9a")
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.432312 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.451036 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.453326 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.533907 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.552951 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeconfig" (UniqueName: "kubernetes.io/host-path/74dea8da17aa6241e5e4f7b2ba4e1d8e-kubeconfig") pod "kube-scheduler-master1" (UID: "74dea8da17aa6241e5e4f7b2ba4e1d8e")
Oct 31 17:39:32 master1 kubelet: W1031 17:39:32.580929 12269 status_manager.go:529] Failed to get status for pod "kube-controller-manager-master1_kube-system(8c026a5b6d81c05befbedbc789d3ae9a)": Get https://192.168.156.211:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-master1: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.616694 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.626077 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.638506 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.660210 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-certs" (UniqueName: "kubernetes.io/host-path/21e652bff56ef230951f4122f16eeb7b-etcd-certs") pod "etcd-master1" (UID: "21e652bff56ef230951f4122f16eeb7b")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.660558 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etcd-data" (UniqueName: "kubernetes.io/host-path/21e652bff56ef230951f4122f16eeb7b-etcd-data") pod "etcd-master1" (UID: "21e652bff56ef230951f4122f16eeb7b")
Oct 31 17:39:32 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.742538 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:32 master1 kubelet: W1031 17:39:32.809375 12269 status_manager.go:529] Failed to get status for pod "kube-scheduler-master1_kube-system(74dea8da17aa6241e5e4f7b2ba4e1d8e)": Get https://192.168.156.211:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-master1: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.850752 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.859346 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://192.168.156.211:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.871117 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.156.211:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.893673 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.896177 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:32 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:32 master1 kubelet: E1031 17:39:32.952328 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:32 master1 dockerd: time="2019-10-31T17:39:32.966944423+08:00" level=debug msg="Calling GET /version"
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.980411 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "ca-certs" (UniqueName: "kubernetes.io/host-path/5866dddc16342a08ee6e5154207433ad-ca-certs") pod "kube-apiserver-master1" (UID: "5866dddc16342a08ee6e5154207433ad")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.980750 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "etc-pki" (UniqueName: "kubernetes.io/host-path/5866dddc16342a08ee6e5154207433ad-etc-pki") pod "kube-apiserver-master1" (UID: "5866dddc16342a08ee6e5154207433ad")
Oct 31 17:39:32 master1 kubelet: I1031 17:39:32.981008 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-certs" (UniqueName: "kubernetes.io/host-path/5866dddc16342a08ee6e5154207433ad-k8s-certs") pod "kube-apiserver-master1" (UID: "5866dddc16342a08ee6e5154207433ad")
Oct 31 17:39:32 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.057663 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: I1031 17:39:33.117774 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.153138 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.158664 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.177670 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.178068 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.260166 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.282395 12269 status_manager.go:529] Failed to get status for pod "etcd-master1_kube-system(21e652bff56ef230951f4122f16eeb7b)": Get https://192.168.156.211:6443/api/v1/namespaces/kube-system/pods/etcd-master1: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.301681 12269 pod_container_deletor.go:75] Container "9d6499c3c52b9a2c49dea34fb9ba590d0e572a96c468e25bd0650c29cfe4b804" not found in pod's containers
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.302203 12269 pod_container_deletor.go:75] Container "fc2445c84f4c1449eafbcbbcc757ff2d2c44be552d9f45a84743fec8c8303588" not found in pod's containers
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.302383 12269 pod_container_deletor.go:75] Container "b4a05eebee8404cdea352c667dbcd4d339b1174183f7904c3c35d73377a460f4" not found in pod's containers
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.302613 12269 pod_container_deletor.go:75] Container "4d4d68dd5d61df56622e61348a03261387189356aca5ccd1dcf5d696e3e090f3" not found in pod's containers
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.302903 12269 pod_container_deletor.go:75] Container "5def88b61e9d79c4f54a123c2942603a0fc7821ec462e0eb80ff4be233f22ed8" not found in pod's containers
Oct 31 17:39:33 master1 kubelet: I1031 17:39:33.303539 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:33 master1 kubelet: I1031 17:39:33.306889 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 containerd: time="2019-10-31T17:39:33.351836135+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/9300e11b91fab949aeeb90b2b7799a81b88cbd47e61626218120d6983630a68c/shim.sock" debug=false pid=12658
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.361804 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 containerd: time="2019-10-31T17:39:33.407141535+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d05b5e41d509d0637a3c834946d6b27a6f8aacc4653fc1d515d3f5f5756f5422/shim.sock" debug=false pid=12681
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.468580 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 kubelet: I1031 17:39:33.540130 12269 kubelet_node_status.go:72] Attempting to register node master1
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.541908 12269 kubelet_node_status.go:94] Unable to register node "master1" with API server: Post https://192.168.156.211:6443/api/v1/nodes: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.579763 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.622126 12269 status_manager.go:529] Failed to get status for pod "kube-apiserver-master1_kube-system(5866dddc16342a08ee6e5154207433ad)": Get https://192.168.156.211:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-master1: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: I1031 17:39:33.629500 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:33 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.680838 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.785739 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 containerd: time="2019-10-31T17:39:33.808612380+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/8b4bd9f2579f91b51c5a053e1e7ed8500661e19c1577b5d574b16ee911588b05/shim.sock" debug=false pid=12767
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.864093 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://192.168.156.211:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.874952 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.156.211:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.887339 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:33 master1 kubelet: W1031 17:39:33.903311 12269 pod_container_deletor.go:75] Container "c4a99d10af7482a9eab350e5337d224044c9920bf817c069316ba7fbe8b00a67" not found in pod's containers
Oct 31 17:39:33 master1 kubelet: I1031 17:39:33.903824 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.912940 12269 controller.go:135] failed to ensure node lease exists, will retry in 3.2s, error: Get https://192.168.156.211:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master1?timeout=10s: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:33 master1 kubelet: E1031 17:39:33.988634 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 containerd: time="2019-10-31T17:39:34.019137413+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/0bb5e5b4653000a3b94562634983ddd31da34cd4227d3f462e544a89a91dccd7/shim.sock" debug=false pid=12827
Oct 31 17:39:34 master1 kubelet: I1031 17:39:34.057713 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.092200 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.157260 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.192795 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.192990 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: W1031 17:39:34.249341 12269 pod_container_deletor.go:75] Container "b2503ec1c7dcc4ca9068554097e832e63a7bbbd7c2d64967b6f6710ef8bb77cd" not found in pod's containers
Oct 31 17:39:34 master1 kubelet: I1031 17:39:34.249636 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.294700 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.384832 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.397269 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.499278 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: I1031 17:39:34.573093 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.606489 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 containerd: time="2019-10-31T17:39:34.613889832+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/5c8a0ea26908945969137736b6dcebc3d7d14279e2e3360034b5a09ddc678588/shim.sock" debug=false pid=12945
Oct 31 17:39:34 master1 containerd: time="2019-10-31T17:39:34.636028307+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c17bad8d6709091fbbb435667490405223e5bc311496089e72e18e65947600a0/shim.sock" debug=false pid=12960
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.707654 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 containerd: time="2019-10-31T17:39:34.709206015+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e4fee0eb9a7ef3590887ecadd1e27d90c3f8723a1b0f96c81cfe74669391b807/shim.sock" debug=false pid=12981
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: W1031 17:39:34.756647 12269 pod_container_deletor.go:75] Container "7b3bf8a52d348f9ac978db338a8296af284d41c798b8cd5ae6a95e7ce76ec5c7" not found in pod's containers
Oct 31 17:39:34 master1 kubelet: I1031 17:39:34.756970 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.812728 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.873108 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://192.168.156.211:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.891595 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.156.211:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:34 master1 kubelet: E1031 17:39:34.914717 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:34 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.025215 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 containerd: time="2019-10-31T17:39:35.060581597+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/35ab48cdcd03a62db46c0277d6ec954b56709f0fce1767d540cc1f9ab60b8ffb/shim.sock" debug=false pid=13058
Oct 31 17:39:35 master1 kubelet: I1031 17:39:35.081207 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.126585 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: I1031 17:39:35.147380 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.164989 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.195636 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.227777 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: W1031 17:39:35.251909 12269 pod_container_deletor.go:75] Container "bc3c7f7470354d57fe7909d4322e9d1c40e2f3cf7c541ab5e18a5b023a25a9bc" not found in pod's containers
Oct 31 17:39:35 master1 kubelet: I1031 17:39:35.252205 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.335462 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.395843 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:35 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.452852 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.554775 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: I1031 17:39:35.615749 12269 kubelet_node_status.go:72] Attempting to register node master1
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.621086 12269 kubelet_node_status.go:94] Unable to register node "master1" with API server: Post https://192.168.156.211:6443/api/v1/nodes: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:35 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.660679 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.767502 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.878125 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.878354 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: Get https://192.168.156.211:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:35 master1 kubelet: W1031 17:39:35.880532 12269 pod_container_deletor.go:75] Container "d05b5e41d509d0637a3c834946d6b27a6f8aacc4653fc1d515d3f5f5756f5422" not found in pod's containers
Oct 31 17:39:35 master1 kubelet: I1031 17:39:35.880733 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.897285 12269 event.go:246] Unable to write event: 'Post https://192.168.156.211:6443/api/v1/namespaces/default/events: dial tcp 192.168.156.211:6443: connect: connection refused' (may retry after sleeping)
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.978758 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:35 master1 kubelet: E1031 17:39:35.985797 12269 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.156.211:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.085538 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:36 master1 kubelet: W1031 17:39:36.096351 12269 docker_container.go:224] Deleted previously existing symlink file: "/var/log/pods/kube-system_kube-scheduler-master1_74dea8da17aa6241e5e4f7b2ba4e1d8e/kube-scheduler/5.log"
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.201168 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.201755 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.156.211:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:36 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.327491 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.392170 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: Get https://192.168.156.211:6443/api/v1/nodes?fieldSelector=metadata.name%3Dmaster1&limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:36 master1 kubelet: I1031 17:39:36.443046 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.474921 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.580653 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.584662 12269 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.156.211:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.156.211:6443: connect: connection refused
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.688537 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 kubelet: W1031 17:39:36.691668 12269 pod_container_deletor.go:75] Container "9300e11b91fab949aeeb90b2b7799a81b88cbd47e61626218120d6983630a68c" not found in pod's containers
Oct 31 17:39:36 master1 kubelet: E1031 17:39:36.800232 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:36 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:37 master1 kubelet: E1031 17:39:37.013789 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:37 master1 kubelet: I1031 17:39:37.016467 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:37 master1 kubelet: E1031 17:39:37.118270 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:37 master1 kubelet: E1031 17:39:37.218720 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:38 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:38 master1 kubelet: E1031 17:39:38.267513 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:38 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:38 master1 kubelet: I1031 17:39:38.377462 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:38 master1 kubelet: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:38 master1 kubelet: I1031 17:39:38.713163 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:38 master1 dockerd: time="2019-10-31T17:39:38.737648834+08:00" level=debug msg="Calling GET /version"
Oct 31 17:39:38 master1 dockerd: time="2019-10-31T17:39:38.737892439+08:00" level=debug msg="Calling GET /containers/json?all=1&filters=%7B%22label%22%3A%7B%22io.kubernetes.docker.type%3Dpodsandbox%22%3Atrue%2C%22io.kubernetes.pod.uid%3D5866dddc16342a08ee6e5154207433ad%22%3Atrue%7D%7D&limit=0"
Oct 31 17:39:38 master1 kubelet: E1031 17:39:38.808260 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:38 master1 kubelet: I1031 17:39:38.825615 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:38 master1 dockerd: time="2019-10-31T17:39:38.828043183+08:00" level=debug msg="Calling GET /version"
Oct 31 17:39:38 master1 dockerd: time="2019-10-31T17:39:38.828463067+08:00" level=debug msg="Calling GET /version"
Oct 31 17:39:38 master1 dockerd: time="2019-10-31T17:39:38.859675003+08:00" level=debug msg="Calling GET /version"
Oct 31 17:39:38 master1 kubelet: E1031 17:39:38.908681 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:38 master1 dockerd: time="2019-10-31T17:39:38.992284316+08:00" level=debug msg="Calling GET /version"
Oct 31 17:39:39 master1 kubelet: E1031 17:39:39.012121 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:39 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:40 master1 kubelet: I1031 17:39:40.218596 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:40 master1 kubelet: I1031 17:39:40.244144 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:40 master1 kubelet: I1031 17:39:40.219389 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:40 master1 kubelet: I1031 17:39:40.245794 12269 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Oct 31 17:39:40 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:42 master1 kubelet: E1031 17:39:42.172654 12269 eviction_manager.go:246] eviction manager: failed to get summary stats: failed to get node info: node "master1" not found
Oct 31 17:39:42 master1 kubelet: E1031 17:39:42.217304 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:42 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（etcd、apiserver等）的pause容器及组件容器
Oct 31 17:39:46 master1 kubelet: E1031 17:39:46.088765 12269 kubelet.go:2267] node "master1" not found
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.174704 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/e7f8835e-ef7e-4557-bee5-85f8841bfac6-config-volume") pod "coredns-5644d7b6d9-rv29w" (UID: "e7f8835e-ef7e-4557-bee5-85f8841bfac6")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.175017 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-xrn7q" (UniqueName: "kubernetes.io/secret/e7f8835e-ef7e-4557-bee5-85f8841bfac6-coredns-token-xrn7q") pod "coredns-5644d7b6d9-rv29w" (UID: "e7f8835e-ef7e-4557-bee5-85f8841bfac6")
以上日志分析：以上大部分日志都在启动apiserver、etcd、scheduler、controller-manager的pause容器及组件容器，启动完成后再执行后续的操作
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.190918 12269 kuberuntime_manager.go:961] updating runtime config through cri with podcidr 10.244.0.0/24
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.191813 12269 docker_service.go:355] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.244.0.0/24,},}
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.192614 12269 kubelet_network.go:77] Setting Pod CIDR: -> 10.244.0.0/24
以上日志分析`设置Pod的CIDR为10.244.0.0/24‘
Oct 31 17:39:46 master1 kubelet: E1031 17:39:46.412399 12269 reflector.go:123] object-"kube-system"/"coredns": Failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:master1" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node "master1" and this object
Oct 31 17:39:46 master1 kubelet: RBAC: [clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "system:certificates.k8s.io:certificatesigningrequests:selfnodeclient" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
Oct 31 17:39:46 master1 kubelet: E1031 17:39:46.459085 12269 reflector.go:123] object-"kube-system"/"coredns-token-xrn7q": Failed to list *v1.Secret: secrets "coredns-token-xrn7q" is forbidden: User "system:node:master1" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node "master1" and this object
Oct 31 17:39:46 master1 kubelet: RBAC: [clusterrole.rbac.authorization.k8s.io "system:certificates.k8s.io:certificatesigningrequests:selfnodeclient" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found, clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found]
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.547131 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/2b2db778-f340-4e9e-b5da-f127acd854ea-kube-proxy") pod "kube-proxy-dhtbj" (UID: "2b2db778-f340-4e9e-b5da-f127acd854ea")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.547670 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "coredns-token-xrn7q" (UniqueName: "kubernetes.io/secret/720a9521-1194-4396-b4e2-7175fa49702a-coredns-token-xrn7q") pod "coredns-5644d7b6d9-bcx8t" (UID: "720a9521-1194-4396-b4e2-7175fa49702a")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.548074 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-p4dxx" (UniqueName: "kubernetes.io/secret/2b2db778-f340-4e9e-b5da-f127acd854ea-kube-proxy-token-p4dxx") pod "kube-proxy-dhtbj" (UID: "2b2db778-f340-4e9e-b5da-f127acd854ea")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.548356 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni" (UniqueName: "kubernetes.io/host-path/31658a4b-0431-43cb-b9a0-5c2419326430-cni") pod "kube-flannel-ds-amd64-24lfb" (UID: "31658a4b-0431-43cb-b9a0-5c2419326430")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.558226 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-cfg" (UniqueName: "kubernetes.io/configmap/31658a4b-0431-43cb-b9a0-5c2419326430-flannel-cfg") pod "kube-flannel-ds-amd64-24lfb" (UID: "31658a4b-0431-43cb-b9a0-5c2419326430")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.558659 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "run" (UniqueName: "kubernetes.io/host-path/31658a4b-0431-43cb-b9a0-5c2419326430-run") pod "kube-flannel-ds-amd64-24lfb" (UID: "31658a4b-0431-43cb-b9a0-5c2419326430")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.558947 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-token-nnqsk" (UniqueName: "kubernetes.io/secret/31658a4b-0431-43cb-b9a0-5c2419326430-flannel-token-nnqsk") pod "kube-flannel-ds-amd64-24lfb" (UID: "31658a4b-0431-43cb-b9a0-5c2419326430")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.559328 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "config-volume" (UniqueName: "kubernetes.io/configmap/720a9521-1194-4396-b4e2-7175fa49702a-config-volume") pod "coredns-5644d7b6d9-bcx8t" (UID: "720a9521-1194-4396-b4e2-7175fa49702a")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.559828 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/2b2db778-f340-4e9e-b5da-f127acd854ea-xtables-lock") pod "kube-proxy-dhtbj" (UID: "2b2db778-f340-4e9e-b5da-f127acd854ea")
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.560075 12269 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/2b2db778-f340-4e9e-b5da-f127acd854ea-lib-modules") pod "kube-proxy-dhtbj" (UID: "2b2db778-f340-4e9e-b5da-f127acd854ea")
以上日志分析`reconciler.go‘：VerifyControllerAttachedVolume检查Attached Volume
Oct 31 17:39:46 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.707686 12269 kubelet_node_status.go:114] Node master1 was previously registered
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.708022 12269 kubelet_node_status.go:75] Successfully registered node master1
Oct 31 17:39:46 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:46 master1 kubelet: I1031 17:39:46.927275 12269 reconciler.go:154] Reconciler: start to sync state
Oct 31 17:39:46 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:47 master1 kubelet: E1031 17:39:47.307171 12269 configmap.go:203] Couldn't get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the condition
Oct 31 17:39:47 master1 kubelet: E1031 17:39:47.335171 12269 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/configmap/e7f8835e-ef7e-4557-bee5-85f8841bfac6-config-volume\" (\"e7f8835e-ef7e-4557-bee5-85f8841bfac6\")" failed. No retries permitted until 2019-10-31 17:39:47.83498694 +0800 CST m=+22.108460783 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e7f8835e-ef7e-4557-bee5-85f8841bfac6-config-volume\") pod \"coredns-5644d7b6d9-rv29w\" (UID: \"e7f8835e-ef7e-4557-bee5-85f8841bfac6\") : failed to sync configmap cache: timed out waiting for the condition"
Oct 31 17:39:47 master1 kubelet: E1031 17:39:47.343631 12269 secret.go:198] Couldn't get secret kube-system/coredns-token-xrn7q: failed to sync secret cache: timed out waiting for the condition
Oct 31 17:39:47 master1 kubelet: E1031 17:39:47.344393 12269 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/e7f8835e-ef7e-4557-bee5-85f8841bfac6-coredns-token-xrn7q\" (\"e7f8835e-ef7e-4557-bee5-85f8841bfac6\")" failed. No retries permitted until 2019-10-31 17:39:47.844014041 +0800 CST m=+22.117466093 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"coredns-token-xrn7q\" (UniqueName: \"kubernetes.io/secret/e7f8835e-ef7e-4557-bee5-85f8841bfac6-coredns-token-xrn7q\") pod \"coredns-5644d7b6d9-rv29w\" (UID: \"e7f8835e-ef7e-4557-bee5-85f8841bfac6\") : failed to sync secret cache: timed out waiting for the condition"
Oct 31 17:39:47 master1 kubelet: W1031 17:39:47.401653 12269 kuberuntime_container.go:696] No ref for container {"docker" "9cdfa09b8400aba0c591f4cb154226767841ad460fa701907edd69309d92c306"}
Oct 31 17:39:47 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:47 master1 kubelet: E1031 17:39:47.766062 12269 secret.go:198] Couldn't get secret kube-system/coredns-token-xrn7q: failed to sync secret cache: timed out waiting for the condition
以上日志分析`secret.go‘： secret.go负责底层管理secret
Oct 31 17:39:47 master1 kubelet: E1031 17:39:47.766353 12269 nestedpendingoperations.go:270] Operation for "\"kubernetes.io/secret/720a9521-1194-4396-b4e2-7175fa49702a-coredns-token-xrn7q\" (\"720a9521-1194-4396-b4e2-7175fa49702a\")" failed. No retries permitted until 2019-10-31 17:39:48.266252869 +0800 CST m=+22.539700171 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"coredns-token-xrn7q\" (UniqueName: \"kubernetes.io/secret/720a9521-1194-4396-b4e2-7175fa49702a-coredns-token-xrn7q\") pod \"coredns-5644d7b6d9-bcx8t\" (UID: \"720a9521-1194-4396-b4e2-7175fa49702a\") : failed to sync secret cache: timed out waiting for the condition"
Oct 31 17:39:47 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:48 master1 containerd: time="2019-10-31T17:39:48.052933776+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ec045a0ddbb864c037bd732d76b5071e8539d70fe5bff02dc1784956cfbdfd05/shim.sock" debug=false pid=13516
Oct 31 17:39:48 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:48 master1 kubelet: W1031 17:39:48.126196 12269 cni.go:328] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "abd2a726d0b830c86e542fce986089a60ae9d09343fa689bfbb54962bdd9831d"
Oct 31 17:39:48 master1 containerd: time="2019-10-31T17:39:48.336506754+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/06e93e5d4cae624a603cc0599f78cd9bbc80ed30735c82c4a26eec0e453cfa03/shim.sock" debug=false pid=13555
Oct 31 17:39:48 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:48 master1 kubelet: W1031 17:39:48.492002 12269 cni.go:328] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "9d6499c3c52b9a2c49dea34fb9ba590d0e572a96c468e25bd0650c29cfe4b804"
以上日志分析`cni.go‘： cni.go负责调用cni网络接口，负责管理网络
Oct 31 17:39:48 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:49 master1 containerd: time="2019-10-31T17:39:49.043750023+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/dced07acac0c6f9c93fa58551f2fc70fc1cd3c2c71c01c92918f71403d078be8/shim.sock" debug=false pid=13673
Oct 31 17:39:49 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:49 master1 kubelet: W1031 17:39:49.197240 12269 pod_container_deletor.go:75] Container "ec045a0ddbb864c037bd732d76b5071e8539d70fe5bff02dc1784956cfbdfd05" not found in pod's containers
以上日志分析`pod_container_deletor.go‘： pod_container_deletor.go负责删除容器
Oct 31 17:39:49 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:49 master1 containerd: time="2019-10-31T17:39:49.450837246+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ee8673826915f94131f14baa2f4a3fa95609b3f61a9d73e192cf3307fb6a5cb6/shim.sock" debug=false pid=13702
Oct 31 17:39:49 master1 containerd: time="2019-10-31T17:39:49.547238063+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/4586344a37e057be027d0af184345c6d39a48b4d27ac7bb50c1287b122c988f9/shim.sock" debug=false pid=13717
Oct 31 17:39:49 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:50 master1 kubelet: W1031 17:39:50.097836 12269 pod_container_deletor.go:75] Container "06e93e5d4cae624a603cc0599f78cd9bbc80ed30735c82c4a26eec0e453cfa03" not found in pod's containers
Oct 31 17:39:50 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:50 master1 containerd: time="2019-10-31T17:39:50.362681102+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d0ba024a1f96a3b2cb4407586ed123475faf46601b2ca90e9e15510b145f70b6/shim.sock" debug=false pid=13782
Oct 31 17:39:50 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:51 master1 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 31 17:39:51 master1 kernel: IPv6: ADDRCONF(NETDEV_UP): veth4cddb4f1: link is not ready
Oct 31 17:39:51 master1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth4cddb4f1: link becomes ready
Oct 31 17:39:51 master1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Oct 31 17:39:51 master1 kernel: cni0: port 1(veth4cddb4f1) entered blocking state
Oct 31 17:39:51 master1 kernel: cni0: port 1(veth4cddb4f1) entered disabled state
Oct 31 17:39:51 master1 kernel: device veth4cddb4f1 entered promiscuous mode
Oct 31 17:39:51 master1 kernel: cni0: port 1(veth4cddb4f1) entered blocking state
Oct 31 17:39:51 master1 kernel: cni0: port 1(veth4cddb4f1) entered forwarding state
Oct 31 17:39:51 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:51 master1 kubelet: W1031 17:39:51.652506 12269 pod_container_deletor.go:75] Container "ee8673826915f94131f14baa2f4a3fa95609b3f61a9d73e192cf3307fb6a5cb6" not found in pod's containers
Oct 31 17:39:51 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:51 master1 kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Oct 31 17:39:51 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:51 master1 kernel: cni0: port 2(vethca9460e6) entered blocking state
Oct 31 17:39:51 master1 kernel: cni0: port 2(vethca9460e6) entered disabled state
Oct 31 17:39:51 master1 kernel: device vethca9460e6 entered promiscuous mode
Oct 31 17:39:51 master1 kernel: cni0: port 2(vethca9460e6) entered blocking state
Oct 31 17:39:51 master1 kernel: cni0: port 2(vethca9460e6) entered forwarding state
Oct 31 17:39:51 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:51 master1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
以上日志分析`kernel‘: 负责管理 cni0 、 eth0 、veth 等网络
Oct 31 17:39:51 master1 containerd: time="2019-10-31T17:39:51.794002416+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/ad83601b24499cdbd30fc8c0a1fa59b4956747a5c2e9884bff5912a29328e530/shim.sock" debug=false pid=13954
Oct 31 17:39:51 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:52 master1 containerd: time="2019-10-31T17:39:52.098672991+08:00" level=info msg="shim reaped" id=d0ba024a1f96a3b2cb4407586ed123475faf46601b2ca90e9e15510b145f70b6
Oct 31 17:39:52 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:52 master1 containerd: time="2019-10-31T17:39:52.168362943+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/c0ad14b699d01e402056e2508a7085afdf0dbfd266a3d5f1332595db11176b13/shim.sock" debug=false pid=14003
Oct 31 17:39:52 master1 kubelet: W1031 17:39:52.186306 12269 container.go:409] Failed to create summary reader for "/kubepods/burstable/pod31658a4b-0431-43cb-b9a0-5c2419326430/d0ba024a1f96a3b2cb4407586ed123475faf46601b2ca90e9e15510b145f70b6": none of the resources are being tracked.
Oct 31 17:39:52 master1 dockerd: 截掉部分调用dockerAPI日志，查询image，并创建k8s基础组件（kube-proxy、flannel、coredns等）的pause容器及组件容器
Oct 31 17:39:53 master1 containerd: time="2019-10-31T17:39:53.954733164+08:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/d7b6fa88f48e5a5700d49e66de59b10280f0eb8a1cbd6bdc0754eb758b6e9984/shim.sock" debug=false pid=14090
Oct 31 17:39:54 master1 dockerd: 截掉部分调用dockerAPI日志，查询容器状态