{"id":4690,"date":"2021-04-15T21:19:36","date_gmt":"2021-04-15T09:19:36","guid":{"rendered":"http:\/\/www.zoyinc.com\/?p=4690"},"modified":"2024-01-04T19:02:03","modified_gmt":"2024-01-04T06:02:03","slug":"restored-okd-vm-not-starting","status":"publish","type":"post","link":"http:\/\/www.zoyinc.com\/?p=4690","title":{"rendered":"Restored OKD Cluster Not Starting"},"content":{"rendered":"\n

I run my OKD cluster on a set of VMs in VMware ESXi. My backups consist of shutting down all VMs and then copying them somewhere else – so full shutdown and backup.<\/p>\n\n\n\n

The cluster setup was done using the steps outlined in OKD 4.5 small cluster on ESX<\/a>.<\/p>\n\n\n\n

This has worked well and I have been able to restart the OKD cluster after a full restore with no issue. But now the cluster doesn’t start, OKD web console doesn’t work and when I monitor the VMs and ESX host the CPU, RAM, Disk and Network resources are all really low. <\/p>\n\n\n\n

I struggled a long time to figure out how to start up what is essentiallya really old VM backup set. I finally got it figured, I think, but this is on the basis of the steps, order and timing I have outlined below.<\/p>\n\n\n\n

So I am now starting up a full backup set that was backed up on 13\/1\/2021 and today is the 14\/3\/2021.<\/p>\n\n\n\n

I have two ESXi servers, “HP3” has the services and control-plane nodes and “Lenovo5” has the computer\/worker node.<\/p>\n\n\n\n

Failed restore<\/h1>\n\n\n\n

To demonstrate the problem I started the services node and the control pane node. These two nodes are both on the “HP3” ESX server. The below is the memory and CPU usage for 24 minutes after I started these nodes. As you can see there was very little activity. I started the nodes at 19:14:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

When you try to login you just get:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Prechecks<\/h1>\n\n\n\n

Before starting check that the date and time are correct on all ESX servers so the initial time for the VMs will all be more or less in sync with each other prior to NTP kicking in.<\/p>\n\n\n\n

Services node<\/h1>\n\n\n\n

To begin with only start the services node as this doesn’t actually start OKD and is important as it needs to provide NTP, DNS and Proxy services. <\/p>\n\n\n\n

There are some below things you should ensure are done prior to starting the other nodes. <\/p>\n\n\n\n

Add a firewall rule for NTP and restart:<\/p>\n\n\n\n

<\/path><\/path><\/svg><\/span>
firewall-cmd <\/span>--permanent<\/span> <\/span>--zone=public<\/span> <\/span>--add-port=123\/udp<\/span><\/span>\nsystemctl <\/span>restart<\/span> <\/span>firewalld<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Then I edit the chrony config. So backup and then edit “\/etc\/chrony.conf”. For me, in New Zealand, this is the chrony.conf file I used:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
#<\/span><\/span>\n# Example chrony file from zoyinc.com<\/span><\/span>\n#<\/span><\/span>\n# Using New Zealand NPT servers - Please set to your local NTP public servers<\/span><\/span>\n#server 43.252.70.34<\/span><\/span>\nserver <\/span>0.pool.ntp.org<\/span><\/span>\nserver <\/span>1.pool.ntp.org<\/span><\/span>\nserver <\/span>2.pool.ntp.org<\/span><\/span>\nserver <\/span>3.pool.ntp.org<\/span><\/span>\nserver <\/span>216.239.35.0<\/span><\/span>\nserver <\/span>216.239.35.4<\/span><\/span>\n# Record the rate at which the system clock gains\/losses time.<\/span><\/span>\ndriftfile <\/span>\/var\/lib\/chrony\/drift<\/span><\/span>\n# Allow the system clock to be stepped in the first three updates<\/span><\/span>\n# if its offset is larger than 1 second.<\/span><\/span>\nmakestep <\/span>1.0<\/span> <\/span>3<\/span><\/span>\n# Enable kernel synchronization of the real-time clock (RTC).<\/span><\/span>\nrtcsync<\/span><\/span>\n# Allow NTP client access from local network.<\/span><\/span>\nallow <\/span>192.168.0.0\/<\/span>16<\/span><\/span>\n# Serve time even if not synchronized to a time source.<\/span><\/span>\nlocal<\/span> <\/span>stratum<\/span> <\/span>10<\/span><\/span>\n# Specify directory for log files.<\/span><\/span>\nlogdir <\/span>\/var\/log\/chrony<\/span><\/span>\n# Select which information is logged.<\/span><\/span>\nlog <\/span>measurements<\/span> <\/span>statistics<\/span> <\/span>tracking<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Now restart:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
systemctl <\/span>restart<\/span> <\/span>chronyd.service<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Check the sources for chrony by running “chronyc sources”: This should return something like:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# chronyc sources<\/span><\/span>\n210 <\/span>Number<\/span> <\/span>of<\/span> <\/span>sources<\/span> <\/span>=<\/span> <\/span>6<\/span><\/span>\nMS <\/span>Name\/IP<\/span> <\/span>address<\/span>         <\/span>Stratum<\/span> <\/span>Poll<\/span> <\/span>Reach<\/span> <\/span>LastRx<\/span> <\/span>Last<\/span> <\/span>sample<\/span><\/span>\n=<\/span>==============================================================================<\/span><\/span>\n^? <\/span>time1.google.com<\/span>              <\/span>1<\/span>   <\/span>6<\/span>     <\/span>1<\/span>    <\/span>28<\/span>   <\/span>-984us[ -984us]<\/span> <\/span>+\/-<\/span>   <\/span>80ms<\/span><\/span>\n^? <\/span>time2.google.com<\/span>              <\/span>1<\/span>   <\/span>6<\/span>     <\/span>1<\/span>    <\/span>28<\/span>    <\/span>-18us[  -18us]<\/span> <\/span>+\/-<\/span>   <\/span>60ms<\/span><\/span>\n^? <\/span>ns1.att.wlg.telesmart.c<\/span>o>     <\/span>2<\/span>   <\/span>6<\/span>     <\/span>1<\/span>    <\/span>29<\/span>  <\/span>-1792us[-1792us]<\/span> <\/span>+\/-<\/span>   <\/span>14ms<\/span><\/span>\n^? <\/span>ip-<\/span>103<\/span>-<\/span>106<\/span>-<\/span>65<\/span>-219.addr.<\/span>l>     <\/span>2<\/span>   <\/span>6<\/span>     <\/span>1<\/span>    <\/span>30<\/span>   <\/span>-576us[ -576us]<\/span> <\/span>+\/-<\/span>   <\/span>38ms<\/span><\/span>\n^? <\/span>101<\/span>-<\/span>100<\/span>-<\/span>146<\/span>-146.myrepub<\/span>l>     <\/span>2<\/span>   <\/span>6<\/span>     <\/span>1<\/span>    <\/span>30<\/span>   <\/span>+962us[<\/span> <\/span>+962us]<\/span> <\/span>+\/-<\/span>   <\/span>52ms<\/span><\/span>\n^? <\/span>ns2.tdc.akl.telesmart.c<\/span>o>     <\/span>2<\/span>   <\/span>6<\/span>     <\/span>1<\/span>    <\/span>30<\/span>   <\/span>-813us[ -813us]<\/span> <\/span>+\/-<\/span> <\/span>6027us<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
It appears that the worker\/master nodes will use UTC so for consistency enable UTC on the services VM by running:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
timedatectl <\/span>set-timezone<\/span> <\/span>UTC<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Initial startup<\/h1>\n\n\n\n
Now that the services node is up and configured start up the control plane node – do NOT start the compute\/worker node yet.<\/p>\n\n\n\n
On the services node run:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
export<\/span> <\/span>KUBECONFIG<\/span>=<\/span>\/opt\/okd4\/install_dir\/auth\/kubeconfig<\/span><\/span>\noc <\/span>get<\/span> <\/span>csr<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Because you have just started the Control Pane this will return:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc get csr<\/span><\/span>\nUnable <\/span>to<\/span> <\/span>connect<\/span> <\/span>to<\/span> <\/span>the<\/span> <\/span>server:<\/span> <\/span>x509:<\/span> <\/span>certificate<\/span> <\/span>has<\/span> <\/span>expired<\/span> <\/span>or<\/span> <\/span>is<\/span> <\/span>not<\/span> <\/span>yet<\/span> <\/span>valid<\/span><\/span>\n[root@okd4-services ~]#<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Keep running “oc get csr” until you get a certificate. You may see the following while you wait for it to start<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc get csr<\/span><\/span>\nNo <\/span>resources<\/span> <\/span>found<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
This could take a few minutes so be patient:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc get csr<\/span><\/span>\nNAME        <\/span>AGE<\/span>   <\/span>SIGNERNAME<\/span>                                    <\/span>REQUESTOR<\/span>                    <\/span>CONDITION<\/span><\/span>\ncsr-jnpdf   <\/span>28s<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Pending<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Approve this certificate using “oc adm certificate approve <csr name>”:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc adm certificate approve csr-d5gt4<\/span><\/span>\ncertificatesigningrequest.certificates.k8s.io\/csr-d5gt4 <\/span>approved<\/span><\/span>\n[root@okd4-services ~]# oc get csr<\/span><\/span>\nNAME        <\/span>AGE<\/span>   <\/span>SIGNERNAME<\/span>                                    <\/span>REQUESTOR<\/span>                                                                   <\/span>CONDITION<\/span><\/span>\ncsr-jnpdf   <\/span>42s<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Approved,Issued<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Keep looking for approvals, we are expecting a “system:node” csr for the control plane. It will look like:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
NAME        <\/span>AGE<\/span>   <\/span>SIGNERNAME<\/span>                                    <\/span>REQUESTOR<\/span>                                                                   <\/span>CONDITION<\/span><\/span>\ncsr-55m7p   <\/span>19s<\/span>   <\/span>kubernetes.io\/kubelet-serving<\/span>                 <\/span>system:node:okd4-control-plane-1.lab.okd.local<\/span>                              <\/span>Pending<\/span><\/span>\ncsr-d5gt4   <\/span>63s<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Approved,Issued<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
You need to also approve this as before:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc adm certificate approve csr-55m7p<\/span><\/span>\ncertificatesigningrequest.certificates.k8s.io\/csr-55m7p <\/span>approved<\/span><\/span>\n[root@okd4-services ~]# oc get csr<\/span><\/span>\nNAME        <\/span>AGE<\/span>   <\/span>SIGNERNAME<\/span>                                    <\/span>REQUESTOR<\/span>                                                                   <\/span>CONDITION<\/span><\/span>\ncsr-55m7p   <\/span>41s<\/span>   <\/span>kubernetes.io\/kubelet-serving<\/span>                 <\/span>system:node:okd4-control-plane-1.lab.okd.local<\/span>                              <\/span>Approved,Issued<\/span><\/span>\ncsr-d5gt4   <\/span>85s<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Approved,Issued<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Wait for web console to come up<\/h1>\n\n\n\n
At this point simply wait for the web console to come up, this could take 10 minutes. Once the web console comes up you will also see a lot more CPU activity and memory usage compared to the earlier screenshots when OKD didn’t start.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Start the compute\/worker node<\/h2>\n\n\n\n
Now that the web console is up you will be able to see some things but some things are still not showing:<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
So now start the compute\/worker node.<\/p>\n\n\n\n
As before keep monitoring for csrs by running “oc get csr” on the services node.<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc get csr<\/span><\/span>\nNAME        <\/span>AGE<\/span>   <\/span>SIGNERNAME<\/span>                                    <\/span>REQUESTOR<\/span>                                                                   <\/span>CONDITION<\/span><\/span>\ncsr-2d68r   <\/span>26s<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Pending<\/span><\/span>\ncsr-jnpdf   <\/span>15m<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Approved,Issued<\/span><\/span>\ncsr-rxwww   <\/span>14m<\/span>   <\/span>kubernetes.io\/kubelet-serving<\/span>                 <\/span>system:node:okd4-control-plane-1.lab.okd.local<\/span>                              <\/span>Approved,Issued<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
Approve the certificates as they come through. Once this is stabilized you should see:<\/p>\n\n\n\n
<\/path><\/path><\/svg><\/span>
[root@okd4-services ~]# oc get csr<\/span><\/span>\nNAME        <\/span>AGE<\/span>    <\/span>SIGNERNAME<\/span>                                    <\/span>REQUESTOR<\/span>                                                                   <\/span>CONDITION<\/span><\/span>\ncsr-2d68r   <\/span>2m4s<\/span>   <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Approved,Issued<\/span><\/span>\ncsr-jnpdf   <\/span>17m<\/span>    <\/span>kubernetes.io\/kube-apiserver-client-kubelet<\/span>   <\/span>system:serviceaccount:openshift-machine-config-operator:node-bootstrapper<\/span>   <\/span>Approved,Issued<\/span><\/span>\ncsr-rxwww   <\/span>15m<\/span>    <\/span>kubernetes.io\/kubelet-serving<\/span>                 <\/span>system:node:okd4-control-plane-1.lab.okd.local<\/span>                              <\/span>Approved,Issued<\/span><\/span>\ncsr-wlbkc   <\/span>61s<\/span>    <\/span>kubernetes.io\/kubelet-serving<\/span>                 <\/span>system:node:okd4-compute-1.lab.okd.local<\/span>                                    <\/span>Approved,Issued<\/span><\/span><\/code><\/pre><\/div>\n\n\n\n
It is worth noting that the above 4 certificate signing requests include one each for the control plane and compute node plus an equal number of bootstrap requests.<\/p>\n\n\n\n
In say 5 minutes you should see the web console looking much healthier:<\/p>\n\n\n\n
\"\"<\/a><\/figure>\n\n\n\n
The CPU and memory on the HP3 ESX host also looks a lot healthier. Note I started the control plane node at 19:33 and the compute node at 19:56:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
I run my OKD cluster on a set of VMs in VMware ESXi. My backups consist of shutting down all VMs and then copying them somewhere else – so full shutdown and backup. The cluster setup was done using the steps outlined in OKD 4.5 small cluster on ESX. This has worked well and I […]<\/p>\n","protected":false},"author":2,"featured_media":5486,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[796,677,552],"tags":[327,519,343,342,721,722,720,679,719,678,497,723,372,308],"class_list":["post-4690","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-home-page","category-openshift-okd","category-redhat","tag-backup","tag-date","tag-esx","tag-esxi","tag-failed","tag-failed-to-start","tag-historical","tag-okd","tag-old","tag-openshift","tag-restore","tag-rollback","tag-vm","tag-vmware"],"_links":{"self":[{"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/posts\/4690","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4690"}],"version-history":[{"count":17,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/posts\/4690\/revisions"}],"predecessor-version":[{"id":5817,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/posts\/4690\/revisions\/5817"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=\/wp\/v2\/media\/5486"}],"wp:attachment":[{"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4690"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4690"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.zoyinc.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4690"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}