如何快速查看Kubernetes Pod崩潰前的日志

當(dāng)pod處于crash狀態(tài)的時(shí)候，容器不斷重啟，此時(shí)用kubelet logs可能出現(xiàn)一直捕捉不到日志。解決方法：

kubectl previous參數(shù)作用：If true, print the logs for the previous instance of the container in a pod if it exists.

單容器pod：

kubectllogspod-name--previous

多容器pod：

kubectllogspod-name--previous-ccontainer-name

比如：

NAMEREADYSTATUSRESTARTSAGE
nginx-7d8b49557c-c2lx92/2Running5

kubectllogsnginx-7d8b49557c-c2lx9--previous
Error:xxxxxxxxxxx

kubelet會(huì)保持pod的前幾個(gè)失敗的容器，這個(gè)是查看的前提條件。kubelet實(shí)現(xiàn)previous的原理：將pod的日志存放在/var/log/pods/podname，并且是鏈接文件，鏈接到docker的容器的日志文件，同時(shí)kubelet還會(huì)保留上一個(gè)容器，同時(shí)有一個(gè)鏈接文件鏈接到pod上一個(gè)崩潰的容器的日志文件，使用previous就是查看的這個(gè)文件

比如查看一個(gè)pod：

ubuntu@~$kubeletgetpod
NAMEREADYSTATUSRESTARTSAGE
busybox1/1Running239499d
nginx-deployment-6wlhd1/1Running079d
redis1/1Running049d

到pod所在node查看kubelet放的兩個(gè)日志文件：

ls/var/log/pods/default_busybox_f72ab71a-5b3b-4ecf-940d-28a5c3b30683/busybox
2393.log2394.log

數(shù)字的含義：2393證明是第 2393 次重啟后的日志，2394代表是第2394次重啟后的日志。

實(shí)際這兩個(gè)日志文件是鏈接文件，指向了docker的日志文件：

/busybox#stat2393.log
File:2393.log->/data/kubernetes/docker/containers/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca/68a5b32c9fdb1ad011b32e6252f9cdb759f69d7850e6b7b8591cb4c2bf00bcca-json.log
Size:173Blocks:8IOBlock:4096symboliclink
Device:fc02h/64514dInode:529958Links:1
Access:(0777/lrwxrwxrwx)Uid:(0/root)Gid:(0/root)
Access:2023-01-311303.751514283+0800
Modify:2023-01-311303.039526838+0800
Change:2023-01-311303.039526838+0800
Birth:-

/busybox#stat2394.log
File:2394.log->/data/kubernetes/docker/containers/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c/2ed9ebf0585215602874b076783e12191dbb010116038b8eb4646273ebfe195c-json.log
Size:173Blocks:8IOBlock:4096symboliclink
Device:fc02h/64514dInode:529955Links:1
Access:(0777/lrwxrwxrwx)Uid:(0/root)Gid:(0/root)
Access:2023-01-311403.991106950+0800
Modify:2023-01-311403.183119308+0800
Change:2023-01-311403.183119308+0800
Birth:-

看到分別指向了這兩個(gè)容器的日志文件，一個(gè)是當(dāng)前pod里在跑的容器，一個(gè)是pod上次跑的容器，現(xiàn)在已經(jīng)退出了

dockerps-a
CONTAINERIDIMAGECOMMANDCREATEDSTATUSPORTSNAMES
2ed9ebf05852ff4a8eb070e1"sleep3600"24minutesagoUp24minutesk8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2394
68a5b32c9fdbff4a8eb070e1"sleep3600"AboutanhouragoExited(0)24minutesagok8s_busybox_busybox_default_f72ab71a-5b3b-4ecf-940d-28a5c3b30683_2393

使用logs的時(shí)候讀的是當(dāng)前容器那個(gè)文件，使用–previous的時(shí)候，讀的是上次退出的容器的日志文件，由于kubelet為pod保留了上次退出的容器。

我們手動(dòng)編輯這兩個(gè)文件的內(nèi)容，看kubelet是否讀的是這兩個(gè)文件

/busybox#cat2393.log
{"log":"lastcrashlogs
","stream":"stderr","time":"2022-11-05T0827.31523845Z"}

/busybox#cat2394.log
{"log":"nowpodlog
","stream":"stderr","time":"2022-11-05T0827.31523845Z"}

ubuntu@10-234-32-51:~$klogsbusybox--previous
lastcrashlogs
ubuntu@10-234-32-51:~$klogsbusybox
nowpodlog

由于是鏈接文件，那么可能實(shí)際是從別的地方讀的，或者說直接讀容器目錄下的，由于鏈接文件我們改了后容器目錄下的日志文件也跟著改了，我們直接創(chuàng)建兩個(gè)文件來做驗(yàn)證：

ubuntu@10-234-32-51:~$kgetpod
NAMEREADYSTATUSRESTARTSAGE
busybox1/1Running239599d
nginx-deployment-6wlhd1/1Running079d
redis1/1Running049d

/busybox#ls
2394.log2395.log

/busybox#rm2394.log2395.log

我們刪除，然后自己創(chuàng)建，這時(shí)是regular file，而不是鏈接文件了：
/busybox#ls
2394.log2395.log

/busybox#stat2394.log
File:2394.log
Size:100Blocks:8IOBlock:4096regularfile
Device:fc02h/64514dInode:529965Links:1
Access:(0640/-rw-r-----)Uid:(0/root)Gid:(0/root)
Access:2023-01-311511.307170422+0800
Modify:2023-01-311507.711225229+0800
Change:2023-01-311507.711225229+0800
Birth:-

/busybox#stat2395.log
File:2395.log
Size:86Blocks:8IOBlock:4096regularfile
Device:fc02h/64514dInode:529967Links:1
Access:(0640/-rw-r-----)Uid:(0/root)Gid:(0/root)
Access:2023-01-311517.539989934+0800
Modify:2023-01-311514.348038586+0800
Change:2023-01-311514.352038525+0800
Birth:-

/busybox#cat2394.log
{"log":"previouslogscreatebymyself
","stream":"stderr","time":"2022-11-05T0827.31523845Z"}
/busybox#cat2395.log
{"log":"createbymyself
","stream":"stderr","time":"2022-11-05T0827.31523845Z"}

ubuntu@10-234-32-51:~$klogsbusybox
createbymyself
ubuntu@10-234-32-51:~$klogsbusybox--previous
previouslogscreatebymyself

得出結(jié)論：kubelet讀的是/var/log/pods/下的日志文件，–previous讀的也是/var/log/pods/下的日志文件，且專門有個(gè)鏈接文件來指向上一個(gè)退出容器的日志文件，以此來獲取容器崩潰前的日志。

審核編輯：劉清