我正在运行几个docker容器,重启=总是在Ubuntu 18.04.1 LTS上.物理服务器每天凌晨2点重启,通过cronjob执行重启.
到目前为止,我在运行该特定设置的过去5或6个月内没有遇到任何问题.
但是今天,每天重启后容器没有启动. docker ps的输出为空,所有容器都处于“退出”状态.
为什么会突然发生这种情况？我的设置从一开始就配置错误,或者最近的docker-ce包升级是否起作用？
以下是重启之前和之后的日志以及docker.service单元和版本信息：
root@skprov2:~# journalctl -b -1 -x -u docker
Nov 15 02:00:02 skprov2 systemd[1]: Stopping Docker Application Container Engine...
-- Subject: Unit docker.service has begun shutting down
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit docker.service has begun shutting down.
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.189764841+01:00" level=info msg="Processing signal 'terminated'"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.595098434+01:00" level=info msg="shim reaped" id=c929d444a6eb59a69a0da738ca782a9feb92ac1f80e5c4576bf85376c3d4c17a
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.601217756+01:00" level=info msg="shim reaped" id=98a8c1b99cf986e6a889474f0fc28fe3635e466b21f8a37ef3c10a1050495c78
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.604880385+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.670918937+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.732991633+01:00" level=info msg="shim reaped" id=9b3badc752786df08d00138c0222042a6bd80bb2c971f5a96b71e57105cea95c
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.748732351+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.843982385+01:00" level=info msg="shim reaped" id=ae7531405113db8b4754491a12c2ababf09fa0c8f501bfe6f1b33e3ff18b6462
Nov 15 02:00:02 skprov2 dockerd[1504]: time="2018-11-15T02:00:02.869023019+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:03 skprov2 dockerd[1504]: time="2018-11-15T02:00:03.863568729+01:00" level=info msg="shim reaped" id=b335536f5f07b1db3f32ba4452fc4aadacc02c6184cef7fc9df619ab81bbf002
Nov 15 02:00:04 skprov2 dockerd[1504]: time="2018-11-15T02:00:04.279347144+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.233635995+01:00" level=info msg="Container 77e00eabebea97357f05f564597d167acad8f2596b25d295b4366baf08ef3127 failed to exit within 10 seconds of signal 15 - using the force"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.253563540+01:00" level=info msg="Container 7f7d2a92bcdbb240a9400942c9301f5cd77bf9d3fbde1d38f41a2bd1226f9b09 failed to exit within 10 seconds of signal 15 - using the force"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.253563179+01:00" level=info msg="Container f6b49cc85eb7f9226ac192498b1e319d68e0de2faff6b4e3e67adabba43a093a failed to exit within 10 seconds of signal 15 - using the force"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.654403249+01:00" level=info msg="shim reaped" id=7f7d2a92bcdbb240a9400942c9301f5cd77bf9d3fbde1d38f41a2bd1226f9b09
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.679675304+01:00" level=info msg="shim reaped" id=f6b49cc85eb7f9226ac192498b1e319d68e0de2faff6b4e3e67adabba43a093a
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.680699340+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:12 skprov2 dockerd[1504]: time="2018-11-15T02:00:12.689801078+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:13 skprov2 dockerd[1504]: time="2018-11-15T02:00:13.088891655+01:00" level=info msg="shim reaped" id=77e00eabebea97357f05f564597d167acad8f2596b25d295b4366baf08ef3127
Nov 15 02:00:13 skprov2 dockerd[1504]: time="2018-11-15T02:00:13.111193244+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 15 02:00:15 skprov2 dockerd[1504]: time="2018-11-15T02:00:15.233286510+01:00" level=info msg="stopping event stream following graceful shutdown" error="
我偶然发现了同样的问题.
在我的情况下,问题是由于Docker没有正确清理.
如Docker的日志中所示：
time="2018-12-31T17:38:54.330555181+02:00" level=error msg="2089c8095e62011b0dc05e66c51ae59d648d909ca7a8e806af0fdf39b2e3006c cleanup: failed to delete container from containerd: transport is closing: unknown"
由于restart = always,这些id在启动时使用.
所以Docker说
time="2018-12-31T17:40:04.648261275+02:00" level=error msg="Failed to start container 2089c8095e62011b0dc05e66c51ae59d648d909ca7a8e806af0fdf39b2e3006c: id already in use"
看起来像Docker守护进程关闭的速度比清理容器的速度快(可能是因为容器忽略了信号或其他东西)
所以似乎我的解决方案是在Docker守护程序配置文件中更改守护程序的shutdown-timeout.默认值为10秒或者其他东西,我将其更改为60秒,我不再遇到这些问题.
我仍然认为这是一个合理的场景,应该开箱即用.
