Bug #50906: rbd-nbd: waiting for process exit timed out - rbd - Ceph

Actions

Copy link

Bug #50906

open

rbd-nbd: waiting for process exit timed out

Added by Ilya Dryomov almost 3 years ago. Updated over 2 years ago.

Status:

New

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/dis-2021-05-20_13:09:57-rbd-wip-dis-testing-distro-basic-smithi/6124830/teuthology.log

2021-05-20T13:29:55.657 INFO:tasks.workunit.client.0.smithi168.stderr:+ unmap_device /dev/nbd0 14492
2021-05-20T13:29:55.657 INFO:tasks.workunit.client.0.smithi168.stderr:+ local dev=/dev/nbd0
2021-05-20T13:29:55.657 INFO:tasks.workunit.client.0.smithi168.stderr:+ local pid=14492
2021-05-20T13:29:55.657 INFO:tasks.workunit.client.0.smithi168.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0
2021-05-20T13:29:55.658 INFO:tasks.workunit.client.0.smithi168.stderr:+ local cmd
2021-05-20T13:29:55.658 INFO:tasks.workunit.client.0.smithi168.stderr:++ id -u
2021-05-20T13:29:55.658 INFO:tasks.workunit.client.0.smithi168.stderr:+ '[' 1000 -eq 0 ']'
2021-05-20T13:29:55.658 INFO:tasks.workunit.client.0.smithi168.stderr:++ which rbd
2021-05-20T13:29:55.659 INFO:tasks.workunit.client.0.smithi168.stderr:+ cmd=/usr/bin/rbd
2021-05-20T13:29:55.659 INFO:tasks.workunit.client.0.smithi168.stderr:+ shift
2021-05-20T13:29:55.659 INFO:tasks.workunit.client.0.smithi168.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0
2021-05-20T13:30:25.564 INFO:tasks.workunit.client.0.smithi168.stderr:rbd-nbd: waiting for process exit timed out
2021-05-20T13:30:25.564 INFO:tasks.workunit.client.0.smithi168.stderr:+ rbd device --device-type nbd list
2021-05-20T13:30:25.564 INFO:tasks.workunit.client.0.smithi168.stderr:+ expect_false grep '^14492\b'
2021-05-20T13:30:25.564 INFO:tasks.workunit.client.0.smithi168.stderr:+ grep '^14492\b'
2021-05-20T13:30:25.567 INFO:tasks.workunit.client.0.smithi168.stderr:+ return 0
2021-05-20T13:30:25.568 INFO:tasks.workunit.client.0.smithi168.stderr:+ ps -C rbd-nbd
2021-05-20T13:30:25.568 INFO:tasks.workunit.client.0.smithi168.stderr:+ expect_false grep '^ *14492\b'
2021-05-20T13:30:25.568 INFO:tasks.workunit.client.0.smithi168.stderr:+ grep '^ *14492\b'
2021-05-20T13:30:25.575 INFO:tasks.workunit.client.0.smithi168.stdout:  14492 ?        00:00:00 rbd-nbd
2021-05-20T13:30:25.576 INFO:tasks.workunit.client.0.smithi168.stderr:+ return 1
2021-05-20T13:30:25.577 INFO:tasks.workunit.client.0.smithi168.stderr:+ return 1
2021-05-20T13:30:25.577 INFO:tasks.workunit.client.0.smithi168.stderr:+ cleanup

Command failed (workunit test rbd/rbd-nbd.sh) on smithi168 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=56464c38ae02b4ed0324b385301a29977b6af7f9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh''

2021-05-20T13:29:29.593650+00:00 smithi168 kernel: [  493.396504] block nbd0: Device being setup by another task
2021-05-20T13:29:29.676424+00:00 smithi168 kernel: [  493.485313] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:29.676439+00:00 smithi168 kernel: [  493.485324] block nbd0: Disconnected due to user request.
2021-05-20T13:29:29.676440+00:00 smithi168 kernel: [  493.485325] block nbd0: shutting down sockets
2021-05-20T13:29:35.387331+00:00 smithi168 kernel: [  499.195904] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:35.387347+00:00 smithi168 kernel: [  499.195918] block nbd0: Disconnected due to user request.
2021-05-20T13:29:35.387347+00:00 smithi168 kernel: [  499.195919] block nbd0: shutting down sockets
2021-05-20T13:29:36.426635+00:00 smithi168 kernel: [  500.235582] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:36.426668+00:00 smithi168 kernel: [  500.235591] block nbd0: Disconnected due to user request.
2021-05-20T13:29:36.426672+00:00 smithi168 kernel: [  500.235592] block nbd0: shutting down sockets
2021-05-20T13:29:37.879647+00:00 smithi168 kernel: [  501.688573] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:37.879661+00:00 smithi168 kernel: [  501.688586] block nbd0: Disconnected due to user request.
2021-05-20T13:29:37.879661+00:00 smithi168 kernel: [  501.688587] block nbd0: shutting down sockets
2021-05-20T13:29:38.838477+00:00 smithi168 kernel: [  502.647358] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:38.838497+00:00 smithi168 kernel: [  502.647371] block nbd0: Disconnected due to user request.
2021-05-20T13:29:38.838498+00:00 smithi168 kernel: [  502.647372] block nbd0: shutting down sockets
2021-05-20T13:29:39.936428+00:00 smithi168 kernel: [  503.745344] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:39.936440+00:00 smithi168 kernel: [  503.745356] block nbd0: Disconnected due to user request.
2021-05-20T13:29:39.936441+00:00 smithi168 kernel: [  503.745357] block nbd0: shutting down sockets
2021-05-20T13:29:41.940051+00:00 smithi168 kernel: [  505.748945] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:41.940074+00:00 smithi168 kernel: [  505.748955] block nbd0: Disconnected due to user request.
2021-05-20T13:29:41.940077+00:00 smithi168 kernel: [  505.748956] block nbd0: shutting down sockets
2021-05-20T13:29:42.881785+00:00 smithi168 kernel: [  506.684507] block nbd0: Device being setup by another task
2021-05-20T13:29:43.075341+00:00 smithi168 kernel: [  506.881127] block nbd1: NBD_DISCONNECT
2021-05-20T13:29:43.075353+00:00 smithi168 kernel: [  506.881140] block nbd1: Disconnected due to user request.
2021-05-20T13:29:43.075354+00:00 smithi168 kernel: [  506.881141] block nbd1: shutting down sockets
2021-05-20T13:29:43.726618+00:00 smithi168 kernel: [  507.535513] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:43.726637+00:00 smithi168 kernel: [  507.535530] block nbd0: Disconnected due to user request.
2021-05-20T13:29:43.726638+00:00 smithi168 kernel: [  507.535531] block nbd0: shutting down sockets
2021-05-20T13:29:44.615381+00:00 smithi168 kernel: [  508.422751] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:44.615405+00:00 smithi168 kernel: [  508.422764] block nbd0: Disconnected due to user request.
2021-05-20T13:29:44.615406+00:00 smithi168 kernel: [  508.422764] block nbd0: shutting down sockets
2021-05-20T13:29:45.619370+00:00 smithi168 kernel: [  509.427091] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:45.619400+00:00 smithi168 kernel: [  509.427104] block nbd0: Disconnected due to user request.
2021-05-20T13:29:45.619404+00:00 smithi168 kernel: [  509.427105] block nbd0: shutting down sockets
2021-05-20T13:29:46.491348+00:00 smithi168 kernel: [  510.298320] block nbd0: shutting down sockets
2021-05-20T13:29:46.491362+00:00 smithi168 kernel: [  510.298650] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:46.491364+00:00 smithi168 kernel: [  510.298655] block nbd0: Send disconnect failed -32
2021-05-20T13:29:46.495330+00:00 smithi168 kernel: [  510.303529] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.511362+00:00 smithi168 kernel: [  510.313540] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.515335+00:00 smithi168 kernel: [  510.320532] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.528396+00:00 smithi168 kernel: [  510.330621] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.531350+00:00 smithi168 kernel: [  510.337578] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.545353+00:00 smithi168 kernel: [  510.347546] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.545368+00:00 smithi168 kernel: [  510.354468] ldm_validate_partition_table(): Disk read failed.
2021-05-20T13:29:46.547344+00:00 smithi168 kernel: [  510.354508] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.562261+00:00 smithi168 kernel: [  510.364492] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.563316+00:00 smithi168 kernel: [  510.371505] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.579282+00:00 smithi168 kernel: [  510.381471] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.579354+00:00 smithi168 kernel: [  510.388446] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.596235+00:00 smithi168 kernel: [  510.398415] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.599346+00:00 smithi168 kernel: [  510.405415] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.613166+00:00 smithi168 kernel: [  510.415397] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.615351+00:00 smithi168 kernel: [  510.422319] Dev nbd0: unable to read RDB block 0
2021-05-20T13:29:46.627843+00:00 smithi168 kernel: [  510.426999] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.627854+00:00 smithi168 kernel: [  510.436947] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.635341+00:00 smithi168 kernel: [  510.443946] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.651803+00:00 smithi168 kernel: [  510.453962] Buffer I/O error on dev nbd0, logical block 0, async page read
2021-05-20T13:29:46.651830+00:00 smithi168 kernel: [  510.460950] blk_update_request: I/O error, dev nbd0, sector 24 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:46.668838+00:00 smithi168 kernel: [  510.471011] Buffer I/O error on dev nbd0, logical block 3, async page read
2021-05-20T13:29:46.669147+00:00 smithi168 kernel: [  510.478179]  nbd0: unable to read partition table
2021-05-20T13:29:46.671358+00:00 smithi168 kernel: [  510.479010] ldm_validate_partition_table(): Disk read failed.
2021-05-20T13:29:46.671368+00:00 smithi168 kernel: [  510.479078] Dev nbd0: unable to read RDB block 0
2021-05-20T13:29:46.675330+00:00 smithi168 kernel: [  510.484161]  nbd0: unable to read partition table
2021-05-20T13:29:55.481387+00:00 smithi168 kernel: [  519.279702] block nbd0: NBD_DISCONNECT
2021-05-20T13:29:55.481426+00:00 smithi168 kernel: [  519.279714] block nbd0: Disconnected due to user request.
2021-05-20T13:29:55.481429+00:00 smithi168 kernel: [  519.279715] block nbd0: shutting down sockets
2021-05-20T13:29:55.481431+00:00 smithi168 kernel: [  519.279735] print_req_error: 14 callbacks suppressed
2021-05-20T13:29:55.481432+00:00 smithi168 kernel: [  519.279737] blk_update_request: I/O error, dev nbd0, sector 130944 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
2021-05-20T13:29:55.483369+00:00 smithi168 kernel: [  519.290571] blk_update_request: I/O error, dev nbd0, sector 130944 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
2021-05-20T13:29:55.499264+00:00 smithi168 kernel: [  519.301011] buffer_io_error: 14 callbacks suppressed
2021-05-20T13:29:55.499282+00:00 smithi168 kernel: [  519.301013] Buffer I/O error on dev nbd0, logical block 130944, async page read
2021-05-20T13:29:55.499284+00:00 smithi168 kernel: [  519.308335] blk_update_request: I/O error, dev nbd0, sector 130945 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
2021-05-20T13:29:55.509626+00:00 smithi168 kernel: [  519.318699] Buffer I/O error on dev nbd0, logical block 130945, async page read
2021-05-20T13:29:55.516946+00:00 smithi168 kernel: [  519.326019] Buffer I/O error on dev nbd0, logical block 130946, async page read
2021-05-20T13:29:55.524263+00:00 smithi168 kernel: [  519.333338] Buffer I/O error on dev nbd0, logical block 130947, async page read
2021-05-20T13:29:55.531551+00:00 smithi168 kernel: [  519.340662] Buffer I/O error on dev nbd0, logical block 130948, async page read
2021-05-20T13:29:55.538905+00:00 smithi168 kernel: [  519.347985] Buffer I/O error on dev nbd0, logical block 130949, async page read
2021-05-20T13:29:55.546225+00:00 smithi168 kernel: [  519.355305] Buffer I/O error on dev nbd0, logical block 130950, async page read
2021-05-20T13:29:55.553542+00:00 smithi168 kernel: [  519.362620] Buffer I/O error on dev nbd0, logical block 130951, async page read

Actions

Copy link

Updated by Deepika Upadhyay over 2 years ago

[  496.635362] block nbd0: Device being setup by another task
[  498.311337] block nbd0: Send disconnect failed -32
[  498.316200] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.326176] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.333105] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.343127] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.350064] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.360035] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.366971] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.376954] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.383880] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.393869] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.400783] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.410760] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.417666] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.427641] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.434543] Dev nbd0: unable to read RDB block 0
[  498.439487] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.449441] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.456371] blk_update_request: I/O error, dev nbd0, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.466350] Buffer I/O error on dev nbd0, logical block 0, async page read
[  498.473323] blk_update_request: I/O error, dev nbd0, sector 24 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  498.483393] Buffer I/O error on dev nbd0, logical block 3, async page read
[  498.491477] Dev nbd0: unable to read RDB block 0
[  498.519603] Dev nbd0: unable to read RDB block 0

/ceph/teuthology-archive/yuriw-2021-07-26_16:38:53-rbd-wip-yuri3-testing-2021-07-26-0800-pacific-distro-basic-smithi/6294949/console_logs/smithi116.log

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » rbd

Custom queries

Bug #50906

rbd-nbd: waiting for process exit timed out

Updated by Deepika Upadhyay over 2 years ago