Project

General

Profile

Actions

Bug #51362

open

qemu_bonnie I/O hang on nautilus

Added by Ilya Dryomov almost 3 years ago. Updated almost 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Failures:

All on centos 7.8:

14.2.21:

http://qa-proxy.ceph.com/teuthology/trociny-2021-06-24_16:42:49-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6190582/teuthology.log

14.2.22 QE branch:

http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-21_15:45:39-rbd-nautilus-distro-basic-smithi/6182709/teuthology.log
http://qa-proxy.ceph.com/teuthology/yuriw-2021-06-22_15:18:36-rbd-nautilus-distro-basic-smithi/6184291/teuthology.log

Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...
2021-06-22T15:44:45.458 INFO:tasks.qemu.client.0.smithi104.stdout:mount.nfs: timeout set for Tue Jun 22 15:46:44 2021
2021-06-22T15:44:45.458 INFO:tasks.qemu.client.0.smithi104.stdout:mount.nfs: trying text-based options 'proto=tcp,vers=4,addr=10.0.2.2,clientaddr=10.0.2.15'
2021-06-22T15:44:45.459 INFO:tasks.qemu.client.0.smithi104.stdout:10.0.2.2:/export/client.0 on /mnt/log type nfs (rw,proto=tcp)
2021-06-22T15:44:45.470 INFO:tasks.qemu.client.0.smithi104.stdout:mount: block device /dev/sr0 is write-protected, mounting read-only
2021-06-22T15:44:47.047 INFO:tasks.qemu.client.0.smithi104.stdout:meta-data=/dev/vdb               isize=256    agcount=4, agsize=655360 blks
2021-06-22T15:44:47.047 INFO:tasks.qemu.client.0.smithi104.stdout:         =                       sectsz=512   attr=2, projid32bit=0
2021-06-22T15:44:47.048 INFO:tasks.qemu.client.0.smithi104.stdout:data     =                       bsize=4096   blocks=2621440, imaxpct=25
2021-06-22T15:44:47.048 INFO:tasks.qemu.client.0.smithi104.stdout:         =                       sunit=0      swidth=0 blks
2021-06-22T15:44:47.048 INFO:tasks.qemu.client.0.smithi104.stdout:naming   =version 2              bsize=4096   ascii-ci=0
2021-06-22T15:44:47.048 INFO:tasks.qemu.client.0.smithi104.stdout:log      =internal log           bsize=4096   blocks=2560, version=2
2021-06-22T15:44:47.049 INFO:tasks.qemu.client.0.smithi104.stdout:         =                       sectsz=512   sunit=0 blks, lazy-count=1
2021-06-22T15:44:47.049 INFO:tasks.qemu.client.0.smithi104.stdout:realtime =none                   extsz=4096   blocks=0, rtextents=0
2021-06-22T15:51:54.425 INFO:tasks.qemu.client.0.smithi104.stdout:[  480.188168] INFO: task kworker/0:2:60 blocked for more than 120 seconds.
2021-06-22T15:51:54.426 INFO:tasks.qemu.client.0.smithi104.stdout:[  480.190562] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:51:54.426 INFO:tasks.qemu.client.0.smithi104.stdout:[  480.193629] INFO: task flush-253:16:1981 blocked for more than 120 seconds.
2021-06-22T15:51:54.427 INFO:tasks.qemu.client.0.smithi104.stdout:[  480.196054] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:53:54.477 INFO:tasks.qemu.client.0.smithi104.stdout:[  600.196160] INFO: task kworker/0:2:60 blocked for more than 120 seconds.
2021-06-22T15:53:54.478 INFO:tasks.qemu.client.0.smithi104.stdout:[  600.198639] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:53:54.478 INFO:tasks.qemu.client.0.smithi104.stdout:[  600.201804] INFO: task flush-253:16:1981 blocked for more than 120 seconds.
2021-06-22T15:53:54.478 INFO:tasks.qemu.client.0.smithi104.stdout:[  600.204420] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:55:54.485 INFO:tasks.qemu.client.0.smithi104.stdout:[  720.204175] INFO: task kworker/0:2:60 blocked for more than 120 seconds.
2021-06-22T15:55:54.486 INFO:tasks.qemu.client.0.smithi104.stdout:[  720.206693] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:55:54.486 INFO:tasks.qemu.client.0.smithi104.stdout:[  720.209843] INFO: task flush-253:16:1981 blocked for more than 120 seconds.
2021-06-22T15:55:54.486 INFO:tasks.qemu.client.0.smithi104.stdout:[  720.212466] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:57:54.508 INFO:tasks.qemu.client.0.smithi104.stdout:[  840.212197] INFO: task kworker/0:2:60 blocked for more than 120 seconds.
2021-06-22T15:57:54.509 INFO:tasks.qemu.client.0.smithi104.stdout:[  840.214679] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:57:54.510 INFO:tasks.qemu.client.0.smithi104.stdout:[  840.217851] INFO: task flush-253:16:1981 blocked for more than 120 seconds.
2021-06-22T15:57:54.510 INFO:tasks.qemu.client.0.smithi104.stdout:[  840.220487] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:59:54.491 INFO:tasks.qemu.client.0.smithi104.stdout:[  960.220158] INFO: task kworker/0:2:60 blocked for more than 120 seconds.
2021-06-22T15:59:54.492 INFO:tasks.qemu.client.0.smithi104.stdout:[  960.222732] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
2021-06-22T15:59:54.492 INFO:tasks.qemu.client.0.smithi104.stdout:[  960.225797] INFO: task flush-253:16:1981 blocked for more than 120 seconds.
2021-06-22T15:59:54.493 INFO:tasks.qemu.client.0.smithi104.stdout:[  960.228237] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

It doesn't seem to reproduce on ubuntu as easily but it's probably just a coincidence.

Actions #1

Updated by Ilya Dryomov almost 3 years ago

A successful rhel 7.9 job for reference:

Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.

http://qa-proxy.ceph.com/teuthology/trociny-2021-06-24_16:43:21-rbd-wip-mgolub-testing-nautilus-distro-basic-smithi/6190585/teuthology.log

Actions #3

Updated by Ilya Dryomov almost 3 years ago

Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...

http://qa-proxy.ceph.com/teuthology/dis-2021-06-28_15:14:21-rbd-nautilus-distro-basic-smithi/6239435/teuthology.log

Actions #4

Updated by Ilya Dryomov almost 3 years ago

Confirmed that the hang is specific to nautilus and appears to have been fixed in octopus. Apart from the already mentioned releases, reproduced on 14.2.18 and 14.2.12.

Actions

Also available in: Atom PDF