Bug #45947: ceph_test_rados_watch_notify hang seen in nautilus - RADOS - Ceph

Actions

Copy link

Bug #45947

open

ceph_test_rados_watch_notify hang seen in nautilus

Added by Brad Hubbard almost 4 years ago. Updated over 3 years ago.

Status:

New

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v14.2.9

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2020-06-08_16:06:08-rados-wip-yuri2-testing-2020-06-08-1458-nautilus-distro-basic-smithi/5129565

failure_reason: 'Command failed (workunit test rados/test.sh) on smithi137 with status 124

2020-06-09T04:50:14.854 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 16278 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""

2020-06-08T22:59:09.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ for t in '"${!pids[@]}"'
2020-06-08T22:59:09.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ pid=16278
2020-06-08T22:59:09.582 INFO:tasks.workunit.client.0.smithi137.stderr:+ wait 16278

2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ echo 'test watch_notify on pid 16278'
2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ pids[$f]=16278
2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ ret=0
2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ '[' 1 -eq 1 ']'
2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ for t in '"${!pids[@]}"'
2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ pid=15803
2020-06-08T22:50:15.186 INFO:tasks.workunit.client.0.smithi137.stderr:+ wait 15803
2020-06-08T22:50:15.187 INFO:tasks.workunit.client.0.smithi137.stderr:+ sed 's/^/    delete_pools_parallel: /'
2020-06-08T22:50:15.191 INFO:tasks.workunit.client.0.smithi137.stderr:+ bash -o pipefail -exc 'ceph_test_rados_watch_notify  2>&1 | tee ceph_test_rados_watch_notify.log | sed "s/^/             watch_notify: /"'
2020-06-08T22:50:15.198 INFO:tasks.workunit.client.0.smithi137.stderr:+ sed 's/^/             watch_notify: /'
2020-06-08T22:50:15.199 INFO:tasks.workunit.client.0.smithi137.stderr:+ ceph_test_rados_watch_notify
2020-06-08T22:50:15.211 INFO:tasks.workunit.client.0.smithi137.stderr:+ tee ceph_test_rados_watch_notify.log

There's no ouput from ceph_test_rados_watch_notify.

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Brad Hubbard almost 4 years ago

Related to Bug #45946: ceph_test_rados_delete_pools_parallel hang seen in octopus added

Actions

Copy link

Updated by Brad Hubbard almost 4 years ago

This is very similar to what is seen in #45946 so they may be related.

Actions

Copy link

Updated by Brad Hubbard almost 4 years ago

Subject changed from ceph_test_rados_watch_notify hangs to ceph_test_rados_watch_notify hang seen in nautilus
Affected Versions v14.2.9 added
Affected Versions deleted (~~v16.0.0~~)

Actions

Copy link

Updated by Brad Hubbard almost 4 years ago

Note https://tracker.ceph.com/issues/43861 removed this test from master because it was hanging.

Actions

Copy link

Updated by Deepika Upadhyay over 3 years ago

2020-11-11T01:12:55.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ '[' 1 -eq 1 ']'
2020-11-11T01:12:55.964 INFO:tasks.workunit.client.0.smithi123.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio  2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/                  api_aio: /"'
2020-11-11T01:12:55.965 INFO:tasks.workunit.client.0.smithi123.stderr:++ printf %25s api_aio_pp
2020-11-11T01:12:55.965 INFO:tasks.workunit.client.0.smithi123.stderr:+ r='               api_aio_pp'
2020-11-11T01:12:55.965 INFO:tasks.workunit.client.0.smithi123.stderr:++ echo api_aio_pp
2020-11-11T01:12:55.965 INFO:tasks.workunit.client.0.smithi123.stderr:++ awk '{print $1}'
2020-11-11T01:12:55.965 INFO:tasks.workunit.client.0.smithi123.stdout:test api_aio on pid 38049
2020-11-11T01:12:55.966 INFO:tasks.workunit.client.0.smithi123.stderr:+ ff=api_aio_pp
2020-11-11T01:12:55.967 INFO:tasks.workunit.client.0.smithi123.stderr:+ ceph_test_rados_api_aio
2020-11-11T01:12:55.967 INFO:tasks.workunit.client.0.smithi123.stderr:+ pid=38057
2020-11-11T01:12:55.967 INFO:tasks.workunit.client.0.smithi123.stderr:+ tee ceph_test_rados_api_aio.log
2020-11-11T01:12:55.967 INFO:tasks.workunit.client.0.smithi123.stderr:+ echo 'test api_aio_pp on pid 38057'
2020-11-11T01:12:55.967 INFO:tasks.workunit.client.0.smithi123.stderr:+ pids[$f]=38057
2020-11-11T01:12:55.967 INFO:tasks.workunit.client.0.smithi123.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel

2020-11-11T07:12:56.081 INFO:tasks.workunit.client.0.smithi123.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 38057 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""

http://qa-proxy.ceph.com/teuthology/yuriw-2020-11-10_16:01:13-rados-wip-yuri7-testing-2020-11-09-0731-nautilus-distro-basic-smithi/5609419/teuthology.log

looks related, Brad can you verify. Otherwise I will move it to a seperate tracker

Actions

Copy link