Actions
Bug #54261
openqa/workunits/rbd/rbd_mirror_stress.sh: compare_image_snaps sporadically fails
Status:
New
Priority:
High
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2022-02-11T07:46:46.278 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ compare_image_snaps mirror test snap10-clean ... 2022-02-11T07:47:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ sdiff -s /tmp/tmp.uzFYBFlCBt/cluster2-mirror-test.export.xxd /tmp/tmp.uzFYBFlCBt/cluster1-mirror-test.export.xxd 2022-02-11T07:47:53.657 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stderr:+ head -n 64 2022-02-11T07:48:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000000: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000010: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.440 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000020: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000020: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000030: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000030: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000040: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000040: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000050: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000050: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000060: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000060: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000070: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000070: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000080: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000080: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000090: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000090: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.442 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000000a0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000000a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000000b0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000000b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000000c0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000000c0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000000d0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000000d0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.443 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000000e0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000000e0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000000f0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000000f0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000100: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000100: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000110: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000110: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000120: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000120: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.444 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000130: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000130: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000140: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000140: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000150: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000150: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000160: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000160: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.445 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000170: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000170: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000180: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000180: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000190: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000190: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000001a0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000001a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000001b0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000001b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000001c0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000001c0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000001d0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000001d0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000001e0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000001e0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000001f0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000001f0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.447 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000200: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000200: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000210: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000210: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000220: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000220: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000230: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000230: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000240: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000240: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000250: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000250: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000260: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000260: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.449 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000270: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000270: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000280: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000280: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000290: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000290: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000002a0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000002a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.450 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000002b0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000002b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000002c0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000002c0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000002d0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000002d0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000002e0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000002e0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.451 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000002f0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000002f0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000300: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000300: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000310: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000310: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000320: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000320: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.452 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000330: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000330: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000340: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000340: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000350: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000350: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000360: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000360: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.453 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000370: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000370: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000380: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000380: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:00000390: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000390: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000003a0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000003a0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.454 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000003b0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000003b0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000003c0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000003c0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000003d0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000003d0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000003e0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000003e0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f ////////// 2022-02-11T07:48:10.455 INFO:tasks.workunit.cluster1.client.mirror.smithi132.stdout:000003f0: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 000003f0: 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f 2f2f //////////
Updated by Ilya Dryomov about 2 years ago
rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} msgr-failures/few objectstore/bluestore-comp-snappy policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{rhel_8} workloads/rbd-mirror-snapshot-stress-workunit-minimum}:
2022-02-23T02:37:05.492 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ cmp /tmp/tmp.V7aKsVP2Kg/cluster2-mirror-test.export /tmp/tmp.V7aKsVP2Kg/cluster1-mirror-test.export 2022-02-23T02:37:05.493 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stdout:/tmp/tmp.V7aKsVP2Kg/cluster2-mirror-test.export /tmp/tmp.V7aKsVP2Kg/cluster1-mirror-test.export differ: byte 1, line 1 2022-02-23T02:37:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ show_diff /tmp/tmp.V7aKsVP2Kg/cluster2-mirror-test.export /tmp/tmp.V7aKsVP2Kg/cluster1-mirror-test.export 2022-02-23T02:37:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ local file1=/tmp/tmp.V7aKsVP2Kg/cluster2-mirror-test.export 2022-02-23T02:37:05.494 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ local file2=/tmp/tmp.V7aKsVP2Kg/cluster1-mirror-test.export 2022-02-23T02:37:05.495 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ xxd /tmp/tmp.V7aKsVP2Kg/cluster2-mirror-test.export 2022-02-23T02:37:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ xxd /tmp/tmp.V7aKsVP2Kg/cluster1-mirror-test.export 2022-02-23T02:38:10.750 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ sdiff -s /tmp/tmp.V7aKsVP2Kg/cluster2-mirror-test.export.xxd /tmp/tmp.V7aKsVP2Kg/cluster1-mirror-test.export.xxd 2022-02-23T02:38:10.751 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stderr:+ head -n 64 2022-02-23T02:38:25.176 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8689810468964587) > weight (0.25) 2022-02-23T02:38:25.176 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4789202242138113) > weight (0.25) 2022-02-23T02:38:25.176 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6248108446384026) > weight (0.25) 2022-02-23T02:38:25.177 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8791741782927245) > weight (0.25) 2022-02-23T02:38:25.177 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2022-02-23T02:38:30.161 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2022-02-23T02:38:30.162 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2022-02-23T02:38:30.163 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2022-02-23T02:38:30.164 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi179.stderr:2022-02-23T02:38:30.152+0000 7f167c4f7700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41052) UID: 1000 2022-02-23T02:38:30.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi179.stderr:daemon-helper: command failed with exit status 1 2022-02-23T02:38:31.164 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2022-02-23T02:38:31.164 INFO:teuthology.orchestra.run:waiting for 600 2022-02-23T02:38:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-02-23T02:38:31.165 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2022-02-23T02:38:31.166 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2022-02-23T02:38:31.166 DEBUG:teuthology.orchestra.run.smithi179:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2022-02-23T02:38:31.169 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2022-02-23T02:38:31.169 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2022-02-23T02:39:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stdout:00000000: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000000: bebe bebe bebe bebe bebe bebe bebe bebe .......... 2022-02-23T02:39:44.675 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stdout:00000010: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000010: bebe bebe bebe bebe bebe bebe bebe bebe .......... 2022-02-23T02:39:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stdout:00000020: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000020: bebe bebe bebe bebe bebe bebe bebe bebe .......... 2022-02-23T02:39:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stdout:00000030: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000030: bebe bebe bebe bebe bebe bebe bebe bebe .......... 2022-02-23T02:39:44.676 INFO:tasks.workunit.cluster1.client.mirror.smithi179.stdout:00000040: 0000 0000 0000 0000 0000 0000 0000 0000 .......... | 00000040: bebe bebe bebe bebe bebe bebe bebe bebe ..........
rerun succeeded:
Updated by Mykola Golub about 2 years ago
Usually I was running with `-filter workloads/rbd-mirror-snapshot-stress-`, which was about 50 jobs, and observed only a couple of failures. So to make sure it is not reproducible one need to rerun multiple times.
On the other hand, taking that we seem not to have changes to librbd/mirror that could affect this for long time, I can imaging it could be a core regression, which is already fixed?
Updated by Ilya Dryomov about 2 years ago
- Priority changed from Normal to High
Mykola Golub wrote:
On the other hand, taking that we seem not to have changes to librbd/mirror that could affect this for long time, I can imaging it could be a core regression, which is already fixed?
I'm not following -- you hit it on master as of a few days ago and I have just hit it on wip-dis-testing (master + latest rbd fixes). I'm not sure whether it's something new or not, but it's definitely still there.
Updated by Mykola Golub about 2 years ago
Ilya Dryomov wrote:
I'm not following -- you hit it on master as of a few days ago and I have just hit it on wip-dis-testing (master + latest rbd fixes). I'm not sure whether it's something new or not, but it's definitely still there.
Ah, sorry. I just read "rerun succeeded" and thought that you reran my tests and they succeeded (I was thinking the output above was still from my run). Agree.
Actions