QA Run #65159: wip-yuri4-testing-2024-03-26-1132-reef - Ceph QA - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
Pending backports
Product Backlog Scrub
QA: All
QA: Component: core
QA: Component: fs
QA: Main
QA: Reef
QA: Squid

Actions

Copy link

QA Run #65159

closed

wip-yuri4-testing-2024-03-26-1132-reef

Added by Yuri Weinstein about 1 month ago. Updated about 1 month ago.

Status:

QA Closed

Priority:

Normal

Assignee:

Yuri Weinstein

Tags:

Shaman Build:

wip-yuri4-testing-2024-03-26-1132-reef

QA Runs:

wip-yuri4-testing-2024-03-26-1132-reef

QA Release:

reef

Git Branch:

Description

--- done. these PRs were included:
https://github.com/ceph/ceph/pull/54258 - reef: os/bluestore: add bluestore fragmentation micros to prometheus
https://github.com/ceph/ceph/pull/55548 - reef: mon: fix health store size growing infinitely
https://github.com/ceph/ceph/pull/55697 - reef: osd: Report health error if OSD public address is not within subnet
https://github.com/ceph/ceph/pull/55777 - reef: os/bluestore: fix free space update after bdev-expand in NCB mode
https://github.com/ceph/ceph/pull/55867 - reef: mon/OSDMonitor: fix get_min_last_epoch_clean()
https://github.com/ceph/ceph/pull/56140 - reef: rgw/notification: Kafka persistent notifications not retried and removed even when the broker is down
https://github.com/ceph/ceph/pull/56197 - reef: os/kv_test: Fix estimate functions
https://github.com/ceph/ceph/pull/56347 - reef: rgw: Add missing empty checks to the split string in is_string_in_set().

History
Notes
Property changes

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

QA Runs set to wip-yuri4-testing-2024-03-26-1132-reef

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

rebasing for rgw rerun per @casey

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

Yuri Weinstein wrote:

rebasing for rgw rerun per @casey

@casey see the run on rebased sha1
https://pulpito.ceph.com/yuriw-2024-03-27_22:55:38-rgw-wip-yuri4-testing-2024-03-26-1132-reef-distro-default-smithi

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

@casey pls review
@laura rerunning rados (on the latest sha1)

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

@laura @casey pls review

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

Assignee changed from Yuri Weinstein to Laura Flores

Actions

Copy link

Updated by Casey Bodley about 1 month ago

rgw approved

Actions

Copy link

Updated by Laura Flores about 1 month ago

Hi Pere, can you please review the rados run?

Actions

Copy link

Updated by Yuri Weinstein about 1 month ago

Assignee changed from Laura Flores to Pere Díaz Bou

Actions

Copy link

#10

Updated by Pere Díaz Bou about 1 month ago

All of these issues seem unrelated and have already been documented as per the links attach to each item in the list:

- 7627636 and 7627655 rgw_multi.tests.test_multi_object_delete
https://tracker.ceph.com/issues/58917
```bash
2024-03-27T23:26:45.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-27T23:26:45.560+0000 7f39edf1e880 1 waiting to reach incremental sync..
2024-03-27T23:26:50.531 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-27T23:26:50.561+0000 7f39edf1e880 1 waiting to reach incremental sync..
2024-03-27T23:26:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-27T23:26:55.561+0000 7f39edf1e880 -1 bucket checkpoint timed out waiting to reach increm
ental sync
2024-03-27T23:26:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-27T23:26:55.561+0000 7f39edf1e880 0 bucket sync checkpoint failed: (110) Connection tim
ed out
2024-03-27T23:26:55.532 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-27T23:26:55.561+0000 7f39edf1e880 -1 bucket sync checkpoint failed: (110) Connection tim

...
2024-03-28T01:58:09.933 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_multi_object_delete
2024-03-28T01:58:09.934 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2024-03-28T01:58:09.934 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/nose/case.py", line 198, in runTest
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/../src/test/rgw/rgw_multi/tests.py", line 687, in test_multi_object_delete
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: zone_bucket_checkpoint(target_conn.zone, source_conn.zone, bucket.name)
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/../src/test/rgw/rgw_multi/tests.py", line 387, in zone_bucket_checkpoint
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: target_zone.cluster.admin(cmd, debug_rgw=1)
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/tasks/rgw_multisite.py", line 197, in admin
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests: assert r == 0
2024-03-28T01:58:09.935 INFO:tasks.rgw_multisite_tests:AssertionError:
...
2024-03-28T01:58:10.239 INFO:tasks.rgw_multisite_tests:FAIL: rgw_multi.tests.test_bucket_delete_notempty
2024-03-28T01:58:10.239 INFO:tasks.rgw_multisite_tests:----------------------------------------------------------------------
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last):
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/py
thon3.8/site-packages/nose/case.py", line 198, in runTest
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests: self.test(*self.arg)
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/../src/test/rgw
/rgw_multi/tests.py", line 900, in test_bucket_delete_notempty
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup)
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/../src/test/rgw
/rgw_multi/tests.py", line 208, in zonegroup_meta_checkpoint
2024-03-28T01:58:10.240 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status)
2024-03-28T01:58:10.241 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph-c_e58b9979847f029f7ab1514b5f5f99b9f03d97f6/qa/../src/test/rgw
/rgw_multi/tests.py", line 197, in zone_meta_checkpoint
2024-03-28T01:58:10.241 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name

```

- 7627670 RGWBucketReshard::execute assert failure

https://tracker.ceph.com/issues/56500

```bash
2024-03-27T23:48:05.346 INFO:tasks.workunit.client.0.smithi037.stderr:*** Caught signal (Aborted) *
2024-03-27T23:48:05.346 INFO:tasks.workunit.client.0.smithi037.stderr: in thread 7ff571d31880 thread_name:radosgw-admin
2024-03-27T23:48:05.364 INFO:tasks.workunit.client.0.smithi037.stderr: ceph version 18.2.2-1053-ge58b9979 (e58b9979847f029f7ab1514b5f5f99b9f03d97f6) reef (stable)
2024-03-27T23:48:05.364 INFO:tasks.workunit.client.0.smithi037.stderr: 1: /lib64/libc.so.6(+0x54db0) [0x7ff572054db0]
2024-03-27T23:48:05.364 INFO:tasks.workunit.client.0.smithi037.stderr: 2: /lib64/libc.so.6(+0xa154c) [0x7ff5720a154c]
2024-03-27T23:48:05.364 INFO:tasks.workunit.client.0.smithi037.stderr: 3: raise()
2024-03-27T23:48:05.365 INFO:tasks.workunit.client.0.smithi037.stderr: 4: abort()
2024-03-27T23:48:05.365 INFO:tasks.workunit.client.0.smithi037.stderr: 5: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x188) [0x7ff574b6b0a7]
2024-03-27T23:48:05.365 INFO:tasks.workunit.client.0.smithi037.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7ff574b6b20b]
2024-03-27T23:48:05.365 INFO:tasks.workunit.client.0.smithi037.stderr: 7: (std::__detail::__variant::__gen_vtable_impl<std::__detail::__variant::_Multi_array<std::__detail::__variant::__deduce_visit_result<int> ()(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)>, std::integer_sequence<unsigned long, 1ul> >::__visit_invoke(FaultInjector<std::basic_string_view<char, std::char_traits<char> > >::check(std::basic_string_view<char, std::char_traits<char> > const&) const::visitor&&, std::variant<std::monostate, InjectAbort, InjectError, InjectDelay> const&)+0x41) [0x55a284573761]
2024-03-27T23:48:05.365 INFO:tasks.workunit.client.0.smithi037.stderr: 8: radosgw-admin(+0x740302) [0x55a284585302]
2024-03-27T23:48:05.365 INFO:tasks.workunit.client.0.smithi037.stderr: 9: (RGWBucketReshard::execute(int, FaultInjector<std::basic_string_view<char, std::char_traits<char> > >&, int, DoutPrefixProvider const, bool, std::ostream*, ceph::Formatter*, RGWReshard*)+0x249) [0x55a284589de9]

```

- 7627667 Package 'python-dev' has no installation candidate

https://tracker.ceph.com/issues/58523

- 7627656

https://tracker.ceph.com/issues/62136

Actions

Copy link

#11