Ceph » bluestore

This may sound crazy but the only explanation that I'm able to come up with for these failures is that the octopus -> pacific upgraded OSDs are not persisting omap updates. I reran another, simpler, job from Yuri's run 5 times and the failures are exactly the same:

https://pulpito.ceph.com/dis-2022-04-28_17:15:07-upgrade:octopus-x-pacific-distro-default-smithi/

2022-04-28T18:14:46.196 INFO:tasks.workunit.client.0.smithi156.stdout:[==========] 27 tests from 1 test suite ran. (2842 ms total)
2022-04-28T18:14:46.196 INFO:tasks.workunit.client.0.smithi156.stdout:[  PASSED  ] 20 tests.
2022-04-28T18:14:46.196 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] 7 tests, listed below:
2022-04-28T18:14:46.196 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.ClientRegister
2022-04-28T18:14:46.196 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.ClientUnregisterPruneTags
2022-04-28T18:14:46.196 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.ClientCommit
2022-04-28T18:14:46.197 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.ClientList
2022-04-28T18:14:46.197 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.TagCreate
2022-04-28T18:14:46.197 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.TagCreatePrunesTags
2022-04-28T18:14:46.197 INFO:tasks.workunit.client.0.smithi156.stdout:[  FAILED  ] TestClsJournal.TagList

Honing in on TestClsJournal.ClientRegister failure, it is really simple:

[ RUN      ] TestClsJournal.ClientRegister
/build/ceph-15.2.16/src/test/cls_journal/test_cls_journal.cc:249: Failure
Expected equality of these values:
  expected_clients
    Which is: { [id=id1, commit_position=[positions=[]], state=connected] }
  clients
    Which is: {}
[  FAILED  ] TestClsJournal.ClientRegister (23 ms)

TEST_F(TestClsJournal, ClientRegister) {
  librados::IoCtx ioctx;
  ASSERT_EQ(0, _rados.ioctx_create(_pool_name.c_str(), ioctx));

  std::string oid = get_temp_image_name();
  ASSERT_EQ(0, client::create(ioctx, oid, 2, 4, ioctx.get_id()));                      // create image1 object, set some keys on it

  ASSERT_EQ(0, client::client_register(ioctx, oid, "id1", bufferlist()));              // set client_id1 key on the same object

  std::set<Client> clients;
  ASSERT_EQ(0, client::client_list(ioctx, oid, &clients));                             // read client_id1 key on the same object -- it appears that such key doesn't exist!
                                                                                       //   and the keys that were supposed to be set by client::create don't seem to be there either...

  std::set<Client> expected_clients = {Client("id1", bufferlist())};
  ASSERT_EQ(expected_clients, clients);                                                // assert fires because |clients| set remained empty
}

Yup, omap is definitely involved. LibRadosAio.OmapPP passes against fresh pacific OSDs and fails against upgraded OSDs:

$ ceph_test_rados_api_aio_pp --gtest_filter=LibRadosAio.OmapPP
Running main() from gmock_main.cc
Note: Google Test filter = LibRadosAio.OmapPP
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from LibRadosAio
[ RUN      ] LibRadosAio.OmapPP
/build/ceph-15.2.16/src/test/librados/aio_cxx.cc:1090: Failure
Expected equality of these values:
  set_got.size()
    Which is: 0
  (unsigned)1
    Which is: 1
[  FAILED  ] LibRadosAio.OmapPP (1725 ms)
[----------] 1 test from LibRadosAio (1726 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (1726 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LibRadosAio.OmapPP

 1 FAILED TEST

Ilya, can you run https://github.com/ceph/ceph-ci/commits/wip-55444-pacific through your reproducer?

Ilya Dryomov wrote:

This may sound crazy but the only explanation that I'm able to come up with for these failures is that the octopus -> pacific upgraded OSDs are not persisting omap updates.

Looking at https://github.com/ceph/ceph/pull/45963, omap updates are probably persisted correctly but listomapkeys just refuses to list anything after the fact.

Can you check if the same test also fails in master, which includes https://github.com/ceph/ceph/pull/45904? I'm going to go ahead with merging the Pacific revert, but would like to understand if we also need to revert it in master.

Ilya Dryomov wrote:

Yup, omap is definitely involved. LibRadosAio.OmapPP passes against fresh pacific OSDs and fails against upgraded OSDs:

[...]

What confuses me is that this test passed in the rados run of the https://github.com/ceph/ceph/pull/45963,

http://pulpito.front.sepia.ceph.com/lflores-2022-04-22_20:48:19-rados-wip-55324-pacific-backport-distro-default-smithi/6801057/

2022-04-23T08:00:25.014 INFO:tasks.workunit.client.0.smithi059.stdout:               api_aio_pp: [ RUN      ] LibRadosAio.OmapPP
2022-04-23T08:00:25.014 INFO:tasks.workunit.client.0.smithi059.stdout:               api_aio_pp: [       OK ] LibRadosAio.OmapPP (2939 ms)

It is triggered only with upgraded OSDs.

Ilya Dryomov wrote:

It is triggered only with upgraded OSDs.

Ah, that makes sense.

Neha Ojha wrote:

Ilya Dryomov wrote:

It is triggered only with upgraded OSDs.

Ah, that makes sense.

Not for me, at least not now.

In /a/yuriw-2022-04-27_14:24:25-upgrade:octopus-x-pacific-distro-default-smithi/6808913, we are using an octopus version of the workunit

  - workunit:
      branch: octopus
      clients:
        client.0:
        - cls/test_cls_rbd.sh
  - print: '**** done cls/test_cls_rbd.sh 5-workload'

Half of the OSDs were upgraded

2022-04-27T18:55:29.564 INFO:teuthology.orchestra.run.smithi006.stdout:{
2022-04-27T18:55:29.564 INFO:teuthology.orchestra.run.smithi006.stdout:    "mon": {
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 16.2.7-969-g4fa079ba (4fa079ba14503defa8dc257d7c2d506ebefcfe6d) pacific (stable)": 3
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:    },
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:    "mgr": {
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 16.2.7-969-g4fa079ba (4fa079ba14503defa8dc257d7c2d506ebefcfe6d) pacific (stable)": 2
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:    },
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:    "osd": {
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 15.2.16-79-gb51d8665 (b51d866559c7cde15d3b75ee17bbba8fdf2584be) octopus (stable)": 4,
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 16.2.7-969-g4fa079ba (4fa079ba14503defa8dc257d7c2d506ebefcfe6d) pacific (stable)": 4
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:    },
2022-04-27T18:55:29.565 INFO:teuthology.orchestra.run.smithi006.stdout:    "mds": {
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 15.2.16-79-gb51d8665 (b51d866559c7cde15d3b75ee17bbba8fdf2584be) octopus (stable)": 2
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:    },
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:    "overall": {
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 15.2.16-79-gb51d8665 (b51d866559c7cde15d3b75ee17bbba8fdf2584be) octopus (stable)": 6,
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:        "ceph version 16.2.7-969-g4fa079ba (4fa079ba14503defa8dc257d7c2d506ebefcfe6d) pacific (stable)": 9
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:    }
2022-04-27T18:55:29.566 INFO:teuthology.orchestra.run.smithi006.stdout:}

2022-04-27T19:24:46.094 INFO:tasks.workunit.client.0.smithi006.stdout:[ RUN      ] TestClsRbd.parents_v1
2022-04-27T19:24:46.117 INFO:tasks.workunit.client.0.smithi006.stdout:/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.16/rpm/el8/BUILD/ceph-15.2.16/src/test/cls_rbd/test_cls_rbd.cc:826: Failure
2022-04-27T19:24:46.117 INFO:tasks.workunit.client.0.smithi006.stdout:Expected equality of these values:
2022-04-27T19:24:46.117 INFO:tasks.workunit.client.0.smithi006.stdout:  0
2022-04-27T19:24:46.117 INFO:tasks.workunit.client.0.smithi006.stdout:  get_parent(&ioctx, oid, 10, &pspec, &size)
2022-04-27T19:24:46.117 INFO:tasks.workunit.client.0.smithi006.stdout:    Which is: -22
2022-04-27T19:24:46.118 INFO:tasks.workunit.client.0.smithi006.stdout:[  FAILED  ] TestClsRbd.parents_v1 (23 ms)

Half of the OSDs were upgraded

Just noting that the "half" part is irrelevant -- in my simpler run linked in https://tracker.ceph.com/issues/55444#note-3 all OSDs were upgraded (which is one of the reasons I chose that job to focus on).

Neha Ojha wrote:

Can you check if the same test also fails in master, which includes https://github.com/ceph/ceph/pull/45904? I'm going to go ahead with merging the Pacific revert, but would like to understand if we also need to revert it in master.

Upgrading octopus OSDs to master also triggers it -- exactly the same as upgrading octopus OSDs to pacific-with-pr-45963.

... however upgrading pacific-without-pr-45963 OSDs to master does NOT trigger it.

Could this be related to the switch to per-PG omap in pacific? Perhaps Cory's improvement just doesn't consider the old naming scheme?

https://github.com/ceph/ceph/pull/46092 merged

also in http://pulpito.front.sepia.ceph.com/yuriw-2022-04-30_04:14:21-upgrade:pacific-p2p-pacific-16.2.8_RC1-distro-default-smithi/