Ceph » RADOS

Something around 10-12 august.

When I tried to debug this on august 12th this was already failing. I can try to find more information in the CI logs.

At the moment I can only tell you that 4fa1eaf is what was tested on august 12th.

Dimitri Savineau wrote:

Something around 10-12 august.

When I tried to debug this on august 12th this was already failing. I can try to find more information in the CI logs.

https://github.com/ceph/ceph/pull/33912 merged in the beginning of July so if you have a successful run from before then and none after that, it would explain why.

At the moment I can only tell you that 4fa1eaf is what was tested on august 12th.

Is reverting 7e4275390c713b8728a3deab7058bf5e21c424cb and testing an option for you?

https://github.com/ceph/ceph/pull/33912 merged in the beginning of July so if you have a successful run from before then and none after that, it would explain why.

I thought about that one but the merge date doesn't really matched.

This PR was merged on 8 Jul.

I found successfull CI jobs until 10 Aug (7ce4f9a4fd).
I also found older failure (compared to 4fa1eaf) on 11 Aug (740480660a)

Is reverting 7e4275390c713b8728a3deab7058bf5e21c424cb and testing an option for you?

I can give it a try

i don't think 7e4275390c713b8728a3deab7058bf5e21c424cb is the offending change.

and i am able to connect my vstart cluster using:

$ export PYTHONPATH=/var/ssd/ceph/src/pybind:/var/ssd/ceph/build/lib/cython_modules/lib.3
$ python3
>>> import rados
>>> cluster = rados.Rados(conffile='/var/ssd/ceph/build/ceph.conf', name='client.admin')
>>> cluster.connect()

There's something quite different about the local environment (vstart cluster) and the teuthology environment. For example:

Teuthology.

$ python -c "import rados;rados = rados.Rados(clustername='ceph', conffile=\"/etc/ceph/ceph.conf\", name='client.admin');rados.connect();print(rados.get_cluster_stats())" 
{'kb': 1132462080, 'kb_used': 13240608, 'kb_avail': 1119221472, 'num_objects': 411}
$ python -c "import rados;rados = rados.Rados(clustername='fred', conffile=\"/etc/ceph/ceph.conf\", name='client.admin');rados.connect();print(rados.get_cluster_stats())" 
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "rados.pyx", line 1003, in rados.Rados.connect
rados.ObjectNotFound: [errno 2] RADOS object not found (error connecting to the cluster)

Local.

$ python -c "import rados;rados = rados.Rados(clustername='ceph', conffile=\"/etc/ceph/ceph.conf\", name='client.admin');rados.connect();print(rados.get_cluster_stats())" 
{'kb': 317718504, 'kb_used': 3146232, 'kb_avail': 314572272, 'num_objects': 1}
python -c "import rados;rados = rados.Rados(clustername='fred', conffile=\"/etc/ceph/ceph.conf\", name='client.admin');rados.connect();print(rados.get_cluster_stats())" 
{'kb': 317718504, 'kb_used': 3146232, 'kb_avail': 314572272, 'num_objects': 1}

So in the teuthology environment the 'clustername' argument is crucial whereas locally it's ignored. Need to think about what's causing that difference.

Progress update.

I eliminated anything to do with python by writing a small C client that just reads the conf file and connects.

The actual cause of the failure to connect is that it can not find a cephx keyring for the client and therefore disables cephx. The ENOENT error comes from the following code.

 459 int MonClient::init()                                                                                                                                                                                                                   
 460 {
 461   ldout(cct, 10) << __func__ << dendl;
 462 
 463   entity_name = cct->_conf->name;
 464 
 465   auth_registry.refresh_config();
 466 
 467   std::lock_guard l(monc_lock);
 468   keyring.reset(new KeyRing);
 469   if (auth_registry.is_supported_method(messenger->get_mytype(),
 470 »·······»·······»·······»·······»·······CEPH_AUTH_CEPHX)) {
 471     // this should succeed, because auth_registry just checked!
 472     int r = keyring->from_ceph_context(cct);
 473     if (r != 0) {
 474       // but be somewhat graceful in case there was a race condition
 475       lderr(cct) << "keyring not found" << dendl;
 476       return r;
 477     }
 478   }
 479   if (!auth_registry.any_supported_methods(messenger->get_mytype())) {
 480     return -ENOENT;
 481   }

The difference between the teuthology test and the local test, at least at one level, is that the local ceph.conf has a 'keyring' line whereas the teuthology ceph.conf does not include any 'keyring' line. This investigation was badly hampered by the fact that I could not get debugging output. Anything I did to enable debugging caused the issue to disappear (just your typical heisenbug). Testing with the commits from https://github.com/ceph/ceph/pull/36516 reverted does not resolve the issue.

Still looking for the answer to 'what happened?' but we are closing in.

I have a candidate patch for this that I should be able to test tomorrow morning APAC time (it's building now).

Ok, I've had a chance to test this. Without my patch in rados_connect we see the following value for 'keyring' and the connection fails with ENOENT.

Breakpoint 1, librados::v14_2_0::RadosClient::connect (this=0x4bd290) at /home/brad/working/src/ceph/src/librados/RadosClient.cc:212
212     {
(gdb)  p cct->_conf->keyring
$1 = "/etc/ceph/.client.admin.keyring,/etc/ceph/.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin," 

With the patch we get the following and the connection succeeds.

Breakpoint 5, librados::v14_2_0::RadosClient::connect (this=0x4bd290) at /home/brad/working/src/ceph/src/librados/RadosClient.cc:212
212     {
(gdb) p cct->_conf->keyring
$7 = "/etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin," 

I've satisfied myself regarding the lingering questions I had about this issue so we are good to go.

A cherry-pick of this PR has caused the ragweed tests to start passing and https://tracker.ceph.com/issues/47060 to go away: http://qa-proxy.ceph.com/teuthology/amaredia-2020-08-26_12:35:49-rgw:verify-wip-s3test-go-distro-basic-smithi/5377877/teuthology.log