Bug #51945: qa/workunits/mon/caps.sh: Error: Expected return 13, got 0 - RADOS - Ceph

Actions

Copy link

Bug #51945

open

qa/workunits/mon/caps.sh: Error: Expected return 13, got 0

Added by Neha Ojha over 2 years ago. Updated over 1 year ago.

Status:

New

Priority:

Normal

Assignee:

Radoslaw Zarzynski

Category:

Target version:

% Done:

Source:

Tags:

medium-hanging-fruit

Backport:

quincy, pacific, octopus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2021-07-27T21:11:15.731 INFO:tasks.workunit.client.0.smithi110.stderr:+ ceph auth del client.bar
2021-07-27T21:11:18.771 INFO:teuthology.orchestra.run.smithi077.stdout:{"election_epoch":190,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138297136906239","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"]},"monmap":{"epoch":1,"fsid":"87a21b71-6325-4bd7-b729-3438c61d134e","modified":"2021-07-27T20:51:11.510526Z","created":"2021-07-27T20:51:11.510526Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3300","nonce":0},{"type":"v1","addr":"172.21.15.77:6789","nonce":0}]},"addr":"172.21.15.77:6789/0","public_addr":"172.21.15.77:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:3301","nonce":0},{"type":"v1","addr":"172.21.15.77:6790","nonce":0}]},"addr":"172.21.15.77:6790/0","public_addr":"172.21.15.77:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}}
2021-07-27T21:11:19.158 INFO:tasks.workunit.client.0.smithi110.stderr:entity client.bar does not exist
2021-07-27T21:11:19.166 INFO:tasks.workunit.client.0.smithi110.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status' 13
2021-07-27T21:11:19.166 INFO:tasks.workunit.client.0.smithi110.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status'
2021-07-27T21:11:19.167 INFO:tasks.workunit.client.0.smithi110.stderr:+ expected_ret=13
2021-07-27T21:11:19.167 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status
2021-07-27T21:11:19.168 INFO:tasks.workunit.client.0.smithi110.stdout:ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status
2021-07-27T21:11:19.169 INFO:tasks.workunit.client.0.smithi110.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.bar.keyring --user bar quorum_status
2021-07-27T21:11:19.597 INFO:tasks.workunit.client.0.smithi110.stderr:+ ret=0
2021-07-27T21:11:19.597 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ 0 -ne 13 ]]
2021-07-27T21:11:19.597 INFO:tasks.workunit.client.0.smithi110.stderr:+ echo 'Error: Expected return 13, got 0'
2021-07-27T21:11:19.598 INFO:tasks.workunit.client.0.smithi110.stdout:Error: Expected return 13, got 0
2021-07-27T21:11:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-07-27T21:11:19.599 INFO:tasks.workunit.client.0.smithi110.stderr:+ [[ 1 -eq 1 ]]
2021-07-27T21:11:19.599 INFO:tasks.workunit.client.0.smithi110.stderr:+ exit 1

/a/yuriw-2021-07-27_17:19:39-rados-wip-yuri-testing-2021-07-27-0830-pacific-distro-basic-smithi/6297204

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Ronen Friedman over 2 years ago

Seems to be the same problem in:
http://pulpito.front.sepia.ceph.com/yuriw-2021-11-20_18:00:22-rados-wip-yuri6-testing-2021-11-20-0807-distro-basic-smithi/6516753

Actions

Copy link

Updated by Laura Flores over 2 years ago

/a/yuriw-2021-12-23_16:50:03-rados-wip-yuri6-testing-2021-12-22-1410-distro-default-smithi/6582413

2021-12-23T18:37:45.913 INFO:tasks.workunit.client.0.smithi117.stderr:entity client.foo does not exist
2021-12-23T18:37:45.919 INFO:tasks.workunit.client.0.smithi117.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13
2021-12-23T18:37:45.919 INFO:tasks.workunit.client.0.smithi117.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status'
2021-12-23T18:37:45.919 INFO:tasks.workunit.client.0.smithi117.stderr:+ expected_ret=13
2021-12-23T18:37:45.920 INFO:tasks.workunit.client.0.smithi117.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status
2021-12-23T18:37:45.920 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status
2021-12-23T18:37:45.920 INFO:tasks.workunit.client.0.smithi117.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status
2021-12-23T18:37:46.438 INFO:tasks.workunit.client.0.smithi117.stderr:+ ret=0
2021-12-23T18:37:46.438 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ 0 -ne 13 ]]
2021-12-23T18:37:46.439 INFO:tasks.workunit.client.0.smithi117.stdout:Error: Expected return 13, got 0
2021-12-23T18:37:46.440 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-12-23T18:37:46.440 INFO:tasks.workunit.client.0.smithi117.stderr:+ echo 'Error: Expected return 13, got 0'
2021-12-23T18:37:46.441 INFO:tasks.workunit.client.0.smithi117.stderr:+ [[ 1 -eq 1 ]]
2021-12-23T18:37:46.441 INFO:tasks.workunit.client.0.smithi117.stderr:+ exit 1

Actions

Copy link

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-11_00:13:58-rados-wip-yuri11-testing-2022-03-10-1443-octopus-distro-default-smithi/6730807

2022-03-11T02:40:17.017 INFO:tasks.workunit.client.0.smithi136.stderr:entity client.foo does not exist
2022-03-11T02:40:17.030 INFO:tasks.workunit.client.0.smithi136.stderr:+ expect 'ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status' 13
2022-03-11T02:40:17.030 INFO:tasks.workunit.client.0.smithi136.stderr:+ cmd='ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status'
2022-03-11T02:40:17.031 INFO:tasks.workunit.client.0.smithi136.stdout:ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status
2022-03-11T02:40:17.031 INFO:tasks.workunit.client.0.smithi136.stderr:+ expected_ret=13
2022-03-11T02:40:17.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status
2022-03-11T02:40:17.032 INFO:tasks.workunit.client.0.smithi136.stderr:+ eval ceph -k /tmp/cephtest-mon-caps-madness.foo.keyring --user foo quorum_status
2022-03-11T02:40:25.846 INFO:tasks.mon_thrash:Sending CONT to mon a
2022-03-11T02:40:25.847 INFO:tasks.ceph.mon.a:Sent signal 18
2022-03-11T02:40:25.847 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3
2022-03-11T02:40:25.848 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status
2022-03-11T02:40:26.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ ret=0
2022-03-11T02:40:26.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ 0 -ne 13 ]]
2022-03-11T02:40:26.912 INFO:tasks.workunit.client.0.smithi136.stderr:+ echo 'Error: Expected return 13, got 0'
2022-03-11T02:40:26.913 INFO:tasks.workunit.client.0.smithi136.stdout:Error: Expected return 13, got 0
2022-03-11T02:40:26.914 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-03-11T02:40:26.914 INFO:tasks.workunit.client.0.smithi136.stderr:+ [[ 1 -eq 1 ]]
2022-03-11T02:40:26.915 INFO:tasks.workunit.client.0.smithi136.stderr:+ exit 1

Actions

Copy link

Updated by Laura Flores about 2 years ago

Backport changed from pacific to quincy,pacific, octopus

Actions

Copy link

Updated by Laura Flores about 2 years ago

Backport changed from quincy,pacific, octopus to quincy, pacific, octopus

Actions

Copy link

Updated by Radoslaw Zarzynski about 2 years ago

Assignee set to Radoslaw Zarzynski

Actions

Copy link

Updated by Laura Flores almost 2 years ago

/a/yuriw-2022-06-16_19:58:30-rados-wip-yuri7-testing-2022-06-16-1051-pacific-distro-default-smithi/6882914

Actions

Copy link

Updated by Radoslaw Zarzynski almost 2 years ago

Tags set to medium-hanging-fruit

Actions

Copy link

Updated by Radoslaw Zarzynski over 1 year ago

The wrong return code is just an echo of a failure with an auth entity deletion:

2022-06-16T22:41:28.399 INFO:tasks.workunit.client.0.smithi103.stderr:+ ceph auth del client.foo
2022-06-16T22:41:39.662 INFO:tasks.workunit.client.0.smithi103.stderr:entity client.foo does not exist

It affects multiple entities in the test:

2022-06-16T22:36:25.463 INFO:tasks.workunit.client.0.smithi103.stderr:+ ceph auth del client.bazar
2022-06-16T22:36:25.931 INFO:tasks.workunit.client.0.smithi103.stderr:entity client.bazar does not exist

Actions

Copy link

#10

Updated by Radoslaw Zarzynski over 1 year ago

  } else if ((prefix == "auth del" || prefix == "auth rm") &&
             !entity_name.empty()) {
    KeyServerData::Incremental auth_inc;
    auth_inc.name = entity;
    if (!mon.key_server.contains(auth_inc.name)) {
      ss << "entity " << entity << " does not exist";
      err = 0;
      goto done;
    }
    auth_inc.op = KeyServerData::AUTH_INC_DEL;
    push_cephx_inc(auth_inc);

    ss << "updated";
    getline(ss, rs);
    wait_for_finished_proposal(op, new Monitor::C_Command(mon, op, 0, rs,
                                              get_last_committed() + 1));
    return true;
  }

Why does the key server lacks these entities?

bool KeyServer::contains(const EntityName& name) const
{
  std::scoped_lock l{lock};

  return data.contains(name);
}

Well, the exact (but not-so-useful) answer is because KeyServerData lacks it:

struct KeyServerData {
  version_t version;

  /* for each entity */
  std::map<EntityName, EntityAuth> secrets;
  KeyRing *extra_secrets;

  /* for each service type */
  version_t rotating_ver;
  std::map<uint32_t, RotatingSecrets> rotating_secrets;

  explicit KeyServerData(KeyRing *extra)
    : version(0),
      extra_secrets(extra),
      rotating_ver(0) {}

  // ...
  bool contains(const EntityName& name) const {
    return (secrets.find(name) != secrets.end());
  }

  void clear_secrets() {
    version = 0;
    secrets.clear();
    rotating_ver = 0;
    rotating_secrets.clear();
  }

  void add_auth(const EntityName& name, EntityAuth& auth) {
    secrets[name] = auth;
  }

  void remove_secret(const EntityName& name) {
    auto iter = secrets.find(name);
    if (iter == secrets.end())
      return;
    secrets.erase(iter);
  }

Is this a monitors' sync issue?

Actions

Copy link

#11

Updated by Laura Flores over 1 year ago

/a/yuriw-2022-11-28_16:10:10-rados-wip-yuri6-testing-2022-11-23-1348-distro-default-smithi/7093588

Actions

Copy link

#12

Updated by Radoslaw Zarzynski over 1 year ago

Related to Bug #58305: src/mon/AuthMonitor.cc: FAILED ceph_assert(version > keys_ver) added

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » RADOS

Custom queries

Bug #51945

qa/workunits/mon/caps.sh: Error: Expected return 13, got 0

Updated by Ronen Friedman over 2 years ago

Updated by Laura Flores over 2 years ago

Updated by Laura Flores about 2 years ago

Updated by Laura Flores about 2 years ago

Updated by Laura Flores about 2 years ago

Updated by Radoslaw Zarzynski about 2 years ago

Updated by Laura Flores almost 2 years ago

Updated by Radoslaw Zarzynski almost 2 years ago

Updated by Radoslaw Zarzynski over 1 year ago

Updated by Radoslaw Zarzynski over 1 year ago

Updated by Laura Flores over 1 year ago

Updated by Radoslaw Zarzynski over 1 year ago