Bug #63789: LibRadosIoEC test failure - RADOS - Ceph

Actions

Copy link

Bug #63789

open

LibRadosIoEC test failure

Added by Laura Flores 5 months ago. Updated 3 days ago.

Status:

In Progress

Priority:

Normal

Assignee:

Nitzan Mordechai

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2023-02-24_17:50:19-rados-main-distro-default-smithi/7186883
/>

2023-02-25T01:26:10.649 INFO:tasks.workunit.client.0.smithi008.stdout:                   api_io: [----------] 10 tests from LibRadosIoEC api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:171: Failure api_io: Expected equality of these values: api_io:   "" api_io:   create_one_ec_pool(pool_name, &s_cluster) api_io:     Which is: "rados_mon_command osd pool create failed with error -22" api_io: [ RUN      ] LibRadosIoEC.SimpleWrite api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.SimpleWrite (1 ms) api_io: [ RUN      ] LibRadosIoEC.RoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.RoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.OverlappingWriteRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.OverlappingWriteRoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.WriteFullRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.WriteFullRoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.AppendRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.AppendRoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.TruncTest api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.TruncTest (0 ms) api_io: [ RUN      ] LibRadosIoEC.RemoveTest api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.RemoveTest (0 ms) api_io: [ RUN      ] LibRadosIoEC.XattrsRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0


	First reported in https://tracker.ceph.com/issues/49428#note-7.














  Related issues
  1 (0 open — 1 closed)



  Related to RADOS - Bug #49428: ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" Duplicate Brad Hubbard 
Actions



Issue #

Delay:  days


Cancel








  
    History
    Notes
    Property changes
  
  
    
    
  


  
  
    
    
      ActionsCopy link
      #1
    
    
      
      Updated by Laura Flores 5 months ago
      
      
    

    /a/yuriw-2023-12-07_01:25:18-rados-wip-yuri2-testing-2023-12-06-1239-distro-default-smithi/7481365
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [----------] 10 tests from LibRadosIoEC
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:171: Failure
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   "" 
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   create_one_ec_pool(pool_name, &s_cluster)
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: "rados_mon_command osd pool create failed with error -22" 
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.SimpleWrite
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.SimpleWrite (0 ms)
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.RoundTrip
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.RoundTrip (0 ms)
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.OverlappingWriteRoundTrip
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.OverlappingWriteRoundTrip (0 ms)
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.WriteFullRoundTrip
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.WriteFullRoundTrip (0 ms)
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.AppendRoundTrip
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.AppendRoundTrip (0 ms)
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.TruncTest
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.TruncTest (0 ms)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.RemoveTest
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.RemoveTest (0 ms)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.XattrsRoundTrip
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.XattrsRoundTrip (0 ms)
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.RmXattr
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.RmXattr (1 ms)
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.XattrIter
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.XattrIter (0 ms)
2023-12-07T05:28:10.363 INFO:tasks.workunit.client.0.smithi082.stderr:bash: line 1: 40350 Segmentation fault      (core dumped) ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1
2023-12-07T05:28:10.363 INFO:tasks.workunit.client.0.smithi082.stderr:     40351 Done                    | tee ceph_test_rados_api_io.log
2023-12-07T05:28:10.363 INFO:tasks.workunit.client.0.smithi082.stderr:     40352 Done                    | sed "s/^/                   api_io: /" 

    
  
  
  
    
    
      ActionsCopy link
      #2
    
    
      
      Updated by Nitzan Mordechai 5 months ago
      
      
    

    the first error is -22 :
2023-12-07T05:28:10.195+0000 7f58944de640  1 -- [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- route(mon_command_ack([{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi082-40350-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi082-40350-16"}]=-22 cannot determine the erasure code plugin because there is no 'plugin' entry in the erasure_code_profile {} v23) v1 tid 171) v3 -- 0x557329b1c2c0 con 0x557328b26400




	we have a missing plugin entry in ec profile. the profile was created before:
2023-12-07T05:28:09.475+0000 7f58944de640 20 mon.a@0(leader).osd e22 erasure code profile set testprofile-LibRadosIoEC_smithi082-40350-16={crush-device-class=,crush-failure-domain=osd,crush-root=default,jerasure-per-chunk-alignment=false,k=2,m=1,plugin=jerasure,technique=reed_sol_van,w=8}

we had a plugin, but then we removed that profile and created it again, which is weird.
we are hitting a few inject socket failures during the creation and remove of the profile, i guess that's the reason we couldn't find a plugin (supposed to be the default)
    
  
  
  
    
    
      ActionsCopy link
      #3
    
    
      
      Updated by Nitzan Mordechai 5 months ago
      
      
    

    
       Status changed from New to In Progress
       Assignee set to Nitzan Mordechai
    
    ohh i didn't realize that if the profile was not created we would still show that error message, that explains why we will get it in all the other tests.
    
  
  
  
    
    
      ActionsCopy link
      #4
    
    
      
      Updated by Nitzan Mordechai 5 months ago
      
      
    

    
       Related to Bug #49428: ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" added
    
    
    
  
  
  
    
    
      ActionsCopy link
      #5
    
    
      
      Updated by Laura Flores 10 days ago
      
      
    

    /a/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687027
    
  
  
  
    
    
      ActionsCopy link
      #6
    
    
      
      Updated by Laura Flores 4 days ago
      
      
    

    @Nitzan Mordechai would you say this is more of a test issue?
    
  
  
  
    
    
      ActionsCopy link
      #7
    
    
      
      Updated by Nitzan Mordechai 3 days ago
      
      
    

    Laura Flores wrote in #note-6:




	@Nitzan Mordechai would you say this is more of a test issue?




	@Laura Flores  yes, we are using mon commands, it's async and we usually execute them in order, but not all the time. I'll need to add some checks before executing the next command, will work on that one.
    
  
  
  
    
    
      ActionsCopy link
      #8
    
    
      
      Updated by Nitzan Mordechai 3 days ago
      
      · Edited
    

    from the client log:
nmordech@teuthology:/a/yuriw-2024-05-02_23:59:28-rados-wip-yuriw11-testing-20240501.200505-squid-distro-default-smithi/7687027$ zgrep "osd erasure-code-profile" ./remote/smithi019/log/ceph-client.admin.44981.log.gz
2024-05-04T14:24:47.023+0000 7f7dc320c740 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]
2024-05-04T14:24:47.023+0000 7f7dc320c740 10 monclient: _send_command 1 [{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]
2024-05-04T14:24:47.023+0000 7f7dc320c740  1 -- 172.21.15.19:0/2256597415 --> [v2:172.21.15.19:3301/0,v1:172.21.15.19:6790/0] -- mon_command({"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"} v 0) -- 0x557d6b1a7cb0 con 0x7f7dac06b9c0
2024-05-04T14:24:47.028+0000 7f7dbf9e0640 10 monclient: _send_command 1 [{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]
2024-05-04T14:24:47.028+0000 7f7dbf9e0640  1 -- 172.21.15.19:0/2256597415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"} v 0) -- 0x7f7db80548c0 con 0x557d6a4a6cc0
2024-05-04T14:24:47.093+0000 7f7dbd9dc640  1 -- 172.21.15.19:0/2256597415 <== mon.0 v2:172.21.15.19:3300/0 4 ==== mon_command_ack([{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]=0 erasure-code-profile testprofile-LibRadosIoEC_smithi019-44981-16 does not exist v53) ==== 209+0+0 (secure 0 0 0) 0x7f7db8001e20 con 0x557d6a4a6cc0
2024-05-04T14:24:47.093+0000 7f7dbd9dc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]
2024-05-04T14:24:47.094+0000 7f7dc320c740 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi019-44981-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]
2024-05-04T14:24:47.094+0000 7f7dc320c740 10 monclient: _send_command 3 [{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi019-44981-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]
2024-05-04T14:24:47.094+0000 7f7dc320c740  1 -- 172.21.15.19:0/2256597415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi019-44981-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} v 0) -- 0x557d6b0d34b0 con 0x557d6a4a6cc0
2024-05-04T14:24:47.697+0000 7f7dbd9dc640  1 -- 172.21.15.19:0/2256597415 <== mon.0 v2:172.21.15.19:3300/0 6 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi019-44981-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]=0  v54) ==== 187+0+0 (secure 0 0 0) 0x7f7db80446e0 con 0x557d6a4a6cc0
2024-05-04T14:24:47.697+0000 7f7dbd9dc640 10 monclient: handle_mon_command_ack 3 [{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi019-44981-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]
2024-05-04T14:24:47.711+0000 7f7dc320c740 10 start_mon_command cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]
2024-05-04T14:24:47.711+0000 7f7dc320c740 10 monclient: _send_command 5 [{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]
2024-05-04T14:24:47.711+0000 7f7dc320c740  1 -- 172.21.15.19:0/2256597415 --> [v2:172.21.15.19:3300/0,v1:172.21.15.19:6789/0] -- mon_command({"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"} v 0) -- 0x557d6a646c20 con 0x557d6a4a6cc0
2024-05-04T14:24:47.727+0000 7f7dbd9dc640  1 -- 172.21.15.19:0/2256597415 <== mon.0 v2:172.21.15.19:3300/0 8 ==== mon_command_ack([{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]=0 erasure-code-profile testprofile-LibRadosIoEC_smithi019-44981-16 does not exist v54) ==== 209+0+0 (secure 0 0 0) 0x7f7db80898d0 con 0x557d6a4a6cc0
2024-05-04T14:24:47.727+0000 7f7dbd9dc640 10 monclient: handle_mon_command_ack 5 [{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi019-44981-16"}]



	we want to create erasure-code-profile named testprofile-LibRadosIoEC_smithi019-44981-16 and erasure code pool with that profile, we first remove any exist profile with that name, but we are sending 2 commands to 2 different mons (mon.c on  14:24:47.023 and to mon.a on 14:24:47.028) command to mon.a ack at 14:24:47.093 just before mon.c command sent, since we got ack we continue to create the profile (14:24:47.094) and got ack (14:24:47.697) then we got ack of the 2nd remove command (14:24:47.727) so finally we have destroyed profile after sending create which cause all the tests to fail. 
checking why we sent 2 remove command, or why we sent the last remove command before we checked for ack
    
  
  


  
  











Actions
  Copy link
  






Also available in:  Atom
  PDF

Project

General

Profile

Ceph » RADOS

Custom queries

Bug #63789

LibRadosIoEC test failure

Updated by Laura Flores 5 months ago

Updated by Nitzan Mordechai 5 months ago

Updated by Nitzan Mordechai 5 months ago

Updated by Nitzan Mordechai 5 months ago

Updated by Laura Flores 10 days ago

Updated by Laura Flores 4 days ago

Updated by Nitzan Mordechai 3 days ago

Updated by Nitzan Mordechai 3 days ago · Edited