Bug #63789: LibRadosIoEC test failure - RADOS - Ceph

Actions

Copy link

Bug #63789

open

LibRadosIoEC test failure

Added by Laura Flores 5 months ago. Updated 5 months ago.

Status:

In Progress

Priority:

Normal

Assignee:

Nitzan Mordechai

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2023-02-24_17:50:19-rados-main-distro-default-smithi/7186883
/>

2023-02-25T01:26:10.649 INFO:tasks.workunit.client.0.smithi008.stdout:                   api_io: [----------] 10 tests from LibRadosIoEC api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:171: Failure api_io: Expected equality of these values: api_io:   "" api_io:   create_one_ec_pool(pool_name, &s_cluster) api_io:     Which is: "rados_mon_command osd pool create failed with error -22" api_io: [ RUN      ] LibRadosIoEC.SimpleWrite api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.SimpleWrite (1 ms) api_io: [ RUN      ] LibRadosIoEC.RoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.RoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.OverlappingWriteRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.OverlappingWriteRoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.WriteFullRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.WriteFullRoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.AppendRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.AppendRoundTrip (0 ms) api_io: [ RUN      ] LibRadosIoEC.TruncTest api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.TruncTest (0 ms) api_io: [ RUN      ] LibRadosIoEC.RemoveTest api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0 api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx) api_io:     Which is: -107 api_io: [  FAILED  ] LibRadosIoEC.RemoveTest (0 ms) api_io: [ RUN      ] LibRadosIoEC.XattrsRoundTrip api_io: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2514-gf9d812a5/rpm/el8/BUILD/ceph-18.0.0-2514-gf9d812a5/src/test/librados/TestCase.cc:184: Failure api_io: Expected equality of these values: api_io:   0


	First reported in https://tracker.ceph.com/issues/49428#note-7.














  Related issues
  1 (0 open — 1 closed)



  Related to RADOS - Bug #49428: ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" Duplicate Brad Hubbard 
Actions



Issue #

Delay:  days


Cancel








  
    History
    Notes
    Property changes
  
  
    
    
  


  
  
    
    
      ActionsCopy link
      #1
    
    
      
      Updated by Laura Flores 5 months ago
      
      
    

    /a/yuriw-2023-12-07_01:25:18-rados-wip-yuri2-testing-2023-12-06-1239-distro-default-smithi/7481365
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [----------] 10 tests from LibRadosIoEC
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:171: Failure
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   "" 
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   create_one_ec_pool(pool_name, &s_cluster)
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: "rados_mon_command osd pool create failed with error -22" 
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.SimpleWrite
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.292 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.SimpleWrite (0 ms)
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.RoundTrip
2023-12-07T05:28:10.293 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.RoundTrip (0 ms)
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.OverlappingWriteRoundTrip
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.294 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.OverlappingWriteRoundTrip (0 ms)
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.WriteFullRoundTrip
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.307 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.WriteFullRoundTrip (0 ms)
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.AppendRoundTrip
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.AppendRoundTrip (0 ms)
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.TruncTest
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.308 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.TruncTest (0 ms)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.RemoveTest
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.RemoveTest (0 ms)
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.XattrsRoundTrip
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.309 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.XattrsRoundTrip (0 ms)
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.RmXattr
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.310 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.RmXattr (1 ms)
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [ RUN      ] LibRadosIoEC.XattrIter
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: ./src/test/librados/TestCase.cc:184: Failure
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: Expected equality of these values:
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   0
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:   rados_ioctx_create(cluster, pool_name.c_str(), &ioctx)
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io:     Which is: -107
2023-12-07T05:28:10.311 INFO:tasks.workunit.client.0.smithi082.stdout:                   api_io: [  FAILED  ] LibRadosIoEC.XattrIter (0 ms)
2023-12-07T05:28:10.363 INFO:tasks.workunit.client.0.smithi082.stderr:bash: line 1: 40350 Segmentation fault      (core dumped) ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/gtest_xml_report-api_io.xml 2>&1
2023-12-07T05:28:10.363 INFO:tasks.workunit.client.0.smithi082.stderr:     40351 Done                    | tee ceph_test_rados_api_io.log
2023-12-07T05:28:10.363 INFO:tasks.workunit.client.0.smithi082.stderr:     40352 Done                    | sed "s/^/                   api_io: /" 

    
  
  
  
    
    
      ActionsCopy link
      #2
    
    
      
      Updated by Nitzan Mordechai 5 months ago
      
      
    

    the first error is -22 :
2023-12-07T05:28:10.195+0000 7f58944de640  1 -- [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] --> [v2:172.21.15.82:3301/0,v1:172.21.15.82:6790/0] -- route(mon_command_ack([{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi082-40350-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi082-40350-16"}]=-22 cannot determine the erasure code plugin because there is no 'plugin' entry in the erasure_code_profile {} v23) v1 tid 171) v3 -- 0x557329b1c2c0 con 0x557328b26400




	we have a missing plugin entry in ec profile. the profile was created before:
2023-12-07T05:28:09.475+0000 7f58944de640 20 mon.a@0(leader).osd e22 erasure code profile set testprofile-LibRadosIoEC_smithi082-40350-16={crush-device-class=,crush-failure-domain=osd,crush-root=default,jerasure-per-chunk-alignment=false,k=2,m=1,plugin=jerasure,technique=reed_sol_van,w=8}

we had a plugin, but then we removed that profile and created it again, which is weird.
we are hitting a few inject socket failures during the creation and remove of the profile, i guess that's the reason we couldn't find a plugin (supposed to be the default)
    
  
  
  
    
    
      ActionsCopy link
      #3
    
    
      
      Updated by Nitzan Mordechai 5 months ago
      
      
    

    
       Status changed from New to In Progress
       Assignee set to Nitzan Mordechai
    
    ohh i didn't realize that if the profile was not created we would still show that error message, that explains why we will get it in all the other tests.
    
  
  
  
    
    
      ActionsCopy link
      #4
    
    
      
      Updated by Nitzan Mordechai 5 months ago
      
      
    

    
       Related to Bug #49428: ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed with error -22" added
    
    
    
  
  


  
  











Actions
  Copy link
  






Also available in:  Atom
  PDF

Project

General

Profile

Ceph » RADOS

Custom queries

Bug #63789

LibRadosIoEC test failure

Updated by Laura Flores 5 months ago

Updated by Nitzan Mordechai 5 months ago

Updated by Nitzan Mordechai 5 months ago

Updated by Nitzan Mordechai 5 months ago