Project

General

Profile

Actions
Copy link

Bug #21842

closed

"repair kvstore failed" in qa/workunits/cephtool/test_kvstore_tool.sh

Added by Kefu Chai over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Kefu Chai
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-10-18T17:18:21.710 INFO:tasks.workunit.client.0.smithi107.stdout:repair rocksdb failed : IO error: stat file: db: No such file or directory
2017-10-18T17:18:21.711 INFO:tasks.workunit.client.0.smithi107.stdout:repair kvstore failed
2017-10-18T17:18:21.740 INFO:tasks.workunit.client.0.smithi107.stderr:2017-10-18 17:18:21.732 7fd721d71380  1 stupidalloc shutdown
2017-10-18T17:18:21.741 INFO:tasks.workunit.client.0.smithi107.stderr:2017-10-18 17:18:21.732 7fd721d71380  1 bdev(0x56084e9d4f40 /tmp/cephtool.UNa/block) close
2017-10-18T17:18:21.982 INFO:tasks.workunit.client.0.smithi107.stderr:2017-10-18 17:18:21.976 7fd721d71380  1 bdev(0x56084e9d4d00 /tmp/cephtool.UNa/block) close
2017-10-18T17:18:22.019 INFO:tasks.workunit.client.0.smithi107.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test_kvstore_tool.sh:1: test_ceph_kvstore_tool:  rm -fr /tmp/cephtool.UNa
2017-10-18T17:18:22.022 INFO:tasks.workunit:Stopping ['cephtool'] on client.0...
2017-10-18T17:18:22.022 INFO:teuthology.orchestra.run.smithi107:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0'
2017-10-18T17:18:22.202 ERROR:teuthology.run_tasks:Saw exception from tasks.

http://qa-proxy.ceph.com/teuthology/kchai-2017-10-18_16:55:07-rados-wip-kefu-testing-2017-10-18-2318-distro-basic-smithi/1746146/teuthology.log

Actions
Copy link
 #1

Updated by Chang Liu over 6 years ago

Working on it

Actions
Copy link
 #2

Updated by Chang Liu over 6 years ago

Rocksdb::RepairDB tries to find all files: https://github.com/facebook/rocksdb/blob/master/db/repair.cc#L168, then it will call `env->GetChildren('db', result)`:https://github.com/facebook/rocksdb/blob/master/db/repair.cc#L271.

Our bluefs tells rocksdb that there is not such a directory called `db`.

I'm reproducing it in my env.

Actions
Copy link
 #3

Updated by Chang Liu over 6 years ago

I can't figure out where creates "stat file: db". I guess we use this "stat file: db" as our dbname.

Actions
Copy link
 #4

Updated by Kefu Chai over 6 years ago

Chang, no worries. i am fixing it.

Actions
Copy link
 #5

Updated by Kefu Chai over 6 years ago

  • Assignee changed from Chang Liu to Kefu Chai
Actions
Copy link
 #6

Updated by Chang Liu over 6 years ago

Kefu, I know why repair failed.

rocksdb's Env imports a new member function called AreSameFiles. But our BlueRocksEnv hasn't this function.

see https://github.com/ceph/ceph/pull/18390

Actions
Copy link
 #7

Updated by Kefu Chai over 6 years ago

I can't figure out where creates "stat file: db". I guess we use this "stat file: db" as our dbname.

and it consistently fails the rados qa suite, so i took it away from you.

https://github.com/ceph/ceph/pull/18392

Actions
Copy link
 #8

Updated by Kefu Chai over 6 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #9

Updated by Kefu Chai over 6 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF