Bug #172
closedOSD and MDS crash on rm -r
0%
Description
I'm still using my test script which unpacks the kernel source and then removes it again with a few steps in between.
Right now the copy back from the snapshot goes fine, but afterwards the rm of the original kernel files fail.
Attached you will find various files with straces and logs in them, i'll try to point out the scenario:
The scripts runs fine and the copy back from the snapshot goes fine, expect for the messages in "ceph_client_script_log.txt"
Why can't those files be found? That seems like a different bug to me?
Well, the bug of the stalling cp seems to have been fixed, but while removing the files afterwards the MDS crashes first, 15 seconds later, all 5 OSD's go down as well.
In the mds strace and osd strace i've added a stat of /core, here you can see the MDS core dump is older then the OSD, thus the MDS crashes first.
After gathering all this information i started my OSD's and the crash MDS (192.168.6.206) again. While doing so, my cluster started to recover, but then mds0 (192.168.6.205) crashed. (Core and log is also attached in mds0_*)
I'm doing a clean mkcephfs right now and running the same test again, expecting the same result as it happened for the second time today.
My Envirioment:- Branch: unstable ( 7c0df0540700fe2816470f5cc2a2fc7a130e4456 )
- OS: Ubuntu 10.04 (AMD64)
- Kernel: 2.6.34
Files
Updated by Wido den Hollander almost 14 years ago
Today i ran the same test again, almost the same result.
Before i ran the test i created a fresh fs with mkcephfs.
Result: MDS crashed again, after which all 5 OSD's followed.
The MDS log gave some more information this the log gave some more information.
Attached you will find the log.
Updated by Sage Weil almost 14 years ago
Wido den Hollander wrote:
Today i ran the same test again, almost the same result.
Before i ran the test i created a fresh fs with mkcephfs.
Result: MDS crashed again, after which all 5 OSD's followed.
The MDS log gave some more information this the log gave some more information.
Attached you will find the log.
Was this with the unstable code from yesterday? There were a zillion little fixes to random stuff. I'm still hitting problems (not done yet!) but it's succeeding much more frequently than before.
Updated by Wido den Hollander almost 14 years ago
Sage Weil wrote:
Wido den Hollander wrote:
Today i ran the same test again, almost the same result.
Before i ran the test i created a fresh fs with mkcephfs.
Result: MDS crashed again, after which all 5 OSD's followed.
The MDS log gave some more information this the log gave some more information.
Attached you will find the log.
Was this with the unstable code from yesterday? There were a zillion little fixes to random stuff. I'm still hitting problems (not done yet!) but it's succeeding much more frequently than before.
This was with:
commit c4e6482d302aa288031ced6cd845d60ba655e5c8
Author: Sage Weil <sage@newdream.net>
Date: Thu Jun 3 17:32:39 2010 -0700
Updated by Sage Weil almost 14 years ago
- Status changed from New to Closed
Closing this one. The osd crash was a snap_trimmer bug fixed a few days ago.
Added a qa workunit that repeats this test.
Updated by John Spray over 7 years ago
- Project changed from Ceph to CephFS
- Category deleted (
1)
Bulk updating project=ceph category=mds bugs so that I can remove the MDS category from the Ceph project to avoid confusion.