Bug #5596: OSD keeps crashing while trying to rewight another OSD - Ceph - Ceph

Actions

Copy link

Bug #5596

closed

OSD keeps crashing while trying to rewight another OSD

Added by Stefan Priebe almost 11 years ago. Updated almost 11 years ago.

Status:

Can't reproduce

Priority:

High

Assignee:

David Zafman

Category:

Target version:

% Done:

Source:

other

Tags:

Backport:

cuttlefish

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

I've added 8 new OSDs with a weight of 0. I then tried to reweight them by adding 0.01 in each turn to the weight of these OSDs. While doing this always the SAME OSDs keeps crashing on one host.

The log looks like this:
2013-07-11 12:30:42.130090 7fba0935f700 0 log [INF] : 4.7d1 restarting backfill on osd.10 from (0'0,0'0] MAX to 112245'2170839
2013-07-11 12:32:03.644426 7fba0935f700 0 log [INF] : 4.7d1 restarting backfill on osd.34 from (0'0,0'0] MAX to 112278'2170860
2013-07-11 12:34:02.153551 7fba08b5e700 0 log [INF] : 4.7d1 restarting backfill on osd.34 from (0'0,0'0] MAX to 112314'2170874
2013-07-11 12:34:10.580863 7fba08b5e700 0 log [INF] : 4.f0a restarting backfill on osd.17 from (0'0,0'0] MAX to 112320'925540
2013-07-11 12:35:00.481146 7fba08b5e700 0 log [INF] : 4.f0a restarting backfill on osd.7 from (0'0,0'0] MAX to 112320'925540
2013-07-11 12:37:31.004458 7fba08b5e700 0 log [INF] : 4.f0a restarting backfill on osd.7 from (0'0,0'0] MAX to 112407'925541
2013-07-11 12:37:42.931338 7fb9da784700 0 -- 10.255.0.84:6814/15519 >> 10.255.0.84:6812/15242 pipe(0x2f56e280 sd=231 :6814 s=0 pgs=
0 cs=0 l=0).accept connect_seq 2 vs existing 1 state standby
2013-07-11 12:38:58.624395 7fba08b5e700 0 log [INF] : 4.f0a restarting backfill on osd.7 from (0'0,0'0] MAX to 112469'925544
2013-07-11 12:39:05.564437 7fba08b5e700 0 log [INF] : 4.a4e restarting backfill on osd.17 from (0'0,0'0] MAX to 112467'819358
2013-07-11 12:39:06.464128 7fba08b5e700 0 log [INF] : 4.aaf restarting backfill on osd.9 from (0'0,0'0] MAX to 112461'705327

then the process is away.

Restarting this OSD looks like this:
2013-07-11 12:40:43.959164 7f00df5e6780 0 filestore(/ceph/osd.14/) mount FIEMAP ioctl is supported and appears to work
2013-07-11 12:40:43.959176 7f00df5e6780 0 filestore(/ceph/osd.14/) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-07-11 12:40:43.959318 7f00df5e6780 0 filestore(/ceph/osd.14/) mount did NOT detect btrfs
2013-07-11 12:40:43.962964 7f00df5e6780 0 filestore(/ceph/osd.14/) mount syscall(_NR_syncfs, fd) fully supported
2013-07-11 12:40:43.963471 7f00df5e6780 0 filestore(/ceph/osd.14/) mount found snaps <>
2013-07-11 12:40:44.416747 7f00df5e6780 -1 filestore(/ceph/osd.14/) Compacting store...
2013-07-11 12:41:31.871297 7f00df5e6780 -1 filestore(/ceph/osd.14/) ...finished compacting store
2013-07-11 12:41:31.871817 7f00df5e6780 0 filestore(/ceph/osd.14/) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-07-11 12:41:31.997018 7f00df5e6780 0 journal kernel version is 3.9.7
2013-07-11 12:41:32.066790 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066818 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066824 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066903 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066916 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066922 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066929 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066936 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066943 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066950 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066956 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.066963 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067038 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067048 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067053 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067065 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067070 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067077 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067083 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067091 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067097 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067105 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067111 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067119 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067125 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067132 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067138 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067144 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067151 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067159 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.067166 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142888 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142909 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142915 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142920 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142926 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142933 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142938 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142945 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142951 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142957 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142963 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142969 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142975 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142981 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142987 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142992 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.142998 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143005 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143013 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143019 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143026 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143034 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143042 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143119 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143128 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143134 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143203 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143212 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143217 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.143223 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199536 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199561 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199571 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199582 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199593 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199603 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199613 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199628 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199642 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199740 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199753 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199763 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199778 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199791 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199803 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199815 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199903 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.199915 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200002 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200014 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200024 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200107 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200120 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200131 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200143 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200156 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200169 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200253 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200265 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.200276 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222598 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222618 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222635 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222645 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222656 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222668 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222682 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222694 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222706 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222720 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222732 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222744 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222835 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222848 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222858 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222871 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222882 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222894 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222906 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222920 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222932 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222948 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222960 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222972 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222985 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.222998 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.223084 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.223097 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.223108 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.223190 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294502 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294526 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294612 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294686 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294695 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294701 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294707 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294713 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294719 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294724 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294729 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294739 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294744 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294749 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294755 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294760 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294766 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294772 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294843 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294852 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294858 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294863 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294932 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294941 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294946 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294951 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294957 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294962 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.294967 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.295038 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366545 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366566 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366572 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366578 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366588 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366593 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366599 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366604 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366609 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366720 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366730 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366736 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366741 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366746 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366752 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366757 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366829 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366839 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366844 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366850 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366861 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366866 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366871 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366876 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366882 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366888 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366894 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366968 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.366977 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.367004 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388163 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388183 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388188 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388193 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388198 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388203 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388209 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388214 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388280 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388289 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.388295 7f00df5e6780 -1 filestore(/ceph/osd.14/) error getting collection index for FORREMOVAL_154_4.bf6: (2) No such file or directory
2013-07-11 12:41:32.437069 7f00df5e6780 0 journal kernel version is 3.9.7
2013-07-11 12:41:33.318490 7f00df5e6780 0 filestore(/ceph/osd.14/) mount FIEMAP ioctl is supported and appears to work
2013-07-11 12:41:33.318500 7f00df5e6780 0 filestore(/ceph/osd.14/) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-07-11 12:41:33.318641 7f00df5e6780 0 filestore(/ceph/osd.14/) mount did NOT detect btrfs
2013-07-11 12:41:33.322477 7f00df5e6780 0 filestore(/ceph/osd.14/) mount syscall(_NR_syncfs, fd) fully supported
2013-07-11 12:41:33.322521 7f00df5e6780 0 filestore(/ceph/osd.14/) mount found snaps <>
2013-07-11 12:41:33.367455 7f00df5e6780 -1 filestore(/ceph/osd.14/) Compacting store...
2013-07-11 12:41:39.029312 7f00df5e6780 -1 filestore(/ceph/osd.14/) ...finished compacting store
2013-07-11 12:41:39.029414 7f00df5e6780 0 filestore(/ceph/osd.14/) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-07-11 12:41:39.043066 7f00df5e6780 0 journal kernel version is 3.9.7
2013-07-11 12:41:39.213055 7f00df5e6780 0 journal kernel version is 3.9.7
2013-07-11 12:41:39.213962 7f00df5e6780 0 osd.14 112506 crush map has features 33816576, adjusting msgr requires for clients
2013-07-11 12:41:39.213972 7f00df5e6780 0 osd.14 112506 crush map has features 33816576, adjusting msgr requires for osds