Bug #2595
filestore: error creating filestore during mkcephfs
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
from ML:
From: Stefan Priebe - Profihost AG <s.priebe@profihost.ag> === osd.32 === pushing conf and monmap to ssdstor002:/tmp/mkfs.ceph.4885 2012-06-14 14:02:09.333309 7f815cd26780 -1 journal FileJournal::_open: unable to open journal: open() failed: (2) No such file or directory 2012-06-14 14:02:09.690097 7f815cd26780 -1 filestore(/srv/osd.32) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2012-06-14 14:02:09.758456 7f815cd26780 -1 created object store /srv/osd.32 journal /journal/osd.32.journal for osd.32 fsid 308b2ada-5c2a-4b54-b094-b7035413d247 creating private key for osd.32 keyring /etc/ceph/osd.32.keyring creating /etc/ceph/osd.32.keyring collecting osd.32 key
History
#1 Updated by Anonymous over 11 years ago
FYI, I saw this once when I was working on the OSD hotplug code paths. Mine might have been caused by a missing "osd journal size", at the time I didn't want to get sidetracked and my memory of the event is already fuzzy.
#2 Updated by Stefan Priebe over 11 years ago
Can you reproduce with 'debug filestore = 20' and attach the log to this
bug?
Log:
2012-06-15 22:51:13.217367 7fe0a4c1b780 1 filestore(/srv/osd.10) mkfs in /srv/osd.10 2012-06-15 22:51:13.217434 7fe0a4c1b780 1 filestore(/srv/osd.10) mkfs generated fsid c2505fd1-de2e-407c-a3bf-692b98e224b4 2012-06-15 22:51:13.220021 7fe0a4c1b780 10 filestore(/srv/osd.10) mkfs fsid is c2505fd1-de2e-407c-a3bf-692b98e224b4 2012-06-15 22:51:13.220091 7fe0a4c1b780 2 filestore(/srv/osd.10) BTRFS_IOC_SUBVOL_CREATE ioctl failed, trying mkdir /srv/osd.10/current 2012-06-15 22:51:13.223389 7fe0a4c1b780 1 filestore(/srv/osd.10) leveldb db exists/created 2012-06-15 22:51:13.223458 7fe0a4c1b780 10 filestore(/srv/osd.10) open_journal at /journal/osd.10.journal 2012-06-15 22:51:13.224448 7fe0a4c1b780 -1 journal FileJournal::_open: unable to open journal: open() failed: (2) No such file or directory 2012-06-15 22:51:13.791658 7fe0a4c1b780 1 journal _open /journal/osd.10.journal fd 9: 2097152000 bytes, block size 4096 bytes, directio = 0, aio = 0 2012-06-15 22:51:13.791716 7fe0a4c1b780 0 filestore(/srv/osd.10) mkjournal created journal on /journal/osd.10.journal 2012-06-15 22:51:13.791729 7fe0a4c1b780 1 filestore(/srv/osd.10) mkfs done in /srv/osd.10 2012-06-15 22:51:13.791737 7fe0a4c1b780 5 filestore(/srv/osd.10) basedir /srv/osd.10 journal /journal/osd.10.journal 2012-06-15 22:51:13.791754 7fe0a4c1b780 10 filestore(/srv/osd.10) mount fsid is c2505fd1-de2e-407c-a3bf-692b98e224b4 2012-06-15 22:51:13.822836 7fe0a4c1b780 0 filestore(/srv/osd.10) mount FIEMAP ioctl is supported and appears to work 2012-06-15 22:51:13.822844 7fe0a4c1b780 0 filestore(/srv/osd.10) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2012-06-15 22:51:13.823005 7fe0a4c1b780 0 filestore(/srv/osd.10) mount did NOT detect btrfs 2012-06-15 22:51:13.863831 7fe0a4c1b780 0 filestore(/srv/osd.10) mount syncfs(2) syscall fully supported (by glibc and kernel) 2012-06-15 22:51:13.863908 7fe0a4c1b780 0 filestore(/srv/osd.10) mount found snaps <> 2012-06-15 22:51:13.863927 7fe0a4c1b780 5 filestore(/srv/osd.10) mount op_seq is 1 2012-06-15 22:51:13.865884 7fe0a4c1b780 20 filestore (init)dbobjectmap: seq is 1 2012-06-15 22:51:13.865892 7fe0a4c1b780 10 filestore(/srv/osd.10) open_journal at /journal/osd.10.journal 2012-06-15 22:51:13.865900 7fe0a4c1b780 0 filestore(/srv/osd.10) mount: enabling WRITEAHEAD journal mode: btrfs not detected 2012-06-15 22:51:13.865904 7fe0a4c1b780 10 filestore(/srv/osd.10) list_collections 2012-06-15 22:51:13.865954 7fe0a4c1b780 1 journal _open /journal/osd.10.journal fd 17: 2097152000 bytes, block size 4096 bytes, directio = 0, aio = 0 2012-06-15 22:51:13.865957 7fe0a0995700 20 filestore(/srv/osd.10) sync_entry waiting for max_interval 5.000000 2012-06-15 22:51:13.865979 7fe0a4c1b780 1 journal _open /journal/osd.10.journal fd 17: 2097152000 bytes, block size 4096 bytes, directio = 0, aio = 0 2012-06-15 22:51:13.866053 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry start 2012-06-15 22:51:13.866058 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry sleeping 2012-06-15 22:51:13.866091 7fe0a4c1b780 15 filestore(/srv/osd.10) read meta/23c2fcde/osd_superblock/0//-1 0~0 2012-06-15 22:51:13.866106 7fe0a4c1b780 -1 filestore(/srv/osd.10) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory 2012-06-15 22:51:13.866114 7fe0a4c1b780 10 filestore(/srv/osd.10) FileStore::read(meta/23c2fcde/osd_superblock/0//-1) open error: (2) No such file or directory 2012-06-15 22:51:13.866126 7fe0a4c1b780 10 filestore(/srv/osd.10) apply queued 2012-06-15 22:51:13.866129 7fe0a4c1b780 5 filestore(/srv/osd.10) queue_transactions new osr(default 0x2654678)/0x2654678 2012-06-15 22:51:13.866137 7fe0a4c1b780 5 filestore(/srv/osd.10) queue_transactions (writeahead) 2 0x7fff7de89ba0 2012-06-15 22:51:13.866229 7fe09f993700 5 filestore(/srv/osd.10) _journaled_ahead 0x263d190 seq 2 osr(default 0x2654678) 0x7fff7de89ba0 2012-06-15 22:51:13.866260 7fe09f993700 5 filestore(/srv/osd.10) queue_op 0x263d190 seq 2 osr(default 0x2654678) 238 bytes (queue has 1 ops and 238 bytes) 2012-06-15 22:51:13.866267 7fe09f192700 5 filestore(/srv/osd.10) _do_op 0x263d190 seq 2 osr(default 0x2654678)/0x2654678 start 2012-06-15 22:51:13.866273 7fe09f192700 10 filestore(/srv/osd.10) _do_transaction on 0x7fff7de89ba0 2012-06-15 22:51:13.866279 7fe09f192700 15 filestore(/srv/osd.10) create_collection /srv/osd.10/current/meta 2012-06-15 22:51:13.866306 7fe09f192700 10 filestore(/srv/osd.10) create_collection /srv/osd.10/current/meta = 0 2012-06-15 22:51:13.866388 7fe09f192700 15 filestore(/srv/osd.10) write meta/23c2fcde/osd_superblock/0//-1 0~106 2012-06-15 22:51:13.866487 7fe09f192700 10 filestore(/srv/osd.10) queue_flusher ep 0 fd 18 0~106 qlen 1 2012-06-15 22:51:13.866511 7fe09f192700 10 filestore(/srv/osd.10) write meta/23c2fcde/osd_superblock/0//-1 0~106 = 106 2012-06-15 22:51:13.866522 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry awoke 2012-06-15 22:51:13.866527 7fe09f192700 10 filestore(/srv/osd.10) _do_op 0x263d190 seq 2 r = 0, finisher 0x264fe10 0 2012-06-15 22:51:13.866533 7fe09e190700 10 filestore(/srv/osd.10) flusher_entry flushing+closing 18 ep 0 2012-06-15 22:51:13.866534 7fe09f192700 10 filestore(/srv/osd.10) _finish_op 0x263d190 seq 2 osr(default 0x2654678)/0x2654678 2012-06-15 22:51:13.866591 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry sleeping 2012-06-15 22:51:13.866596 7fe0a4c1b780 10 filestore(/srv/osd.10) apply done r = 0 2012-06-15 22:51:13.866598 7fe0a4c1b780 10 filestore(/srv/osd.10) sync_and_flush 2012-06-15 22:51:13.866600 7fe0a4c1b780 10 filestore(/srv/osd.10) _flush_op_queue draining op tp 2012-06-15 22:51:13.866601 7fe0a4c1b780 10 filestore(/srv/osd.10) _flush_op_queue waiting for apply finisher 2012-06-15 22:51:13.866602 7fe0a4c1b780 10 filestore(/srv/osd.10) sync_and_flush done 2012-06-15 22:51:13.932680 7fe0a4c1b780 5 filestore(/srv/osd.10) umount /srv/osd.10 2012-06-15 22:51:13.932717 7fe0a0995700 20 filestore(/srv/osd.10) sync_entry force_sync set 2012-06-15 22:51:13.932739 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry awoke 2012-06-15 22:51:13.932746 7fe09e190700 20 filestore(/srv/osd.10) flusher_entry finish 2012-06-15 22:51:13.932752 7fe0a0995700 15 filestore(/srv/osd.10) sync_entry committing 2 sync_epoch 1 2012-06-15 22:51:13.932778 7fe0a0995700 15 filestore(/srv/osd.10) sync_entry doing a full sync (syncfs(2) if possible) 2012-06-15 22:51:13.933625 7fe0a0995700 10 filestore(/srv/osd.10) sync_entry commit took 0.000887, interval was 0.067667 2012-06-15 22:51:13.933653 7fe0a0995700 15 filestore(/srv/osd.10) sync_entry committed to op_seq 2 2012-06-15 22:51:13.933797 7fe0a4c1b780 1 journal close /journal/osd.10.journal 2012-06-15 22:51:13.934117 7fe0a4c1b780 -1 created object store /srv/osd.10 journal /journal/osd.10.journal for osd.10 fsid 72571270-b891-4dca-ad4d-19e4996dc509
#3 Updated by Sage Weil over 11 years ago
Oh, it looks like it's just noise from checking the journal. The mkcephfs succeeded, right?
#4 Updated by Stefan Priebe over 11 years ago
Yes
#5 Updated by Sage Weil over 11 years ago
#6 Updated by Sage Weil over 11 years ago
- Status changed from 12 to Resolved
#7 Updated by Stefan Priebe over 11 years ago
2012-06-18 17:42:16.232924 7f54292fb780 -1 filestore(/srv/osd.20) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory