Ceph

upgraded to latest master (commit 961260d3a) and cosd began automatic upgrade of the filestore, which seemed to complete fine but then failed to start the osds. All the osds refuse to start reporting "unable to read osd superblock".

2011-08-31 10:08:25.769905 7f701f117720 Updating collection 2.e1_head current version is 0
2011-08-31 10:08:25.783932 7f701f117720 collection 2.e1_head updated
2011-08-31 10:08:25.783971 7f701f117720 480/486 processed
2011-08-31 10:08:25.784022 7f701f117720 Updating collection 2.40_head current version is 0
2011-08-31 10:08:25.797404 7f701f117720 collection 2.40_head updated
2011-08-31 10:08:25.797440 7f701f117720 481/486 processed
2011-08-31 10:08:25.797486 7f701f117720 Updating collection 1.ea_head current version is 0
2011-08-31 10:08:25.813655 7f701f117720 collection 1.ea_head updated
2011-08-31 10:08:25.813688 7f701f117720 482/486 processed
2011-08-31 10:08:25.813727 7f701f117720 Updating collection 1.14_head current version is 0
2011-08-31 10:08:25.834735 7f701f117720 collection 1.14_head updated
2011-08-31 10:08:25.834779 7f701f117720 483/486 processed
2011-08-31 10:08:25.834823 7f701f117720 Updating collection 1.9b_head current version is 0
2011-08-31 10:08:25.860830 7f701f117720 collection 1.9b_head updated
2011-08-31 10:08:25.860863 7f701f117720 484/486 processed
2011-08-31 10:08:25.860889 7f701f117720 Updating collection 1.6a_head current version is 0
2011-08-31 10:08:25.879706 7f701f117720 collection 1.6a_head updated
2011-08-31 10:08:25.879737 7f701f117720 485/486 processed
2011-08-31 10:08:25.879773 7f701f117720 Updating collection 0.80_head current version is 0
2011-08-31 10:08:30.019669 7f701f117720 collection 0.80_head updated
2011-08-31 10:08:30.029900 7f701f117720 journal close /data/osd/journal
2011-08-31 10:08:30.077887 7f701f117720 filestore(/data/osd) mount FIEMAP ioctl is supported
2011-08-31 10:08:30.077926 7f701f117720 filestore(/data/osd) mount did NOT detect btrfs
2011-08-31 10:08:30.078420 7f701f117720 filestore(/data/osd) mount found snaps <>
2011-08-31 10:08:30.078453 7f701f117720 filestore(/data/osd) mount: WRITEAHEAD journal mode explicitly enabled in conf
2011-08-31 10:08:30.085258 7f701f117720 journal _open /data/osd/journal fd 16: 536870912 bytes, block size 4096 bytes, directio = 1
2011-08-31 10:08:30.095640 7f701f117720 journal read_entry 77070336 : seq 785930 207906 bytes
2011-08-31 10:08:30.095715 7f701f117720 journal read_entry 77283328 : seq 781861 1923 bytes
2011-08-31 10:08:30.095731 7f701f117720 journal read_entry 77283328 : seq 781861 1923 bytes
2011-08-31 10:08:30.095751 7f701f117720 journal _open /data/osd/journal fd 16: 536870912 bytes, block size 4096 bytes, directio = 1
2011-08-31 10:08:30.096167 7f701f117720 osd1 0 OSD::init() : unable to read osd superblock
2011-08-31 10:08:30.096718 7f701f117720 journal close /data/osd/journal
2011-08-31 10:08:30.098546 7f701f117720  ** ERROR: initializing osd failed: error 1: Operation not permitted

upping the debug filestore level gets:

2011-08-31 10:26:42.757501 7f22ae768720 ceph version .commit: . process: cosd. pid: 19822
2011-08-31 10:26:42.757647 7f22ae768720 filestore(/data/osd) basedir /data/osd journal /data/osd/journal
2011-08-31 10:26:42.766997 7f22ae768720 filestore(/data/osd) mount FIEMAP ioctl is supported
2011-08-31 10:26:42.767046 7f22ae768720 filestore(/data/osd) mount did NOT detect btrfs
2011-08-31 10:26:42.767206 7f22ae768720 filestore(/data/osd) mount fsid is 1461041640
2011-08-31 10:26:42.767280 7f22ae768720 filestore(/data/osd) mount found snaps <>
2011-08-31 10:26:42.767308 7f22ae768720 filestore(/data/osd) mount op_seq is 785930
2011-08-31 10:26:42.767314 7f22ae768720 filestore(/data/osd) open_journal at /data/osd/journal
2011-08-31 10:26:42.767348 7f22ae768720 filestore(/data/osd) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2011-08-31 10:26:42.767366 7f22ae768720 filestore(/data/osd) list_collections
2011-08-31 10:26:42.775765 7f22ae768720 journal _open /data/osd/journal fd 9: 536870912 bytes, block size 4096 bytes, directio = 1
2011-08-31 10:26:42.776001 7f22ae768720 journal read_entry 77070336 : seq 785930 207906 bytes
2011-08-31 10:26:42.776023 7f22ae768720 journal read_entry 77283328 : seq 781861 1923 bytes
2011-08-31 10:26:42.776032 7f22ae768720 journal read_entry 77283328 : seq 781861 1923 bytes
2011-08-31 10:26:42.776045 7f22ae768720 journal _open /data/osd/journal fd 9: 536870912 bytes, block size 4096 bytes, directio = 1
2011-08-31 10:26:42.776374 7f22ab8bf700 filestore(/data/osd) sync_entry waiting for max_interval 5.000000
2011-08-31 10:26:42.776492 7f22ae768720 FileStore is up to date.
2011-08-31 10:26:42.776511 7f22ae768720 filestore(/data/osd) umount /data/osd
2011-08-31 10:26:42.776569 7f22aa0bc700 filestore(/data/osd) flusher_entry start
2011-08-31 10:26:42.776580 7f22aa0bc700 filestore(/data/osd) flusher_entry sleeping
2011-08-31 10:26:42.776607 7f22ab8bf700 filestore(/data/osd) sync_entry force_sync set
2011-08-31 10:26:42.776719 7f22aa0bc700 filestore(/data/osd) flusher_entry awoke
2011-08-31 10:26:42.776729 7f22aa0bc700 filestore(/data/osd) flusher_entry finish
2011-08-31 10:26:42.776850 7f22ae768720 journal close /data/osd/journal
2011-08-31 10:26:42.776906 7f22ae768720 filestore(/data/osd) stop_logger
2011-08-31 10:26:42.778024 7f22ae768720 filestore(/data/osd) test_mount basedir /data/osd journal /data/osd/journal
2011-08-31 10:26:42.778689 7f22ae768720 osd1 0 mounting /data/osd /data/osd/journal
2011-08-31 10:26:42.778726 7f22ae768720 filestore(/data/osd) basedir /data/osd journal /data/osd/journal
2011-08-31 10:26:42.786620 7f22ae768720 filestore(/data/osd) mount FIEMAP ioctl is supported
2011-08-31 10:26:42.786640 7f22ae768720 filestore(/data/osd) mount did NOT detect btrfs
2011-08-31 10:26:42.786677 7f22ae768720 filestore(/data/osd) mount fsid is 1461041640
2011-08-31 10:26:42.786730 7f22ae768720 filestore(/data/osd) mount found snaps <>
2011-08-31 10:26:42.786746 7f22ae768720 filestore(/data/osd) mount op_seq is 785930
2011-08-31 10:26:42.786751 7f22ae768720 filestore(/data/osd) open_journal at /data/osd/journal
2011-08-31 10:26:42.786763 7f22ae768720 filestore(/data/osd) mount: WRITEAHEAD journal mode explicitly enabled in conf
2011-08-31 10:26:42.786772 7f22ae768720 filestore(/data/osd) list_collections
2011-08-31 10:26:42.792436 7f22ae768720 journal _open /data/osd/journal fd 17: 536870912 bytes, block size 4096 bytes, directio = 1
2011-08-31 10:26:42.792578 7f22ae768720 journal read_entry 77070336 : seq 785930 207906 bytes
2011-08-31 10:26:42.792594 7f22ae768720 journal read_entry 77283328 : seq 781861 1923 bytes
2011-08-31 10:26:42.792601 7f22ae768720 journal read_entry 77283328 : seq 781861 1923 bytes
2011-08-31 10:26:42.792614 7f22ae768720 journal _open /data/osd/journal fd 17: 536870912 bytes, block size 4096 bytes, directio = 1
2011-08-31 10:26:42.792888 7f22a60b4700 filestore(/data/osd) sync_entry waiting for max_interval 5.000000
2011-08-31 10:26:42.792984 7f22a48b1700 filestore(/data/osd) flusher_entry start
2011-08-31 10:26:42.792997 7f22a48b1700 filestore(/data/osd) flusher_entry sleeping
2011-08-31 10:26:42.793051 7f22ae768720 osd1 0 boot
2011-08-31 10:26:42.793078 7f22ae768720 filestore(/data/osd) read meta/osd_superblock/0 0~0
2011-08-31 10:26:42.793229 7f22ae768720 filestore(/data/osd) FileStore::read(meta/osd_superblock/0): open error error 2: No such file or directory
2011-08-31 10:26:42.793242 7f22ae768720 osd1 0 OSD::init() : unable to read osd superblock
2011-08-31 10:26:42.793247 7f22ae768720 filestore(/data/osd) umount /data/osd
2011-08-31 10:26:42.793284 7f22a48b1700 filestore(/data/osd) flusher_entry awoke
2011-08-31 10:26:42.793295 7f22a48b1700 filestore(/data/osd) flusher_entry finish
2011-08-31 10:26:42.793327 7f22a60b4700 filestore(/data/osd) sync_entry force_sync set
2011-08-31 10:26:42.793503 7f22ae768720 journal close /data/osd/journal
2011-08-31 10:26:42.793544 7f22ae768720 filestore(/data/osd) stop_logger
2011-08-31 10:26:42.793625 7f22ae768720  ** ERROR: initializing osd failed: error 1: Operation not permitted