Ceph

Getting a crash on one OSD when it tries to start up after upgrading to 0.38.

Here is the log of start up to crash.

starting osd.11 at 10.10.20.12:6801/9443 osd_data /srv/osd11 /dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR112407B1080BGN-part2
2011-11-15 14:56:56.720731 7f7047d60760 -- 192.168.91.12:6802/9443 accepter.bind ms_addr is 192.168.91.12:6802/9443 need_addr=0
2011-11-15 14:56:56.721365 7f7047d60760 filestore(/srv/osd11) basedir /srv/osd11 journal /dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR112407B1080BGN-part2
2011-11-15 14:56:56.721572 7f7047d60760 filestore(/srv/osd11) mount FIEMAP ioctl is NOT supported
2011-11-15 14:56:56.721603 7f7047d60760 filestore(/srv/osd11) mount detected btrfs
2011-11-15 14:56:56.721615 7f7047d60760 filestore(/srv/osd11) _do_clone_range 0~1 to 0
2011-11-15 14:56:56.721623 7f7047d60760 filestore(/srv/osd11) mount btrfs CLONE_RANGE ioctl is supported
2011-11-15 14:56:56.906320 7f7047d60760 filestore(/srv/osd11) mount btrfs SNAP_CREATE is supported
2011-11-15 14:56:56.952538 7f7047d60760 filestore(/srv/osd11) mount btrfs SNAP_DESTROY is supported
2011-11-15 14:56:56.956104 7f7047d60760 filestore(/srv/osd11) mount btrfs START_SYNC got 0 Success
2011-11-15 14:56:56.956140 7f7047d60760 filestore(/srv/osd11) mount btrfs START_SYNC is supported (transid 121346)
2011-11-15 14:56:57.239715 7f7047d60760 filestore(/srv/osd11) mount btrfs WAIT_SYNC is supported
2011-11-15 14:56:57.260929 7f7047d60760 filestore(/srv/osd11) mount btrfs SNAP_CREATE_V2 got 0 Success
2011-11-15 14:56:57.260944 7f7047d60760 filestore(/srv/osd11) mount btrfs SNAP_CREATE_V2 is supported
2011-11-15 14:56:57.264809 7f7047d60760 filestore(/srv/osd11) mount fsid is 1302018284
2011-11-15 14:56:57.264855 7f7047d60760 filestore(/srv/osd11) mount found snaps <9537365,9537376>
2011-11-15 14:56:57.264874 7f7047d60760 filestore(/srv/osd11)  current/ seq was 9537376
2011-11-15 14:56:57.264880 7f7047d60760 filestore(/srv/osd11)  most recent snap from <9537365,9537376> is 9537376
2011-11-15 14:56:57.264884 7f7047d60760 filestore(/srv/osd11) mount rolling back to consistent snap 9537376
2011-11-15 14:56:57.748062 7f7047d60760 filestore(/srv/osd11) mount op_seq is 9537376
2011-11-15 14:56:57.748100 7f7047d60760 filestore(/srv/osd11) open_journal at /dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR112407B1080BGN-part2
2011-11-15 14:56:57.748129 7f7047d60760 filestore(/srv/osd11) mount: enabling PARALLEL journal mode: btrfs, SNAP_CREATE_V2 detected and 'filestore btrfs snap' mode is enabled
2011-11-15 14:56:57.748139 7f7047d60760 filestore(/srv/osd11) list_collections
2011-11-15 14:56:57.761500 7f7047d60760 journal journal_replay fs op_seq 9537376
2011-11-15 14:56:57.764918 7f7047d60760 journal _open /dev/disk/by-id/ata-INTEL_SSDSA2CW080G3_CVPR112407B1080BGN-part2 fd 14: 4095737856 bytes, block size 4096 bytes, directio = 1
2011-11-15 14:56:57.764925 7f7047d60760 journal read_header
2011-11-15 14:56:57.765368 7f7047d60760 journal header: block_size 4096 alignment 4096 max_size 4095737856
2011-11-15 14:56:57.765388 7f7047d60760 journal header: start 250503168
2011-11-15 14:56:57.765396 7f7047d60760 journal  write_pos 4096
2011-11-15 14:56:57.765405 7f7047d60760 journal open header.fsid = 1302018284
2011-11-15 14:56:57.766197 7f7047d60760 journal read_entry 250503168 : seq 9537377 7225 bytes
2011-11-15 14:56:57.766226 7f7047d60760 journal open reached seq 9537377
2011-11-15 14:56:57.766257 7f7047d60760 journal read_entry 250503168 : seq 9537377 7225 bytes
2011-11-15 14:56:57.766267 7f7047d60760 journal journal_replay: applying op seq 9537377 (op_seq 9537376)
2011-11-15 14:56:57.766290 7f7047d60760 filestore(/srv/osd11) _do_transaction on 0x27bb000
2011-11-15 14:56:57.766313 7f7047d60760 filestore(/srv/osd11) write 7.7_head/rb.0.0.000000000329/head 696320~4096
2011-11-15 14:56:57.766522 7f7047d60760 filestore(/srv/osd11) queue_flusher ep 0 fd 15 696320~4096 qlen 1
2011-11-15 14:56:57.766536 7f7047d60760 filestore(/srv/osd11) write 7.7_head/rb.0.0.000000000329/head 696320~4096 = 4096
2011-11-15 14:56:57.766548 7f7047d60760 filestore(/srv/osd11) setattr 7.7_head/rb.0.0.000000000329/head '_' len 179
2011-11-15 14:56:57.806983 7f7047d60760 filestore(/srv/osd11) setattr 7.7_head/rb.0.0.000000000329/head '_' len 179 = 179
2011-11-15 14:56:57.807011 7f7047d60760 filestore(/srv/osd11) setattr 7.7_head/rb.0.0.000000000329/head 'snapset' len 1926
2011-11-15 14:56:57.807060 7f7047d60760 filestore(/srv/osd11) setattr 7.7_head/rb.0.0.000000000329/head 'snapset' len 1926 = -75
2011-11-15 14:56:57.807081 7f7047d60760 filestore(/srv/osd11)  error error 75: Value too large for defined data type not handled
os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&)', in thread '7f7047d60760'
os/FileStore.cc: 2426: FAILED assert(0 == "unexpected error")
 ceph version 0.38 (commit:b600ec2ac7c0f2e508720f8e8bb87c3db15509b9)
 1: (FileStore::_do_transaction(ObjectStore::Transaction&)+0x1f88) [0x6adff8]
 2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x76) [0x6afd06]
 3: (JournalingObjectStore::journal_replay(unsigned long)+0x938) [0x6bae38]
 4: (FileStore::mount()+0x294b) [0x6a71db]
 5: /usr/bin/ceph-osd() [0x51c7ca]
 6: (OSD::convertfs(std::string const&, std::string const&)+0xbb) [0x51d73b]
 7: (main()+0x2213) [0x4ad993]
 8: (__libc_start_main()+0xed) [0x7f70460ca30d]
 9: /usr/bin/ceph-osd() [0x4af45d]
 ceph version 0.38 (commit:b600ec2ac7c0f2e508720f8e8bb87c3db15509b9)
 1: (FileStore::_do_transaction(ObjectStore::Transaction&)+0x1f88) [0x6adff8]
 2: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x76) [0x6afd06]
 3: (JournalingObjectStore::journal_replay(unsigned long)+0x938) [0x6bae38]
 4: (FileStore::mount()+0x294b) [0x6a71db]
 5: /usr/bin/ceph-osd() [0x51c7ca]
 6: (OSD::convertfs(std::string const&, std::string const&)+0xbb) [0x51d73b]
 7: (main()+0x2213) [0x4ad993]
 8: (__libc_start_main()+0xed) [0x7f70460ca30d]
 9: /usr/bin/ceph-osd() [0x4af45d]
*** Caught signal (Aborted) **
 in thread 7f7047d60760
 ceph version 0.38 (commit:b600ec2ac7c0f2e508720f8e8bb87c3db15509b9)
 1: /usr/bin/ceph-osd() [0x5ee736]
 2: (()+0x10060) [0x7f704795d060]
 3: (gsignal()+0x35) [0x7f70460df3a5]
 4: (abort()+0x17b) [0x7f70460e2b0b]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f704699dd7d]
 6: (()+0xb9f26) [0x7f704699bf26]
 7: (()+0xb9f53) [0x7f704699bf53]
 8: (()+0xba04e) [0x7f704699c04e]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x193) [0x5c30c3]
 10: (FileStore::_do_transaction(ObjectStore::Transaction&)+0x1f88) [0x6adff8]
 11: (FileStore::do_transactions(std::list<ObjectStore::Transaction*, std::allocator<ObjectStore::Transaction*> >&, unsigned long)+0x76) [0x6afd06]
 12: (JournalingObjectStore::journal_replay(unsigned long)+0x938) [0x6bae38]
 13: (FileStore::mount()+0x294b) [0x6a71db]
 14: /usr/bin/ceph-osd() [0x51c7ca]
 15: (OSD::convertfs(std::string const&, std::string const&)+0xbb) [0x51d73b]
 16: (main()+0x2213) [0x4ad993]
 17: (__libc_start_main()+0xed) [0x7f70460ca30d]
 18: /usr/bin/ceph-osd() [0x4af45d]