Project

General

Profile

Bug #9073 » ceph-osd.0.log

Mark Kirkwood, 08/17/2014 11:40 PM

 
2014-08-18 18:14:12.103485 7f233f154800 0 ceph version 0.83-428-g0279503 (02795034ea5b03e2c38dcb6f21de33f9949f9624), process ceph-osd, pid 6071
2014-08-18 18:14:12.103716 7f233f154800 10 filestore(/ceph0) dump_stop
2014-08-18 18:14:12.103842 7f233f154800 10 monclient(hunting): build_initial_monmap
2014-08-18 18:14:12.103874 7f233f154800 10 monclient(hunting): get_monmap_privately
2014-08-18 18:14:12.103985 7f233f154800 10 monclient(hunting): have 0 fsid 390c6db4-7737-40e4-9c30-837e95db7055
2014-08-18 18:14:12.104128 7f233f154800 1 filestore(/ceph0) mkfs in /ceph0
2014-08-18 18:14:12.104244 7f233f154800 1 filestore(/ceph0) mkfs generated fsid a77fd6da-4c53-441c-b248-ccb59f712fba
2014-08-18 18:14:12.121583 7f233f154800 10 filestore(/ceph0) mkfs fsid is a77fd6da-4c53-441c-b248-ccb59f712fba
2014-08-18 18:14:12.182272 7f233f154800 0 filestore(/ceph0) backend generic (magic 0xef53)
2014-08-18 18:14:12.249308 7f233f154800 1 filestore(/ceph0) leveldb db exists/created
2014-08-18 18:14:12.249343 7f233f154800 10 filestore(/ceph0) open_journal at /dev/sdc1
2014-08-18 18:14:12.249397 7f233f154800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 15032385536)
2014-08-18 18:14:12.262601 7f233f154800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sdc1)
2014-08-18 18:14:12.262820 7f233f154800 1 journal _open /dev/sdc1 fd 10: 15032385536 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-18 18:14:12.262850 7f233f154800 10 journal read_header
2014-08-18 18:14:12.263106 7f233f154800 10 journal read_headercount=0x2b7b000 ,0x=0
2014-08-18 18:14:12.263243 7f233f154800 -1 journal read_header error decoding journal header
2014-08-18 18:14:12.263328 7f233f154800 2 journal create /dev/sdc1 fsid a77fd6da-4c53-441c-b248-ccb59f712fba
2014-08-18 18:14:12.263427 7f233f154800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 15032385536)
2014-08-18 18:14:12.272568 7f233f154800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sdc1)
2014-08-18 18:14:12.272793 7f233f154800 1 journal _open /dev/sdc1 fd 10: 15032385536 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-18 18:14:12.272819 7f233f154800 10 journal header: block_size 4096 alignment 4096 max_size 15032385536
2014-08-18 18:14:12.272822 7f233f154800 10 journal header: start 4096
2014-08-18 18:14:12.272824 7f233f154800 10 journal write_pos 0
2014-08-18 18:14:12.272841 7f233f154800 10 journal prepare_header ptr= 0x237b000
2014-08-18 18:14:12.272843 7f233f154800 10 journal prepare_header content=0x4 ,0x=40
2014-08-18 18:14:12.273635 7f233f154800 2 journal create done
2014-08-18 18:14:12.273674 7f233f154800 0 filestore(/ceph0) mkjournal created journal on /dev/sdc1
2014-08-18 18:14:12.273707 7f233f154800 1 filestore(/ceph0) mkfs done in /ceph0
2014-08-18 18:14:12.273744 7f233f154800 5 filestore(/ceph0) basedir /ceph0 journal /dev/sdc1
2014-08-18 18:14:12.273797 7f233f154800 10 filestore(/ceph0) mount fsid is a77fd6da-4c53-441c-b248-ccb59f712fba
2014-08-18 18:14:12.273874 7f233f154800 0 filestore(/ceph0) backend generic (magic 0xef53)
2014-08-18 18:14:12.303736 7f233f154800 0 genericfilestorebackend(/ceph0) detect_features: FIEMAP ioctl is supported and appears to work
2014-08-18 18:14:12.303755 7f233f154800 0 genericfilestorebackend(/ceph0) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-08-18 18:14:12.334048 7f233f154800 0 genericfilestorebackend(/ceph0) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-08-18 18:14:12.400293 7f233f154800 0 filestore(/ceph0) limited size xattrs
2014-08-18 18:14:12.400426 7f233f154800 5 filestore(/ceph0) mount op_seq is 1
2014-08-18 18:14:12.450448 7f233f154800 20 filestore (init)dbobjectmap: seq is 1
2014-08-18 18:14:12.450475 7f233f154800 10 filestore(/ceph0) open_journal at /dev/sdc1
2014-08-18 18:14:12.450498 7f233f154800 0 filestore(/ceph0) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-08-18 18:14:12.450507 7f233f154800 10 filestore(/ceph0) list_collections
2014-08-18 18:14:12.450602 7f233f154800 10 journal journal_replay fs op_seq 1
2014-08-18 18:14:12.450614 7f233f154800 2 journal open /dev/sdc1 fsid a77fd6da-4c53-441c-b248-ccb59f712fba fs_op_seq 1
2014-08-18 18:14:12.450635 7f233f154800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 15032385536)
2014-08-18 18:14:12.450622 7f233ab2e700 20 filestore(/ceph0) sync_entry waiting for max_interval 5.000000
2014-08-18 18:14:12.454214 7f233f154800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sdc1)
2014-08-18 18:14:12.454459 7f233f154800 1 journal _open /dev/sdc1 fd 16: 15032385536 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-18 18:14:12.454479 7f233f154800 10 journal read_header
2014-08-18 18:14:12.454678 7f233f154800 10 journal read_headercount=0x4 ,0x=40
2014-08-18 18:14:12.454697 7f233f154800 10 journal header: block_size 4096 alignment 4096 max_size 15032385536
2014-08-18 18:14:12.454702 7f233f154800 10 journal header: start 4096
2014-08-18 18:14:12.454705 7f233f154800 10 journal write_pos 4096
2014-08-18 18:14:12.454709 7f233f154800 10 journal open header.fsid = a77fd6da-4c53-441c-b248-ccb59f712fba
2014-08-18 18:14:12.454813 7f233f154800 2 journal No further valid entries found, journal is most likely valid
2014-08-18 18:14:12.454817 7f233f154800 10 journal open reached end of journal.
2014-08-18 18:14:12.454832 7f233f154800 2 journal No further valid entries found, journal is most likely valid
2014-08-18 18:14:12.454840 7f233f154800 3 journal journal_replay: end of journal, done.
2014-08-18 18:14:12.454844 7f233f154800 10 journal make_writeable
2014-08-18 18:14:12.454982 7f233f154800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 15032385536)
2014-08-18 18:14:12.464204 7f233f154800 20 journal _check_disk_write_cache: disk write cache is on, but your kernel is new enough to handle it correctly. (fn:/dev/sdc1)
2014-08-18 18:14:12.464412 7f233f154800 1 journal _open /dev/sdc1 fd 16: 15032385536 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-08-18 18:14:12.464508 7f233f154800 10 journal journal_start
2014-08-18 18:14:12.464513 7f233a32d700 10 journal write_thread_entry start
2014-08-18 18:14:12.464534 7f233a32d700 20 journal prepare_multi_write queue_pos now 4096
2014-08-18 18:14:12.464539 7f233932b700 10 journal write_finish_thread_entry enter
2014-08-18 18:14:12.464557 7f233932b700 20 journal write_finish_thread_entry sleeping
2014-08-18 18:14:12.464593 7f233a32d700 10 journal prepare_header ptr= 0x23ab000
2014-08-18 18:14:12.464603 7f233a32d700 10 journal prepare_header content=0x4 ,0x=40
2014-08-18 18:14:12.464608 7f233a32d700 15 journal do_aio_write writing 4096~0 + header
2014-08-18 18:14:12.464615 7f233a32d700 20 journal write_aio_bl 0~4096 seq 0
2014-08-18 18:14:12.464617 7f233a32d700 10 journal write_aio_bl ptr= 0x23ab000
2014-08-18 18:14:12.464619 7f233a32d700 10 journal write_aio_bl content=0x4 ,0x=40
2014-08-18 18:14:12.464629 7f233a32d700 20 journal write_aio_bl .. 0~4096 in 1
2014-08-18 18:14:12.464632 7f233a32d700 10 journal write_aio_bl ptr= 0x23ab000
2014-08-18 18:14:12.464633 7f233a32d700 10 journal write_aio_bl content=0x4 ,0x=40
2014-08-18 18:14:12.464693 7f233a32d700 20 journal write_aio_bl 4096~0 seq 0
2014-08-18 18:14:12.464712 7f233932b700 20 journal write_finish_thread_entry waiting for aio(s)
2014-08-18 18:14:12.464711 7f233a32d700 5 journal put_throttle finished 0 ops and 0 bytes, now 0 ops and 0 bytes
2014-08-18 18:14:12.464722 7f233a32d700 20 journal write_thread_entry going to sleep
2014-08-18 18:14:12.464750 7f233f154800 15 filestore(/ceph0) read meta/23c2fcde/osd_superblock/0//-1 0~0
2014-08-18 18:14:12.464839 7f233932b700 10 journal write_finish_thread_entry aio 0~4096 done
2014-08-18 18:14:12.464848 7f233932b700 10 journal write_finish_thread_entry ptr= 0x23ab000
2014-08-18 18:14:12.464849 7f233932b700 10 journal write_finish_thread_entry content=0x4 ,0x=40
2014-08-18 18:14:12.464829 7f233f154800 -1 filestore(/ceph0) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2014-08-18 18:14:12.464858 7f233932b700 20 journal check_aio_completion
2014-08-18 18:14:12.464861 7f233932b700 20 journal check_aio_completion completed seq 0 0~4096
2014-08-18 18:14:12.464859 7f233f154800 10 filestore(/ceph0) FileStore::read(meta/23c2fcde/osd_superblock/0//-1) open error: (2) No such file or directory
2014-08-18 18:14:12.464865 7f233932b700 20 journal write_finish_thread_entry sleeping
2014-08-18 18:14:12.464906 7f233f154800 5 filestore(/ceph0) queue_transactions new osr(default 0x2362b90)/0x2362b90
2014-08-18 18:14:12.464922 7f233f154800 10 journal op_submit_start 2
2014-08-18 18:14:12.464924 7f233f154800 5 filestore(/ceph0) queue_transactions (writeahead) 2 0x7fff33fe5b00
2014-08-18 18:14:12.464928 7f233f154800 10 journal op_journal_transactions 2 0x7fff33fe5b00
2014-08-18 18:14:12.464933 7f233f154800 5 journal submit_entry seq 2 len 505 (0x2323540)
2014-08-18 18:14:12.464943 7f233f154800 10 journal op_submit_finish 2
2014-08-18 18:14:12.464980 7f233a32d700 20 journal write_thread_entry woke up
2014-08-18 18:14:12.464999 7f233a32d700 10 journal room 15032381439 max_size 15032385536 pos 4096 header.start 4096 top 4096
2014-08-18 18:14:12.465005 7f233a32d700 10 journal check_for_full at 4096 : 4096 < 15032381439
2014-08-18 18:14:12.465006 7f233a32d700 15 journal prepare_single_write 1 will write 4096 : seq 2 len 505 -> 4096 (head 40 pre_pad 0 ebl 505 post_pad 3511 tail 40) (ebl alignment -1)
2014-08-18 18:14:12.465028 7f233a32d700 20 journal prepare_multi_write queue_pos now 8192
2014-08-18 18:14:12.465031 7f233a32d700 15 journal do_aio_write writing 4096~4096
2014-08-18 18:14:12.465039 7f233a32d700 20 journal write_aio_bl 4096~4096 seq 2
2014-08-18 18:14:12.465042 7f233a32d700 10 journal write_aio_bl ptr= 0x23ac000
2014-08-18 18:14:12.465044 7f233a32d700 10 journal write_aio_bl content=0x2 ,0x=0
2014-08-18 18:14:12.465046 7f233a32d700 20 journal write_aio_bl .. 4096~4096 in 1
2014-08-18 18:14:12.465048 7f233a32d700 10 journal write_aio_bl ptr= 0x23ac000
2014-08-18 18:14:12.465049 7f233a32d700 10 journal write_aio_bl content=0x2 ,0x=0
2014-08-18 18:14:12.465081 7f233a32d700 5 journal put_throttle finished 1 ops and 505 bytes, now 0 ops and 0 bytes
2014-08-18 18:14:12.465086 7f233a32d700 20 journal write_thread_entry going to sleep
2014-08-18 18:14:12.465094 7f233932b700 20 journal write_finish_thread_entry waiting for aio(s)
2014-08-18 18:14:12.465242 7f233932b700 10 journal write_finish_thread_entry aio 4096~4096 done
2014-08-18 18:14:12.465250 7f233932b700 10 journal write_finish_thread_entry ptr= 0x23ac000
2014-08-18 18:14:12.465252 7f233932b700 10 journal write_finish_thread_entry content=0x2 ,0x=0
2014-08-18 18:14:12.465255 7f233932b700 20 journal check_aio_completion
2014-08-18 18:14:12.465256 7f233932b700 20 journal check_aio_completion completed seq 2 4096~4096
2014-08-18 18:14:12.465261 7f233932b700 20 journal check_aio_completion queueing finishers through seq 2
2014-08-18 18:14:12.465270 7f233932b700 10 journal queue_completions_thru seq 2 queueing seq 2 0x2323540 lat 0.000332
2014-08-18 18:14:12.465290 7f233932b700 20 journal write_finish_thread_entry sleeping
2014-08-18 18:14:12.465302 7f2338b2a700 5 filestore(/ceph0) _journaled_ahead 0x23246e0 seq 2 osr(default 0x2362b90) 0x7fff33fe5b00
2014-08-18 18:14:12.465316 7f2338b2a700 5 filestore(/ceph0) queue_op 0x23246e0 seq 2 osr(default 0x2362b90) 499 bytes (queue has 1 ops and 499 bytes)
2014-08-18 18:14:12.465376 7f2337b28700 10 journal op_apply_start 2 open_ops 0 -> 1
2014-08-18 18:14:12.465387 7f2337b28700 5 filestore(/ceph0) _do_op 0x23246e0 seq 2 osr(default 0x2362b90)/0x2362b90 start
2014-08-18 18:14:12.465394 7f2337b28700 10 filestore(/ceph0) _do_transaction on 0x7fff33fe5b00
2014-08-18 18:14:12.465420 7f2337b28700 15 filestore(/ceph0) create_collection /ceph0/current/meta
2014-08-18 18:14:12.465482 7f2337b28700 10 filestore(/ceph0) create_collection /ceph0/current/meta = 0
2014-08-18 18:14:12.465559 7f2337b28700 10 filestore(/ceph0) _set_replay_guard 2.0.0
2014-08-18 18:14:12.474554 7f2337b28700 20 filestore dbobjectmap: seq is 1
2014-08-18 18:14:12.504819 7f2337b28700 10 filestore(/ceph0) _set_replay_guard 2.0.0 done
2014-08-18 18:14:12.504867 7f2337b28700 15 filestore(/ceph0) write meta/23c2fcde/osd_superblock/0//-1 0~358
2014-08-18 18:14:12.505073 7f2337b28700 10 filestore(/ceph0) write meta/23c2fcde/osd_superblock/0//-1 0~358 = 358
2014-08-18 18:14:12.505086 7f2337b28700 10 journal op_apply_finish 2 open_ops 1 -> 0, max_applied_seq 1 -> 2
2014-08-18 18:14:12.505089 7f2337b28700 10 filestore(/ceph0) _do_op 0x23246e0 seq 2 r = 0, finisher 0x23234e0 0
2014-08-18 18:14:12.505093 7f2337b28700 10 filestore(/ceph0) _finish_op 0x23246e0 seq 2 osr(default 0x2362b90)/0x2362b90
2014-08-18 18:14:12.505169 7f233f154800 10 filestore(/ceph0) sync_and_flush
2014-08-18 18:14:12.505177 7f233f154800 10 journal waiting for completions to empty
2014-08-18 18:14:12.505179 7f233f154800 10 journal flush waiting for finisher
2014-08-18 18:14:12.505181 7f233f154800 10 journal flush done
2014-08-18 18:14:12.505182 7f233f154800 10 filestore(/ceph0) _flush_op_queue draining op tp
2014-08-18 18:14:12.505187 7f233f154800 10 filestore(/ceph0) _flush_op_queue waiting for apply finisher
2014-08-18 18:14:12.505189 7f233f154800 10 filestore(/ceph0) sync_and_flush done
2014-08-18 18:14:12.626307 7f233f154800 5 filestore(/ceph0) umount /ceph0
2014-08-18 18:14:12.626391 7f233ab2e700 20 filestore(/ceph0) sync_entry force_sync set
2014-08-18 18:14:12.626415 7f233ab2e700 10 journal commit_start max_applied_seq 2, open_ops 0
2014-08-18 18:14:12.626419 7f233ab2e700 10 journal commit_start blocked, all open_ops have completed
2014-08-18 18:14:12.626421 7f233ab2e700 10 journal commit_start committing 2, still blocked
2014-08-18 18:14:12.626423 7f233ab2e700 10 journal commit_start
2014-08-18 18:14:12.626437 7f233ab2e700 15 filestore(/ceph0) sync_entry committing 2 sync_epoch 1
2014-08-18 18:14:12.626443 7f233ab2e700 10 journal commit_started committing 2, unblocking
2014-08-18 18:14:12.626449 7f233ab2e700 15 genericfilestorebackend(/ceph0) syncfs: doing a full sync (syncfs(2) if possible)
2014-08-18 18:14:12.686547 7f233ab2e700 10 filestore(/ceph0) sync_entry commit took 0.060123, interval was 0.235925
2014-08-18 18:14:12.686558 7f233ab2e700 10 journal commit_finish thru 2
2014-08-18 18:14:12.686560 7f233ab2e700 5 journal committed_thru 2 (last_committed_seq 1)
2014-08-18 18:14:12.686562 7f233ab2e700 10 journal header: block_size 4096 alignment 4096 max_size 15032385536
2014-08-18 18:14:12.686563 7f233ab2e700 10 journal header: start 8192
2014-08-18 18:14:12.686563 7f233ab2e700 10 journal write_pos 8192
2014-08-18 18:14:12.686564 7f233ab2e700 10 journal committed_thru done
2014-08-18 18:14:12.686569 7f233ab2e700 15 filestore(/ceph0) sync_entry committed to op_seq 2
2014-08-18 18:14:12.686910 7f233f154800 10 journal journal_stop
2014-08-18 18:14:12.687048 7f233f154800 1 journal close /dev/sdc1
2014-08-18 18:14:12.687108 7f233a32d700 20 journal write_thread_entry woke up
2014-08-18 18:14:12.687111 7f233932b700 10 journal write_finish_thread_entry exit
2014-08-18 18:14:12.687120 7f233a32d700 20 journal prepare_multi_write queue_pos now 8192
2014-08-18 18:14:12.687132 7f233a32d700 10 journal prepare_header ptr= 0x23ac000
2014-08-18 18:14:12.687134 7f233a32d700 10 journal prepare_header content=0x4 ,0x=40
2014-08-18 18:14:12.687137 7f233a32d700 15 journal do_aio_write writing 8192~0 + header
2014-08-18 18:14:12.687139 7f233a32d700 20 journal write_aio_bl 0~4096 seq 0
2014-08-18 18:14:12.687142 7f233a32d700 10 journal write_aio_bl ptr= 0x23ac000
2014-08-18 18:14:12.687144 7f233a32d700 10 journal write_aio_bl content=0x4 ,0x=40
2014-08-18 18:14:12.687146 7f233a32d700 20 journal write_aio_bl .. 0~4096 in 1
2014-08-18 18:14:12.687148 7f233a32d700 10 journal write_aio_bl ptr= 0x23ac000
2014-08-18 18:14:12.687150 7f233a32d700 10 journal write_aio_bl content=0x4 ,0x=40
2014-08-18 18:14:12.687176 7f233a32d700 20 journal write_aio_bl 8192~0 seq 2
2014-08-18 18:14:12.687181 7f233a32d700 5 journal put_throttle finished 0 ops and 0 bytes, now 0 ops and 0 bytes
2014-08-18 18:14:12.687183 7f233a32d700 10 journal write_thread_entry finish
2014-08-18 18:14:12.687842 7f233f154800 -1 created object store /ceph0 journal /dev/sdc1 for osd.0 fsid 390c6db4-7737-40e4-9c30-837e95db7055
(15-15/17)