|
2013-08-15 06:47:28.611094 7f5bb48817c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3452
|
|
2013-08-15 06:47:28.612089 7f5bb48817c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.615322 7f5bb48817c0 0 filestore(/var/lib/ceph/osd/ceph-9) lock_fsid failed to lock /var/lib/ceph/osd/ceph-9/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.615341 7f5bb48817c0 -1 filestore(/var/lib/ceph/osd/ceph-9) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.615367 7f5bb48817c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-9: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.656410 7f49ec40f7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3713
|
|
2013-08-15 06:47:28.657397 7f49ec40f7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.660406 7f49ec40f7c0 0 filestore(/var/lib/ceph/osd/ceph-9) lock_fsid failed to lock /var/lib/ceph/osd/ceph-9/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.660422 7f49ec40f7c0 -1 filestore(/var/lib/ceph/osd/ceph-9) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.660452 7f49ec40f7c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-9: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.707684 7fbd22c2e7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3957
|
|
2013-08-15 06:47:28.708279 7fbd22c2e7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.710951 7fbd22c2e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) lock_fsid failed to lock /var/lib/ceph/osd/ceph-9/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.710974 7fbd22c2e7c0 -1 filestore(/var/lib/ceph/osd/ceph-9) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.711013 7fbd22c2e7c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-9: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.759408 7f91847427c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4155
|
|
2013-08-15 06:47:28.759978 7f91847427c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.762407 7f91847427c0 0 filestore(/var/lib/ceph/osd/ceph-9) lock_fsid failed to lock /var/lib/ceph/osd/ceph-9/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.762424 7f91847427c0 -1 filestore(/var/lib/ceph/osd/ceph-9) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.762463 7f91847427c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-9: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.809842 7f5fbb9177c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4351
|
|
2013-08-15 06:47:28.810575 7f5fbb9177c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.813218 7f5fbb9177c0 0 filestore(/var/lib/ceph/osd/ceph-9) lock_fsid failed to lock /var/lib/ceph/osd/ceph-9/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.813235 7f5fbb9177c0 -1 filestore(/var/lib/ceph/osd/ceph-9) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.813275 7f5fbb9177c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-9: (16) Device or resource busy[0m
|
|
2013-08-15 20:46:26.419097 7fa91d8a87c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3272
|
|
2013-08-15 20:46:26.421281 7fa91d8a87c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.473400 7f0ffcf457c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3472
|
|
2013-08-15 20:46:26.473875 7f0ffcf457c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.521089 7f14893f37c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3675
|
|
2013-08-15 20:46:26.521851 7f14893f37c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.564623 7fb200e687c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3879
|
|
2013-08-15 20:46:26.565454 7fb200e687c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.607884 7face15f57c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4078
|
|
2013-08-15 20:46:26.608370 7face15f57c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.655986 7f65045de7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4291
|
|
2013-08-15 20:46:26.656510 7f65045de7c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:50:22.473057 7ffd2348c7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4117
|
|
2013-08-15 20:50:22.474297 7ffd2348c7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:22.530335 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:22.530351 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:22.530622 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:22.536283 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:22.536599 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:22.666978 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:22.698560 7ffd2348c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:22.698575 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:23.410670 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.492235 7ffd2348c7c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
2013-08-15 20:50:24.527924 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:24.527941 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:24.528236 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:24.551835 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:24.551912 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:24.606093 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:24.606724 7ffd2348c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:24.606745 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.708580 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.267189 7ffd19575700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:25.267205 7ffd19575700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
2013-08-15 20:50:25.268318 7ffd19575700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7ffd19575700 time 2013-08-15 20:50:25.267213
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7ffd22e68f8e]
|
|
5: (clone()+0x6d) [0x7ffd2111ae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-327> 2013-08-15 20:50:22.471517 7ffd2348c7c0 5 asok(0x1d300e0) register_command perfcounters_dump hook 0x1d2a010
|
|
-326> 2013-08-15 20:50:22.471554 7ffd2348c7c0 5 asok(0x1d300e0) register_command 1 hook 0x1d2a010
|
|
-325> 2013-08-15 20:50:22.471562 7ffd2348c7c0 5 asok(0x1d300e0) register_command perf dump hook 0x1d2a010
|
|
-324> 2013-08-15 20:50:22.471570 7ffd2348c7c0 5 asok(0x1d300e0) register_command perfcounters_schema hook 0x1d2a010
|
|
-323> 2013-08-15 20:50:22.471576 7ffd2348c7c0 5 asok(0x1d300e0) register_command 2 hook 0x1d2a010
|
|
-322> 2013-08-15 20:50:22.471581 7ffd2348c7c0 5 asok(0x1d300e0) register_command perf schema hook 0x1d2a010
|
|
-321> 2013-08-15 20:50:22.471592 7ffd2348c7c0 5 asok(0x1d300e0) register_command config show hook 0x1d2a010
|
|
-320> 2013-08-15 20:50:22.471600 7ffd2348c7c0 5 asok(0x1d300e0) register_command config set hook 0x1d2a010
|
|
-319> 2013-08-15 20:50:22.471606 7ffd2348c7c0 5 asok(0x1d300e0) register_command log flush hook 0x1d2a010
|
|
-318> 2013-08-15 20:50:22.471613 7ffd2348c7c0 5 asok(0x1d300e0) register_command log dump hook 0x1d2a010
|
|
-317> 2013-08-15 20:50:22.471619 7ffd2348c7c0 5 asok(0x1d300e0) register_command log reopen hook 0x1d2a010
|
|
-316> 2013-08-15 20:50:22.473057 7ffd2348c7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4117
|
|
-315> 2013-08-15 20:50:22.474297 7ffd2348c7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-314> 2013-08-15 20:50:22.474479 7ffd2348c7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/4117 need_addr=1
|
|
-313> 2013-08-15 20:50:22.474501 7ffd2348c7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-312> 2013-08-15 20:50:22.474511 7ffd2348c7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6807/4117 need_addr=0
|
|
-311> 2013-08-15 20:50:22.474530 7ffd2348c7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-310> 2013-08-15 20:50:22.474536 7ffd2348c7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6808/4117 need_addr=0
|
|
-309> 2013-08-15 20:50:22.479031 7ffd2348c7c0 5 asok(0x1d300e0) init /var/run/ceph/ceph-osd.9.asok
|
|
-308> 2013-08-15 20:50:22.479049 7ffd2348c7c0 5 asok(0x1d300e0) bind_and_listen /var/run/ceph/ceph-osd.9.asok
|
|
-307> 2013-08-15 20:50:22.479072 7ffd2348c7c0 5 asok(0x1d300e0) register_command 0 hook 0x1d280b8
|
|
-306> 2013-08-15 20:50:22.479078 7ffd2348c7c0 5 asok(0x1d300e0) register_command version hook 0x1d280b8
|
|
-305> 2013-08-15 20:50:22.479085 7ffd2348c7c0 5 asok(0x1d300e0) register_command git_version hook 0x1d280b8
|
|
-304> 2013-08-15 20:50:22.479091 7ffd2348c7c0 5 asok(0x1d300e0) register_command help hook 0x1d2a0d0
|
|
-303> 2013-08-15 20:50:22.479123 7ffd1ee66700 5 asok(0x1d300e0) entry start
|
|
-302> 2013-08-15 20:50:22.530335 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-301> 2013-08-15 20:50:22.530351 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-300> 2013-08-15 20:50:22.530622 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-299> 2013-08-15 20:50:22.536283 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-298> 2013-08-15 20:50:22.536599 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-297> 2013-08-15 20:50:22.666978 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-296> 2013-08-15 20:50:22.698516 7ffd2348c7c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2506
|
|
-295> 2013-08-15 20:50:22.698560 7ffd2348c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-294> 2013-08-15 20:50:22.698575 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-293> 2013-08-15 20:50:22.733081 7ffd2348c7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-292> 2013-08-15 20:50:22.737606 7ffd2348c7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-291> 2013-08-15 20:50:22.737624 7ffd2348c7c0 3 journal journal_replay: applying op seq 2507
|
|
-290> 2013-08-15 20:50:22.761304 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2507
|
|
-289> 2013-08-15 20:50:22.779497 7ffd2348c7c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-288> 2013-08-15 20:50:22.779513 7ffd2348c7c0 3 journal journal_replay: applying op seq 2508
|
|
-287> 2013-08-15 20:50:22.793922 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2508
|
|
-286> 2013-08-15 20:50:22.820167 7ffd2348c7c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-285> 2013-08-15 20:50:22.820192 7ffd2348c7c0 3 journal journal_replay: applying op seq 2509
|
|
-284> 2013-08-15 20:50:22.822760 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2509
|
|
-283> 2013-08-15 20:50:22.847216 7ffd2348c7c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-282> 2013-08-15 20:50:22.847239 7ffd2348c7c0 3 journal journal_replay: applying op seq 2510
|
|
-281> 2013-08-15 20:50:22.850434 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2510
|
|
-280> 2013-08-15 20:50:22.874900 7ffd2348c7c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-279> 2013-08-15 20:50:22.874924 7ffd2348c7c0 3 journal journal_replay: applying op seq 2511
|
|
-278> 2013-08-15 20:50:22.879701 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2511
|
|
-277> 2013-08-15 20:50:22.909040 7ffd2348c7c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-276> 2013-08-15 20:50:22.909064 7ffd2348c7c0 3 journal journal_replay: applying op seq 2512
|
|
-275> 2013-08-15 20:50:22.920425 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2512
|
|
-274> 2013-08-15 20:50:22.949415 7ffd2348c7c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-273> 2013-08-15 20:50:22.949437 7ffd2348c7c0 3 journal journal_replay: applying op seq 2513
|
|
-272> 2013-08-15 20:50:22.951964 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2513
|
|
-271> 2013-08-15 20:50:22.979751 7ffd2348c7c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-270> 2013-08-15 20:50:22.979772 7ffd2348c7c0 3 journal journal_replay: applying op seq 2514
|
|
-269> 2013-08-15 20:50:22.982641 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2514
|
|
-268> 2013-08-15 20:50:23.014134 7ffd2348c7c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-267> 2013-08-15 20:50:23.014152 7ffd2348c7c0 3 journal journal_replay: applying op seq 2515
|
|
-266> 2013-08-15 20:50:23.016827 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2515
|
|
-265> 2013-08-15 20:50:23.047936 7ffd2348c7c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-264> 2013-08-15 20:50:23.047954 7ffd2348c7c0 3 journal journal_replay: applying op seq 2516
|
|
-263> 2013-08-15 20:50:23.050811 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2516
|
|
-262> 2013-08-15 20:50:23.081112 7ffd2348c7c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-261> 2013-08-15 20:50:23.081134 7ffd2348c7c0 3 journal journal_replay: applying op seq 2517
|
|
-260> 2013-08-15 20:50:23.083539 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2517
|
|
-259> 2013-08-15 20:50:23.112087 7ffd2348c7c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-258> 2013-08-15 20:50:23.112106 7ffd2348c7c0 3 journal journal_replay: applying op seq 2518
|
|
-257> 2013-08-15 20:50:23.114527 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2518
|
|
-256> 2013-08-15 20:50:23.133064 7ffd2348c7c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-255> 2013-08-15 20:50:23.133085 7ffd2348c7c0 3 journal journal_replay: applying op seq 2519
|
|
-254> 2013-08-15 20:50:23.136575 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2519
|
|
-253> 2013-08-15 20:50:23.159380 7ffd2348c7c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-252> 2013-08-15 20:50:23.159398 7ffd2348c7c0 3 journal journal_replay: applying op seq 2520
|
|
-251> 2013-08-15 20:50:23.162038 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2520
|
|
-250> 2013-08-15 20:50:23.186003 7ffd2348c7c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-249> 2013-08-15 20:50:23.186024 7ffd2348c7c0 3 journal journal_replay: applying op seq 2521
|
|
-248> 2013-08-15 20:50:23.190244 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2521
|
|
-247> 2013-08-15 20:50:23.213526 7ffd2348c7c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-246> 2013-08-15 20:50:23.213544 7ffd2348c7c0 3 journal journal_replay: applying op seq 2522
|
|
-245> 2013-08-15 20:50:23.216258 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2522
|
|
-244> 2013-08-15 20:50:23.237623 7ffd2348c7c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-243> 2013-08-15 20:50:23.237640 7ffd2348c7c0 3 journal journal_replay: applying op seq 2523
|
|
-242> 2013-08-15 20:50:23.240203 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2523
|
|
-241> 2013-08-15 20:50:23.261037 7ffd2348c7c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-240> 2013-08-15 20:50:23.261054 7ffd2348c7c0 3 journal journal_replay: applying op seq 2524
|
|
-239> 2013-08-15 20:50:23.264353 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2524
|
|
-238> 2013-08-15 20:50:23.286795 7ffd2348c7c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-237> 2013-08-15 20:50:23.286817 7ffd2348c7c0 3 journal journal_replay: applying op seq 2525
|
|
-236> 2013-08-15 20:50:23.289737 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2525
|
|
-235> 2013-08-15 20:50:23.311973 7ffd2348c7c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-234> 2013-08-15 20:50:23.312021 7ffd2348c7c0 3 journal journal_replay: applying op seq 2526
|
|
-233> 2013-08-15 20:50:23.315034 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2526
|
|
-232> 2013-08-15 20:50:23.338328 7ffd2348c7c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-231> 2013-08-15 20:50:23.338364 7ffd2348c7c0 3 journal journal_replay: applying op seq 2527
|
|
-230> 2013-08-15 20:50:23.341501 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2527
|
|
-229> 2013-08-15 20:50:23.364809 7ffd2348c7c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-228> 2013-08-15 20:50:23.364828 7ffd2348c7c0 3 journal journal_replay: applying op seq 2528
|
|
-227> 2013-08-15 20:50:23.368379 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2528
|
|
-226> 2013-08-15 20:50:23.386995 7ffd2348c7c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-225> 2013-08-15 20:50:23.387015 7ffd2348c7c0 3 journal journal_replay: applying op seq 2529
|
|
-224> 2013-08-15 20:50:23.389969 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2529
|
|
-223> 2013-08-15 20:50:23.407627 7ffd2348c7c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-222> 2013-08-15 20:50:23.407644 7ffd2348c7c0 3 journal journal_replay: applying op seq 2530
|
|
-221> 2013-08-15 20:50:23.410611 7ffd2348c7c0 3 journal journal_replay: r = 0, op_seq now 2530
|
|
-220> 2013-08-15 20:50:23.410634 7ffd2348c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-219> 2013-08-15 20:50:23.410638 7ffd2348c7c0 3 journal journal_replay: end of journal, done.
|
|
-218> 2013-08-15 20:50:23.410670 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-217> 2013-08-15 20:50:24.492046 7ffd1c57b700 1 FileStore::op_tp worker finish
|
|
-216> 2013-08-15 20:50:24.492073 7ffd1bd7a700 1 FileStore::op_tp worker finish
|
|
-215> 2013-08-15 20:50:24.492235 7ffd2348c7c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
-214> 2013-08-15 20:50:24.492672 7ffd2348c7c0 10 monclient(hunting): build_initial_monmap
|
|
-213> 2013-08-15 20:50:24.492730 7ffd2348c7c0 5 adding auth protocol: cephx
|
|
-212> 2013-08-15 20:50:24.492736 7ffd2348c7c0 5 adding auth protocol: cephx
|
|
-211> 2013-08-15 20:50:24.492820 7ffd2348c7c0 1 -- 0.0.0.0:6805/4117 messenger.start
|
|
-210> 2013-08-15 20:50:24.492857 7ffd2348c7c0 1 -- :/0 messenger.start
|
|
-209> 2013-08-15 20:50:24.492881 7ffd2348c7c0 1 -- 9.4.196.3:6808/4117 messenger.start
|
|
-208> 2013-08-15 20:50:24.492895 7ffd2348c7c0 1 -- 9.4.196.3:6807/4117 messenger.start
|
|
-207> 2013-08-15 20:50:24.492986 7ffd2348c7c0 2 osd.9 0 mounting /var/lib/ceph/osd/ceph-9 /var/lib/ceph/osd/ceph-9/journal
|
|
-206> 2013-08-15 20:50:24.527924 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-205> 2013-08-15 20:50:24.527941 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-204> 2013-08-15 20:50:24.528236 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-203> 2013-08-15 20:50:24.551835 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-202> 2013-08-15 20:50:24.551912 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-201> 2013-08-15 20:50:24.606093 7ffd2348c7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-200> 2013-08-15 20:50:24.606696 7ffd2348c7c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-199> 2013-08-15 20:50:24.606724 7ffd2348c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-198> 2013-08-15 20:50:24.606745 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-197> 2013-08-15 20:50:24.610984 7ffd2348c7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-196> 2013-08-15 20:50:24.615314 7ffd2348c7c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-195> 2013-08-15 20:50:24.619565 7ffd2348c7c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-194> 2013-08-15 20:50:24.623541 7ffd2348c7c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-193> 2013-08-15 20:50:24.627688 7ffd2348c7c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-192> 2013-08-15 20:50:24.632073 7ffd2348c7c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-191> 2013-08-15 20:50:24.636457 7ffd2348c7c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-190> 2013-08-15 20:50:24.640885 7ffd2348c7c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-189> 2013-08-15 20:50:24.645151 7ffd2348c7c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-188> 2013-08-15 20:50:24.649525 7ffd2348c7c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-187> 2013-08-15 20:50:24.653703 7ffd2348c7c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-186> 2013-08-15 20:50:24.657855 7ffd2348c7c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-185> 2013-08-15 20:50:24.662061 7ffd2348c7c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-184> 2013-08-15 20:50:24.666325 7ffd2348c7c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-183> 2013-08-15 20:50:24.670488 7ffd2348c7c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-182> 2013-08-15 20:50:24.674650 7ffd2348c7c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-181> 2013-08-15 20:50:24.678878 7ffd2348c7c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-180> 2013-08-15 20:50:24.683123 7ffd2348c7c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-179> 2013-08-15 20:50:24.687291 7ffd2348c7c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-178> 2013-08-15 20:50:24.691550 7ffd2348c7c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-177> 2013-08-15 20:50:24.695829 7ffd2348c7c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-176> 2013-08-15 20:50:24.700274 7ffd2348c7c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-175> 2013-08-15 20:50:24.704384 7ffd2348c7c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-174> 2013-08-15 20:50:24.708518 7ffd2348c7c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-173> 2013-08-15 20:50:24.708538 7ffd2348c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-172> 2013-08-15 20:50:24.708543 7ffd2348c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-171> 2013-08-15 20:50:24.708544 7ffd2348c7c0 3 journal journal_replay: end of journal, done.
|
|
-170> 2013-08-15 20:50:24.708580 7ffd2348c7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-169> 2013-08-15 20:50:24.708766 7ffd2348c7c0 2 osd.9 0 boot
|
|
-168> 2013-08-15 20:50:24.736792 7ffd2348c7c0 2 osd.9 63 superblock: i am osd.9
|
|
-167> 2013-08-15 20:50:24.736867 7ffd2348c7c0 1 accepter.accepter.start
|
|
-166> 2013-08-15 20:50:24.736915 7ffd2348c7c0 1 accepter.accepter.start
|
|
-165> 2013-08-15 20:50:24.736971 7ffd2348c7c0 1 accepter.accepter.start
|
|
-164> 2013-08-15 20:50:24.736991 7ffd2348c7c0 10 monclient(hunting): init
|
|
-163> 2013-08-15 20:50:24.737000 7ffd2348c7c0 5 adding auth protocol: cephx
|
|
-162> 2013-08-15 20:50:24.737002 7ffd2348c7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-161> 2013-08-15 20:50:24.737160 7ffd2348c7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-9/keyring
|
|
-160> 2013-08-15 20:50:24.737401 7ffd2348c7c0 5 asok(0x1d300e0) register_command dump_ops_in_flight hook 0x1d2a820
|
|
-159> 2013-08-15 20:50:24.737413 7ffd2348c7c0 5 asok(0x1d300e0) register_command dump_historic_ops hook 0x1d2a820
|
|
-158> 2013-08-15 20:50:24.737416 7ffd2348c7c0 5 asok(0x1d300e0) register_command dump_op_pq_state hook 0x1d2a820
|
|
-157> 2013-08-15 20:50:24.737419 7ffd2348c7c0 5 asok(0x1d300e0) register_command setomapval hook 0x20c72e0
|
|
-156> 2013-08-15 20:50:24.737434 7ffd2348c7c0 5 asok(0x1d300e0) register_command rmomapkey hook 0x20c72e0
|
|
-155> 2013-08-15 20:50:24.737437 7ffd2348c7c0 5 asok(0x1d300e0) register_command setomapheader hook 0x20c72e0
|
|
-154> 2013-08-15 20:50:24.737439 7ffd2348c7c0 5 asok(0x1d300e0) register_command getomap hook 0x20c72e0
|
|
-153> 2013-08-15 20:50:24.737442 7ffd2348c7c0 5 asok(0x1d300e0) register_command truncobj hook 0x20c72e0
|
|
-152> 2013-08-15 20:50:24.737444 7ffd2348c7c0 5 asok(0x1d300e0) register_command injectdataerr hook 0x20c72e0
|
|
-151> 2013-08-15 20:50:24.737446 7ffd2348c7c0 5 asok(0x1d300e0) register_command injectmdataerr hook 0x20c72e0
|
|
-150> 2013-08-15 20:50:24.737455 7ffd0dd5e700 5 osd.9 63 heartbeat: osd_stat(1026 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-149> 2013-08-15 20:50:24.737502 7ffd2348c7c0 10 monclient(hunting): _reopen_session
|
|
-148> 2013-08-15 20:50:24.737551 7ffd2348c7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x2138580 addr 9.4.196.1:6789/0
|
|
-147> 2013-08-15 20:50:24.737580 7ffd2348c7c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-146> 2013-08-15 20:50:24.737584 7ffd2348c7c0 1 -- 0.0.0.0:6805/4117 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1d3cb40 con 0x2138580
|
|
-145> 2013-08-15 20:50:24.737596 7ffd2348c7c0 10 monclient(hunting): renew_subs
|
|
-144> 2013-08-15 20:50:24.738123 7ffd1bd7a700 1 -- 9.4.196.3:6805/4117 learned my addr 9.4.196.3:6805/4117
|
|
-143> 2013-08-15 20:50:24.738356 7ffd14d6c700 10 monclient(hunting): renew_subs
|
|
-142> 2013-08-15 20:50:24.738845 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1db21e0 con 0x2138580
|
|
-141> 2013-08-15 20:50:24.738870 7ffd14d6c700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-140> 2013-08-15 20:50:24.738885 7ffd14d6c700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-139> 2013-08-15 20:50:24.738889 7ffd14d6c700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-138> 2013-08-15 20:50:24.738909 7ffd14d6c700 1 monclient(hunting): found mon.zc2store
|
|
-137> 2013-08-15 20:50:24.738916 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3296819727 0 0) 0x1f62600 con 0x2138580
|
|
-136> 2013-08-15 20:50:24.738939 7ffd14d6c700 10 monclient: my global_id is 4362
|
|
-135> 2013-08-15 20:50:24.739083 7ffd14d6c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-134> 2013-08-15 20:50:24.739092 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1d3dd40 con 0x2138580
|
|
-133> 2013-08-15 20:50:24.739717 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1074977128 0 0) 0x1f62400 con 0x2138580
|
|
-132> 2013-08-15 20:50:24.739827 7ffd14d6c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-131> 2013-08-15 20:50:24.739835 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x213e240 con 0x2138580
|
|
-130> 2013-08-15 20:50:24.740550 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (616994107 0 0) 0x1f62800 con 0x2138580
|
|
-129> 2013-08-15 20:50:24.740615 7ffd14d6c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-128> 2013-08-15 20:50:24.740622 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1ea2000 con 0x2138580
|
|
-127> 2013-08-15 20:50:24.740639 7ffd14d6c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-126> 2013-08-15 20:50:24.740643 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x1ea2380 con 0x2138580
|
|
-125> 2013-08-15 20:50:24.740655 7ffd14d6c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:54.740654)
|
|
-124> 2013-08-15 20:50:24.740676 7ffd14d6c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-123> 2013-08-15 20:50:24.740681 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x213e000 con 0x2138580
|
|
-122> 2013-08-15 20:50:24.740706 7ffd2348c7c0 5 monclient: authenticate success, global_id 4362
|
|
-121> 2013-08-15 20:50:24.740715 7ffd2348c7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:54.740714)
|
|
-120> 2013-08-15 20:50:24.741117 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1db25a0 con 0x2138580
|
|
-119> 2013-08-15 20:50:24.741129 7ffd14d6c700 10 monclient: handle_monmap mon_map v1
|
|
-118> 2013-08-15 20:50:24.741138 7ffd14d6c700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-117> 2013-08-15 20:50:24.741142 7ffd14d6c700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-116> 2013-08-15 20:50:24.741160 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1ea2000 con 0x2138580
|
|
-115> 2013-08-15 20:50:24.741171 7ffd14d6c700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:24.737597 renew after 2013-08-15 20:52:54.737597
|
|
-114> 2013-08-15 20:50:24.741181 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1db23c0 con 0x2138580
|
|
-113> 2013-08-15 20:50:24.741189 7ffd14d6c700 10 monclient: handle_monmap mon_map v1
|
|
-112> 2013-08-15 20:50:24.741194 7ffd14d6c700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-111> 2013-08-15 20:50:24.741197 7ffd14d6c700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-110> 2013-08-15 20:50:24.741211 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1ea2540 con 0x2138580
|
|
-109> 2013-08-15 20:50:24.741218 7ffd14d6c700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-108> 2013-08-15 20:50:24.741223 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (2296613975 0 0) 0x1f62a00 con 0x2138580
|
|
-107> 2013-08-15 20:50:24.741265 7ffd14d6c700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:54.741265)
|
|
-106> 2013-08-15 20:50:24.741281 7ffd2348c7c0 10 monclient: wait_auth_rotating done
|
|
-105> 2013-08-15 20:50:24.741878 7ffd2348c7c0 10 monclient: get_version osdmap
|
|
-104> 2013-08-15 20:50:24.741885 7ffd2348c7c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-103> 2013-08-15 20:50:24.741892 7ffd2348c7c0 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x1d981a0 con 0x2138580
|
|
-102> 2013-08-15 20:50:24.742175 7ffd14d6c700 1 -- 9.4.196.3:6805/4117 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=83) v2 ==== 24+0+0 (2922730777 0 0) 0x1ea2380 con 0x2138580
|
|
-101> 2013-08-15 20:50:24.742236 7ffd10d64700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-100> 2013-08-15 20:50:24.742251 7ffd10d64700 1 -- 9.4.196.3:6805/4117 --> 9.4.196.1:6789/0 -- osd_boot(osd.9 booted 0 v63) v3 -- ?+0 0x2148000 con 0x2138580
|
|
-99> 2013-08-15 20:50:25.255496 7ffd0c056700 1 -- 9.4.196.3:6807/4117 >> :/0 pipe(0x1f61900 sd=33 :6807 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:48763/0
|
|
-98> 2013-08-15 20:50:25.257314 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x213f440 con 0x2139ce0
|
|
-97> 2013-08-15 20:50:25.257357 7ffd13d6a700 3 osd.9 63 handle_osd_map epochs [64,84], i have 63, src has [1,84]
|
|
-96> 2013-08-15 20:50:25.257842 7ffd0be54700 1 -- 9.4.196.3:6807/4117 >> :/0 pipe(0x1f61b80 sd=34 :6807 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:48767/0
|
|
-95> 2013-08-15 20:50:25.259676 7ffd0bc52700 1 -- 9.4.196.3:6808/4117 >> :/0 pipe(0x21a8000 sd=35 :6808 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:45811/0
|
|
-94> 2013-08-15 20:50:25.262652 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-93> 2013-08-15 20:50:25.262676 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-92> 2013-08-15 20:50:25.262707 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-91> 2013-08-15 20:50:25.262750 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-90> 2013-08-15 20:50:25.262771 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-89> 2013-08-15 20:50:25.262778 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-88> 2013-08-15 20:50:25.262811 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-87> 2013-08-15 20:50:25.262820 7ffd13d6a700 1 osd.9 84 state: booting -> active
|
|
-86> 2013-08-15 20:50:25.263065 7ffd0ba50700 1 -- 9.4.196.3:6808/4117 >> :/0 pipe(0x21a8500 sd=36 :6808 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:45813/0
|
|
-85> 2013-08-15 20:50:25.263647 7ffd0b94f700 1 -- 9.4.196.3:6807/4117 >> :/0 pipe(0x21a8c80 sd=37 :6807 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.3:48769/0
|
|
-84> 2013-08-15 20:50:25.263695 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_query(0.7,1.6,2.5 epoch 84) v2 ==== 387+0+0 (4109240191 0 0) 0x1db21e0 con 0x2139ce0
|
|
-83> 2013-08-15 20:50:25.263719 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.257638, event: header_read, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-82> 2013-08-15 20:50:25.263733 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.257640, event: throttled, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-81> 2013-08-15 20:50:25.263740 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.257684, event: all_read, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-80> 2013-08-15 20:50:25.263747 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.263711, event: dispatched, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-79> 2013-08-15 20:50:25.263755 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.263755, event: waiting_for_osdmap, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-78> 2013-08-15 20:50:25.263768 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.263767, event: started, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-77> 2013-08-15 20:50:25.263807 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.263807, event: done, request: pg_query(0.7,1.6,2.5 epoch 84) v2
|
|
-76> 2013-08-15 20:50:25.263822 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x213fd40 con 0x2144420
|
|
-75> 2013-08-15 20:50:25.263837 7ffd13d6a700 3 osd.9 84 handle_osd_map epochs [64,84], i have 84, src has [1,84]
|
|
-74> 2013-08-15 20:50:25.263848 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.123(6) epoch 84) v4 ==== 830+0+0 (3809005805 0 0) 0x1ea2c40 con 0x2144dc0
|
|
-73> 2013-08-15 20:50:25.263861 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.261303, event: header_read, request: pg_notify(3.123(6) epoch 84) v4
|
|
-72> 2013-08-15 20:50:25.263870 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.261304, event: throttled, request: pg_notify(3.123(6) epoch 84) v4
|
|
-71> 2013-08-15 20:50:25.263876 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.261361, event: all_read, request: pg_notify(3.123(6) epoch 84) v4
|
|
-70> 2013-08-15 20:50:25.263883 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.263859, event: dispatched, request: pg_notify(3.123(6) epoch 84) v4
|
|
-69> 2013-08-15 20:50:25.263891 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.263891, event: waiting_for_osdmap, request: pg_notify(3.123(6) epoch 84) v4
|
|
-68> 2013-08-15 20:50:25.263902 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.263902, event: started, request: pg_notify(3.123(6) epoch 84) v4
|
|
-67> 2013-08-15 20:50:25.263887 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.64( v 63'17 (0'0,63'17] local-les=62 n=5 ec=56 les/c 62/62 84/84/56) [4,9] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-66> 2013-08-15 20:50:25.264046 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.264046, event: done, request: pg_notify(3.123(6) epoch 84) v4
|
|
-65> 2013-08-15 20:50:25.264064 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.2 9.4.196.1:6807/6887 2 ==== pg_notify(2.9(5) epoch 84) v4 ==== 787+0+0 (1815781942 0 0) 0x1ea2700 con 0x2144420
|
|
-64> 2013-08-15 20:50:25.264079 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.259620, event: header_read, request: pg_notify(2.9(5) epoch 84) v4
|
|
-63> 2013-08-15 20:50:25.264087 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.259621, event: throttled, request: pg_notify(2.9(5) epoch 84) v4
|
|
-62> 2013-08-15 20:50:25.264094 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.259689, event: all_read, request: pg_notify(2.9(5) epoch 84) v4
|
|
-61> 2013-08-15 20:50:25.264101 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.264077, event: dispatched, request: pg_notify(2.9(5) epoch 84) v4
|
|
-60> 2013-08-15 20:50:25.264107 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.264107, event: waiting_for_osdmap, request: pg_notify(2.9(5) epoch 84) v4
|
|
-59> 2013-08-15 20:50:25.264116 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.264116, event: started, request: pg_notify(2.9(5) epoch 84) v4
|
|
-58> 2013-08-15 20:50:25.264139 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.264139, event: done, request: pg_notify(2.9(5) epoch 84) v4
|
|
-57> 2013-08-15 20:50:25.264152 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.2 9.4.196.1:6807/6887 3 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x21b0240 con 0x2144420
|
|
-56> 2013-08-15 20:50:25.264167 7ffd13d6a700 3 osd.9 84 handle_osd_map epochs [64,84], i have 84, src has [1,84]
|
|
-55> 2013-08-15 20:50:25.264180 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.2 9.4.196.1:6807/6887 4 ==== pg_notify(1.a(5) epoch 84) v4 ==== 787+0+0 (3077272268 0 0) 0x1ea2a80 con 0x2144420
|
|
-54> 2013-08-15 20:50:25.264193 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.259874, event: header_read, request: pg_notify(1.a(5) epoch 84) v4
|
|
-53> 2013-08-15 20:50:25.264202 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.259875, event: throttled, request: pg_notify(1.a(5) epoch 84) v4
|
|
-52> 2013-08-15 20:50:25.264209 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.259948, event: all_read, request: pg_notify(1.a(5) epoch 84) v4
|
|
-51> 2013-08-15 20:50:25.264216 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.264191, event: dispatched, request: pg_notify(1.a(5) epoch 84) v4
|
|
-50> 2013-08-15 20:50:25.264223 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.264223, event: waiting_for_osdmap, request: pg_notify(1.a(5) epoch 84) v4
|
|
-49> 2013-08-15 20:50:25.264231 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.264231, event: started, request: pg_notify(1.a(5) epoch 84) v4
|
|
-48> 2013-08-15 20:50:25.264384 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.264384, event: done, request: pg_notify(1.a(5) epoch 84) v4
|
|
-47> 2013-08-15 20:50:25.264449 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.1 9.4.196.1:6801/5083 3 ==== pg_notify(3.14a(3) epoch 84) v4 ==== 713+0+0 (3944878718 0 0) 0x1ea2e00 con 0x2139ce0
|
|
-46> 2013-08-15 20:50:25.264468 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264296, event: header_read, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-45> 2013-08-15 20:50:25.264475 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264297, event: throttled, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-44> 2013-08-15 20:50:25.264482 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264392, event: all_read, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-43> 2013-08-15 20:50:25.264489 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264466, event: dispatched, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-42> 2013-08-15 20:50:25.264496 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264496, event: waiting_for_osdmap, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-41> 2013-08-15 20:50:25.264504 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264504, event: started, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-40> 2013-08-15 20:50:25.264652 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.1eb( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 84/84/84) [9,3] r=0 lpr=84 pi=61-83/8 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-39> 2013-08-15 20:50:25.264934 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.264934, event: done, request: pg_notify(3.14a(3) epoch 84) v4
|
|
-38> 2013-08-15 20:50:25.264953 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.5 9.4.196.3:6819/4129 1 ==== pg_notify(0.b(6) epoch 84) v4 ==== 830+0+0 (994748206 0 0) 0x1ea2fc0 con 0x2145340
|
|
-37> 2013-08-15 20:50:25.264969 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.264515, event: header_read, request: pg_notify(0.b(6) epoch 84) v4
|
|
-36> 2013-08-15 20:50:25.264977 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.264517, event: throttled, request: pg_notify(0.b(6) epoch 84) v4
|
|
-35> 2013-08-15 20:50:25.264984 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.264600, event: all_read, request: pg_notify(0.b(6) epoch 84) v4
|
|
-34> 2013-08-15 20:50:25.264993 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.264967, event: dispatched, request: pg_notify(0.b(6) epoch 84) v4
|
|
-33> 2013-08-15 20:50:25.265001 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.265000, event: waiting_for_osdmap, request: pg_notify(0.b(6) epoch 84) v4
|
|
-32> 2013-08-15 20:50:25.265010 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.265010, event: started, request: pg_notify(0.b(6) epoch 84) v4
|
|
-31> 2013-08-15 20:50:25.265032 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.265032, event: done, request: pg_notify(0.b(6) epoch 84) v4
|
|
-30> 2013-08-15 20:50:25.265045 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.5 9.4.196.3:6819/4129 2 ==== pg_notify(2.9(6) epoch 84) v4 ==== 830+0+0 (3838464258 0 0) 0x1ea3340 con 0x2145340
|
|
-29> 2013-08-15 20:50:25.265060 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.264615, event: header_read, request: pg_notify(2.9(6) epoch 84) v4
|
|
-28> 2013-08-15 20:50:25.265067 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.264616, event: throttled, request: pg_notify(2.9(6) epoch 84) v4
|
|
-27> 2013-08-15 20:50:25.265074 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.264669, event: all_read, request: pg_notify(2.9(6) epoch 84) v4
|
|
-26> 2013-08-15 20:50:25.265080 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.265058, event: dispatched, request: pg_notify(2.9(6) epoch 84) v4
|
|
-25> 2013-08-15 20:50:25.265087 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.265087, event: waiting_for_osdmap, request: pg_notify(2.9(6) epoch 84) v4
|
|
-24> 2013-08-15 20:50:25.265096 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.265096, event: started, request: pg_notify(2.9(6) epoch 84) v4
|
|
-23> 2013-08-15 20:50:25.265118 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.265118, event: done, request: pg_notify(2.9(6) epoch 84) v4
|
|
-22> 2013-08-15 20:50:25.265130 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.5 9.4.196.3:6819/4129 3 ==== pg_notify(1.a(6) epoch 84) v4 ==== 830+0+0 (2379271390 0 0) 0x1ea3180 con 0x2145340
|
|
-21> 2013-08-15 20:50:25.265144 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.264677, event: header_read, request: pg_notify(1.a(6) epoch 84) v4
|
|
-20> 2013-08-15 20:50:25.265151 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.264678, event: throttled, request: pg_notify(1.a(6) epoch 84) v4
|
|
-19> 2013-08-15 20:50:25.265158 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.264726, event: all_read, request: pg_notify(1.a(6) epoch 84) v4
|
|
-18> 2013-08-15 20:50:25.265164 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.265142, event: dispatched, request: pg_notify(1.a(6) epoch 84) v4
|
|
-17> 2013-08-15 20:50:25.265171 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.265171, event: waiting_for_osdmap, request: pg_notify(1.a(6) epoch 84) v4
|
|
-16> 2013-08-15 20:50:25.265179 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.265179, event: started, request: pg_notify(1.a(6) epoch 84) v4
|
|
-15> 2013-08-15 20:50:25.265203 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.265203, event: done, request: pg_notify(1.a(6) epoch 84) v4
|
|
-14> 2013-08-15 20:50:25.265217 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.5 9.4.196.3:6819/4129 4 ==== pg_notify(3.8(6) epoch 84) v4 ==== 830+0+0 (2764059677 0 0) 0x1ea3880 con 0x2145340
|
|
-13> 2013-08-15 20:50:25.265232 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.264735, event: header_read, request: pg_notify(3.8(6) epoch 84) v4
|
|
-12> 2013-08-15 20:50:25.265240 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.264736, event: throttled, request: pg_notify(3.8(6) epoch 84) v4
|
|
-11> 2013-08-15 20:50:25.265247 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.264784, event: all_read, request: pg_notify(3.8(6) epoch 84) v4
|
|
-10> 2013-08-15 20:50:25.265253 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.265229, event: dispatched, request: pg_notify(3.8(6) epoch 84) v4
|
|
-9> 2013-08-15 20:50:25.265263 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.265263, event: waiting_for_osdmap, request: pg_notify(3.8(6) epoch 84) v4
|
|
-8> 2013-08-15 20:50:25.265271 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.265271, event: started, request: pg_notify(3.8(6) epoch 84) v4
|
|
-7> 2013-08-15 20:50:25.265293 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:25.265292, event: done, request: pg_notify(3.8(6) epoch 84) v4
|
|
-6> 2013-08-15 20:50:25.265369 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.127( v 63'13 (0'0,63'13] local-les=62 n=5 ec=56 les/c 62/62 84/84/56) [3,9] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:25.265984 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.1e7( v 63'12 (0'0,63'12] local-les=62 n=2 ec=56 les/c 62/62 84/84/56) [0,9] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:25.266490 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.121( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 84/84/56) [1,9] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:25.267189 7ffd19575700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-2> 2013-08-15 20:50:25.267205 7ffd19575700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
-1> 2013-08-15 20:50:25.267302 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.123( v 63'16 (0'0,63'16] local-les=62 n=2 ec=56 les/c 62/62 84/84/84) [9,4] r=0 lpr=84 pi=61-83/8 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
0> 2013-08-15 20:50:25.268318 7ffd19575700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7ffd19575700 time 2013-08-15 20:50:25.267213
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7ffd22e68f8e]
|
|
5: (clone()+0x6d) [0x7ffd2111ae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:25.273343 7ffd19575700 -1 *** Caught signal (Aborted) **
|
|
in thread 7ffd19575700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7ffd22e70bd0]
|
|
3: (gsignal()+0x37) [0x7ffd21058037]
|
|
4: (abort()+0x148) [0x7ffd2105b698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7ffd21964e8d]
|
|
6: (()+0x5ef76) [0x7ffd21962f76]
|
|
7: (()+0x5efa3) [0x7ffd21962fa3]
|
|
8: (()+0x5f1de) [0x7ffd219631de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7ffd22e68f8e]
|
|
14: (clone()+0x6d) [0x7ffd2111ae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-54> 2013-08-15 20:50:25.268461 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.11e( v 63'14 (0'0,63'14] local-les=62 n=8 ec=56 les/c 62/62 84/84/56) [0,9] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-53> 2013-08-15 20:50:25.269210 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.1d7( v 63'14 (0'0,63'14] local-les=62 n=2 ec=56 les/c 62/62 84/84/84) [9,0] r=0 lpr=84 pi=61-83/8 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-52> 2013-08-15 20:50:25.269863 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.110( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 84/84/56) [1,9] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-51> 2013-08-15 20:50:25.270568 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.1d1( v 63'17 (0'0,63'17] local-les=62 n=7 ec=56 les/c 62/62 84/84/56) [2,9] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-50> 2013-08-15 20:50:25.271276 7ffd0fd62700 1 osd.9 pg_epoch: 84 pg[3.53( v 63'9 (0'0,63'9] local-les=62 n=1 ec=56 les/c 62/62 84/84/84) [9,2] r=0 lpr=84 pi=61-83/8 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-49> 2013-08-15 20:50:25.271918 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.2 9.4.196.1:6807/6887 5 ==== pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4 ==== 3677+0+0 (677549616 0 0) 0x1ea28c0 con 0x2144420
|
|
-48> 2013-08-15 20:50:25.271941 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.266741, event: header_read, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-47> 2013-08-15 20:50:25.271953 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.266742, event: throttled, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-46> 2013-08-15 20:50:25.271962 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.266843, event: all_read, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-45> 2013-08-15 20:50:25.272020 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.271938, event: dispatched, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-44> 2013-08-15 20:50:25.272031 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.272031, event: waiting_for_osdmap, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-43> 2013-08-15 20:50:25.272044 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.272044, event: started, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-42> 2013-08-15 20:50:25.272115 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:25.272115, event: done, request: pg_notify(0.b(5),3.8(5),3.13f(3),3.123(3),3.f2(3) epoch 84) v4
|
|
-41> 2013-08-15 20:50:25.272133 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x21b1440 con 0x21394a0
|
|
-40> 2013-08-15 20:50:25.272149 7ffd13d6a700 3 osd.9 84 handle_osd_map epochs [64,84], i have 84, src has [1,84]
|
|
-39> 2013-08-15 20:50:25.272162 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.1 9.4.196.1:6801/5083 4 ==== pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2 ==== 762+0+0 (3952600624 0 0) 0x1db2780 con 0x2139ce0
|
|
-38> 2013-08-15 20:50:25.272178 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.266876, event: header_read, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-37> 2013-08-15 20:50:25.272189 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.266878, event: throttled, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-36> 2013-08-15 20:50:25.272199 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.266929, event: all_read, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-35> 2013-08-15 20:50:25.272208 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.272176, event: dispatched, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-34> 2013-08-15 20:50:25.272216 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.272216, event: waiting_for_osdmap, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-33> 2013-08-15 20:50:25.272226 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.272226, event: started, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-32> 2013-08-15 20:50:25.272276 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:25.272276, event: done, request: pg_query(3.4,3.110,3.121,3.15a,3.17b,3.185 epoch 84) v2
|
|
-31> 2013-08-15 20:50:25.272290 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.2 9.4.196.1:6807/6887 6 ==== pg_query(3.cc,3.d5,3.16e epoch 84) v2 ==== 387+0+0 (2431999090 0 0) 0x1db2960 con 0x2144420
|
|
-30> 2013-08-15 20:50:25.272304 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.266942, event: header_read, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-29> 2013-08-15 20:50:25.272313 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.266943, event: throttled, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-28> 2013-08-15 20:50:25.272320 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.266998, event: all_read, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-27> 2013-08-15 20:50:25.272327 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.272302, event: dispatched, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-26> 2013-08-15 20:50:25.272336 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.272336, event: waiting_for_osdmap, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-25> 2013-08-15 20:50:25.272344 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.272344, event: started, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-24> 2013-08-15 20:50:25.272494 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 --> 9.4.196.1:6807/6887 -- pg_notify(3.16e epoch 84) v4 -- ?+0 0x22868c0 con 0x2144420
|
|
-23> 2013-08-15 20:50:25.272596 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:25.272596, event: done, request: pg_query(3.cc,3.d5,3.16e epoch 84) v2
|
|
-22> 2013-08-15 20:50:25.272660 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.5 9.4.196.3:6819/4129 5 ==== pg_notify(3.255(1),3.c6(6) epoch 84) v4 ==== 1453+0+0 (35297354 0 0) 0x1ea36c0 con 0x2145340
|
|
-21> 2013-08-15 20:50:25.272678 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.271879, event: header_read, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-20> 2013-08-15 20:50:25.272687 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.271881, event: throttled, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-19> 2013-08-15 20:50:25.272694 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.271943, event: all_read, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-18> 2013-08-15 20:50:25.272701 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.272675, event: dispatched, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-17> 2013-08-15 20:50:25.272710 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.272709, event: waiting_for_osdmap, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-16> 2013-08-15 20:50:25.272718 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.272718, event: started, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-15> 2013-08-15 20:50:25.272752 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:25.272752, event: done, request: pg_notify(3.255(1),3.c6(6) epoch 84) v4
|
|
-14> 2013-08-15 20:50:25.272766 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_notify(3.93(5),3.206(5) epoch 84) v4 ==== 1562+0+0 (414341506 0 0) 0x1ea3c00 con 0x21394a0
|
|
-13> 2013-08-15 20:50:25.272780 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.267295, event: header_read, request: pg_notify(3.93(5),3.206(5) epoch 84) v4
|
|
-12> 2013-08-15 20:50:25.272788 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.267296, event: throttled, request: pg_notify(3.93(5),3.206(5) epoch 84) v4
|
|
-11> 2013-08-15 20:50:25.272796 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.267366, event: all_read, request: pg_notify(3.93(5),3.206(5) epoch 84) v4
|
|
-10> 2013-08-15 20:50:25.272802 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.272778, event: dispatched, request: pg_notify(3.93(5),3.206(5) epoch 84) v4
|
|
-9> 2013-08-15 20:50:25.272809 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.272809, event: waiting_for_osdmap, request: pg_notify(3.93(5),3.206(5) epoch 84) v4
|
|
-8> 2013-08-15 20:50:25.272818 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.272818, event: started, request: pg_notify(3.93(5),3.206(5) epoch 84) v4
|
|
-7> 2013-08-15 20:50:25.272970 7ffd13d6a700 1 osd.9 pg_epoch: 84 pg[3.206( empty local-les=0 n=0 ec=56 les/c 80/80 84/84/84) [9,0] r=0 lpr=84 pi=73-83/5 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-6> 2013-08-15 20:50:25.273007 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 --> 9.4.196.1:6804/5992 -- pg_query(3.206 epoch 84) v2 -- ?+0 0x1db2f00 con 0x21394a0
|
|
-5> 2013-08-15 20:50:25.273028 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 --> 9.4.196.1:6810/7799 -- pg_query(3.206 epoch 84) v2 -- ?+0 0x1db32c0 con 0x2139760
|
|
-4> 2013-08-15 20:50:25.273312 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:25.273312, event: done, request: pg_notify(3.93(5),3.206 epoch 84) v4
|
|
-3> 2013-08-15 20:50:25.273327 7ffd13d6a700 1 -- 9.4.196.3:6807/4117 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_notify(3.8(3),0.b(3),1.a(3),2.9(3),3.17d(5),3.f2(5),3.255(2) epoch 84) v4 ==== 5040+0+0 (136058629 0 0) 0x2286700 con 0x21394a0
|
|
-2> 2013-08-15 20:50:25.273347 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:25.271802, event: header_read, request: pg_notify(3.8(3),0.b(3),1.a(3),2.9(3),3.17d(5),3.f2(5),3.255(2) epoch 84) v4
|
|
-1> 2013-08-15 20:50:25.273358 7ffd13d6a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:25.271804, event: throttled, request: pg_notify(3.8(3),0.b(3),1.a(3),2.9(3),3.17d(5),3.f2(5),3.255(2) epoch 84) v4
|
|
0> 2013-08-15 20:50:25.273343 7ffd19575700 -1 *** Caught signal (Aborted) **
|
|
in thread 7ffd19575700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7ffd22e70bd0]
|
|
3: (gsignal()+0x37) [0x7ffd21058037]
|
|
4: (abort()+0x148) [0x7ffd2105b698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7ffd21964e8d]
|
|
6: (()+0x5ef76) [0x7ffd21962f76]
|
|
7: (()+0x5efa3) [0x7ffd21962fa3]
|
|
8: (()+0x5f1de) [0x7ffd219631de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7ffd22e68f8e]
|
|
14: (clone()+0x6d) [0x7ffd2111ae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:25.448274 7f88a53fa7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8356
|
|
2013-08-15 20:50:25.448822 7f88a53fa7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:25.490888 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:25.490907 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:25.491169 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:25.514787 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:25.514890 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:25.539040 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:25.539962 7f88a53fa7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:25.539979 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.644192 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.644576 7f88a53fa7c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
2013-08-15 20:50:25.676368 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:25.676384 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:25.676656 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:25.694312 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:25.694364 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:25.724580 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:25.725161 7f88a53fa7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:25.725179 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.826459 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.272805 7f889ddce700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:27.272826 7f889ddce700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
2013-08-15 20:50:27.273756 7f889ddce700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f889ddce700 time 2013-08-15 20:50:27.272830
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f88a4dd6f8e]
|
|
5: (clone()+0x6d) [0x7f88a3088e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-243> 2013-08-15 20:50:25.446920 7f88a53fa7c0 5 asok(0x180a0e0) register_command perfcounters_dump hook 0x1804010
|
|
-242> 2013-08-15 20:50:25.446951 7f88a53fa7c0 5 asok(0x180a0e0) register_command 1 hook 0x1804010
|
|
-241> 2013-08-15 20:50:25.446955 7f88a53fa7c0 5 asok(0x180a0e0) register_command perf dump hook 0x1804010
|
|
-240> 2013-08-15 20:50:25.446962 7f88a53fa7c0 5 asok(0x180a0e0) register_command perfcounters_schema hook 0x1804010
|
|
-239> 2013-08-15 20:50:25.446966 7f88a53fa7c0 5 asok(0x180a0e0) register_command 2 hook 0x1804010
|
|
-238> 2013-08-15 20:50:25.446969 7f88a53fa7c0 5 asok(0x180a0e0) register_command perf schema hook 0x1804010
|
|
-237> 2013-08-15 20:50:25.446975 7f88a53fa7c0 5 asok(0x180a0e0) register_command config show hook 0x1804010
|
|
-236> 2013-08-15 20:50:25.446980 7f88a53fa7c0 5 asok(0x180a0e0) register_command config set hook 0x1804010
|
|
-235> 2013-08-15 20:50:25.446984 7f88a53fa7c0 5 asok(0x180a0e0) register_command log flush hook 0x1804010
|
|
-234> 2013-08-15 20:50:25.446988 7f88a53fa7c0 5 asok(0x180a0e0) register_command log dump hook 0x1804010
|
|
-233> 2013-08-15 20:50:25.446993 7f88a53fa7c0 5 asok(0x180a0e0) register_command log reopen hook 0x1804010
|
|
-232> 2013-08-15 20:50:25.448274 7f88a53fa7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8356
|
|
-231> 2013-08-15 20:50:25.448822 7f88a53fa7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-230> 2013-08-15 20:50:25.448965 7f88a53fa7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/8356 need_addr=1
|
|
-229> 2013-08-15 20:50:25.448990 7f88a53fa7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-228> 2013-08-15 20:50:25.448998 7f88a53fa7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6807/8356 need_addr=0
|
|
-227> 2013-08-15 20:50:25.449015 7f88a53fa7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-226> 2013-08-15 20:50:25.449021 7f88a53fa7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6812/8356 need_addr=0
|
|
-225> 2013-08-15 20:50:25.450952 7f88a53fa7c0 5 asok(0x180a0e0) init /var/run/ceph/ceph-osd.9.asok
|
|
-224> 2013-08-15 20:50:25.450968 7f88a53fa7c0 5 asok(0x180a0e0) bind_and_listen /var/run/ceph/ceph-osd.9.asok
|
|
-223> 2013-08-15 20:50:25.451003 7f88a53fa7c0 5 asok(0x180a0e0) register_command 0 hook 0x18020b8
|
|
-222> 2013-08-15 20:50:25.451009 7f88a53fa7c0 5 asok(0x180a0e0) register_command version hook 0x18020b8
|
|
-221> 2013-08-15 20:50:25.451016 7f88a53fa7c0 5 asok(0x180a0e0) register_command git_version hook 0x18020b8
|
|
-220> 2013-08-15 20:50:25.451021 7f88a53fa7c0 5 asok(0x180a0e0) register_command help hook 0x18040d0
|
|
-219> 2013-08-15 20:50:25.451049 7f88a0dd4700 5 asok(0x180a0e0) entry start
|
|
-218> 2013-08-15 20:50:25.490888 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-217> 2013-08-15 20:50:25.490907 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-216> 2013-08-15 20:50:25.491169 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-215> 2013-08-15 20:50:25.514787 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-214> 2013-08-15 20:50:25.514890 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-213> 2013-08-15 20:50:25.539040 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-212> 2013-08-15 20:50:25.539935 7f88a53fa7c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-211> 2013-08-15 20:50:25.539962 7f88a53fa7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-210> 2013-08-15 20:50:25.539979 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-209> 2013-08-15 20:50:25.545677 7f88a53fa7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-208> 2013-08-15 20:50:25.549715 7f88a53fa7c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-207> 2013-08-15 20:50:25.554077 7f88a53fa7c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-206> 2013-08-15 20:50:25.558453 7f88a53fa7c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-205> 2013-08-15 20:50:25.562729 7f88a53fa7c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-204> 2013-08-15 20:50:25.567027 7f88a53fa7c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-203> 2013-08-15 20:50:25.571286 7f88a53fa7c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-202> 2013-08-15 20:50:25.575606 7f88a53fa7c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-201> 2013-08-15 20:50:25.579904 7f88a53fa7c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-200> 2013-08-15 20:50:25.584155 7f88a53fa7c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-199> 2013-08-15 20:50:25.588419 7f88a53fa7c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-198> 2013-08-15 20:50:25.592709 7f88a53fa7c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-197> 2013-08-15 20:50:25.596989 7f88a53fa7c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-196> 2013-08-15 20:50:25.601258 7f88a53fa7c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-195> 2013-08-15 20:50:25.605542 7f88a53fa7c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-194> 2013-08-15 20:50:25.609892 7f88a53fa7c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-193> 2013-08-15 20:50:25.614284 7f88a53fa7c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-192> 2013-08-15 20:50:25.618534 7f88a53fa7c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-191> 2013-08-15 20:50:25.622947 7f88a53fa7c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-190> 2013-08-15 20:50:25.627212 7f88a53fa7c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-189> 2013-08-15 20:50:25.631473 7f88a53fa7c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-188> 2013-08-15 20:50:25.635649 7f88a53fa7c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-187> 2013-08-15 20:50:25.639849 7f88a53fa7c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-186> 2013-08-15 20:50:25.644089 7f88a53fa7c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-185> 2013-08-15 20:50:25.644132 7f88a53fa7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-184> 2013-08-15 20:50:25.644139 7f88a53fa7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-183> 2013-08-15 20:50:25.644148 7f88a53fa7c0 3 journal journal_replay: end of journal, done.
|
|
-182> 2013-08-15 20:50:25.644192 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-181> 2013-08-15 20:50:25.644487 7f889ddce700 1 FileStore::op_tp worker finish
|
|
-180> 2013-08-15 20:50:25.644517 7f889e5cf700 1 FileStore::op_tp worker finish
|
|
-179> 2013-08-15 20:50:25.644576 7f88a53fa7c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
-178> 2013-08-15 20:50:25.644825 7f88a53fa7c0 10 monclient(hunting): build_initial_monmap
|
|
-177> 2013-08-15 20:50:25.644875 7f88a53fa7c0 5 adding auth protocol: cephx
|
|
-176> 2013-08-15 20:50:25.644882 7f88a53fa7c0 5 adding auth protocol: cephx
|
|
-175> 2013-08-15 20:50:25.644964 7f88a53fa7c0 1 -- 0.0.0.0:6805/8356 messenger.start
|
|
-174> 2013-08-15 20:50:25.644991 7f88a53fa7c0 1 -- :/0 messenger.start
|
|
-173> 2013-08-15 20:50:25.645003 7f88a53fa7c0 1 -- 9.4.196.3:6812/8356 messenger.start
|
|
-172> 2013-08-15 20:50:25.645014 7f88a53fa7c0 1 -- 9.4.196.3:6807/8356 messenger.start
|
|
-171> 2013-08-15 20:50:25.645096 7f88a53fa7c0 2 osd.9 0 mounting /var/lib/ceph/osd/ceph-9 /var/lib/ceph/osd/ceph-9/journal
|
|
-170> 2013-08-15 20:50:25.676368 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-169> 2013-08-15 20:50:25.676384 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-168> 2013-08-15 20:50:25.676656 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-167> 2013-08-15 20:50:25.694312 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-166> 2013-08-15 20:50:25.694364 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-165> 2013-08-15 20:50:25.724580 7f88a53fa7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-164> 2013-08-15 20:50:25.725140 7f88a53fa7c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-163> 2013-08-15 20:50:25.725161 7f88a53fa7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-162> 2013-08-15 20:50:25.725179 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-161> 2013-08-15 20:50:25.729494 7f88a53fa7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-160> 2013-08-15 20:50:25.733769 7f88a53fa7c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-159> 2013-08-15 20:50:25.738054 7f88a53fa7c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-158> 2013-08-15 20:50:25.742279 7f88a53fa7c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-157> 2013-08-15 20:50:25.746501 7f88a53fa7c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-156> 2013-08-15 20:50:25.750748 7f88a53fa7c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-155> 2013-08-15 20:50:25.754978 7f88a53fa7c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-154> 2013-08-15 20:50:25.759158 7f88a53fa7c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-153> 2013-08-15 20:50:25.763288 7f88a53fa7c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-152> 2013-08-15 20:50:25.767546 7f88a53fa7c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-151> 2013-08-15 20:50:25.772037 7f88a53fa7c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-150> 2013-08-15 20:50:25.776176 7f88a53fa7c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-149> 2013-08-15 20:50:25.780550 7f88a53fa7c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-148> 2013-08-15 20:50:25.784805 7f88a53fa7c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-147> 2013-08-15 20:50:25.789146 7f88a53fa7c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-146> 2013-08-15 20:50:25.793339 7f88a53fa7c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-145> 2013-08-15 20:50:25.797498 7f88a53fa7c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-144> 2013-08-15 20:50:25.801609 7f88a53fa7c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-143> 2013-08-15 20:50:25.805767 7f88a53fa7c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-142> 2013-08-15 20:50:25.809802 7f88a53fa7c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-141> 2013-08-15 20:50:25.813856 7f88a53fa7c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-140> 2013-08-15 20:50:25.817942 7f88a53fa7c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-139> 2013-08-15 20:50:25.821959 7f88a53fa7c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-138> 2013-08-15 20:50:25.826393 7f88a53fa7c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-137> 2013-08-15 20:50:25.826424 7f88a53fa7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-136> 2013-08-15 20:50:25.826431 7f88a53fa7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-135> 2013-08-15 20:50:25.826432 7f88a53fa7c0 3 journal journal_replay: end of journal, done.
|
|
-134> 2013-08-15 20:50:25.826459 7f88a53fa7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-133> 2013-08-15 20:50:25.826620 7f88a53fa7c0 2 osd.9 0 boot
|
|
-132> 2013-08-15 20:50:25.853757 7f88a53fa7c0 2 osd.9 63 superblock: i am osd.9
|
|
-131> 2013-08-15 20:50:25.853822 7f88a53fa7c0 1 accepter.accepter.start
|
|
-130> 2013-08-15 20:50:25.853875 7f88a53fa7c0 1 accepter.accepter.start
|
|
-129> 2013-08-15 20:50:25.853927 7f88a53fa7c0 1 accepter.accepter.start
|
|
-128> 2013-08-15 20:50:25.853958 7f88a53fa7c0 10 monclient(hunting): init
|
|
-127> 2013-08-15 20:50:25.853970 7f88a53fa7c0 5 adding auth protocol: cephx
|
|
-126> 2013-08-15 20:50:25.853972 7f88a53fa7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-125> 2013-08-15 20:50:25.854175 7f88943b9700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62c80 sd=30 :6807 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:60251/0
|
|
-124> 2013-08-15 20:50:25.854178 7f88945bb700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62500 sd=29 :6807 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.3:48787/0
|
|
-123> 2013-08-15 20:50:25.854196 7f88944ba700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62280 sd=28 :6807 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.3:48789/0
|
|
-122> 2013-08-15 20:50:25.854179 7f88a538f700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62000 sd=27 :6807 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.3:48786/0
|
|
-121> 2013-08-15 20:50:25.854217 7f88942b8700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62a00 sd=31 :6807 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:60252/0
|
|
-120> 2013-08-15 20:50:25.854365 7f88a53fa7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-9/keyring
|
|
-119> 2013-08-15 20:50:25.854482 7f88941b7700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62780 sd=32 :6807 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:60254/0
|
|
-118> 2013-08-15 20:50:25.854562 7f88938b5700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a63680 sd=33 :6807 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60255/0
|
|
-117> 2013-08-15 20:50:25.854735 7f88a53fa7c0 5 asok(0x180a0e0) register_command dump_ops_in_flight hook 0x1804710
|
|
-116> 2013-08-15 20:50:25.854749 7f88a53fa7c0 5 asok(0x180a0e0) register_command dump_historic_ops hook 0x1804710
|
|
-115> 2013-08-15 20:50:25.854754 7f88a53fa7c0 5 asok(0x180a0e0) register_command dump_op_pq_state hook 0x1804710
|
|
-114> 2013-08-15 20:50:25.854759 7f88a53fa7c0 5 asok(0x180a0e0) register_command setomapval hook 0x19f7360
|
|
-113> 2013-08-15 20:50:25.854764 7f88a53fa7c0 5 asok(0x180a0e0) register_command rmomapkey hook 0x19f7360
|
|
-112> 2013-08-15 20:50:25.854769 7f88a53fa7c0 5 asok(0x180a0e0) register_command setomapheader hook 0x19f7360
|
|
-111> 2013-08-15 20:50:25.854772 7f88a53fa7c0 5 asok(0x180a0e0) register_command getomap hook 0x19f7360
|
|
-110> 2013-08-15 20:50:25.854784 7f88a53fa7c0 5 asok(0x180a0e0) register_command truncobj hook 0x19f7360
|
|
-109> 2013-08-15 20:50:25.854793 7f88a53fa7c0 5 asok(0x180a0e0) register_command injectdataerr hook 0x19f7360
|
|
-108> 2013-08-15 20:50:25.854743 7f8892fb3700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a63400 sd=34 :6807 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:60256/0
|
|
-107> 2013-08-15 20:50:25.854797 7f88a53fa7c0 5 asok(0x180a0e0) register_command injectmdataerr hook 0x19f7360
|
|
-106> 2013-08-15 20:50:25.854801 7f88906ad700 5 osd.9 63 heartbeat: osd_stat(1026 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-105> 2013-08-15 20:50:25.854854 7f88a53fa7c0 10 monclient(hunting): _reopen_session
|
|
-104> 2013-08-15 20:50:25.854880 7f88a53fa7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x184bb80 addr 9.4.196.1:6789/0
|
|
-103> 2013-08-15 20:50:25.854893 7f88a53fa7c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-102> 2013-08-15 20:50:25.854899 7f88a53fa7c0 1 -- 0.0.0.0:6805/8356 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1816b40 con 0x184bb80
|
|
-101> 2013-08-15 20:50:25.854913 7f88a53fa7c0 10 monclient(hunting): renew_subs
|
|
-100> 2013-08-15 20:50:25.855169 7f88938b5700 1 -- 9.4.196.3:6805/8356 learned my addr 9.4.196.3:6805/8356
|
|
-99> 2013-08-15 20:50:25.855324 7f8897dc2700 10 monclient(hunting): renew_subs
|
|
-98> 2013-08-15 20:50:25.855711 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x188e1e0 con 0x184bb80
|
|
-97> 2013-08-15 20:50:25.855738 7f8897dc2700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-96> 2013-08-15 20:50:25.855754 7f8897dc2700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-95> 2013-08-15 20:50:25.855758 7f8897dc2700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-94> 2013-08-15 20:50:25.855775 7f8897dc2700 1 monclient(hunting): found mon.zc2store
|
|
-93> 2013-08-15 20:50:25.855794 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2773490275 0 0) 0x1a70200 con 0x184bb80
|
|
-92> 2013-08-15 20:50:25.855814 7f8897dc2700 10 monclient: my global_id is 4371
|
|
-91> 2013-08-15 20:50:25.855967 7f8897dc2700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-90> 2013-08-15 20:50:25.855977 7f8897dc2700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1817d40 con 0x184bb80
|
|
-89> 2013-08-15 20:50:25.856597 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (105823370 0 0) 0x1a70000 con 0x184bb80
|
|
-88> 2013-08-15 20:50:25.856666 7f8897dc2700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-87> 2013-08-15 20:50:25.856674 7f8897dc2700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1a76240 con 0x184bb80
|
|
-86> 2013-08-15 20:50:25.857340 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (527651788 0 0) 0x1a70400 con 0x184bb80
|
|
-85> 2013-08-15 20:50:25.857404 7f8897dc2700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-84> 2013-08-15 20:50:25.857411 7f8897dc2700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x184c380 con 0x184bb80
|
|
-83> 2013-08-15 20:50:25.857430 7f8897dc2700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-82> 2013-08-15 20:50:25.857433 7f8897dc2700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x184c540 con 0x184bb80
|
|
-81> 2013-08-15 20:50:25.857441 7f8897dc2700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:55.857441)
|
|
-80> 2013-08-15 20:50:25.857449 7f8897dc2700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-79> 2013-08-15 20:50:25.857451 7f8897dc2700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1a76000 con 0x184bb80
|
|
-78> 2013-08-15 20:50:25.857486 7f88a53fa7c0 5 monclient: authenticate success, global_id 4371
|
|
-77> 2013-08-15 20:50:25.857498 7f88a53fa7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:55.857497)
|
|
-76> 2013-08-15 20:50:25.857913 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x188e5a0 con 0x184bb80
|
|
-75> 2013-08-15 20:50:25.857932 7f8897dc2700 10 monclient: handle_monmap mon_map v1
|
|
-74> 2013-08-15 20:50:25.857941 7f8897dc2700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-73> 2013-08-15 20:50:25.857943 7f8897dc2700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-72> 2013-08-15 20:50:25.857961 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x184c380 con 0x184bb80
|
|
-71> 2013-08-15 20:50:25.857970 7f8897dc2700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:25.854916 renew after 2013-08-15 20:52:55.854916
|
|
-70> 2013-08-15 20:50:25.857984 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x188e3c0 con 0x184bb80
|
|
-69> 2013-08-15 20:50:25.857991 7f8897dc2700 10 monclient: handle_monmap mon_map v1
|
|
-68> 2013-08-15 20:50:25.857996 7f8897dc2700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-67> 2013-08-15 20:50:25.857998 7f8897dc2700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-66> 2013-08-15 20:50:25.858010 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x184c700 con 0x184bb80
|
|
-65> 2013-08-15 20:50:25.858018 7f8897dc2700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-64> 2013-08-15 20:50:25.858053 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1029180693 0 0) 0x1a78600 con 0x184bb80
|
|
-63> 2013-08-15 20:50:25.858118 7f8897dc2700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:55.858117)
|
|
-62> 2013-08-15 20:50:25.858158 7f88a53fa7c0 10 monclient: wait_auth_rotating done
|
|
-61> 2013-08-15 20:50:25.858660 7f88a53fa7c0 10 monclient: get_version osdmap
|
|
-60> 2013-08-15 20:50:25.858668 7f88a53fa7c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-59> 2013-08-15 20:50:25.858705 7f88a53fa7c0 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x18729c0 con 0x184bb80
|
|
-58> 2013-08-15 20:50:25.858989 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=84) v2 ==== 24+0+0 (2076809473 0 0) 0x184c540 con 0x184bb80
|
|
-57> 2013-08-15 20:50:25.859038 7f88937b4700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-56> 2013-08-15 20:50:25.859052 7f88937b4700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- osd_boot(osd.9 booted 0 v63) v3 -- ?+0 0x1a9a000 con 0x184bb80
|
|
-55> 2013-08-15 20:50:26.254526 7f888eaa6700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a63180 sd=33 :6807 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60299/0
|
|
-54> 2013-08-15 20:50:26.254583 7f888e9a5700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62f00 sd=34 :6807 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:48811/0
|
|
-53> 2013-08-15 20:50:26.254614 7f888e8a4700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a63b80 sd=35 :6807 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:48812/0
|
|
-52> 2013-08-15 20:50:26.254674 7f888e6a2700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a62a00 sd=37 :6807 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:60300/0
|
|
-51> 2013-08-15 20:50:26.254696 7f888e7a3700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a63900 sd=36 :6807 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.3:48814/0
|
|
-50> 2013-08-15 20:50:26.254731 7f888e5a1700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a96c80 sd=38 :6807 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:60301/0
|
|
-49> 2013-08-15 20:50:26.254852 7f888e9a5700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a96a00 sd=39 :6807 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.1:60302/0
|
|
-48> 2013-08-15 20:50:26.255028 7f888e5a1700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a96780 sd=33 :6807 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60303/0
|
|
-47> 2013-08-15 20:50:27.262765 7f888e5a1700 1 -- 9.4.196.3:6812/8356 >> :/0 pipe(0x1a62a00 sd=33 :6812 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:40064/0
|
|
-46> 2013-08-15 20:50:27.264287 7f888e6a2700 1 -- 9.4.196.3:6812/8356 >> :/0 pipe(0x1a63180 sd=34 :6812 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:40065/0
|
|
-45> 2013-08-15 20:50:27.267361 7f8896dc0700 1 -- 9.4.196.3:6807/8356 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(85..86 src has 1..86) v3 ==== 6767+0+0 (1565733723 0 0) 0x1a77200 con 0x1a651e0
|
|
-44> 2013-08-15 20:50:27.267418 7f8896dc0700 3 osd.9 63 handle_osd_map epochs [85,86], i have 63, src has [1,86]
|
|
-43> 2013-08-15 20:50:27.267428 7f8896dc0700 10 monclient: renew_subs
|
|
-42> 2013-08-15 20:50:27.267432 7f8896dc0700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-41> 2013-08-15 20:50:27.267442 7f8896dc0700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x184c8c0 con 0x184bb80
|
|
-40> 2013-08-15 20:50:27.267463 7f8896dc0700 1 -- 9.4.196.3:6807/8356 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(0.1,1.0 epoch 86) v2 ==== 262+0+0 (689273710 0 0) 0x188e1e0 con 0x1a651e0
|
|
-39> 2013-08-15 20:50:27.267478 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.267347, event: header_read, request: pg_query(0.1,1.0 epoch 86) v2
|
|
-38> 2013-08-15 20:50:27.267492 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.267348, event: throttled, request: pg_query(0.1,1.0 epoch 86) v2
|
|
-37> 2013-08-15 20:50:27.267501 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.267383, event: all_read, request: pg_query(0.1,1.0 epoch 86) v2
|
|
-36> 2013-08-15 20:50:27.267522 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.267472, event: dispatched, request: pg_query(0.1,1.0 epoch 86) v2
|
|
-35> 2013-08-15 20:50:27.267528 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.267528, event: waiting_for_osdmap, request: pg_query(0.1,1.0 epoch 86) v2
|
|
-34> 2013-08-15 20:50:27.267540 7f8896dc0700 10 monclient: renew_subs
|
|
-33> 2013-08-15 20:50:27.267544 7f8896dc0700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-32> 2013-08-15 20:50:27.267549 7f8896dc0700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x184cc40 con 0x184bb80
|
|
-31> 2013-08-15 20:50:27.267560 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.267560, event: wait for new map, request: pg_query(0.1,1.0 epoch 86) v2
|
|
-30> 2013-08-15 20:50:27.267770 7f8896dc0700 1 -- 9.4.196.3:6807/8356 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(85..86 src has 1..86) v3 ==== 6767+0+0 (1565733723 0 0) 0x1a778c0 con 0x1a64420
|
|
-29> 2013-08-15 20:50:27.267794 7f8896dc0700 3 osd.9 63 handle_osd_map epochs [85,86], i have 63, src has [1,86]
|
|
-28> 2013-08-15 20:50:27.267805 7f8896dc0700 10 monclient: renew_subs
|
|
-27> 2013-08-15 20:50:27.267811 7f8896dc0700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-26> 2013-08-15 20:50:27.267819 7f8896dc0700 1 -- 9.4.196.3:6805/8356 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x184ca80 con 0x184bb80
|
|
-25> 2013-08-15 20:50:27.267843 7f8896dc0700 1 -- 9.4.196.3:6807/8356 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(3.177,3.17d epoch 86) v2 ==== 262+0+0 (3426006378 0 0) 0x188e5a0 con 0x1a64420
|
|
-24> 2013-08-15 20:50:27.267857 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.267767, event: header_read, request: pg_query(3.177,3.17d epoch 86) v2
|
|
-23> 2013-08-15 20:50:27.267868 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.267768, event: throttled, request: pg_query(3.177,3.17d epoch 86) v2
|
|
-22> 2013-08-15 20:50:27.267878 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.267812, event: all_read, request: pg_query(3.177,3.17d epoch 86) v2
|
|
-21> 2013-08-15 20:50:27.267887 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.267854, event: dispatched, request: pg_query(3.177,3.17d epoch 86) v2
|
|
-20> 2013-08-15 20:50:27.267897 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.267897, event: waiting_for_osdmap, request: pg_query(3.177,3.17d epoch 86) v2
|
|
-19> 2013-08-15 20:50:27.267908 7f8896dc0700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.267908, event: wait for new map, request: pg_query(3.177,3.17d epoch 86) v2
|
|
-18> 2013-08-15 20:50:27.268557 7f888eaa6700 1 -- 9.4.196.3:6807/8356 >> :/0 pipe(0x1a96500 sd=35 :6807 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:48836/0
|
|
-17> 2013-08-15 20:50:27.268907 7f8897dc2700 1 -- 9.4.196.3:6805/8356 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..85 src has 1..85) v3 ==== 18379+0+0 (600021662 0 0) 0x1a778c0 con 0x184bb80
|
|
-16> 2013-08-15 20:50:27.268929 7f8897dc2700 3 osd.9 63 handle_osd_map epochs [64,85], i have 63, src has [1,85]
|
|
-15> 2013-08-15 20:50:27.270083 7f8896dc0700 1 -- 9.4.196.3:6807/8356 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(85..86 src has 1..86) v3 ==== 6767+0+0 (1565733723 0 0) 0x1ad8480 con 0x1a64160
|
|
-14> 2013-08-15 20:50:27.271472 7f888e4a0700 1 -- 9.4.196.3:6812/8356 >> :/0 pipe(0x1a63900 sd=36 :6812 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:40066/0
|
|
-13> 2013-08-15 20:50:27.271670 7f888e39f700 1 -- 9.4.196.3:6812/8356 >> :/0 pipe(0x1a96780 sd=37 :6812 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:40067/0
|
|
-12> 2013-08-15 20:50:27.272278 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-11> 2013-08-15 20:50:27.272290 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-10> 2013-08-15 20:50:27.272295 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-9> 2013-08-15 20:50:27.272311 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-8> 2013-08-15 20:50:27.272321 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-7> 2013-08-15 20:50:27.272325 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-6> 2013-08-15 20:50:27.272346 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-5> 2013-08-15 20:50:27.272351 7f8897dc2700 1 -- 9.4.196.3:6807/8356 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-4> 2013-08-15 20:50:27.272805 7f889ddce700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-3> 2013-08-15 20:50:27.272826 7f889ddce700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
-2> 2013-08-15 20:50:27.273275 7f88926b1700 1 osd.9 pg_epoch: 85 pg[3.127( v 63'13 (0'0,63'13] local-les=62 n=5 ec=56 les/c 62/62 85/85/56) [3] r=-1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:27.273398 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.64( v 63'17 (0'0,63'17] local-les=62 n=5 ec=56 les/c 62/62 85/85/56) [4] r=-1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:27.273756 7f889ddce700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f889ddce700 time 2013-08-15 20:50:27.272830
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f88a4dd6f8e]
|
|
5: (clone()+0x6d) [0x7f88a3088e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:27.277392 7f889ddce700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f889ddce700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f88a4ddebd0]
|
|
3: (gsignal()+0x37) [0x7f88a2fc6037]
|
|
4: (abort()+0x148) [0x7f88a2fc9698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f88a38d2e8d]
|
|
6: (()+0x5ef76) [0x7f88a38d0f76]
|
|
7: (()+0x5efa3) [0x7f88a38d0fa3]
|
|
8: (()+0x5f1de) [0x7f88a38d11de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f88a4dd6f8e]
|
|
14: (clone()+0x6d) [0x7f88a3088e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-7> 2013-08-15 20:50:27.274323 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.1eb( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 85/85/85) [3] r=-1 lpr=85 pi=61-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-6> 2013-08-15 20:50:27.274970 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.123( v 63'16 (0'0,63'16] local-les=62 n=2 ec=56 les/c 62/62 85/85/85) [4] r=-1 lpr=85 pi=61-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:27.275464 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.11e( v 63'14 (0'0,63'14] local-les=62 n=8 ec=56 les/c 62/62 85/85/56) [0] r=-1 lpr=85 pi=56-84/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:27.275982 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.1d7( v 63'14 (0'0,63'14] local-les=62 n=2 ec=56 les/c 62/62 85/85/85) [0] r=-1 lpr=85 pi=61-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:27.276437 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.110( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 85/85/56) [1] r=-1 lpr=85 pi=56-84/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:27.276911 7f88926b1700 1 osd.9 pg_epoch: 85 pg[3.1e7( v 63'12 (0'0,63'12] local-les=62 n=2 ec=56 les/c 62/62 85/85/56) [0] r=-1 lpr=85 pi=56-84/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:27.276955 7f8892eb2700 1 osd.9 pg_epoch: 85 pg[3.1d1( v 63'17 (0'0,63'17] local-les=62 n=7 ec=56 les/c 62/62 85/85/56) [2] r=-1 lpr=85 pi=56-84/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:27.277392 7f889ddce700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f889ddce700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f88a4ddebd0]
|
|
3: (gsignal()+0x37) [0x7f88a2fc6037]
|
|
4: (abort()+0x148) [0x7f88a2fc9698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f88a38d2e8d]
|
|
6: (()+0x5ef76) [0x7f88a38d0f76]
|
|
7: (()+0x5efa3) [0x7f88a38d0fa3]
|
|
8: (()+0x5f1de) [0x7f88a38d11de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f88a4dd6f8e]
|
|
14: (clone()+0x6d) [0x7f88a3088e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:27.428300 7fa3477527c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8749
|
|
2013-08-15 20:50:27.428848 7fa3477527c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:27.476486 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:27.476498 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:27.476749 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:27.494395 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:27.494466 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:27.518838 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:27.519782 7fa3477527c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:27.519830 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.616707 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.617192 7fa3477527c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
2013-08-15 20:50:27.650130 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:27.650140 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:27.650376 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:27.667999 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:27.668079 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:27.698327 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:27.698928 7fa3477527c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:27.698959 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.793798 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:29.766522 7fa340126700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:29.766540 7fa340126700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
2013-08-15 20:50:29.767537 7fa340126700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fa340126700 time 2013-08-15 20:50:29.766545
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fa34712ef8e]
|
|
5: (clone()+0x6d) [0x7fa3453e0e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-217> 2013-08-15 20:50:27.426961 7fa3477527c0 5 asok(0x34120e0) register_command perfcounters_dump hook 0x340c010
|
|
-216> 2013-08-15 20:50:27.426989 7fa3477527c0 5 asok(0x34120e0) register_command 1 hook 0x340c010
|
|
-215> 2013-08-15 20:50:27.426996 7fa3477527c0 5 asok(0x34120e0) register_command perf dump hook 0x340c010
|
|
-214> 2013-08-15 20:50:27.427006 7fa3477527c0 5 asok(0x34120e0) register_command perfcounters_schema hook 0x340c010
|
|
-213> 2013-08-15 20:50:27.427010 7fa3477527c0 5 asok(0x34120e0) register_command 2 hook 0x340c010
|
|
-212> 2013-08-15 20:50:27.427012 7fa3477527c0 5 asok(0x34120e0) register_command perf schema hook 0x340c010
|
|
-211> 2013-08-15 20:50:27.427018 7fa3477527c0 5 asok(0x34120e0) register_command config show hook 0x340c010
|
|
-210> 2013-08-15 20:50:27.427023 7fa3477527c0 5 asok(0x34120e0) register_command config set hook 0x340c010
|
|
-209> 2013-08-15 20:50:27.427027 7fa3477527c0 5 asok(0x34120e0) register_command log flush hook 0x340c010
|
|
-208> 2013-08-15 20:50:27.427030 7fa3477527c0 5 asok(0x34120e0) register_command log dump hook 0x340c010
|
|
-207> 2013-08-15 20:50:27.427035 7fa3477527c0 5 asok(0x34120e0) register_command log reopen hook 0x340c010
|
|
-206> 2013-08-15 20:50:27.428300 7fa3477527c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8749
|
|
-205> 2013-08-15 20:50:27.428848 7fa3477527c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-204> 2013-08-15 20:50:27.428992 7fa3477527c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/8749 need_addr=1
|
|
-203> 2013-08-15 20:50:27.429022 7fa3477527c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-202> 2013-08-15 20:50:27.429030 7fa3477527c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6813/8749 need_addr=0
|
|
-201> 2013-08-15 20:50:27.429052 7fa3477527c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-200> 2013-08-15 20:50:27.429057 7fa3477527c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6824/8749 need_addr=0
|
|
-199> 2013-08-15 20:50:27.430842 7fa3477527c0 5 asok(0x34120e0) init /var/run/ceph/ceph-osd.9.asok
|
|
-198> 2013-08-15 20:50:27.430868 7fa3477527c0 5 asok(0x34120e0) bind_and_listen /var/run/ceph/ceph-osd.9.asok
|
|
-197> 2013-08-15 20:50:27.430900 7fa3477527c0 5 asok(0x34120e0) register_command 0 hook 0x340a0b8
|
|
-196> 2013-08-15 20:50:27.430906 7fa3477527c0 5 asok(0x34120e0) register_command version hook 0x340a0b8
|
|
-195> 2013-08-15 20:50:27.430912 7fa3477527c0 5 asok(0x34120e0) register_command git_version hook 0x340a0b8
|
|
-194> 2013-08-15 20:50:27.430917 7fa3477527c0 5 asok(0x34120e0) register_command help hook 0x340c0d0
|
|
-193> 2013-08-15 20:50:27.430945 7fa34312c700 5 asok(0x34120e0) entry start
|
|
-192> 2013-08-15 20:50:27.476486 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-191> 2013-08-15 20:50:27.476498 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-190> 2013-08-15 20:50:27.476749 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-189> 2013-08-15 20:50:27.494395 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-188> 2013-08-15 20:50:27.494466 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-187> 2013-08-15 20:50:27.518838 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-186> 2013-08-15 20:50:27.519744 7fa3477527c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-185> 2013-08-15 20:50:27.519782 7fa3477527c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-184> 2013-08-15 20:50:27.519830 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-183> 2013-08-15 20:50:27.525246 7fa3477527c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-182> 2013-08-15 20:50:27.529387 7fa3477527c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-181> 2013-08-15 20:50:27.533410 7fa3477527c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-180> 2013-08-15 20:50:27.537408 7fa3477527c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-179> 2013-08-15 20:50:27.541407 7fa3477527c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-178> 2013-08-15 20:50:27.545393 7fa3477527c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-177> 2013-08-15 20:50:27.549377 7fa3477527c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-176> 2013-08-15 20:50:27.553398 7fa3477527c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-175> 2013-08-15 20:50:27.557388 7fa3477527c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-174> 2013-08-15 20:50:27.561392 7fa3477527c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-173> 2013-08-15 20:50:27.565395 7fa3477527c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-172> 2013-08-15 20:50:27.569377 7fa3477527c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-171> 2013-08-15 20:50:27.573336 7fa3477527c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-170> 2013-08-15 20:50:27.577312 7fa3477527c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-169> 2013-08-15 20:50:27.581418 7fa3477527c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-168> 2013-08-15 20:50:27.585392 7fa3477527c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-167> 2013-08-15 20:50:27.589333 7fa3477527c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-166> 2013-08-15 20:50:27.593325 7fa3477527c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-165> 2013-08-15 20:50:27.597301 7fa3477527c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-164> 2013-08-15 20:50:27.601202 7fa3477527c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-163> 2013-08-15 20:50:27.605083 7fa3477527c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-162> 2013-08-15 20:50:27.608909 7fa3477527c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-161> 2013-08-15 20:50:27.612800 7fa3477527c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-160> 2013-08-15 20:50:27.616607 7fa3477527c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-159> 2013-08-15 20:50:27.616649 7fa3477527c0 2 journal No further valid entries found, journal is most likely valid
|
|
-158> 2013-08-15 20:50:27.616661 7fa3477527c0 2 journal No further valid entries found, journal is most likely valid
|
|
-157> 2013-08-15 20:50:27.616668 7fa3477527c0 3 journal journal_replay: end of journal, done.
|
|
-156> 2013-08-15 20:50:27.616707 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-155> 2013-08-15 20:50:27.617082 7fa340126700 1 FileStore::op_tp worker finish
|
|
-154> 2013-08-15 20:50:27.617098 7fa340927700 1 FileStore::op_tp worker finish
|
|
-153> 2013-08-15 20:50:27.617192 7fa3477527c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
-152> 2013-08-15 20:50:27.617438 7fa3477527c0 10 monclient(hunting): build_initial_monmap
|
|
-151> 2013-08-15 20:50:27.617485 7fa3477527c0 5 adding auth protocol: cephx
|
|
-150> 2013-08-15 20:50:27.617491 7fa3477527c0 5 adding auth protocol: cephx
|
|
-149> 2013-08-15 20:50:27.617573 7fa3477527c0 1 -- 0.0.0.0:6805/8749 messenger.start
|
|
-148> 2013-08-15 20:50:27.617599 7fa3477527c0 1 -- :/0 messenger.start
|
|
-147> 2013-08-15 20:50:27.617610 7fa3477527c0 1 -- 9.4.196.3:6824/8749 messenger.start
|
|
-146> 2013-08-15 20:50:27.617621 7fa3477527c0 1 -- 9.4.196.3:6813/8749 messenger.start
|
|
-145> 2013-08-15 20:50:27.617698 7fa3477527c0 2 osd.9 0 mounting /var/lib/ceph/osd/ceph-9 /var/lib/ceph/osd/ceph-9/journal
|
|
-144> 2013-08-15 20:50:27.650130 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-143> 2013-08-15 20:50:27.650140 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-142> 2013-08-15 20:50:27.650376 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-141> 2013-08-15 20:50:27.667999 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-140> 2013-08-15 20:50:27.668079 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-139> 2013-08-15 20:50:27.698327 7fa3477527c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-138> 2013-08-15 20:50:27.698894 7fa3477527c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-137> 2013-08-15 20:50:27.698928 7fa3477527c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-136> 2013-08-15 20:50:27.698959 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-135> 2013-08-15 20:50:27.703086 7fa3477527c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-134> 2013-08-15 20:50:27.707047 7fa3477527c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-133> 2013-08-15 20:50:27.710997 7fa3477527c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-132> 2013-08-15 20:50:27.715001 7fa3477527c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-131> 2013-08-15 20:50:27.718951 7fa3477527c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-130> 2013-08-15 20:50:27.722937 7fa3477527c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-129> 2013-08-15 20:50:27.726875 7fa3477527c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-128> 2013-08-15 20:50:27.730826 7fa3477527c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-127> 2013-08-15 20:50:27.734706 7fa3477527c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-126> 2013-08-15 20:50:27.738617 7fa3477527c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-125> 2013-08-15 20:50:27.742514 7fa3477527c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-124> 2013-08-15 20:50:27.746358 7fa3477527c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-123> 2013-08-15 20:50:27.750218 7fa3477527c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-122> 2013-08-15 20:50:27.754496 7fa3477527c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-121> 2013-08-15 20:50:27.758470 7fa3477527c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-120> 2013-08-15 20:50:27.762660 7fa3477527c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-119> 2013-08-15 20:50:27.766724 7fa3477527c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-118> 2013-08-15 20:50:27.770684 7fa3477527c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-117> 2013-08-15 20:50:27.774504 7fa3477527c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-116> 2013-08-15 20:50:27.778392 7fa3477527c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-115> 2013-08-15 20:50:27.782218 7fa3477527c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-114> 2013-08-15 20:50:27.786114 7fa3477527c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-113> 2013-08-15 20:50:27.789891 7fa3477527c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-112> 2013-08-15 20:50:27.793713 7fa3477527c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-111> 2013-08-15 20:50:27.793752 7fa3477527c0 2 journal No further valid entries found, journal is most likely valid
|
|
-110> 2013-08-15 20:50:27.793757 7fa3477527c0 2 journal No further valid entries found, journal is most likely valid
|
|
-109> 2013-08-15 20:50:27.793758 7fa3477527c0 3 journal journal_replay: end of journal, done.
|
|
-108> 2013-08-15 20:50:27.793798 7fa3477527c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-107> 2013-08-15 20:50:27.794020 7fa3477527c0 2 osd.9 0 boot
|
|
-106> 2013-08-15 20:50:27.819652 7fa3477527c0 2 osd.9 63 superblock: i am osd.9
|
|
-105> 2013-08-15 20:50:27.819693 7fa3477527c0 1 accepter.accepter.start
|
|
-104> 2013-08-15 20:50:27.819764 7fa3477527c0 1 accepter.accepter.start
|
|
-103> 2013-08-15 20:50:27.819805 7fa3477527c0 1 accepter.accepter.start
|
|
-102> 2013-08-15 20:50:27.819823 7fa3477527c0 10 monclient(hunting): init
|
|
-101> 2013-08-15 20:50:27.819829 7fa3477527c0 5 adding auth protocol: cephx
|
|
-100> 2013-08-15 20:50:27.819832 7fa3477527c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-99> 2013-08-15 20:50:27.819972 7fa3477527c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-9/keyring
|
|
-98> 2013-08-15 20:50:27.820125 7fa3477527c0 5 asok(0x34120e0) register_command dump_ops_in_flight hook 0x340c710
|
|
-97> 2013-08-15 20:50:27.820146 7fa3477527c0 5 asok(0x34120e0) register_command dump_historic_ops hook 0x340c710
|
|
-96> 2013-08-15 20:50:27.820149 7fa3477527c0 5 asok(0x34120e0) register_command dump_op_pq_state hook 0x340c710
|
|
-95> 2013-08-15 20:50:27.820153 7fa3477527c0 5 asok(0x34120e0) register_command setomapval hook 0x35ff360
|
|
-94> 2013-08-15 20:50:27.820155 7fa3477527c0 5 asok(0x34120e0) register_command rmomapkey hook 0x35ff360
|
|
-93> 2013-08-15 20:50:27.820158 7fa3477527c0 5 asok(0x34120e0) register_command setomapheader hook 0x35ff360
|
|
-92> 2013-08-15 20:50:27.820160 7fa3477527c0 5 asok(0x34120e0) register_command getomap hook 0x35ff360
|
|
-91> 2013-08-15 20:50:27.820163 7fa3477527c0 5 asok(0x34120e0) register_command truncobj hook 0x35ff360
|
|
-90> 2013-08-15 20:50:27.820169 7fa3477527c0 5 asok(0x34120e0) register_command injectdataerr hook 0x35ff360
|
|
-89> 2013-08-15 20:50:27.820171 7fa3477527c0 5 asok(0x34120e0) register_command injectmdataerr hook 0x35ff360
|
|
-88> 2013-08-15 20:50:27.820211 7fa3477527c0 10 monclient(hunting): _reopen_session
|
|
-87> 2013-08-15 20:50:27.820220 7fa33310c700 5 osd.9 63 heartbeat: osd_stat(1026 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-86> 2013-08-15 20:50:27.820272 7fa3477527c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x366e840 addr 9.4.196.1:6789/0
|
|
-85> 2013-08-15 20:50:27.820287 7fa3477527c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-84> 2013-08-15 20:50:27.820292 7fa3477527c0 1 -- 0.0.0.0:6805/8749 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x341eb40 con 0x366e840
|
|
-83> 2013-08-15 20:50:27.820305 7fa3477527c0 10 monclient(hunting): renew_subs
|
|
-82> 2013-08-15 20:50:27.820799 7fa3476e7700 1 -- 9.4.196.3:6805/8749 learned my addr 9.4.196.3:6805/8749
|
|
-81> 2013-08-15 20:50:27.821023 7fa33a11a700 10 monclient(hunting): renew_subs
|
|
-80> 2013-08-15 20:50:27.873244 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x34941e0 con 0x366e840
|
|
-79> 2013-08-15 20:50:27.873276 7fa33a11a700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-78> 2013-08-15 20:50:27.873306 7fa33a11a700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-77> 2013-08-15 20:50:27.873308 7fa33a11a700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-76> 2013-08-15 20:50:27.873329 7fa33a11a700 1 monclient(hunting): found mon.zc2store
|
|
-75> 2013-08-15 20:50:27.873335 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (397523502 0 0) 0x3676200 con 0x366e840
|
|
-74> 2013-08-15 20:50:27.873369 7fa33a11a700 10 monclient: my global_id is 4377
|
|
-73> 2013-08-15 20:50:27.873485 7fa33a11a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-72> 2013-08-15 20:50:27.873495 7fa33a11a700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x341fd40 con 0x366e840
|
|
-71> 2013-08-15 20:50:27.873950 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (2589160638 0 0) 0x3676000 con 0x366e840
|
|
-70> 2013-08-15 20:50:27.874040 7fa33a11a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-69> 2013-08-15 20:50:27.874046 7fa33a11a700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x367c240 con 0x366e840
|
|
-68> 2013-08-15 20:50:27.874530 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (3735137220 0 0) 0x3676400 con 0x366e840
|
|
-67> 2013-08-15 20:50:27.874586 7fa33a11a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-66> 2013-08-15 20:50:27.874591 7fa33a11a700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x3454380 con 0x366e840
|
|
-65> 2013-08-15 20:50:27.874607 7fa33a11a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-64> 2013-08-15 20:50:27.874621 7fa33a11a700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x3454540 con 0x366e840
|
|
-63> 2013-08-15 20:50:27.874633 7fa33a11a700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:57.874632)
|
|
-62> 2013-08-15 20:50:27.874648 7fa33a11a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-61> 2013-08-15 20:50:27.874653 7fa33a11a700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x367c000 con 0x366e840
|
|
-60> 2013-08-15 20:50:27.874765 7fa3477527c0 5 monclient: authenticate success, global_id 4377
|
|
-59> 2013-08-15 20:50:27.874778 7fa3477527c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:57.874777)
|
|
-58> 2013-08-15 20:50:27.874944 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x34945a0 con 0x366e840
|
|
-57> 2013-08-15 20:50:27.874957 7fa33a11a700 10 monclient: handle_monmap mon_map v1
|
|
-56> 2013-08-15 20:50:27.874965 7fa33a11a700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-55> 2013-08-15 20:50:27.874967 7fa33a11a700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-54> 2013-08-15 20:50:27.874995 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x3454380 con 0x366e840
|
|
-53> 2013-08-15 20:50:27.875002 7fa33a11a700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:27.820308 renew after 2013-08-15 20:52:57.820308
|
|
-52> 2013-08-15 20:50:27.875011 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x34943c0 con 0x366e840
|
|
-51> 2013-08-15 20:50:27.875019 7fa33a11a700 10 monclient: handle_monmap mon_map v1
|
|
-50> 2013-08-15 20:50:27.875024 7fa33a11a700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-49> 2013-08-15 20:50:27.875027 7fa33a11a700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-48> 2013-08-15 20:50:27.875038 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x3454700 con 0x366e840
|
|
-47> 2013-08-15 20:50:27.875058 7fa33a11a700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-46> 2013-08-15 20:50:27.875062 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (2803631360 0 0) 0x3676c00 con 0x366e840
|
|
-45> 2013-08-15 20:50:27.875096 7fa33a11a700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:57.875096)
|
|
-44> 2013-08-15 20:50:27.875198 7fa3477527c0 10 monclient: wait_auth_rotating done
|
|
-43> 2013-08-15 20:50:27.875901 7fa3477527c0 10 monclient: get_version osdmap
|
|
-42> 2013-08-15 20:50:27.875910 7fa3477527c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-41> 2013-08-15 20:50:27.875915 7fa3477527c0 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x347a9c0 con 0x366e840
|
|
-40> 2013-08-15 20:50:27.876225 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=87) v2 ==== 24+0+0 (1748105714 0 0) 0x3454540 con 0x366e840
|
|
-39> 2013-08-15 20:50:27.876269 7fa336112700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-38> 2013-08-15 20:50:27.876288 7fa336112700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- osd_boot(osd.9 booted 0 v63) v3 -- ?+0 0x36a4000 con 0x366e840
|
|
-37> 2013-08-15 20:50:29.757818 7fa330dfe700 1 -- 9.4.196.3:6813/8749 >> :/0 pipe(0x3671400 sd=33 :6813 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44093/0
|
|
-36> 2013-08-15 20:50:29.759663 7fa330bfc700 1 -- 9.4.196.3:6824/8749 >> :/0 pipe(0x3671680 sd=34 :6824 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:37097/0
|
|
-35> 2013-08-15 20:50:29.760108 7fa339118700 1 -- 9.4.196.3:6813/8749 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(88..89 src has 1..89) v3 ==== 1979+0+0 (1011610323 0 0) 0x367d440 con 0x366fe40
|
|
-34> 2013-08-15 20:50:29.760144 7fa339118700 3 osd.9 63 handle_osd_map epochs [88,89], i have 63, src has [1,89]
|
|
-33> 2013-08-15 20:50:29.760154 7fa339118700 10 monclient: renew_subs
|
|
-32> 2013-08-15 20:50:29.760157 7fa339118700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-31> 2013-08-15 20:50:29.760163 7fa339118700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x34548c0 con 0x366e840
|
|
-30> 2013-08-15 20:50:29.760180 7fa339118700 1 -- 9.4.196.3:6813/8749 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_query(3.17b epoch 89) v2 ==== 137+0+0 (3929970161 0 0) 0x34941e0 con 0x366fe40
|
|
-29> 2013-08-15 20:50:29.760194 7fa339118700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.760085, event: header_read, request: pg_query(3.17b epoch 89) v2
|
|
-28> 2013-08-15 20:50:29.760205 7fa339118700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.760086, event: throttled, request: pg_query(3.17b epoch 89) v2
|
|
-27> 2013-08-15 20:50:29.760211 7fa339118700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.760124, event: all_read, request: pg_query(3.17b epoch 89) v2
|
|
-26> 2013-08-15 20:50:29.760217 7fa339118700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.760189, event: dispatched, request: pg_query(3.17b epoch 89) v2
|
|
-25> 2013-08-15 20:50:29.760223 7fa339118700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.760223, event: waiting_for_osdmap, request: pg_query(3.17b epoch 89) v2
|
|
-24> 2013-08-15 20:50:29.760234 7fa339118700 10 monclient: renew_subs
|
|
-23> 2013-08-15 20:50:29.760238 7fa339118700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-22> 2013-08-15 20:50:29.760242 7fa339118700 1 -- 9.4.196.3:6805/8749 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x3454c40 con 0x366e840
|
|
-21> 2013-08-15 20:50:29.760254 7fa339118700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.760254, event: wait for new map, request: pg_query(3.17b epoch 89) v2
|
|
-20> 2013-08-15 20:50:29.761704 7fa33a11a700 1 -- 9.4.196.3:6805/8749 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..88 src has 1..88) v3 ==== 24497+0+0 (52318603 0 0) 0x367db00 con 0x366e840
|
|
-19> 2013-08-15 20:50:29.761729 7fa33a11a700 3 osd.9 63 handle_osd_map epochs [64,88], i have 63, src has [1,88]
|
|
-18> 2013-08-15 20:50:29.761884 7fa339118700 1 -- 9.4.196.3:6813/8749 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(88..89 src has 1..89) v3 ==== 1979+0+0 (1011610323 0 0) 0x367dd40 con 0x3684000
|
|
-17> 2013-08-15 20:50:29.762310 7fa3309fa700 1 -- 9.4.196.3:6824/8749 >> :/0 pipe(0x3671b80 sd=35 :6824 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:37098/0
|
|
-16> 2013-08-15 20:50:29.762919 7fa3308f9700 1 -- 9.4.196.3:6813/8749 >> :/0 pipe(0x36f8280 sd=36 :6813 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.3:44097/0
|
|
-15> 2013-08-15 20:50:29.765917 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-14> 2013-08-15 20:50:29.765932 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-13> 2013-08-15 20:50:29.765938 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-12> 2013-08-15 20:50:29.765956 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-11> 2013-08-15 20:50:29.765968 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-10> 2013-08-15 20:50:29.765973 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-9> 2013-08-15 20:50:29.765997 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-8> 2013-08-15 20:50:29.766005 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-7> 2013-08-15 20:50:29.766012 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-6> 2013-08-15 20:50:29.766021 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-5> 2013-08-15 20:50:29.766026 7fa33a11a700 1 -- 9.4.196.3:6813/8749 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-4> 2013-08-15 20:50:29.766522 7fa340126700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-3> 2013-08-15 20:50:29.766540 7fa340126700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
-2> 2013-08-15 20:50:29.767108 7fa335911700 1 osd.9 pg_epoch: 88 pg[3.127( v 63'13 (0'0,63'13] local-les=62 n=5 ec=56 les/c 62/62 88/88/56) [3] r=-1 lpr=88 pi=56-87/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:29.767146 7fa335110700 1 osd.9 pg_epoch: 88 pg[3.64( v 63'17 (0'0,63'17] local-les=62 n=5 ec=56 les/c 62/62 88/88/56) [4] r=-1 lpr=88 pi=56-87/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:29.767537 7fa340126700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fa340126700 time 2013-08-15 20:50:29.766545
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fa34712ef8e]
|
|
5: (clone()+0x6d) [0x7fa3453e0e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:29.770532 7fa340126700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fa340126700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fa347136bd0]
|
|
3: (gsignal()+0x37) [0x7fa34531e037]
|
|
4: (abort()+0x148) [0x7fa345321698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa345c2ae8d]
|
|
6: (()+0x5ef76) [0x7fa345c28f76]
|
|
7: (()+0x5efa3) [0x7fa345c28fa3]
|
|
8: (()+0x5f1de) [0x7fa345c291de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fa34712ef8e]
|
|
14: (clone()+0x6d) [0x7fa3453e0e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-6> 2013-08-15 20:50:29.768218 7fa335911700 1 osd.9 pg_epoch: 88 pg[3.1e7( v 63'12 (0'0,63'12] local-les=62 n=2 ec=56 les/c 62/62 88/88/56) [0]/[0,10] r=-1 lpr=88 pi=56-87/10 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:29.768241 7fa335110700 1 osd.9 pg_epoch: 88 pg[3.1eb( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 88/88/85) [3]/[3,1] r=-1 lpr=88 pi=61-87/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:29.768649 7fa335911700 1 osd.9 pg_epoch: 88 pg[3.121( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 88/88/56) [1] r=-1 lpr=88 pi=56-87/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:29.769234 7fa335911700 1 osd.9 pg_epoch: 88 pg[3.123( v 63'16 (0'0,63'16] local-les=62 n=2 ec=56 les/c 62/62 88/88/85) [4]/[4,2] r=-1 lpr=88 pi=61-87/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:29.769890 7fa335911700 1 osd.9 pg_epoch: 88 pg[3.11e( v 63'14 (0'0,63'14] local-les=62 n=8 ec=56 les/c 62/62 88/88/56) [0]/[0,10] r=-1 lpr=88 pi=56-87/10 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:29.770469 7fa335911700 1 osd.9 pg_epoch: 88 pg[3.1d7( v 63'14 (0'0,63'14] local-les=62 n=2 ec=56 les/c 62/62 88/88/85) [0]/[0,1] r=-1 lpr=88 pi=61-87/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:29.770532 7fa340126700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fa340126700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fa347136bd0]
|
|
3: (gsignal()+0x37) [0x7fa34531e037]
|
|
4: (abort()+0x148) [0x7fa345321698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa345c2ae8d]
|
|
6: (()+0x5ef76) [0x7fa345c28f76]
|
|
7: (()+0x5efa3) [0x7fa345c28fa3]
|
|
8: (()+0x5f1de) [0x7fa345c291de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fa34712ef8e]
|
|
14: (clone()+0x6d) [0x7fa3453e0e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:29.950099 7f59270b47c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9079
|
|
2013-08-15 20:50:29.950626 7f59270b47c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:30.000340 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:30.000360 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:30.000668 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:30.018242 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:30.018339 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:30.042502 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:30.043344 7f59270b47c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:30.043358 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.147947 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.148583 7f59270b47c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
2013-08-15 20:50:30.179875 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:30.179894 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:30.180265 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:30.203760 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:30.203801 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:30.233992 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:30.234552 7f59270b47c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:30.234564 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.333917 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:31.800142 7f591fa88700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:31.800156 7f591fa88700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
2013-08-15 20:50:31.801185 7f591fa88700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f591fa88700 time 2013-08-15 20:50:31.800162
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f5926a90f8e]
|
|
5: (clone()+0x6d) [0x7f5924d42e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-234> 2013-08-15 20:50:29.948822 7f59270b47c0 5 asok(0x1f480e0) register_command perfcounters_dump hook 0x1f42010
|
|
-233> 2013-08-15 20:50:29.948855 7f59270b47c0 5 asok(0x1f480e0) register_command 1 hook 0x1f42010
|
|
-232> 2013-08-15 20:50:29.948858 7f59270b47c0 5 asok(0x1f480e0) register_command perf dump hook 0x1f42010
|
|
-231> 2013-08-15 20:50:29.948864 7f59270b47c0 5 asok(0x1f480e0) register_command perfcounters_schema hook 0x1f42010
|
|
-230> 2013-08-15 20:50:29.948869 7f59270b47c0 5 asok(0x1f480e0) register_command 2 hook 0x1f42010
|
|
-229> 2013-08-15 20:50:29.948871 7f59270b47c0 5 asok(0x1f480e0) register_command perf schema hook 0x1f42010
|
|
-228> 2013-08-15 20:50:29.948875 7f59270b47c0 5 asok(0x1f480e0) register_command config show hook 0x1f42010
|
|
-227> 2013-08-15 20:50:29.948880 7f59270b47c0 5 asok(0x1f480e0) register_command config set hook 0x1f42010
|
|
-226> 2013-08-15 20:50:29.948886 7f59270b47c0 5 asok(0x1f480e0) register_command log flush hook 0x1f42010
|
|
-225> 2013-08-15 20:50:29.948892 7f59270b47c0 5 asok(0x1f480e0) register_command log dump hook 0x1f42010
|
|
-224> 2013-08-15 20:50:29.948898 7f59270b47c0 5 asok(0x1f480e0) register_command log reopen hook 0x1f42010
|
|
-223> 2013-08-15 20:50:29.950099 7f59270b47c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9079
|
|
-222> 2013-08-15 20:50:29.950626 7f59270b47c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-221> 2013-08-15 20:50:29.950738 7f59270b47c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/9079 need_addr=1
|
|
-220> 2013-08-15 20:50:29.950762 7f59270b47c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-219> 2013-08-15 20:50:29.950771 7f59270b47c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6809/9079 need_addr=0
|
|
-218> 2013-08-15 20:50:29.950796 7f59270b47c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-217> 2013-08-15 20:50:29.950802 7f59270b47c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6828/9079 need_addr=0
|
|
-216> 2013-08-15 20:50:29.952574 7f59270b47c0 5 asok(0x1f480e0) init /var/run/ceph/ceph-osd.9.asok
|
|
-215> 2013-08-15 20:50:29.952589 7f59270b47c0 5 asok(0x1f480e0) bind_and_listen /var/run/ceph/ceph-osd.9.asok
|
|
-214> 2013-08-15 20:50:29.952623 7f59270b47c0 5 asok(0x1f480e0) register_command 0 hook 0x1f400b8
|
|
-213> 2013-08-15 20:50:29.952629 7f59270b47c0 5 asok(0x1f480e0) register_command version hook 0x1f400b8
|
|
-212> 2013-08-15 20:50:29.952635 7f59270b47c0 5 asok(0x1f480e0) register_command git_version hook 0x1f400b8
|
|
-211> 2013-08-15 20:50:29.952641 7f59270b47c0 5 asok(0x1f480e0) register_command help hook 0x1f420d0
|
|
-210> 2013-08-15 20:50:29.952669 7f5922a8e700 5 asok(0x1f480e0) entry start
|
|
-209> 2013-08-15 20:50:30.000340 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-208> 2013-08-15 20:50:30.000360 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-207> 2013-08-15 20:50:30.000668 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-206> 2013-08-15 20:50:30.018242 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-205> 2013-08-15 20:50:30.018339 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-204> 2013-08-15 20:50:30.042502 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-203> 2013-08-15 20:50:30.043321 7f59270b47c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-202> 2013-08-15 20:50:30.043344 7f59270b47c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-201> 2013-08-15 20:50:30.043358 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-200> 2013-08-15 20:50:30.049662 7f59270b47c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-199> 2013-08-15 20:50:30.053900 7f59270b47c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-198> 2013-08-15 20:50:30.058153 7f59270b47c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-197> 2013-08-15 20:50:30.062803 7f59270b47c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-196> 2013-08-15 20:50:30.067066 7f59270b47c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-195> 2013-08-15 20:50:30.071455 7f59270b47c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-194> 2013-08-15 20:50:30.075671 7f59270b47c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-193> 2013-08-15 20:50:30.079905 7f59270b47c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-192> 2013-08-15 20:50:30.084221 7f59270b47c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-191> 2013-08-15 20:50:30.088626 7f59270b47c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-190> 2013-08-15 20:50:30.093011 7f59270b47c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-189> 2013-08-15 20:50:30.097374 7f59270b47c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-188> 2013-08-15 20:50:30.101728 7f59270b47c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-187> 2013-08-15 20:50:30.105780 7f59270b47c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-186> 2013-08-15 20:50:30.109809 7f59270b47c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-185> 2013-08-15 20:50:30.113815 7f59270b47c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-184> 2013-08-15 20:50:30.117993 7f59270b47c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-183> 2013-08-15 20:50:30.122255 7f59270b47c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-182> 2013-08-15 20:50:30.126508 7f59270b47c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-181> 2013-08-15 20:50:30.130754 7f59270b47c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-180> 2013-08-15 20:50:30.134997 7f59270b47c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-179> 2013-08-15 20:50:30.139242 7f59270b47c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-178> 2013-08-15 20:50:30.143557 7f59270b47c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-177> 2013-08-15 20:50:30.147814 7f59270b47c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-176> 2013-08-15 20:50:30.147874 7f59270b47c0 2 journal No further valid entries found, journal is most likely valid
|
|
-175> 2013-08-15 20:50:30.147889 7f59270b47c0 2 journal No further valid entries found, journal is most likely valid
|
|
-174> 2013-08-15 20:50:30.147899 7f59270b47c0 3 journal journal_replay: end of journal, done.
|
|
-173> 2013-08-15 20:50:30.147947 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-172> 2013-08-15 20:50:30.148396 7f5920289700 1 FileStore::op_tp worker finish
|
|
-171> 2013-08-15 20:50:30.148462 7f591fa88700 1 FileStore::op_tp worker finish
|
|
-170> 2013-08-15 20:50:30.148583 7f59270b47c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
-169> 2013-08-15 20:50:30.149257 7f59270b47c0 10 monclient(hunting): build_initial_monmap
|
|
-168> 2013-08-15 20:50:30.149316 7f59270b47c0 5 adding auth protocol: cephx
|
|
-167> 2013-08-15 20:50:30.149324 7f59270b47c0 5 adding auth protocol: cephx
|
|
-166> 2013-08-15 20:50:30.149412 7f59270b47c0 1 -- 0.0.0.0:6805/9079 messenger.start
|
|
-165> 2013-08-15 20:50:30.149442 7f59270b47c0 1 -- :/0 messenger.start
|
|
-164> 2013-08-15 20:50:30.149454 7f59270b47c0 1 -- 9.4.196.3:6828/9079 messenger.start
|
|
-163> 2013-08-15 20:50:30.149465 7f59270b47c0 1 -- 9.4.196.3:6809/9079 messenger.start
|
|
-162> 2013-08-15 20:50:30.149552 7f59270b47c0 2 osd.9 0 mounting /var/lib/ceph/osd/ceph-9 /var/lib/ceph/osd/ceph-9/journal
|
|
-161> 2013-08-15 20:50:30.179875 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-160> 2013-08-15 20:50:30.179894 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-159> 2013-08-15 20:50:30.180265 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-158> 2013-08-15 20:50:30.203760 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-157> 2013-08-15 20:50:30.203801 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-156> 2013-08-15 20:50:30.233992 7f59270b47c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-155> 2013-08-15 20:50:30.234521 7f59270b47c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-154> 2013-08-15 20:50:30.234552 7f59270b47c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-153> 2013-08-15 20:50:30.234564 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-152> 2013-08-15 20:50:30.238684 7f59270b47c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-151> 2013-08-15 20:50:30.242730 7f59270b47c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-150> 2013-08-15 20:50:30.246763 7f59270b47c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-149> 2013-08-15 20:50:30.250739 7f59270b47c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-148> 2013-08-15 20:50:30.254831 7f59270b47c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-147> 2013-08-15 20:50:30.258877 7f59270b47c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-146> 2013-08-15 20:50:30.262894 7f59270b47c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-145> 2013-08-15 20:50:30.266990 7f59270b47c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-144> 2013-08-15 20:50:30.271183 7f59270b47c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-143> 2013-08-15 20:50:30.275376 7f59270b47c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-142> 2013-08-15 20:50:30.279706 7f59270b47c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-141> 2013-08-15 20:50:30.283971 7f59270b47c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-140> 2013-08-15 20:50:30.288242 7f59270b47c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-139> 2013-08-15 20:50:30.292558 7f59270b47c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-138> 2013-08-15 20:50:30.296860 7f59270b47c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-137> 2013-08-15 20:50:30.301078 7f59270b47c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-136> 2013-08-15 20:50:30.305302 7f59270b47c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-135> 2013-08-15 20:50:30.309495 7f59270b47c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-134> 2013-08-15 20:50:30.313697 7f59270b47c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-133> 2013-08-15 20:50:30.317782 7f59270b47c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-132> 2013-08-15 20:50:30.321837 7f59270b47c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-131> 2013-08-15 20:50:30.325853 7f59270b47c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-130> 2013-08-15 20:50:30.329895 7f59270b47c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-129> 2013-08-15 20:50:30.333869 7f59270b47c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-128> 2013-08-15 20:50:30.333889 7f59270b47c0 2 journal No further valid entries found, journal is most likely valid
|
|
-127> 2013-08-15 20:50:30.333894 7f59270b47c0 2 journal No further valid entries found, journal is most likely valid
|
|
-126> 2013-08-15 20:50:30.333895 7f59270b47c0 3 journal journal_replay: end of journal, done.
|
|
-125> 2013-08-15 20:50:30.333917 7f59270b47c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-124> 2013-08-15 20:50:30.334084 7f59270b47c0 2 osd.9 0 boot
|
|
-123> 2013-08-15 20:50:30.360872 7f59270b47c0 2 osd.9 63 superblock: i am osd.9
|
|
-122> 2013-08-15 20:50:30.360914 7f59270b47c0 1 accepter.accepter.start
|
|
-121> 2013-08-15 20:50:30.360962 7f59270b47c0 1 accepter.accepter.start
|
|
-120> 2013-08-15 20:50:30.361002 7f59270b47c0 1 accepter.accepter.start
|
|
-119> 2013-08-15 20:50:30.361019 7f59270b47c0 10 monclient(hunting): init
|
|
-118> 2013-08-15 20:50:30.361026 7f59270b47c0 5 adding auth protocol: cephx
|
|
-117> 2013-08-15 20:50:30.361031 7f59270b47c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-116> 2013-08-15 20:50:30.361153 7f5927049700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4000 sd=27 :6828 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.3:56121/0
|
|
-115> 2013-08-15 20:50:30.361187 7f5916174700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4280 sd=29 :6828 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.1:43185/0
|
|
-114> 2013-08-15 20:50:30.361241 7f59270b47c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-9/keyring
|
|
-113> 2013-08-15 20:50:30.361236 7f5916073700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4c80 sd=30 :6828 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:43187/0
|
|
-112> 2013-08-15 20:50:30.361217 7f5916275700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4500 sd=28 :6828 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.3:56123/0
|
|
-111> 2013-08-15 20:50:30.361264 7f5915f72700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4a00 sd=31 :6828 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:43188/0
|
|
-110> 2013-08-15 20:50:30.361300 7f5915e71700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4780 sd=32 :6828 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:43195/0
|
|
-109> 2013-08-15 20:50:30.361352 7f5915d70700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a5680 sd=33 :6828 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:43197/0
|
|
-108> 2013-08-15 20:50:30.361544 7f59270b47c0 5 asok(0x1f480e0) register_command dump_ops_in_flight hook 0x1f42710
|
|
-107> 2013-08-15 20:50:30.361558 7f59270b47c0 5 asok(0x1f480e0) register_command dump_historic_ops hook 0x1f42710
|
|
-106> 2013-08-15 20:50:30.361567 7f59270b47c0 5 asok(0x1f480e0) register_command dump_op_pq_state hook 0x1f42710
|
|
-105> 2013-08-15 20:50:30.361574 7f59270b47c0 5 asok(0x1f480e0) register_command setomapval hook 0x2139360
|
|
-104> 2013-08-15 20:50:30.361579 7f59270b47c0 5 asok(0x1f480e0) register_command rmomapkey hook 0x2139360
|
|
-103> 2013-08-15 20:50:30.361567 7f5916174700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a5400 sd=34 :6828 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:56124/0
|
|
-102> 2013-08-15 20:50:30.361575 7f5912468700 5 osd.9 63 heartbeat: osd_stat(1026 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-101> 2013-08-15 20:50:30.361583 7f59270b47c0 5 asok(0x1f480e0) register_command setomapheader hook 0x2139360
|
|
-100> 2013-08-15 20:50:30.361597 7f59270b47c0 5 asok(0x1f480e0) register_command getomap hook 0x2139360
|
|
-99> 2013-08-15 20:50:30.361603 7f59270b47c0 5 asok(0x1f480e0) register_command truncobj hook 0x2139360
|
|
-98> 2013-08-15 20:50:30.361607 7f59270b47c0 5 asok(0x1f480e0) register_command injectdataerr hook 0x2139360
|
|
-97> 2013-08-15 20:50:30.361611 7f59270b47c0 5 asok(0x1f480e0) register_command injectmdataerr hook 0x2139360
|
|
-96> 2013-08-15 20:50:30.361660 7f59270b47c0 10 monclient(hunting): _reopen_session
|
|
-95> 2013-08-15 20:50:30.361684 7f59270b47c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x21a6420 addr 9.4.196.1:6789/0
|
|
-94> 2013-08-15 20:50:30.361697 7f59270b47c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-93> 2013-08-15 20:50:30.361702 7f59270b47c0 1 -- 0.0.0.0:6805/9079 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1f54b40 con 0x21a6420
|
|
-92> 2013-08-15 20:50:30.361714 7f59270b47c0 10 monclient(hunting): renew_subs
|
|
-91> 2013-08-15 20:50:30.361960 7f5916073700 1 -- 9.4.196.3:6805/9079 learned my addr 9.4.196.3:6805/9079
|
|
-90> 2013-08-15 20:50:30.362130 7f5919a7c700 10 monclient(hunting): renew_subs
|
|
-89> 2013-08-15 20:50:30.362442 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1fca1e0 con 0x21a6420
|
|
-88> 2013-08-15 20:50:30.362459 7f5919a7c700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-87> 2013-08-15 20:50:30.362472 7f5919a7c700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-86> 2013-08-15 20:50:30.362476 7f5919a7c700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-85> 2013-08-15 20:50:30.362494 7f5919a7c700 1 monclient(hunting): found mon.zc2store
|
|
-84> 2013-08-15 20:50:30.362501 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3272392107 0 0) 0x21b4200 con 0x21a6420
|
|
-83> 2013-08-15 20:50:30.362522 7f5919a7c700 10 monclient: my global_id is 4395
|
|
-82> 2013-08-15 20:50:30.362617 7f5919a7c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-81> 2013-08-15 20:50:30.362627 7f5919a7c700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1f55d40 con 0x21a6420
|
|
-80> 2013-08-15 20:50:30.363079 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3549654017 0 0) 0x21b4000 con 0x21a6420
|
|
-79> 2013-08-15 20:50:30.363143 7f5919a7c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-78> 2013-08-15 20:50:30.363151 7f5919a7c700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x21b8240 con 0x21a6420
|
|
-77> 2013-08-15 20:50:30.363590 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (1333693387 0 0) 0x21b4400 con 0x21a6420
|
|
-76> 2013-08-15 20:50:30.363653 7f5919a7c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-75> 2013-08-15 20:50:30.363660 7f5919a7c700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1f8a380 con 0x21a6420
|
|
-74> 2013-08-15 20:50:30.363685 7f5919a7c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-73> 2013-08-15 20:50:30.363689 7f5919a7c700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x1f8a540 con 0x21a6420
|
|
-72> 2013-08-15 20:50:30.363697 7f5919a7c700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:00.363697)
|
|
-71> 2013-08-15 20:50:30.363706 7f5919a7c700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-70> 2013-08-15 20:50:30.363711 7f5919a7c700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x21b8000 con 0x21a6420
|
|
-69> 2013-08-15 20:50:30.363724 7f59270b47c0 5 monclient: authenticate success, global_id 4395
|
|
-68> 2013-08-15 20:50:30.363731 7f59270b47c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:00.363730)
|
|
-67> 2013-08-15 20:50:30.364065 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1fca5a0 con 0x21a6420
|
|
-66> 2013-08-15 20:50:30.364076 7f5919a7c700 10 monclient: handle_monmap mon_map v1
|
|
-65> 2013-08-15 20:50:30.364095 7f5919a7c700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-64> 2013-08-15 20:50:30.364098 7f5919a7c700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-63> 2013-08-15 20:50:30.364113 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1f8a380 con 0x21a6420
|
|
-62> 2013-08-15 20:50:30.364122 7f5919a7c700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:30.361717 renew after 2013-08-15 20:53:00.361717
|
|
-61> 2013-08-15 20:50:30.364132 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1fca3c0 con 0x21a6420
|
|
-60> 2013-08-15 20:50:30.364140 7f5919a7c700 10 monclient: handle_monmap mon_map v1
|
|
-59> 2013-08-15 20:50:30.364157 7f5919a7c700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-58> 2013-08-15 20:50:30.364161 7f5919a7c700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-57> 2013-08-15 20:50:30.364175 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1f8a700 con 0x21a6420
|
|
-56> 2013-08-15 20:50:30.364183 7f5919a7c700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-55> 2013-08-15 20:50:30.364190 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (3381096821 0 0) 0x21ba600 con 0x21a6420
|
|
-54> 2013-08-15 20:50:30.364238 7f5919a7c700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:00.364238)
|
|
-53> 2013-08-15 20:50:30.364251 7f59270b47c0 10 monclient: wait_auth_rotating done
|
|
-52> 2013-08-15 20:50:30.364624 7f59270b47c0 10 monclient: get_version osdmap
|
|
-51> 2013-08-15 20:50:30.364629 7f59270b47c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-50> 2013-08-15 20:50:30.364636 7f59270b47c0 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x1fb09c0 con 0x21a6420
|
|
-49> 2013-08-15 20:50:30.364874 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=89) v2 ==== 24+0+0 (3324254003 0 0) 0x1f8a540 con 0x21a6420
|
|
-48> 2013-08-15 20:50:30.364947 7f591546e700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-47> 2013-08-15 20:50:30.364963 7f591546e700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- osd_boot(osd.9 booted 0 v63) v3 -- ?+0 0x21d8000 con 0x21a6420
|
|
-46> 2013-08-15 20:50:30.761541 7f591065f700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a4f00 sd=34 :6828 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:56146/0
|
|
-45> 2013-08-15 20:50:30.761568 7f5910760700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a5180 sd=33 :6828 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:56145/0
|
|
-44> 2013-08-15 20:50:30.761575 7f591045d700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a5900 sd=36 :6828 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:43237/0
|
|
-43> 2013-08-15 20:50:30.761559 7f591055e700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21a5b80 sd=35 :6828 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:43234/0
|
|
-42> 2013-08-15 20:50:30.761629 7f591035c700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21dac80 sd=37 :6828 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:43235/0
|
|
-41> 2013-08-15 20:50:30.761665 7f591025b700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21daa00 sd=38 :6828 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:43236/0
|
|
-40> 2013-08-15 20:50:30.761700 7f591015a700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21da780 sd=39 :6828 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.1:43238/0
|
|
-39> 2013-08-15 20:50:30.761922 7f591015a700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21da500 sd=33 :6828 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:56147/0
|
|
-38> 2013-08-15 20:50:31.783942 7f591015a700 1 -- 9.4.196.3:6809/9079 >> :/0 pipe(0x21a5b80 sd=33 :6809 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:42786/0
|
|
-37> 2013-08-15 20:50:31.785566 7f591035c700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21da280 sd=34 :6828 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:43239/0
|
|
-36> 2013-08-15 20:50:31.788861 7f5918a7a700 1 -- 9.4.196.3:6809/9079 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x21b9440 con 0x21be000
|
|
-35> 2013-08-15 20:50:31.788896 7f5918a7a700 3 osd.9 63 handle_osd_map epochs [90,91], i have 63, src has [1,91]
|
|
-34> 2013-08-15 20:50:31.788916 7f5918a7a700 10 monclient: renew_subs
|
|
-33> 2013-08-15 20:50:31.788925 7f5918a7a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-32> 2013-08-15 20:50:31.788933 7f5918a7a700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x1f8a8c0 con 0x21a6420
|
|
-31> 2013-08-15 20:50:31.788953 7f5918a7a700 1 -- 9.4.196.3:6809/9079 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_query(3.17b epoch 91) v2 ==== 137+0+0 (3579298369 0 0) 0x1fca1e0 con 0x21be000
|
|
-30> 2013-08-15 20:50:31.788968 7f5918a7a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.788878, event: header_read, request: pg_query(3.17b epoch 91) v2
|
|
-29> 2013-08-15 20:50:31.788979 7f5918a7a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.788879, event: throttled, request: pg_query(3.17b epoch 91) v2
|
|
-28> 2013-08-15 20:50:31.788986 7f5918a7a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.788921, event: all_read, request: pg_query(3.17b epoch 91) v2
|
|
-27> 2013-08-15 20:50:31.788992 7f5918a7a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.788962, event: dispatched, request: pg_query(3.17b epoch 91) v2
|
|
-26> 2013-08-15 20:50:31.788996 7f5918a7a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.788996, event: waiting_for_osdmap, request: pg_query(3.17b epoch 91) v2
|
|
-25> 2013-08-15 20:50:31.789007 7f5918a7a700 10 monclient: renew_subs
|
|
-24> 2013-08-15 20:50:31.789011 7f5918a7a700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-23> 2013-08-15 20:50:31.789016 7f5918a7a700 1 -- 9.4.196.3:6805/9079 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x1f8ac40 con 0x21a6420
|
|
-22> 2013-08-15 20:50:31.789028 7f5918a7a700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.789028, event: wait for new map, request: pg_query(3.17b epoch 91) v2
|
|
-21> 2013-08-15 20:50:31.792161 7f591045d700 1 -- 9.4.196.3:6809/9079 >> :/0 pipe(0x21a4f00 sd=35 :6809 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:42790/0
|
|
-20> 2013-08-15 20:50:31.792596 7f5919a7c700 1 -- 9.4.196.3:6805/9079 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..90 src has 1..90) v3 ==== 26633+0+0 (225616071 0 0) 0x21b9d40 con 0x21a6420
|
|
-19> 2013-08-15 20:50:31.792617 7f5919a7c700 3 osd.9 63 handle_osd_map epochs [64,90], i have 63, src has [1,90]
|
|
-18> 2013-08-15 20:50:31.797194 7f5918a7a700 1 -- 9.4.196.3:6809/9079 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x221c480 con 0x21be160
|
|
-17> 2013-08-15 20:50:31.799374 7f5910760700 1 -- 9.4.196.3:6828/9079 >> :/0 pipe(0x21da000 sd=36 :6828 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:43242/0
|
|
-16> 2013-08-15 20:50:31.799395 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-15> 2013-08-15 20:50:31.799418 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-14> 2013-08-15 20:50:31.799425 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-13> 2013-08-15 20:50:31.799449 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-12> 2013-08-15 20:50:31.799467 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-11> 2013-08-15 20:50:31.799473 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-10> 2013-08-15 20:50:31.799509 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-9> 2013-08-15 20:50:31.799520 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-8> 2013-08-15 20:50:31.799532 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-7> 2013-08-15 20:50:31.799545 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-6> 2013-08-15 20:50:31.799555 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-5> 2013-08-15 20:50:31.799567 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6826/8751 -- pipe dne
|
|
-4> 2013-08-15 20:50:31.799573 7f5919a7c700 1 -- 9.4.196.3:6809/9079 mark_down 9.4.196.3:6828/8914 -- pipe dne
|
|
-3> 2013-08-15 20:50:31.800142 7f591fa88700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-2> 2013-08-15 20:50:31.800156 7f591fa88700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
-1> 2013-08-15 20:50:31.800441 7f591446c700 1 osd.9 pg_epoch: 90 pg[3.64( v 63'17 (0'0,63'17] local-les=62 n=5 ec=56 les/c 62/62 90/90/56) [4] r=-1 lpr=90 pi=56-89/13 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:31.801185 7f591fa88700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f591fa88700 time 2013-08-15 20:50:31.800162
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f5926a90f8e]
|
|
5: (clone()+0x6d) [0x7f5924d42e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:31.805053 7f591fa88700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f591fa88700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f5926a98bd0]
|
|
3: (gsignal()+0x37) [0x7f5924c80037]
|
|
4: (abort()+0x148) [0x7f5924c83698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f592558ce8d]
|
|
6: (()+0x5ef76) [0x7f592558af76]
|
|
7: (()+0x5efa3) [0x7f592558afa3]
|
|
8: (()+0x5f1de) [0x7f592558b1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f5926a90f8e]
|
|
14: (clone()+0x6d) [0x7f5924d42e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-7> 2013-08-15 20:50:31.801704 7f591446c700 1 osd.9 pg_epoch: 90 pg[3.127( v 63'13 (0'0,63'13] local-les=62 n=5 ec=56 les/c 62/62 90/90/56) [3] r=-1 lpr=90 pi=56-89/13 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-6> 2013-08-15 20:50:31.802433 7f591446c700 1 osd.9 pg_epoch: 90 pg[3.1e7( v 63'12 (0'0,63'12] local-les=62 n=2 ec=56 les/c 62/62 90/90/56) [0]/[0,10] r=-1 lpr=90 pi=56-89/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:31.802583 7f5914c6d700 1 osd.9 pg_epoch: 90 pg[3.1eb( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 90/90/85) [3]/[3,1] r=-1 lpr=90 pi=61-89/13 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:31.802845 7f591446c700 1 osd.9 pg_epoch: 90 pg[3.121( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 90/90/56) [1] r=-1 lpr=90 pi=56-89/13 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:31.803606 7f591446c700 1 osd.9 pg_epoch: 90 pg[3.123( v 63'16 (0'0,63'16] local-les=62 n=2 ec=56 les/c 62/62 90/90/85) [4]/[4,2] r=-1 lpr=90 pi=61-89/13 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:31.803720 7f590ff58700 1 -- 9.4.196.3:6809/9079 >> :/0 pipe(0x21a5900 sd=37 :6809 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.3:42794/0
|
|
-1> 2013-08-15 20:50:31.804416 7f591446c700 1 osd.9 pg_epoch: 90 pg[3.11e( v 63'14 (0'0,63'14] local-les=62 n=8 ec=56 les/c 62/62 90/90/56) [0]/[0,10] r=-1 lpr=90 pi=56-89/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:31.805053 7f591fa88700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f591fa88700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f5926a98bd0]
|
|
3: (gsignal()+0x37) [0x7f5924c80037]
|
|
4: (abort()+0x148) [0x7f5924c83698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f592558ce8d]
|
|
6: (()+0x5ef76) [0x7f592558af76]
|
|
7: (()+0x5efa3) [0x7f592558afa3]
|
|
8: (()+0x5f1de) [0x7f592558b1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f5926a90f8e]
|
|
14: (clone()+0x6d) [0x7f5924d42e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:32.113128 7fd94e0e37c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9435
|
|
2013-08-15 20:50:32.113632 7fd94e0e37c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:32.159415 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:32.159427 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:32.159695 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:32.177317 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:32.177388 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:32.201609 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:32.202519 7fd94e0e37c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:32.202541 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.304026 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.304518 7fd94e0e37c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
2013-08-15 20:50:32.332982 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:32.332992 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:32.333228 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:32.356829 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:32.356901 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:32.387158 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:32.387749 7fd94e0e37c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:32.387766 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.486830 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:33.763182 7fd946ab7700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:33.763206 7fd946ab7700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
2013-08-15 20:50:33.764271 7fd946ab7700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fd946ab7700 time 2013-08-15 20:50:33.763213
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fd94dabff8e]
|
|
5: (clone()+0x6d) [0x7fd94bd71e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-221> 2013-08-15 20:50:32.111802 7fd94e0e37c0 5 asok(0x25f20e0) register_command perfcounters_dump hook 0x25ec010
|
|
-220> 2013-08-15 20:50:32.111829 7fd94e0e37c0 5 asok(0x25f20e0) register_command 1 hook 0x25ec010
|
|
-219> 2013-08-15 20:50:32.111832 7fd94e0e37c0 5 asok(0x25f20e0) register_command perf dump hook 0x25ec010
|
|
-218> 2013-08-15 20:50:32.111839 7fd94e0e37c0 5 asok(0x25f20e0) register_command perfcounters_schema hook 0x25ec010
|
|
-217> 2013-08-15 20:50:32.111843 7fd94e0e37c0 5 asok(0x25f20e0) register_command 2 hook 0x25ec010
|
|
-216> 2013-08-15 20:50:32.111845 7fd94e0e37c0 5 asok(0x25f20e0) register_command perf schema hook 0x25ec010
|
|
-215> 2013-08-15 20:50:32.111851 7fd94e0e37c0 5 asok(0x25f20e0) register_command config show hook 0x25ec010
|
|
-214> 2013-08-15 20:50:32.111855 7fd94e0e37c0 5 asok(0x25f20e0) register_command config set hook 0x25ec010
|
|
-213> 2013-08-15 20:50:32.111859 7fd94e0e37c0 5 asok(0x25f20e0) register_command log flush hook 0x25ec010
|
|
-212> 2013-08-15 20:50:32.111863 7fd94e0e37c0 5 asok(0x25f20e0) register_command log dump hook 0x25ec010
|
|
-211> 2013-08-15 20:50:32.111867 7fd94e0e37c0 5 asok(0x25f20e0) register_command log reopen hook 0x25ec010
|
|
-210> 2013-08-15 20:50:32.113128 7fd94e0e37c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9435
|
|
-209> 2013-08-15 20:50:32.113632 7fd94e0e37c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-208> 2013-08-15 20:50:32.113791 7fd94e0e37c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6835/9435 need_addr=1
|
|
-207> 2013-08-15 20:50:32.113827 7fd94e0e37c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-206> 2013-08-15 20:50:32.113836 7fd94e0e37c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6836/9435 need_addr=0
|
|
-205> 2013-08-15 20:50:32.113860 7fd94e0e37c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-204> 2013-08-15 20:50:32.113865 7fd94e0e37c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6837/9435 need_addr=0
|
|
-203> 2013-08-15 20:50:32.116205 7fd94e0e37c0 5 asok(0x25f20e0) init /var/run/ceph/ceph-osd.9.asok
|
|
-202> 2013-08-15 20:50:32.116220 7fd94e0e37c0 5 asok(0x25f20e0) bind_and_listen /var/run/ceph/ceph-osd.9.asok
|
|
-201> 2013-08-15 20:50:32.116249 7fd94e0e37c0 5 asok(0x25f20e0) register_command 0 hook 0x25ea0b8
|
|
-200> 2013-08-15 20:50:32.116255 7fd94e0e37c0 5 asok(0x25f20e0) register_command version hook 0x25ea0b8
|
|
-199> 2013-08-15 20:50:32.116258 7fd94e0e37c0 5 asok(0x25f20e0) register_command git_version hook 0x25ea0b8
|
|
-198> 2013-08-15 20:50:32.116261 7fd94e0e37c0 5 asok(0x25f20e0) register_command help hook 0x25ec0d0
|
|
-197> 2013-08-15 20:50:32.116315 7fd949abd700 5 asok(0x25f20e0) entry start
|
|
-196> 2013-08-15 20:50:32.159415 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-195> 2013-08-15 20:50:32.159427 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-194> 2013-08-15 20:50:32.159695 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-193> 2013-08-15 20:50:32.177317 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-192> 2013-08-15 20:50:32.177388 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-191> 2013-08-15 20:50:32.201609 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-190> 2013-08-15 20:50:32.202489 7fd94e0e37c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-189> 2013-08-15 20:50:32.202519 7fd94e0e37c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-188> 2013-08-15 20:50:32.202541 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-187> 2013-08-15 20:50:32.207945 7fd94e0e37c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-186> 2013-08-15 20:50:32.211945 7fd94e0e37c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-185> 2013-08-15 20:50:32.216051 7fd94e0e37c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-184> 2013-08-15 20:50:32.220295 7fd94e0e37c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-183> 2013-08-15 20:50:32.224640 7fd94e0e37c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-182> 2013-08-15 20:50:32.228893 7fd94e0e37c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-181> 2013-08-15 20:50:32.233172 7fd94e0e37c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-180> 2013-08-15 20:50:32.237408 7fd94e0e37c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-179> 2013-08-15 20:50:32.241686 7fd94e0e37c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-178> 2013-08-15 20:50:32.245937 7fd94e0e37c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-177> 2013-08-15 20:50:32.250180 7fd94e0e37c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-176> 2013-08-15 20:50:32.254403 7fd94e0e37c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-175> 2013-08-15 20:50:32.258646 7fd94e0e37c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-174> 2013-08-15 20:50:32.262887 7fd94e0e37c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-173> 2013-08-15 20:50:32.267100 7fd94e0e37c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-172> 2013-08-15 20:50:32.271425 7fd94e0e37c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-171> 2013-08-15 20:50:32.275596 7fd94e0e37c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-170> 2013-08-15 20:50:32.279720 7fd94e0e37c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-169> 2013-08-15 20:50:32.283803 7fd94e0e37c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-168> 2013-08-15 20:50:32.287904 7fd94e0e37c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-167> 2013-08-15 20:50:32.291967 7fd94e0e37c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-166> 2013-08-15 20:50:32.296020 7fd94e0e37c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-165> 2013-08-15 20:50:32.299972 7fd94e0e37c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-164> 2013-08-15 20:50:32.303941 7fd94e0e37c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-163> 2013-08-15 20:50:32.303972 7fd94e0e37c0 2 journal No further valid entries found, journal is most likely valid
|
|
-162> 2013-08-15 20:50:32.303980 7fd94e0e37c0 2 journal No further valid entries found, journal is most likely valid
|
|
-161> 2013-08-15 20:50:32.303988 7fd94e0e37c0 3 journal journal_replay: end of journal, done.
|
|
-160> 2013-08-15 20:50:32.304026 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-159> 2013-08-15 20:50:32.304446 7fd946ab7700 1 FileStore::op_tp worker finish
|
|
-158> 2013-08-15 20:50:32.304460 7fd9472b8700 1 FileStore::op_tp worker finish
|
|
-157> 2013-08-15 20:50:32.304518 7fd94e0e37c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
-156> 2013-08-15 20:50:32.304872 7fd94e0e37c0 10 monclient(hunting): build_initial_monmap
|
|
-155> 2013-08-15 20:50:32.304923 7fd94e0e37c0 5 adding auth protocol: cephx
|
|
-154> 2013-08-15 20:50:32.304929 7fd94e0e37c0 5 adding auth protocol: cephx
|
|
-153> 2013-08-15 20:50:32.305018 7fd94e0e37c0 1 -- 0.0.0.0:6835/9435 messenger.start
|
|
-152> 2013-08-15 20:50:32.305049 7fd94e0e37c0 1 -- :/0 messenger.start
|
|
-151> 2013-08-15 20:50:32.305061 7fd94e0e37c0 1 -- 9.4.196.3:6837/9435 messenger.start
|
|
-150> 2013-08-15 20:50:32.305073 7fd94e0e37c0 1 -- 9.4.196.3:6836/9435 messenger.start
|
|
-149> 2013-08-15 20:50:32.305153 7fd94e0e37c0 2 osd.9 0 mounting /var/lib/ceph/osd/ceph-9 /var/lib/ceph/osd/ceph-9/journal
|
|
-148> 2013-08-15 20:50:32.332982 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-147> 2013-08-15 20:50:32.332992 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-146> 2013-08-15 20:50:32.333228 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-145> 2013-08-15 20:50:32.356829 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-144> 2013-08-15 20:50:32.356901 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-143> 2013-08-15 20:50:32.387158 7fd94e0e37c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-142> 2013-08-15 20:50:32.387727 7fd94e0e37c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-141> 2013-08-15 20:50:32.387749 7fd94e0e37c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-140> 2013-08-15 20:50:32.387766 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-139> 2013-08-15 20:50:32.392248 7fd94e0e37c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-138> 2013-08-15 20:50:32.396689 7fd94e0e37c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-137> 2013-08-15 20:50:32.401052 7fd94e0e37c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-136> 2013-08-15 20:50:32.405414 7fd94e0e37c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-135> 2013-08-15 20:50:32.409767 7fd94e0e37c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-134> 2013-08-15 20:50:32.414076 7fd94e0e37c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-133> 2013-08-15 20:50:32.418372 7fd94e0e37c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-132> 2013-08-15 20:50:32.422650 7fd94e0e37c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-131> 2013-08-15 20:50:32.426929 7fd94e0e37c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-130> 2013-08-15 20:50:32.431014 7fd94e0e37c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-129> 2013-08-15 20:50:32.435006 7fd94e0e37c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-128> 2013-08-15 20:50:32.439092 7fd94e0e37c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-127> 2013-08-15 20:50:32.443113 7fd94e0e37c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-126> 2013-08-15 20:50:32.447114 7fd94e0e37c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-125> 2013-08-15 20:50:32.451223 7fd94e0e37c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-124> 2013-08-15 20:50:32.455425 7fd94e0e37c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-123> 2013-08-15 20:50:32.459587 7fd94e0e37c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-122> 2013-08-15 20:50:32.463598 7fd94e0e37c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-121> 2013-08-15 20:50:32.467442 7fd94e0e37c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-120> 2013-08-15 20:50:32.471422 7fd94e0e37c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-119> 2013-08-15 20:50:32.475278 7fd94e0e37c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-118> 2013-08-15 20:50:32.479142 7fd94e0e37c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-117> 2013-08-15 20:50:32.482954 7fd94e0e37c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-116> 2013-08-15 20:50:32.486758 7fd94e0e37c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-115> 2013-08-15 20:50:32.486784 7fd94e0e37c0 2 journal No further valid entries found, journal is most likely valid
|
|
-114> 2013-08-15 20:50:32.486789 7fd94e0e37c0 2 journal No further valid entries found, journal is most likely valid
|
|
-113> 2013-08-15 20:50:32.486791 7fd94e0e37c0 3 journal journal_replay: end of journal, done.
|
|
-112> 2013-08-15 20:50:32.486830 7fd94e0e37c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-111> 2013-08-15 20:50:32.487025 7fd94e0e37c0 2 osd.9 0 boot
|
|
-110> 2013-08-15 20:50:32.512399 7fd94e0e37c0 2 osd.9 63 superblock: i am osd.9
|
|
-109> 2013-08-15 20:50:32.512452 7fd94e0e37c0 1 accepter.accepter.start
|
|
-108> 2013-08-15 20:50:32.512486 7fd94e0e37c0 1 accepter.accepter.start
|
|
-107> 2013-08-15 20:50:32.512525 7fd94e0e37c0 1 accepter.accepter.start
|
|
-106> 2013-08-15 20:50:32.512542 7fd94e0e37c0 10 monclient(hunting): init
|
|
-105> 2013-08-15 20:50:32.512549 7fd94e0e37c0 5 adding auth protocol: cephx
|
|
-104> 2013-08-15 20:50:32.512550 7fd94e0e37c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-103> 2013-08-15 20:50:32.512683 7fd94e0e37c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-9/keyring
|
|
-102> 2013-08-15 20:50:32.512860 7fd94e0e37c0 5 asok(0x25f20e0) register_command dump_ops_in_flight hook 0x25ec710
|
|
-101> 2013-08-15 20:50:32.512869 7fd94e0e37c0 5 asok(0x25f20e0) register_command dump_historic_ops hook 0x25ec710
|
|
-100> 2013-08-15 20:50:32.512874 7fd94e0e37c0 5 asok(0x25f20e0) register_command dump_op_pq_state hook 0x25ec710
|
|
-99> 2013-08-15 20:50:32.512879 7fd94e0e37c0 5 asok(0x25f20e0) register_command setomapval hook 0x27df360
|
|
-98> 2013-08-15 20:50:32.512884 7fd94e0e37c0 5 asok(0x25f20e0) register_command rmomapkey hook 0x27df360
|
|
-97> 2013-08-15 20:50:32.512888 7fd94e0e37c0 5 asok(0x25f20e0) register_command setomapheader hook 0x27df360
|
|
-96> 2013-08-15 20:50:32.512891 7fd94e0e37c0 5 asok(0x25f20e0) register_command getomap hook 0x27df360
|
|
-95> 2013-08-15 20:50:32.512895 7fd94e0e37c0 5 asok(0x25f20e0) register_command truncobj hook 0x27df360
|
|
-94> 2013-08-15 20:50:32.512900 7fd94e0e37c0 5 asok(0x25f20e0) register_command injectdataerr hook 0x27df360
|
|
-93> 2013-08-15 20:50:32.512904 7fd94e0e37c0 5 asok(0x25f20e0) register_command injectmdataerr hook 0x27df360
|
|
-92> 2013-08-15 20:50:32.512907 7fd939a9d700 5 osd.9 63 heartbeat: osd_stat(1026 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-91> 2013-08-15 20:50:32.512945 7fd94e0e37c0 10 monclient(hunting): _reopen_session
|
|
-90> 2013-08-15 20:50:32.512987 7fd94e0e37c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x284e840 addr 9.4.196.1:6789/0
|
|
-89> 2013-08-15 20:50:32.513001 7fd94e0e37c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-88> 2013-08-15 20:50:32.513006 7fd94e0e37c0 1 -- 0.0.0.0:6835/9435 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x25feb40 con 0x284e840
|
|
-87> 2013-08-15 20:50:32.513019 7fd94e0e37c0 10 monclient(hunting): renew_subs
|
|
-86> 2013-08-15 20:50:32.513399 7fd94e078700 1 -- 9.4.196.3:6835/9435 learned my addr 9.4.196.3:6835/9435
|
|
-85> 2013-08-15 20:50:32.513567 7fd940aab700 10 monclient(hunting): renew_subs
|
|
-84> 2013-08-15 20:50:32.513915 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x26741e0 con 0x284e840
|
|
-83> 2013-08-15 20:50:32.513933 7fd940aab700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-82> 2013-08-15 20:50:32.513949 7fd940aab700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-81> 2013-08-15 20:50:32.513953 7fd940aab700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-80> 2013-08-15 20:50:32.513971 7fd940aab700 1 monclient(hunting): found mon.zc2store
|
|
-79> 2013-08-15 20:50:32.513978 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2930406418 0 0) 0x2856200 con 0x284e840
|
|
-78> 2013-08-15 20:50:32.514001 7fd940aab700 10 monclient: my global_id is 4404
|
|
-77> 2013-08-15 20:50:32.514122 7fd940aab700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-76> 2013-08-15 20:50:32.514131 7fd940aab700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x25ffd40 con 0x284e840
|
|
-75> 2013-08-15 20:50:32.514610 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1622163214 0 0) 0x2856000 con 0x284e840
|
|
-74> 2013-08-15 20:50:32.514672 7fd940aab700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-73> 2013-08-15 20:50:32.514679 7fd940aab700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x285c240 con 0x284e840
|
|
-72> 2013-08-15 20:50:32.515188 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (1386130298 0 0) 0x2856400 con 0x284e840
|
|
-71> 2013-08-15 20:50:32.515231 7fd940aab700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-70> 2013-08-15 20:50:32.515237 7fd940aab700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x2634380 con 0x284e840
|
|
-69> 2013-08-15 20:50:32.515252 7fd940aab700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-68> 2013-08-15 20:50:32.515256 7fd940aab700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x2634540 con 0x284e840
|
|
-67> 2013-08-15 20:50:32.515269 7fd940aab700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:02.515268)
|
|
-66> 2013-08-15 20:50:32.515277 7fd940aab700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-65> 2013-08-15 20:50:32.515282 7fd940aab700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x285c000 con 0x284e840
|
|
-64> 2013-08-15 20:50:32.515327 7fd94e0e37c0 5 monclient: authenticate success, global_id 4404
|
|
-63> 2013-08-15 20:50:32.515340 7fd94e0e37c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:02.515339)
|
|
-62> 2013-08-15 20:50:32.515598 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x26745a0 con 0x284e840
|
|
-61> 2013-08-15 20:50:32.515618 7fd940aab700 10 monclient: handle_monmap mon_map v1
|
|
-60> 2013-08-15 20:50:32.515629 7fd940aab700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-59> 2013-08-15 20:50:32.515633 7fd940aab700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-58> 2013-08-15 20:50:32.515651 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x2634380 con 0x284e840
|
|
-57> 2013-08-15 20:50:32.515660 7fd940aab700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:32.513022 renew after 2013-08-15 20:53:02.513022
|
|
-56> 2013-08-15 20:50:32.515672 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x26743c0 con 0x284e840
|
|
-55> 2013-08-15 20:50:32.515679 7fd940aab700 10 monclient: handle_monmap mon_map v1
|
|
-54> 2013-08-15 20:50:32.515685 7fd940aab700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-53> 2013-08-15 20:50:32.515688 7fd940aab700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-52> 2013-08-15 20:50:32.515701 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x2634700 con 0x284e840
|
|
-51> 2013-08-15 20:50:32.515709 7fd940aab700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-50> 2013-08-15 20:50:32.515715 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (167835588 0 0) 0x2856600 con 0x284e840
|
|
-49> 2013-08-15 20:50:32.515758 7fd940aab700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:02.515758)
|
|
-48> 2013-08-15 20:50:32.515798 7fd94e0e37c0 10 monclient: wait_auth_rotating done
|
|
-47> 2013-08-15 20:50:32.516444 7fd94e0e37c0 10 monclient: get_version osdmap
|
|
-46> 2013-08-15 20:50:32.516455 7fd94e0e37c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-45> 2013-08-15 20:50:32.516463 7fd94e0e37c0 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x265a9c0 con 0x284e840
|
|
-44> 2013-08-15 20:50:32.516856 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=91) v2 ==== 24+0+0 (667974206 0 0) 0x2634540 con 0x284e840
|
|
-43> 2013-08-15 20:50:32.516927 7fd93caa3700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-42> 2013-08-15 20:50:32.516947 7fd93caa3700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- osd_boot(osd.9 booted 0 v63) v3 -- ?+0 0x2880000 con 0x284e840
|
|
-41> 2013-08-15 20:50:33.754838 7fd93faa9700 1 -- 9.4.196.3:6836/9435 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x285d200 con 0x284fb80
|
|
-40> 2013-08-15 20:50:33.754881 7fd93faa9700 3 osd.9 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-39> 2013-08-15 20:50:33.754891 7fd93faa9700 10 monclient: renew_subs
|
|
-38> 2013-08-15 20:50:33.754894 7fd93faa9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-37> 2013-08-15 20:50:33.754900 7fd93faa9700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x26348c0 con 0x284e840
|
|
-36> 2013-08-15 20:50:33.754918 7fd93faa9700 1 -- 9.4.196.3:6836/9435 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(3.17d epoch 93) v2 ==== 137+0+0 (1150340524 0 0) 0x26741e0 con 0x284fb80
|
|
-35> 2013-08-15 20:50:33.754933 7fd93faa9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.754803, event: header_read, request: pg_query(3.17d epoch 93) v2
|
|
-34> 2013-08-15 20:50:33.754943 7fd93faa9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.754803, event: throttled, request: pg_query(3.17d epoch 93) v2
|
|
-33> 2013-08-15 20:50:33.754949 7fd93faa9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.754844, event: all_read, request: pg_query(3.17d epoch 93) v2
|
|
-32> 2013-08-15 20:50:33.754955 7fd93faa9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.754927, event: dispatched, request: pg_query(3.17d epoch 93) v2
|
|
-31> 2013-08-15 20:50:33.754961 7fd93faa9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.754961, event: waiting_for_osdmap, request: pg_query(3.17d epoch 93) v2
|
|
-30> 2013-08-15 20:50:33.754972 7fd93faa9700 10 monclient: renew_subs
|
|
-29> 2013-08-15 20:50:33.754976 7fd93faa9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-28> 2013-08-15 20:50:33.754981 7fd93faa9700 1 -- 9.4.196.3:6835/9435 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x2634c40 con 0x284e840
|
|
-27> 2013-08-15 20:50:33.754991 7fd93faa9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.754991, event: wait for new map, request: pg_query(3.17d epoch 93) v2
|
|
-26> 2013-08-15 20:50:33.755387 7fd93778f700 1 -- 9.4.196.3:6837/9435 >> :/0 pipe(0x2851180 sd=33 :6837 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:49158/0
|
|
-25> 2013-08-15 20:50:33.756248 7fd940aab700 1 -- 9.4.196.3:6835/9435 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..92 src has 1..92) v3 ==== 28799+0+0 (1553658085 0 0) 0x285d8c0 con 0x284e840
|
|
-24> 2013-08-15 20:50:33.756274 7fd940aab700 3 osd.9 63 handle_osd_map epochs [64,92], i have 63, src has [1,92]
|
|
-23> 2013-08-15 20:50:33.757848 7fd93758d700 1 -- 9.4.196.3:6837/9435 >> :/0 pipe(0x2851680 sd=34 :6837 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:49159/0
|
|
-22> 2013-08-15 20:50:33.761769 7fd93faa9700 1 -- 9.4.196.3:6836/9435 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_query(0.25,1.24,2.23,3.22 epoch 93) v2 ==== 512+0+0 (3999158791 0 0) 0x2674780 con 0x284fb80
|
|
-21> 2013-08-15 20:50:33.762035 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-20> 2013-08-15 20:50:33.762052 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-19> 2013-08-15 20:50:33.762063 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-18> 2013-08-15 20:50:33.762082 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-17> 2013-08-15 20:50:33.762094 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-16> 2013-08-15 20:50:33.762099 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-15> 2013-08-15 20:50:33.762121 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-14> 2013-08-15 20:50:33.762128 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-13> 2013-08-15 20:50:33.762135 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-12> 2013-08-15 20:50:33.762143 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-11> 2013-08-15 20:50:33.762148 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-10> 2013-08-15 20:50:33.762156 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6826/8751 -- pipe dne
|
|
-9> 2013-08-15 20:50:33.762161 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6828/8914 -- pipe dne
|
|
-8> 2013-08-15 20:50:33.762169 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6833/9115 -- pipe dne
|
|
-7> 2013-08-15 20:50:33.762174 7fd940aab700 1 -- 9.4.196.3:6836/9435 mark_down 9.4.196.3:6830/9101 -- pipe dne
|
|
-6> 2013-08-15 20:50:33.763182 7fd946ab7700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-5> 2013-08-15 20:50:33.763188 7fd93738b700 1 -- 9.4.196.3:6837/9435 >> :/0 pipe(0x2851400 sd=35 :6837 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:49160/0
|
|
-4> 2013-08-15 20:50:33.763206 7fd946ab7700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
-3> 2013-08-15 20:50:33.763218 7fd93c2a2700 1 osd.9 pg_epoch: 92 pg[3.64( v 63'17 (0'0,63'17] local-les=62 n=5 ec=56 les/c 62/62 92/92/56) [4] r=-1 lpr=92 pi=56-91/15 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:33.763290 7fd93baa1700 1 osd.9 pg_epoch: 92 pg[3.1e7( v 63'12 (0'0,63'12] local-les=62 n=2 ec=56 les/c 62/62 92/92/56) [0]/[0,10] r=-1 lpr=92 pi=56-91/14 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:33.763995 7fd93baa1700 1 osd.9 pg_epoch: 92 pg[3.121( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 92/92/56) [1] r=-1 lpr=92 pi=56-91/15 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:33.764271 7fd946ab7700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fd946ab7700 time 2013-08-15 20:50:33.763213
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fd94dabff8e]
|
|
5: (clone()+0x6d) [0x7fd94bd71e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:33.767605 7fd946ab7700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fd946ab7700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fd94dac7bd0]
|
|
3: (gsignal()+0x37) [0x7fd94bcaf037]
|
|
4: (abort()+0x148) [0x7fd94bcb2698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fd94c5bbe8d]
|
|
6: (()+0x5ef76) [0x7fd94c5b9f76]
|
|
7: (()+0x5efa3) [0x7fd94c5b9fa3]
|
|
8: (()+0x5f1de) [0x7fd94c5ba1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fd94dabff8e]
|
|
14: (clone()+0x6d) [0x7fd94bd71e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-6> 2013-08-15 20:50:33.765119 7fd93c2a2700 1 osd.9 pg_epoch: 92 pg[3.1eb( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 92/92/85) [3]/[3,1] r=-1 lpr=92 pi=61-91/15 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:33.765701 7fd93c2a2700 1 osd.9 pg_epoch: 92 pg[3.127( v 63'13 (0'0,63'13] local-les=62 n=5 ec=56 les/c 62/62 92/92/56) [3] r=-1 lpr=92 pi=56-91/15 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:33.765828 7fd936f87700 1 -- 9.4.196.3:6837/9435 >> :/0 pipe(0x29ea500 sd=37 :6837 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:49162/0
|
|
-3> 2013-08-15 20:50:33.766376 7fd93c2a2700 1 osd.9 pg_epoch: 92 pg[3.1d7( v 63'14 (0'0,63'14] local-les=62 n=2 ec=56 les/c 62/62 92/92/85) [0]/[0,1] r=-1 lpr=92 pi=61-91/15 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:33.766922 7fd93c2a2700 1 osd.9 pg_epoch: 92 pg[3.110( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 92/92/56) [1]/[1,10] r=-1 lpr=92 pi=56-91/14 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:33.767459 7fd93c2a2700 1 osd.9 pg_epoch: 92 pg[3.1d1( v 63'17 (0'0,63'17] local-les=62 n=7 ec=56 les/c 62/62 92/92/56) [2]/[2,10] r=-1 lpr=92 pi=56-91/14 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:33.767605 7fd946ab7700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fd946ab7700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fd94dac7bd0]
|
|
3: (gsignal()+0x37) [0x7fd94bcaf037]
|
|
4: (abort()+0x148) [0x7fd94bcb2698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fd94c5bbe8d]
|
|
6: (()+0x5ef76) [0x7fd94c5b9f76]
|
|
7: (()+0x5efa3) [0x7fd94c5b9fa3]
|
|
8: (()+0x5f1de) [0x7fd94c5ba1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fd94dabff8e]
|
|
14: (clone()+0x6d) [0x7fd94bd71e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:33.932104 7f974cc4e7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9675
|
|
2013-08-15 20:50:33.932664 7f974cc4e7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:33.977517 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:33.977534 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:33.977828 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:33.989478 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:33.989575 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:34.013727 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:34.014625 7f974cc4e7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:34.014639 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:34.112747 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:34.113335 7f974cc4e7c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
2013-08-15 20:50:34.145131 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:34.145159 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:34.145494 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
2013-08-15 20:50:34.169018 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:34.169077 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
2013-08-15 20:50:34.193237 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:34.193785 7f974cc4e7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:34.193814 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:34.289857 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:35.772438 7f9745622700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:35.772529 7f9745622700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
2013-08-15 20:50:35.773908 7f9745622700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f9745622700 time 2013-08-15 20:50:35.772540
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f974c62af8e]
|
|
5: (clone()+0x6d) [0x7f974a8dce1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-275> 2013-08-15 20:50:33.930719 7f974cc4e7c0 5 asok(0x21cc0e0) register_command perfcounters_dump hook 0x21c6010
|
|
-274> 2013-08-15 20:50:33.930747 7f974cc4e7c0 5 asok(0x21cc0e0) register_command 1 hook 0x21c6010
|
|
-273> 2013-08-15 20:50:33.930761 7f974cc4e7c0 5 asok(0x21cc0e0) register_command perf dump hook 0x21c6010
|
|
-272> 2013-08-15 20:50:33.930767 7f974cc4e7c0 5 asok(0x21cc0e0) register_command perfcounters_schema hook 0x21c6010
|
|
-271> 2013-08-15 20:50:33.930769 7f974cc4e7c0 5 asok(0x21cc0e0) register_command 2 hook 0x21c6010
|
|
-270> 2013-08-15 20:50:33.930771 7f974cc4e7c0 5 asok(0x21cc0e0) register_command perf schema hook 0x21c6010
|
|
-269> 2013-08-15 20:50:33.930775 7f974cc4e7c0 5 asok(0x21cc0e0) register_command config show hook 0x21c6010
|
|
-268> 2013-08-15 20:50:33.930777 7f974cc4e7c0 5 asok(0x21cc0e0) register_command config set hook 0x21c6010
|
|
-267> 2013-08-15 20:50:33.930780 7f974cc4e7c0 5 asok(0x21cc0e0) register_command log flush hook 0x21c6010
|
|
-266> 2013-08-15 20:50:33.930782 7f974cc4e7c0 5 asok(0x21cc0e0) register_command log dump hook 0x21c6010
|
|
-265> 2013-08-15 20:50:33.930784 7f974cc4e7c0 5 asok(0x21cc0e0) register_command log reopen hook 0x21c6010
|
|
-264> 2013-08-15 20:50:33.932104 7f974cc4e7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9675
|
|
-263> 2013-08-15 20:50:33.932664 7f974cc4e7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-262> 2013-08-15 20:50:33.932816 7f974cc4e7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/9675 need_addr=1
|
|
-261> 2013-08-15 20:50:33.932847 7f974cc4e7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-260> 2013-08-15 20:50:33.932854 7f974cc4e7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6825/9675 need_addr=0
|
|
-259> 2013-08-15 20:50:33.932876 7f974cc4e7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-258> 2013-08-15 20:50:33.932880 7f974cc4e7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6835/9675 need_addr=0
|
|
-257> 2013-08-15 20:50:33.934613 7f974cc4e7c0 5 asok(0x21cc0e0) init /var/run/ceph/ceph-osd.9.asok
|
|
-256> 2013-08-15 20:50:33.934628 7f974cc4e7c0 5 asok(0x21cc0e0) bind_and_listen /var/run/ceph/ceph-osd.9.asok
|
|
-255> 2013-08-15 20:50:33.934656 7f974cc4e7c0 5 asok(0x21cc0e0) register_command 0 hook 0x21c40b8
|
|
-254> 2013-08-15 20:50:33.934662 7f974cc4e7c0 5 asok(0x21cc0e0) register_command version hook 0x21c40b8
|
|
-253> 2013-08-15 20:50:33.934665 7f974cc4e7c0 5 asok(0x21cc0e0) register_command git_version hook 0x21c40b8
|
|
-252> 2013-08-15 20:50:33.934667 7f974cc4e7c0 5 asok(0x21cc0e0) register_command help hook 0x21c60d0
|
|
-251> 2013-08-15 20:50:33.934711 7f9748628700 5 asok(0x21cc0e0) entry start
|
|
-250> 2013-08-15 20:50:33.977517 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-249> 2013-08-15 20:50:33.977534 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-248> 2013-08-15 20:50:33.977828 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-247> 2013-08-15 20:50:33.989478 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-246> 2013-08-15 20:50:33.989575 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-245> 2013-08-15 20:50:34.013727 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-244> 2013-08-15 20:50:34.014601 7f974cc4e7c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-243> 2013-08-15 20:50:34.014625 7f974cc4e7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-242> 2013-08-15 20:50:34.014639 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-241> 2013-08-15 20:50:34.020639 7f974cc4e7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-240> 2013-08-15 20:50:34.024768 7f974cc4e7c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-239> 2013-08-15 20:50:34.028819 7f974cc4e7c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-238> 2013-08-15 20:50:34.033028 7f974cc4e7c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-237> 2013-08-15 20:50:34.037037 7f974cc4e7c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-236> 2013-08-15 20:50:34.041101 7f974cc4e7c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-235> 2013-08-15 20:50:34.045277 7f974cc4e7c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-234> 2013-08-15 20:50:34.049320 7f974cc4e7c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-233> 2013-08-15 20:50:34.053248 7f974cc4e7c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-232> 2013-08-15 20:50:34.057186 7f974cc4e7c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-231> 2013-08-15 20:50:34.061085 7f974cc4e7c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-230> 2013-08-15 20:50:34.065003 7f974cc4e7c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-229> 2013-08-15 20:50:34.068925 7f974cc4e7c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-228> 2013-08-15 20:50:34.072808 7f974cc4e7c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-227> 2013-08-15 20:50:34.076725 7f974cc4e7c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-226> 2013-08-15 20:50:34.080579 7f974cc4e7c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-225> 2013-08-15 20:50:34.084428 7f974cc4e7c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-224> 2013-08-15 20:50:34.088298 7f974cc4e7c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-223> 2013-08-15 20:50:34.092396 7f974cc4e7c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-222> 2013-08-15 20:50:34.096480 7f974cc4e7c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-221> 2013-08-15 20:50:34.100598 7f974cc4e7c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-220> 2013-08-15 20:50:34.104673 7f974cc4e7c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-219> 2013-08-15 20:50:34.108645 7f974cc4e7c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-218> 2013-08-15 20:50:34.112617 7f974cc4e7c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-217> 2013-08-15 20:50:34.112673 7f974cc4e7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-216> 2013-08-15 20:50:34.112682 7f974cc4e7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-215> 2013-08-15 20:50:34.112690 7f974cc4e7c0 3 journal journal_replay: end of journal, done.
|
|
-214> 2013-08-15 20:50:34.112747 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-213> 2013-08-15 20:50:34.113160 7f9745622700 1 FileStore::op_tp worker finish
|
|
-212> 2013-08-15 20:50:34.113223 7f9745e23700 1 FileStore::op_tp worker finish
|
|
-211> 2013-08-15 20:50:34.113335 7f974cc4e7c0 1 journal close /var/lib/ceph/osd/ceph-9/journal
|
|
-210> 2013-08-15 20:50:34.113735 7f974cc4e7c0 10 monclient(hunting): build_initial_monmap
|
|
-209> 2013-08-15 20:50:34.113785 7f974cc4e7c0 5 adding auth protocol: cephx
|
|
-208> 2013-08-15 20:50:34.113792 7f974cc4e7c0 5 adding auth protocol: cephx
|
|
-207> 2013-08-15 20:50:34.113878 7f974cc4e7c0 1 -- 0.0.0.0:6805/9675 messenger.start
|
|
-206> 2013-08-15 20:50:34.113907 7f974cc4e7c0 1 -- :/0 messenger.start
|
|
-205> 2013-08-15 20:50:34.113918 7f974cc4e7c0 1 -- 9.4.196.3:6835/9675 messenger.start
|
|
-204> 2013-08-15 20:50:34.113929 7f974cc4e7c0 1 -- 9.4.196.3:6825/9675 messenger.start
|
|
-203> 2013-08-15 20:50:34.114008 7f974cc4e7c0 2 osd.9 0 mounting /var/lib/ceph/osd/ceph-9 /var/lib/ceph/osd/ceph-9/journal
|
|
-202> 2013-08-15 20:50:34.145131 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is supported and appears to work
|
|
-201> 2013-08-15 20:50:34.145159 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-200> 2013-08-15 20:50:34.145494 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount did NOT detect btrfs
|
|
-199> 2013-08-15 20:50:34.169018 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-198> 2013-08-15 20:50:34.169077 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount found snaps <>
|
|
-197> 2013-08-15 20:50:34.193237 7f974cc4e7c0 0 filestore(/var/lib/ceph/osd/ceph-9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-196> 2013-08-15 20:50:34.193755 7f974cc4e7c0 2 journal open /var/lib/ceph/osd/ceph-9/journal fsid 36289fb7-a31f-42f7-8ff1-f9d763da0a67 fs_op_seq 2530
|
|
-195> 2013-08-15 20:50:34.193785 7f974cc4e7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-194> 2013-08-15 20:50:34.193814 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-193> 2013-08-15 20:50:34.198137 7f974cc4e7c0 2 journal read_entry 3398758400 : seq 2507 4195925 bytes
|
|
-192> 2013-08-15 20:50:34.202031 7f974cc4e7c0 2 journal read_entry 3402960896 : seq 2508 4195919 bytes
|
|
-191> 2013-08-15 20:50:34.205963 7f974cc4e7c0 2 journal read_entry 3407163392 : seq 2509 4195925 bytes
|
|
-190> 2013-08-15 20:50:34.209976 7f974cc4e7c0 2 journal read_entry 3411365888 : seq 2510 4195925 bytes
|
|
-189> 2013-08-15 20:50:34.213886 7f974cc4e7c0 2 journal read_entry 3415568384 : seq 2511 4195919 bytes
|
|
-188> 2013-08-15 20:50:34.217720 7f974cc4e7c0 2 journal read_entry 3419770880 : seq 2512 4195925 bytes
|
|
-187> 2013-08-15 20:50:34.221612 7f974cc4e7c0 2 journal read_entry 3423973376 : seq 2513 4195925 bytes
|
|
-186> 2013-08-15 20:50:34.225565 7f974cc4e7c0 2 journal read_entry 3428175872 : seq 2514 4195925 bytes
|
|
-185> 2013-08-15 20:50:34.229421 7f974cc4e7c0 2 journal read_entry 3432378368 : seq 2515 4195919 bytes
|
|
-184> 2013-08-15 20:50:34.233422 7f974cc4e7c0 2 journal read_entry 3436580864 : seq 2516 4195919 bytes
|
|
-183> 2013-08-15 20:50:34.237308 7f974cc4e7c0 2 journal read_entry 3440783360 : seq 2517 4195919 bytes
|
|
-182> 2013-08-15 20:50:34.241326 7f974cc4e7c0 2 journal read_entry 3444985856 : seq 2518 4195913 bytes
|
|
-181> 2013-08-15 20:50:34.245414 7f974cc4e7c0 2 journal read_entry 3449188352 : seq 2519 4195925 bytes
|
|
-180> 2013-08-15 20:50:34.249466 7f974cc4e7c0 2 journal read_entry 3453390848 : seq 2520 4195919 bytes
|
|
-179> 2013-08-15 20:50:34.253511 7f974cc4e7c0 2 journal read_entry 3457593344 : seq 2521 4195925 bytes
|
|
-178> 2013-08-15 20:50:34.257562 7f974cc4e7c0 2 journal read_entry 3461795840 : seq 2522 4195925 bytes
|
|
-177> 2013-08-15 20:50:34.261588 7f974cc4e7c0 2 journal read_entry 3465998336 : seq 2523 4195925 bytes
|
|
-176> 2013-08-15 20:50:34.265608 7f974cc4e7c0 2 journal read_entry 3470200832 : seq 2524 4195925 bytes
|
|
-175> 2013-08-15 20:50:34.269651 7f974cc4e7c0 2 journal read_entry 3474403328 : seq 2525 4195925 bytes
|
|
-174> 2013-08-15 20:50:34.273697 7f974cc4e7c0 2 journal read_entry 3478605824 : seq 2526 4195925 bytes
|
|
-173> 2013-08-15 20:50:34.277742 7f974cc4e7c0 2 journal read_entry 3482808320 : seq 2527 4195925 bytes
|
|
-172> 2013-08-15 20:50:34.281754 7f974cc4e7c0 2 journal read_entry 3487010816 : seq 2528 4195925 bytes
|
|
-171> 2013-08-15 20:50:34.285763 7f974cc4e7c0 2 journal read_entry 3491213312 : seq 2529 4195925 bytes
|
|
-170> 2013-08-15 20:50:34.289773 7f974cc4e7c0 2 journal read_entry 3495415808 : seq 2530 4195925 bytes
|
|
-169> 2013-08-15 20:50:34.289806 7f974cc4e7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-168> 2013-08-15 20:50:34.289811 7f974cc4e7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-167> 2013-08-15 20:50:34.289812 7f974cc4e7c0 3 journal journal_replay: end of journal, done.
|
|
-166> 2013-08-15 20:50:34.289857 7f974cc4e7c0 1 journal _open /var/lib/ceph/osd/ceph-9/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-165> 2013-08-15 20:50:34.290081 7f974cc4e7c0 2 osd.9 0 boot
|
|
-164> 2013-08-15 20:50:34.316503 7f974cc4e7c0 2 osd.9 63 superblock: i am osd.9
|
|
-163> 2013-08-15 20:50:34.316554 7f974cc4e7c0 1 accepter.accepter.start
|
|
-162> 2013-08-15 20:50:34.316609 7f974cc4e7c0 1 accepter.accepter.start
|
|
-161> 2013-08-15 20:50:34.316664 7f974cc4e7c0 1 accepter.accepter.start
|
|
-160> 2013-08-15 20:50:34.316708 7f974cc4e7c0 10 monclient(hunting): init
|
|
-159> 2013-08-15 20:50:34.316716 7f974cc4e7c0 5 adding auth protocol: cephx
|
|
-158> 2013-08-15 20:50:34.316718 7f974cc4e7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-157> 2013-08-15 20:50:34.316857 7f974cbe3700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424000 sd=27 :6825 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.3:44281/0
|
|
-156> 2013-08-15 20:50:34.316858 7f973bd0e700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424280 sd=29 :6825 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.1:38039/0
|
|
-155> 2013-08-15 20:50:34.316861 7f973be0f700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424500 sd=28 :6825 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.3:44282/0
|
|
-154> 2013-08-15 20:50:34.316906 7f973bc0d700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424c80 sd=30 :6825 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:38040/0
|
|
-153> 2013-08-15 20:50:34.316964 7f973bb0c700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424a00 sd=31 :6825 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:38042/0
|
|
-152> 2013-08-15 20:50:34.316984 7f974cc4e7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-9/keyring
|
|
-151> 2013-08-15 20:50:34.317048 7f973ba0b700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424780 sd=32 :6825 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:38041/0
|
|
-150> 2013-08-15 20:50:34.317057 7f973b90a700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425680 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38044/0
|
|
-149> 2013-08-15 20:50:34.317229 7f974cc4e7c0 5 asok(0x21cc0e0) register_command dump_ops_in_flight hook 0x21c6710
|
|
-148> 2013-08-15 20:50:34.317240 7f974cc4e7c0 5 asok(0x21cc0e0) register_command dump_historic_ops hook 0x21c6710
|
|
-147> 2013-08-15 20:50:34.317245 7f974cc4e7c0 5 asok(0x21cc0e0) register_command dump_op_pq_state hook 0x21c6710
|
|
-146> 2013-08-15 20:50:34.317252 7f974cc4e7c0 5 asok(0x21cc0e0) register_command setomapval hook 0x23b9360
|
|
-145> 2013-08-15 20:50:34.317257 7f974cc4e7c0 5 asok(0x21cc0e0) register_command rmomapkey hook 0x23b9360
|
|
-144> 2013-08-15 20:50:34.317261 7f974cc4e7c0 5 asok(0x21cc0e0) register_command setomapheader hook 0x23b9360
|
|
-143> 2013-08-15 20:50:34.317266 7f974cc4e7c0 5 asok(0x21cc0e0) register_command getomap hook 0x23b9360
|
|
-142> 2013-08-15 20:50:34.317271 7f974cc4e7c0 5 asok(0x21cc0e0) register_command truncobj hook 0x23b9360
|
|
-141> 2013-08-15 20:50:34.317275 7f974cc4e7c0 5 asok(0x21cc0e0) register_command injectdataerr hook 0x23b9360
|
|
-140> 2013-08-15 20:50:34.317279 7f974cc4e7c0 5 asok(0x21cc0e0) register_command injectmdataerr hook 0x23b9360
|
|
-139> 2013-08-15 20:50:34.317277 7f9738002700 5 osd.9 63 heartbeat: osd_stat(1026 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-138> 2013-08-15 20:50:34.317337 7f974cc4e7c0 10 monclient(hunting): _reopen_session
|
|
-137> 2013-08-15 20:50:34.317359 7f974cc4e7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x2426160 addr 9.4.196.1:6789/0
|
|
-136> 2013-08-15 20:50:34.317372 7f974cc4e7c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-135> 2013-08-15 20:50:34.317378 7f974cc4e7c0 1 -- 0.0.0.0:6805/9675 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x21d8b40 con 0x2426160
|
|
-134> 2013-08-15 20:50:34.317391 7f974cc4e7c0 10 monclient(hunting): renew_subs
|
|
-133> 2013-08-15 20:50:34.317763 7f973b90a700 1 -- 9.4.196.3:6805/9675 learned my addr 9.4.196.3:6805/9675
|
|
-132> 2013-08-15 20:50:34.317948 7f973f616700 10 monclient(hunting): renew_subs
|
|
-131> 2013-08-15 20:50:34.318357 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x224e1e0 con 0x2426160
|
|
-130> 2013-08-15 20:50:34.318378 7f973f616700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-129> 2013-08-15 20:50:34.318392 7f973f616700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-128> 2013-08-15 20:50:34.318397 7f973f616700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-127> 2013-08-15 20:50:34.318416 7f973f616700 1 monclient(hunting): found mon.zc2store
|
|
-126> 2013-08-15 20:50:34.318424 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3438544984 0 0) 0x2432200 con 0x2426160
|
|
-125> 2013-08-15 20:50:34.318446 7f973f616700 10 monclient: my global_id is 4410
|
|
-124> 2013-08-15 20:50:34.318575 7f973f616700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-123> 2013-08-15 20:50:34.318584 7f973f616700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x21d9d40 con 0x2426160
|
|
-122> 2013-08-15 20:50:34.319076 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3938001559 0 0) 0x2432000 con 0x2426160
|
|
-121> 2013-08-15 20:50:34.319149 7f973f616700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-120> 2013-08-15 20:50:34.319156 7f973f616700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2438240 con 0x2426160
|
|
-119> 2013-08-15 20:50:34.319717 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (3364168134 0 0) 0x2432400 con 0x2426160
|
|
-118> 2013-08-15 20:50:34.319781 7f973f616700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-117> 2013-08-15 20:50:34.319788 7f973f616700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x220e380 con 0x2426160
|
|
-116> 2013-08-15 20:50:34.319805 7f973f616700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-115> 2013-08-15 20:50:34.319810 7f973f616700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x220e540 con 0x2426160
|
|
-114> 2013-08-15 20:50:34.319834 7f973f616700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:04.319833)
|
|
-113> 2013-08-15 20:50:34.319843 7f973f616700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-112> 2013-08-15 20:50:34.319847 7f973f616700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2438000 con 0x2426160
|
|
-111> 2013-08-15 20:50:34.319885 7f974cc4e7c0 5 monclient: authenticate success, global_id 4410
|
|
-110> 2013-08-15 20:50:34.319899 7f974cc4e7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:04.319898)
|
|
-109> 2013-08-15 20:50:34.320193 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x224e5a0 con 0x2426160
|
|
-108> 2013-08-15 20:50:34.320212 7f973f616700 10 monclient: handle_monmap mon_map v1
|
|
-107> 2013-08-15 20:50:34.320223 7f973f616700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-106> 2013-08-15 20:50:34.320227 7f973f616700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-105> 2013-08-15 20:50:34.320246 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x220e380 con 0x2426160
|
|
-104> 2013-08-15 20:50:34.320255 7f973f616700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:34.317394 renew after 2013-08-15 20:53:04.317394
|
|
-103> 2013-08-15 20:50:34.320268 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x224e3c0 con 0x2426160
|
|
-102> 2013-08-15 20:50:34.320276 7f973f616700 10 monclient: handle_monmap mon_map v1
|
|
-101> 2013-08-15 20:50:34.320284 7f973f616700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-100> 2013-08-15 20:50:34.320287 7f973f616700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-99> 2013-08-15 20:50:34.320301 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x220e700 con 0x2426160
|
|
-98> 2013-08-15 20:50:34.320309 7f973f616700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-97> 2013-08-15 20:50:34.320313 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (926361820 0 0) 0x243a600 con 0x2426160
|
|
-96> 2013-08-15 20:50:34.320359 7f973f616700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:04.320359)
|
|
-95> 2013-08-15 20:50:34.320398 7f974cc4e7c0 10 monclient: wait_auth_rotating done
|
|
-94> 2013-08-15 20:50:34.320869 7f974cc4e7c0 10 monclient: get_version osdmap
|
|
-93> 2013-08-15 20:50:34.320874 7f974cc4e7c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-92> 2013-08-15 20:50:34.320882 7f974cc4e7c0 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x22349c0 con 0x2426160
|
|
-91> 2013-08-15 20:50:34.321199 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=93) v2 ==== 24+0+0 (2145240 0 0) 0x220e540 con 0x2426160
|
|
-90> 2013-08-15 20:50:34.321247 7f973b008700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-89> 2013-08-15 20:50:34.321263 7f973b008700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- osd_boot(osd.9 booted 0 v63) v3 -- ?+0 0x2460000 con 0x2426160
|
|
-88> 2013-08-15 20:50:34.717216 7f97362fa700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425400 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44297/0
|
|
-87> 2013-08-15 20:50:34.717233 7f97361f9700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425180 sd=34 :6825 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:38074/0
|
|
-86> 2013-08-15 20:50:34.717264 7f9735ef6700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425900 sd=37 :6825 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:38076/0
|
|
-85> 2013-08-15 20:50:34.717292 7f9735df5700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2446c80 sd=38 :6825 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:38075/0
|
|
-84> 2013-08-15 20:50:34.717356 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425b80 sd=36 :6825 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:38073/0
|
|
-83> 2013-08-15 20:50:34.717364 7f97360f8700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424f00 sd=35 :6825 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:44298/0
|
|
-82> 2013-08-15 20:50:34.717362 7f9735cf4700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2446a00 sd=39 :6825 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.1:38078/0
|
|
-81> 2013-08-15 20:50:34.762273 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2446780 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44300/0
|
|
-80> 2013-08-15 20:50:34.767518 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2446500 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44301/0
|
|
-79> 2013-08-15 20:50:34.773735 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2446280 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38085/0
|
|
-78> 2013-08-15 20:50:34.774274 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2446000 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38086/0
|
|
-77> 2013-08-15 20:50:34.774775 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425400 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44302/0
|
|
-76> 2013-08-15 20:50:34.777067 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425180 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38087/0
|
|
-75> 2013-08-15 20:50:34.777456 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425900 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44303/0
|
|
-74> 2013-08-15 20:50:34.779332 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424f00 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38088/0
|
|
-73> 2013-08-15 20:50:34.783102 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425b80 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38089/0
|
|
-72> 2013-08-15 20:50:35.178449 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425400 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44304/0
|
|
-71> 2013-08-15 20:50:35.183112 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425180 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44305/0
|
|
-70> 2013-08-15 20:50:35.183327 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425900 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44306/0
|
|
-69> 2013-08-15 20:50:35.184871 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424f00 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44307/0
|
|
-68> 2013-08-15 20:50:35.517590 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425b80 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:38096/0
|
|
-67> 2013-08-15 20:50:35.585198 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425400 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44308/0
|
|
-66> 2013-08-15 20:50:35.760820 7f9735ff7700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425180 sd=33 :6825 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:44309/0
|
|
-65> 2013-08-15 20:50:35.762924 7f973e614700 1 -- 9.4.196.3:6825/9675 <== osd.6 9.4.196.3:6801/4121 1 ==== pg_notify(3.14a(14) epoch 95) v4 ==== 1146+0+0 (1797934205 0 0) 0x220e8c0 con 0x243f1e0
|
|
-64> 2013-08-15 20:50:35.762958 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.762839, event: header_read, request: pg_notify(3.14a(14) epoch 95) v4
|
|
-63> 2013-08-15 20:50:35.762970 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.762841, event: throttled, request: pg_notify(3.14a(14) epoch 95) v4
|
|
-62> 2013-08-15 20:50:35.762977 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.762906, event: all_read, request: pg_notify(3.14a(14) epoch 95) v4
|
|
-61> 2013-08-15 20:50:35.762983 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.762949, event: dispatched, request: pg_notify(3.14a(14) epoch 95) v4
|
|
-60> 2013-08-15 20:50:35.762990 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.762990, event: waiting_for_osdmap, request: pg_notify(3.14a(14) epoch 95) v4
|
|
-59> 2013-08-15 20:50:35.763002 7f973e614700 10 monclient: renew_subs
|
|
-58> 2013-08-15 20:50:35.763007 7f973e614700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-57> 2013-08-15 20:50:35.763015 7f973e614700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x220ea80 con 0x2426160
|
|
-56> 2013-08-15 20:50:35.763029 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.763028, event: wait for new map, request: pg_notify(3.14a(14) epoch 95) v4
|
|
-55> 2013-08-15 20:50:35.763292 7f973e614700 1 -- 9.4.196.3:6825/9675 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(94..95 src has 1..95) v3 ==== 1978+0+0 (4104616668 0 0) 0x2439440 con 0x243e2c0
|
|
-54> 2013-08-15 20:50:35.763320 7f973e614700 3 osd.9 63 handle_osd_map epochs [94,95], i have 63, src has [1,95]
|
|
-53> 2013-08-15 20:50:35.763329 7f973e614700 10 monclient: renew_subs
|
|
-52> 2013-08-15 20:50:35.763332 7f973e614700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-51> 2013-08-15 20:50:35.763338 7f973e614700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x220ee00 con 0x2426160
|
|
-50> 2013-08-15 20:50:35.763373 7f973e614700 1 -- 9.4.196.3:6825/9675 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_query(3.17b,3.185 epoch 95) v2 ==== 262+0+0 (2824360025 0 0) 0x224e1e0 con 0x243e2c0
|
|
-49> 2013-08-15 20:50:35.763401 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:35.763289, event: header_read, request: pg_query(3.17b,3.185 epoch 95) v2
|
|
-48> 2013-08-15 20:50:35.763420 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:35.763289, event: throttled, request: pg_query(3.17b,3.185 epoch 95) v2
|
|
-47> 2013-08-15 20:50:35.763434 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:35.763352, event: all_read, request: pg_query(3.17b,3.185 epoch 95) v2
|
|
-46> 2013-08-15 20:50:35.763444 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:35.763398, event: dispatched, request: pg_query(3.17b,3.185 epoch 95) v2
|
|
-45> 2013-08-15 20:50:35.763455 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:35.763455, event: waiting_for_osdmap, request: pg_query(3.17b,3.185 epoch 95) v2
|
|
-44> 2013-08-15 20:50:35.763470 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:35.763469, event: wait for new map, request: pg_query(3.17b,3.185 epoch 95) v2
|
|
-43> 2013-08-15 20:50:35.763708 7f973e614700 1 -- 9.4.196.3:6825/9675 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(94..95 src has 1..95) v3 ==== 1978+0+0 (4104616668 0 0) 0x2439b00 con 0x243e580
|
|
-42> 2013-08-15 20:50:35.763735 7f973e614700 3 osd.9 63 handle_osd_map epochs [94,95], i have 63, src has [1,95]
|
|
-41> 2013-08-15 20:50:35.763745 7f973e614700 10 monclient: renew_subs
|
|
-40> 2013-08-15 20:50:35.763750 7f973e614700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-39> 2013-08-15 20:50:35.763757 7f973e614700 1 -- 9.4.196.3:6805/9675 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x220ec40 con 0x2426160
|
|
-38> 2013-08-15 20:50:35.763827 7f97360f8700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425900 sd=34 :6825 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:38098/0
|
|
-37> 2013-08-15 20:50:35.763858 7f973e614700 1 -- 9.4.196.3:6825/9675 <== osd.1 9.4.196.1:6801/5083 3 ==== pg_query(0.7,1.6,2.5,3.4 epoch 95) v2 ==== 512+0+0 (780570937 0 0) 0x224e780 con 0x243e2c0
|
|
-36> 2013-08-15 20:50:35.763873 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:35.763797, event: header_read, request: pg_query(0.7,1.6,2.5,3.4 epoch 95) v2
|
|
-35> 2013-08-15 20:50:35.763884 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:35.763799, event: throttled, request: pg_query(0.7,1.6,2.5,3.4 epoch 95) v2
|
|
-34> 2013-08-15 20:50:35.763894 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:35.763844, event: all_read, request: pg_query(0.7,1.6,2.5,3.4 epoch 95) v2
|
|
-33> 2013-08-15 20:50:35.763904 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:35.763869, event: dispatched, request: pg_query(0.7,1.6,2.5,3.4 epoch 95) v2
|
|
-32> 2013-08-15 20:50:35.763915 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:35.763915, event: waiting_for_osdmap, request: pg_query(0.7,1.6,2.5,3.4 epoch 95) v2
|
|
-31> 2013-08-15 20:50:35.763926 7f973e614700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:35.763926, event: wait for new map, request: pg_query(0.7,1.6,2.5,3.4 epoch 95) v2
|
|
-30> 2013-08-15 20:50:35.764604 7f973f616700 1 -- 9.4.196.3:6805/9675 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..94 src has 1..94) v3 ==== 30872+0+0 (824186616 0 0) 0x2439440 con 0x2426160
|
|
-29> 2013-08-15 20:50:35.764628 7f973f616700 3 osd.9 63 handle_osd_map epochs [64,94], i have 63, src has [1,94]
|
|
-28> 2013-08-15 20:50:35.764743 7f973e614700 1 -- 9.4.196.3:6825/9675 <== osd.1 9.4.196.1:6801/5083 4 ==== pg_query(3.15a epoch 95) v2 ==== 137+0+0 (4174127534 0 0) 0x224e5a0 con 0x243e2c0
|
|
-27> 2013-08-15 20:50:35.765195 7f97360f8700 1 -- 9.4.196.3:6835/9675 >> :/0 pipe(0x2425900 sd=34 :6835 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:51500/0
|
|
-26> 2013-08-15 20:50:35.765653 7f9735ef6700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2424f00 sd=35 :6825 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:44311/0
|
|
-25> 2013-08-15 20:50:35.766807 7f9735ef6700 1 -- 9.4.196.3:6835/9675 >> :/0 pipe(0x2425400 sd=35 :6835 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:51501/0
|
|
-24> 2013-08-15 20:50:35.766844 7f97361f9700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2425b80 sd=36 :6825 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.3:44313/0
|
|
-23> 2013-08-15 20:50:35.767160 7f97362fa700 1 -- 9.4.196.3:6835/9675 >> :/0 pipe(0x2424f00 sd=37 :6835 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:51502/0
|
|
-22> 2013-08-15 20:50:35.768521 7f97359f1700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2503680 sd=38 :6825 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.3:44314/0
|
|
-21> 2013-08-15 20:50:35.771152 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-20> 2013-08-15 20:50:35.771166 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-19> 2013-08-15 20:50:35.771171 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-18> 2013-08-15 20:50:35.771189 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-17> 2013-08-15 20:50:35.771203 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-16> 2013-08-15 20:50:35.771208 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-15> 2013-08-15 20:50:35.771231 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-14> 2013-08-15 20:50:35.771239 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-13> 2013-08-15 20:50:35.771246 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-12> 2013-08-15 20:50:35.771256 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-11> 2013-08-15 20:50:35.771262 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-10> 2013-08-15 20:50:35.771273 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6826/8751 -- pipe dne
|
|
-9> 2013-08-15 20:50:35.771278 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6828/8914 -- pipe dne
|
|
-8> 2013-08-15 20:50:35.771285 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6833/9115 -- pipe dne
|
|
-7> 2013-08-15 20:50:35.771289 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6830/9101 -- pipe dne
|
|
-6> 2013-08-15 20:50:35.771295 7f973f616700 1 -- 9.4.196.3:6825/9675 mark_down 9.4.196.3:6839/9437 -- pipe dne
|
|
-5> 2013-08-15 20:50:35.772438 7f9745622700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-4> 2013-08-15 20:50:35.772529 7f9745622700 -1 journal FileJournal::do_write: write_bl(pos=3495415808) failed
|
|
-3> 2013-08-15 20:50:35.772520 7f97358f0700 1 -- 9.4.196.3:6835/9675 >> :/0 pipe(0x2503b80 sd=38 :6835 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:51506/0
|
|
-2> 2013-08-15 20:50:35.772992 7f97356ee700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2503400 sd=39 :6825 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.3:44315/0
|
|
-1> 2013-08-15 20:50:35.773538 7f97356ee700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2503180 sd=39 :6825 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.3:44316/0
|
|
0> 2013-08-15 20:50:35.773908 7f9745622700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f9745622700 time 2013-08-15 20:50:35.772540
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f974c62af8e]
|
|
5: (clone()+0x6d) [0x7f974a8dce1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:35.779286 7f9745622700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f9745622700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f974c632bd0]
|
|
3: (gsignal()+0x37) [0x7f974a81a037]
|
|
4: (abort()+0x148) [0x7f974a81d698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f974b126e8d]
|
|
6: (()+0x5ef76) [0x7f974b124f76]
|
|
7: (()+0x5efa3) [0x7f974b124fa3]
|
|
8: (()+0x5f1de) [0x7f974b1251de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f974c62af8e]
|
|
14: (clone()+0x6d) [0x7f974a8dce1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-13> 2013-08-15 20:50:35.773915 7f97355ed700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2502f00 sd=40 :6825 s=0 pgs=0 cs=0 l=0).accept sd=40 9.4.196.3:44317/0
|
|
-12> 2013-08-15 20:50:35.773938 7f973a006700 1 osd.9 pg_epoch: 94 pg[3.64( v 63'17 (0'0,63'17] local-les=62 n=5 ec=56 les/c 62/62 94/94/56) [4] r=-1 lpr=94 pi=56-93/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-11> 2013-08-15 20:50:35.774913 7f973a006700 1 osd.9 pg_epoch: 94 pg[3.1eb( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 94/94/85) [3]/[3,1] r=-1 lpr=94 pi=61-93/17 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-10> 2013-08-15 20:50:35.775475 7f97354ec700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2502c80 sd=39 :6825 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.1:38106/0
|
|
-9> 2013-08-15 20:50:35.775786 7f973a006700 1 osd.9 pg_epoch: 94 pg[3.127( v 63'13 (0'0,63'13] local-les=62 n=5 ec=56 les/c 62/62 94/94/56) [3] r=-1 lpr=94 pi=56-93/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-8> 2013-08-15 20:50:35.775910 7f97353eb700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2502a00 sd=41 :6825 s=0 pgs=0 cs=0 l=0).accept sd=41 9.4.196.3:44318/0
|
|
-7> 2013-08-15 20:50:35.776535 7f97353eb700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2502780 sd=41 :6825 s=0 pgs=0 cs=0 l=0).accept sd=41 9.4.196.1:38107/0
|
|
-6> 2013-08-15 20:50:35.776595 7f973a006700 1 osd.9 pg_epoch: 94 pg[3.1e7( v 63'12 (0'0,63'12] local-les=62 n=2 ec=56 les/c 62/62 94/94/56) [0]/[0,10] r=-1 lpr=94 pi=56-93/16 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:35.777781 7f973a807700 1 osd.9 pg_epoch: 94 pg[3.121( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 94/94/56) [1] r=-1 lpr=94 pi=56-93/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:35.777872 7f973a006700 1 osd.9 pg_epoch: 94 pg[3.123( v 63'16 (0'0,63'16] local-les=62 n=2 ec=56 les/c 62/62 94/94/85) [4]/[4,2] r=-1 lpr=94 pi=61-93/17 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:35.778426 7f97354ec700 1 -- 9.4.196.3:6825/9675 >> :/0 pipe(0x2502500 sd=39 :6825 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.1:38108/0
|
|
-2> 2013-08-15 20:50:35.778505 7f973a807700 1 osd.9 pg_epoch: 94 pg[3.11e( v 63'14 (0'0,63'14] local-les=62 n=8 ec=56 les/c 62/62 94/94/56) [0]/[0,10] r=-1 lpr=94 pi=56-93/16 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:35.779168 7f973a807700 1 osd.9 pg_epoch: 94 pg[3.1d7( v 63'14 (0'0,63'14] local-les=62 n=2 ec=56 les/c 62/62 94/94/85) [0]/[0,1] r=-1 lpr=94 pi=61-93/17 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:35.779286 7f9745622700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f9745622700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f974c632bd0]
|
|
3: (gsignal()+0x37) [0x7f974a81a037]
|
|
4: (abort()+0x148) [0x7f974a81d698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f974b126e8d]
|
|
6: (()+0x5ef76) [0x7f974b124f76]
|
|
7: (()+0x5efa3) [0x7f974b124fa3]
|
|
8: (()+0x5f1de) [0x7f974b1251de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f974c62af8e]
|
|
14: (clone()+0x6d) [0x7f974a8dce1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.9.log
|
|
--- end dump of recent events ---
|