|
2013-08-15 06:47:28.620946 7f356f4217c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3487
|
|
2013-08-15 06:47:28.621571 7f356f4217c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.623930 7f356f4217c0 0 filestore(/var/lib/ceph/osd/ceph-7) lock_fsid failed to lock /var/lib/ceph/osd/ceph-7/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.623946 7f356f4217c0 -1 filestore(/var/lib/ceph/osd/ceph-7) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.623972 7f356f4217c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-7: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.668448 7f549af647c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3765
|
|
2013-08-15 06:47:28.669004 7f549af647c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.671461 7f549af647c0 0 filestore(/var/lib/ceph/osd/ceph-7) lock_fsid failed to lock /var/lib/ceph/osd/ceph-7/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.671478 7f549af647c0 -1 filestore(/var/lib/ceph/osd/ceph-7) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.671502 7f549af647c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-7: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.719719 7f12c7ae97c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4000
|
|
2013-08-15 06:47:28.720266 7f12c7ae97c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.722860 7f12c7ae97c0 0 filestore(/var/lib/ceph/osd/ceph-7) lock_fsid failed to lock /var/lib/ceph/osd/ceph-7/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.722877 7f12c7ae97c0 -1 filestore(/var/lib/ceph/osd/ceph-7) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.722903 7f12c7ae97c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-7: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.770845 7f561e9a47c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4189
|
|
2013-08-15 06:47:28.771420 7f561e9a47c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.773664 7f561e9a47c0 0 filestore(/var/lib/ceph/osd/ceph-7) lock_fsid failed to lock /var/lib/ceph/osd/ceph-7/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.773681 7f561e9a47c0 -1 filestore(/var/lib/ceph/osd/ceph-7) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.773708 7f561e9a47c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-7: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.814659 7fe677d427c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4399
|
|
2013-08-15 06:47:28.815304 7fe677d427c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.818610 7fe677d427c0 0 filestore(/var/lib/ceph/osd/ceph-7) lock_fsid failed to lock /var/lib/ceph/osd/ceph-7/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.818626 7fe677d427c0 -1 filestore(/var/lib/ceph/osd/ceph-7) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.818652 7fe677d427c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-7: (16) Device or resource busy[0m
|
|
2013-08-15 20:46:26.417736 7f8662ec37c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3268
|
|
2013-08-15 20:46:26.421182 7f8662ec37c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.481303 7f05738ff7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3487
|
|
2013-08-15 20:46:26.481913 7f05738ff7c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.524840 7f34f19077c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3684
|
|
2013-08-15 20:46:26.525321 7f34f19077c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.572983 7f7548aea7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3894
|
|
2013-08-15 20:46:26.573536 7f7548aea7c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.613703 7f1d1b6267c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4092
|
|
2013-08-15 20:46:26.614214 7f1d1b6267c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.653561 7f80bc0227c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4288
|
|
2013-08-15 20:46:26.654019 7f80bc0227c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:50:22.474780 7f5accc987c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4125
|
|
2013-08-15 20:50:22.475415 7f5accc987c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:22.518953 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:22.518974 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:22.519250 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:22.524909 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:22.531606 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:22.667900 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:22.703820 7f5accc987c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:22.703839 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:23.365952 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.024501 7f5accc987c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
2013-08-15 20:50:24.054355 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:24.054370 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:24.054650 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:24.078330 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:24.078387 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:24.120968 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:24.121504 7f5accc987c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:24.121531 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 27: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.207251 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.269693 7f5ac2d54700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:25.269712 7f5ac2d54700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
2013-08-15 20:50:25.270808 7f5ac2d54700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f5ac2d54700 time 2013-08-15 20:50:25.269720
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f5acc674f8e]
|
|
5: (clone()+0x6d) [0x7f5aca926e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-253> 2013-08-15 20:50:22.473039 7f5accc987c0 5 asok(0x19e60e0) register_command perfcounters_dump hook 0x19e0010
|
|
-252> 2013-08-15 20:50:22.473072 7f5accc987c0 5 asok(0x19e60e0) register_command 1 hook 0x19e0010
|
|
-251> 2013-08-15 20:50:22.473097 7f5accc987c0 5 asok(0x19e60e0) register_command perf dump hook 0x19e0010
|
|
-250> 2013-08-15 20:50:22.473108 7f5accc987c0 5 asok(0x19e60e0) register_command perfcounters_schema hook 0x19e0010
|
|
-249> 2013-08-15 20:50:22.473113 7f5accc987c0 5 asok(0x19e60e0) register_command 2 hook 0x19e0010
|
|
-248> 2013-08-15 20:50:22.473118 7f5accc987c0 5 asok(0x19e60e0) register_command perf schema hook 0x19e0010
|
|
-247> 2013-08-15 20:50:22.473125 7f5accc987c0 5 asok(0x19e60e0) register_command config show hook 0x19e0010
|
|
-246> 2013-08-15 20:50:22.473131 7f5accc987c0 5 asok(0x19e60e0) register_command config set hook 0x19e0010
|
|
-245> 2013-08-15 20:50:22.473137 7f5accc987c0 5 asok(0x19e60e0) register_command log flush hook 0x19e0010
|
|
-244> 2013-08-15 20:50:22.473142 7f5accc987c0 5 asok(0x19e60e0) register_command log dump hook 0x19e0010
|
|
-243> 2013-08-15 20:50:22.473148 7f5accc987c0 5 asok(0x19e60e0) register_command log reopen hook 0x19e0010
|
|
-242> 2013-08-15 20:50:22.474780 7f5accc987c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4125
|
|
-241> 2013-08-15 20:50:22.475415 7f5accc987c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-240> 2013-08-15 20:50:22.475612 7f5accc987c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6812/4125 need_addr=1
|
|
-239> 2013-08-15 20:50:22.475649 7f5accc987c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-238> 2013-08-15 20:50:22.475658 7f5accc987c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6813/4125 need_addr=0
|
|
-237> 2013-08-15 20:50:22.475688 7f5accc987c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-236> 2013-08-15 20:50:22.475694 7f5accc987c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6814/4125 need_addr=0
|
|
-235> 2013-08-15 20:50:22.480781 7f5accc987c0 5 asok(0x19e60e0) init /var/run/ceph/ceph-osd.7.asok
|
|
-234> 2013-08-15 20:50:22.480801 7f5accc987c0 5 asok(0x19e60e0) bind_and_listen /var/run/ceph/ceph-osd.7.asok
|
|
-233> 2013-08-15 20:50:22.480825 7f5accc987c0 5 asok(0x19e60e0) register_command 0 hook 0x19de0b8
|
|
-232> 2013-08-15 20:50:22.480833 7f5accc987c0 5 asok(0x19e60e0) register_command version hook 0x19de0b8
|
|
-231> 2013-08-15 20:50:22.480840 7f5accc987c0 5 asok(0x19e60e0) register_command git_version hook 0x19de0b8
|
|
-230> 2013-08-15 20:50:22.480847 7f5accc987c0 5 asok(0x19e60e0) register_command help hook 0x19e00d0
|
|
-229> 2013-08-15 20:50:22.480882 7f5ac8672700 5 asok(0x19e60e0) entry start
|
|
-228> 2013-08-15 20:50:22.518953 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-227> 2013-08-15 20:50:22.518974 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-226> 2013-08-15 20:50:22.519250 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-225> 2013-08-15 20:50:22.524909 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-224> 2013-08-15 20:50:22.531606 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-223> 2013-08-15 20:50:22.667900 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-222> 2013-08-15 20:50:22.703777 7f5accc987c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2477
|
|
-221> 2013-08-15 20:50:22.703820 7f5accc987c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-220> 2013-08-15 20:50:22.703839 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-219> 2013-08-15 20:50:22.738617 7f5accc987c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-218> 2013-08-15 20:50:22.742866 7f5accc987c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-217> 2013-08-15 20:50:22.742888 7f5accc987c0 3 journal journal_replay: applying op seq 2478
|
|
-216> 2013-08-15 20:50:22.769954 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2478
|
|
-215> 2013-08-15 20:50:22.797792 7f5accc987c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-214> 2013-08-15 20:50:22.797817 7f5accc987c0 3 journal journal_replay: applying op seq 2479
|
|
-213> 2013-08-15 20:50:22.804414 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2479
|
|
-212> 2013-08-15 20:50:22.835320 7f5accc987c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-211> 2013-08-15 20:50:22.835338 7f5accc987c0 3 journal journal_replay: applying op seq 2480
|
|
-210> 2013-08-15 20:50:22.847555 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2480
|
|
-209> 2013-08-15 20:50:22.872744 7f5accc987c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-208> 2013-08-15 20:50:22.872766 7f5accc987c0 3 journal journal_replay: applying op seq 2481
|
|
-207> 2013-08-15 20:50:22.875853 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2481
|
|
-206> 2013-08-15 20:50:22.906496 7f5accc987c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-205> 2013-08-15 20:50:22.906513 7f5accc987c0 3 journal journal_replay: applying op seq 2482
|
|
-204> 2013-08-15 20:50:22.909405 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2482
|
|
-203> 2013-08-15 20:50:22.937259 7f5accc987c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-202> 2013-08-15 20:50:22.937276 7f5accc987c0 3 journal journal_replay: applying op seq 2483
|
|
-201> 2013-08-15 20:50:22.940207 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2483
|
|
-200> 2013-08-15 20:50:22.971896 7f5accc987c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-199> 2013-08-15 20:50:22.971917 7f5accc987c0 3 journal journal_replay: applying op seq 2484
|
|
-198> 2013-08-15 20:50:22.981536 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2484
|
|
-197> 2013-08-15 20:50:23.014478 7f5accc987c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-196> 2013-08-15 20:50:23.014498 7f5accc987c0 3 journal journal_replay: applying op seq 2485
|
|
-195> 2013-08-15 20:50:23.017158 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2485
|
|
-194> 2013-08-15 20:50:23.048299 7f5accc987c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-193> 2013-08-15 20:50:23.048318 7f5accc987c0 3 journal journal_replay: applying op seq 2486
|
|
-192> 2013-08-15 20:50:23.051463 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2486
|
|
-191> 2013-08-15 20:50:23.082906 7f5accc987c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-190> 2013-08-15 20:50:23.082925 7f5accc987c0 3 journal journal_replay: applying op seq 2487
|
|
-189> 2013-08-15 20:50:23.085809 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2487
|
|
-188> 2013-08-15 20:50:23.116300 7f5accc987c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-187> 2013-08-15 20:50:23.116320 7f5accc987c0 3 journal journal_replay: applying op seq 2488
|
|
-186> 2013-08-15 20:50:23.129460 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2488
|
|
-185> 2013-08-15 20:50:23.153046 7f5accc987c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-184> 2013-08-15 20:50:23.153063 7f5accc987c0 3 journal journal_replay: applying op seq 2489
|
|
-183> 2013-08-15 20:50:23.155464 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2489
|
|
-182> 2013-08-15 20:50:23.179958 7f5accc987c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-181> 2013-08-15 20:50:23.179975 7f5accc987c0 3 journal journal_replay: applying op seq 2490
|
|
-180> 2013-08-15 20:50:23.183813 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2490
|
|
-179> 2013-08-15 20:50:23.206326 7f5accc987c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-178> 2013-08-15 20:50:23.206345 7f5accc987c0 3 journal journal_replay: applying op seq 2491
|
|
-177> 2013-08-15 20:50:23.220482 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2491
|
|
-176> 2013-08-15 20:50:23.245184 7f5accc987c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-175> 2013-08-15 20:50:23.245200 7f5accc987c0 3 journal journal_replay: applying op seq 2492
|
|
-174> 2013-08-15 20:50:23.248307 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2492
|
|
-173> 2013-08-15 20:50:23.276740 7f5accc987c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-172> 2013-08-15 20:50:23.276759 7f5accc987c0 3 journal journal_replay: applying op seq 2493
|
|
-171> 2013-08-15 20:50:23.279641 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2493
|
|
-170> 2013-08-15 20:50:23.307309 7f5accc987c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-169> 2013-08-15 20:50:23.307330 7f5accc987c0 3 journal journal_replay: applying op seq 2494
|
|
-168> 2013-08-15 20:50:23.311282 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2494
|
|
-167> 2013-08-15 20:50:23.332960 7f5accc987c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-166> 2013-08-15 20:50:23.332979 7f5accc987c0 3 journal journal_replay: applying op seq 2495
|
|
-165> 2013-08-15 20:50:23.337525 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2495
|
|
-164> 2013-08-15 20:50:23.361504 7f5accc987c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-163> 2013-08-15 20:50:23.361523 7f5accc987c0 3 journal journal_replay: applying op seq 2496
|
|
-162> 2013-08-15 20:50:23.365850 7f5accc987c0 3 journal journal_replay: r = 0, op_seq now 2496
|
|
-161> 2013-08-15 20:50:23.365877 7f5accc987c0 2 journal No further valid entries found, journal is most likely valid
|
|
-160> 2013-08-15 20:50:23.365883 7f5accc987c0 3 journal journal_replay: end of journal, done.
|
|
-159> 2013-08-15 20:50:23.365952 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-158> 2013-08-15 20:50:24.024376 7f5ac5559700 1 FileStore::op_tp worker finish
|
|
-157> 2013-08-15 20:50:24.024418 7f5ac5d5a700 1 FileStore::op_tp worker finish
|
|
-156> 2013-08-15 20:50:24.024501 7f5accc987c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
-155> 2013-08-15 20:50:24.024843 7f5accc987c0 10 monclient(hunting): build_initial_monmap
|
|
-154> 2013-08-15 20:50:24.024889 7f5accc987c0 5 adding auth protocol: cephx
|
|
-153> 2013-08-15 20:50:24.024896 7f5accc987c0 5 adding auth protocol: cephx
|
|
-152> 2013-08-15 20:50:24.024982 7f5accc987c0 1 -- 0.0.0.0:6812/4125 messenger.start
|
|
-151> 2013-08-15 20:50:24.025009 7f5accc987c0 1 -- :/0 messenger.start
|
|
-150> 2013-08-15 20:50:24.025020 7f5accc987c0 1 -- 9.4.196.3:6814/4125 messenger.start
|
|
-149> 2013-08-15 20:50:24.025032 7f5accc987c0 1 -- 9.4.196.3:6813/4125 messenger.start
|
|
-148> 2013-08-15 20:50:24.025110 7f5accc987c0 2 osd.7 0 mounting /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
|
|
-147> 2013-08-15 20:50:24.054355 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-146> 2013-08-15 20:50:24.054370 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-145> 2013-08-15 20:50:24.054650 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-144> 2013-08-15 20:50:24.078330 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-143> 2013-08-15 20:50:24.078387 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-142> 2013-08-15 20:50:24.120968 7f5accc987c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-141> 2013-08-15 20:50:24.121483 7f5accc987c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-140> 2013-08-15 20:50:24.121504 7f5accc987c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-139> 2013-08-15 20:50:24.121531 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 27: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-138> 2013-08-15 20:50:24.126084 7f5accc987c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-137> 2013-08-15 20:50:24.130625 7f5accc987c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-136> 2013-08-15 20:50:24.135060 7f5accc987c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-135> 2013-08-15 20:50:24.139696 7f5accc987c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-134> 2013-08-15 20:50:24.144204 7f5accc987c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-133> 2013-08-15 20:50:24.148596 7f5accc987c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-132> 2013-08-15 20:50:24.153111 7f5accc987c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-131> 2013-08-15 20:50:24.157548 7f5accc987c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-130> 2013-08-15 20:50:24.161858 7f5accc987c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-129> 2013-08-15 20:50:24.166445 7f5accc987c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-128> 2013-08-15 20:50:24.171163 7f5accc987c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-127> 2013-08-15 20:50:24.175633 7f5accc987c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-126> 2013-08-15 20:50:24.180387 7f5accc987c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-125> 2013-08-15 20:50:24.184942 7f5accc987c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-124> 2013-08-15 20:50:24.189527 7f5accc987c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-123> 2013-08-15 20:50:24.193827 7f5accc987c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-122> 2013-08-15 20:50:24.198116 7f5accc987c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-121> 2013-08-15 20:50:24.202595 7f5accc987c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-120> 2013-08-15 20:50:24.207184 7f5accc987c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-119> 2013-08-15 20:50:24.207205 7f5accc987c0 2 journal No further valid entries found, journal is most likely valid
|
|
-118> 2013-08-15 20:50:24.207211 7f5accc987c0 2 journal No further valid entries found, journal is most likely valid
|
|
-117> 2013-08-15 20:50:24.207214 7f5accc987c0 3 journal journal_replay: end of journal, done.
|
|
-116> 2013-08-15 20:50:24.207251 7f5accc987c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-115> 2013-08-15 20:50:24.207419 7f5accc987c0 2 osd.7 0 boot
|
|
-114> 2013-08-15 20:50:24.265399 7f5accc987c0 2 osd.7 63 superblock: i am osd.7
|
|
-113> 2013-08-15 20:50:24.265623 7f5accc987c0 1 accepter.accepter.start
|
|
-112> 2013-08-15 20:50:24.265677 7f5accc987c0 1 accepter.accepter.start
|
|
-111> 2013-08-15 20:50:24.265746 7f5accc987c0 1 accepter.accepter.start
|
|
-110> 2013-08-15 20:50:24.265796 7f5accc987c0 10 monclient(hunting): init
|
|
-109> 2013-08-15 20:50:24.265817 7f5accc987c0 5 adding auth protocol: cephx
|
|
-108> 2013-08-15 20:50:24.265823 7f5accc987c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-107> 2013-08-15 20:50:24.265984 7f5accc987c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring
|
|
-106> 2013-08-15 20:50:24.266180 7f5accc987c0 5 asok(0x19e60e0) register_command dump_ops_in_flight hook 0x19e0760
|
|
-105> 2013-08-15 20:50:24.266192 7f5accc987c0 5 asok(0x19e60e0) register_command dump_historic_ops hook 0x19e0760
|
|
-104> 2013-08-15 20:50:24.266199 7f5accc987c0 5 asok(0x19e60e0) register_command dump_op_pq_state hook 0x19e0760
|
|
-103> 2013-08-15 20:50:24.266207 7f5accc987c0 5 asok(0x19e60e0) register_command setomapval hook 0x1d2ca40
|
|
-102> 2013-08-15 20:50:24.266213 7f5accc987c0 5 asok(0x19e60e0) register_command rmomapkey hook 0x1d2ca40
|
|
-101> 2013-08-15 20:50:24.266219 7f5accc987c0 5 asok(0x19e60e0) register_command setomapheader hook 0x1d2ca40
|
|
-100> 2013-08-15 20:50:24.266226 7f5accc987c0 5 asok(0x19e60e0) register_command getomap hook 0x1d2ca40
|
|
-99> 2013-08-15 20:50:24.266232 7f5accc987c0 5 asok(0x19e60e0) register_command truncobj hook 0x1d2ca40
|
|
-98> 2013-08-15 20:50:24.266223 7f5ab753d700 5 osd.7 63 heartbeat: osd_stat(734 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-97> 2013-08-15 20:50:24.266237 7f5accc987c0 5 asok(0x19e60e0) register_command injectdataerr hook 0x1d2ca40
|
|
-96> 2013-08-15 20:50:24.266243 7f5accc987c0 5 asok(0x19e60e0) register_command injectmdataerr hook 0x1d2ca40
|
|
-95> 2013-08-15 20:50:24.266321 7f5accc987c0 10 monclient(hunting): _reopen_session
|
|
-94> 2013-08-15 20:50:24.266373 7f5accc987c0 10 monclient(hunting): _pick_new_mon picked mon.noname-b con 0x1d78580 addr 9.4.196.5:6789/0
|
|
-93> 2013-08-15 20:50:24.266394 7f5accc987c0 10 monclient(hunting): _send_mon_message to mon.noname-b at 9.4.196.5:6789/0
|
|
-92> 2013-08-15 20:50:24.266413 7f5accc987c0 1 -- 0.0.0.0:6812/4125 --> 9.4.196.5:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x19f2b40 con 0x1d78580
|
|
-91> 2013-08-15 20:50:24.266441 7f5accc987c0 10 monclient(hunting): renew_subs
|
|
-90> 2013-08-15 20:50:24.266787 7f5accc7f700 1 -- 9.4.196.3:6812/4125 learned my addr 9.4.196.3:6812/4125
|
|
-89> 2013-08-15 20:50:24.267050 7f5abe54b700 10 monclient(hunting): renew_subs
|
|
-88> 2013-08-15 20:50:24.465590 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1adc1e0 con 0x1d78580
|
|
-87> 2013-08-15 20:50:24.465644 7f5abe54b700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-86> 2013-08-15 20:50:24.465673 7f5abe54b700 10 monclient(hunting): got monmap 1, mon.noname-b is now rank -1
|
|
-85> 2013-08-15 20:50:24.465676 7f5abe54b700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-84> 2013-08-15 20:50:24.465707 7f5abe54b700 1 monclient(hunting): found mon.ceph-dmon2
|
|
-83> 2013-08-15 20:50:24.465717 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (999625149 0 0) 0x1fd6600 con 0x1d78580
|
|
-82> 2013-08-15 20:50:24.465747 7f5abe54b700 10 monclient: my global_id is 4313
|
|
-81> 2013-08-15 20:50:24.465905 7f5abe54b700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-80> 2013-08-15 20:50:24.465917 7f5abe54b700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x19f3d40 con 0x1d78580
|
|
-79> 2013-08-15 20:50:24.467179 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3876097611 0 0) 0x1fd6400 con 0x1d78580
|
|
-78> 2013-08-15 20:50:24.467283 7f5abe54b700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-77> 2013-08-15 20:50:24.467292 7f5abe54b700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1d7c240 con 0x1d78580
|
|
-76> 2013-08-15 20:50:24.468768 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (2515574859 0 0) 0x1fd6800 con 0x1d78580
|
|
-75> 2013-08-15 20:50:24.468847 7f5abe54b700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-74> 2013-08-15 20:50:24.468856 7f5abe54b700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1ac6000 con 0x1d78580
|
|
-73> 2013-08-15 20:50:24.468877 7f5abe54b700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-72> 2013-08-15 20:50:24.468881 7f5abe54b700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x1ac6380 con 0x1d78580
|
|
-71> 2013-08-15 20:50:24.468905 7f5abe54b700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:54.468904)
|
|
-70> 2013-08-15 20:50:24.468917 7f5abe54b700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-69> 2013-08-15 20:50:24.468921 7f5abe54b700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1d7c000 con 0x1d78580
|
|
-68> 2013-08-15 20:50:24.468944 7f5accc987c0 5 monclient: authenticate success, global_id 4313
|
|
-67> 2013-08-15 20:50:24.468954 7f5accc987c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:54.468953)
|
|
-66> 2013-08-15 20:50:24.469643 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1adc5a0 con 0x1d78580
|
|
-65> 2013-08-15 20:50:24.469657 7f5abe54b700 10 monclient: handle_monmap mon_map v1
|
|
-64> 2013-08-15 20:50:24.469666 7f5abe54b700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-63> 2013-08-15 20:50:24.469669 7f5abe54b700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-62> 2013-08-15 20:50:24.469689 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1ac6000 con 0x1d78580
|
|
-61> 2013-08-15 20:50:24.469701 7f5abe54b700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:24.266454 renew after 2013-08-15 20:52:54.266454
|
|
-60> 2013-08-15 20:50:24.469913 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x1adc3c0 con 0x1d78580
|
|
-59> 2013-08-15 20:50:24.469925 7f5abe54b700 10 monclient: handle_monmap mon_map v1
|
|
-58> 2013-08-15 20:50:24.469933 7f5abe54b700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-57> 2013-08-15 20:50:24.469935 7f5abe54b700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-56> 2013-08-15 20:50:24.469953 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1ac6540 con 0x1d78580
|
|
-55> 2013-08-15 20:50:24.469966 7f5abe54b700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-54> 2013-08-15 20:50:24.470300 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (2214353505 0 0) 0x1fd6a00 con 0x1d78580
|
|
-53> 2013-08-15 20:50:24.470354 7f5abe54b700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:54.470354)
|
|
-52> 2013-08-15 20:50:24.470371 7f5accc987c0 10 monclient: wait_auth_rotating done
|
|
-51> 2013-08-15 20:50:24.470897 7f5accc987c0 10 monclient: get_version osdmap
|
|
-50> 2013-08-15 20:50:24.470904 7f5accc987c0 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-49> 2013-08-15 20:50:24.470912 7f5accc987c0 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x1a4e9c0 con 0x1d78580
|
|
-48> 2013-08-15 20:50:24.471535 7f5abe54b700 1 -- 9.4.196.3:6812/4125 <== mon.2 9.4.196.5:6789/0 10 ==== mon_check_map_ack(handle=1 version=83) v2 ==== 24+0+0 (2922730777 0 0) 0x1ac6380 con 0x1d78580
|
|
-47> 2013-08-15 20:50:24.471583 7f5aba543700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-46> 2013-08-15 20:50:24.471597 7f5aba543700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- osd_boot(osd.7 booted 0 v63) v3 -- ?+0 0x1d86000 con 0x1d78580
|
|
-45> 2013-08-15 20:50:25.255724 7f5ab5a37700 1 -- 9.4.196.3:6813/4125 >> :/0 pipe(0x1a29680 sd=33 :6813 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:43954/0
|
|
-44> 2013-08-15 20:50:25.256389 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.6 9.4.196.3:6801/4121 1 ==== pg_notify(3.175(4) epoch 84) v4 ==== 756+0+0 (2523743656 0 0) 0x1ac6700 con 0x1d84c60
|
|
-43> 2013-08-15 20:50:25.256433 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.256300, event: header_read, request: pg_notify(3.175(4) epoch 84) v4
|
|
-42> 2013-08-15 20:50:25.256454 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.256303, event: throttled, request: pg_notify(3.175(4) epoch 84) v4
|
|
-41> 2013-08-15 20:50:25.256465 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.256373, event: all_read, request: pg_notify(3.175(4) epoch 84) v4
|
|
-40> 2013-08-15 20:50:25.256474 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.256421, event: dispatched, request: pg_notify(3.175(4) epoch 84) v4
|
|
-39> 2013-08-15 20:50:25.256483 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.256483, event: waiting_for_osdmap, request: pg_notify(3.175(4) epoch 84) v4
|
|
-38> 2013-08-15 20:50:25.256498 7f5abd549700 10 monclient: renew_subs
|
|
-37> 2013-08-15 20:50:25.256507 7f5abd549700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-36> 2013-08-15 20:50:25.256519 7f5abd549700 1 -- 9.4.196.3:6812/4125 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x1ac6c40 con 0x1d78580
|
|
-35> 2013-08-15 20:50:25.256537 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.256537, event: wait for new map, request: pg_notify(3.175(4) epoch 84) v4
|
|
-34> 2013-08-15 20:50:25.256549 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.6 9.4.196.3:6801/4121 2 ==== pg_notify(3.176(6) epoch 84) v4 ==== 830+0+0 (4183081250 0 0) 0x1ac6a80 con 0x1d84c60
|
|
-33> 2013-08-15 20:50:25.256560 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.256398, event: header_read, request: pg_notify(3.176(6) epoch 84) v4
|
|
-32> 2013-08-15 20:50:25.256567 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.256399, event: throttled, request: pg_notify(3.176(6) epoch 84) v4
|
|
-31> 2013-08-15 20:50:25.256576 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.256452, event: all_read, request: pg_notify(3.176(6) epoch 84) v4
|
|
-30> 2013-08-15 20:50:25.256587 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.256558, event: dispatched, request: pg_notify(3.176(6) epoch 84) v4
|
|
-29> 2013-08-15 20:50:25.256596 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.256595, event: waiting_for_osdmap, request: pg_notify(3.176(6) epoch 84) v4
|
|
-28> 2013-08-15 20:50:25.256604 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.256604, event: wait for new map, request: pg_notify(3.176(6) epoch 84) v4
|
|
-27> 2013-08-15 20:50:25.258813 7f5ab5835700 1 -- 9.4.196.3:6813/4125 >> :/0 pipe(0x1a29b80 sd=34 :6813 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:43958/0
|
|
-26> 2013-08-15 20:50:25.259244 7f5ab5734700 1 -- 9.4.196.3:6814/4125 >> :/0 pipe(0x1da0000 sd=35 :6814 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:59402/0
|
|
-25> 2013-08-15 20:50:25.260047 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.6 9.4.196.3:6801/4121 3 ==== pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4 ==== 4102+0+0 (2587202093 0 0) 0x1ac68c0 con 0x1d84c60
|
|
-24> 2013-08-15 20:50:25.260073 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.259918, event: header_read, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-23> 2013-08-15 20:50:25.260087 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.259920, event: throttled, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-22> 2013-08-15 20:50:25.260097 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.260026, event: all_read, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-21> 2013-08-15 20:50:25.260106 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.260069, event: dispatched, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-20> 2013-08-15 20:50:25.260118 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.260118, event: waiting_for_osdmap, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-19> 2013-08-15 20:50:25.260129 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.260128, event: wait for new map, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-18> 2013-08-15 20:50:25.262676 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x1d7d680 con 0x1d79ce0
|
|
-17> 2013-08-15 20:50:25.262700 7f5abd549700 3 osd.7 63 handle_osd_map epochs [64,84], i have 63, src has [1,84]
|
|
-16> 2013-08-15 20:50:25.263127 7f5ab5431700 1 -- 9.4.196.3:6814/4125 >> :/0 pipe(0x1da0500 sd=36 :6814 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:59404/0
|
|
-15> 2013-08-15 20:50:25.264839 7f5ab522f700 1 -- 9.4.196.3:6813/4125 >> :/0 pipe(0x1a29900 sd=37 :6813 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.3:43960/0
|
|
-14> 2013-08-15 20:50:25.268547 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-13> 2013-08-15 20:50:25.268567 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-12> 2013-08-15 20:50:25.268577 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-11> 2013-08-15 20:50:25.268607 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-10> 2013-08-15 20:50:25.268631 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-9> 2013-08-15 20:50:25.268643 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-8> 2013-08-15 20:50:25.268681 7f5abd549700 1 -- 9.4.196.3:6813/4125 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-7> 2013-08-15 20:50:25.268696 7f5abd549700 1 osd.7 84 state: booting -> active
|
|
-6> 2013-08-15 20:50:25.269429 7f5ab9d42700 1 osd.7 pg_epoch: 84 pg[3.1ee( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 84/84/56) [2,7] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:25.269693 7f5ac2d54700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-4> 2013-08-15 20:50:25.269712 7f5ac2d54700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
-3> 2013-08-15 20:50:25.270093 7f5ab9d42700 1 osd.7 pg_epoch: 84 pg[3.1e8( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 84/84/56) [1,7] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:25.270178 7f5ab9d42700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e8(7) epoch 84) v4 -- ?+0 0x1ac7500 con 0x1d79340
|
|
-1> 2013-08-15 20:50:25.270210 7f5ab9d42700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ee(7) epoch 84) v4 -- ?+0 0x1ac7880 con 0x1d79760
|
|
0> 2013-08-15 20:50:25.270808 7f5ac2d54700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f5ac2d54700 time 2013-08-15 20:50:25.269720
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f5acc674f8e]
|
|
5: (clone()+0x6d) [0x7f5aca926e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:25.276211 7f5ac2d54700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f5ac2d54700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f5acc67cbd0]
|
|
3: (gsignal()+0x37) [0x7f5aca864037]
|
|
4: (abort()+0x148) [0x7f5aca867698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f5acb170e8d]
|
|
6: (()+0x5ef76) [0x7f5acb16ef76]
|
|
7: (()+0x5efa3) [0x7f5acb16efa3]
|
|
8: (()+0x5f1de) [0x7f5acb16f1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f5acc674f8e]
|
|
14: (clone()+0x6d) [0x7f5aca926e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-60> 2013-08-15 20:50:25.271007 7f5ab9541700 1 osd.7 pg_epoch: 84 pg[3.1e2( v 63'18 (0'0,63'18] local-les=62 n=6 ec=56 les/c 62/62 84/84/56) [1,7] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-59> 2013-08-15 20:50:25.271732 7f5ab9541700 1 osd.7 pg_epoch: 84 pg[3.5e( v 63'10 (0'0,63'10] local-les=62 n=0 ec=56 les/c 62/62 84/84/56) [4,7] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-58> 2013-08-15 20:50:25.271807 7f5ab9541700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e2(8) epoch 84) v4 -- ?+0 0x1ac7dc0 con 0x1d79340
|
|
-57> 2013-08-15 20:50:25.271834 7f5ab9541700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6813/8739 -- pg_notify(3.5e(8) epoch 84) v4 -- ?+0 0x1ac7c00 con 0x1d794a0
|
|
-56> 2013-08-15 20:50:25.272491 7f5ab9541700 1 osd.7 pg_epoch: 84 pg[3.11f( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 84/84/56) [1,7] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-55> 2013-08-15 20:50:25.272911 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.272911, event: started, request: pg_notify(3.175(4) epoch 84) v4
|
|
-54> 2013-08-15 20:50:25.272943 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:25.272943, event: done, request: pg_notify(3.175(4) epoch 84) v4
|
|
-53> 2013-08-15 20:50:25.272956 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.272956, event: started, request: pg_notify(3.176(6) epoch 84) v4
|
|
-52> 2013-08-15 20:50:25.272976 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:25.272976, event: done, request: pg_notify(3.176(6) epoch 84) v4
|
|
-51> 2013-08-15 20:50:25.272987 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.272987, event: started, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-50> 2013-08-15 20:50:25.273048 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:25.273048, event: done, request: pg_notify(3.1f(6),0.f(6),1.e(6),2.d(6),3.c(6) epoch 84) v4
|
|
-49> 2013-08-15 20:50:25.273072 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_notify(3.a2(5) epoch 84) v4 ==== 787+0+0 (2764250569 0 0) 0x1ac6e00 con 0x1d79ce0
|
|
-48> 2013-08-15 20:50:25.273089 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.262677, event: header_read, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-47> 2013-08-15 20:50:25.273097 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.262678, event: throttled, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-46> 2013-08-15 20:50:25.273104 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.262760, event: all_read, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-45> 2013-08-15 20:50:25.273110 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.273087, event: dispatched, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-44> 2013-08-15 20:50:25.273117 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.273117, event: waiting_for_osdmap, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-43> 2013-08-15 20:50:25.273125 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.273125, event: started, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-42> 2013-08-15 20:50:25.273146 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:25.273146, event: done, request: pg_notify(3.a2(5) epoch 84) v4
|
|
-41> 2013-08-15 20:50:25.273158 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.175(3) epoch 84) v4 ==== 713+0+0 (2737411217 0 0) 0x1ac7180 con 0x1d84dc0
|
|
-40> 2013-08-15 20:50:25.273172 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.266302, event: header_read, request: pg_notify(3.175(3) epoch 84) v4
|
|
-39> 2013-08-15 20:50:25.273179 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.266304, event: throttled, request: pg_notify(3.175(3) epoch 84) v4
|
|
-38> 2013-08-15 20:50:25.273185 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.266363, event: all_read, request: pg_notify(3.175(3) epoch 84) v4
|
|
-37> 2013-08-15 20:50:25.273192 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.273170, event: dispatched, request: pg_notify(3.175(3) epoch 84) v4
|
|
-36> 2013-08-15 20:50:25.273199 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.273199, event: waiting_for_osdmap, request: pg_notify(3.175(3) epoch 84) v4
|
|
-35> 2013-08-15 20:50:25.273208 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.273208, event: started, request: pg_notify(3.175(3) epoch 84) v4
|
|
-34> 2013-08-15 20:50:25.273230 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:25.273230, event: done, request: pg_notify(3.175(3) epoch 84) v4
|
|
-33> 2013-08-15 20:50:25.273242 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.5 9.4.196.3:6819/4129 1 ==== pg_notify(3.24a(4) epoch 84) v4 ==== 756+0+0 (2570003036 0 0) 0x1e96000 con 0x1d85340
|
|
-32> 2013-08-15 20:50:25.273256 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.272512, event: header_read, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-31> 2013-08-15 20:50:25.273263 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.272514, event: throttled, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-30> 2013-08-15 20:50:25.273270 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.272572, event: all_read, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-29> 2013-08-15 20:50:25.273276 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.273254, event: dispatched, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-28> 2013-08-15 20:50:25.273282 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.273282, event: waiting_for_osdmap, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-27> 2013-08-15 20:50:25.273290 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.273290, event: started, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-26> 2013-08-15 20:50:25.273312 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:25.273312, event: done, request: pg_notify(3.24a(4) epoch 84) v4
|
|
-25> 2013-08-15 20:50:25.273325 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x1d7dd40 con 0x1d79340
|
|
-24> 2013-08-15 20:50:25.273338 7f5abd549700 3 osd.7 84 handle_osd_map epochs [64,84], i have 84, src has [1,84]
|
|
-23> 2013-08-15 20:50:25.273350 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(64..84 src has 1..84) v3 ==== 16823+0+0 (396616316 0 0) 0x1df86c0 con 0x1d79760
|
|
-22> 2013-08-15 20:50:25.273362 7f5abd549700 3 osd.7 84 handle_osd_map epochs [64,84], i have 84, src has [1,84]
|
|
-21> 2013-08-15 20:50:25.273373 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_query(3.207 epoch 84) v2 ==== 137+0+0 (2477302200 0 0) 0x1adc1e0 con 0x1d79ce0
|
|
-20> 2013-08-15 20:50:25.273386 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.263261, event: header_read, request: pg_query(3.207 epoch 84) v2
|
|
-19> 2013-08-15 20:50:25.273395 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.263263, event: throttled, request: pg_query(3.207 epoch 84) v2
|
|
-18> 2013-08-15 20:50:25.273401 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.263330, event: all_read, request: pg_query(3.207 epoch 84) v2
|
|
-17> 2013-08-15 20:50:25.273407 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.273385, event: dispatched, request: pg_query(3.207 epoch 84) v2
|
|
-16> 2013-08-15 20:50:25.273414 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.273414, event: waiting_for_osdmap, request: pg_query(3.207 epoch 84) v2
|
|
-15> 2013-08-15 20:50:25.273425 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.273424, event: started, request: pg_query(3.207 epoch 84) v2
|
|
-14> 2013-08-15 20:50:25.273447 7f5abd549700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6804/5992 -- pg_notify(3.207 epoch 84) v4 -- ?+0 0x1e96380 con 0x1d79ce0
|
|
-13> 2013-08-15 20:50:25.273467 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:25.273467, event: done, request: pg_query(3.207 epoch 84) v2
|
|
-12> 2013-08-15 20:50:25.273481 7f5abd549700 1 -- 9.4.196.3:6813/4125 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4 ==== 2193+0+0 (760992370 0 0) 0x1ac6fc0 con 0x1d79340
|
|
-11> 2013-08-15 20:50:25.273496 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.264702, event: header_read, request: pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4
|
|
-10> 2013-08-15 20:50:25.273504 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.264703, event: throttled, request: pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4
|
|
-9> 2013-08-15 20:50:25.273511 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.264778, event: all_read, request: pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4
|
|
-8> 2013-08-15 20:50:25.273519 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.273494, event: dispatched, request: pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4
|
|
-7> 2013-08-15 20:50:25.273526 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.273526, event: waiting_for_osdmap, request: pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4
|
|
-6> 2013-08-15 20:50:25.273536 7f5abd549700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:25.273536, event: started, request: pg_notify(3.175(3),3.145(3),3.102(5) epoch 84) v4
|
|
-5> 2013-08-15 20:50:25.274199 7f5ab9541700 1 osd.7 pg_epoch: 84 pg[3.11a( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 84/84/56) [2,7] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:25.274274 7f5ab9541700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6801/5083 -- pg_notify(3.11f(8) epoch 84) v4 -- ?+0 0x1e968c0 con 0x1d79340
|
|
-3> 2013-08-15 20:50:25.274328 7f5ab9541700 1 -- 9.4.196.3:6813/4125 --> 9.4.196.1:6807/6887 -- pg_notify(3.11a(8) epoch 84) v4 -- ?+0 0x1e96700 con 0x1d79760
|
|
-2> 2013-08-15 20:50:25.275183 7f5ab9d42700 1 osd.7 pg_epoch: 84 pg[3.5b( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 84/84/56) [0,7] r=1 lpr=84 pi=56-83/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:25.275807 7f5ab9d42700 1 osd.7 pg_epoch: 84 pg[3.1d4( v 63'15 (0'0,63'15] local-les=62 n=3 ec=56 les/c 62/62 84/84/56) [0,7] r=1 lpr=84 pi=56-83/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:25.276211 7f5ac2d54700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f5ac2d54700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f5acc67cbd0]
|
|
3: (gsignal()+0x37) [0x7f5aca864037]
|
|
4: (abort()+0x148) [0x7f5aca867698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f5acb170e8d]
|
|
6: (()+0x5ef76) [0x7f5acb16ef76]
|
|
7: (()+0x5efa3) [0x7f5acb16efa3]
|
|
8: (()+0x5f1de) [0x7f5acb16f1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f5acc674f8e]
|
|
14: (clone()+0x6d) [0x7f5aca926e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:25.484675 7fa8b60017c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8361
|
|
2013-08-15 20:50:25.485203 7fa8b60017c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:25.517816 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:25.517833 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:25.518101 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:25.535765 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:25.535857 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:25.560189 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:25.561065 7fa8b60017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:25.561080 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.646882 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.647194 7fa8b60017c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
2013-08-15 20:50:25.673825 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:25.673841 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:25.674120 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:25.685822 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:25.685877 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:25.716060 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:25.716610 7fa8b60017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:25.716658 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:25.796376 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.277240 7fa8ae9d5700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:27.277258 7fa8ae9d5700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
2013-08-15 20:50:27.278258 7fa8ae9d5700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fa8ae9d5700 time 2013-08-15 20:50:27.277266
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fa8b59ddf8e]
|
|
5: (clone()+0x6d) [0x7fa8b3c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-213> 2013-08-15 20:50:25.483328 7fa8b60017c0 5 asok(0x29f40e0) register_command perfcounters_dump hook 0x29ee010
|
|
-212> 2013-08-15 20:50:25.483358 7fa8b60017c0 5 asok(0x29f40e0) register_command 1 hook 0x29ee010
|
|
-211> 2013-08-15 20:50:25.483365 7fa8b60017c0 5 asok(0x29f40e0) register_command perf dump hook 0x29ee010
|
|
-210> 2013-08-15 20:50:25.483374 7fa8b60017c0 5 asok(0x29f40e0) register_command perfcounters_schema hook 0x29ee010
|
|
-209> 2013-08-15 20:50:25.483378 7fa8b60017c0 5 asok(0x29f40e0) register_command 2 hook 0x29ee010
|
|
-208> 2013-08-15 20:50:25.483382 7fa8b60017c0 5 asok(0x29f40e0) register_command perf schema hook 0x29ee010
|
|
-207> 2013-08-15 20:50:25.483388 7fa8b60017c0 5 asok(0x29f40e0) register_command config show hook 0x29ee010
|
|
-206> 2013-08-15 20:50:25.483392 7fa8b60017c0 5 asok(0x29f40e0) register_command config set hook 0x29ee010
|
|
-205> 2013-08-15 20:50:25.483396 7fa8b60017c0 5 asok(0x29f40e0) register_command log flush hook 0x29ee010
|
|
-204> 2013-08-15 20:50:25.483402 7fa8b60017c0 5 asok(0x29f40e0) register_command log dump hook 0x29ee010
|
|
-203> 2013-08-15 20:50:25.483417 7fa8b60017c0 5 asok(0x29f40e0) register_command log reopen hook 0x29ee010
|
|
-202> 2013-08-15 20:50:25.484675 7fa8b60017c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8361
|
|
-201> 2013-08-15 20:50:25.485203 7fa8b60017c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-200> 2013-08-15 20:50:25.485325 7fa8b60017c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6813/8361 need_addr=1
|
|
-199> 2013-08-15 20:50:25.485353 7fa8b60017c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-198> 2013-08-15 20:50:25.485360 7fa8b60017c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6822/8361 need_addr=0
|
|
-197> 2013-08-15 20:50:25.485376 7fa8b60017c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-196> 2013-08-15 20:50:25.485381 7fa8b60017c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6823/8361 need_addr=0
|
|
-195> 2013-08-15 20:50:25.487222 7fa8b60017c0 5 asok(0x29f40e0) init /var/run/ceph/ceph-osd.7.asok
|
|
-194> 2013-08-15 20:50:25.487237 7fa8b60017c0 5 asok(0x29f40e0) bind_and_listen /var/run/ceph/ceph-osd.7.asok
|
|
-193> 2013-08-15 20:50:25.487266 7fa8b60017c0 5 asok(0x29f40e0) register_command 0 hook 0x29ec0b8
|
|
-192> 2013-08-15 20:50:25.487271 7fa8b60017c0 5 asok(0x29f40e0) register_command version hook 0x29ec0b8
|
|
-191> 2013-08-15 20:50:25.487276 7fa8b60017c0 5 asok(0x29f40e0) register_command git_version hook 0x29ec0b8
|
|
-190> 2013-08-15 20:50:25.487281 7fa8b60017c0 5 asok(0x29f40e0) register_command help hook 0x29ee0d0
|
|
-189> 2013-08-15 20:50:25.487305 7fa8b19db700 5 asok(0x29f40e0) entry start
|
|
-188> 2013-08-15 20:50:25.517816 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-187> 2013-08-15 20:50:25.517833 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-186> 2013-08-15 20:50:25.518101 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-185> 2013-08-15 20:50:25.535765 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-184> 2013-08-15 20:50:25.535857 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-183> 2013-08-15 20:50:25.560189 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-182> 2013-08-15 20:50:25.561037 7fa8b60017c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-181> 2013-08-15 20:50:25.561065 7fa8b60017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-180> 2013-08-15 20:50:25.561080 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-179> 2013-08-15 20:50:25.567923 7fa8b60017c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-178> 2013-08-15 20:50:25.572343 7fa8b60017c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-177> 2013-08-15 20:50:25.576824 7fa8b60017c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-176> 2013-08-15 20:50:25.581301 7fa8b60017c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-175> 2013-08-15 20:50:25.585730 7fa8b60017c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-174> 2013-08-15 20:50:25.590084 7fa8b60017c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-173> 2013-08-15 20:50:25.594603 7fa8b60017c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-172> 2013-08-15 20:50:25.599057 7fa8b60017c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-171> 2013-08-15 20:50:25.603467 7fa8b60017c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-170> 2013-08-15 20:50:25.607707 7fa8b60017c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-169> 2013-08-15 20:50:25.612175 7fa8b60017c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-168> 2013-08-15 20:50:25.616539 7fa8b60017c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-167> 2013-08-15 20:50:25.620842 7fa8b60017c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-166> 2013-08-15 20:50:25.625230 7fa8b60017c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-165> 2013-08-15 20:50:25.629632 7fa8b60017c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-164> 2013-08-15 20:50:25.633936 7fa8b60017c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-163> 2013-08-15 20:50:25.638127 7fa8b60017c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-162> 2013-08-15 20:50:25.642456 7fa8b60017c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-161> 2013-08-15 20:50:25.646808 7fa8b60017c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-160> 2013-08-15 20:50:25.646841 7fa8b60017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-159> 2013-08-15 20:50:25.646848 7fa8b60017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-158> 2013-08-15 20:50:25.646854 7fa8b60017c0 3 journal journal_replay: end of journal, done.
|
|
-157> 2013-08-15 20:50:25.646882 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-156> 2013-08-15 20:50:25.647126 7fa8ae9d5700 1 FileStore::op_tp worker finish
|
|
-155> 2013-08-15 20:50:25.647140 7fa8af1d6700 1 FileStore::op_tp worker finish
|
|
-154> 2013-08-15 20:50:25.647194 7fa8b60017c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
-153> 2013-08-15 20:50:25.647427 7fa8b60017c0 10 monclient(hunting): build_initial_monmap
|
|
-152> 2013-08-15 20:50:25.647470 7fa8b60017c0 5 adding auth protocol: cephx
|
|
-151> 2013-08-15 20:50:25.647476 7fa8b60017c0 5 adding auth protocol: cephx
|
|
-150> 2013-08-15 20:50:25.647545 7fa8b60017c0 1 -- 0.0.0.0:6813/8361 messenger.start
|
|
-149> 2013-08-15 20:50:25.647570 7fa8b60017c0 1 -- :/0 messenger.start
|
|
-148> 2013-08-15 20:50:25.647581 7fa8b60017c0 1 -- 9.4.196.3:6823/8361 messenger.start
|
|
-147> 2013-08-15 20:50:25.647592 7fa8b60017c0 1 -- 9.4.196.3:6822/8361 messenger.start
|
|
-146> 2013-08-15 20:50:25.647690 7fa8b60017c0 2 osd.7 0 mounting /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
|
|
-145> 2013-08-15 20:50:25.673825 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-144> 2013-08-15 20:50:25.673841 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-143> 2013-08-15 20:50:25.674120 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-142> 2013-08-15 20:50:25.685822 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-141> 2013-08-15 20:50:25.685877 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-140> 2013-08-15 20:50:25.716060 7fa8b60017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-139> 2013-08-15 20:50:25.716578 7fa8b60017c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-138> 2013-08-15 20:50:25.716610 7fa8b60017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-137> 2013-08-15 20:50:25.716658 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-136> 2013-08-15 20:50:25.721063 7fa8b60017c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-135> 2013-08-15 20:50:25.725038 7fa8b60017c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-134> 2013-08-15 20:50:25.729212 7fa8b60017c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-133> 2013-08-15 20:50:25.733416 7fa8b60017c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-132> 2013-08-15 20:50:25.737713 7fa8b60017c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-131> 2013-08-15 20:50:25.741901 7fa8b60017c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-130> 2013-08-15 20:50:25.746066 7fa8b60017c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-129> 2013-08-15 20:50:25.750249 7fa8b60017c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-128> 2013-08-15 20:50:25.754462 7fa8b60017c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-127> 2013-08-15 20:50:25.758595 7fa8b60017c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-126> 2013-08-15 20:50:25.762682 7fa8b60017c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-125> 2013-08-15 20:50:25.766836 7fa8b60017c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-124> 2013-08-15 20:50:25.771178 7fa8b60017c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-123> 2013-08-15 20:50:25.775291 7fa8b60017c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-122> 2013-08-15 20:50:25.779587 7fa8b60017c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-121> 2013-08-15 20:50:25.783729 7fa8b60017c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-120> 2013-08-15 20:50:25.787998 7fa8b60017c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-119> 2013-08-15 20:50:25.792211 7fa8b60017c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-118> 2013-08-15 20:50:25.796315 7fa8b60017c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-117> 2013-08-15 20:50:25.796336 7fa8b60017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-116> 2013-08-15 20:50:25.796341 7fa8b60017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-115> 2013-08-15 20:50:25.796343 7fa8b60017c0 3 journal journal_replay: end of journal, done.
|
|
-114> 2013-08-15 20:50:25.796376 7fa8b60017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-113> 2013-08-15 20:50:25.796567 7fa8b60017c0 2 osd.7 0 boot
|
|
-112> 2013-08-15 20:50:25.821787 7fa8b60017c0 2 osd.7 63 superblock: i am osd.7
|
|
-111> 2013-08-15 20:50:25.821858 7fa8b60017c0 1 accepter.accepter.start
|
|
-110> 2013-08-15 20:50:25.821904 7fa8b60017c0 1 accepter.accepter.start
|
|
-109> 2013-08-15 20:50:25.821997 7fa8b60017c0 1 accepter.accepter.start
|
|
-108> 2013-08-15 20:50:25.822039 7fa8b60017c0 10 monclient(hunting): init
|
|
-107> 2013-08-15 20:50:25.822059 7fa8b60017c0 5 adding auth protocol: cephx
|
|
-106> 2013-08-15 20:50:25.822061 7fa8b60017c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-105> 2013-08-15 20:50:25.822213 7fa8a4fc0700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3ac80 sd=30 :6813 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:39867/0
|
|
-104> 2013-08-15 20:50:25.822210 7fa8a50c1700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3a280 sd=29 :6813 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.3:43982/0
|
|
-103> 2013-08-15 20:50:25.822210 7fa8b5f95700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3a000 sd=27 :6813 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.3:43980/0
|
|
-102> 2013-08-15 20:50:25.822254 7fa8a51c2700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3a500 sd=28 :6813 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.3:43981/0
|
|
-101> 2013-08-15 20:50:25.822263 7fa8a4ebf700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3aa00 sd=31 :6813 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:39868/0
|
|
-100> 2013-08-15 20:50:25.822392 7fa8b60017c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring
|
|
-99> 2013-08-15 20:50:25.822345 7fa8a4dbe700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3a780 sd=32 :6813 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:39869/0
|
|
-98> 2013-08-15 20:50:25.822381 7fa8a4cbd700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3b680 sd=33 :6813 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:39870/0
|
|
-97> 2013-08-15 20:50:25.822482 7fa8a4bbc700 1 -- 0.0.0.0:6813/8361 >> :/0 pipe(0x2c3b400 sd=34 :6813 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:39872/0
|
|
-96> 2013-08-15 20:50:25.822692 7fa8b60017c0 5 asok(0x29f40e0) register_command dump_ops_in_flight hook 0x29ee770
|
|
-95> 2013-08-15 20:50:25.822704 7fa8b60017c0 5 asok(0x29f40e0) register_command dump_historic_ops hook 0x29ee770
|
|
-94> 2013-08-15 20:50:25.822709 7fa8b60017c0 5 asok(0x29f40e0) register_command dump_op_pq_state hook 0x29ee770
|
|
-93> 2013-08-15 20:50:25.822715 7fa8b60017c0 5 asok(0x29f40e0) register_command setomapval hook 0x2beaae0
|
|
-92> 2013-08-15 20:50:25.822720 7fa8b60017c0 5 asok(0x29f40e0) register_command rmomapkey hook 0x2beaae0
|
|
-91> 2013-08-15 20:50:25.822725 7fa8b60017c0 5 asok(0x29f40e0) register_command setomapheader hook 0x2beaae0
|
|
-90> 2013-08-15 20:50:25.822729 7fa8b60017c0 5 asok(0x29f40e0) register_command getomap hook 0x2beaae0
|
|
-89> 2013-08-15 20:50:25.822734 7fa8b60017c0 5 asok(0x29f40e0) register_command truncobj hook 0x2beaae0
|
|
-88> 2013-08-15 20:50:25.822738 7fa8b60017c0 5 asok(0x29f40e0) register_command injectdataerr hook 0x2beaae0
|
|
-87> 2013-08-15 20:50:25.822742 7fa8b60017c0 5 asok(0x29f40e0) register_command injectmdataerr hook 0x2beaae0
|
|
-86> 2013-08-15 20:50:25.822736 7fa8a12b4700 5 osd.7 63 heartbeat: osd_stat(734 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-85> 2013-08-15 20:50:25.822793 7fa8b60017c0 10 monclient(hunting): _reopen_session
|
|
-84> 2013-08-15 20:50:25.822815 7fa8b60017c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x2c3e420 addr 9.4.196.1:6789/0
|
|
-83> 2013-08-15 20:50:25.822828 7fa8b60017c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-82> 2013-08-15 20:50:25.822835 7fa8b60017c0 1 -- 0.0.0.0:6813/8361 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x2a00b40 con 0x2c3e420
|
|
-81> 2013-08-15 20:50:25.822849 7fa8b60017c0 10 monclient(hunting): renew_subs
|
|
-80> 2013-08-15 20:50:25.823202 7fa8a4bbc700 1 -- 9.4.196.3:6813/8361 learned my addr 9.4.196.3:6813/8361
|
|
-79> 2013-08-15 20:50:25.823414 7fa8a89c9700 10 monclient(hunting): renew_subs
|
|
-78> 2013-08-15 20:50:25.823770 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x2a261e0 con 0x2c3e420
|
|
-77> 2013-08-15 20:50:25.823797 7fa8a89c9700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-76> 2013-08-15 20:50:25.823813 7fa8a89c9700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-75> 2013-08-15 20:50:25.823818 7fa8a89c9700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-74> 2013-08-15 20:50:25.823837 7fa8a89c9700 1 monclient(hunting): found mon.zc2store
|
|
-73> 2013-08-15 20:50:25.823846 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2998019622 0 0) 0x2c4a200 con 0x2c3e420
|
|
-72> 2013-08-15 20:50:25.823869 7fa8a89c9700 10 monclient: my global_id is 4368
|
|
-71> 2013-08-15 20:50:25.824014 7fa8a89c9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-70> 2013-08-15 20:50:25.824023 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2a01d40 con 0x2c3e420
|
|
-69> 2013-08-15 20:50:25.824552 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (692884377 0 0) 0x2c4a000 con 0x2c3e420
|
|
-68> 2013-08-15 20:50:25.824633 7fa8a89c9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-67> 2013-08-15 20:50:25.824640 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2c4e240 con 0x2c3e420
|
|
-66> 2013-08-15 20:50:25.825365 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (130348059 0 0) 0x2c4a400 con 0x2c3e420
|
|
-65> 2013-08-15 20:50:25.825445 7fa8a89c9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-64> 2013-08-15 20:50:25.825452 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x2a24000 con 0x2c3e420
|
|
-63> 2013-08-15 20:50:25.825470 7fa8a89c9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-62> 2013-08-15 20:50:25.825486 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x2a24380 con 0x2c3e420
|
|
-61> 2013-08-15 20:50:25.825499 7fa8a89c9700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:55.825498)
|
|
-60> 2013-08-15 20:50:25.825509 7fa8a89c9700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-59> 2013-08-15 20:50:25.825514 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2c4e000 con 0x2c3e420
|
|
-58> 2013-08-15 20:50:25.825542 7fa8b60017c0 5 monclient: authenticate success, global_id 4368
|
|
-57> 2013-08-15 20:50:25.825554 7fa8b60017c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:55.825553)
|
|
-56> 2013-08-15 20:50:25.825867 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x2a265a0 con 0x2c3e420
|
|
-55> 2013-08-15 20:50:25.825887 7fa8a89c9700 10 monclient: handle_monmap mon_map v1
|
|
-54> 2013-08-15 20:50:25.825899 7fa8a89c9700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-53> 2013-08-15 20:50:25.825903 7fa8a89c9700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-52> 2013-08-15 20:50:25.825921 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x2a24000 con 0x2c3e420
|
|
-51> 2013-08-15 20:50:25.825930 7fa8a89c9700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:25.822852 renew after 2013-08-15 20:52:55.822852
|
|
-50> 2013-08-15 20:50:25.825943 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x2a263c0 con 0x2c3e420
|
|
-49> 2013-08-15 20:50:25.825951 7fa8a89c9700 10 monclient: handle_monmap mon_map v1
|
|
-48> 2013-08-15 20:50:25.825959 7fa8a89c9700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-47> 2013-08-15 20:50:25.825963 7fa8a89c9700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-46> 2013-08-15 20:50:25.825976 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x2a24540 con 0x2c3e420
|
|
-45> 2013-08-15 20:50:25.825984 7fa8a89c9700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-44> 2013-08-15 20:50:25.825992 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (523159842 0 0) 0x2c4ac00 con 0x2c3e420
|
|
-43> 2013-08-15 20:50:25.826038 7fa8a89c9700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:55.826038)
|
|
-42> 2013-08-15 20:50:25.826052 7fa8b60017c0 10 monclient: wait_auth_rotating done
|
|
-41> 2013-08-15 20:50:25.826492 7fa8b60017c0 10 monclient: get_version osdmap
|
|
-40> 2013-08-15 20:50:25.826498 7fa8b60017c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-39> 2013-08-15 20:50:25.826509 7fa8b60017c0 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x2a5c9c0 con 0x2c3e420
|
|
-38> 2013-08-15 20:50:25.826757 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=84) v2 ==== 24+0+0 (2076809473 0 0) 0x2a24380 con 0x2c3e420
|
|
-37> 2013-08-15 20:50:25.826804 7fa8a42ba700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-36> 2013-08-15 20:50:25.826817 7fa8a42ba700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- osd_boot(osd.7 booted 0 v63) v3 -- ?+0 0x2c70000 con 0x2c3e420
|
|
-35> 2013-08-15 20:50:26.222669 7fa89f6ad700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c3b180 sd=33 :6813 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:43998/0
|
|
-34> 2013-08-15 20:50:26.222683 7fa89f4ab700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c3bb80 sd=35 :6813 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:39899/0
|
|
-33> 2013-08-15 20:50:26.222683 7fa89f5ac700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c3af00 sd=34 :6813 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:39898/0
|
|
-32> 2013-08-15 20:50:26.222736 7fa89f3aa700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c3b900 sd=36 :6813 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.3:43999/0
|
|
-31> 2013-08-15 20:50:26.222862 7fa89f1a8700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c62a00 sd=38 :6813 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:39901/0
|
|
-30> 2013-08-15 20:50:26.222815 7fa89f2a9700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c62c80 sd=37 :6813 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.3:44000/0
|
|
-29> 2013-08-15 20:50:26.222912 7fa89f0a7700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c62780 sd=39 :6813 s=0 pgs=0 cs=0 l=0).accept sd=39 9.4.196.1:39900/0
|
|
-28> 2013-08-15 20:50:26.222986 7fa89f5ac700 1 -- 9.4.196.3:6813/8361 >> :/0 pipe(0x2c62500 sd=40 :6813 s=0 pgs=0 cs=0 l=0).accept sd=40 9.4.196.1:39902/0
|
|
-27> 2013-08-15 20:50:27.270330 7fa89f5ac700 1 -- 9.4.196.3:6822/8361 >> :/0 pipe(0x2c3b900 sd=33 :6822 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:53021/0
|
|
-26> 2013-08-15 20:50:27.270941 7fa8a79c7700 1 -- 9.4.196.3:6822/8361 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.175(5) epoch 86) v4 ==== 791+0+0 (413623918 0 0) 0x2a24700 con 0x2c3ec60
|
|
-25> 2013-08-15 20:50:27.270985 7fa8a79c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.270860, event: header_read, request: pg_notify(3.175(5) epoch 86) v4
|
|
-24> 2013-08-15 20:50:27.271002 7fa8a79c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.270862, event: throttled, request: pg_notify(3.175(5) epoch 86) v4
|
|
-23> 2013-08-15 20:50:27.271012 7fa8a79c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.270923, event: all_read, request: pg_notify(3.175(5) epoch 86) v4
|
|
-22> 2013-08-15 20:50:27.271022 7fa8a79c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.270973, event: dispatched, request: pg_notify(3.175(5) epoch 86) v4
|
|
-21> 2013-08-15 20:50:27.271032 7fa8a79c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.271032, event: waiting_for_osdmap, request: pg_notify(3.175(5) epoch 86) v4
|
|
-20> 2013-08-15 20:50:27.271051 7fa8a79c7700 10 monclient: renew_subs
|
|
-19> 2013-08-15 20:50:27.271059 7fa8a79c7700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-18> 2013-08-15 20:50:27.271072 7fa8a79c7700 1 -- 9.4.196.3:6813/8361 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x2a248c0 con 0x2c3e420
|
|
-17> 2013-08-15 20:50:27.271093 7fa8a79c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.271093, event: wait for new map, request: pg_notify(3.175(5) epoch 86) v4
|
|
-16> 2013-08-15 20:50:27.271658 7fa89f0a7700 1 -- 9.4.196.3:6823/8361 >> :/0 pipe(0x2c3af00 sd=34 :6823 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:44807/0
|
|
-15> 2013-08-15 20:50:27.272731 7fa8a89c9700 1 -- 9.4.196.3:6813/8361 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..85 src has 1..85) v3 ==== 18379+0+0 (600021662 0 0) 0x2c4f440 con 0x2c3e420
|
|
-14> 2013-08-15 20:50:27.272756 7fa8a89c9700 3 osd.7 63 handle_osd_map epochs [64,85], i have 63, src has [1,85]
|
|
-13> 2013-08-15 20:50:27.276571 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-12> 2013-08-15 20:50:27.276586 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-11> 2013-08-15 20:50:27.276590 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-10> 2013-08-15 20:50:27.276626 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-9> 2013-08-15 20:50:27.276650 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-8> 2013-08-15 20:50:27.276653 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-7> 2013-08-15 20:50:27.276674 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-6> 2013-08-15 20:50:27.276687 7fa8a89c9700 1 -- 9.4.196.3:6822/8361 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-5> 2013-08-15 20:50:27.277240 7fa8ae9d5700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-4> 2013-08-15 20:50:27.277258 7fa8ae9d5700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
-3> 2013-08-15 20:50:27.277303 7fa8a79c7700 1 -- 9.4.196.3:6822/8361 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(85..86 src has 1..86) v3 ==== 6767+0+0 (1565733723 0 0) 0x2d36000 con 0x2c52000
|
|
-2> 2013-08-15 20:50:27.277519 7fa8a32b8700 1 osd.7 pg_epoch: 85 pg[3.1ee( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 85/85/56) [2] r=-1 lpr=85 pi=56-84/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:27.278190 7fa8a32b8700 1 osd.7 pg_epoch: 85 pg[3.1e8( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 85/85/56) [1] r=-1 lpr=85 pi=56-84/8 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:27.278258 7fa8ae9d5700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fa8ae9d5700 time 2013-08-15 20:50:27.277266
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fa8b59ddf8e]
|
|
5: (clone()+0x6d) [0x7fa8b3c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:27.281238 7fa8ae9d5700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fa8ae9d5700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fa8b59e5bd0]
|
|
3: (gsignal()+0x37) [0x7fa8b3bcd037]
|
|
4: (abort()+0x148) [0x7fa8b3bd0698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa8b44d9e8d]
|
|
6: (()+0x5ef76) [0x7fa8b44d7f76]
|
|
7: (()+0x5efa3) [0x7fa8b44d7fa3]
|
|
8: (()+0x5f1de) [0x7fa8b44d81de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fa8b59ddf8e]
|
|
14: (clone()+0x6d) [0x7fa8b3c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-5> 2013-08-15 20:50:27.278971 7fa8a32b8700 1 osd.7 pg_epoch: 85 pg[3.5e( v 63'10 (0'0,63'10] local-les=62 n=0 ec=56 les/c 62/62 85/85/56) [4] r=-1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:27.279463 7fa89f3aa700 1 -- 9.4.196.3:6822/8361 >> :/0 pipe(0x2c62780 sd=35 :6822 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:53023/0
|
|
-3> 2013-08-15 20:50:27.279624 7fa8a32b8700 1 osd.7 pg_epoch: 85 pg[3.11f( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 85/85/56) [1] r=-1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:27.280339 7fa8a32b8700 1 osd.7 pg_epoch: 85 pg[3.11a( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 85/85/56) [2] r=-1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:27.281009 7fa8a32b8700 1 osd.7 pg_epoch: 85 pg[3.5b( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 85/85/56) [0] r=-1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:27.281238 7fa8ae9d5700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fa8ae9d5700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fa8b59e5bd0]
|
|
3: (gsignal()+0x37) [0x7fa8b3bcd037]
|
|
4: (abort()+0x148) [0x7fa8b3bd0698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fa8b44d9e8d]
|
|
6: (()+0x5ef76) [0x7fa8b44d7f76]
|
|
7: (()+0x5efa3) [0x7fa8b44d7fa3]
|
|
8: (()+0x5f1de) [0x7fa8b44d81de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fa8b59ddf8e]
|
|
14: (clone()+0x6d) [0x7fa8b3c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:27.432359 7ff6f91c87c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8751
|
|
2013-08-15 20:50:27.432881 7ff6f91c87c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:27.461139 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:27.461152 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:27.461425 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:27.479033 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:27.479119 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:27.503383 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:27.504136 7ff6f91c87c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:27.504153 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.580693 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.581459 7ff6f91c87c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
2013-08-15 20:50:27.611144 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:27.611162 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:27.611503 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:27.623094 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:27.623147 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:27.653418 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:27.653874 7ff6f91c87c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:27.653881 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.728113 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:29.847437 7ff6f1b9c700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:29.847459 7ff6f1b9c700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
2013-08-15 20:50:29.848757 7ff6f1b9c700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7ff6f1b9c700 time 2013-08-15 20:50:29.847467
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7ff6f8ba4f8e]
|
|
5: (clone()+0x6d) [0x7ff6f6e56e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-392> 2013-08-15 20:50:27.431085 7ff6f91c87c0 5 asok(0x23980e0) register_command perfcounters_dump hook 0x2392010
|
|
-391> 2013-08-15 20:50:27.431112 7ff6f91c87c0 5 asok(0x23980e0) register_command 1 hook 0x2392010
|
|
-390> 2013-08-15 20:50:27.431115 7ff6f91c87c0 5 asok(0x23980e0) register_command perf dump hook 0x2392010
|
|
-389> 2013-08-15 20:50:27.431122 7ff6f91c87c0 5 asok(0x23980e0) register_command perfcounters_schema hook 0x2392010
|
|
-388> 2013-08-15 20:50:27.431125 7ff6f91c87c0 5 asok(0x23980e0) register_command 2 hook 0x2392010
|
|
-387> 2013-08-15 20:50:27.431128 7ff6f91c87c0 5 asok(0x23980e0) register_command perf schema hook 0x2392010
|
|
-386> 2013-08-15 20:50:27.431134 7ff6f91c87c0 5 asok(0x23980e0) register_command config show hook 0x2392010
|
|
-385> 2013-08-15 20:50:27.431139 7ff6f91c87c0 5 asok(0x23980e0) register_command config set hook 0x2392010
|
|
-384> 2013-08-15 20:50:27.431143 7ff6f91c87c0 5 asok(0x23980e0) register_command log flush hook 0x2392010
|
|
-383> 2013-08-15 20:50:27.431147 7ff6f91c87c0 5 asok(0x23980e0) register_command log dump hook 0x2392010
|
|
-382> 2013-08-15 20:50:27.431151 7ff6f91c87c0 5 asok(0x23980e0) register_command log reopen hook 0x2392010
|
|
-381> 2013-08-15 20:50:27.432359 7ff6f91c87c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8751
|
|
-380> 2013-08-15 20:50:27.432881 7ff6f91c87c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-379> 2013-08-15 20:50:27.433011 7ff6f91c87c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6825/8751 need_addr=1
|
|
-378> 2013-08-15 20:50:27.433039 7ff6f91c87c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-377> 2013-08-15 20:50:27.433047 7ff6f91c87c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6826/8751 need_addr=0
|
|
-376> 2013-08-15 20:50:27.433065 7ff6f91c87c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-375> 2013-08-15 20:50:27.433071 7ff6f91c87c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6827/8751 need_addr=0
|
|
-374> 2013-08-15 20:50:27.434813 7ff6f91c87c0 5 asok(0x23980e0) init /var/run/ceph/ceph-osd.7.asok
|
|
-373> 2013-08-15 20:50:27.434828 7ff6f91c87c0 5 asok(0x23980e0) bind_and_listen /var/run/ceph/ceph-osd.7.asok
|
|
-372> 2013-08-15 20:50:27.434854 7ff6f91c87c0 5 asok(0x23980e0) register_command 0 hook 0x23900b8
|
|
-371> 2013-08-15 20:50:27.434860 7ff6f91c87c0 5 asok(0x23980e0) register_command version hook 0x23900b8
|
|
-370> 2013-08-15 20:50:27.434865 7ff6f91c87c0 5 asok(0x23980e0) register_command git_version hook 0x23900b8
|
|
-369> 2013-08-15 20:50:27.434869 7ff6f91c87c0 5 asok(0x23980e0) register_command help hook 0x23920d0
|
|
-368> 2013-08-15 20:50:27.434897 7ff6f4ba2700 5 asok(0x23980e0) entry start
|
|
-367> 2013-08-15 20:50:27.461139 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-366> 2013-08-15 20:50:27.461152 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-365> 2013-08-15 20:50:27.461425 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-364> 2013-08-15 20:50:27.479033 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-363> 2013-08-15 20:50:27.479119 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-362> 2013-08-15 20:50:27.503383 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-361> 2013-08-15 20:50:27.504115 7ff6f91c87c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-360> 2013-08-15 20:50:27.504136 7ff6f91c87c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-359> 2013-08-15 20:50:27.504153 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-358> 2013-08-15 20:50:27.509184 7ff6f91c87c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-357> 2013-08-15 20:50:27.513081 7ff6f91c87c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-356> 2013-08-15 20:50:27.516988 7ff6f91c87c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-355> 2013-08-15 20:50:27.520989 7ff6f91c87c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-354> 2013-08-15 20:50:27.525112 7ff6f91c87c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-353> 2013-08-15 20:50:27.529167 7ff6f91c87c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-352> 2013-08-15 20:50:27.533152 7ff6f91c87c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-351> 2013-08-15 20:50:27.537127 7ff6f91c87c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-350> 2013-08-15 20:50:27.541079 7ff6f91c87c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-349> 2013-08-15 20:50:27.545024 7ff6f91c87c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-348> 2013-08-15 20:50:27.548961 7ff6f91c87c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-347> 2013-08-15 20:50:27.552953 7ff6f91c87c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-346> 2013-08-15 20:50:27.556909 7ff6f91c87c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-345> 2013-08-15 20:50:27.560858 7ff6f91c87c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-344> 2013-08-15 20:50:27.564814 7ff6f91c87c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-343> 2013-08-15 20:50:27.568738 7ff6f91c87c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-342> 2013-08-15 20:50:27.572681 7ff6f91c87c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-341> 2013-08-15 20:50:27.576619 7ff6f91c87c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-340> 2013-08-15 20:50:27.580567 7ff6f91c87c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-339> 2013-08-15 20:50:27.580621 7ff6f91c87c0 2 journal No further valid entries found, journal is most likely valid
|
|
-338> 2013-08-15 20:50:27.580631 7ff6f91c87c0 2 journal No further valid entries found, journal is most likely valid
|
|
-337> 2013-08-15 20:50:27.580640 7ff6f91c87c0 3 journal journal_replay: end of journal, done.
|
|
-336> 2013-08-15 20:50:27.580693 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-335> 2013-08-15 20:50:27.581374 7ff6f239d700 1 FileStore::op_tp worker finish
|
|
-334> 2013-08-15 20:50:27.581397 7ff6f1b9c700 1 FileStore::op_tp worker finish
|
|
-333> 2013-08-15 20:50:27.581459 7ff6f91c87c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
-332> 2013-08-15 20:50:27.581755 7ff6f91c87c0 10 monclient(hunting): build_initial_monmap
|
|
-331> 2013-08-15 20:50:27.581807 7ff6f91c87c0 5 adding auth protocol: cephx
|
|
-330> 2013-08-15 20:50:27.581814 7ff6f91c87c0 5 adding auth protocol: cephx
|
|
-329> 2013-08-15 20:50:27.581901 7ff6f91c87c0 1 -- 0.0.0.0:6825/8751 messenger.start
|
|
-328> 2013-08-15 20:50:27.581929 7ff6f91c87c0 1 -- :/0 messenger.start
|
|
-327> 2013-08-15 20:50:27.581940 7ff6f91c87c0 1 -- 9.4.196.3:6827/8751 messenger.start
|
|
-326> 2013-08-15 20:50:27.581951 7ff6f91c87c0 1 -- 9.4.196.3:6826/8751 messenger.start
|
|
-325> 2013-08-15 20:50:27.582037 7ff6f91c87c0 2 osd.7 0 mounting /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
|
|
-324> 2013-08-15 20:50:27.611144 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-323> 2013-08-15 20:50:27.611162 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-322> 2013-08-15 20:50:27.611503 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-321> 2013-08-15 20:50:27.623094 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-320> 2013-08-15 20:50:27.623147 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-319> 2013-08-15 20:50:27.653418 7ff6f91c87c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-318> 2013-08-15 20:50:27.653857 7ff6f91c87c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-317> 2013-08-15 20:50:27.653874 7ff6f91c87c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-316> 2013-08-15 20:50:27.653881 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-315> 2013-08-15 20:50:27.657896 7ff6f91c87c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-314> 2013-08-15 20:50:27.661757 7ff6f91c87c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-313> 2013-08-15 20:50:27.665614 7ff6f91c87c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-312> 2013-08-15 20:50:27.669550 7ff6f91c87c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-311> 2013-08-15 20:50:27.673377 7ff6f91c87c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-310> 2013-08-15 20:50:27.677208 7ff6f91c87c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-309> 2013-08-15 20:50:27.681022 7ff6f91c87c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-308> 2013-08-15 20:50:27.684814 7ff6f91c87c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-307> 2013-08-15 20:50:27.688589 7ff6f91c87c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-306> 2013-08-15 20:50:27.692363 7ff6f91c87c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-305> 2013-08-15 20:50:27.696136 7ff6f91c87c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-304> 2013-08-15 20:50:27.700065 7ff6f91c87c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-303> 2013-08-15 20:50:27.704077 7ff6f91c87c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-302> 2013-08-15 20:50:27.708064 7ff6f91c87c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-301> 2013-08-15 20:50:27.712045 7ff6f91c87c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-300> 2013-08-15 20:50:27.716051 7ff6f91c87c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-299> 2013-08-15 20:50:27.720086 7ff6f91c87c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-298> 2013-08-15 20:50:27.724035 7ff6f91c87c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-297> 2013-08-15 20:50:27.728015 7ff6f91c87c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-296> 2013-08-15 20:50:27.728051 7ff6f91c87c0 2 journal No further valid entries found, journal is most likely valid
|
|
-295> 2013-08-15 20:50:27.728056 7ff6f91c87c0 2 journal No further valid entries found, journal is most likely valid
|
|
-294> 2013-08-15 20:50:27.728057 7ff6f91c87c0 3 journal journal_replay: end of journal, done.
|
|
-293> 2013-08-15 20:50:27.728113 7ff6f91c87c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-292> 2013-08-15 20:50:27.728312 7ff6f91c87c0 2 osd.7 0 boot
|
|
-291> 2013-08-15 20:50:27.752593 7ff6f91c87c0 2 osd.7 63 superblock: i am osd.7
|
|
-290> 2013-08-15 20:50:27.752653 7ff6f91c87c0 1 accepter.accepter.start
|
|
-289> 2013-08-15 20:50:27.752698 7ff6f91c87c0 1 accepter.accepter.start
|
|
-288> 2013-08-15 20:50:27.752789 7ff6f91c87c0 1 accepter.accepter.start
|
|
-287> 2013-08-15 20:50:27.752812 7ff6f91c87c0 10 monclient(hunting): init
|
|
-286> 2013-08-15 20:50:27.752819 7ff6f91c87c0 5 adding auth protocol: cephx
|
|
-285> 2013-08-15 20:50:27.752821 7ff6f91c87c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-284> 2013-08-15 20:50:27.752966 7ff6f91c87c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring
|
|
-283> 2013-08-15 20:50:27.753147 7ff6f91c87c0 5 asok(0x23980e0) register_command dump_ops_in_flight hook 0x2392770
|
|
-282> 2013-08-15 20:50:27.753157 7ff6f91c87c0 5 asok(0x23980e0) register_command dump_historic_ops hook 0x2392770
|
|
-281> 2013-08-15 20:50:27.753163 7ff6f91c87c0 5 asok(0x23980e0) register_command dump_op_pq_state hook 0x2392770
|
|
-280> 2013-08-15 20:50:27.753168 7ff6f91c87c0 5 asok(0x23980e0) register_command setomapval hook 0x258eae0
|
|
-279> 2013-08-15 20:50:27.753171 7ff6f91c87c0 5 asok(0x23980e0) register_command rmomapkey hook 0x258eae0
|
|
-278> 2013-08-15 20:50:27.753175 7ff6f91c87c0 5 asok(0x23980e0) register_command setomapheader hook 0x258eae0
|
|
-277> 2013-08-15 20:50:27.753180 7ff6f91c87c0 5 asok(0x23980e0) register_command getomap hook 0x258eae0
|
|
-276> 2013-08-15 20:50:27.753184 7ff6f91c87c0 5 asok(0x23980e0) register_command truncobj hook 0x258eae0
|
|
-275> 2013-08-15 20:50:27.753186 7ff6f91c87c0 5 asok(0x23980e0) register_command injectdataerr hook 0x258eae0
|
|
-274> 2013-08-15 20:50:27.753190 7ff6f91c87c0 5 asok(0x23980e0) register_command injectmdataerr hook 0x258eae0
|
|
-273> 2013-08-15 20:50:27.753203 7ff6e4b82700 5 osd.7 63 heartbeat: osd_stat(734 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-272> 2013-08-15 20:50:27.753244 7ff6f91c87c0 10 monclient(hunting): _reopen_session
|
|
-271> 2013-08-15 20:50:27.753293 7ff6f91c87c0 10 monclient(hunting): _pick_new_mon picked mon.noname-b con 0x25e4840 addr 9.4.196.5:6789/0
|
|
-270> 2013-08-15 20:50:27.753320 7ff6f91c87c0 10 monclient(hunting): _send_mon_message to mon.noname-b at 9.4.196.5:6789/0
|
|
-269> 2013-08-15 20:50:27.753325 7ff6f91c87c0 1 -- 0.0.0.0:6825/8751 --> 9.4.196.5:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x23a4b40 con 0x25e4840
|
|
-268> 2013-08-15 20:50:27.753337 7ff6f91c87c0 10 monclient(hunting): renew_subs
|
|
-267> 2013-08-15 20:50:27.753816 7ff6f915c700 1 -- 9.4.196.3:6825/8751 learned my addr 9.4.196.3:6825/8751
|
|
-266> 2013-08-15 20:50:27.754085 7ff6ebb90700 10 monclient(hunting): renew_subs
|
|
-265> 2013-08-15 20:50:27.904555 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x23ca1e0 con 0x25e4840
|
|
-264> 2013-08-15 20:50:27.904604 7ff6ebb90700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-263> 2013-08-15 20:50:27.904620 7ff6ebb90700 10 monclient(hunting): got monmap 1, mon.noname-b is now rank -1
|
|
-262> 2013-08-15 20:50:27.904624 7ff6ebb90700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-261> 2013-08-15 20:50:27.904660 7ff6ebb90700 1 monclient(hunting): found mon.ceph-dmon2
|
|
-260> 2013-08-15 20:50:27.904670 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3692731459 0 0) 0x25ec200 con 0x25e4840
|
|
-259> 2013-08-15 20:50:27.904691 7ff6ebb90700 10 monclient: my global_id is 4328
|
|
-258> 2013-08-15 20:50:27.904816 7ff6ebb90700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-257> 2013-08-15 20:50:27.904840 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x23a5d40 con 0x25e4840
|
|
-256> 2013-08-15 20:50:27.908119 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (441092808 0 0) 0x25ec000 con 0x25e4840
|
|
-255> 2013-08-15 20:50:27.908205 7ff6ebb90700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-254> 2013-08-15 20:50:27.908214 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x25f2240 con 0x25e4840
|
|
-253> 2013-08-15 20:50:27.912079 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (2683247234 0 0) 0x25ec400 con 0x25e4840
|
|
-252> 2013-08-15 20:50:27.912132 7ff6ebb90700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-251> 2013-08-15 20:50:27.912139 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x23c8000 con 0x25e4840
|
|
-250> 2013-08-15 20:50:27.912157 7ff6ebb90700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-249> 2013-08-15 20:50:27.912162 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x23c8380 con 0x25e4840
|
|
-248> 2013-08-15 20:50:27.912174 7ff6ebb90700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:57.912174)
|
|
-247> 2013-08-15 20:50:27.912183 7ff6ebb90700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-246> 2013-08-15 20:50:27.912189 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x25f2000 con 0x25e4840
|
|
-245> 2013-08-15 20:50:27.912206 7ff6f91c87c0 5 monclient: authenticate success, global_id 4328
|
|
-244> 2013-08-15 20:50:27.912218 7ff6f91c87c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:57.912217)
|
|
-243> 2013-08-15 20:50:27.914981 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x23ca5a0 con 0x25e4840
|
|
-242> 2013-08-15 20:50:27.914992 7ff6ebb90700 10 monclient: handle_monmap mon_map v1
|
|
-241> 2013-08-15 20:50:27.915000 7ff6ebb90700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-240> 2013-08-15 20:50:27.915002 7ff6ebb90700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-239> 2013-08-15 20:50:27.915021 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x23c8000 con 0x25e4840
|
|
-238> 2013-08-15 20:50:27.915030 7ff6ebb90700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:27.753341 renew after 2013-08-15 20:52:57.753341
|
|
-237> 2013-08-15 20:50:27.915085 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x23ca3c0 con 0x25e4840
|
|
-236> 2013-08-15 20:50:27.915095 7ff6ebb90700 10 monclient: handle_monmap mon_map v1
|
|
-235> 2013-08-15 20:50:27.915100 7ff6ebb90700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-234> 2013-08-15 20:50:27.915104 7ff6ebb90700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-233> 2013-08-15 20:50:27.915187 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x23c8540 con 0x25e4840
|
|
-232> 2013-08-15 20:50:27.915196 7ff6ebb90700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-231> 2013-08-15 20:50:27.915510 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1418663573 0 0) 0x25f4600 con 0x25e4840
|
|
-230> 2013-08-15 20:50:27.915553 7ff6ebb90700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:57.915553)
|
|
-229> 2013-08-15 20:50:27.915567 7ff6f91c87c0 10 monclient: wait_auth_rotating done
|
|
-228> 2013-08-15 20:50:27.916134 7ff6f91c87c0 10 monclient: get_version osdmap
|
|
-227> 2013-08-15 20:50:27.916139 7ff6f91c87c0 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-226> 2013-08-15 20:50:27.916145 7ff6f91c87c0 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x24009c0 con 0x25e4840
|
|
-225> 2013-08-15 20:50:27.916541 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 10 ==== mon_check_map_ack(handle=1 version=87) v2 ==== 24+0+0 (1748105714 0 0) 0x23c8380 con 0x25e4840
|
|
-224> 2013-08-15 20:50:27.916582 7ff6e7b88700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-223> 2013-08-15 20:50:27.916601 7ff6e7b88700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- osd_boot(osd.7 booted 0 v63) v3 -- ?+0 0x2614000 con 0x25e4840
|
|
-222> 2013-08-15 20:50:29.758407 7ff6e2874700 1 -- 9.4.196.3:6826/8751 >> :/0 pipe(0x25e7180 sd=33 :6826 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:54031/0
|
|
-221> 2013-08-15 20:50:29.758948 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.6 9.4.196.3:6801/4121 1 ==== pg_notify(3.175(8) epoch 89) v4 ==== 912+0+0 (3258197322 0 0) 0x23c8700 con 0x25fab00
|
|
-220> 2013-08-15 20:50:29.758987 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.758865, event: header_read, request: pg_notify(3.175(8) epoch 89) v4
|
|
-219> 2013-08-15 20:50:29.758999 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.758867, event: throttled, request: pg_notify(3.175(8) epoch 89) v4
|
|
-218> 2013-08-15 20:50:29.759006 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.758934, event: all_read, request: pg_notify(3.175(8) epoch 89) v4
|
|
-217> 2013-08-15 20:50:29.759012 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.758977, event: dispatched, request: pg_notify(3.175(8) epoch 89) v4
|
|
-216> 2013-08-15 20:50:29.759018 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.759018, event: waiting_for_osdmap, request: pg_notify(3.175(8) epoch 89) v4
|
|
-215> 2013-08-15 20:50:29.759034 7ff6eab8e700 10 monclient: renew_subs
|
|
-214> 2013-08-15 20:50:29.759039 7ff6eab8e700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-213> 2013-08-15 20:50:29.759047 7ff6eab8e700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x23c88c0 con 0x25e4840
|
|
-212> 2013-08-15 20:50:29.759061 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.759061, event: wait for new map, request: pg_notify(3.175(8) epoch 89) v4
|
|
-211> 2013-08-15 20:50:29.759839 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(88..89 src has 1..89) v3 ==== 1979+0+0 (1011610323 0 0) 0x25f3440 con 0x25e54a0
|
|
-210> 2013-08-15 20:50:29.759868 7ff6eab8e700 3 osd.7 63 handle_osd_map epochs [88,89], i have 63, src has [1,89]
|
|
-209> 2013-08-15 20:50:29.759875 7ff6eab8e700 10 monclient: renew_subs
|
|
-208> 2013-08-15 20:50:29.759878 7ff6eab8e700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-207> 2013-08-15 20:50:29.759881 7ff6eab8e700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x23c8e00 con 0x25e4840
|
|
-206> 2013-08-15 20:50:29.759908 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_notify(3.175(7) epoch 89) v4 ==== 869+0+0 (1512933645 0 0) 0x23c8a80 con 0x25e54a0
|
|
-205> 2013-08-15 20:50:29.759925 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.759826, event: header_read, request: pg_notify(3.175(7) epoch 89) v4
|
|
-204> 2013-08-15 20:50:29.759939 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.759827, event: throttled, request: pg_notify(3.175(7) epoch 89) v4
|
|
-203> 2013-08-15 20:50:29.759945 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.759894, event: all_read, request: pg_notify(3.175(7) epoch 89) v4
|
|
-202> 2013-08-15 20:50:29.759951 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.759922, event: dispatched, request: pg_notify(3.175(7) epoch 89) v4
|
|
-201> 2013-08-15 20:50:29.759957 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.759957, event: waiting_for_osdmap, request: pg_notify(3.175(7) epoch 89) v4
|
|
-200> 2013-08-15 20:50:29.759964 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.759964, event: wait for new map, request: pg_notify(3.175(7) epoch 89) v4
|
|
-199> 2013-08-15 20:50:29.763338 7ff6e2672700 1 -- 9.4.196.3:6826/8751 >> :/0 pipe(0x25e7680 sd=34 :6826 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:54035/0
|
|
-198> 2013-08-15 20:50:29.763798 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.175(7) epoch 89) v4 ==== 869+0+0 (3510264094 0 0) 0x23c8fc0 con 0x25fac60
|
|
-197> 2013-08-15 20:50:29.763821 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.763715, event: header_read, request: pg_notify(3.175(7) epoch 89) v4
|
|
-196> 2013-08-15 20:50:29.763830 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.763725, event: throttled, request: pg_notify(3.175(7) epoch 89) v4
|
|
-195> 2013-08-15 20:50:29.763836 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.763782, event: all_read, request: pg_notify(3.175(7) epoch 89) v4
|
|
-194> 2013-08-15 20:50:29.763842 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.763818, event: dispatched, request: pg_notify(3.175(7) epoch 89) v4
|
|
-193> 2013-08-15 20:50:29.763848 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.763848, event: waiting_for_osdmap, request: pg_notify(3.175(7) epoch 89) v4
|
|
-192> 2013-08-15 20:50:29.763855 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.763855, event: wait for new map, request: pg_notify(3.175(7) epoch 89) v4
|
|
-191> 2013-08-15 20:50:29.765546 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(87..89 src has 1..89) v3 ==== 2626+0+0 (3924186462 0 0) 0x25f3d40 con 0x25e5340
|
|
-190> 2013-08-15 20:50:29.765568 7ff6eab8e700 3 osd.7 63 handle_osd_map epochs [87,89], i have 63, src has [1,89]
|
|
-189> 2013-08-15 20:50:29.765576 7ff6eab8e700 10 monclient: renew_subs
|
|
-188> 2013-08-15 20:50:29.765580 7ff6eab8e700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-187> 2013-08-15 20:50:29.765586 7ff6eab8e700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x23c8c40 con 0x25e4840
|
|
-186> 2013-08-15 20:50:29.765605 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.2 9.4.196.1:6807/6887 2 ==== pg_query(2.4 epoch 89) v2 ==== 137+0+0 (2755468864 0 0) 0x23ca1e0 con 0x25e5340
|
|
-185> 2013-08-15 20:50:29.765622 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.765518, event: header_read, request: pg_query(2.4 epoch 89) v2
|
|
-184> 2013-08-15 20:50:29.765635 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.765518, event: throttled, request: pg_query(2.4 epoch 89) v2
|
|
-183> 2013-08-15 20:50:29.765646 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.765564, event: all_read, request: pg_query(2.4 epoch 89) v2
|
|
-182> 2013-08-15 20:50:29.765654 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.765619, event: dispatched, request: pg_query(2.4 epoch 89) v2
|
|
-181> 2013-08-15 20:50:29.765659 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.765659, event: waiting_for_osdmap, request: pg_query(2.4 epoch 89) v2
|
|
-180> 2013-08-15 20:50:29.765669 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.765669, event: wait for new map, request: pg_query(2.4 epoch 89) v2
|
|
-179> 2013-08-15 20:50:29.766415 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.2 9.4.196.1:6807/6887 3 ==== pg_query(0.6 epoch 89) v2 ==== 137+0+0 (789891283 0 0) 0x23ca780 con 0x25e5340
|
|
-178> 2013-08-15 20:50:29.766438 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.766313, event: header_read, request: pg_query(0.6 epoch 89) v2
|
|
-177> 2013-08-15 20:50:29.766447 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.766315, event: throttled, request: pg_query(0.6 epoch 89) v2
|
|
-176> 2013-08-15 20:50:29.766453 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.766372, event: all_read, request: pg_query(0.6 epoch 89) v2
|
|
-175> 2013-08-15 20:50:29.766459 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.766435, event: dispatched, request: pg_query(0.6 epoch 89) v2
|
|
-174> 2013-08-15 20:50:29.766466 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.766466, event: waiting_for_osdmap, request: pg_query(0.6 epoch 89) v2
|
|
-173> 2013-08-15 20:50:29.766472 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.766471, event: wait for new map, request: pg_query(0.6 epoch 89) v2
|
|
-172> 2013-08-15 20:50:29.766873 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(87..89 src has 1..89) v3 ==== 2626+0+0 (3924186462 0 0) 0x25f3440 con 0x25e5b80
|
|
-171> 2013-08-15 20:50:29.766895 7ff6eab8e700 3 osd.7 63 handle_osd_map epochs [87,89], i have 63, src has [1,89]
|
|
-170> 2013-08-15 20:50:29.766902 7ff6eab8e700 10 monclient: renew_subs
|
|
-169> 2013-08-15 20:50:29.766906 7ff6eab8e700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-168> 2013-08-15 20:50:29.766911 7ff6eab8e700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x23c8c40 con 0x25e4840
|
|
-167> 2013-08-15 20:50:29.766928 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(3.16d epoch 89) v2 ==== 137+0+0 (2894385370 0 0) 0x23ca960 con 0x25e5b80
|
|
-166> 2013-08-15 20:50:29.766945 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.766859, event: header_read, request: pg_query(3.16d epoch 89) v2
|
|
-165> 2013-08-15 20:50:29.766957 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.766860, event: throttled, request: pg_query(3.16d epoch 89) v2
|
|
-164> 2013-08-15 20:50:29.766967 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.766889, event: all_read, request: pg_query(3.16d epoch 89) v2
|
|
-163> 2013-08-15 20:50:29.766976 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.766942, event: dispatched, request: pg_query(3.16d epoch 89) v2
|
|
-162> 2013-08-15 20:50:29.766983 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.766983, event: waiting_for_osdmap, request: pg_query(3.16d epoch 89) v2
|
|
-161> 2013-08-15 20:50:29.766990 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.766990, event: wait for new map, request: pg_query(3.16d epoch 89) v2
|
|
-160> 2013-08-15 20:50:29.767683 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 3 ==== pg_query(3.14e epoch 89) v2 ==== 137+0+0 (1872877779 0 0) 0x23cad20 con 0x25e5b80
|
|
-159> 2013-08-15 20:50:29.767707 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.767587, event: header_read, request: pg_query(3.14e epoch 89) v2
|
|
-158> 2013-08-15 20:50:29.767715 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.767589, event: throttled, request: pg_query(3.14e epoch 89) v2
|
|
-157> 2013-08-15 20:50:29.767719 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.767638, event: all_read, request: pg_query(3.14e epoch 89) v2
|
|
-156> 2013-08-15 20:50:29.767723 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.767704, event: dispatched, request: pg_query(3.14e epoch 89) v2
|
|
-155> 2013-08-15 20:50:29.767727 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.767727, event: waiting_for_osdmap, request: pg_query(3.14e epoch 89) v2
|
|
-154> 2013-08-15 20:50:29.767732 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.767732, event: wait for new map, request: pg_query(3.14e epoch 89) v2
|
|
-153> 2013-08-15 20:50:29.770672 7ff6e2470700 1 -- 9.4.196.3:6826/8751 >> :/0 pipe(0x25e7400 sd=35 :6826 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:54036/0
|
|
-152> 2013-08-15 20:50:29.771042 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(87..89 src has 1..89) v3 ==== 2626+0+0 (3924186462 0 0) 0x2636900 con 0x25e5ce0
|
|
-151> 2013-08-15 20:50:29.771068 7ff6eab8e700 3 osd.7 63 handle_osd_map epochs [87,89], i have 63, src has [1,89]
|
|
-150> 2013-08-15 20:50:29.771099 7ff6eab8e700 10 monclient: renew_subs
|
|
-149> 2013-08-15 20:50:29.771107 7ff6eab8e700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-148> 2013-08-15 20:50:29.771117 7ff6eab8e700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x23c8e00 con 0x25e4840
|
|
-147> 2013-08-15 20:50:29.771167 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2 ==== 512+0+0 (1853417528 0 0) 0x23caf00 con 0x25e5ce0
|
|
-146> 2013-08-15 20:50:29.771186 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.771100, event: header_read, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-145> 2013-08-15 20:50:29.771196 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.771102, event: throttled, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-144> 2013-08-15 20:50:29.771202 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.771155, event: all_read, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-143> 2013-08-15 20:50:29.771209 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.771184, event: dispatched, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-142> 2013-08-15 20:50:29.771216 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.771216, event: waiting_for_osdmap, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-141> 2013-08-15 20:50:29.771223 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.771223, event: wait for new map, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-140> 2013-08-15 20:50:29.771337 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.5 9.4.196.3:6819/4129 1 ==== pg_notify(3.24a(8) epoch 89) v4 ==== 912+0+0 (2274624305 0 0) 0x23c9180 con 0x25fb1e0
|
|
-139> 2013-08-15 20:50:29.771362 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.771269, event: header_read, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-138> 2013-08-15 20:50:29.771387 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.771271, event: throttled, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-137> 2013-08-15 20:50:29.771396 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.771318, event: all_read, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-136> 2013-08-15 20:50:29.771400 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.771359, event: dispatched, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-135> 2013-08-15 20:50:29.771415 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.771415, event: waiting_for_osdmap, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-134> 2013-08-15 20:50:29.771421 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.771421, event: wait for new map, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-133> 2013-08-15 20:50:29.772020 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 4 ==== pg_query(3.cb,3.10b epoch 89) v2 ==== 262+0+0 (1968573178 0 0) 0x23cab40 con 0x25e5b80
|
|
-132> 2013-08-15 20:50:29.772040 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.771966, event: header_read, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-131> 2013-08-15 20:50:29.772048 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.771967, event: throttled, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-130> 2013-08-15 20:50:29.772054 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.772004, event: all_read, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-129> 2013-08-15 20:50:29.772060 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.772037, event: dispatched, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-128> 2013-08-15 20:50:29.772075 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.772075, event: waiting_for_osdmap, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-127> 2013-08-15 20:50:29.772083 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.772083, event: wait for new map, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-126> 2013-08-15 20:50:29.772464 7ff6e226e700 1 -- 9.4.196.3:6827/8751 >> :/0 pipe(0x25e7900 sd=36 :6827 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:51789/0
|
|
-125> 2013-08-15 20:50:29.773018 7ff6e206c700 1 -- 9.4.196.3:6827/8751 >> :/0 pipe(0x25e7b80 sd=37 :6827 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:51792/0
|
|
-124> 2013-08-15 20:50:29.773864 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 5 ==== pg_query(3.1fe epoch 89) v2 ==== 137+0+0 (2502459247 0 0) 0x23cb4a0 con 0x25e5b80
|
|
-123> 2013-08-15 20:50:29.773890 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.773785, event: header_read, request: pg_query(3.1fe epoch 89) v2
|
|
-122> 2013-08-15 20:50:29.773904 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.773787, event: throttled, request: pg_query(3.1fe epoch 89) v2
|
|
-121> 2013-08-15 20:50:29.773913 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.773841, event: all_read, request: pg_query(3.1fe epoch 89) v2
|
|
-120> 2013-08-15 20:50:29.773923 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.773885, event: dispatched, request: pg_query(3.1fe epoch 89) v2
|
|
-119> 2013-08-15 20:50:29.773934 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.773934, event: waiting_for_osdmap, request: pg_query(3.1fe epoch 89) v2
|
|
-118> 2013-08-15 20:50:29.773946 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.773945, event: wait for new map, request: pg_query(3.1fe epoch 89) v2
|
|
-117> 2013-08-15 20:50:29.775939 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(87..89 src has 1..89) v3 ==== 2626+0+0 (3924186462 0 0) 0x2636fc0 con 0x25e58c0
|
|
-116> 2013-08-15 20:50:29.775963 7ff6eab8e700 3 osd.7 63 handle_osd_map epochs [87,89], i have 63, src has [1,89]
|
|
-115> 2013-08-15 20:50:29.775973 7ff6eab8e700 10 monclient: renew_subs
|
|
-114> 2013-08-15 20:50:29.775979 7ff6eab8e700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-113> 2013-08-15 20:50:29.775988 7ff6eab8e700 1 -- 9.4.196.3:6825/8751 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x23c88c0 con 0x25e4840
|
|
-112> 2013-08-15 20:50:29.776128 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.3 9.4.196.1:6810/7799 2 ==== pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2 ==== 1137+0+0 (640736252 0 0) 0x23cb680 con 0x25e58c0
|
|
-111> 2013-08-15 20:50:29.776155 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.776049, event: header_read, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-110> 2013-08-15 20:50:29.776168 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.776050, event: throttled, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-109> 2013-08-15 20:50:29.776180 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.776111, event: all_read, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-108> 2013-08-15 20:50:29.776191 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.776152, event: dispatched, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-107> 2013-08-15 20:50:29.776201 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.776201, event: waiting_for_osdmap, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-106> 2013-08-15 20:50:29.776212 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.776212, event: wait for new map, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-105> 2013-08-15 20:50:29.777826 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 6 ==== pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2 ==== 387+0+0 (2336248245 0 0) 0x23cb2c0 con 0x25e5b80
|
|
-104> 2013-08-15 20:50:29.777850 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.777726, event: header_read, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-103> 2013-08-15 20:50:29.777862 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.777728, event: throttled, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-102> 2013-08-15 20:50:29.777872 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.777805, event: all_read, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-101> 2013-08-15 20:50:29.777881 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.777847, event: dispatched, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-100> 2013-08-15 20:50:29.777891 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.777891, event: waiting_for_osdmap, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-99> 2013-08-15 20:50:29.777902 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.777902, event: wait for new map, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-98> 2013-08-15 20:50:29.778806 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.1 9.4.196.1:6801/5083 3 ==== pg_notify(3.24a(7) epoch 89) v4 ==== 869+0+0 (2328628780 0 0) 0x23c9500 con 0x25e54a0
|
|
-97> 2013-08-15 20:50:29.778831 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.778702, event: header_read, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-96> 2013-08-15 20:50:29.778843 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.778704, event: throttled, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-95> 2013-08-15 20:50:29.778853 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.778787, event: all_read, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-94> 2013-08-15 20:50:29.778863 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.778828, event: dispatched, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-93> 2013-08-15 20:50:29.778875 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.778875, event: waiting_for_osdmap, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-92> 2013-08-15 20:50:29.778886 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.778886, event: wait for new map, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-91> 2013-08-15 20:50:29.779796 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.1 9.4.196.1:6801/5083 4 ==== pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2 ==== 2387+0+0 (4044763928 0 0) 0x23cb860 con 0x25e54a0
|
|
-90> 2013-08-15 20:50:29.779826 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.779704, event: header_read, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-89> 2013-08-15 20:50:29.779845 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.779705, event: throttled, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-88> 2013-08-15 20:50:29.779861 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.779775, event: all_read, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-87> 2013-08-15 20:50:29.779880 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.779822, event: dispatched, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-86> 2013-08-15 20:50:29.779898 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.779898, event: waiting_for_osdmap, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-85> 2013-08-15 20:50:29.779918 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.779918, event: wait for new map, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-84> 2013-08-15 20:50:29.780183 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.3 9.4.196.1:6810/7799 3 ==== pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2 ==== 762+0+0 (3559476600 0 0) 0x23cbc20 con 0x25e58c0
|
|
-83> 2013-08-15 20:50:29.780209 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.780107, event: header_read, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-82> 2013-08-15 20:50:29.780225 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.780109, event: throttled, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-81> 2013-08-15 20:50:29.780237 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.780165, event: all_read, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-80> 2013-08-15 20:50:29.780248 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.780205, event: dispatched, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-79> 2013-08-15 20:50:29.780262 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.780262, event: waiting_for_osdmap, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-78> 2013-08-15 20:50:29.780277 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.780276, event: wait for new map, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-77> 2013-08-15 20:50:29.780842 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.2 9.4.196.1:6807/6887 4 ==== pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2 ==== 2387+0+0 (1609276427 0 0) 0x23ca5a0 con 0x25e5340
|
|
-76> 2013-08-15 20:50:29.780874 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.780735, event: header_read, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-75> 2013-08-15 20:50:29.780893 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.780737, event: throttled, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-74> 2013-08-15 20:50:29.780910 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.780821, event: all_read, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-73> 2013-08-15 20:50:29.780927 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.780870, event: dispatched, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-72> 2013-08-15 20:50:29.780945 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.780945, event: waiting_for_osdmap, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-71> 2013-08-15 20:50:29.780964 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.780964, event: wait for new map, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-70> 2013-08-15 20:50:29.782212 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.2 9.4.196.1:6807/6887 5 ==== pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2 ==== 762+0+0 (1286310374 0 0) 0x23cbe00 con 0x25e5340
|
|
-69> 2013-08-15 20:50:29.782238 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.782145, event: header_read, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-68> 2013-08-15 20:50:29.782255 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.782146, event: throttled, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-67> 2013-08-15 20:50:29.782264 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.782191, event: all_read, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-66> 2013-08-15 20:50:29.782274 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.782235, event: dispatched, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-65> 2013-08-15 20:50:29.782284 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.782284, event: waiting_for_osdmap, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-64> 2013-08-15 20:50:29.782295 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.782295, event: wait for new map, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-63> 2013-08-15 20:50:29.783415 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2 ==== 2012+0+0 (2078503555 0 0) 0x26601e0 con 0x25e5ce0
|
|
-62> 2013-08-15 20:50:29.783476 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.783309, event: header_read, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-61> 2013-08-15 20:50:29.783493 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.783311, event: throttled, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-60> 2013-08-15 20:50:29.783505 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.783390, event: all_read, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-59> 2013-08-15 20:50:29.783517 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.783472, event: dispatched, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-58> 2013-08-15 20:50:29.783531 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.783530, event: waiting_for_osdmap, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-57> 2013-08-15 20:50:29.783545 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.783545, event: wait for new map, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-56> 2013-08-15 20:50:29.842046 7ff6ebb90700 1 -- 9.4.196.3:6825/8751 <== mon.2 9.4.196.5:6789/0 11 ==== osd_map(64..89 src has 1..89) v3 ==== 26152+0+0 (4292010592 0 0) 0x25f3440 con 0x25e4840
|
|
-55> 2013-08-15 20:50:29.842072 7ff6ebb90700 3 osd.7 63 handle_osd_map epochs [64,89], i have 63, src has [1,89]
|
|
-54> 2013-08-15 20:50:29.846273 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-53> 2013-08-15 20:50:29.846285 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-52> 2013-08-15 20:50:29.846289 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-51> 2013-08-15 20:50:29.846304 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-50> 2013-08-15 20:50:29.846320 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-49> 2013-08-15 20:50:29.846325 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-48> 2013-08-15 20:50:29.846348 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-47> 2013-08-15 20:50:29.846355 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-46> 2013-08-15 20:50:29.846362 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-45> 2013-08-15 20:50:29.846371 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-44> 2013-08-15 20:50:29.846376 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-43> 2013-08-15 20:50:29.846384 7ff6ebb90700 1 osd.7 89 state: booting -> active
|
|
-42> 2013-08-15 20:50:29.846698 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.846698, event: started, request: pg_notify(3.175(8) epoch 89) v4
|
|
-41> 2013-08-15 20:50:29.846731 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.846731, event: done, request: pg_notify(3.175(8) epoch 89) v4
|
|
-40> 2013-08-15 20:50:29.846741 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.846741, event: started, request: pg_notify(3.175(7) epoch 89) v4
|
|
-39> 2013-08-15 20:50:29.846762 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.846761, event: done, request: pg_notify(3.175(7) epoch 89) v4
|
|
-38> 2013-08-15 20:50:29.846770 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.846770, event: started, request: pg_notify(3.175(7) epoch 89) v4
|
|
-37> 2013-08-15 20:50:29.846781 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.846781, event: done, request: pg_notify(3.175(7) epoch 89) v4
|
|
-36> 2013-08-15 20:50:29.846790 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.846790, event: started, request: pg_query(2.4 epoch 89) v2
|
|
-35> 2013-08-15 20:50:29.846870 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6807/6887 -- pg_notify(2.4 epoch 89) v4 -- ?+0 0x23c8540 con 0x25e5340
|
|
-34> 2013-08-15 20:50:29.846887 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.846887, event: done, request: pg_query(2.4 epoch 89) v2
|
|
-33> 2013-08-15 20:50:29.846899 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.846899, event: started, request: pg_query(0.6 epoch 89) v2
|
|
-32> 2013-08-15 20:50:29.846994 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6807/6887 -- pg_notify(0.6 epoch 89) v4 -- ?+0 0x23c8000 con 0x25e5340
|
|
-31> 2013-08-15 20:50:29.847014 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:29.847014, event: done, request: pg_query(0.6 epoch 89) v2
|
|
-30> 2013-08-15 20:50:29.847027 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.847027, event: started, request: pg_query(3.16d epoch 89) v2
|
|
-29> 2013-08-15 20:50:29.847050 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:29.847049, event: done, request: pg_query(3.16d epoch 89) v2
|
|
-28> 2013-08-15 20:50:29.847064 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.847064, event: started, request: pg_query(3.14e epoch 89) v2
|
|
-27> 2013-08-15 20:50:29.847187 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6813/8739 -- pg_notify(3.14e epoch 89) v4 -- ?+0 0x23c9880 con 0x25e5b80
|
|
-26> 2013-08-15 20:50:29.847203 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:29.847203, event: done, request: pg_query(3.14e epoch 89) v2
|
|
-25> 2013-08-15 20:50:29.847213 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.847213, event: started, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-24> 2013-08-15 20:50:29.847237 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:29.847237, event: done, request: pg_query(3.d6,3.e6,3.145,3.249 epoch 89) v2
|
|
-23> 2013-08-15 20:50:29.847246 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.847245, event: started, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-22> 2013-08-15 20:50:29.847259 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:29.847259, event: done, request: pg_notify(3.24a(8) epoch 89) v4
|
|
-21> 2013-08-15 20:50:29.847265 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.847265, event: started, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-20> 2013-08-15 20:50:29.847277 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:29.847277, event: done, request: pg_query(3.cb,3.10b epoch 89) v2
|
|
-19> 2013-08-15 20:50:29.847284 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.847284, event: started, request: pg_query(3.1fe epoch 89) v2
|
|
-18> 2013-08-15 20:50:29.847295 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:29.847295, event: done, request: pg_query(3.1fe epoch 89) v2
|
|
-17> 2013-08-15 20:50:29.847303 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.847303, event: started, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-16> 2013-08-15 20:50:29.847341 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6810/7799 -- pg_notify(3.b1 epoch 89) v4 -- ?+0 0x23c96c0 con 0x25e58c0
|
|
-15> 2013-08-15 20:50:29.847355 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:29.847355, event: done, request: pg_query(3.b1,3.13d,3.147,3.176,3.1ec,3.1f4,3.21d,3.21e,3.244 epoch 89) v2
|
|
-14> 2013-08-15 20:50:29.847369 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.847369, event: started, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-13> 2013-08-15 20:50:29.847437 7ff6f1b9c700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-12> 2013-08-15 20:50:29.847436 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.5e( v 63'10 (0'0,63'10] local-les=62 n=0 ec=56 les/c 62/62 89/89/56) [4,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-11> 2013-08-15 20:50:29.847459 7ff6f1b9c700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
-10> 2013-08-15 20:50:29.847456 7ff6e7387700 1 osd.7 pg_epoch: 89 pg[3.1ee( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 89/89/56) [2,7]/[2,5] r=-1 lpr=89 pi=56-88/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-9> 2013-08-15 20:50:29.847522 7ff6e6b86700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6813/8739 -- pg_notify(3.5e(12) epoch 89) v4 -- ?+0 0x23c9a40 con 0x25e5b80
|
|
-8> 2013-08-15 20:50:29.847792 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6813/8739 -- pg_notify(3.1d9 epoch 89) v4 -- ?+0 0x23c9dc0 con 0x25e5b80
|
|
-7> 2013-08-15 20:50:29.847813 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:29.847813, event: done, request: pg_query(3.5e,3.1bb,3.1d9 epoch 89) v2
|
|
-6> 2013-08-15 20:50:29.847827 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.847827, event: started, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-5> 2013-08-15 20:50:29.847847 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:29.847847, event: done, request: pg_notify(3.24a(7) epoch 89) v4
|
|
-4> 2013-08-15 20:50:29.847856 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.847856, event: started, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-3> 2013-08-15 20:50:29.848342 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.4 9.4.196.1:6813/8739 7 ==== pg_query(3.5e epoch 89) v2 ==== 137+0+0 (1097631480 0 0) 0x23cb0e0 con 0x25e5b80
|
|
-2> 2013-08-15 20:50:29.848390 7ff6e7387700 1 osd.7 pg_epoch: 89 pg[3.1e8( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 89/89/56) [1,7]/[1,5] r=-1 lpr=89 pi=56-88/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:29.848587 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.11f( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 89/89/56) [1,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:29.848757 7ff6f1b9c700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7ff6f1b9c700 time 2013-08-15 20:50:29.847467
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7ff6f8ba4f8e]
|
|
5: (clone()+0x6d) [0x7ff6f6e56e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:29.855535 7ff6f1b9c700 -1 *** Caught signal (Aborted) **
|
|
in thread 7ff6f1b9c700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7ff6f8bacbd0]
|
|
3: (gsignal()+0x37) [0x7ff6f6d94037]
|
|
4: (abort()+0x148) [0x7ff6f6d97698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7ff6f76a0e8d]
|
|
6: (()+0x5ef76) [0x7ff6f769ef76]
|
|
7: (()+0x5efa3) [0x7ff6f769efa3]
|
|
8: (()+0x5f1de) [0x7ff6f769f1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7ff6f8ba4f8e]
|
|
14: (clone()+0x6d) [0x7ff6f6e56e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-49> 2013-08-15 20:50:29.849173 7ff6e7387700 1 osd.7 pg_epoch: 89 pg[3.1e2( v 63'18 (0'0,63'18] local-les=62 n=6 ec=56 les/c 62/62 89/89/56) [1,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-48> 2013-08-15 20:50:29.849227 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e8(11),3.1e2(12) epoch 89) v4 -- ?+0 0x23c9a40 con 0x25e54a0
|
|
-47> 2013-08-15 20:50:29.849252 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ee(11) epoch 89) v4 -- ?+0 0x2794380 con 0x25e5340
|
|
-46> 2013-08-15 20:50:29.849343 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6801/5083 -- pg_notify(3.c5,3.102,3.152,3.1bc epoch 89) v4 -- ?+0 0x23c9c00 con 0x25e54a0
|
|
-45> 2013-08-15 20:50:29.849369 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:29.849369, event: done, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.195,3.1a3,3.1a9,3.1bc,3.1e2,3.1e8 epoch 89) v2
|
|
-44> 2013-08-15 20:50:29.849388 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.849388, event: started, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-43> 2013-08-15 20:50:29.849429 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e2(12) epoch 89) v4 -- ?+0 0x27941c0 con 0x25e54a0
|
|
-42> 2013-08-15 20:50:29.849452 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6813/8739 -- pg_notify(3.5e(12) epoch 89) v4 -- ?+0 0x2794000 con 0x25e5b80
|
|
-41> 2013-08-15 20:50:29.849550 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6810/7799 -- pg_notify(3.1b4 epoch 89) v4 -- ?+0 0x2794a80 con 0x25e58c0
|
|
-40> 2013-08-15 20:50:29.849553 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e8(11) epoch 89) v4 -- ?+0 0x2795180 con 0x25e54a0
|
|
-39> 2013-08-15 20:50:29.849570 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:29.849570, event: done, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 89) v2
|
|
-38> 2013-08-15 20:50:29.849588 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.849588, event: started, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-37> 2013-08-15 20:50:29.849645 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.11a( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 89/89/56) [2,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-36> 2013-08-15 20:50:29.849736 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:29.849736, event: done, request: pg_query(0.f,0.1f,0.28,0.29,1.e,1.1e,1.27,1.28,2.d,2.1d,2.26,2.27,3.c,3.1c,3.25,3.26,3.1bd,3.1c4,3.1ee epoch 89) v2
|
|
-35> 2013-08-15 20:50:29.849750 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.849750, event: started, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-34> 2013-08-15 20:50:29.849764 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ee(11) epoch 89) v4 -- ?+0 0x2794fc0 con 0x25e5340
|
|
-33> 2013-08-15 20:50:29.849840 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6807/6887 -- pg_notify(1.5,3.3 epoch 89) v4 -- ?+0 0x27948c0 con 0x25e5340
|
|
-32> 2013-08-15 20:50:29.849862 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:29.849862, event: done, request: pg_query(1.5,3.3,3.f9,3.11a,3.187,3.18b epoch 89) v2
|
|
-31> 2013-08-15 20:50:29.849878 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.849878, event: started, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-30> 2013-08-15 20:50:29.850171 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.5b( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 89/89/56) [0,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-29> 2013-08-15 20:50:29.850665 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.1d4( v 63'15 (0'0,63'15] local-les=62 n=3 ec=56 les/c 62/62 89/89/56) [0,7]/[0,10] r=-1 lpr=89 pi=56-88/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-28> 2013-08-15 20:50:29.850740 7ff6ebb90700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6804/5992 -- pg_notify(3.207 epoch 89) v4 -- ?+0 0x2794700 con 0x25e5ce0
|
|
-27> 2013-08-15 20:50:29.850754 7ff6ebb90700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:29.850754, event: done, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.a2,3.1bf,3.1d4,3.1f2,3.207 epoch 89) v2
|
|
-26> 2013-08-15 20:50:29.850775 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.848230, event: header_read, request: pg_query(3.5e epoch 89) v2
|
|
-25> 2013-08-15 20:50:29.850788 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.848231, event: throttled, request: pg_query(3.5e epoch 89) v2
|
|
-24> 2013-08-15 20:50:29.850792 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.848289, event: all_read, request: pg_query(3.5e epoch 89) v2
|
|
-23> 2013-08-15 20:50:29.850796 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.848361, event: dispatched, request: pg_query(3.5e epoch 89) v2
|
|
-22> 2013-08-15 20:50:29.850801 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.850801, event: waiting_for_osdmap, request: pg_query(3.5e epoch 89) v2
|
|
-21> 2013-08-15 20:50:29.850807 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.850807, event: started, request: pg_query(3.5e epoch 89) v2
|
|
-20> 2013-08-15 20:50:29.850825 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:29.850825, event: done, request: pg_query(3.5e epoch 89) v2
|
|
-19> 2013-08-15 20:50:29.850838 7ff6eab8e700 1 -- 9.4.196.3:6826/8751 <== osd.1 9.4.196.1:6801/5083 5 ==== pg_query(3.1e2 epoch 89) v2 ==== 137+0+0 (542011626 0 0) 0x26605a0 con 0x25e54a0
|
|
-18> 2013-08-15 20:50:29.850849 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850079, event: header_read, request: pg_query(3.1e2 epoch 89) v2
|
|
-17> 2013-08-15 20:50:29.850853 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850080, event: throttled, request: pg_query(3.1e2 epoch 89) v2
|
|
-16> 2013-08-15 20:50:29.850857 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850122, event: all_read, request: pg_query(3.1e2 epoch 89) v2
|
|
-15> 2013-08-15 20:50:29.850850 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6813/8739 -- pg_log(3.5e epoch 89 query_epoch 89) v3 -- ?+0 0x23d1600 con 0x25e5b80
|
|
-14> 2013-08-15 20:50:29.850860 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850848, event: dispatched, request: pg_query(3.1e2 epoch 89) v2
|
|
-13> 2013-08-15 20:50:29.850864 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850864, event: waiting_for_osdmap, request: pg_query(3.1e2 epoch 89) v2
|
|
-12> 2013-08-15 20:50:29.850868 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850868, event: started, request: pg_query(3.1e2 epoch 89) v2
|
|
-11> 2013-08-15 20:50:29.850892 7ff6eab8e700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:29.850892, event: done, request: pg_query(3.1e2 epoch 89) v2
|
|
-10> 2013-08-15 20:50:29.850909 7ff6e7387700 1 -- 9.4.196.3:6826/8751 --> 9.4.196.1:6801/5083 -- pg_log(3.1e2 epoch 89 query_epoch 89) v3 -- ?+0 0x23d2680 con 0x25e54a0
|
|
-9> 2013-08-15 20:50:29.851240 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.4c( v 63'3 (0'0,63'3] local-les=62 n=1 ec=56 les/c 62/62 89/89/56) [0,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-8> 2013-08-15 20:50:29.851670 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.10b( v 63'11 (0'0,63'11] local-les=62 n=9 ec=56 les/c 62/62 89/89/56) [4,7]/[4,5] r=-1 lpr=89 pi=56-88/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-7> 2013-08-15 20:50:29.852159 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.1c4( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 89/89/56) [2,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-6> 2013-08-15 20:50:29.852646 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.46( v 63'5 (0'0,63'5] local-les=62 n=1 ec=56 les/c 62/62 89/89/56) [3,7]/[3,10] r=-1 lpr=89 pi=56-88/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:29.853164 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.47( v 63'17 (0'0,63'17] local-les=62 n=3 ec=56 les/c 62/62 89/89/56) [0,7]/[0,10] r=-1 lpr=89 pi=56-88/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:29.853677 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.3e( v 63'13 (0'0,63'13] local-les=62 n=1 ec=56 les/c 62/62 89/89/85) [7,1]/[1,4] r=-1 lpr=89 pi=61-88/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:29.854187 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[2.3f( v 55'102 (0'0,55'102] local-les=62 n=0 ec=1 les/c 62/62 89/89/85) [7,1]/[1,4] r=-1 lpr=89 pi=61-88/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:29.854690 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.1bd( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 89/89/85) [7,2]/[2,0] r=-1 lpr=89 pi=61-88/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:29.855202 7ff6e6b86700 1 osd.7 pg_epoch: 89 pg[3.1bf( v 63'24 (0'0,63'24] local-les=62 n=8 ec=56 les/c 62/62 89/89/56) [0,7] r=1 lpr=89 pi=56-88/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:29.855535 7ff6f1b9c700 -1 *** Caught signal (Aborted) **
|
|
in thread 7ff6f1b9c700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7ff6f8bacbd0]
|
|
3: (gsignal()+0x37) [0x7ff6f6d94037]
|
|
4: (abort()+0x148) [0x7ff6f6d97698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7ff6f76a0e8d]
|
|
6: (()+0x5ef76) [0x7ff6f769ef76]
|
|
7: (()+0x5efa3) [0x7ff6f769efa3]
|
|
8: (()+0x5f1de) [0x7ff6f769f1de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7ff6f8ba4f8e]
|
|
14: (clone()+0x6d) [0x7ff6f6e56e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:30.010609 7f05a80017c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9115
|
|
2013-08-15 20:50:30.011112 7f05a80017c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:30.039957 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:30.039968 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:30.040245 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:30.057985 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:30.058103 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:30.082253 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:30.083070 7f05a80017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:30.083091 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.163341 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.163773 7f05a80017c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
2013-08-15 20:50:30.189987 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:30.190000 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:30.190246 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:30.201968 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:30.202029 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:30.232334 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:30.232829 7f05a80017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:30.232856 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.312292 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:31.793595 7f05a09d5700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:31.793614 7f05a09d5700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
2013-08-15 20:50:31.794863 7f05a09d5700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f05a09d5700 time 2013-08-15 20:50:31.793621
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f05a79ddf8e]
|
|
5: (clone()+0x6d) [0x7f05a5c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-216> 2013-08-15 20:50:30.009313 7f05a80017c0 5 asok(0x25a20e0) register_command perfcounters_dump hook 0x259c010
|
|
-215> 2013-08-15 20:50:30.009340 7f05a80017c0 5 asok(0x25a20e0) register_command 1 hook 0x259c010
|
|
-214> 2013-08-15 20:50:30.009343 7f05a80017c0 5 asok(0x25a20e0) register_command perf dump hook 0x259c010
|
|
-213> 2013-08-15 20:50:30.009349 7f05a80017c0 5 asok(0x25a20e0) register_command perfcounters_schema hook 0x259c010
|
|
-212> 2013-08-15 20:50:30.009377 7f05a80017c0 5 asok(0x25a20e0) register_command 2 hook 0x259c010
|
|
-211> 2013-08-15 20:50:30.009384 7f05a80017c0 5 asok(0x25a20e0) register_command perf schema hook 0x259c010
|
|
-210> 2013-08-15 20:50:30.009389 7f05a80017c0 5 asok(0x25a20e0) register_command config show hook 0x259c010
|
|
-209> 2013-08-15 20:50:30.009394 7f05a80017c0 5 asok(0x25a20e0) register_command config set hook 0x259c010
|
|
-208> 2013-08-15 20:50:30.009397 7f05a80017c0 5 asok(0x25a20e0) register_command log flush hook 0x259c010
|
|
-207> 2013-08-15 20:50:30.009402 7f05a80017c0 5 asok(0x25a20e0) register_command log dump hook 0x259c010
|
|
-206> 2013-08-15 20:50:30.009406 7f05a80017c0 5 asok(0x25a20e0) register_command log reopen hook 0x259c010
|
|
-205> 2013-08-15 20:50:30.010609 7f05a80017c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9115
|
|
-204> 2013-08-15 20:50:30.011112 7f05a80017c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-203> 2013-08-15 20:50:30.011231 7f05a80017c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6832/9115 need_addr=1
|
|
-202> 2013-08-15 20:50:30.011261 7f05a80017c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-201> 2013-08-15 20:50:30.011269 7f05a80017c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6833/9115 need_addr=0
|
|
-200> 2013-08-15 20:50:30.011293 7f05a80017c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-199> 2013-08-15 20:50:30.011299 7f05a80017c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6834/9115 need_addr=0
|
|
-198> 2013-08-15 20:50:30.013097 7f05a80017c0 5 asok(0x25a20e0) init /var/run/ceph/ceph-osd.7.asok
|
|
-197> 2013-08-15 20:50:30.013112 7f05a80017c0 5 asok(0x25a20e0) bind_and_listen /var/run/ceph/ceph-osd.7.asok
|
|
-196> 2013-08-15 20:50:30.013142 7f05a80017c0 5 asok(0x25a20e0) register_command 0 hook 0x259a0b8
|
|
-195> 2013-08-15 20:50:30.013148 7f05a80017c0 5 asok(0x25a20e0) register_command version hook 0x259a0b8
|
|
-194> 2013-08-15 20:50:30.013151 7f05a80017c0 5 asok(0x25a20e0) register_command git_version hook 0x259a0b8
|
|
-193> 2013-08-15 20:50:30.013154 7f05a80017c0 5 asok(0x25a20e0) register_command help hook 0x259c0d0
|
|
-192> 2013-08-15 20:50:30.013182 7f05a39db700 5 asok(0x25a20e0) entry start
|
|
-191> 2013-08-15 20:50:30.039957 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-190> 2013-08-15 20:50:30.039968 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-189> 2013-08-15 20:50:30.040245 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-188> 2013-08-15 20:50:30.057985 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-187> 2013-08-15 20:50:30.058103 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-186> 2013-08-15 20:50:30.082253 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-185> 2013-08-15 20:50:30.083045 7f05a80017c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-184> 2013-08-15 20:50:30.083070 7f05a80017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-183> 2013-08-15 20:50:30.083091 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-182> 2013-08-15 20:50:30.088322 7f05a80017c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-181> 2013-08-15 20:50:30.092313 7f05a80017c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-180> 2013-08-15 20:50:30.096254 7f05a80017c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-179> 2013-08-15 20:50:30.100297 7f05a80017c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-178> 2013-08-15 20:50:30.104386 7f05a80017c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-177> 2013-08-15 20:50:30.108364 7f05a80017c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-176> 2013-08-15 20:50:30.112312 7f05a80017c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-175> 2013-08-15 20:50:30.116374 7f05a80017c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-174> 2013-08-15 20:50:30.120625 7f05a80017c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-173> 2013-08-15 20:50:30.124945 7f05a80017c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-172> 2013-08-15 20:50:30.129261 7f05a80017c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-171> 2013-08-15 20:50:30.133542 7f05a80017c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-170> 2013-08-15 20:50:30.137823 7f05a80017c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-169> 2013-08-15 20:50:30.142124 7f05a80017c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-168> 2013-08-15 20:50:30.146400 7f05a80017c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-167> 2013-08-15 20:50:30.150858 7f05a80017c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-166> 2013-08-15 20:50:30.155042 7f05a80017c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-165> 2013-08-15 20:50:30.159141 7f05a80017c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-164> 2013-08-15 20:50:30.163234 7f05a80017c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-163> 2013-08-15 20:50:30.163276 7f05a80017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-162> 2013-08-15 20:50:30.163281 7f05a80017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-161> 2013-08-15 20:50:30.163297 7f05a80017c0 3 journal journal_replay: end of journal, done.
|
|
-160> 2013-08-15 20:50:30.163341 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-159> 2013-08-15 20:50:30.163688 7f05a09d5700 1 FileStore::op_tp worker finish
|
|
-158> 2013-08-15 20:50:30.163712 7f05a11d6700 1 FileStore::op_tp worker finish
|
|
-157> 2013-08-15 20:50:30.163773 7f05a80017c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
-156> 2013-08-15 20:50:30.164017 7f05a80017c0 10 monclient(hunting): build_initial_monmap
|
|
-155> 2013-08-15 20:50:30.164070 7f05a80017c0 5 adding auth protocol: cephx
|
|
-154> 2013-08-15 20:50:30.164077 7f05a80017c0 5 adding auth protocol: cephx
|
|
-153> 2013-08-15 20:50:30.164177 7f05a80017c0 1 -- 0.0.0.0:6832/9115 messenger.start
|
|
-152> 2013-08-15 20:50:30.164206 7f05a80017c0 1 -- :/0 messenger.start
|
|
-151> 2013-08-15 20:50:30.164216 7f05a80017c0 1 -- 9.4.196.3:6834/9115 messenger.start
|
|
-150> 2013-08-15 20:50:30.164227 7f05a80017c0 1 -- 9.4.196.3:6833/9115 messenger.start
|
|
-149> 2013-08-15 20:50:30.164308 7f05a80017c0 2 osd.7 0 mounting /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
|
|
-148> 2013-08-15 20:50:30.189987 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-147> 2013-08-15 20:50:30.190000 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-146> 2013-08-15 20:50:30.190246 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-145> 2013-08-15 20:50:30.201968 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-144> 2013-08-15 20:50:30.202029 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-143> 2013-08-15 20:50:30.232334 7f05a80017c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-142> 2013-08-15 20:50:30.232798 7f05a80017c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-141> 2013-08-15 20:50:30.232829 7f05a80017c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-140> 2013-08-15 20:50:30.232856 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-139> 2013-08-15 20:50:30.237053 7f05a80017c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-138> 2013-08-15 20:50:30.241094 7f05a80017c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-137> 2013-08-15 20:50:30.245110 7f05a80017c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-136> 2013-08-15 20:50:30.249094 7f05a80017c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-135> 2013-08-15 20:50:30.253147 7f05a80017c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-134> 2013-08-15 20:50:30.257183 7f05a80017c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-133> 2013-08-15 20:50:30.261226 7f05a80017c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-132> 2013-08-15 20:50:30.265293 7f05a80017c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-131> 2013-08-15 20:50:30.269511 7f05a80017c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-130> 2013-08-15 20:50:30.273727 7f05a80017c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-129> 2013-08-15 20:50:30.278008 7f05a80017c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-128> 2013-08-15 20:50:30.282319 7f05a80017c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-127> 2013-08-15 20:50:30.286608 7f05a80017c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-126> 2013-08-15 20:50:30.290894 7f05a80017c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-125> 2013-08-15 20:50:30.295233 7f05a80017c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-124> 2013-08-15 20:50:30.299552 7f05a80017c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-123> 2013-08-15 20:50:30.303789 7f05a80017c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-122> 2013-08-15 20:50:30.307998 7f05a80017c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-121> 2013-08-15 20:50:30.312215 7f05a80017c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-120> 2013-08-15 20:50:30.312238 7f05a80017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-119> 2013-08-15 20:50:30.312244 7f05a80017c0 2 journal No further valid entries found, journal is most likely valid
|
|
-118> 2013-08-15 20:50:30.312245 7f05a80017c0 3 journal journal_replay: end of journal, done.
|
|
-117> 2013-08-15 20:50:30.312292 7f05a80017c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-116> 2013-08-15 20:50:30.312494 7f05a80017c0 2 osd.7 0 boot
|
|
-115> 2013-08-15 20:50:30.337576 7f05a80017c0 2 osd.7 63 superblock: i am osd.7
|
|
-114> 2013-08-15 20:50:30.337628 7f05a80017c0 1 accepter.accepter.start
|
|
-113> 2013-08-15 20:50:30.337677 7f05a80017c0 1 accepter.accepter.start
|
|
-112> 2013-08-15 20:50:30.337719 7f05a80017c0 1 accepter.accepter.start
|
|
-111> 2013-08-15 20:50:30.337738 7f05a80017c0 10 monclient(hunting): init
|
|
-110> 2013-08-15 20:50:30.337744 7f05a80017c0 5 adding auth protocol: cephx
|
|
-109> 2013-08-15 20:50:30.337755 7f05a80017c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-108> 2013-08-15 20:50:30.337909 7f05a80017c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring
|
|
-107> 2013-08-15 20:50:30.338068 7f05a80017c0 5 asok(0x25a20e0) register_command dump_ops_in_flight hook 0x259c770
|
|
-106> 2013-08-15 20:50:30.338077 7f05a80017c0 5 asok(0x25a20e0) register_command dump_historic_ops hook 0x259c770
|
|
-105> 2013-08-15 20:50:30.338081 7f05a80017c0 5 asok(0x25a20e0) register_command dump_op_pq_state hook 0x259c770
|
|
-104> 2013-08-15 20:50:30.338084 7f05a80017c0 5 asok(0x25a20e0) register_command setomapval hook 0x2798ae0
|
|
-103> 2013-08-15 20:50:30.338087 7f05a80017c0 5 asok(0x25a20e0) register_command rmomapkey hook 0x2798ae0
|
|
-102> 2013-08-15 20:50:30.338090 7f05a80017c0 5 asok(0x25a20e0) register_command setomapheader hook 0x2798ae0
|
|
-101> 2013-08-15 20:50:30.338093 7f05a80017c0 5 asok(0x25a20e0) register_command getomap hook 0x2798ae0
|
|
-100> 2013-08-15 20:50:30.338095 7f05a80017c0 5 asok(0x25a20e0) register_command truncobj hook 0x2798ae0
|
|
-99> 2013-08-15 20:50:30.338098 7f05a80017c0 5 asok(0x25a20e0) register_command injectdataerr hook 0x2798ae0
|
|
-98> 2013-08-15 20:50:30.338104 7f05a80017c0 5 asok(0x25a20e0) register_command injectmdataerr hook 0x2798ae0
|
|
-97> 2013-08-15 20:50:30.338161 7f05a80017c0 10 monclient(hunting): _reopen_session
|
|
-96> 2013-08-15 20:50:30.338168 7f05939bb700 5 osd.7 63 heartbeat: osd_stat(734 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-95> 2013-08-15 20:50:30.338220 7f05a80017c0 10 monclient(hunting): _pick_new_mon picked mon.noname-a con 0x27ee840 addr 9.4.196.4:6789/0
|
|
-94> 2013-08-15 20:50:30.338236 7f05a80017c0 10 monclient(hunting): _send_mon_message to mon.noname-a at 9.4.196.4:6789/0
|
|
-93> 2013-08-15 20:50:30.338242 7f05a80017c0 1 -- 0.0.0.0:6832/9115 --> 9.4.196.4:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x25aeb40 con 0x27ee840
|
|
-92> 2013-08-15 20:50:30.338256 7f05a80017c0 10 monclient(hunting): renew_subs
|
|
-91> 2013-08-15 20:50:30.338866 7f05a7f95700 1 -- 9.4.196.3:6832/9115 learned my addr 9.4.196.3:6832/9115
|
|
-90> 2013-08-15 20:50:30.339155 7f059a9c9700 10 monclient(hunting): renew_subs
|
|
-89> 2013-08-15 20:50:30.339643 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 1 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x25d41e0 con 0x27ee840
|
|
-88> 2013-08-15 20:50:30.339677 7f059a9c9700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-87> 2013-08-15 20:50:30.339692 7f059a9c9700 10 monclient(hunting): got monmap 1, mon.noname-a is now rank -1
|
|
-86> 2013-08-15 20:50:30.339695 7f059a9c9700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-85> 2013-08-15 20:50:30.339714 7f059a9c9700 1 monclient(hunting): found mon.ceph-dmon1
|
|
-84> 2013-08-15 20:50:30.339722 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (1913914157 0 0) 0x27f6200 con 0x27ee840
|
|
-83> 2013-08-15 20:50:30.339745 7f059a9c9700 10 monclient: my global_id is 4321
|
|
-82> 2013-08-15 20:50:30.339884 7f059a9c9700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-81> 2013-08-15 20:50:30.339893 7f059a9c9700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x25afd40 con 0x27ee840
|
|
-80> 2013-08-15 20:50:30.340490 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3280048206 0 0) 0x27f6000 con 0x27ee840
|
|
-79> 2013-08-15 20:50:30.340556 7f059a9c9700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-78> 2013-08-15 20:50:30.340575 7f059a9c9700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x27fc240 con 0x27ee840
|
|
-77> 2013-08-15 20:50:30.341266 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (1837960775 0 0) 0x27f6400 con 0x27ee840
|
|
-76> 2013-08-15 20:50:30.341311 7f059a9c9700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-75> 2013-08-15 20:50:30.341317 7f059a9c9700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x25d2000 con 0x27ee840
|
|
-74> 2013-08-15 20:50:30.341331 7f059a9c9700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-73> 2013-08-15 20:50:30.341335 7f059a9c9700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x25d2380 con 0x27ee840
|
|
-72> 2013-08-15 20:50:30.341345 7f059a9c9700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:00.341344)
|
|
-71> 2013-08-15 20:50:30.341354 7f059a9c9700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-70> 2013-08-15 20:50:30.341359 7f059a9c9700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x27fc000 con 0x27ee840
|
|
-69> 2013-08-15 20:50:30.341383 7f05a80017c0 5 monclient: authenticate success, global_id 4321
|
|
-68> 2013-08-15 20:50:30.341396 7f05a80017c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:00.341395)
|
|
-67> 2013-08-15 20:50:30.341754 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 5 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x25d45a0 con 0x27ee840
|
|
-66> 2013-08-15 20:50:30.341764 7f059a9c9700 10 monclient: handle_monmap mon_map v1
|
|
-65> 2013-08-15 20:50:30.341772 7f059a9c9700 10 monclient: got monmap 1, mon.ceph-dmon1 is now rank 1
|
|
-64> 2013-08-15 20:50:30.341776 7f059a9c9700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-63> 2013-08-15 20:50:30.341793 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x25d2000 con 0x27ee840
|
|
-62> 2013-08-15 20:50:30.341802 7f059a9c9700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:30.338259 renew after 2013-08-15 20:53:00.338259
|
|
-61> 2013-08-15 20:50:30.341842 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 7 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x25d43c0 con 0x27ee840
|
|
-60> 2013-08-15 20:50:30.341851 7f059a9c9700 10 monclient: handle_monmap mon_map v1
|
|
-59> 2013-08-15 20:50:30.341857 7f059a9c9700 10 monclient: got monmap 1, mon.ceph-dmon1 is now rank 1
|
|
-58> 2013-08-15 20:50:30.341860 7f059a9c9700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-57> 2013-08-15 20:50:30.341893 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x25d2540 con 0x27ee840
|
|
-56> 2013-08-15 20:50:30.341902 7f059a9c9700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-55> 2013-08-15 20:50:30.342035 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1998792193 0 0) 0x27fe600 con 0x27ee840
|
|
-54> 2013-08-15 20:50:30.342074 7f059a9c9700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:00.342074)
|
|
-53> 2013-08-15 20:50:30.342086 7f05a80017c0 10 monclient: wait_auth_rotating done
|
|
-52> 2013-08-15 20:50:30.342662 7f05a80017c0 10 monclient: get_version osdmap
|
|
-51> 2013-08-15 20:50:30.342674 7f05a80017c0 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-50> 2013-08-15 20:50:30.342685 7f05a80017c0 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x260a9c0 con 0x27ee840
|
|
-49> 2013-08-15 20:50:30.343008 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 10 ==== mon_check_map_ack(handle=1 version=89) v2 ==== 24+0+0 (3324254003 0 0) 0x25d2380 con 0x27ee840
|
|
-48> 2013-08-15 20:50:30.343062 7f05969c1700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-47> 2013-08-15 20:50:30.343078 7f05969c1700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- osd_boot(osd.7 booted 0 v63) v3 -- ?+0 0x281a000 con 0x27ee840
|
|
-46> 2013-08-15 20:50:31.784545 7f05916ad700 1 -- 9.4.196.3:6833/9115 >> :/0 pipe(0x27f1400 sd=33 :6833 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:42106/0
|
|
-45> 2013-08-15 20:50:31.785088 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.6 9.4.196.3:6801/4121 1 ==== pg_notify(3.175(10) epoch 91) v4 ==== 990+0+0 (1991842462 0 0) 0x25d2700 con 0x2802c60
|
|
-44> 2013-08-15 20:50:31.785147 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.785006, event: header_read, request: pg_notify(3.175(10) epoch 91) v4
|
|
-43> 2013-08-15 20:50:31.785160 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.785007, event: throttled, request: pg_notify(3.175(10) epoch 91) v4
|
|
-42> 2013-08-15 20:50:31.785167 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.785069, event: all_read, request: pg_notify(3.175(10) epoch 91) v4
|
|
-41> 2013-08-15 20:50:31.785186 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.785135, event: dispatched, request: pg_notify(3.175(10) epoch 91) v4
|
|
-40> 2013-08-15 20:50:31.785193 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.785193, event: waiting_for_osdmap, request: pg_notify(3.175(10) epoch 91) v4
|
|
-39> 2013-08-15 20:50:31.785214 7f05999c7700 10 monclient: renew_subs
|
|
-38> 2013-08-15 20:50:31.785223 7f05999c7700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-37> 2013-08-15 20:50:31.785234 7f05999c7700 1 -- 9.4.196.3:6832/9115 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x25d28c0 con 0x27ee840
|
|
-36> 2013-08-15 20:50:31.785256 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.785256, event: wait for new map, request: pg_notify(3.175(10) epoch 91) v4
|
|
-35> 2013-08-15 20:50:31.786788 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 11 ==== osd_map(64..91 src has 1..91) v3 ==== 28549+0+0 (322815474 0 0) 0x27fd440 con 0x27ee840
|
|
-34> 2013-08-15 20:50:31.786811 7f059a9c9700 3 osd.7 63 handle_osd_map epochs [64,91], i have 63, src has [1,91]
|
|
-33> 2013-08-15 20:50:31.787425 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x27fd680 con 0x27ef760
|
|
-32> 2013-08-15 20:50:31.792702 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-31> 2013-08-15 20:50:31.792719 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-30> 2013-08-15 20:50:31.792728 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-29> 2013-08-15 20:50:31.792756 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-28> 2013-08-15 20:50:31.792777 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-27> 2013-08-15 20:50:31.792788 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-26> 2013-08-15 20:50:31.792824 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-25> 2013-08-15 20:50:31.792839 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-24> 2013-08-15 20:50:31.792852 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-23> 2013-08-15 20:50:31.792869 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-22> 2013-08-15 20:50:31.792879 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-21> 2013-08-15 20:50:31.792896 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6828/8914 -- pipe dne
|
|
-20> 2013-08-15 20:50:31.792908 7f059a9c9700 1 -- 9.4.196.3:6833/9115 mark_down 9.4.196.3:6813/8749 -- pipe dne
|
|
-19> 2013-08-15 20:50:31.792923 7f059a9c9700 1 osd.7 91 state: booting -> active
|
|
-18> 2013-08-15 20:50:31.793413 7f059a9c9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.793413, event: started, request: pg_notify(3.175(10) epoch 91) v4
|
|
-17> 2013-08-15 20:50:31.793452 7f059a9c9700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.793452, event: done, request: pg_notify(3.175(10) epoch 91) v4
|
|
-16> 2013-08-15 20:50:31.793473 7f05999c7700 3 osd.7 91 handle_osd_map epochs [90,91], i have 91, src has [1,91]
|
|
-15> 2013-08-15 20:50:31.793485 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_notify(3.175(9) epoch 91) v4 ==== 947+0+0 (3129455416 0 0) 0x25d2fc0 con 0x27ef760
|
|
-14> 2013-08-15 20:50:31.793503 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787425, event: header_read, request: pg_notify(3.175(9) epoch 91) v4
|
|
-13> 2013-08-15 20:50:31.793512 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787426, event: throttled, request: pg_notify(3.175(9) epoch 91) v4
|
|
-12> 2013-08-15 20:50:31.793519 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787550, event: all_read, request: pg_notify(3.175(9) epoch 91) v4
|
|
-11> 2013-08-15 20:50:31.793526 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.793500, event: dispatched, request: pg_notify(3.175(9) epoch 91) v4
|
|
-10> 2013-08-15 20:50:31.793533 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.793533, event: waiting_for_osdmap, request: pg_notify(3.175(9) epoch 91) v4
|
|
-9> 2013-08-15 20:50:31.793542 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.793542, event: started, request: pg_notify(3.175(9) epoch 91) v4
|
|
-8> 2013-08-15 20:50:31.793566 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.793566, event: done, request: pg_notify(3.175(9) epoch 91) v4
|
|
-7> 2013-08-15 20:50:31.793589 7f059a9c9700 1 -- 9.4.196.3:6832/9115 <== mon.1 9.4.196.4:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x25d2e00 con 0x27ee840
|
|
-6> 2013-08-15 20:50:31.793604 7f059a9c9700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:31.785221 renew after 2013-08-15 20:53:01.785221
|
|
-5> 2013-08-15 20:50:31.793595 7f05a09d5700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-4> 2013-08-15 20:50:31.793614 7f05a09d5700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
-3> 2013-08-15 20:50:31.793927 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.1ee( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 91/91/56) [2,7]/[2,5] r=-1 lpr=91 pi=56-90/13 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:31.794053 7f05959bf700 1 -- 9.4.196.3:6833/9115 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ee(13) epoch 91) v4 -- ?+0 0x25d2000 con 0x27ef340
|
|
-1> 2013-08-15 20:50:31.794850 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.5e( v 63'10 (0'0,63'10] local-les=62 n=0 ec=56 les/c 62/62 91/91/56) [4,7] r=1 lpr=91 pi=56-90/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:31.794863 7f05a09d5700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f05a09d5700 time 2013-08-15 20:50:31.793621
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f05a79ddf8e]
|
|
5: (clone()+0x6d) [0x7f05a5c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:31.800475 7f05a09d5700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f05a09d5700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f05a79e5bd0]
|
|
3: (gsignal()+0x37) [0x7f05a5bcd037]
|
|
4: (abort()+0x148) [0x7f05a5bd0698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f05a64d9e8d]
|
|
6: (()+0x5ef76) [0x7f05a64d7f76]
|
|
7: (()+0x5efa3) [0x7f05a64d7fa3]
|
|
8: (()+0x5f1de) [0x7f05a64d81de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f05a79ddf8e]
|
|
14: (clone()+0x6d) [0x7f05a5c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-48> 2013-08-15 20:50:31.795531 7f05912a9700 1 -- 9.4.196.3:6833/9115 >> :/0 pipe(0x2950280 sd=35 :6833 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:42111/0
|
|
-47> 2013-08-15 20:50:31.795578 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.11f( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 91/91/56) [1,7] r=1 lpr=91 pi=56-90/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-46> 2013-08-15 20:50:31.796176 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.175(9) epoch 91) v4 ==== 947+0+0 (2393759481 0 0) 0x25d3180 con 0x2802dc0
|
|
-45> 2013-08-15 20:50:31.796196 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796091, event: header_read, request: pg_notify(3.175(9) epoch 91) v4
|
|
-44> 2013-08-15 20:50:31.796206 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796093, event: throttled, request: pg_notify(3.175(9) epoch 91) v4
|
|
-43> 2013-08-15 20:50:31.796213 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796155, event: all_read, request: pg_notify(3.175(9) epoch 91) v4
|
|
-42> 2013-08-15 20:50:31.796219 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796193, event: dispatched, request: pg_notify(3.175(9) epoch 91) v4
|
|
-41> 2013-08-15 20:50:31.796225 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796225, event: waiting_for_osdmap, request: pg_notify(3.175(9) epoch 91) v4
|
|
-40> 2013-08-15 20:50:31.796234 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796234, event: started, request: pg_notify(3.175(9) epoch 91) v4
|
|
-39> 2013-08-15 20:50:31.796257 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.796257, event: done, request: pg_notify(3.175(9) epoch 91) v4
|
|
-38> 2013-08-15 20:50:31.796388 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.11a( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 91/91/56) [2,7] r=1 lpr=91 pi=56-90/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-37> 2013-08-15 20:50:31.796701 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x28e9680 con 0x27efce0
|
|
-36> 2013-08-15 20:50:31.796720 7f05999c7700 3 osd.7 91 handle_osd_map epochs [90,91], i have 91, src has [1,91]
|
|
-35> 2013-08-15 20:50:31.796843 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2 ==== 1637+0+0 (1436022769 0 0) 0x25d41e0 con 0x27efce0
|
|
-34> 2013-08-15 20:50:31.796871 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.796753, event: header_read, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-33> 2013-08-15 20:50:31.796885 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.796755, event: throttled, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-32> 2013-08-15 20:50:31.796897 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.796824, event: all_read, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-31> 2013-08-15 20:50:31.796907 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.796867, event: dispatched, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-30> 2013-08-15 20:50:31.796919 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.796919, event: waiting_for_osdmap, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-29> 2013-08-15 20:50:31.796936 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.796936, event: started, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-28> 2013-08-15 20:50:31.797160 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.5b( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 91/91/56) [0,7] r=1 lpr=91 pi=56-90/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-27> 2013-08-15 20:50:31.797937 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.1d4( v 63'15 (0'0,63'15] local-les=62 n=3 ec=56 les/c 62/62 91/91/56) [0,7]/[0,10] r=-1 lpr=91 pi=56-90/13 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-26> 2013-08-15 20:50:31.798038 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:31.798038, event: done, request: pg_query(0.1e,0.31,1.1d,1.30,2.1c,2.2f,3.1b,3.2e,3.47,3.4c,3.5b,3.1bf,3.1d4 epoch 91) v2
|
|
-25> 2013-08-15 20:50:31.798752 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.4c( v 63'3 (0'0,63'3] local-les=62 n=1 ec=56 les/c 62/62 91/91/56) [0,7] r=1 lpr=91 pi=56-90/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-24> 2013-08-15 20:50:31.798980 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x28e9d40 con 0x27ef8c0
|
|
-23> 2013-08-15 20:50:31.799003 7f05999c7700 3 osd.7 91 handle_osd_map epochs [90,91], i have 91, src has [1,91]
|
|
-22> 2013-08-15 20:50:31.799018 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(3.1bb epoch 91) v2 ==== 137+0+0 (3182266282 0 0) 0x25d45a0 con 0x27ef8c0
|
|
-21> 2013-08-15 20:50:31.799034 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.798805, event: header_read, request: pg_query(3.1bb epoch 91) v2
|
|
-20> 2013-08-15 20:50:31.799046 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.798806, event: throttled, request: pg_query(3.1bb epoch 91) v2
|
|
-19> 2013-08-15 20:50:31.799056 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.798850, event: all_read, request: pg_query(3.1bb epoch 91) v2
|
|
-18> 2013-08-15 20:50:31.799065 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.799032, event: dispatched, request: pg_query(3.1bb epoch 91) v2
|
|
-17> 2013-08-15 20:50:31.799075 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.799075, event: waiting_for_osdmap, request: pg_query(3.1bb epoch 91) v2
|
|
-16> 2013-08-15 20:50:31.799086 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.799086, event: started, request: pg_query(3.1bb epoch 91) v2
|
|
-15> 2013-08-15 20:50:31.799107 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:31.799107, event: done, request: pg_query(3.1bb epoch 91) v2
|
|
-14> 2013-08-15 20:50:31.799413 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.10b( v 63'11 (0'0,63'11] local-les=62 n=9 ec=56 les/c 62/62 91/91/56) [4,7]/[4,5] r=-1 lpr=91 pi=56-90/13 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-13> 2013-08-15 20:50:31.799430 7f05910a7700 1 -- 9.4.196.3:6834/9115 >> :/0 pipe(0x2950500 sd=36 :6834 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:43920/0
|
|
-12> 2013-08-15 20:50:31.799894 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x25f8480 con 0x2802420
|
|
-11> 2013-08-15 20:50:31.799915 7f05999c7700 3 osd.7 91 handle_osd_map epochs [90,91], i have 91, src has [1,91]
|
|
-10> 2013-08-15 20:50:31.799973 7f05999c7700 1 -- 9.4.196.3:6833/9115 <== osd.3 9.4.196.1:6810/7799 2 ==== pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2 ==== 637+0+0 (978920974 0 0) 0x25d4780 con 0x2802420
|
|
-9> 2013-08-15 20:50:31.799996 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.799897, event: header_read, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-8> 2013-08-15 20:50:31.800006 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.799898, event: throttled, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-7> 2013-08-15 20:50:31.800014 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.799958, event: all_read, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-6> 2013-08-15 20:50:31.800021 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.799992, event: dispatched, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-5> 2013-08-15 20:50:31.800030 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.800029, event: waiting_for_osdmap, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-4> 2013-08-15 20:50:31.800039 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.800039, event: started, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-3> 2013-08-15 20:50:31.800093 7f05999c7700 1 -- 9.4.196.3:6833/9115 --> 9.4.196.1:6810/7799 -- pg_notify(3.b1 epoch 91) v4 -- ?+0 0x25d3500 con 0x2802420
|
|
-2> 2013-08-15 20:50:31.800111 7f05999c7700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:31.800110, event: done, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 91) v2
|
|
-1> 2013-08-15 20:50:31.800198 7f05959bf700 1 osd.7 pg_epoch: 91 pg[3.1c4( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 91/91/56) [2,7] r=1 lpr=91 pi=56-90/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:31.800475 7f05a09d5700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f05a09d5700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f05a79e5bd0]
|
|
3: (gsignal()+0x37) [0x7f05a5bcd037]
|
|
4: (abort()+0x148) [0x7f05a5bd0698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f05a64d9e8d]
|
|
6: (()+0x5ef76) [0x7f05a64d7f76]
|
|
7: (()+0x5efa3) [0x7f05a64d7fa3]
|
|
8: (()+0x5f1de) [0x7f05a64d81de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f05a79ddf8e]
|
|
14: (clone()+0x6d) [0x7f05a5c8fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:32.113798 7fbe3f4a17c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9437
|
|
2013-08-15 20:50:32.114291 7fbe3f4a17c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:32.145145 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:32.145158 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:32.145432 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:32.163167 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:32.163242 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:32.187504 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:32.188237 7fbe3f4a17c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:32.188269 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.269012 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.269831 7fbe3f4a17c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
2013-08-15 20:50:32.301197 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:32.301213 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:32.301519 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:32.313146 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:32.313191 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:32.343366 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:32.343888 7fbe3f4a17c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:32.343896 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.423382 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:33.848738 7fbe37e75700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:33.848751 7fbe37e75700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
2013-08-15 20:50:33.849756 7fbe37e75700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fbe37e75700 time 2013-08-15 20:50:33.848756
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fbe3ee7df8e]
|
|
5: (clone()+0x6d) [0x7fbe3d12fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-351> 2013-08-15 20:50:32.112506 7fbe3f4a17c0 5 asok(0x29a20e0) register_command perfcounters_dump hook 0x299c010
|
|
-350> 2013-08-15 20:50:32.112532 7fbe3f4a17c0 5 asok(0x29a20e0) register_command 1 hook 0x299c010
|
|
-349> 2013-08-15 20:50:32.112548 7fbe3f4a17c0 5 asok(0x29a20e0) register_command perf dump hook 0x299c010
|
|
-348> 2013-08-15 20:50:32.112553 7fbe3f4a17c0 5 asok(0x29a20e0) register_command perfcounters_schema hook 0x299c010
|
|
-347> 2013-08-15 20:50:32.112555 7fbe3f4a17c0 5 asok(0x29a20e0) register_command 2 hook 0x299c010
|
|
-346> 2013-08-15 20:50:32.112557 7fbe3f4a17c0 5 asok(0x29a20e0) register_command perf schema hook 0x299c010
|
|
-345> 2013-08-15 20:50:32.112561 7fbe3f4a17c0 5 asok(0x29a20e0) register_command config show hook 0x299c010
|
|
-344> 2013-08-15 20:50:32.112563 7fbe3f4a17c0 5 asok(0x29a20e0) register_command config set hook 0x299c010
|
|
-343> 2013-08-15 20:50:32.112566 7fbe3f4a17c0 5 asok(0x29a20e0) register_command log flush hook 0x299c010
|
|
-342> 2013-08-15 20:50:32.112568 7fbe3f4a17c0 5 asok(0x29a20e0) register_command log dump hook 0x299c010
|
|
-341> 2013-08-15 20:50:32.112570 7fbe3f4a17c0 5 asok(0x29a20e0) register_command log reopen hook 0x299c010
|
|
-340> 2013-08-15 20:50:32.113798 7fbe3f4a17c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9437
|
|
-339> 2013-08-15 20:50:32.114291 7fbe3f4a17c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-338> 2013-08-15 20:50:32.114410 7fbe3f4a17c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6838/9437 need_addr=1
|
|
-337> 2013-08-15 20:50:32.114445 7fbe3f4a17c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-336> 2013-08-15 20:50:32.114452 7fbe3f4a17c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6839/9437 need_addr=0
|
|
-335> 2013-08-15 20:50:32.114476 7fbe3f4a17c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-334> 2013-08-15 20:50:32.114481 7fbe3f4a17c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6840/9437 need_addr=0
|
|
-333> 2013-08-15 20:50:32.116608 7fbe3f4a17c0 5 asok(0x29a20e0) init /var/run/ceph/ceph-osd.7.asok
|
|
-332> 2013-08-15 20:50:32.116634 7fbe3f4a17c0 5 asok(0x29a20e0) bind_and_listen /var/run/ceph/ceph-osd.7.asok
|
|
-331> 2013-08-15 20:50:32.116660 7fbe3f4a17c0 5 asok(0x29a20e0) register_command 0 hook 0x299a0b8
|
|
-330> 2013-08-15 20:50:32.116663 7fbe3f4a17c0 5 asok(0x29a20e0) register_command version hook 0x299a0b8
|
|
-329> 2013-08-15 20:50:32.116666 7fbe3f4a17c0 5 asok(0x29a20e0) register_command git_version hook 0x299a0b8
|
|
-328> 2013-08-15 20:50:32.116668 7fbe3f4a17c0 5 asok(0x29a20e0) register_command help hook 0x299c0d0
|
|
-327> 2013-08-15 20:50:32.116719 7fbe3ae7b700 5 asok(0x29a20e0) entry start
|
|
-326> 2013-08-15 20:50:32.145145 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-325> 2013-08-15 20:50:32.145158 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-324> 2013-08-15 20:50:32.145432 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-323> 2013-08-15 20:50:32.163167 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-322> 2013-08-15 20:50:32.163242 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-321> 2013-08-15 20:50:32.187504 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-320> 2013-08-15 20:50:32.188203 7fbe3f4a17c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-319> 2013-08-15 20:50:32.188237 7fbe3f4a17c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-318> 2013-08-15 20:50:32.188269 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-317> 2013-08-15 20:50:32.193869 7fbe3f4a17c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-316> 2013-08-15 20:50:32.197777 7fbe3f4a17c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-315> 2013-08-15 20:50:32.201736 7fbe3f4a17c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-314> 2013-08-15 20:50:32.205846 7fbe3f4a17c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-313> 2013-08-15 20:50:32.209826 7fbe3f4a17c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-312> 2013-08-15 20:50:32.213894 7fbe3f4a17c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-311> 2013-08-15 20:50:32.218077 7fbe3f4a17c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-310> 2013-08-15 20:50:32.222257 7fbe3f4a17c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-309> 2013-08-15 20:50:32.226680 7fbe3f4a17c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-308> 2013-08-15 20:50:32.230897 7fbe3f4a17c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-307> 2013-08-15 20:50:32.235132 7fbe3f4a17c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-306> 2013-08-15 20:50:32.239347 7fbe3f4a17c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-305> 2013-08-15 20:50:32.243603 7fbe3f4a17c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-304> 2013-08-15 20:50:32.247811 7fbe3f4a17c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-303> 2013-08-15 20:50:32.252018 7fbe3f4a17c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-302> 2013-08-15 20:50:32.256212 7fbe3f4a17c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-301> 2013-08-15 20:50:32.260465 7fbe3f4a17c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-300> 2013-08-15 20:50:32.264682 7fbe3f4a17c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-299> 2013-08-15 20:50:32.268888 7fbe3f4a17c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-298> 2013-08-15 20:50:32.268936 7fbe3f4a17c0 2 journal No further valid entries found, journal is most likely valid
|
|
-297> 2013-08-15 20:50:32.268954 7fbe3f4a17c0 2 journal No further valid entries found, journal is most likely valid
|
|
-296> 2013-08-15 20:50:32.268962 7fbe3f4a17c0 3 journal journal_replay: end of journal, done.
|
|
-295> 2013-08-15 20:50:32.269012 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-294> 2013-08-15 20:50:32.269736 7fbe38676700 1 FileStore::op_tp worker finish
|
|
-293> 2013-08-15 20:50:32.269756 7fbe37e75700 1 FileStore::op_tp worker finish
|
|
-292> 2013-08-15 20:50:32.269831 7fbe3f4a17c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
-291> 2013-08-15 20:50:32.270090 7fbe3f4a17c0 10 monclient(hunting): build_initial_monmap
|
|
-290> 2013-08-15 20:50:32.270143 7fbe3f4a17c0 5 adding auth protocol: cephx
|
|
-289> 2013-08-15 20:50:32.270150 7fbe3f4a17c0 5 adding auth protocol: cephx
|
|
-288> 2013-08-15 20:50:32.270237 7fbe3f4a17c0 1 -- 0.0.0.0:6838/9437 messenger.start
|
|
-287> 2013-08-15 20:50:32.270266 7fbe3f4a17c0 1 -- :/0 messenger.start
|
|
-286> 2013-08-15 20:50:32.270276 7fbe3f4a17c0 1 -- 9.4.196.3:6840/9437 messenger.start
|
|
-285> 2013-08-15 20:50:32.270288 7fbe3f4a17c0 1 -- 9.4.196.3:6839/9437 messenger.start
|
|
-284> 2013-08-15 20:50:32.270374 7fbe3f4a17c0 2 osd.7 0 mounting /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
|
|
-283> 2013-08-15 20:50:32.301197 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-282> 2013-08-15 20:50:32.301213 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-281> 2013-08-15 20:50:32.301519 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-280> 2013-08-15 20:50:32.313146 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-279> 2013-08-15 20:50:32.313191 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-278> 2013-08-15 20:50:32.343366 7fbe3f4a17c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-277> 2013-08-15 20:50:32.343870 7fbe3f4a17c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-276> 2013-08-15 20:50:32.343888 7fbe3f4a17c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-275> 2013-08-15 20:50:32.343896 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-274> 2013-08-15 20:50:32.347986 7fbe3f4a17c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-273> 2013-08-15 20:50:32.351913 7fbe3f4a17c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-272> 2013-08-15 20:50:32.355803 7fbe3f4a17c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-271> 2013-08-15 20:50:32.359789 7fbe3f4a17c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-270> 2013-08-15 20:50:32.363677 7fbe3f4a17c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-269> 2013-08-15 20:50:32.367844 7fbe3f4a17c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-268> 2013-08-15 20:50:32.371937 7fbe3f4a17c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-267> 2013-08-15 20:50:32.375965 7fbe3f4a17c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-266> 2013-08-15 20:50:32.380003 7fbe3f4a17c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-265> 2013-08-15 20:50:32.384015 7fbe3f4a17c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-264> 2013-08-15 20:50:32.388074 7fbe3f4a17c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-263> 2013-08-15 20:50:32.392507 7fbe3f4a17c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-262> 2013-08-15 20:50:32.396943 7fbe3f4a17c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-261> 2013-08-15 20:50:32.401340 7fbe3f4a17c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-260> 2013-08-15 20:50:32.405760 7fbe3f4a17c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-259> 2013-08-15 20:50:32.410169 7fbe3f4a17c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-258> 2013-08-15 20:50:32.414551 7fbe3f4a17c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-257> 2013-08-15 20:50:32.418926 7fbe3f4a17c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-256> 2013-08-15 20:50:32.423272 7fbe3f4a17c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-255> 2013-08-15 20:50:32.423306 7fbe3f4a17c0 2 journal No further valid entries found, journal is most likely valid
|
|
-254> 2013-08-15 20:50:32.423311 7fbe3f4a17c0 2 journal No further valid entries found, journal is most likely valid
|
|
-253> 2013-08-15 20:50:32.423313 7fbe3f4a17c0 3 journal journal_replay: end of journal, done.
|
|
-252> 2013-08-15 20:50:32.423382 7fbe3f4a17c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-251> 2013-08-15 20:50:32.423639 7fbe3f4a17c0 2 osd.7 0 boot
|
|
-250> 2013-08-15 20:50:32.448926 7fbe3f4a17c0 2 osd.7 63 superblock: i am osd.7
|
|
-249> 2013-08-15 20:50:32.448999 7fbe3f4a17c0 1 accepter.accepter.start
|
|
-248> 2013-08-15 20:50:32.449075 7fbe3f4a17c0 1 accepter.accepter.start
|
|
-247> 2013-08-15 20:50:32.449129 7fbe3f4a17c0 1 accepter.accepter.start
|
|
-246> 2013-08-15 20:50:32.449149 7fbe3f4a17c0 10 monclient(hunting): init
|
|
-245> 2013-08-15 20:50:32.449156 7fbe3f4a17c0 5 adding auth protocol: cephx
|
|
-244> 2013-08-15 20:50:32.449161 7fbe3f4a17c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-243> 2013-08-15 20:50:32.449310 7fbe3f4a17c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring
|
|
-242> 2013-08-15 20:50:32.449485 7fbe3f4a17c0 5 asok(0x29a20e0) register_command dump_ops_in_flight hook 0x299c770
|
|
-241> 2013-08-15 20:50:32.449507 7fbe3f4a17c0 5 asok(0x29a20e0) register_command dump_historic_ops hook 0x299c770
|
|
-240> 2013-08-15 20:50:32.449512 7fbe3f4a17c0 5 asok(0x29a20e0) register_command dump_op_pq_state hook 0x299c770
|
|
-239> 2013-08-15 20:50:32.449518 7fbe3f4a17c0 5 asok(0x29a20e0) register_command setomapval hook 0x2b98ae0
|
|
-238> 2013-08-15 20:50:32.449522 7fbe3f4a17c0 5 asok(0x29a20e0) register_command rmomapkey hook 0x2b98ae0
|
|
-237> 2013-08-15 20:50:32.449525 7fbe3f4a17c0 5 asok(0x29a20e0) register_command setomapheader hook 0x2b98ae0
|
|
-236> 2013-08-15 20:50:32.449529 7fbe3f4a17c0 5 asok(0x29a20e0) register_command getomap hook 0x2b98ae0
|
|
-235> 2013-08-15 20:50:32.449533 7fbe3f4a17c0 5 asok(0x29a20e0) register_command truncobj hook 0x2b98ae0
|
|
-234> 2013-08-15 20:50:32.449536 7fbe3f4a17c0 5 asok(0x29a20e0) register_command injectdataerr hook 0x2b98ae0
|
|
-233> 2013-08-15 20:50:32.449538 7fbe3f4a17c0 5 asok(0x29a20e0) register_command injectmdataerr hook 0x2b98ae0
|
|
-232> 2013-08-15 20:50:32.449554 7fbe2ae5b700 5 osd.7 63 heartbeat: osd_stat(734 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-231> 2013-08-15 20:50:32.449597 7fbe3f4a17c0 10 monclient(hunting): _reopen_session
|
|
-230> 2013-08-15 20:50:32.449648 7fbe3f4a17c0 10 monclient(hunting): _pick_new_mon picked mon.noname-b con 0x2bee840 addr 9.4.196.5:6789/0
|
|
-229> 2013-08-15 20:50:32.449676 7fbe3f4a17c0 10 monclient(hunting): _send_mon_message to mon.noname-b at 9.4.196.5:6789/0
|
|
-228> 2013-08-15 20:50:32.449682 7fbe3f4a17c0 1 -- 0.0.0.0:6838/9437 --> 9.4.196.5:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x29aeb40 con 0x2bee840
|
|
-227> 2013-08-15 20:50:32.449695 7fbe3f4a17c0 10 monclient(hunting): renew_subs
|
|
-226> 2013-08-15 20:50:32.450265 7fbe3f435700 1 -- 9.4.196.3:6838/9437 learned my addr 9.4.196.3:6838/9437
|
|
-225> 2013-08-15 20:50:32.450561 7fbe31e69700 10 monclient(hunting): renew_subs
|
|
-224> 2013-08-15 20:50:32.451172 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x29d41e0 con 0x2bee840
|
|
-223> 2013-08-15 20:50:32.451191 7fbe31e69700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-222> 2013-08-15 20:50:32.451220 7fbe31e69700 10 monclient(hunting): got monmap 1, mon.noname-b is now rank -1
|
|
-221> 2013-08-15 20:50:32.451225 7fbe31e69700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-220> 2013-08-15 20:50:32.451261 7fbe31e69700 1 monclient(hunting): found mon.ceph-dmon2
|
|
-219> 2013-08-15 20:50:32.451271 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3820426168 0 0) 0x2bf6200 con 0x2bee840
|
|
-218> 2013-08-15 20:50:32.451310 7fbe31e69700 10 monclient: my global_id is 4334
|
|
-217> 2013-08-15 20:50:32.451478 7fbe31e69700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-216> 2013-08-15 20:50:32.451489 7fbe31e69700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x29afd40 con 0x2bee840
|
|
-215> 2013-08-15 20:50:32.452348 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3419635816 0 0) 0x2bf6000 con 0x2bee840
|
|
-214> 2013-08-15 20:50:32.452434 7fbe31e69700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-213> 2013-08-15 20:50:32.452442 7fbe31e69700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2bfc240 con 0x2bee840
|
|
-212> 2013-08-15 20:50:32.453588 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (4157860860 0 0) 0x2bf6400 con 0x2bee840
|
|
-211> 2013-08-15 20:50:32.453640 7fbe31e69700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-210> 2013-08-15 20:50:32.453659 7fbe31e69700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x29d2000 con 0x2bee840
|
|
-209> 2013-08-15 20:50:32.453681 7fbe31e69700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-208> 2013-08-15 20:50:32.453688 7fbe31e69700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x29d2380 con 0x2bee840
|
|
-207> 2013-08-15 20:50:32.453717 7fbe31e69700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:02.453716)
|
|
-206> 2013-08-15 20:50:32.453729 7fbe31e69700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-205> 2013-08-15 20:50:32.453736 7fbe31e69700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2bfc000 con 0x2bee840
|
|
-204> 2013-08-15 20:50:32.453783 7fbe3f4a17c0 5 monclient: authenticate success, global_id 4334
|
|
-203> 2013-08-15 20:50:32.453796 7fbe3f4a17c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:02.453795)
|
|
-202> 2013-08-15 20:50:32.454246 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x29d45a0 con 0x2bee840
|
|
-201> 2013-08-15 20:50:32.454263 7fbe31e69700 10 monclient: handle_monmap mon_map v1
|
|
-200> 2013-08-15 20:50:32.454279 7fbe31e69700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-199> 2013-08-15 20:50:32.454283 7fbe31e69700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-198> 2013-08-15 20:50:32.454309 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x29d2000 con 0x2bee840
|
|
-197> 2013-08-15 20:50:32.454319 7fbe31e69700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:32.449698 renew after 2013-08-15 20:53:02.449698
|
|
-196> 2013-08-15 20:50:32.454405 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x29d43c0 con 0x2bee840
|
|
-195> 2013-08-15 20:50:32.454429 7fbe31e69700 10 monclient: handle_monmap mon_map v1
|
|
-194> 2013-08-15 20:50:32.454444 7fbe31e69700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-193> 2013-08-15 20:50:32.454451 7fbe31e69700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-192> 2013-08-15 20:50:32.454476 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x29d2540 con 0x2bee840
|
|
-191> 2013-08-15 20:50:32.454485 7fbe31e69700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-190> 2013-08-15 20:50:32.454752 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (582160742 0 0) 0x2bfe600 con 0x2bee840
|
|
-189> 2013-08-15 20:50:32.454809 7fbe31e69700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:02.454809)
|
|
-188> 2013-08-15 20:50:32.454824 7fbe3f4a17c0 10 monclient: wait_auth_rotating done
|
|
-187> 2013-08-15 20:50:32.455340 7fbe3f4a17c0 10 monclient: get_version osdmap
|
|
-186> 2013-08-15 20:50:32.455347 7fbe3f4a17c0 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-185> 2013-08-15 20:50:32.455355 7fbe3f4a17c0 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x2a0a9c0 con 0x2bee840
|
|
-184> 2013-08-15 20:50:32.455754 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 10 ==== mon_check_map_ack(handle=1 version=91) v2 ==== 24+0+0 (667974206 0 0) 0x29d2380 con 0x2bee840
|
|
-183> 2013-08-15 20:50:32.455808 7fbe2de61700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-182> 2013-08-15 20:50:32.455826 7fbe2de61700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- osd_boot(osd.7 booted 0 v63) v3 -- ?+0 0x2c1a000 con 0x2bee840
|
|
-181> 2013-08-15 20:50:33.761641 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x2bfd200 con 0x2befce0
|
|
-180> 2013-08-15 20:50:33.761685 7fbe30e67700 3 osd.7 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-179> 2013-08-15 20:50:33.761698 7fbe30e67700 10 monclient: renew_subs
|
|
-178> 2013-08-15 20:50:33.761703 7fbe30e67700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-177> 2013-08-15 20:50:33.761710 7fbe30e67700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x29d2700 con 0x2bee840
|
|
-176> 2013-08-15 20:50:33.761728 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2 ==== 762+0+0 (2595181954 0 0) 0x29d41e0 con 0x2befce0
|
|
-175> 2013-08-15 20:50:33.761744 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.761596, event: header_read, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-174> 2013-08-15 20:50:33.761755 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.761597, event: throttled, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-173> 2013-08-15 20:50:33.761760 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.761665, event: all_read, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-172> 2013-08-15 20:50:33.761764 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.761739, event: dispatched, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-171> 2013-08-15 20:50:33.761770 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.761770, event: waiting_for_osdmap, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-170> 2013-08-15 20:50:33.761782 7fbe30e67700 10 monclient: renew_subs
|
|
-169> 2013-08-15 20:50:33.761786 7fbe30e67700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-168> 2013-08-15 20:50:33.761791 7fbe30e67700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x29d2a80 con 0x2bee840
|
|
-167> 2013-08-15 20:50:33.761802 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.761801, event: wait for new map, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-166> 2013-08-15 20:50:33.764541 7fbe28b4d700 1 -- 9.4.196.3:6840/9437 >> :/0 pipe(0x2bf1400 sd=33 :6840 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:35755/0
|
|
-165> 2013-08-15 20:50:33.767109 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x2bfd8c0 con 0x2bef4a0
|
|
-164> 2013-08-15 20:50:33.767135 7fbe30e67700 3 osd.7 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-163> 2013-08-15 20:50:33.767143 7fbe30e67700 10 monclient: renew_subs
|
|
-162> 2013-08-15 20:50:33.767146 7fbe30e67700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-161> 2013-08-15 20:50:33.767157 7fbe30e67700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x29d28c0 con 0x2bee840
|
|
-160> 2013-08-15 20:50:33.767183 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(3.16d epoch 93) v2 ==== 137+0+0 (3528884154 0 0) 0x29d45a0 con 0x2bef4a0
|
|
-159> 2013-08-15 20:50:33.767204 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.767121, event: header_read, request: pg_query(3.16d epoch 93) v2
|
|
-158> 2013-08-15 20:50:33.767218 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.767122, event: throttled, request: pg_query(3.16d epoch 93) v2
|
|
-157> 2013-08-15 20:50:33.767225 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.767158, event: all_read, request: pg_query(3.16d epoch 93) v2
|
|
-156> 2013-08-15 20:50:33.767230 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.767201, event: dispatched, request: pg_query(3.16d epoch 93) v2
|
|
-155> 2013-08-15 20:50:33.767236 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.767236, event: waiting_for_osdmap, request: pg_query(3.16d epoch 93) v2
|
|
-154> 2013-08-15 20:50:33.767243 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.767243, event: wait for new map, request: pg_query(3.16d epoch 93) v2
|
|
-153> 2013-08-15 20:50:33.768243 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x2bfd200 con 0x2bef760
|
|
-152> 2013-08-15 20:50:33.768267 7fbe30e67700 3 osd.7 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-151> 2013-08-15 20:50:33.768276 7fbe30e67700 10 monclient: renew_subs
|
|
-150> 2013-08-15 20:50:33.768279 7fbe30e67700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-149> 2013-08-15 20:50:33.768284 7fbe30e67700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x29d28c0 con 0x2bee840
|
|
-148> 2013-08-15 20:50:33.768321 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.2 9.4.196.1:6807/6887 2 ==== pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2 ==== 637+0+0 (1808341177 0 0) 0x29d4780 con 0x2bef760
|
|
-147> 2013-08-15 20:50:33.768341 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.768255, event: header_read, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-146> 2013-08-15 20:50:33.768355 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.768256, event: throttled, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-145> 2013-08-15 20:50:33.768363 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.768307, event: all_read, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-144> 2013-08-15 20:50:33.768370 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.768338, event: dispatched, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-143> 2013-08-15 20:50:33.768377 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.768377, event: waiting_for_osdmap, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-142> 2013-08-15 20:50:33.768385 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.768385, event: wait for new map, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-141> 2013-08-15 20:50:33.768988 7fbe2894b700 1 -- 9.4.196.3:6839/9437 >> :/0 pipe(0x2bf1900 sd=34 :6839 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:60995/0
|
|
-140> 2013-08-15 20:50:33.769128 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2 ==== 1762+0+0 (1098414387 0 0) 0x29d4b40 con 0x2befce0
|
|
-139> 2013-08-15 20:50:33.769159 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.769004, event: header_read, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-138> 2013-08-15 20:50:33.769204 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.769006, event: throttled, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-137> 2013-08-15 20:50:33.769242 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.769110, event: all_read, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-136> 2013-08-15 20:50:33.769253 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.769155, event: dispatched, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-135> 2013-08-15 20:50:33.769262 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.769262, event: waiting_for_osdmap, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-134> 2013-08-15 20:50:33.769293 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.769293, event: wait for new map, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-133> 2013-08-15 20:50:33.769720 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.175(11) epoch 93) v4 ==== 1025+0+0 (2656751121 0 0) 0x29d2c40 con 0x2c031e0
|
|
-132> 2013-08-15 20:50:33.769745 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.769519, event: header_read, request: pg_notify(3.175(11) epoch 93) v4
|
|
-131> 2013-08-15 20:50:33.769755 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.769521, event: throttled, request: pg_notify(3.175(11) epoch 93) v4
|
|
-130> 2013-08-15 20:50:33.769762 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.769593, event: all_read, request: pg_notify(3.175(11) epoch 93) v4
|
|
-129> 2013-08-15 20:50:33.769769 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.769742, event: dispatched, request: pg_notify(3.175(11) epoch 93) v4
|
|
-128> 2013-08-15 20:50:33.769776 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.769776, event: waiting_for_osdmap, request: pg_notify(3.175(11) epoch 93) v4
|
|
-127> 2013-08-15 20:50:33.769787 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.769787, event: wait for new map, request: pg_notify(3.175(11) epoch 93) v4
|
|
-126> 2013-08-15 20:50:33.769799 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.4 9.4.196.1:6813/8739 3 ==== pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2 ==== 887+0+0 (3537717236 0 0) 0x29d4f00 con 0x2bef4a0
|
|
-125> 2013-08-15 20:50:33.769817 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.769592, event: header_read, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-124> 2013-08-15 20:50:33.769845 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.769594, event: throttled, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-123> 2013-08-15 20:50:33.769854 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.769689, event: all_read, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-122> 2013-08-15 20:50:33.769863 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.769815, event: dispatched, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-121> 2013-08-15 20:50:33.769873 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.769873, event: waiting_for_osdmap, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-120> 2013-08-15 20:50:33.769897 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.769897, event: wait for new map, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-119> 2013-08-15 20:50:33.769916 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x2c40240 con 0x2bef340
|
|
-118> 2013-08-15 20:50:33.769934 7fbe30e67700 3 osd.7 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-117> 2013-08-15 20:50:33.769945 7fbe30e67700 10 monclient: renew_subs
|
|
-116> 2013-08-15 20:50:33.769981 7fbe30e67700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-115> 2013-08-15 20:50:33.769990 7fbe30e67700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x29d2a80 con 0x2bee840
|
|
-114> 2013-08-15 20:50:33.770013 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2 ==== 1387+0+0 (612307893 0 0) 0x29d50e0 con 0x2bef340
|
|
-113> 2013-08-15 20:50:33.770034 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.769843, event: header_read, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-112> 2013-08-15 20:50:33.770047 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.769844, event: throttled, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-111> 2013-08-15 20:50:33.770058 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.769918, event: all_read, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-110> 2013-08-15 20:50:33.770069 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.770031, event: dispatched, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-109> 2013-08-15 20:50:33.770080 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.770080, event: waiting_for_osdmap, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-108> 2013-08-15 20:50:33.770089 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.770089, event: wait for new map, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-107> 2013-08-15 20:50:33.770376 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x2c40900 con 0x2c02420
|
|
-106> 2013-08-15 20:50:33.770399 7fbe30e67700 3 osd.7 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-105> 2013-08-15 20:50:33.770410 7fbe30e67700 10 monclient: renew_subs
|
|
-104> 2013-08-15 20:50:33.770415 7fbe30e67700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-103> 2013-08-15 20:50:33.770421 7fbe30e67700 1 -- 9.4.196.3:6838/9437 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x29d2700 con 0x2bee840
|
|
-102> 2013-08-15 20:50:33.770437 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.3 9.4.196.1:6810/7799 2 ==== pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2 ==== 512+0+0 (4144900829 0 0) 0x29d52c0 con 0x2c02420
|
|
-101> 2013-08-15 20:50:33.770448 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.770317, event: header_read, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-100> 2013-08-15 20:50:33.770458 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.770318, event: throttled, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-99> 2013-08-15 20:50:33.770469 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.770364, event: all_read, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-98> 2013-08-15 20:50:33.770480 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.770447, event: dispatched, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-97> 2013-08-15 20:50:33.770491 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.770491, event: waiting_for_osdmap, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-96> 2013-08-15 20:50:33.770504 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.770504, event: wait for new map, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-95> 2013-08-15 20:50:33.774720 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.3 9.4.196.1:6810/7799 3 ==== pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2 ==== 637+0+0 (2001836301 0 0) 0x29d5680 con 0x2c02420
|
|
-94> 2013-08-15 20:50:33.774752 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.774637, event: header_read, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-93> 2013-08-15 20:50:33.774789 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.774639, event: throttled, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-92> 2013-08-15 20:50:33.774801 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.774703, event: all_read, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-91> 2013-08-15 20:50:33.774812 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.774748, event: dispatched, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-90> 2013-08-15 20:50:33.774825 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.774825, event: waiting_for_osdmap, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-89> 2013-08-15 20:50:33.774866 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.774841, event: wait for new map, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-88> 2013-08-15 20:50:33.775312 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.1 9.4.196.1:6801/5083 3 ==== pg_notify(3.175(11),3.24a(11) epoch 93) v4 ==== 2038+0+0 (4246739521 0 0) 0x29d2e00 con 0x2bef340
|
|
-87> 2013-08-15 20:50:33.775336 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.775162, event: header_read, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-86> 2013-08-15 20:50:33.775346 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.775164, event: throttled, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-85> 2013-08-15 20:50:33.775352 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.775288, event: all_read, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-84> 2013-08-15 20:50:33.775358 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.775334, event: dispatched, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-83> 2013-08-15 20:50:33.775365 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.775365, event: waiting_for_osdmap, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-82> 2013-08-15 20:50:33.775377 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.775377, event: wait for new map, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-81> 2013-08-15 20:50:33.775974 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.1 9.4.196.1:6801/5083 4 ==== pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2 ==== 1012+0+0 (759960074 0 0) 0x29d5a40 con 0x2bef340
|
|
-80> 2013-08-15 20:50:33.775997 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.775893, event: header_read, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-79> 2013-08-15 20:50:33.776007 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.775894, event: throttled, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-78> 2013-08-15 20:50:33.776015 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.775957, event: all_read, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-77> 2013-08-15 20:50:33.776022 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.775994, event: dispatched, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-76> 2013-08-15 20:50:33.776029 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.776029, event: waiting_for_osdmap, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-75> 2013-08-15 20:50:33.776038 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.776038, event: wait for new map, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-74> 2013-08-15 20:50:33.776200 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.3 9.4.196.1:6810/7799 4 ==== pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2 ==== 762+0+0 (1233794662 0 0) 0x29d54a0 con 0x2c02420
|
|
-73> 2013-08-15 20:50:33.776227 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.776139, event: header_read, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-72> 2013-08-15 20:50:33.776239 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.776141, event: throttled, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-71> 2013-08-15 20:50:33.776247 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.776184, event: all_read, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-70> 2013-08-15 20:50:33.776256 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.776224, event: dispatched, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-69> 2013-08-15 20:50:33.776264 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.776264, event: waiting_for_osdmap, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-68> 2013-08-15 20:50:33.776278 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.776277, event: wait for new map, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-67> 2013-08-15 20:50:33.776437 7fbe28749700 1 -- 9.4.196.3:6839/9437 >> :/0 pipe(0x2c5c280 sd=35 :6839 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:60998/0
|
|
-66> 2013-08-15 20:50:33.777106 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.5 9.4.196.3:6819/4129 1 ==== pg_notify(3.24a(12) epoch 93) v4 ==== 1068+0+0 (2762759628 0 0) 0x29d2fc0 con 0x2c03340
|
|
-65> 2013-08-15 20:50:33.777132 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.777038, event: header_read, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-64> 2013-08-15 20:50:33.777142 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.777039, event: throttled, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-63> 2013-08-15 20:50:33.777148 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.777089, event: all_read, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-62> 2013-08-15 20:50:33.777153 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.777128, event: dispatched, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-61> 2013-08-15 20:50:33.777159 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.777159, event: waiting_for_osdmap, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-60> 2013-08-15 20:50:33.777168 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.777167, event: wait for new map, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-59> 2013-08-15 20:50:33.779815 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.2 9.4.196.1:6807/6887 3 ==== pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2 ==== 2762+0+0 (2542665017 0 0) 0x29d5e00 con 0x2bef760
|
|
-58> 2013-08-15 20:50:33.779847 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.779713, event: header_read, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-57> 2013-08-15 20:50:33.779862 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.779715, event: throttled, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-56> 2013-08-15 20:50:33.779876 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.779794, event: all_read, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-55> 2013-08-15 20:50:33.779894 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.779844, event: dispatched, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-54> 2013-08-15 20:50:33.779912 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.779912, event: waiting_for_osdmap, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-53> 2013-08-15 20:50:33.779927 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.779927, event: wait for new map, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-52> 2013-08-15 20:50:33.842309 7fbe31e69700 1 -- 9.4.196.3:6838/9437 <== mon.2 9.4.196.5:6789/0 11 ==== osd_map(64..93 src has 1..93) v3 ==== 30396+0+0 (1597789786 0 0) 0x2c40240 con 0x2bee840
|
|
-51> 2013-08-15 20:50:33.842361 7fbe31e69700 3 osd.7 63 handle_osd_map epochs [64,93], i have 63, src has [1,93]
|
|
-50> 2013-08-15 20:50:33.847519 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-49> 2013-08-15 20:50:33.847531 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-48> 2013-08-15 20:50:33.847536 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-47> 2013-08-15 20:50:33.847551 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-46> 2013-08-15 20:50:33.847561 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-45> 2013-08-15 20:50:33.847567 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-44> 2013-08-15 20:50:33.847590 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-43> 2013-08-15 20:50:33.847597 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-42> 2013-08-15 20:50:33.847602 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-41> 2013-08-15 20:50:33.847609 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-40> 2013-08-15 20:50:33.847612 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-39> 2013-08-15 20:50:33.847619 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6828/8914 -- pipe dne
|
|
-38> 2013-08-15 20:50:33.847622 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6813/8749 -- pipe dne
|
|
-37> 2013-08-15 20:50:33.847630 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6830/9101 -- pipe dne
|
|
-36> 2013-08-15 20:50:33.847637 7fbe31e69700 1 -- 9.4.196.3:6839/9437 mark_down 9.4.196.3:6809/9079 -- pipe dne
|
|
-35> 2013-08-15 20:50:33.847645 7fbe31e69700 1 osd.7 93 state: booting -> active
|
|
-34> 2013-08-15 20:50:33.847984 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.847984, event: started, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-33> 2013-08-15 20:50:33.848027 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.848027, event: done, request: pg_query(0.1e,1.1d,2.1c,2.2f,3.1b,3.2e epoch 93) v2
|
|
-32> 2013-08-15 20:50:33.848039 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.848039, event: started, request: pg_query(3.16d epoch 93) v2
|
|
-31> 2013-08-15 20:50:33.848052 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.848052, event: done, request: pg_query(3.16d epoch 93) v2
|
|
-30> 2013-08-15 20:50:33.848060 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.848060, event: started, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-29> 2013-08-15 20:50:33.848083 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.848083, event: done, request: pg_query(0.f,1.e,2.d,3.c,3.1c epoch 93) v2
|
|
-28> 2013-08-15 20:50:33.848092 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.848091, event: started, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-27> 2013-08-15 20:50:33.848737 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6804/5992 -- pg_notify(3.207 epoch 93) v4 -- ?+0 0x29d2540 con 0x2befce0
|
|
-26> 2013-08-15 20:50:33.848738 7fbe37e75700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-25> 2013-08-15 20:50:33.848751 7fbe37e75700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
-24> 2013-08-15 20:50:33.848757 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.848757, event: done, request: pg_query(0.31,1.30,3.47,3.4c,3.5b,3.a2,3.d6,3.e6,3.145,3.1bf,3.1d4,3.1f2,3.207,3.249 epoch 93) v2
|
|
-23> 2013-08-15 20:50:33.848775 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.848775, event: started, request: pg_notify(3.175(11) epoch 93) v4
|
|
-22> 2013-08-15 20:50:33.848796 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:33.848795, event: done, request: pg_notify(3.175(11) epoch 93) v4
|
|
-21> 2013-08-15 20:50:33.848802 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.848802, event: started, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-20> 2013-08-15 20:50:33.848852 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6813/8739 -- pg_notify(3.14e,3.1d9 epoch 93) v4 -- ?+0 0x29d2000 con 0x2bef4a0
|
|
-19> 2013-08-15 20:50:33.848866 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:33.848866, event: done, request: pg_query(3.5e,3.cb,3.10b,3.14e,3.1bb,3.1d9,3.1fe epoch 93) v2
|
|
-18> 2013-08-15 20:50:33.848882 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.848882, event: started, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-17> 2013-08-15 20:50:33.848916 7fbe2ce5f700 1 osd.7 pg_epoch: 93 pg[3.1ee( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 93/93/56) [2,7]/[2,5] r=-1 lpr=93 pi=56-92/15 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-16> 2013-08-15 20:50:33.848956 7fbe2d660700 1 osd.7 pg_epoch: 93 pg[3.11a( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 93/93/56) [2,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-15> 2013-08-15 20:50:33.848968 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6801/5083 -- pg_notify(3.1bc epoch 93) v4 -- ?+0 0x29d3340 con 0x2bef340
|
|
-14> 2013-08-15 20:50:33.848990 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:33.848990, event: done, request: pg_query(0.22,1.21,2.20,2.3f,3.1f,3.3e,3.195,3.1a3,3.1a9,3.1bc,3.1e2 epoch 93) v2
|
|
-13> 2013-08-15 20:50:33.849004 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.849004, event: started, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-12> 2013-08-15 20:50:33.849025 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:33.849025, event: done, request: pg_query(3.1ec,3.1f4,3.21d,3.21e epoch 93) v2
|
|
-11> 2013-08-15 20:50:33.849034 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.849034, event: started, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-10> 2013-08-15 20:50:33.849027 7fbe2d660700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6807/6887 -- pg_notify(3.11a(16) epoch 93) v4 -- ?+0 0x29d3500 con 0x2bef760
|
|
-9> 2013-08-15 20:50:33.849122 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6810/7799 -- pg_notify(3.b1 epoch 93) v4 -- ?+0 0x29d3180 con 0x2c02420
|
|
-8> 2013-08-15 20:50:33.849142 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:33.849142, event: done, request: pg_query(3.b1,3.13d,3.147,3.176,3.244 epoch 93) v2
|
|
-7> 2013-08-15 20:50:33.849159 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.849159, event: started, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-6> 2013-08-15 20:50:33.849196 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:33.849196, event: done, request: pg_notify(3.175(11),3.24a(11) epoch 93) v4
|
|
-5> 2013-08-15 20:50:33.849211 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.849211, event: started, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-4> 2013-08-15 20:50:33.849583 7fbe2ce5f700 1 osd.7 pg_epoch: 93 pg[3.1e8( v 63'10 (0'0,63'10] local-les=62 n=2 ec=56 les/c 62/62 93/93/56) [1,7]/[1,5] r=-1 lpr=93 pi=56-92/15 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:33.849657 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6801/5083 -- pg_notify(3.c5,3.102,3.152 epoch 93) v4 -- ?+0 0x29d3c00 con 0x2bef340
|
|
-2> 2013-08-15 20:50:33.849672 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:33.849672, event: done, request: pg_query(3.a6,3.a7,3.c5,3.e4,3.102,3.11f,3.152,3.1e8 epoch 93) v2
|
|
-1> 2013-08-15 20:50:33.849685 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.849684, event: started, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
0> 2013-08-15 20:50:33.849756 7fbe37e75700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fbe37e75700 time 2013-08-15 20:50:33.848756
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fbe3ee7df8e]
|
|
5: (clone()+0x6d) [0x7fbe3d12fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:33.853715 7fbe37e75700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fbe37e75700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fbe3ee85bd0]
|
|
3: (gsignal()+0x37) [0x7fbe3d06d037]
|
|
4: (abort()+0x148) [0x7fbe3d070698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fbe3d979e8d]
|
|
6: (()+0x5ef76) [0x7fbe3d977f76]
|
|
7: (()+0x5efa3) [0x7fbe3d977fa3]
|
|
8: (()+0x5f1de) [0x7fbe3d9781de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fbe3ee7df8e]
|
|
14: (clone()+0x6d) [0x7fbe3d12fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-56> 2013-08-15 20:50:33.849785 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6810/7799 -- pg_notify(3.1b4 epoch 93) v4 -- ?+0 0x29d3a40 con 0x2c02420
|
|
-55> 2013-08-15 20:50:33.849807 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:33.849806, event: done, request: pg_query(0.27,1.26,2.25,3.24,3.46,3.1b4 epoch 93) v2
|
|
-54> 2013-08-15 20:50:33.849822 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.849822, event: started, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-53> 2013-08-15 20:50:33.849818 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.2 9.4.196.1:6807/6887 4 ==== pg_query(3.11a epoch 93) v2 ==== 137+0+0 (1890217985 0 0) 0x29d5c20 con 0x2bef760
|
|
-52> 2013-08-15 20:50:33.849846 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:33.849846, event: done, request: pg_notify(3.24a(12) epoch 93) v4
|
|
-51> 2013-08-15 20:50:33.849857 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.849857, event: started, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-50> 2013-08-15 20:50:33.850035 7fbe31e69700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6807/6887 -- pg_notify(0.6,1.5,2.4,3.3 epoch 93) v4 -- ?+0 0x29d3880 con 0x2bef760
|
|
-49> 2013-08-15 20:50:33.850048 7fbe31e69700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:33.850048, event: done, request: pg_query(0.6,0.1f,0.28,0.29,1.5,1.1e,1.27,1.28,2.4,2.1d,2.26,2.27,3.3,3.25,3.26,3.f9,3.11a,3.187,3.18b,3.1bd,3.1c4,3.1ee epoch 93) v2
|
|
-48> 2013-08-15 20:50:33.850099 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.849746, event: header_read, request: pg_query(3.11a epoch 93) v2
|
|
-47> 2013-08-15 20:50:33.850114 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.849748, event: throttled, request: pg_query(3.11a epoch 93) v2
|
|
-46> 2013-08-15 20:50:33.850118 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.849789, event: all_read, request: pg_query(3.11a epoch 93) v2
|
|
-45> 2013-08-15 20:50:33.850121 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.849840, event: dispatched, request: pg_query(3.11a epoch 93) v2
|
|
-44> 2013-08-15 20:50:33.850126 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.850126, event: waiting_for_osdmap, request: pg_query(3.11a epoch 93) v2
|
|
-43> 2013-08-15 20:50:33.850132 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.850132, event: started, request: pg_query(3.11a epoch 93) v2
|
|
-42> 2013-08-15 20:50:33.850146 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:33.850146, event: done, request: pg_query(3.11a epoch 93) v2
|
|
-41> 2013-08-15 20:50:33.850448 7fbe2d660700 1 osd.7 pg_epoch: 93 pg[3.5b( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 93/93/56) [0,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-40> 2013-08-15 20:50:33.850498 7fbe2ce5f700 1 osd.7 pg_epoch: 93 pg[3.1e2( v 63'18 (0'0,63'18] local-les=62 n=6 ec=56 les/c 62/62 93/93/56) [1,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-39> 2013-08-15 20:50:33.851394 7fbe2d660700 1 osd.7 pg_epoch: 93 pg[3.1d4( v 63'15 (0'0,63'15] local-les=62 n=3 ec=56 les/c 62/62 93/93/56) [0,7]/[0,10] r=-1 lpr=93 pi=56-92/15 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-38> 2013-08-15 20:50:33.851478 7fbe2ce5f700 1 osd.7 pg_epoch: 93 pg[3.5e( v 63'10 (0'0,63'10] local-les=62 n=0 ec=56 les/c 62/62 93/93/56) [4,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-37> 2013-08-15 20:50:33.852321 7fbe2d660700 1 osd.7 pg_epoch: 93 pg[3.4c( v 63'3 (0'0,63'3] local-les=62 n=1 ec=56 les/c 62/62 93/93/56) [0,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-36> 2013-08-15 20:50:33.852394 7fbe2ce5f700 1 osd.7 pg_epoch: 93 pg[3.11f( v 63'9 (0'0,63'9] local-les=62 n=3 ec=56 les/c 62/62 93/93/56) [1,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-35> 2013-08-15 20:50:33.852457 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e8(15),3.1e2(16),3.11f(16) epoch 93) v4 -- ?+0 0x2c4c380 con 0x2bef340
|
|
-34> 2013-08-15 20:50:33.852480 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ee(15) epoch 93) v4 -- ?+0 0x2c4c1c0 con 0x2bef760
|
|
-33> 2013-08-15 20:50:33.852499 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6813/8739 -- pg_notify(3.5e(16) epoch 93) v4 -- ?+0 0x2c4c000 con 0x2bef4a0
|
|
-32> 2013-08-15 20:50:33.852658 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6807/6887 -- pg_log(3.11a epoch 93 query_epoch 93) v3 -- ?+0 0x29dc100 con 0x2bef760
|
|
-31> 2013-08-15 20:50:33.852755 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6801/5083 -- pg_notify(3.1e2(16),3.11f(16),3.1e8(15) epoch 93) v4 -- ?+0 0x2c4ca80 con 0x2bef340
|
|
-30> 2013-08-15 20:50:33.852771 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6807/6887 -- pg_notify(3.11a(16),3.1ee(15) epoch 93) v4 -- ?+0 0x2c4c8c0 con 0x2bef760
|
|
-29> 2013-08-15 20:50:33.852786 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6813/8739 -- pg_notify(3.5e(16) epoch 93) v4 -- ?+0 0x2c4c700 con 0x2bef4a0
|
|
-28> 2013-08-15 20:50:33.852790 7fbe2d660700 1 osd.7 pg_epoch: 93 pg[3.10b( v 63'11 (0'0,63'11] local-les=62 n=9 ec=56 les/c 62/62 93/93/56) [4,7]/[4,5] r=-1 lpr=93 pi=56-92/15 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-27> 2013-08-15 20:50:33.853360 7fbe2d660700 1 osd.7 pg_epoch: 93 pg[3.1c4( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 93/93/56) [2,7] r=1 lpr=93 pi=56-92/16 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-26> 2013-08-15 20:50:33.853395 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.4 9.4.196.1:6813/8739 4 ==== pg_query(3.5e epoch 93) v2 ==== 137+0+0 (1069427096 0 0) 0x29d4d20 con 0x2bef4a0
|
|
-25> 2013-08-15 20:50:33.853418 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853334, event: header_read, request: pg_query(3.5e epoch 93) v2
|
|
-24> 2013-08-15 20:50:33.853430 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853335, event: throttled, request: pg_query(3.5e epoch 93) v2
|
|
-23> 2013-08-15 20:50:33.853440 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853373, event: all_read, request: pg_query(3.5e epoch 93) v2
|
|
-22> 2013-08-15 20:50:33.853449 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853414, event: dispatched, request: pg_query(3.5e epoch 93) v2
|
|
-21> 2013-08-15 20:50:33.853459 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853459, event: waiting_for_osdmap, request: pg_query(3.5e epoch 93) v2
|
|
-20> 2013-08-15 20:50:33.853471 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853471, event: started, request: pg_query(3.5e epoch 93) v2
|
|
-19> 2013-08-15 20:50:33.853496 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:33.853496, event: done, request: pg_query(3.5e epoch 93) v2
|
|
-18> 2013-08-15 20:50:33.853513 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6813/8739 -- pg_log(3.5e epoch 93 query_epoch 93) v3 -- ?+0 0x29dcc00 con 0x2bef4a0
|
|
-17> 2013-08-15 20:50:33.853516 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.1 9.4.196.1:6801/5083 5 ==== pg_query(3.1e2 epoch 93) v2 ==== 137+0+0 (1587339146 0 0) 0x29d5860 con 0x2bef340
|
|
-16> 2013-08-15 20:50:33.853530 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853341, event: header_read, request: pg_query(3.1e2 epoch 93) v2
|
|
-15> 2013-08-15 20:50:33.853542 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853342, event: throttled, request: pg_query(3.1e2 epoch 93) v2
|
|
-14> 2013-08-15 20:50:33.853551 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853398, event: all_read, request: pg_query(3.1e2 epoch 93) v2
|
|
-13> 2013-08-15 20:50:33.853559 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853528, event: dispatched, request: pg_query(3.1e2 epoch 93) v2
|
|
-12> 2013-08-15 20:50:33.853569 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853569, event: waiting_for_osdmap, request: pg_query(3.1e2 epoch 93) v2
|
|
-11> 2013-08-15 20:50:33.853580 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853580, event: started, request: pg_query(3.1e2 epoch 93) v2
|
|
-10> 2013-08-15 20:50:33.853618 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:33.853618, event: done, request: pg_query(3.1e2 epoch 93) v2
|
|
-9> 2013-08-15 20:50:33.853631 7fbe2ce5f700 1 -- 9.4.196.3:6839/9437 --> 9.4.196.1:6801/5083 -- pg_log(3.1e2 epoch 93 query_epoch 93) v3 -- ?+0 0x29dc680 con 0x2bef340
|
|
-8> 2013-08-15 20:50:33.853635 7fbe30e67700 1 -- 9.4.196.3:6839/9437 <== osd.1 9.4.196.1:6801/5083 6 ==== pg_query(3.11f epoch 93) v2 ==== 137+0+0 (2742127544 0 0) 0x29f85a0 con 0x2bef340
|
|
-7> 2013-08-15 20:50:33.853651 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853504, event: header_read, request: pg_query(3.11f epoch 93) v2
|
|
-6> 2013-08-15 20:50:33.853661 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853505, event: throttled, request: pg_query(3.11f epoch 93) v2
|
|
-5> 2013-08-15 20:50:33.853670 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853538, event: all_read, request: pg_query(3.11f epoch 93) v2
|
|
-4> 2013-08-15 20:50:33.853678 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853649, event: dispatched, request: pg_query(3.11f epoch 93) v2
|
|
-3> 2013-08-15 20:50:33.853687 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853687, event: waiting_for_osdmap, request: pg_query(3.11f epoch 93) v2
|
|
-2> 2013-08-15 20:50:33.853698 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853698, event: started, request: pg_query(3.11f epoch 93) v2
|
|
-1> 2013-08-15 20:50:33.853718 7fbe30e67700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:33.853718, event: done, request: pg_query(3.11f epoch 93) v2
|
|
0> 2013-08-15 20:50:33.853715 7fbe37e75700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fbe37e75700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fbe3ee85bd0]
|
|
3: (gsignal()+0x37) [0x7fbe3d06d037]
|
|
4: (abort()+0x148) [0x7fbe3d070698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fbe3d979e8d]
|
|
6: (()+0x5ef76) [0x7fbe3d977f76]
|
|
7: (()+0x5efa3) [0x7fbe3d977fa3]
|
|
8: (()+0x5f1de) [0x7fbe3d9781de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fbe3ee7df8e]
|
|
14: (clone()+0x6d) [0x7fbe3d12fe1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:34.013092 7f106ea027c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9706
|
|
2013-08-15 20:50:34.013621 7f106ea027c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:34.046455 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:34.046476 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:34.046798 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:34.064405 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:34.064530 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:34.088882 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:34.089677 7f106ea027c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:34.089694 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:34.166518 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:34.166924 7f106ea027c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
2013-08-15 20:50:34.196466 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:34.196480 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:34.196849 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
2013-08-15 20:50:34.208443 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:34.208518 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
2013-08-15 20:50:34.238773 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:34.239305 7f106ea027c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:34.239332 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:34.315900 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:35.773706 7f10673d6700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:35.773721 7f10673d6700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
2013-08-15 20:50:35.774790 7f10673d6700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f10673d6700 time 2013-08-15 20:50:35.773727
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f106e3def8e]
|
|
5: (clone()+0x6d) [0x7f106c690e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-218> 2013-08-15 20:50:34.011781 7f106ea027c0 5 asok(0x19640e0) register_command perfcounters_dump hook 0x195e010
|
|
-217> 2013-08-15 20:50:34.011809 7f106ea027c0 5 asok(0x19640e0) register_command 1 hook 0x195e010
|
|
-216> 2013-08-15 20:50:34.011812 7f106ea027c0 5 asok(0x19640e0) register_command perf dump hook 0x195e010
|
|
-215> 2013-08-15 20:50:34.011819 7f106ea027c0 5 asok(0x19640e0) register_command perfcounters_schema hook 0x195e010
|
|
-214> 2013-08-15 20:50:34.011824 7f106ea027c0 5 asok(0x19640e0) register_command 2 hook 0x195e010
|
|
-213> 2013-08-15 20:50:34.011826 7f106ea027c0 5 asok(0x19640e0) register_command perf schema hook 0x195e010
|
|
-212> 2013-08-15 20:50:34.011833 7f106ea027c0 5 asok(0x19640e0) register_command config show hook 0x195e010
|
|
-211> 2013-08-15 20:50:34.011837 7f106ea027c0 5 asok(0x19640e0) register_command config set hook 0x195e010
|
|
-210> 2013-08-15 20:50:34.011842 7f106ea027c0 5 asok(0x19640e0) register_command log flush hook 0x195e010
|
|
-209> 2013-08-15 20:50:34.011846 7f106ea027c0 5 asok(0x19640e0) register_command log dump hook 0x195e010
|
|
-208> 2013-08-15 20:50:34.011850 7f106ea027c0 5 asok(0x19640e0) register_command log reopen hook 0x195e010
|
|
-207> 2013-08-15 20:50:34.013092 7f106ea027c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9706
|
|
-206> 2013-08-15 20:50:34.013621 7f106ea027c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-205> 2013-08-15 20:50:34.013749 7f106ea027c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6836/9706 need_addr=1
|
|
-204> 2013-08-15 20:50:34.013779 7f106ea027c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-203> 2013-08-15 20:50:34.013788 7f106ea027c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6838/9706 need_addr=0
|
|
-202> 2013-08-15 20:50:34.013817 7f106ea027c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-201> 2013-08-15 20:50:34.013823 7f106ea027c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6841/9706 need_addr=0
|
|
-200> 2013-08-15 20:50:34.015628 7f106ea027c0 5 asok(0x19640e0) init /var/run/ceph/ceph-osd.7.asok
|
|
-199> 2013-08-15 20:50:34.015643 7f106ea027c0 5 asok(0x19640e0) bind_and_listen /var/run/ceph/ceph-osd.7.asok
|
|
-198> 2013-08-15 20:50:34.015673 7f106ea027c0 5 asok(0x19640e0) register_command 0 hook 0x195c0b8
|
|
-197> 2013-08-15 20:50:34.015679 7f106ea027c0 5 asok(0x19640e0) register_command version hook 0x195c0b8
|
|
-196> 2013-08-15 20:50:34.015684 7f106ea027c0 5 asok(0x19640e0) register_command git_version hook 0x195c0b8
|
|
-195> 2013-08-15 20:50:34.015689 7f106ea027c0 5 asok(0x19640e0) register_command help hook 0x195e0d0
|
|
-194> 2013-08-15 20:50:34.015715 7f106a3dc700 5 asok(0x19640e0) entry start
|
|
-193> 2013-08-15 20:50:34.046455 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-192> 2013-08-15 20:50:34.046476 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-191> 2013-08-15 20:50:34.046798 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-190> 2013-08-15 20:50:34.064405 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-189> 2013-08-15 20:50:34.064530 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-188> 2013-08-15 20:50:34.088882 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-187> 2013-08-15 20:50:34.089651 7f106ea027c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-186> 2013-08-15 20:50:34.089677 7f106ea027c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-185> 2013-08-15 20:50:34.089694 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-184> 2013-08-15 20:50:34.095427 7f106ea027c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-183> 2013-08-15 20:50:34.099514 7f106ea027c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-182> 2013-08-15 20:50:34.103510 7f106ea027c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-181> 2013-08-15 20:50:34.107463 7f106ea027c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-180> 2013-08-15 20:50:34.111358 7f106ea027c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-179> 2013-08-15 20:50:34.115584 7f106ea027c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-178> 2013-08-15 20:50:34.119478 7f106ea027c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-177> 2013-08-15 20:50:34.123504 7f106ea027c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-176> 2013-08-15 20:50:34.127414 7f106ea027c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-175> 2013-08-15 20:50:34.131390 7f106ea027c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-174> 2013-08-15 20:50:34.135230 7f106ea027c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-173> 2013-08-15 20:50:34.139121 7f106ea027c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-172> 2013-08-15 20:50:34.143030 7f106ea027c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-171> 2013-08-15 20:50:34.146974 7f106ea027c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-170> 2013-08-15 20:50:34.150814 7f106ea027c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-169> 2013-08-15 20:50:34.154668 7f106ea027c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-168> 2013-08-15 20:50:34.158655 7f106ea027c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-167> 2013-08-15 20:50:34.162534 7f106ea027c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-166> 2013-08-15 20:50:34.166426 7f106ea027c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-165> 2013-08-15 20:50:34.166458 7f106ea027c0 2 journal No further valid entries found, journal is most likely valid
|
|
-164> 2013-08-15 20:50:34.166463 7f106ea027c0 2 journal No further valid entries found, journal is most likely valid
|
|
-163> 2013-08-15 20:50:34.166476 7f106ea027c0 3 journal journal_replay: end of journal, done.
|
|
-162> 2013-08-15 20:50:34.166518 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-161> 2013-08-15 20:50:34.166818 7f10673d6700 1 FileStore::op_tp worker finish
|
|
-160> 2013-08-15 20:50:34.166856 7f1067bd7700 1 FileStore::op_tp worker finish
|
|
-159> 2013-08-15 20:50:34.166924 7f106ea027c0 1 journal close /var/lib/ceph/osd/ceph-7/journal
|
|
-158> 2013-08-15 20:50:34.167164 7f106ea027c0 10 monclient(hunting): build_initial_monmap
|
|
-157> 2013-08-15 20:50:34.167214 7f106ea027c0 5 adding auth protocol: cephx
|
|
-156> 2013-08-15 20:50:34.167221 7f106ea027c0 5 adding auth protocol: cephx
|
|
-155> 2013-08-15 20:50:34.167307 7f106ea027c0 1 -- 0.0.0.0:6836/9706 messenger.start
|
|
-154> 2013-08-15 20:50:34.167335 7f106ea027c0 1 -- :/0 messenger.start
|
|
-153> 2013-08-15 20:50:34.167346 7f106ea027c0 1 -- 9.4.196.3:6841/9706 messenger.start
|
|
-152> 2013-08-15 20:50:34.167357 7f106ea027c0 1 -- 9.4.196.3:6838/9706 messenger.start
|
|
-151> 2013-08-15 20:50:34.167462 7f106ea027c0 2 osd.7 0 mounting /var/lib/ceph/osd/ceph-7 /var/lib/ceph/osd/ceph-7/journal
|
|
-150> 2013-08-15 20:50:34.196466 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is supported and appears to work
|
|
-149> 2013-08-15 20:50:34.196480 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-148> 2013-08-15 20:50:34.196849 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount did NOT detect btrfs
|
|
-147> 2013-08-15 20:50:34.208443 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-146> 2013-08-15 20:50:34.208518 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount found snaps <>
|
|
-145> 2013-08-15 20:50:34.238773 7f106ea027c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-144> 2013-08-15 20:50:34.239273 7f106ea027c0 2 journal open /var/lib/ceph/osd/ceph-7/journal fsid 371849f3-3106-4663-b1fe-50e886b026a2 fs_op_seq 2496
|
|
-143> 2013-08-15 20:50:34.239305 7f106ea027c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-142> 2013-08-15 20:50:34.239332 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-141> 2013-08-15 20:50:34.243620 7f106ea027c0 2 journal read_entry 3415433216 : seq 2478 4195925 bytes
|
|
-140> 2013-08-15 20:50:34.247684 7f106ea027c0 2 journal read_entry 3419635712 : seq 2479 4195925 bytes
|
|
-139> 2013-08-15 20:50:34.251733 7f106ea027c0 2 journal read_entry 3423838208 : seq 2480 4195925 bytes
|
|
-138> 2013-08-15 20:50:34.255803 7f106ea027c0 2 journal read_entry 3428040704 : seq 2481 4195919 bytes
|
|
-137> 2013-08-15 20:50:34.259820 7f106ea027c0 2 journal read_entry 3432243200 : seq 2482 4195925 bytes
|
|
-136> 2013-08-15 20:50:34.263842 7f106ea027c0 2 journal read_entry 3436445696 : seq 2483 4195919 bytes
|
|
-135> 2013-08-15 20:50:34.267882 7f106ea027c0 2 journal read_entry 3440648192 : seq 2484 4195925 bytes
|
|
-134> 2013-08-15 20:50:34.271922 7f106ea027c0 2 journal read_entry 3444850688 : seq 2485 4195925 bytes
|
|
-133> 2013-08-15 20:50:34.275954 7f106ea027c0 2 journal read_entry 3449053184 : seq 2486 4195925 bytes
|
|
-132> 2013-08-15 20:50:34.279961 7f106ea027c0 2 journal read_entry 3453255680 : seq 2487 4195925 bytes
|
|
-131> 2013-08-15 20:50:34.283974 7f106ea027c0 2 journal read_entry 3457458176 : seq 2488 4195925 bytes
|
|
-130> 2013-08-15 20:50:34.287982 7f106ea027c0 2 journal read_entry 3461660672 : seq 2489 4195925 bytes
|
|
-129> 2013-08-15 20:50:34.292026 7f106ea027c0 2 journal read_entry 3465863168 : seq 2490 4195925 bytes
|
|
-128> 2013-08-15 20:50:34.296026 7f106ea027c0 2 journal read_entry 3470065664 : seq 2491 4195925 bytes
|
|
-127> 2013-08-15 20:50:34.300008 7f106ea027c0 2 journal read_entry 3474268160 : seq 2492 4195919 bytes
|
|
-126> 2013-08-15 20:50:34.303980 7f106ea027c0 2 journal read_entry 3478470656 : seq 2493 4195919 bytes
|
|
-125> 2013-08-15 20:50:34.307948 7f106ea027c0 2 journal read_entry 3482673152 : seq 2494 4195919 bytes
|
|
-124> 2013-08-15 20:50:34.311902 7f106ea027c0 2 journal read_entry 3486875648 : seq 2495 4195919 bytes
|
|
-123> 2013-08-15 20:50:34.315851 7f106ea027c0 2 journal read_entry 3491078144 : seq 2496 4195919 bytes
|
|
-122> 2013-08-15 20:50:34.315863 7f106ea027c0 2 journal No further valid entries found, journal is most likely valid
|
|
-121> 2013-08-15 20:50:34.315868 7f106ea027c0 2 journal No further valid entries found, journal is most likely valid
|
|
-120> 2013-08-15 20:50:34.315869 7f106ea027c0 3 journal journal_replay: end of journal, done.
|
|
-119> 2013-08-15 20:50:34.315900 7f106ea027c0 1 journal _open /var/lib/ceph/osd/ceph-7/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-118> 2013-08-15 20:50:34.316090 7f106ea027c0 2 osd.7 0 boot
|
|
-117> 2013-08-15 20:50:34.340973 7f106ea027c0 2 osd.7 63 superblock: i am osd.7
|
|
-116> 2013-08-15 20:50:34.341013 7f106ea027c0 1 accepter.accepter.start
|
|
-115> 2013-08-15 20:50:34.341077 7f106ea027c0 1 accepter.accepter.start
|
|
-114> 2013-08-15 20:50:34.341156 7f106ea027c0 1 accepter.accepter.start
|
|
-113> 2013-08-15 20:50:34.341245 7f106ea027c0 10 monclient(hunting): init
|
|
-112> 2013-08-15 20:50:34.341270 7f106ea027c0 5 adding auth protocol: cephx
|
|
-111> 2013-08-15 20:50:34.341275 7f106ea027c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-110> 2013-08-15 20:50:34.341306 7f105dac2700 1 -- 0.0.0.0:6836/9706 >> :/0 pipe(0x1baa280 sd=29 :6836 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.1:35042/0
|
|
-109> 2013-08-15 20:50:34.341286 7f105dbc3700 1 -- 0.0.0.0:6836/9706 >> :/0 pipe(0x1baa500 sd=28 :6836 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.1:35041/0
|
|
-108> 2013-08-15 20:50:34.341303 7f106e996700 1 -- 0.0.0.0:6836/9706 >> :/0 pipe(0x1baa000 sd=27 :6836 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.3:47855/0
|
|
-107> 2013-08-15 20:50:34.341381 7f105d9c1700 1 -- 0.0.0.0:6836/9706 >> :/0 pipe(0x1baac80 sd=30 :6836 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:35045/0
|
|
-106> 2013-08-15 20:50:34.341417 7f105d7bf700 1 -- 0.0.0.0:6836/9706 >> :/0 pipe(0x1baa780 sd=31 :6836 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:35049/0
|
|
-105> 2013-08-15 20:50:34.341452 7f105d8c0700 1 -- 0.0.0.0:6836/9706 >> :/0 pipe(0x1baaa00 sd=32 :6836 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:35044/0
|
|
-104> 2013-08-15 20:50:34.341475 7f106ea027c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-7/keyring
|
|
-103> 2013-08-15 20:50:34.341651 7f106ea027c0 5 asok(0x19640e0) register_command dump_ops_in_flight hook 0x195e770
|
|
-102> 2013-08-15 20:50:34.341661 7f106ea027c0 5 asok(0x19640e0) register_command dump_historic_ops hook 0x195e770
|
|
-101> 2013-08-15 20:50:34.341664 7f106ea027c0 5 asok(0x19640e0) register_command dump_op_pq_state hook 0x195e770
|
|
-100> 2013-08-15 20:50:34.341673 7f106ea027c0 5 asok(0x19640e0) register_command setomapval hook 0x1b5aae0
|
|
-99> 2013-08-15 20:50:34.341677 7f106ea027c0 5 asok(0x19640e0) register_command rmomapkey hook 0x1b5aae0
|
|
-98> 2013-08-15 20:50:34.341682 7f106ea027c0 5 asok(0x19640e0) register_command setomapheader hook 0x1b5aae0
|
|
-97> 2013-08-15 20:50:34.341686 7f106ea027c0 5 asok(0x19640e0) register_command getomap hook 0x1b5aae0
|
|
-96> 2013-08-15 20:50:34.341689 7f106ea027c0 5 asok(0x19640e0) register_command truncobj hook 0x1b5aae0
|
|
-95> 2013-08-15 20:50:34.341693 7f106ea027c0 5 asok(0x19640e0) register_command injectdataerr hook 0x1b5aae0
|
|
-94> 2013-08-15 20:50:34.341697 7f106ea027c0 5 asok(0x19640e0) register_command injectmdataerr hook 0x1b5aae0
|
|
-93> 2013-08-15 20:50:34.341690 7f1059eb7700 5 osd.7 63 heartbeat: osd_stat(734 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-92> 2013-08-15 20:50:34.341739 7f106ea027c0 10 monclient(hunting): _reopen_session
|
|
-91> 2013-08-15 20:50:34.341763 7f106ea027c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x1baec60 addr 9.4.196.1:6789/0
|
|
-90> 2013-08-15 20:50:34.341775 7f106ea027c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-89> 2013-08-15 20:50:34.341781 7f106ea027c0 1 -- 0.0.0.0:6836/9706 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1970b40 con 0x1baec60
|
|
-88> 2013-08-15 20:50:34.341793 7f106ea027c0 10 monclient(hunting): renew_subs
|
|
-87> 2013-08-15 20:50:34.342118 7f105d7bf700 1 -- 9.4.196.3:6836/9706 learned my addr 9.4.196.3:6836/9706
|
|
-86> 2013-08-15 20:50:34.342280 7f10613ca700 10 monclient(hunting): renew_subs
|
|
-85> 2013-08-15 20:50:34.342687 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x19961e0 con 0x1baec60
|
|
-84> 2013-08-15 20:50:34.342702 7f10613ca700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-83> 2013-08-15 20:50:34.342713 7f10613ca700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-82> 2013-08-15 20:50:34.342717 7f10613ca700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-81> 2013-08-15 20:50:34.342746 7f10613ca700 1 monclient(hunting): found mon.zc2store
|
|
-80> 2013-08-15 20:50:34.342754 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (3844928420 0 0) 0x1bba200 con 0x1baec60
|
|
-79> 2013-08-15 20:50:34.342774 7f10613ca700 10 monclient: my global_id is 4413
|
|
-78> 2013-08-15 20:50:34.342867 7f10613ca700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-77> 2013-08-15 20:50:34.342876 7f10613ca700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1971d40 con 0x1baec60
|
|
-76> 2013-08-15 20:50:34.343342 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (295241349 0 0) 0x1bba000 con 0x1baec60
|
|
-75> 2013-08-15 20:50:34.343417 7f10613ca700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-74> 2013-08-15 20:50:34.343428 7f10613ca700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1bbe240 con 0x1baec60
|
|
-73> 2013-08-15 20:50:34.343948 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (2372738017 0 0) 0x1bba400 con 0x1baec60
|
|
-72> 2013-08-15 20:50:34.343995 7f10613ca700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-71> 2013-08-15 20:50:34.344013 7f10613ca700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1994000 con 0x1baec60
|
|
-70> 2013-08-15 20:50:34.344025 7f10613ca700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-69> 2013-08-15 20:50:34.344028 7f10613ca700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x1994380 con 0x1baec60
|
|
-68> 2013-08-15 20:50:34.344038 7f10613ca700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:04.344037)
|
|
-67> 2013-08-15 20:50:34.344045 7f10613ca700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-66> 2013-08-15 20:50:34.344048 7f10613ca700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1bbe000 con 0x1baec60
|
|
-65> 2013-08-15 20:50:34.344085 7f106ea027c0 5 monclient: authenticate success, global_id 4413
|
|
-64> 2013-08-15 20:50:34.344098 7f106ea027c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:04.344097)
|
|
-63> 2013-08-15 20:50:34.344430 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x19965a0 con 0x1baec60
|
|
-62> 2013-08-15 20:50:34.344444 7f10613ca700 10 monclient: handle_monmap mon_map v1
|
|
-61> 2013-08-15 20:50:34.344452 7f10613ca700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-60> 2013-08-15 20:50:34.344454 7f10613ca700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-59> 2013-08-15 20:50:34.344467 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1994000 con 0x1baec60
|
|
-58> 2013-08-15 20:50:34.344486 7f10613ca700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:34.341796 renew after 2013-08-15 20:53:04.341796
|
|
-57> 2013-08-15 20:50:34.344497 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x19963c0 con 0x1baec60
|
|
-56> 2013-08-15 20:50:34.344504 7f10613ca700 10 monclient: handle_monmap mon_map v1
|
|
-55> 2013-08-15 20:50:34.344509 7f10613ca700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-54> 2013-08-15 20:50:34.344512 7f10613ca700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-53> 2013-08-15 20:50:34.344525 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1994540 con 0x1baec60
|
|
-52> 2013-08-15 20:50:34.344532 7f10613ca700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-51> 2013-08-15 20:50:34.344547 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (2989636582 0 0) 0x1bba600 con 0x1baec60
|
|
-50> 2013-08-15 20:50:34.344585 7f10613ca700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:04.344584)
|
|
-49> 2013-08-15 20:50:34.344641 7f106ea027c0 10 monclient: wait_auth_rotating done
|
|
-48> 2013-08-15 20:50:34.345218 7f106ea027c0 10 monclient: get_version osdmap
|
|
-47> 2013-08-15 20:50:34.345223 7f106ea027c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-46> 2013-08-15 20:50:34.345231 7f106ea027c0 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x19cc9c0 con 0x1baec60
|
|
-45> 2013-08-15 20:50:34.345554 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=93) v2 ==== 24+0+0 (2145240 0 0) 0x1994380 con 0x1baec60
|
|
-44> 2013-08-15 20:50:34.345765 7f105cebd700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-43> 2013-08-15 20:50:34.345781 7f105cebd700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- osd_boot(osd.7 booted 0 v63) v3 -- ?+0 0x1bca000 con 0x1baec60
|
|
-42> 2013-08-15 20:50:34.741693 7f1057fad700 1 -- 9.4.196.3:6836/9706 >> :/0 pipe(0x1bab680 sd=34 :6836 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:35097/0
|
|
-41> 2013-08-15 20:50:34.741758 7f1057eac700 1 -- 9.4.196.3:6836/9706 >> :/0 pipe(0x1bab400 sd=35 :6836 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:35093/0
|
|
-40> 2013-08-15 20:50:34.741777 7f10580ae700 1 -- 9.4.196.3:6836/9706 >> :/0 pipe(0x1bab900 sd=33 :6836 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:47874/0
|
|
-39> 2013-08-15 20:50:34.741812 7f1057dab700 1 -- 9.4.196.3:6836/9706 >> :/0 pipe(0x1bab180 sd=36 :6836 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:35095/0
|
|
-38> 2013-08-15 20:50:34.741815 7f1057caa700 1 -- 9.4.196.3:6836/9706 >> :/0 pipe(0x1babb80 sd=37 :6836 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:35098/0
|
|
-37> 2013-08-15 20:50:34.741934 7f1057ba9700 1 -- 9.4.196.3:6836/9706 >> :/0 pipe(0x1be0c80 sd=38 :6836 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:35094/0
|
|
-36> 2013-08-15 20:50:35.760896 7f1057ba9700 1 -- 9.4.196.3:6838/9706 >> :/0 pipe(0x1babb80 sd=33 :6838 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:46874/0
|
|
-35> 2013-08-15 20:50:35.761566 7f10603c8700 1 -- 9.4.196.3:6838/9706 <== osd.6 9.4.196.3:6801/4121 1 ==== pg_notify(3.175(14) epoch 95) v4 ==== 1146+0+0 (3066341953 0 0) 0x1994700 con 0x1bae580
|
|
-34> 2013-08-15 20:50:35.761607 7f10603c8700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.761456, event: header_read, request: pg_notify(3.175(14) epoch 95) v4
|
|
-33> 2013-08-15 20:50:35.761622 7f10603c8700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.761459, event: throttled, request: pg_notify(3.175(14) epoch 95) v4
|
|
-32> 2013-08-15 20:50:35.761632 7f10603c8700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.761548, event: all_read, request: pg_notify(3.175(14) epoch 95) v4
|
|
-31> 2013-08-15 20:50:35.761643 7f10603c8700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.761593, event: dispatched, request: pg_notify(3.175(14) epoch 95) v4
|
|
-30> 2013-08-15 20:50:35.761654 7f10603c8700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.761654, event: waiting_for_osdmap, request: pg_notify(3.175(14) epoch 95) v4
|
|
-29> 2013-08-15 20:50:35.761673 7f10603c8700 10 monclient: renew_subs
|
|
-28> 2013-08-15 20:50:35.761680 7f10603c8700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-27> 2013-08-15 20:50:35.761691 7f10603c8700 1 -- 9.4.196.3:6836/9706 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x19948c0 con 0x1baec60
|
|
-26> 2013-08-15 20:50:35.761711 7f10603c8700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:35.761711, event: wait for new map, request: pg_notify(3.175(14) epoch 95) v4
|
|
-25> 2013-08-15 20:50:35.763330 7f10613ca700 1 -- 9.4.196.3:6836/9706 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..94 src has 1..94) v3 ==== 30872+0+0 (824186616 0 0) 0x1bbf440 con 0x1baec60
|
|
-24> 2013-08-15 20:50:35.763358 7f10613ca700 3 osd.7 63 handle_osd_map epochs [64,94], i have 63, src has [1,94]
|
|
-23> 2013-08-15 20:50:35.765289 7f10603c8700 1 -- 9.4.196.3:6838/9706 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(94..95 src has 1..95) v3 ==== 1978+0+0 (4104616668 0 0) 0x1bbf680 con 0x1bae420
|
|
-22> 2013-08-15 20:50:35.766154 7f1057eac700 1 -- 9.4.196.3:6838/9706 >> :/0 pipe(0x1bab180 sd=34 :6838 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:46876/0
|
|
-21> 2013-08-15 20:50:35.769346 7f1057fad700 1 -- 9.4.196.3:6841/9706 >> :/0 pipe(0x1bab680 sd=35 :6841 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:44371/0
|
|
-20> 2013-08-15 20:50:35.771123 7f1057aa8700 1 -- 9.4.196.3:6841/9706 >> :/0 pipe(0x1be0c80 sd=36 :6841 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:44372/0
|
|
-19> 2013-08-15 20:50:35.772236 7f10578a6700 1 -- 9.4.196.3:6841/9706 >> :/0 pipe(0x1bab900 sd=37 :6841 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:44373/0
|
|
-18> 2013-08-15 20:50:35.772778 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6815/4367 -- pipe dne
|
|
-17> 2013-08-15 20:50:35.772789 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-16> 2013-08-15 20:50:35.772796 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-15> 2013-08-15 20:50:35.772819 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-14> 2013-08-15 20:50:35.772840 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-13> 2013-08-15 20:50:35.772847 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-12> 2013-08-15 20:50:35.772883 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6810/4119 -- pipe dne
|
|
-11> 2013-08-15 20:50:35.772901 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-10> 2013-08-15 20:50:35.772914 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6811/8147 -- pipe dne
|
|
-9> 2013-08-15 20:50:35.772930 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6811/8539 -- pipe dne
|
|
-8> 2013-08-15 20:50:35.772940 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-7> 2013-08-15 20:50:35.772954 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6828/8914 -- pipe dne
|
|
-6> 2013-08-15 20:50:35.772964 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6813/8749 -- pipe dne
|
|
-5> 2013-08-15 20:50:35.772978 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6830/9101 -- pipe dne
|
|
-4> 2013-08-15 20:50:35.772987 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6809/9079 -- pipe dne
|
|
-3> 2013-08-15 20:50:35.772999 7f10613ca700 1 -- 9.4.196.3:6838/9706 mark_down 9.4.196.3:6836/9435 -- pipe dne
|
|
-2> 2013-08-15 20:50:35.773706 7f10673d6700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-1> 2013-08-15 20:50:35.773721 7f10673d6700 -1 journal FileJournal::do_write: write_bl(pos=3491078144) failed
|
|
0> 2013-08-15 20:50:35.774790 7f10673d6700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f10673d6700 time 2013-08-15 20:50:35.773727
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f106e3def8e]
|
|
5: (clone()+0x6d) [0x7f106c690e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:35.778825 7f10673d6700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f10673d6700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f106e3e6bd0]
|
|
3: (gsignal()+0x37) [0x7f106c5ce037]
|
|
4: (abort()+0x148) [0x7f106c5d1698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f106cedae8d]
|
|
6: (()+0x5ef76) [0x7f106ced8f76]
|
|
7: (()+0x5efa3) [0x7f106ced8fa3]
|
|
8: (()+0x5f1de) [0x7f106ced91de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f106e3def8e]
|
|
14: (clone()+0x6d) [0x7f106c690e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-1> 2013-08-15 20:50:35.777212 7f10576a4700 1 -- 9.4.196.3:6838/9706 >> :/0 pipe(0x1bab400 sd=38 :6838 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.3:46883/0
|
|
0> 2013-08-15 20:50:35.778825 7f10673d6700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f10673d6700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f106e3e6bd0]
|
|
3: (gsignal()+0x37) [0x7f106c5ce037]
|
|
4: (abort()+0x148) [0x7f106c5d1698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f106cedae8d]
|
|
6: (()+0x5ef76) [0x7f106ced8f76]
|
|
7: (()+0x5efa3) [0x7f106ced8fa3]
|
|
8: (()+0x5f1de) [0x7f106ced91de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f106e3def8e]
|
|
14: (clone()+0x6d) [0x7f106c690e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.7.log
|
|
--- end dump of recent events ---
|