|
2013-08-15 06:47:28.611386 7fab304c07c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3462
|
|
2013-08-15 06:47:28.612116 7fab304c07c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.615278 7fab304c07c0 0 filestore(/var/lib/ceph/osd/ceph-8) lock_fsid failed to lock /var/lib/ceph/osd/ceph-8/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.615298 7fab304c07c0 -1 filestore(/var/lib/ceph/osd/ceph-8) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.615338 7fab304c07c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-8: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.656492 7fba4ec9d7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3723
|
|
2013-08-15 06:47:28.657397 7fba4ec9d7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.660449 7fba4ec9d7c0 0 filestore(/var/lib/ceph/osd/ceph-8) lock_fsid failed to lock /var/lib/ceph/osd/ceph-8/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.660472 7fba4ec9d7c0 -1 filestore(/var/lib/ceph/osd/ceph-8) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.660506 7fba4ec9d7c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-8: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.708167 7f73014c47c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3966
|
|
2013-08-15 06:47:28.708728 7f73014c47c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.711448 7f73014c47c0 0 filestore(/var/lib/ceph/osd/ceph-8) lock_fsid failed to lock /var/lib/ceph/osd/ceph-8/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.711464 7f73014c47c0 -1 filestore(/var/lib/ceph/osd/ceph-8) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.711489 7f73014c47c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-8: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.760372 7fa2c833f7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4157
|
|
2013-08-15 06:47:28.761000 7fa2c833f7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.764302 7fa2c833f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) lock_fsid failed to lock /var/lib/ceph/osd/ceph-8/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.764319 7fa2c833f7c0 -1 filestore(/var/lib/ceph/osd/ceph-8) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.764353 7fa2c833f7c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-8: (16) Device or resource busy[0m
|
|
2013-08-15 06:47:28.811436 7fea9cad27c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4394
|
|
2013-08-15 06:47:28.811999 7fea9cad27c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 06:47:28.815167 7fea9cad27c0 0 filestore(/var/lib/ceph/osd/ceph-8) lock_fsid failed to lock /var/lib/ceph/osd/ceph-8/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
|
|
2013-08-15 06:47:28.815191 7fea9cad27c0 -1 filestore(/var/lib/ceph/osd/ceph-8) FileStore::mount: lock_fsid failed
|
|
2013-08-15 06:47:28.815226 7fea9cad27c0 -1 [0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-8: (16) Device or resource busy[0m
|
|
2013-08-15 20:46:26.421171 7f2ade0057c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3274
|
|
2013-08-15 20:46:26.422152 7f2ade0057c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.481048 7f45f47717c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3484
|
|
2013-08-15 20:46:26.481913 7f45f47717c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.528691 7fe95c7957c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3691
|
|
2013-08-15 20:46:26.529140 7fe95c7957c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.569398 7fb3e634a7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 3887
|
|
2013-08-15 20:46:26.569862 7fb3e634a7c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.615622 7f6a27aac7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4097
|
|
2013-08-15 20:46:26.616090 7f6a27aac7c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:46:26.660220 7f22441327c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4299
|
|
2013-08-15 20:46:26.660659 7f22441327c0 -1 unable to find any IP address in networks: 9.4.196.0/24
|
|
2013-08-15 20:50:22.474318 7f0a4821f7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4119
|
|
2013-08-15 20:50:22.474935 7f0a4821f7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:22.535789 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:22.535804 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:22.536084 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:22.541742 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:22.542051 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:22.635105 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:22.657551 7f0a4821f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:22.657583 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:23.126026 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:23.732099 7f0a4821f7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
2013-08-15 20:50:23.773865 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:23.773880 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:23.774181 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:23.791784 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:23.791839 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:23.834162 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:23.834894 7f0a4821f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:23.834906 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 27: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:23.902350 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.285476 7f0a3e305700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:24.285497 7f0a3e305700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
2013-08-15 20:50:24.286482 7f0a3e305700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f0a3e305700 time 2013-08-15 20:50:24.285504
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f0a47bfbf8e]
|
|
5: (clone()+0x6d) [0x7f0a45eade1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-247> 2013-08-15 20:50:22.472514 7f0a4821f7c0 5 asok(0x183c0e0) register_command perfcounters_dump hook 0x1836010
|
|
-246> 2013-08-15 20:50:22.472548 7f0a4821f7c0 5 asok(0x183c0e0) register_command 1 hook 0x1836010
|
|
-245> 2013-08-15 20:50:22.472551 7f0a4821f7c0 5 asok(0x183c0e0) register_command perf dump hook 0x1836010
|
|
-244> 2013-08-15 20:50:22.472558 7f0a4821f7c0 5 asok(0x183c0e0) register_command perfcounters_schema hook 0x1836010
|
|
-243> 2013-08-15 20:50:22.472567 7f0a4821f7c0 5 asok(0x183c0e0) register_command 2 hook 0x1836010
|
|
-242> 2013-08-15 20:50:22.472573 7f0a4821f7c0 5 asok(0x183c0e0) register_command perf schema hook 0x1836010
|
|
-241> 2013-08-15 20:50:22.472579 7f0a4821f7c0 5 asok(0x183c0e0) register_command config show hook 0x1836010
|
|
-240> 2013-08-15 20:50:22.472584 7f0a4821f7c0 5 asok(0x183c0e0) register_command config set hook 0x1836010
|
|
-239> 2013-08-15 20:50:22.472589 7f0a4821f7c0 5 asok(0x183c0e0) register_command log flush hook 0x1836010
|
|
-238> 2013-08-15 20:50:22.472595 7f0a4821f7c0 5 asok(0x183c0e0) register_command log dump hook 0x1836010
|
|
-237> 2013-08-15 20:50:22.472600 7f0a4821f7c0 5 asok(0x183c0e0) register_command log reopen hook 0x1836010
|
|
-236> 2013-08-15 20:50:22.474318 7f0a4821f7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 4119
|
|
-235> 2013-08-15 20:50:22.474935 7f0a4821f7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-234> 2013-08-15 20:50:22.475092 7f0a4821f7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6809/4119 need_addr=1
|
|
-233> 2013-08-15 20:50:22.475125 7f0a4821f7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-232> 2013-08-15 20:50:22.475139 7f0a4821f7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6810/4119 need_addr=0
|
|
-231> 2013-08-15 20:50:22.475166 7f0a4821f7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-230> 2013-08-15 20:50:22.475176 7f0a4821f7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6811/4119 need_addr=0
|
|
-229> 2013-08-15 20:50:22.480425 7f0a4821f7c0 5 asok(0x183c0e0) init /var/run/ceph/ceph-osd.8.asok
|
|
-228> 2013-08-15 20:50:22.480444 7f0a4821f7c0 5 asok(0x183c0e0) bind_and_listen /var/run/ceph/ceph-osd.8.asok
|
|
-227> 2013-08-15 20:50:22.480470 7f0a4821f7c0 5 asok(0x183c0e0) register_command 0 hook 0x18340b8
|
|
-226> 2013-08-15 20:50:22.480477 7f0a4821f7c0 5 asok(0x183c0e0) register_command version hook 0x18340b8
|
|
-225> 2013-08-15 20:50:22.480484 7f0a4821f7c0 5 asok(0x183c0e0) register_command git_version hook 0x18340b8
|
|
-224> 2013-08-15 20:50:22.480491 7f0a4821f7c0 5 asok(0x183c0e0) register_command help hook 0x18360d0
|
|
-223> 2013-08-15 20:50:22.480539 7f0a43bf9700 5 asok(0x183c0e0) entry start
|
|
-222> 2013-08-15 20:50:22.535789 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-221> 2013-08-15 20:50:22.535804 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-220> 2013-08-15 20:50:22.536084 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-219> 2013-08-15 20:50:22.541742 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-218> 2013-08-15 20:50:22.542051 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-217> 2013-08-15 20:50:22.635105 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-216> 2013-08-15 20:50:22.657506 7f0a4821f7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2413
|
|
-215> 2013-08-15 20:50:22.657551 7f0a4821f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-214> 2013-08-15 20:50:22.657583 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-213> 2013-08-15 20:50:22.672304 7f0a4821f7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-212> 2013-08-15 20:50:22.676357 7f0a4821f7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-211> 2013-08-15 20:50:22.676369 7f0a4821f7c0 3 journal journal_replay: applying op seq 2414
|
|
-210> 2013-08-15 20:50:22.689253 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2414
|
|
-209> 2013-08-15 20:50:22.702863 7f0a4821f7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-208> 2013-08-15 20:50:22.702877 7f0a4821f7c0 3 journal journal_replay: applying op seq 2415
|
|
-207> 2013-08-15 20:50:22.705311 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2415
|
|
-206> 2013-08-15 20:50:22.735935 7f0a4821f7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-205> 2013-08-15 20:50:22.735962 7f0a4821f7c0 3 journal journal_replay: applying op seq 2416
|
|
-204> 2013-08-15 20:50:22.743335 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2416
|
|
-203> 2013-08-15 20:50:22.762636 7f0a4821f7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-202> 2013-08-15 20:50:22.762658 7f0a4821f7c0 3 journal journal_replay: applying op seq 2417
|
|
-201> 2013-08-15 20:50:22.773205 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2417
|
|
-200> 2013-08-15 20:50:22.800563 7f0a4821f7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-199> 2013-08-15 20:50:22.800580 7f0a4821f7c0 3 journal journal_replay: applying op seq 2418
|
|
-198> 2013-08-15 20:50:22.802921 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2418
|
|
-197> 2013-08-15 20:50:22.833649 7f0a4821f7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-196> 2013-08-15 20:50:22.833670 7f0a4821f7c0 3 journal journal_replay: applying op seq 2419
|
|
-195> 2013-08-15 20:50:22.844314 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2419
|
|
-194> 2013-08-15 20:50:22.869301 7f0a4821f7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-193> 2013-08-15 20:50:22.869327 7f0a4821f7c0 3 journal journal_replay: applying op seq 2420
|
|
-192> 2013-08-15 20:50:22.872301 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2420
|
|
-191> 2013-08-15 20:50:22.902579 7f0a4821f7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-190> 2013-08-15 20:50:22.902600 7f0a4821f7c0 3 journal journal_replay: applying op seq 2421
|
|
-189> 2013-08-15 20:50:22.905730 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2421
|
|
-188> 2013-08-15 20:50:22.933719 7f0a4821f7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-187> 2013-08-15 20:50:22.933740 7f0a4821f7c0 3 journal journal_replay: applying op seq 2422
|
|
-186> 2013-08-15 20:50:22.936499 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2422
|
|
-185> 2013-08-15 20:50:22.968386 7f0a4821f7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-184> 2013-08-15 20:50:22.968403 7f0a4821f7c0 3 journal journal_replay: applying op seq 2423
|
|
-183> 2013-08-15 20:50:22.971255 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2423
|
|
-182> 2013-08-15 20:50:22.997837 7f0a4821f7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-181> 2013-08-15 20:50:22.997854 7f0a4821f7c0 3 journal journal_replay: applying op seq 2424
|
|
-180> 2013-08-15 20:50:23.000445 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2424
|
|
-179> 2013-08-15 20:50:23.029054 7f0a4821f7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-178> 2013-08-15 20:50:23.029073 7f0a4821f7c0 3 journal journal_replay: applying op seq 2425
|
|
-177> 2013-08-15 20:50:23.031486 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2425
|
|
-176> 2013-08-15 20:50:23.058367 7f0a4821f7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-175> 2013-08-15 20:50:23.058385 7f0a4821f7c0 3 journal journal_replay: applying op seq 2426
|
|
-174> 2013-08-15 20:50:23.061011 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2426
|
|
-173> 2013-08-15 20:50:23.089806 7f0a4821f7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-172> 2013-08-15 20:50:23.089824 7f0a4821f7c0 3 journal journal_replay: applying op seq 2427
|
|
-171> 2013-08-15 20:50:23.092467 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2427
|
|
-170> 2013-08-15 20:50:23.123270 7f0a4821f7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-169> 2013-08-15 20:50:23.123289 7f0a4821f7c0 3 journal journal_replay: applying op seq 2428
|
|
-168> 2013-08-15 20:50:23.125970 7f0a4821f7c0 3 journal journal_replay: r = 0, op_seq now 2428
|
|
-167> 2013-08-15 20:50:23.125992 7f0a4821f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-166> 2013-08-15 20:50:23.125997 7f0a4821f7c0 3 journal journal_replay: end of journal, done.
|
|
-165> 2013-08-15 20:50:23.126026 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-164> 2013-08-15 20:50:23.731981 7f0a40b0a700 1 FileStore::op_tp worker finish
|
|
-163> 2013-08-15 20:50:23.732007 7f0a4130b700 1 FileStore::op_tp worker finish
|
|
-162> 2013-08-15 20:50:23.732099 7f0a4821f7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
-161> 2013-08-15 20:50:23.732472 7f0a4821f7c0 10 monclient(hunting): build_initial_monmap
|
|
-160> 2013-08-15 20:50:23.732517 7f0a4821f7c0 5 adding auth protocol: cephx
|
|
-159> 2013-08-15 20:50:23.732526 7f0a4821f7c0 5 adding auth protocol: cephx
|
|
-158> 2013-08-15 20:50:23.732650 7f0a4821f7c0 1 -- 0.0.0.0:6809/4119 messenger.start
|
|
-157> 2013-08-15 20:50:23.732688 7f0a4821f7c0 1 -- :/0 messenger.start
|
|
-156> 2013-08-15 20:50:23.732707 7f0a4821f7c0 1 -- 9.4.196.3:6811/4119 messenger.start
|
|
-155> 2013-08-15 20:50:23.732727 7f0a4821f7c0 1 -- 9.4.196.3:6810/4119 messenger.start
|
|
-154> 2013-08-15 20:50:23.732828 7f0a4821f7c0 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
|
|
-153> 2013-08-15 20:50:23.773865 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-152> 2013-08-15 20:50:23.773880 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-151> 2013-08-15 20:50:23.774181 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-150> 2013-08-15 20:50:23.791784 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-149> 2013-08-15 20:50:23.791839 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-148> 2013-08-15 20:50:23.834162 7f0a4821f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-147> 2013-08-15 20:50:23.834872 7f0a4821f7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-146> 2013-08-15 20:50:23.834894 7f0a4821f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-145> 2013-08-15 20:50:23.834906 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 27: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-144> 2013-08-15 20:50:23.839527 7f0a4821f7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-143> 2013-08-15 20:50:23.844206 7f0a4821f7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-142> 2013-08-15 20:50:23.848767 7f0a4821f7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-141> 2013-08-15 20:50:23.853106 7f0a4821f7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-140> 2013-08-15 20:50:23.857387 7f0a4821f7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-139> 2013-08-15 20:50:23.861767 7f0a4821f7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-138> 2013-08-15 20:50:23.866539 7f0a4821f7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-137> 2013-08-15 20:50:23.871014 7f0a4821f7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-136> 2013-08-15 20:50:23.875534 7f0a4821f7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-135> 2013-08-15 20:50:23.880297 7f0a4821f7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-134> 2013-08-15 20:50:23.884666 7f0a4821f7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-133> 2013-08-15 20:50:23.888949 7f0a4821f7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-132> 2013-08-15 20:50:23.893417 7f0a4821f7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-131> 2013-08-15 20:50:23.897914 7f0a4821f7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-130> 2013-08-15 20:50:23.902286 7f0a4821f7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-129> 2013-08-15 20:50:23.902304 7f0a4821f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-128> 2013-08-15 20:50:23.902309 7f0a4821f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-127> 2013-08-15 20:50:23.902312 7f0a4821f7c0 3 journal journal_replay: end of journal, done.
|
|
-126> 2013-08-15 20:50:23.902350 7f0a4821f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-125> 2013-08-15 20:50:23.902535 7f0a4821f7c0 2 osd.8 0 boot
|
|
-124> 2013-08-15 20:50:23.941201 7f0a4821f7c0 2 osd.8 63 superblock: i am osd.8
|
|
-123> 2013-08-15 20:50:23.941259 7f0a4821f7c0 1 accepter.accepter.start
|
|
-122> 2013-08-15 20:50:23.941300 7f0a4821f7c0 1 accepter.accepter.start
|
|
-121> 2013-08-15 20:50:23.941341 7f0a4821f7c0 1 accepter.accepter.start
|
|
-120> 2013-08-15 20:50:23.941359 7f0a4821f7c0 10 monclient(hunting): init
|
|
-119> 2013-08-15 20:50:23.941367 7f0a4821f7c0 5 adding auth protocol: cephx
|
|
-118> 2013-08-15 20:50:23.941370 7f0a4821f7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-117> 2013-08-15 20:50:23.941517 7f0a4821f7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-8/keyring
|
|
-116> 2013-08-15 20:50:23.941703 7f0a4821f7c0 5 asok(0x183c0e0) register_command dump_ops_in_flight hook 0x1836780
|
|
-115> 2013-08-15 20:50:23.941713 7f0a4821f7c0 5 asok(0x183c0e0) register_command dump_historic_ops hook 0x1836780
|
|
-114> 2013-08-15 20:50:23.941718 7f0a4821f7c0 5 asok(0x183c0e0) register_command dump_op_pq_state hook 0x1836780
|
|
-113> 2013-08-15 20:50:23.941724 7f0a4821f7c0 5 asok(0x183c0e0) register_command setomapval hook 0x1abede0
|
|
-112> 2013-08-15 20:50:23.941728 7f0a4821f7c0 5 asok(0x183c0e0) register_command rmomapkey hook 0x1abede0
|
|
-111> 2013-08-15 20:50:23.941733 7f0a4821f7c0 5 asok(0x183c0e0) register_command setomapheader hook 0x1abede0
|
|
-110> 2013-08-15 20:50:23.941738 7f0a4821f7c0 5 asok(0x183c0e0) register_command getomap hook 0x1abede0
|
|
-109> 2013-08-15 20:50:23.941742 7f0a4821f7c0 5 asok(0x183c0e0) register_command truncobj hook 0x1abede0
|
|
-108> 2013-08-15 20:50:23.941745 7f0a4821f7c0 5 asok(0x183c0e0) register_command injectdataerr hook 0x1abede0
|
|
-107> 2013-08-15 20:50:23.941750 7f0a4821f7c0 5 asok(0x183c0e0) register_command injectmdataerr hook 0x1abede0
|
|
-106> 2013-08-15 20:50:23.941752 7f0a32aee700 5 osd.8 63 heartbeat: osd_stat(1022 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-105> 2013-08-15 20:50:23.941817 7f0a4821f7c0 10 monclient(hunting): _reopen_session
|
|
-104> 2013-08-15 20:50:23.941857 7f0a4821f7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-b con 0x1b24580 addr 9.4.196.5:6789/0
|
|
-103> 2013-08-15 20:50:23.941871 7f0a4821f7c0 10 monclient(hunting): _send_mon_message to mon.noname-b at 9.4.196.5:6789/0
|
|
-102> 2013-08-15 20:50:23.941877 7f0a4821f7c0 1 -- 0.0.0.0:6809/4119 --> 9.4.196.5:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1848b40 con 0x1b24580
|
|
-101> 2013-08-15 20:50:23.941890 7f0a4821f7c0 10 monclient(hunting): renew_subs
|
|
-100> 2013-08-15 20:50:23.942261 7f0a40215700 1 -- 9.4.196.3:6809/4119 learned my addr 9.4.196.3:6809/4119
|
|
-99> 2013-08-15 20:50:23.942517 7f0a39afc700 10 monclient(hunting): renew_subs
|
|
-98> 2013-08-15 20:50:23.943059 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x18be3c0 con 0x1b24580
|
|
-97> 2013-08-15 20:50:23.943080 7f0a39afc700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-96> 2013-08-15 20:50:23.943093 7f0a39afc700 10 monclient(hunting): got monmap 1, mon.noname-b is now rank -1
|
|
-95> 2013-08-15 20:50:23.943097 7f0a39afc700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-94> 2013-08-15 20:50:23.943122 7f0a39afc700 1 monclient(hunting): found mon.ceph-dmon2
|
|
-93> 2013-08-15 20:50:23.943166 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (965475137 0 0) 0x1e50600 con 0x1b24580
|
|
-92> 2013-08-15 20:50:23.943193 7f0a39afc700 10 monclient: my global_id is 4310
|
|
-91> 2013-08-15 20:50:23.943339 7f0a39afc700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-90> 2013-08-15 20:50:23.943350 7f0a39afc700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1849d40 con 0x1b24580
|
|
-89> 2013-08-15 20:50:23.944176 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (2450457762 0 0) 0x1e50400 con 0x1b24580
|
|
-88> 2013-08-15 20:50:23.944254 7f0a39afc700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-87> 2013-08-15 20:50:23.944262 7f0a39afc700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1b2c240 con 0x1b24580
|
|
-86> 2013-08-15 20:50:23.945429 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (1630421911 0 0) 0x1e50800 con 0x1b24580
|
|
-85> 2013-08-15 20:50:23.945495 7f0a39afc700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-84> 2013-08-15 20:50:23.945503 7f0a39afc700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x19ae000 con 0x1b24580
|
|
-83> 2013-08-15 20:50:23.945521 7f0a39afc700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-82> 2013-08-15 20:50:23.945528 7f0a39afc700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x19ae380 con 0x1b24580
|
|
-81> 2013-08-15 20:50:23.945550 7f0a39afc700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:53.945549)
|
|
-80> 2013-08-15 20:50:23.945563 7f0a39afc700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-79> 2013-08-15 20:50:23.945572 7f0a39afc700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1b2c000 con 0x1b24580
|
|
-78> 2013-08-15 20:50:23.945598 7f0a4821f7c0 5 monclient: authenticate success, global_id 4310
|
|
-77> 2013-08-15 20:50:23.945608 7f0a4821f7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:53.945607)
|
|
-76> 2013-08-15 20:50:23.946095 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x18be780 con 0x1b24580
|
|
-75> 2013-08-15 20:50:23.946111 7f0a39afc700 10 monclient: handle_monmap mon_map v1
|
|
-74> 2013-08-15 20:50:23.946121 7f0a39afc700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-73> 2013-08-15 20:50:23.946125 7f0a39afc700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-72> 2013-08-15 20:50:23.946144 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x19ae000 con 0x1b24580
|
|
-71> 2013-08-15 20:50:23.946153 7f0a39afc700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:23.941893 renew after 2013-08-15 20:52:53.941893
|
|
-70> 2013-08-15 20:50:23.946255 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x18be5a0 con 0x1b24580
|
|
-69> 2013-08-15 20:50:23.946271 7f0a39afc700 10 monclient: handle_monmap mon_map v1
|
|
-68> 2013-08-15 20:50:23.946282 7f0a39afc700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-67> 2013-08-15 20:50:23.946286 7f0a39afc700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-66> 2013-08-15 20:50:23.946318 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x19ae540 con 0x1b24580
|
|
-65> 2013-08-15 20:50:23.946332 7f0a39afc700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-64> 2013-08-15 20:50:23.946744 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1065813796 0 0) 0x1e50a00 con 0x1b24580
|
|
-63> 2013-08-15 20:50:23.946795 7f0a39afc700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:53.946794)
|
|
-62> 2013-08-15 20:50:23.946810 7f0a4821f7c0 10 monclient: wait_auth_rotating done
|
|
-61> 2013-08-15 20:50:23.947476 7f0a4821f7c0 10 monclient: get_version osdmap
|
|
-60> 2013-08-15 20:50:23.947485 7f0a4821f7c0 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-59> 2013-08-15 20:50:23.947492 7f0a4821f7c0 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x18a49c0 con 0x1b24580
|
|
-58> 2013-08-15 20:50:23.947910 7f0a39afc700 1 -- 9.4.196.3:6809/4119 <== mon.2 9.4.196.5:6789/0 10 ==== mon_check_map_ack(handle=1 version=82) v2 ==== 24+0+0 (1547262183 0 0) 0x19ae380 con 0x1b24580
|
|
-57> 2013-08-15 20:50:23.947961 7f0a35af4700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-56> 2013-08-15 20:50:23.947974 7f0a35af4700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- osd_boot(osd.8 booted 0 v63) v3 -- ?+0 0x1b42000 con 0x1b24580
|
|
-55> 2013-08-15 20:50:24.263600 7f0a30de6700 1 -- 9.4.196.3:6810/4119 >> :/0 pipe(0x1b29400 sd=33 :6810 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57652/0
|
|
-54> 2013-08-15 20:50:24.264583 7f0a30be4700 1 -- 9.4.196.3:6810/4119 >> :/0 pipe(0x1b29900 sd=34 :6810 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:57654/0
|
|
-53> 2013-08-15 20:50:24.268400 7f0a38afa700 1 -- 9.4.196.3:6810/4119 <== osd.5 9.4.196.3:6819/4129 1 ==== pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4 ==== 2964+0+0 (1404680434 0 0) 0x19ae700 con 0x1b38b00
|
|
-52> 2013-08-15 20:50:24.268459 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.268289, event: header_read, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-51> 2013-08-15 20:50:24.268479 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.268291, event: throttled, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-50> 2013-08-15 20:50:24.268492 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.268380, event: all_read, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-49> 2013-08-15 20:50:24.268505 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.268446, event: dispatched, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-48> 2013-08-15 20:50:24.268517 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.268517, event: waiting_for_osdmap, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-47> 2013-08-15 20:50:24.268536 7f0a38afa700 10 monclient: renew_subs
|
|
-46> 2013-08-15 20:50:24.268544 7f0a38afa700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-45> 2013-08-15 20:50:24.268557 7f0a38afa700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x19ae8c0 con 0x1b24580
|
|
-44> 2013-08-15 20:50:24.268580 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.268580, event: wait for new map, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-43> 2013-08-15 20:50:24.269793 7f0a309e2700 1 -- 9.4.196.3:6810/4119 >> :/0 pipe(0x1b29680 sd=35 :6810 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:57665/0
|
|
-42> 2013-08-15 20:50:24.270281 7f0a38afa700 1 -- 9.4.196.3:6810/4119 <== osd.5 9.4.196.3:6819/4129 2 ==== pg_notify(3.24a(3) epoch 83) v4 ==== 713+0+0 (2045658353 0 0) 0x19aec40 con 0x1b38b00
|
|
-41> 2013-08-15 20:50:24.270305 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.270206, event: header_read, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-40> 2013-08-15 20:50:24.270317 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.270207, event: throttled, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-39> 2013-08-15 20:50:24.270327 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.270264, event: all_read, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-38> 2013-08-15 20:50:24.270336 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.270301, event: dispatched, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-37> 2013-08-15 20:50:24.270347 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.270346, event: waiting_for_osdmap, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-36> 2013-08-15 20:50:24.270358 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.270358, event: wait for new map, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-35> 2013-08-15 20:50:24.272671 7f0a38afa700 1 -- 9.4.196.3:6810/4119 <== osd.10 9.4.196.3:6804/4123 1 ==== pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4 ==== 1488+0+0 (1198686570 0 0) 0x19aee00 con 0x1b38c60
|
|
-34> 2013-08-15 20:50:24.272699 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.272594, event: header_read, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-33> 2013-08-15 20:50:24.272714 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.272596, event: throttled, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-32> 2013-08-15 20:50:24.272723 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.272653, event: all_read, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-31> 2013-08-15 20:50:24.272731 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.272695, event: dispatched, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-30> 2013-08-15 20:50:24.272736 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.272736, event: waiting_for_osdmap, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-29> 2013-08-15 20:50:24.272744 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.272744, event: wait for new map, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-28> 2013-08-15 20:50:24.280789 7f0a38afa700 1 -- 9.4.196.3:6810/4119 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(64..83 src has 1..83) v3 ==== 15682+0+0 (3550349950 0 0) 0x1b2d8c0 con 0x1b25760
|
|
-27> 2013-08-15 20:50:24.280821 7f0a38afa700 3 osd.8 63 handle_osd_map epochs [64,83], i have 63, src has [1,83]
|
|
-26> 2013-08-15 20:50:24.284947 7f0a38afa700 1 -- 9.4.196.3:6810/4119 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-25> 2013-08-15 20:50:24.284963 7f0a38afa700 1 -- 9.4.196.3:6810/4119 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-24> 2013-08-15 20:50:24.284969 7f0a38afa700 1 -- 9.4.196.3:6810/4119 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-23> 2013-08-15 20:50:24.284986 7f0a38afa700 1 -- 9.4.196.3:6810/4119 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-22> 2013-08-15 20:50:24.284998 7f0a38afa700 1 -- 9.4.196.3:6810/4119 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-21> 2013-08-15 20:50:24.285003 7f0a38afa700 1 -- 9.4.196.3:6810/4119 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-20> 2013-08-15 20:50:24.285025 7f0a38afa700 1 osd.8 83 state: booting -> active
|
|
-19> 2013-08-15 20:50:24.285476 7f0a3e305700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-18> 2013-08-15 20:50:24.285497 7f0a3e305700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
-17> 2013-08-15 20:50:24.285490 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.285490, event: started, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b(3) epoch 83) v4
|
|
-16> 2013-08-15 20:50:24.285910 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[3.16b( empty local-les=0 n=0 ec=56 les/c 74/74 83/83/83) [8,3] r=0 lpr=83 pi=67-82/3 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-15> 2013-08-15 20:50:24.285960 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6810/7799 -- pg_query(3.16b epoch 83) v2 -- ?+0 0x18be780 con 0x1b254a0
|
|
-14> 2013-08-15 20:50:24.285960 7f0a352f3700 1 osd.8 pg_epoch: 83 pg[3.69( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 83/83/56) [1,8] r=1 lpr=83 pi=56-82/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-13> 2013-08-15 20:50:24.286018 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:24.286017, event: done, request: pg_notify(3.18d(3),3.182(5),3.178(5),3.16b epoch 83) v4
|
|
-12> 2013-08-15 20:50:24.286039 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.286039, event: started, request: pg_notify(3.24a(3) epoch 83) v4
|
|
-11> 2013-08-15 20:50:24.286176 7f0a34af2700 1 osd.8 pg_epoch: 83 pg[3.12b( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 83/83/56) [3,8] r=1 lpr=83 pi=56-82/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-10> 2013-08-15 20:50:24.286200 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[3.24a( empty local-les=0 n=0 ec=56 les/c 74/74 83/83/83) [8,1] r=0 lpr=83 pi=67-82/3 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-9> 2013-08-15 20:50:24.286238 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_query(3.24a epoch 83) v2 -- ?+0 0x18beb40 con 0x1b25340
|
|
-8> 2013-08-15 20:50:24.286291 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:24.286291, event: done, request: pg_notify(3.24a epoch 83) v4
|
|
-7> 2013-08-15 20:50:24.286308 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.286308, event: started, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-6> 2013-08-15 20:50:24.286343 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:24.286343, event: done, request: pg_notify(3.1a8(5),3.1a0(3) epoch 83) v4
|
|
-5> 2013-08-15 20:50:24.286365 7f0a38afa700 1 -- 9.4.196.3:6810/4119 <== osd.2 9.4.196.1:6807/6887 2 ==== pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4 ==== 9906+0+0 (1731453686 0 0) 0x19aefc0 con 0x1b25760
|
|
-4> 2013-08-15 20:50:24.286392 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.280785, event: header_read, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4
|
|
-3> 2013-08-15 20:50:24.286411 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.280786, event: throttled, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4
|
|
-2> 2013-08-15 20:50:24.286429 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.280957, event: all_read, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4
|
|
-1> 2013-08-15 20:50:24.286470 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.286390, event: dispatched, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4
|
|
0> 2013-08-15 20:50:24.286482 7f0a3e305700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f0a3e305700 time 2013-08-15 20:50:24.285504
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f0a47bfbf8e]
|
|
5: (clone()+0x6d) [0x7f0a45eade1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:24.290112 7f0a3e305700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f0a3e305700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f0a47c03bd0]
|
|
3: (gsignal()+0x37) [0x7f0a45deb037]
|
|
4: (abort()+0x148) [0x7f0a45dee698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f0a466f7e8d]
|
|
6: (()+0x5ef76) [0x7f0a466f5f76]
|
|
7: (()+0x5efa3) [0x7f0a466f5fa3]
|
|
8: (()+0x5f1de) [0x7f0a466f61de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f0a47bfbf8e]
|
|
14: (clone()+0x6d) [0x7f0a45eade1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-39> 2013-08-15 20:50:24.286489 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.286489, event: waiting_for_osdmap, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4
|
|
-38> 2013-08-15 20:50:24.286511 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.286511, event: started, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f(4),1.e(4),2.d(4),3.c(4) epoch 83) v4
|
|
-37> 2013-08-15 20:50:24.286707 7f0a352f3700 1 osd.8 pg_epoch: 83 pg[3.1ea( v 63'13 (0'0,63'13] local-les=62 n=9 ec=56 les/c 62/62 83/83/83) [8,2] r=0 lpr=83 pi=61-82/7 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-36> 2013-08-15 20:50:24.286732 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[0.f( empty local-les=0 n=0 ec=1 les/c 80/80 83/83/83) [8,2] r=0 lpr=83 pi=73-82/4 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-35> 2013-08-15 20:50:24.286761 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_query(0.f epoch 83) v2 -- ?+0 0x18be960 con 0x1b25340
|
|
-34> 2013-08-15 20:50:24.286783 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6807/6887 -- pg_query(0.f epoch 83) v2 -- ?+0 0x18bf0e0 con 0x1b25760
|
|
-33> 2013-08-15 20:50:24.286796 7f0a352f3700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-32> 2013-08-15 20:50:24.286804 7f0a352f3700 1 -- 9.4.196.3:6809/4119 --> 9.4.196.5:6789/0 -- osd_alive(want up_thru 83 have 83) v1 -- ?+0 0x19af6c0 con 0x1b24580
|
|
-31> 2013-08-15 20:50:24.286828 7f0a352f3700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_notify(3.69(7) epoch 83) v4 -- ?+0 0x19afa40 con 0x1b25340
|
|
-30> 2013-08-15 20:50:24.286842 7f0a352f3700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6804/5992 -- pg_query(3.1ea epoch 83) v2 -- ?+0 0x18bf2c0 con 0x1b25ce0
|
|
-29> 2013-08-15 20:50:24.286862 7f0a352f3700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6807/6887 -- pg_query(3.1ea epoch 83) v2 -- ?+0 0x18bf680 con 0x1b25760
|
|
-28> 2013-08-15 20:50:24.286872 7f0a352f3700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.3:6804/4123 -- pg_query(3.1ea epoch 83) v2 -- ?+0 0x18bf4a0 con 0x1b38c60
|
|
-27> 2013-08-15 20:50:24.286976 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[1.e( empty local-les=0 n=0 ec=1 les/c 80/80 83/83/83) [8,2] r=0 lpr=83 pi=73-82/4 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-26> 2013-08-15 20:50:24.287005 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_query(1.e epoch 83) v2 -- ?+0 0x18bef00 con 0x1b25340
|
|
-25> 2013-08-15 20:50:24.287021 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6807/6887 -- pg_query(1.e epoch 83) v2 -- ?+0 0x18bed20 con 0x1b25760
|
|
-24> 2013-08-15 20:50:24.287104 7f0a34af2700 1 osd.8 pg_epoch: 83 pg[3.125( v 63'23 (0'0,63'23] local-les=62 n=7 ec=56 les/c 62/62 83/83/56) [1,8] r=1 lpr=83 pi=56-82/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-23> 2013-08-15 20:50:24.287163 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[2.d( empty local-les=0 n=0 ec=1 les/c 80/80 83/83/83) [8,2] r=0 lpr=83 pi=73-82/4 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-22> 2013-08-15 20:50:24.287196 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_query(2.d epoch 83) v2 -- ?+0 0x18bfc20 con 0x1b25340
|
|
-21> 2013-08-15 20:50:24.287215 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6807/6887 -- pg_query(2.d epoch 83) v2 -- ?+0 0x18bfa40 con 0x1b25760
|
|
-20> 2013-08-15 20:50:24.287485 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[3.c( empty local-les=0 n=0 ec=56 les/c 80/80 83/83/83) [8,2] r=0 lpr=83 pi=73-82/4 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-19> 2013-08-15 20:50:24.287517 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_query(3.c epoch 83) v2 -- ?+0 0x18bf860 con 0x1b25340
|
|
-18> 2013-08-15 20:50:24.287535 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6807/6887 -- pg_query(3.c epoch 83) v2 -- ?+0 0x18beb40 con 0x1b25760
|
|
-17> 2013-08-15 20:50:24.287577 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:24.287577, event: done, request: pg_notify(0.37(4),1.36(4),2.35(4),3.34(4),0.34(2),3.31(2),1.33(2),2.32(2),3.1a0(2),3.19e(4),0.f,1.e,2.d,3.c epoch 83) v4
|
|
-16> 2013-08-15 20:50:24.287599 7f0a38afa700 1 -- 9.4.196.3:6810/4119 <== osd.5 9.4.196.3:6819/4129 3 ==== pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4 ==== 9121+0+0 (3681020135 0 0) 0x19aea80 con 0x1b38b00
|
|
-15> 2013-08-15 20:50:24.287620 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:24.282909, event: header_read, request: pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4
|
|
-14> 2013-08-15 20:50:24.287633 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:24.282911, event: throttled, request: pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4
|
|
-13> 2013-08-15 20:50:24.287646 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:24.283077, event: all_read, request: pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4
|
|
-12> 2013-08-15 20:50:24.287659 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:24.287617, event: dispatched, request: pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4
|
|
-11> 2013-08-15 20:50:24.287665 7f0a352f3700 1 osd.8 pg_epoch: 83 pg[2.35( v 55'90 (0'0,55'90] local-les=62 n=0 ec=1 les/c 62/62 83/83/83) [8,2] r=0 lpr=83 pi=61-82/7 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-10> 2013-08-15 20:50:24.287672 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:24.287672, event: waiting_for_osdmap, request: pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4
|
|
-9> 2013-08-15 20:50:24.287686 7f0a38afa700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:24.287686, event: started, request: pg_notify(3.8f(5),3.86(2),3.200(5),3.44(3),0.37(5),1.36(5),2.35(5),3.34(5),0.34(5),1.33(5),3.31(5),2.32(5) epoch 83) v4
|
|
-8> 2013-08-15 20:50:24.287933 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[3.86( empty local-les=0 n=0 ec=56 les/c 74/74 83/83/83) [8,1] r=0 lpr=83 pi=73-82/2 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-7> 2013-08-15 20:50:24.287968 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6801/5083 -- pg_query(3.86 epoch 83) v2 -- ?+0 0x1c70780 con 0x1b25340
|
|
-6> 2013-08-15 20:50:24.288209 7f0a38afa700 1 osd.8 pg_epoch: 83 pg[3.44( empty local-les=0 n=0 ec=56 les/c 76/76 83/83/83) [8,2] r=0 lpr=83 pi=67-82/3 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-5> 2013-08-15 20:50:24.288255 7f0a34af2700 1 osd.8 pg_epoch: 83 pg[3.66( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 83/83/83) [8,0] r=0 lpr=83 pi=61-82/7 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-4> 2013-08-15 20:50:24.288263 7f0a38afa700 1 -- 9.4.196.3:6810/4119 --> 9.4.196.1:6807/6887 -- pg_query(3.44 epoch 83) v2 -- ?+0 0x1c705a0 con 0x1b25760
|
|
-3> 2013-08-15 20:50:24.288862 7f0a34af2700 1 osd.8 pg_epoch: 83 pg[3.11c( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 83/83/56) [3,8] r=1 lpr=83 pi=56-82/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:24.289368 7f0a34af2700 1 osd.8 pg_epoch: 83 pg[3.11d( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 83/83/83) [8,0] r=0 lpr=83 pi=61-82/7 lcod 0'0 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-1> 2013-08-15 20:50:24.289929 7f0a34af2700 1 osd.8 pg_epoch: 83 pg[3.58( v 63'21 (0'0,63'21] local-les=62 n=5 ec=56 les/c 62/62 83/83/56) [3,8] r=1 lpr=83 pi=56-82/7 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:24.290112 7f0a3e305700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f0a3e305700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f0a47c03bd0]
|
|
3: (gsignal()+0x37) [0x7f0a45deb037]
|
|
4: (abort()+0x148) [0x7f0a45dee698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f0a466f7e8d]
|
|
6: (()+0x5ef76) [0x7f0a466f5f76]
|
|
7: (()+0x5efa3) [0x7f0a466f5fa3]
|
|
8: (()+0x5f1de) [0x7f0a466f61de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f0a47bfbf8e]
|
|
14: (clone()+0x6d) [0x7f0a45eade1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:24.534445 7f6c5584c7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8147
|
|
2013-08-15 20:50:24.534983 7f6c5584c7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:24.581374 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:24.581393 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:24.581681 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:24.599285 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:24.599389 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:24.629660 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:24.630549 7f6c5584c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:24.630569 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.697403 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.697862 7f6c5584c7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
2013-08-15 20:50:24.736958 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:24.736976 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:24.737283 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:24.754903 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:24.754978 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:24.785093 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:24.785713 7f6c5584c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:24.785745 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:24.845194 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:26.262994 7f6c4e220700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:26.263012 7f6c4e220700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
2013-08-15 20:50:26.263907 7f6c4e220700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f6c4e220700 time 2013-08-15 20:50:26.263017
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f6c55228f8e]
|
|
5: (clone()+0x6d) [0x7f6c534dae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-195> 2013-08-15 20:50:24.533108 7f6c5584c7c0 5 asok(0x35800e0) register_command perfcounters_dump hook 0x357a010
|
|
-194> 2013-08-15 20:50:24.533137 7f6c5584c7c0 5 asok(0x35800e0) register_command 1 hook 0x357a010
|
|
-193> 2013-08-15 20:50:24.533140 7f6c5584c7c0 5 asok(0x35800e0) register_command perf dump hook 0x357a010
|
|
-192> 2013-08-15 20:50:24.533146 7f6c5584c7c0 5 asok(0x35800e0) register_command perfcounters_schema hook 0x357a010
|
|
-191> 2013-08-15 20:50:24.533148 7f6c5584c7c0 5 asok(0x35800e0) register_command 2 hook 0x357a010
|
|
-190> 2013-08-15 20:50:24.533150 7f6c5584c7c0 5 asok(0x35800e0) register_command perf schema hook 0x357a010
|
|
-189> 2013-08-15 20:50:24.533154 7f6c5584c7c0 5 asok(0x35800e0) register_command config show hook 0x357a010
|
|
-188> 2013-08-15 20:50:24.533157 7f6c5584c7c0 5 asok(0x35800e0) register_command config set hook 0x357a010
|
|
-187> 2013-08-15 20:50:24.533160 7f6c5584c7c0 5 asok(0x35800e0) register_command log flush hook 0x357a010
|
|
-186> 2013-08-15 20:50:24.533162 7f6c5584c7c0 5 asok(0x35800e0) register_command log dump hook 0x357a010
|
|
-185> 2013-08-15 20:50:24.533165 7f6c5584c7c0 5 asok(0x35800e0) register_command log reopen hook 0x357a010
|
|
-184> 2013-08-15 20:50:24.534445 7f6c5584c7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8147
|
|
-183> 2013-08-15 20:50:24.534983 7f6c5584c7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-182> 2013-08-15 20:50:24.535121 7f6c5584c7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6809/8147 need_addr=1
|
|
-181> 2013-08-15 20:50:24.535149 7f6c5584c7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-180> 2013-08-15 20:50:24.535157 7f6c5584c7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6811/8147 need_addr=0
|
|
-179> 2013-08-15 20:50:24.535180 7f6c5584c7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-178> 2013-08-15 20:50:24.535186 7f6c5584c7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6821/8147 need_addr=0
|
|
-177> 2013-08-15 20:50:24.537119 7f6c5584c7c0 5 asok(0x35800e0) init /var/run/ceph/ceph-osd.8.asok
|
|
-176> 2013-08-15 20:50:24.537135 7f6c5584c7c0 5 asok(0x35800e0) bind_and_listen /var/run/ceph/ceph-osd.8.asok
|
|
-175> 2013-08-15 20:50:24.537168 7f6c5584c7c0 5 asok(0x35800e0) register_command 0 hook 0x35780b8
|
|
-174> 2013-08-15 20:50:24.537175 7f6c5584c7c0 5 asok(0x35800e0) register_command version hook 0x35780b8
|
|
-173> 2013-08-15 20:50:24.537181 7f6c5584c7c0 5 asok(0x35800e0) register_command git_version hook 0x35780b8
|
|
-172> 2013-08-15 20:50:24.537188 7f6c5584c7c0 5 asok(0x35800e0) register_command help hook 0x357a0d0
|
|
-171> 2013-08-15 20:50:24.537217 7f6c51226700 5 asok(0x35800e0) entry start
|
|
-170> 2013-08-15 20:50:24.581374 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-169> 2013-08-15 20:50:24.581393 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-168> 2013-08-15 20:50:24.581681 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-167> 2013-08-15 20:50:24.599285 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-166> 2013-08-15 20:50:24.599389 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-165> 2013-08-15 20:50:24.629660 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-164> 2013-08-15 20:50:24.630521 7f6c5584c7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-163> 2013-08-15 20:50:24.630549 7f6c5584c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-162> 2013-08-15 20:50:24.630569 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-161> 2013-08-15 20:50:24.637177 7f6c5584c7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-160> 2013-08-15 20:50:24.641705 7f6c5584c7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-159> 2013-08-15 20:50:24.646003 7f6c5584c7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-158> 2013-08-15 20:50:24.650423 7f6c5584c7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-157> 2013-08-15 20:50:24.654680 7f6c5584c7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-156> 2013-08-15 20:50:24.658833 7f6c5584c7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-155> 2013-08-15 20:50:24.663098 7f6c5584c7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-154> 2013-08-15 20:50:24.667440 7f6c5584c7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-153> 2013-08-15 20:50:24.671665 7f6c5584c7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-152> 2013-08-15 20:50:24.675856 7f6c5584c7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-151> 2013-08-15 20:50:24.680132 7f6c5584c7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-150> 2013-08-15 20:50:24.684410 7f6c5584c7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-149> 2013-08-15 20:50:24.688667 7f6c5584c7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-148> 2013-08-15 20:50:24.692898 7f6c5584c7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-147> 2013-08-15 20:50:24.697285 7f6c5584c7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-146> 2013-08-15 20:50:24.697335 7f6c5584c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-145> 2013-08-15 20:50:24.697342 7f6c5584c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-144> 2013-08-15 20:50:24.697349 7f6c5584c7c0 3 journal journal_replay: end of journal, done.
|
|
-143> 2013-08-15 20:50:24.697403 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-142> 2013-08-15 20:50:24.697734 7f6c4e220700 1 FileStore::op_tp worker finish
|
|
-141> 2013-08-15 20:50:24.697748 7f6c4ea21700 1 FileStore::op_tp worker finish
|
|
-140> 2013-08-15 20:50:24.697862 7f6c5584c7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
-139> 2013-08-15 20:50:24.698279 7f6c5584c7c0 10 monclient(hunting): build_initial_monmap
|
|
-138> 2013-08-15 20:50:24.698345 7f6c5584c7c0 5 adding auth protocol: cephx
|
|
-137> 2013-08-15 20:50:24.698352 7f6c5584c7c0 5 adding auth protocol: cephx
|
|
-136> 2013-08-15 20:50:24.698441 7f6c5584c7c0 1 -- 0.0.0.0:6809/8147 messenger.start
|
|
-135> 2013-08-15 20:50:24.698472 7f6c5584c7c0 1 -- :/0 messenger.start
|
|
-134> 2013-08-15 20:50:24.698483 7f6c5584c7c0 1 -- 9.4.196.3:6821/8147 messenger.start
|
|
-133> 2013-08-15 20:50:24.698496 7f6c5584c7c0 1 -- 9.4.196.3:6811/8147 messenger.start
|
|
-132> 2013-08-15 20:50:24.698584 7f6c5584c7c0 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
|
|
-131> 2013-08-15 20:50:24.736958 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-130> 2013-08-15 20:50:24.736976 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-129> 2013-08-15 20:50:24.737283 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-128> 2013-08-15 20:50:24.754903 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-127> 2013-08-15 20:50:24.754978 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-126> 2013-08-15 20:50:24.785093 7f6c5584c7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-125> 2013-08-15 20:50:24.785680 7f6c5584c7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-124> 2013-08-15 20:50:24.785713 7f6c5584c7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-123> 2013-08-15 20:50:24.785745 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-122> 2013-08-15 20:50:24.789883 7f6c5584c7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-121> 2013-08-15 20:50:24.793822 7f6c5584c7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-120> 2013-08-15 20:50:24.797748 7f6c5584c7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-119> 2013-08-15 20:50:24.801809 7f6c5584c7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-118> 2013-08-15 20:50:24.805770 7f6c5584c7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-117> 2013-08-15 20:50:24.809691 7f6c5584c7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-116> 2013-08-15 20:50:24.813713 7f6c5584c7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-115> 2013-08-15 20:50:24.817659 7f6c5584c7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-114> 2013-08-15 20:50:24.821556 7f6c5584c7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-113> 2013-08-15 20:50:24.825495 7f6c5584c7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-112> 2013-08-15 20:50:24.829363 7f6c5584c7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-111> 2013-08-15 20:50:24.833322 7f6c5584c7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-110> 2013-08-15 20:50:24.837224 7f6c5584c7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-109> 2013-08-15 20:50:24.841187 7f6c5584c7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-108> 2013-08-15 20:50:24.845102 7f6c5584c7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-107> 2013-08-15 20:50:24.845135 7f6c5584c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-106> 2013-08-15 20:50:24.845140 7f6c5584c7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-105> 2013-08-15 20:50:24.845141 7f6c5584c7c0 3 journal journal_replay: end of journal, done.
|
|
-104> 2013-08-15 20:50:24.845194 7f6c5584c7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-103> 2013-08-15 20:50:24.845396 7f6c5584c7c0 2 osd.8 0 boot
|
|
-102> 2013-08-15 20:50:24.871487 7f6c5584c7c0 2 osd.8 63 superblock: i am osd.8
|
|
-101> 2013-08-15 20:50:24.871562 7f6c5584c7c0 1 accepter.accepter.start
|
|
-100> 2013-08-15 20:50:24.871609 7f6c5584c7c0 1 accepter.accepter.start
|
|
-99> 2013-08-15 20:50:24.871665 7f6c5584c7c0 1 accepter.accepter.start
|
|
-98> 2013-08-15 20:50:24.871691 7f6c5584c7c0 10 monclient(hunting): init
|
|
-97> 2013-08-15 20:50:24.871711 7f6c5584c7c0 5 adding auth protocol: cephx
|
|
-96> 2013-08-15 20:50:24.871718 7f6c5584c7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-95> 2013-08-15 20:50:24.871837 7f6c557de700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d4000 sd=27 :6811 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.1:60457/0
|
|
-94> 2013-08-15 20:50:24.871867 7f6c44a0d700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d4500 sd=28 :6811 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.1:60459/0
|
|
-93> 2013-08-15 20:50:24.871881 7f6c4490c700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d4280 sd=29 :6811 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.1:60458/0
|
|
-92> 2013-08-15 20:50:24.871937 7f6c5584c7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-8/keyring
|
|
-91> 2013-08-15 20:50:24.871929 7f6c4480b700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d4c80 sd=30 :6811 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:60462/0
|
|
-90> 2013-08-15 20:50:24.872392 7f6c5584c7c0 5 asok(0x35800e0) register_command dump_ops_in_flight hook 0x357a750
|
|
-89> 2013-08-15 20:50:24.872404 7f6c5584c7c0 5 asok(0x35800e0) register_command dump_historic_ops hook 0x357a750
|
|
-88> 2013-08-15 20:50:24.872410 7f6c5584c7c0 5 asok(0x35800e0) register_command dump_op_pq_state hook 0x357a750
|
|
-87> 2013-08-15 20:50:24.872416 7f6c5584c7c0 5 asok(0x35800e0) register_command setomapval hook 0x376ee80
|
|
-86> 2013-08-15 20:50:24.872422 7f6c5584c7c0 5 asok(0x35800e0) register_command rmomapkey hook 0x376ee80
|
|
-85> 2013-08-15 20:50:24.872427 7f6c5584c7c0 5 asok(0x35800e0) register_command setomapheader hook 0x376ee80
|
|
-84> 2013-08-15 20:50:24.872442 7f6c5584c7c0 5 asok(0x35800e0) register_command getomap hook 0x376ee80
|
|
-83> 2013-08-15 20:50:24.872447 7f6c5584c7c0 5 asok(0x35800e0) register_command truncobj hook 0x376ee80
|
|
-82> 2013-08-15 20:50:24.872423 7f6c44a0d700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d4a00 sd=31 :6811 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:60469/0
|
|
-81> 2013-08-15 20:50:24.872446 7f6c40f03700 5 osd.8 63 heartbeat: osd_stat(1022 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-80> 2013-08-15 20:50:24.872451 7f6c4480b700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d4780 sd=27 :6811 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.1:60468/0
|
|
-79> 2013-08-15 20:50:24.872449 7f6c5584c7c0 5 asok(0x35800e0) register_command injectdataerr hook 0x376ee80
|
|
-78> 2013-08-15 20:50:24.872474 7f6c5584c7c0 5 asok(0x35800e0) register_command injectmdataerr hook 0x376ee80
|
|
-77> 2013-08-15 20:50:24.872561 7f6c5584c7c0 10 monclient(hunting): _reopen_session
|
|
-76> 2013-08-15 20:50:24.872587 7f6c5584c7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-a con 0x37d66e0 addr 9.4.196.4:6789/0
|
|
-75> 2013-08-15 20:50:24.872614 7f6c5584c7c0 10 monclient(hunting): _send_mon_message to mon.noname-a at 9.4.196.4:6789/0
|
|
-74> 2013-08-15 20:50:24.872621 7f6c5584c7c0 1 -- 0.0.0.0:6809/8147 --> 9.4.196.4:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x358cb40 con 0x37d66e0
|
|
-73> 2013-08-15 20:50:24.872636 7f6c5584c7c0 10 monclient(hunting): renew_subs
|
|
-72> 2013-08-15 20:50:24.873052 7f6c44a0d700 1 -- 9.4.196.3:6809/8147 learned my addr 9.4.196.3:6809/8147
|
|
-71> 2013-08-15 20:50:24.873318 7f6c48214700 10 monclient(hunting): renew_subs
|
|
-70> 2013-08-15 20:50:24.873725 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 1 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x36041e0 con 0x37d66e0
|
|
-69> 2013-08-15 20:50:24.873747 7f6c48214700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-68> 2013-08-15 20:50:24.873764 7f6c48214700 10 monclient(hunting): got monmap 1, mon.noname-a is now rank -1
|
|
-67> 2013-08-15 20:50:24.873769 7f6c48214700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-66> 2013-08-15 20:50:24.873803 7f6c48214700 1 monclient(hunting): found mon.ceph-dmon1
|
|
-65> 2013-08-15 20:50:24.873812 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (1192085734 0 0) 0x37e2200 con 0x37d66e0
|
|
-64> 2013-08-15 20:50:24.873838 7f6c48214700 10 monclient: my global_id is 4318
|
|
-63> 2013-08-15 20:50:24.873965 7f6c48214700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-62> 2013-08-15 20:50:24.873975 7f6c48214700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x358dd40 con 0x37d66e0
|
|
-61> 2013-08-15 20:50:24.874606 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3909889861 0 0) 0x37e2000 con 0x37d66e0
|
|
-60> 2013-08-15 20:50:24.874686 7f6c48214700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-59> 2013-08-15 20:50:24.874694 7f6c48214700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x37e6240 con 0x37d66e0
|
|
-58> 2013-08-15 20:50:24.875373 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (1824883726 0 0) 0x37e2400 con 0x37d66e0
|
|
-57> 2013-08-15 20:50:24.875445 7f6c48214700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-56> 2013-08-15 20:50:24.875452 7f6c48214700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x35c2380 con 0x37d66e0
|
|
-55> 2013-08-15 20:50:24.875475 7f6c48214700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-54> 2013-08-15 20:50:24.875482 7f6c48214700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x35c2540 con 0x37d66e0
|
|
-53> 2013-08-15 20:50:24.875501 7f6c48214700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:54.875500)
|
|
-52> 2013-08-15 20:50:24.875513 7f6c48214700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-51> 2013-08-15 20:50:24.875520 7f6c48214700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x37e6000 con 0x37d66e0
|
|
-50> 2013-08-15 20:50:24.875569 7f6c5584c7c0 5 monclient: authenticate success, global_id 4318
|
|
-49> 2013-08-15 20:50:24.875582 7f6c5584c7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:54.875581)
|
|
-48> 2013-08-15 20:50:24.875868 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 5 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x36045a0 con 0x37d66e0
|
|
-47> 2013-08-15 20:50:24.875882 7f6c48214700 10 monclient: handle_monmap mon_map v1
|
|
-46> 2013-08-15 20:50:24.875896 7f6c48214700 10 monclient: got monmap 1, mon.ceph-dmon1 is now rank 1
|
|
-45> 2013-08-15 20:50:24.875899 7f6c48214700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-44> 2013-08-15 20:50:24.875925 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x35c2380 con 0x37d66e0
|
|
-43> 2013-08-15 20:50:24.875934 7f6c48214700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:24.872639 renew after 2013-08-15 20:52:54.872639
|
|
-42> 2013-08-15 20:50:24.875949 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 7 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x36043c0 con 0x37d66e0
|
|
-41> 2013-08-15 20:50:24.875958 7f6c48214700 10 monclient: handle_monmap mon_map v1
|
|
-40> 2013-08-15 20:50:24.875967 7f6c48214700 10 monclient: got monmap 1, mon.ceph-dmon1 is now rank 1
|
|
-39> 2013-08-15 20:50:24.875970 7f6c48214700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-38> 2013-08-15 20:50:24.875989 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x35c2700 con 0x37d66e0
|
|
-37> 2013-08-15 20:50:24.875999 7f6c48214700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-36> 2013-08-15 20:50:24.876051 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (1955389391 0 0) 0x37e2600 con 0x37d66e0
|
|
-35> 2013-08-15 20:50:24.876094 7f6c48214700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:54.876094)
|
|
-34> 2013-08-15 20:50:24.876133 7f6c5584c7c0 10 monclient: wait_auth_rotating done
|
|
-33> 2013-08-15 20:50:24.876701 7f6c5584c7c0 10 monclient: get_version osdmap
|
|
-32> 2013-08-15 20:50:24.876712 7f6c5584c7c0 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-31> 2013-08-15 20:50:24.876719 7f6c5584c7c0 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x35e89c0 con 0x37d66e0
|
|
-30> 2013-08-15 20:50:24.877046 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 10 ==== mon_check_map_ack(handle=1 version=83) v2 ==== 24+0+0 (2922730777 0 0) 0x35c2540 con 0x37d66e0
|
|
-29> 2013-08-15 20:50:24.877225 7f6c43f09700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-28> 2013-08-15 20:50:24.877248 7f6c43f09700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- osd_boot(osd.8 booted 0 v63) v3 -- ?+0 0x3806000 con 0x37d66e0
|
|
-27> 2013-08-15 20:50:25.073010 7f6c3eef8700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d5900 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60470/0
|
|
-26> 2013-08-15 20:50:25.073051 7f6c3edf7700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d5680 sd=34 :6811 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:60471/0
|
|
-25> 2013-08-15 20:50:25.239188 7f6c3edf7700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d5400 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60478/0
|
|
-24> 2013-08-15 20:50:25.239672 7f6c3edf7700 1 -- 9.4.196.3:6811/8147 >> :/0 pipe(0x37d5180 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60479/0
|
|
-23> 2013-08-15 20:50:26.257375 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 11 ==== osd_map(84..85 src has 1..85) v3 ==== 2729+0+0 (424509592 0 0) 0x37e7200 con 0x37d66e0
|
|
-22> 2013-08-15 20:50:26.257424 7f6c48214700 3 osd.8 63 handle_osd_map epochs [84,85], i have 63, src has [1,85]
|
|
-21> 2013-08-15 20:50:26.257433 7f6c48214700 10 monclient: renew_subs
|
|
-20> 2013-08-15 20:50:26.257438 7f6c48214700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-19> 2013-08-15 20:50:26.257441 7f6c48214700 1 -- 9.4.196.3:6809/8147 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x35c2700 con 0x37d66e0
|
|
-18> 2013-08-15 20:50:26.258798 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 12 ==== osd_map(64..85 src has 1..85) v3 ==== 18379+0+0 (600021662 0 0) 0x37e7680 con 0x37d66e0
|
|
-17> 2013-08-15 20:50:26.258820 7f6c48214700 3 osd.8 63 handle_osd_map epochs [64,85], i have 63, src has [1,85]
|
|
-16> 2013-08-15 20:50:26.262019 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-15> 2013-08-15 20:50:26.262031 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-14> 2013-08-15 20:50:26.262047 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-13> 2013-08-15 20:50:26.262063 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-12> 2013-08-15 20:50:26.262079 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-11> 2013-08-15 20:50:26.262084 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-10> 2013-08-15 20:50:26.262107 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-9> 2013-08-15 20:50:26.262112 7f6c48214700 1 -- 9.4.196.3:6811/8147 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-8> 2013-08-15 20:50:26.262118 7f6c48214700 1 osd.8 85 state: booting -> active
|
|
-7> 2013-08-15 20:50:26.262440 7f6c48214700 1 -- 9.4.196.3:6809/8147 <== mon.1 9.4.196.4:6789/0 13 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x35c2c40 con 0x37d66e0
|
|
-6> 2013-08-15 20:50:26.262454 7f6c48214700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:26.257435 renew after 2013-08-15 20:52:56.257435
|
|
-5> 2013-08-15 20:50:26.262994 7f6c4e220700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-4> 2013-08-15 20:50:26.263012 7f6c4e220700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
-3> 2013-08-15 20:50:26.263251 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.12b( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 85/85/56) [3,8] r=1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:26.263252 7f6c42f07700 1 osd.8 pg_epoch: 85 pg[3.69( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 85/85/56) [1,8] r=1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:26.263366 7f6c43708700 1 -- 9.4.196.3:6811/8147 --> 9.4.196.1:6810/7799 -- pg_notify(3.12b(9) epoch 85) v4 -- ?+0 0x35c2e00 con 0x37d78c0
|
|
0> 2013-08-15 20:50:26.263907 7f6c4e220700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f6c4e220700 time 2013-08-15 20:50:26.263017
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f6c55228f8e]
|
|
5: (clone()+0x6d) [0x7f6c534dae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:26.266688 7f6c4e220700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f6c4e220700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f6c55230bd0]
|
|
3: (gsignal()+0x37) [0x7f6c53418037]
|
|
4: (abort()+0x148) [0x7f6c5341b698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f6c53d24e8d]
|
|
6: (()+0x5ef76) [0x7f6c53d22f76]
|
|
7: (()+0x5efa3) [0x7f6c53d22fa3]
|
|
8: (()+0x5f1de) [0x7f6c53d231de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f6c55228f8e]
|
|
14: (clone()+0x6d) [0x7f6c534dae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-11> 2013-08-15 20:50:26.264094 7f6c42f07700 1 osd.8 pg_epoch: 85 pg[3.1ea( v 63'13 (0'0,63'13] local-les=62 n=9 ec=56 les/c 62/62 85/85/84) [8,2]/[2,0] r=-1 lpr=85 pi=61-84/9 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-10> 2013-08-15 20:50:26.264176 7f6c42f07700 1 -- 9.4.196.3:6811/8147 --> 9.4.196.1:6801/5083 -- pg_notify(3.69(9) epoch 85) v4 -- ?+0 0x35c3340 con 0x37d6420
|
|
-9> 2013-08-15 20:50:26.264202 7f6c42f07700 1 -- 9.4.196.3:6811/8147 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ea(9) epoch 85) v4 -- ?+0 0x35c3180 con 0x37d7600
|
|
-8> 2013-08-15 20:50:26.264239 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.125( v 63'23 (0'0,63'23] local-les=62 n=7 ec=56 les/c 62/62 85/85/56) [1,8] r=1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-7> 2013-08-15 20:50:26.264851 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.66( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 85/85/84) [8,0]/[0,3] r=-1 lpr=85 pi=61-84/9 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-6> 2013-08-15 20:50:26.265274 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.11c( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 85/85/56) [3,8] r=1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:26.265720 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.11d( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 85/85/84) [8,0]/[0,1] r=-1 lpr=85 pi=61-84/9 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:26.266212 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.58( v 63'21 (0'0,63'21] local-les=62 n=5 ec=56 les/c 62/62 85/85/56) [3,8] r=1 lpr=85 pi=56-84/9 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:26.266397 7f6c47212700 1 -- 9.4.196.3:6811/8147 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(84..85 src has 1..85) v3 ==== 2729+0+0 (424509592 0 0) 0x35d6000 con 0x37d6420
|
|
-2> 2013-08-15 20:50:26.266429 7f6c47212700 3 osd.8 85 handle_osd_map epochs [84,85], i have 85, src has [1,85]
|
|
-1> 2013-08-15 20:50:26.266601 7f6c43708700 1 osd.8 pg_epoch: 85 pg[3.118( v 63'17 (0'0,63'17] local-les=62 n=7 ec=56 les/c 62/62 85/85/84) [8,3]/[3,1] r=-1 lpr=85 pi=61-84/9 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:26.266688 7f6c4e220700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f6c4e220700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f6c55230bd0]
|
|
3: (gsignal()+0x37) [0x7f6c53418037]
|
|
4: (abort()+0x148) [0x7f6c5341b698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f6c53d24e8d]
|
|
6: (()+0x5ef76) [0x7f6c53d22f76]
|
|
7: (()+0x5efa3) [0x7f6c53d22fa3]
|
|
8: (()+0x5f1de) [0x7f6c53d231de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f6c55228f8e]
|
|
14: (clone()+0x6d) [0x7f6c534dae1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:26.477883 7f2b416707c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8539
|
|
2013-08-15 20:50:26.478504 7f2b416707c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:26.543072 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:26.543091 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:26.543397 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:26.560983 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:26.561080 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:26.597252 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:26.598105 7f2b416707c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:26.598120 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:26.660737 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:26.661302 7f2b416707c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
2013-08-15 20:50:26.698706 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:26.698723 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:26.699011 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:26.728555 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:26.728627 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:26.752757 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:26.753319 7f2b416707c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:26.753350 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:26.813469 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:27.845539 7f2b3a044700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:27.845556 7f2b3a044700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
2013-08-15 20:50:27.846501 7f2b3a044700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f2b3a044700 time 2013-08-15 20:50:27.845561
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f2b4104cf8e]
|
|
5: (clone()+0x6d) [0x7f2b3f2fee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-403> 2013-08-15 20:50:26.476474 7f2b416707c0 5 asok(0x33fa0e0) register_command perfcounters_dump hook 0x33f4010
|
|
-402> 2013-08-15 20:50:26.476507 7f2b416707c0 5 asok(0x33fa0e0) register_command 1 hook 0x33f4010
|
|
-401> 2013-08-15 20:50:26.476509 7f2b416707c0 5 asok(0x33fa0e0) register_command perf dump hook 0x33f4010
|
|
-400> 2013-08-15 20:50:26.476515 7f2b416707c0 5 asok(0x33fa0e0) register_command perfcounters_schema hook 0x33f4010
|
|
-399> 2013-08-15 20:50:26.476522 7f2b416707c0 5 asok(0x33fa0e0) register_command 2 hook 0x33f4010
|
|
-398> 2013-08-15 20:50:26.476524 7f2b416707c0 5 asok(0x33fa0e0) register_command perf schema hook 0x33f4010
|
|
-397> 2013-08-15 20:50:26.476528 7f2b416707c0 5 asok(0x33fa0e0) register_command config show hook 0x33f4010
|
|
-396> 2013-08-15 20:50:26.476533 7f2b416707c0 5 asok(0x33fa0e0) register_command config set hook 0x33f4010
|
|
-395> 2013-08-15 20:50:26.476535 7f2b416707c0 5 asok(0x33fa0e0) register_command log flush hook 0x33f4010
|
|
-394> 2013-08-15 20:50:26.476537 7f2b416707c0 5 asok(0x33fa0e0) register_command log dump hook 0x33f4010
|
|
-393> 2013-08-15 20:50:26.476543 7f2b416707c0 5 asok(0x33fa0e0) register_command log reopen hook 0x33f4010
|
|
-392> 2013-08-15 20:50:26.477883 7f2b416707c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8539
|
|
-391> 2013-08-15 20:50:26.478504 7f2b416707c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-390> 2013-08-15 20:50:26.478640 7f2b416707c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6809/8539 need_addr=1
|
|
-389> 2013-08-15 20:50:26.478665 7f2b416707c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-388> 2013-08-15 20:50:26.478674 7f2b416707c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6811/8539 need_addr=0
|
|
-387> 2013-08-15 20:50:26.478695 7f2b416707c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-386> 2013-08-15 20:50:26.478700 7f2b416707c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6821/8539 need_addr=0
|
|
-385> 2013-08-15 20:50:26.480631 7f2b416707c0 5 asok(0x33fa0e0) init /var/run/ceph/ceph-osd.8.asok
|
|
-384> 2013-08-15 20:50:26.480646 7f2b416707c0 5 asok(0x33fa0e0) bind_and_listen /var/run/ceph/ceph-osd.8.asok
|
|
-383> 2013-08-15 20:50:26.480679 7f2b416707c0 5 asok(0x33fa0e0) register_command 0 hook 0x33f20b8
|
|
-382> 2013-08-15 20:50:26.480684 7f2b416707c0 5 asok(0x33fa0e0) register_command version hook 0x33f20b8
|
|
-381> 2013-08-15 20:50:26.480691 7f2b416707c0 5 asok(0x33fa0e0) register_command git_version hook 0x33f20b8
|
|
-380> 2013-08-15 20:50:26.480698 7f2b416707c0 5 asok(0x33fa0e0) register_command help hook 0x33f40d0
|
|
-379> 2013-08-15 20:50:26.480727 7f2b3d04a700 5 asok(0x33fa0e0) entry start
|
|
-378> 2013-08-15 20:50:26.543072 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-377> 2013-08-15 20:50:26.543091 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-376> 2013-08-15 20:50:26.543397 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-375> 2013-08-15 20:50:26.560983 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-374> 2013-08-15 20:50:26.561080 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-373> 2013-08-15 20:50:26.597252 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-372> 2013-08-15 20:50:26.598068 7f2b416707c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-371> 2013-08-15 20:50:26.598105 7f2b416707c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-370> 2013-08-15 20:50:26.598120 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-369> 2013-08-15 20:50:26.603885 7f2b416707c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-368> 2013-08-15 20:50:26.607871 7f2b416707c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-367> 2013-08-15 20:50:26.611957 7f2b416707c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-366> 2013-08-15 20:50:26.615975 7f2b416707c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-365> 2013-08-15 20:50:26.620004 7f2b416707c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-364> 2013-08-15 20:50:26.623962 7f2b416707c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-363> 2013-08-15 20:50:26.628211 7f2b416707c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-362> 2013-08-15 20:50:26.632296 7f2b416707c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-361> 2013-08-15 20:50:26.636356 7f2b416707c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-360> 2013-08-15 20:50:26.640319 7f2b416707c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-359> 2013-08-15 20:50:26.644410 7f2b416707c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-358> 2013-08-15 20:50:26.648498 7f2b416707c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-357> 2013-08-15 20:50:26.652506 7f2b416707c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-356> 2013-08-15 20:50:26.656536 7f2b416707c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-355> 2013-08-15 20:50:26.660627 7f2b416707c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-354> 2013-08-15 20:50:26.660668 7f2b416707c0 2 journal No further valid entries found, journal is most likely valid
|
|
-353> 2013-08-15 20:50:26.660675 7f2b416707c0 2 journal No further valid entries found, journal is most likely valid
|
|
-352> 2013-08-15 20:50:26.660686 7f2b416707c0 3 journal journal_replay: end of journal, done.
|
|
-351> 2013-08-15 20:50:26.660737 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-350> 2013-08-15 20:50:26.661167 7f2b3a044700 1 FileStore::op_tp worker finish
|
|
-349> 2013-08-15 20:50:26.661185 7f2b3a845700 1 FileStore::op_tp worker finish
|
|
-348> 2013-08-15 20:50:26.661302 7f2b416707c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
-347> 2013-08-15 20:50:26.661657 7f2b416707c0 10 monclient(hunting): build_initial_monmap
|
|
-346> 2013-08-15 20:50:26.661708 7f2b416707c0 5 adding auth protocol: cephx
|
|
-345> 2013-08-15 20:50:26.661715 7f2b416707c0 5 adding auth protocol: cephx
|
|
-344> 2013-08-15 20:50:26.661801 7f2b416707c0 1 -- 0.0.0.0:6809/8539 messenger.start
|
|
-343> 2013-08-15 20:50:26.661832 7f2b416707c0 1 -- :/0 messenger.start
|
|
-342> 2013-08-15 20:50:26.661856 7f2b416707c0 1 -- 9.4.196.3:6821/8539 messenger.start
|
|
-341> 2013-08-15 20:50:26.661869 7f2b416707c0 1 -- 9.4.196.3:6811/8539 messenger.start
|
|
-340> 2013-08-15 20:50:26.661964 7f2b416707c0 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
|
|
-339> 2013-08-15 20:50:26.698706 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-338> 2013-08-15 20:50:26.698723 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-337> 2013-08-15 20:50:26.699011 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-336> 2013-08-15 20:50:26.728555 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-335> 2013-08-15 20:50:26.728627 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-334> 2013-08-15 20:50:26.752757 7f2b416707c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-333> 2013-08-15 20:50:26.753287 7f2b416707c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-332> 2013-08-15 20:50:26.753319 7f2b416707c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-331> 2013-08-15 20:50:26.753350 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-330> 2013-08-15 20:50:26.757510 7f2b416707c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-329> 2013-08-15 20:50:26.761536 7f2b416707c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-328> 2013-08-15 20:50:26.765614 7f2b416707c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-327> 2013-08-15 20:50:26.769617 7f2b416707c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-326> 2013-08-15 20:50:26.773552 7f2b416707c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-325> 2013-08-15 20:50:26.777504 7f2b416707c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-324> 2013-08-15 20:50:26.781586 7f2b416707c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-323> 2013-08-15 20:50:26.785536 7f2b416707c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-322> 2013-08-15 20:50:26.789455 7f2b416707c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-321> 2013-08-15 20:50:26.793409 7f2b416707c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-320> 2013-08-15 20:50:26.797466 7f2b416707c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-319> 2013-08-15 20:50:26.801459 7f2b416707c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-318> 2013-08-15 20:50:26.805425 7f2b416707c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-317> 2013-08-15 20:50:26.809392 7f2b416707c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-316> 2013-08-15 20:50:26.813398 7f2b416707c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-315> 2013-08-15 20:50:26.813430 7f2b416707c0 2 journal No further valid entries found, journal is most likely valid
|
|
-314> 2013-08-15 20:50:26.813435 7f2b416707c0 2 journal No further valid entries found, journal is most likely valid
|
|
-313> 2013-08-15 20:50:26.813436 7f2b416707c0 3 journal journal_replay: end of journal, done.
|
|
-312> 2013-08-15 20:50:26.813469 7f2b416707c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-311> 2013-08-15 20:50:26.813664 7f2b416707c0 2 osd.8 0 boot
|
|
-310> 2013-08-15 20:50:26.839940 7f2b416707c0 2 osd.8 63 superblock: i am osd.8
|
|
-309> 2013-08-15 20:50:26.840005 7f2b416707c0 1 accepter.accepter.start
|
|
-308> 2013-08-15 20:50:26.840040 7f2b416707c0 1 accepter.accepter.start
|
|
-307> 2013-08-15 20:50:26.840087 7f2b416707c0 1 accepter.accepter.start
|
|
-306> 2013-08-15 20:50:26.840127 7f2b416707c0 10 monclient(hunting): init
|
|
-305> 2013-08-15 20:50:26.840165 7f2b416707c0 5 adding auth protocol: cephx
|
|
-304> 2013-08-15 20:50:26.840175 7f2b416707c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-303> 2013-08-15 20:50:26.840319 7f2b41602700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e000 sd=27 :6811 s=0 pgs=0 cs=0 l=0).accept sd=27 9.4.196.1:60582/0
|
|
-302> 2013-08-15 20:50:26.840280 7f2b30831700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364e280 sd=28 :6821 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.1:51958/0
|
|
-301> 2013-08-15 20:50:26.840283 7f2b3062f700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364ec80 sd=30 :6821 s=0 pgs=0 cs=0 l=0).accept sd=30 9.4.196.1:51960/0
|
|
-300> 2013-08-15 20:50:26.840348 7f2b30730700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e780 sd=29 :6811 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.3:57895/0
|
|
-299> 2013-08-15 20:50:26.840403 7f2b3052e700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e500 sd=31 :6811 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.3:57894/0
|
|
-298> 2013-08-15 20:50:26.840419 7f2b3032c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f400 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60584/0
|
|
-297> 2013-08-15 20:50:26.840425 7f2b3042d700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364ea00 sd=32 :6821 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:51965/0
|
|
-296> 2013-08-15 20:50:26.840524 7f2b416707c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-8/keyring
|
|
-295> 2013-08-15 20:50:26.840509 7f2b3012a700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364ef00 sd=35 :6811 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:60586/0
|
|
-294> 2013-08-15 20:50:26.840523 7f2b3022b700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f180 sd=34 :6811 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:60585/0
|
|
-293> 2013-08-15 20:50:26.840573 7f2b30029700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364fb80 sd=36 :6811 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:60587/0
|
|
-292> 2013-08-15 20:50:26.840722 7f2b416707c0 5 asok(0x33fa0e0) register_command dump_ops_in_flight hook 0x33f4750
|
|
-291> 2013-08-15 20:50:26.840733 7f2b416707c0 5 asok(0x33fa0e0) register_command dump_historic_ops hook 0x33f4750
|
|
-290> 2013-08-15 20:50:26.840738 7f2b416707c0 5 asok(0x33fa0e0) register_command dump_op_pq_state hook 0x33f4750
|
|
-289> 2013-08-15 20:50:26.840743 7f2b416707c0 5 asok(0x33fa0e0) register_command setomapval hook 0x35e8e80
|
|
-288> 2013-08-15 20:50:26.840749 7f2b416707c0 5 asok(0x33fa0e0) register_command rmomapkey hook 0x35e8e80
|
|
-287> 2013-08-15 20:50:26.840752 7f2b416707c0 5 asok(0x33fa0e0) register_command setomapheader hook 0x35e8e80
|
|
-286> 2013-08-15 20:50:26.840757 7f2b416707c0 5 asok(0x33fa0e0) register_command getomap hook 0x35e8e80
|
|
-285> 2013-08-15 20:50:26.840760 7f2b416707c0 5 asok(0x33fa0e0) register_command truncobj hook 0x35e8e80
|
|
-284> 2013-08-15 20:50:26.840764 7f2b416707c0 5 asok(0x33fa0e0) register_command injectdataerr hook 0x35e8e80
|
|
-283> 2013-08-15 20:50:26.840768 7f2b416707c0 5 asok(0x33fa0e0) register_command injectmdataerr hook 0x35e8e80
|
|
-282> 2013-08-15 20:50:26.840766 7f2b2c721700 5 osd.8 63 heartbeat: osd_stat(1022 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-281> 2013-08-15 20:50:26.840782 7f2b3042d700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364e500 sd=28 :6821 s=0 pgs=0 cs=0 l=0).accept sd=28 9.4.196.1:51968/0
|
|
-280> 2013-08-15 20:50:26.840832 7f2b416707c0 10 monclient(hunting): _reopen_session
|
|
-279> 2013-08-15 20:50:26.840836 7f2b30730700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f900 sd=29 :6811 s=0 pgs=0 cs=0 l=0).accept sd=29 9.4.196.1:60592/0
|
|
-278> 2013-08-15 20:50:26.840844 7f2b30831700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364e780 sd=32 :6821 s=0 pgs=0 cs=0 l=0).accept sd=32 9.4.196.1:51970/0
|
|
-277> 2013-08-15 20:50:26.840842 7f2b3052e700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f680 sd=31 :6811 s=0 pgs=0 cs=0 l=0).accept sd=31 9.4.196.1:60594/0
|
|
-276> 2013-08-15 20:50:26.840910 7f2b416707c0 10 monclient(hunting): _pick_new_mon picked mon.noname-b con 0x3667e40 addr 9.4.196.5:6789/0
|
|
-275> 2013-08-15 20:50:26.840927 7f2b416707c0 10 monclient(hunting): _send_mon_message to mon.noname-b at 9.4.196.5:6789/0
|
|
-274> 2013-08-15 20:50:26.840934 7f2b416707c0 1 -- 0.0.0.0:6809/8539 --> 9.4.196.5:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x3406b40 con 0x3667e40
|
|
-273> 2013-08-15 20:50:26.840948 7f2b416707c0 10 monclient(hunting): renew_subs
|
|
-272> 2013-08-15 20:50:26.841324 7f2b2af1e700 1 -- 9.4.196.3:6809/8539 learned my addr 9.4.196.3:6809/8539
|
|
-271> 2013-08-15 20:50:26.841573 7f2b34038700 10 monclient(hunting): renew_subs
|
|
-270> 2013-08-15 20:50:26.842059 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x347c1e0 con 0x3667e40
|
|
-269> 2013-08-15 20:50:26.842082 7f2b34038700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-268> 2013-08-15 20:50:26.842101 7f2b34038700 10 monclient(hunting): got monmap 1, mon.noname-b is now rank -1
|
|
-267> 2013-08-15 20:50:26.842105 7f2b34038700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-266> 2013-08-15 20:50:26.842133 7f2b34038700 1 monclient(hunting): found mon.ceph-dmon2
|
|
-265> 2013-08-15 20:50:26.842145 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (560101738 0 0) 0x3656200 con 0x3667e40
|
|
-264> 2013-08-15 20:50:26.842170 7f2b34038700 10 monclient: my global_id is 4322
|
|
-263> 2013-08-15 20:50:26.842296 7f2b34038700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-262> 2013-08-15 20:50:26.842306 7f2b34038700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x3407d40 con 0x3667e40
|
|
-261> 2013-08-15 20:50:26.843051 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (3083947143 0 0) 0x3656000 con 0x3667e40
|
|
-260> 2013-08-15 20:50:26.843121 7f2b34038700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-259> 2013-08-15 20:50:26.843129 7f2b34038700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x3662240 con 0x3667e40
|
|
-258> 2013-08-15 20:50:26.844173 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (547848044 0 0) 0x3656400 con 0x3667e40
|
|
-257> 2013-08-15 20:50:26.844234 7f2b34038700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-256> 2013-08-15 20:50:26.844242 7f2b34038700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x343c380 con 0x3667e40
|
|
-255> 2013-08-15 20:50:26.844265 7f2b34038700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-254> 2013-08-15 20:50:26.844270 7f2b34038700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x343c540 con 0x3667e40
|
|
-253> 2013-08-15 20:50:26.844287 7f2b34038700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:56.844286)
|
|
-252> 2013-08-15 20:50:26.844297 7f2b34038700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-251> 2013-08-15 20:50:26.844304 7f2b34038700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x3662000 con 0x3667e40
|
|
-250> 2013-08-15 20:50:26.844325 7f2b416707c0 5 monclient: authenticate success, global_id 4322
|
|
-249> 2013-08-15 20:50:26.844336 7f2b416707c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:56.844335)
|
|
-248> 2013-08-15 20:50:26.844788 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x347c5a0 con 0x3667e40
|
|
-247> 2013-08-15 20:50:26.844803 7f2b34038700 10 monclient: handle_monmap mon_map v1
|
|
-246> 2013-08-15 20:50:26.844817 7f2b34038700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-245> 2013-08-15 20:50:26.844821 7f2b34038700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-244> 2013-08-15 20:50:26.844855 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x343c380 con 0x3667e40
|
|
-243> 2013-08-15 20:50:26.844865 7f2b34038700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:26.840952 renew after 2013-08-15 20:52:56.840952
|
|
-242> 2013-08-15 20:50:26.844945 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x347c3c0 con 0x3667e40
|
|
-241> 2013-08-15 20:50:26.844963 7f2b34038700 10 monclient: handle_monmap mon_map v1
|
|
-240> 2013-08-15 20:50:26.844976 7f2b34038700 10 monclient: got monmap 1, mon.ceph-dmon2 is now rank 2
|
|
-239> 2013-08-15 20:50:26.844980 7f2b34038700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-238> 2013-08-15 20:50:26.844996 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x343c700 con 0x3667e40
|
|
-237> 2013-08-15 20:50:26.845005 7f2b34038700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-236> 2013-08-15 20:50:26.845217 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (656438434 0 0) 0x3656600 con 0x3667e40
|
|
-235> 2013-08-15 20:50:26.845262 7f2b34038700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:56.845262)
|
|
-234> 2013-08-15 20:50:26.845292 7f2b416707c0 10 monclient: wait_auth_rotating done
|
|
-233> 2013-08-15 20:50:26.846048 7f2b416707c0 10 monclient: get_version osdmap
|
|
-232> 2013-08-15 20:50:26.846054 7f2b416707c0 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-231> 2013-08-15 20:50:26.846061 7f2b416707c0 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x34629c0 con 0x3667e40
|
|
-230> 2013-08-15 20:50:26.846473 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 10 ==== mon_check_map_ack(handle=1 version=85) v2 ==== 24+0+0 (2311427327 0 0) 0x343c540 con 0x3667e40
|
|
-229> 2013-08-15 20:50:26.846516 7f2b2f727700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-228> 2013-08-15 20:50:26.846529 7f2b2f727700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- osd_boot(osd.8 booted 0 v63) v3 -- ?+0 0x3682000 con 0x3667e40
|
|
-227> 2013-08-15 20:50:26.867999 7f2b2ad1c700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364ea00 sd=33 :6821 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:51973/0
|
|
-226> 2013-08-15 20:50:26.868450 7f2b2ad1c700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364f180 sd=33 :6821 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:51975/0
|
|
-225> 2013-08-15 20:50:26.968770 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e000 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60599/0
|
|
-224> 2013-08-15 20:50:26.974953 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f180 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60600/0
|
|
-223> 2013-08-15 20:50:26.976868 7f2b2ad1c700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364f400 sd=33 :6821 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:51978/0
|
|
-222> 2013-08-15 20:50:26.978864 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e000 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57903/0
|
|
-221> 2013-08-15 20:50:26.980184 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f180 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57904/0
|
|
-220> 2013-08-15 20:50:26.980376 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f400 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57905/0
|
|
-219> 2013-08-15 20:50:26.982700 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e000 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57906/0
|
|
-218> 2013-08-15 20:50:27.041301 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f180 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:60602/0
|
|
-217> 2013-08-15 20:50:27.041317 7f2b2ac1b700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364e500 sd=34 :6821 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:51979/0
|
|
-216> 2013-08-15 20:50:27.041354 7f2b2ab1a700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364ea00 sd=35 :6821 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:51981/0
|
|
-215> 2013-08-15 20:50:27.238581 7f2b2ab1a700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364e000 sd=33 :6821 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:51982/0
|
|
-214> 2013-08-15 20:50:27.239039 7f2b2ab1a700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364f180 sd=33 :6821 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:51984/0
|
|
-213> 2013-08-15 20:50:27.240696 7f2b2ab1a700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f400 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57907/0
|
|
-212> 2013-08-15 20:50:27.240755 7f2b2ac1b700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364e000 sd=34 :6811 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:60608/0
|
|
-211> 2013-08-15 20:50:27.240807 7f2b2ad1c700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x364f180 sd=35 :6811 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.3:57908/0
|
|
-210> 2013-08-15 20:50:27.241024 7f2b2a918700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x368d400 sd=36 :6811 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:60610/0
|
|
-209> 2013-08-15 20:50:27.241024 7f2b2aa19700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364e500 sd=37 :6821 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:51986/0
|
|
-208> 2013-08-15 20:50:27.383118 7f2b2a918700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x368d180 sd=33 :6811 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:57913/0
|
|
-207> 2013-08-15 20:50:27.752072 7f2b2a918700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364ea00 sd=33 :6821 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:52049/0
|
|
-206> 2013-08-15 20:50:27.752149 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(86..87 src has 1..87) v3 ==== 5858+0+0 (2545451241 0 0) 0x3663200 con 0x3650000
|
|
-205> 2013-08-15 20:50:27.752190 7f2b33036700 3 osd.8 63 handle_osd_map epochs [86,87], i have 63, src has [1,87]
|
|
-204> 2013-08-15 20:50:27.752199 7f2b33036700 10 monclient: renew_subs
|
|
-203> 2013-08-15 20:50:27.752203 7f2b33036700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-202> 2013-08-15 20:50:27.752208 7f2b33036700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343c8c0 con 0x3667e40
|
|
-201> 2013-08-15 20:50:27.752234 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(3.160,3.17f epoch 87) v2 ==== 262+0+0 (4236935764 0 0) 0x347c1e0 con 0x3650000
|
|
-200> 2013-08-15 20:50:27.752249 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.752083, event: header_read, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-199> 2013-08-15 20:50:27.752261 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.752084, event: throttled, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-198> 2013-08-15 20:50:27.752265 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.752201, event: all_read, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-197> 2013-08-15 20:50:27.752269 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.752243, event: dispatched, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-196> 2013-08-15 20:50:27.752273 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.752273, event: waiting_for_osdmap, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-195> 2013-08-15 20:50:27.752291 7f2b33036700 10 monclient: renew_subs
|
|
-194> 2013-08-15 20:50:27.752307 7f2b33036700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-193> 2013-08-15 20:50:27.752314 7f2b33036700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343cc40 con 0x3667e40
|
|
-192> 2013-08-15 20:50:27.752324 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.752324, event: wait for new map, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-191> 2013-08-15 20:50:27.752333 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_query(2.b,3.a epoch 87) v2 ==== 262+0+0 (1639718063 0 0) 0x347c780 con 0x3650000
|
|
-190> 2013-08-15 20:50:27.752341 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.752212, event: header_read, request: pg_query(2.b,3.a epoch 87) v2
|
|
-189> 2013-08-15 20:50:27.752347 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.752213, event: throttled, request: pg_query(2.b,3.a epoch 87) v2
|
|
-188> 2013-08-15 20:50:27.752353 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.752251, event: all_read, request: pg_query(2.b,3.a epoch 87) v2
|
|
-187> 2013-08-15 20:50:27.752358 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.752340, event: dispatched, request: pg_query(2.b,3.a epoch 87) v2
|
|
-186> 2013-08-15 20:50:27.752363 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.752363, event: waiting_for_osdmap, request: pg_query(2.b,3.a epoch 87) v2
|
|
-185> 2013-08-15 20:50:27.752369 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.752369, event: wait for new map, request: pg_query(2.b,3.a epoch 87) v2
|
|
-184> 2013-08-15 20:50:27.753790 7f2b2ad1c700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x368d400 sd=34 :6821 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:52051/0
|
|
-183> 2013-08-15 20:50:27.754063 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.2 9.4.196.1:6807/6887 1 ==== osd_map(86..87 src has 1..87) v3 ==== 5858+0+0 (2545451241 0 0) 0x36638c0 con 0x36662c0
|
|
-182> 2013-08-15 20:50:27.754084 7f2b33036700 3 osd.8 63 handle_osd_map epochs [86,87], i have 63, src has [1,87]
|
|
-181> 2013-08-15 20:50:27.754092 7f2b33036700 10 monclient: renew_subs
|
|
-180> 2013-08-15 20:50:27.754095 7f2b33036700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-179> 2013-08-15 20:50:27.754100 7f2b33036700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343ca80 con 0x3667e40
|
|
-178> 2013-08-15 20:50:27.754118 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.2 9.4.196.1:6807/6887 2 ==== pg_query(3.14d epoch 87) v2 ==== 137+0+0 (2267521637 0 0) 0x347c960 con 0x36662c0
|
|
-177> 2013-08-15 20:50:27.754128 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.754062, event: header_read, request: pg_query(3.14d epoch 87) v2
|
|
-176> 2013-08-15 20:50:27.754147 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.754063, event: throttled, request: pg_query(3.14d epoch 87) v2
|
|
-175> 2013-08-15 20:50:27.754153 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.754092, event: all_read, request: pg_query(3.14d epoch 87) v2
|
|
-174> 2013-08-15 20:50:27.754158 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.754126, event: dispatched, request: pg_query(3.14d epoch 87) v2
|
|
-173> 2013-08-15 20:50:27.754164 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.754163, event: waiting_for_osdmap, request: pg_query(3.14d epoch 87) v2
|
|
-172> 2013-08-15 20:50:27.754169 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.754169, event: wait for new map, request: pg_query(3.14d epoch 87) v2
|
|
-171> 2013-08-15 20:50:27.755334 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(86..87 src has 1..87) v3 ==== 5858+0+0 (2545451241 0 0) 0x3663200 con 0x3666c60
|
|
-170> 2013-08-15 20:50:27.755352 7f2b33036700 3 osd.8 63 handle_osd_map epochs [86,87], i have 63, src has [1,87]
|
|
-169> 2013-08-15 20:50:27.755371 7f2b33036700 10 monclient: renew_subs
|
|
-168> 2013-08-15 20:50:27.755374 7f2b33036700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-167> 2013-08-15 20:50:27.755376 7f2b33036700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343ce00 con 0x3667e40
|
|
-166> 2013-08-15 20:50:27.755433 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_notify(3.18d(6) epoch 87) v4 ==== 826+0+0 (1102052700 0 0) 0x343c8c0 con 0x3666c60
|
|
-165> 2013-08-15 20:50:27.755461 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.755332, event: header_read, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-164> 2013-08-15 20:50:27.755467 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.755333, event: throttled, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-163> 2013-08-15 20:50:27.755471 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.755399, event: all_read, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-162> 2013-08-15 20:50:27.755474 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.755458, event: dispatched, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-161> 2013-08-15 20:50:27.755478 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.755478, event: waiting_for_osdmap, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-160> 2013-08-15 20:50:27.755486 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.755486, event: wait for new map, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-159> 2013-08-15 20:50:27.756192 7f2b2ab1a700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x368d180 sd=35 :6821 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:52053/0
|
|
-158> 2013-08-15 20:50:27.756357 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 3 ==== pg_notify(3.86(6) epoch 87) v4 ==== 826+0+0 (2360900112 0 0) 0x343d180 con 0x3666c60
|
|
-157> 2013-08-15 20:50:27.756374 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.756231, event: header_read, request: pg_notify(3.86(6) epoch 87) v4
|
|
-156> 2013-08-15 20:50:27.756379 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.756232, event: throttled, request: pg_notify(3.86(6) epoch 87) v4
|
|
-155> 2013-08-15 20:50:27.756382 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.756324, event: all_read, request: pg_notify(3.86(6) epoch 87) v4
|
|
-154> 2013-08-15 20:50:27.756385 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.756371, event: dispatched, request: pg_notify(3.86(6) epoch 87) v4
|
|
-153> 2013-08-15 20:50:27.756389 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.756389, event: waiting_for_osdmap, request: pg_notify(3.86(6) epoch 87) v4
|
|
-152> 2013-08-15 20:50:27.756393 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.756393, event: wait for new map, request: pg_notify(3.86(6) epoch 87) v4
|
|
-151> 2013-08-15 20:50:27.756455 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 4 ==== pg_query(3.80,3.8f epoch 87) v2 ==== 262+0+0 (3394511258 0 0) 0x347cb40 con 0x3666c60
|
|
-150> 2013-08-15 20:50:27.756482 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.756394, event: header_read, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-149> 2013-08-15 20:50:27.756488 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.756395, event: throttled, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-148> 2013-08-15 20:50:27.756491 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.756437, event: all_read, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-147> 2013-08-15 20:50:27.756495 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.756480, event: dispatched, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-146> 2013-08-15 20:50:27.756503 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.756503, event: waiting_for_osdmap, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-145> 2013-08-15 20:50:27.756511 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.756511, event: wait for new map, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-144> 2013-08-15 20:50:27.759427 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.2 9.4.196.1:6807/6887 3 ==== pg_notify(3.1a0(6) epoch 87) v4 ==== 826+0+0 (1602759696 0 0) 0x343d340 con 0x36662c0
|
|
-143> 2013-08-15 20:50:27.759452 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.759325, event: header_read, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-142> 2013-08-15 20:50:27.759464 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.759327, event: throttled, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-141> 2013-08-15 20:50:27.759485 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.759374, event: all_read, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-140> 2013-08-15 20:50:27.759495 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.759448, event: dispatched, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-139> 2013-08-15 20:50:27.759504 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.759504, event: waiting_for_osdmap, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-138> 2013-08-15 20:50:27.759511 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.759511, event: wait for new map, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-137> 2013-08-15 20:50:27.759719 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 5 ==== pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2 ==== 512+0+0 (1942191021 0 0) 0x347cf00 con 0x3666c60
|
|
-136> 2013-08-15 20:50:27.759743 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.759631, event: header_read, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-135> 2013-08-15 20:50:27.759766 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.759632, event: throttled, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-134> 2013-08-15 20:50:27.759772 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.759675, event: all_read, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-133> 2013-08-15 20:50:27.759778 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.759740, event: dispatched, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-132> 2013-08-15 20:50:27.759784 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.759784, event: waiting_for_osdmap, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-131> 2013-08-15 20:50:27.759793 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.759793, event: wait for new map, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-130> 2013-08-15 20:50:27.759805 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.2 9.4.196.1:6807/6887 4 ==== pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2 ==== 2387+0+0 (2609206201 0 0) 0x347d2c0 con 0x36662c0
|
|
-129> 2013-08-15 20:50:27.759819 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.759653, event: header_read, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-128> 2013-08-15 20:50:27.759829 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.759655, event: throttled, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-127> 2013-08-15 20:50:27.759838 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.759740, event: all_read, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-126> 2013-08-15 20:50:27.759847 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.759818, event: dispatched, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-125> 2013-08-15 20:50:27.759856 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.759856, event: waiting_for_osdmap, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-124> 2013-08-15 20:50:27.759866 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.759866, event: wait for new map, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-123> 2013-08-15 20:50:27.760141 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(86..87 src has 1..87) v3 ==== 5858+0+0 (2545451241 0 0) 0x36b0000 con 0x3651080
|
|
-122> 2013-08-15 20:50:27.760164 7f2b33036700 3 osd.8 63 handle_osd_map epochs [86,87], i have 63, src has [1,87]
|
|
-121> 2013-08-15 20:50:27.760184 7f2b33036700 10 monclient: renew_subs
|
|
-120> 2013-08-15 20:50:27.760189 7f2b33036700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-119> 2013-08-15 20:50:27.760194 7f2b33036700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343ca80 con 0x3667e40
|
|
-118> 2013-08-15 20:50:27.760212 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(0.9,1.8,2.7,3.6 epoch 87) v2 ==== 512+0+0 (178656435 0 0) 0x347d4a0 con 0x3651080
|
|
-117> 2013-08-15 20:50:27.760223 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.760124, event: header_read, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-116> 2013-08-15 20:50:27.760232 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.760125, event: throttled, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-115> 2013-08-15 20:50:27.760239 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.760185, event: all_read, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-114> 2013-08-15 20:50:27.760245 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.760221, event: dispatched, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-113> 2013-08-15 20:50:27.760252 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.760252, event: waiting_for_osdmap, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-112> 2013-08-15 20:50:27.760260 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.760259, event: wait for new map, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-111> 2013-08-15 20:50:27.760339 7f2b2a716700 1 -- 9.4.196.3:6821/8539 >> :/0 pipe(0x364f400 sd=36 :6821 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.1:52059/0
|
|
-110> 2013-08-15 20:50:27.760397 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.0 9.4.196.1:6804/5992 4 ==== pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2 ==== 2512+0+0 (1855732680 0 0) 0x347c5a0 con 0x3650000
|
|
-109> 2013-08-15 20:50:27.760438 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.760258, event: header_read, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-108> 2013-08-15 20:50:27.760451 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.760259, event: throttled, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-107> 2013-08-15 20:50:27.760462 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.760326, event: all_read, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-106> 2013-08-15 20:50:27.760471 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.760423, event: dispatched, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-105> 2013-08-15 20:50:27.760481 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.760481, event: waiting_for_osdmap, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-104> 2013-08-15 20:50:27.760505 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.760504, event: wait for new map, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-103> 2013-08-15 20:50:27.760638 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 6 ==== pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2 ==== 1012+0+0 (1533877767 0 0) 0x347cd20 con 0x3666c60
|
|
-102> 2013-08-15 20:50:27.760663 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.760563, event: header_read, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-101> 2013-08-15 20:50:27.760674 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.760564, event: throttled, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-100> 2013-08-15 20:50:27.760681 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.760599, event: all_read, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-99> 2013-08-15 20:50:27.760687 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.760660, event: dispatched, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-98> 2013-08-15 20:50:27.760694 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.760694, event: waiting_for_osdmap, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-97> 2013-08-15 20:50:27.760702 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.760702, event: wait for new map, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-96> 2013-08-15 20:50:27.761563 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(86..87 src has 1..87) v3 ==== 5858+0+0 (2545451241 0 0) 0x36b06c0 con 0x3651340
|
|
-95> 2013-08-15 20:50:27.761588 7f2b33036700 3 osd.8 63 handle_osd_map epochs [86,87], i have 63, src has [1,87]
|
|
-94> 2013-08-15 20:50:27.761595 7f2b33036700 10 monclient: renew_subs
|
|
-93> 2013-08-15 20:50:27.761600 7f2b33036700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-92> 2013-08-15 20:50:27.761605 7f2b33036700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343cc40 con 0x3667e40
|
|
-91> 2013-08-15 20:50:27.761624 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.3 9.4.196.1:6810/7799 2 ==== pg_notify(3.23e(6) epoch 87) v4 ==== 826+0+0 (1380794048 0 0) 0x343d500 con 0x3651340
|
|
-90> 2013-08-15 20:50:27.761634 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.761523, event: header_read, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-89> 2013-08-15 20:50:27.761642 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.761524, event: throttled, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-88> 2013-08-15 20:50:27.761648 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.761571, event: all_read, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-87> 2013-08-15 20:50:27.761654 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.761632, event: dispatched, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-86> 2013-08-15 20:50:27.761660 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.761660, event: waiting_for_osdmap, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-85> 2013-08-15 20:50:27.761666 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.761666, event: wait for new map, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-84> 2013-08-15 20:50:27.761851 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.3 9.4.196.1:6810/7799 3 ==== pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2 ==== 762+0+0 (4268587238 0 0) 0x347d680 con 0x3651340
|
|
-83> 2013-08-15 20:50:27.761877 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.761770, event: header_read, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-82> 2013-08-15 20:50:27.761899 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.761771, event: throttled, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-81> 2013-08-15 20:50:27.761905 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.761808, event: all_read, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-80> 2013-08-15 20:50:27.761912 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.761874, event: dispatched, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-79> 2013-08-15 20:50:27.761919 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.761919, event: waiting_for_osdmap, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-78> 2013-08-15 20:50:27.761927 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.761927, event: wait for new map, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-77> 2013-08-15 20:50:27.763504 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.4 9.4.196.1:6813/8739 3 ==== pg_query(3.178 epoch 87) v2 ==== 137+0+0 (3874275074 0 0) 0x347da40 con 0x3651080
|
|
-76> 2013-08-15 20:50:27.763529 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.763445, event: header_read, request: pg_query(3.178 epoch 87) v2
|
|
-75> 2013-08-15 20:50:27.763552 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.763446, event: throttled, request: pg_query(3.178 epoch 87) v2
|
|
-74> 2013-08-15 20:50:27.763557 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.763491, event: all_read, request: pg_query(3.178 epoch 87) v2
|
|
-73> 2013-08-15 20:50:27.763561 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.763525, event: dispatched, request: pg_query(3.178 epoch 87) v2
|
|
-72> 2013-08-15 20:50:27.763569 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.763568, event: waiting_for_osdmap, request: pg_query(3.178 epoch 87) v2
|
|
-71> 2013-08-15 20:50:27.763577 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.763577, event: wait for new map, request: pg_query(3.178 epoch 87) v2
|
|
-70> 2013-08-15 20:50:27.763666 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.3 9.4.196.1:6810/7799 4 ==== pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2 ==== 1012+0+0 (3706359968 0 0) 0x347de00 con 0x3651340
|
|
-69> 2013-08-15 20:50:27.763680 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.763620, event: header_read, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-68> 2013-08-15 20:50:27.763688 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.763621, event: throttled, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-67> 2013-08-15 20:50:27.763695 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.763655, event: all_read, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-66> 2013-08-15 20:50:27.763704 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.763678, event: dispatched, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-65> 2013-08-15 20:50:27.763711 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.763711, event: waiting_for_osdmap, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-64> 2013-08-15 20:50:27.763718 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.763718, event: wait for new map, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-63> 2013-08-15 20:50:27.764817 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.4 9.4.196.1:6813/8739 4 ==== pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2 ==== 1012+0+0 (1088534330 0 0) 0x347d860 con 0x3651080
|
|
-62> 2013-08-15 20:50:27.764830 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.764736, event: header_read, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-61> 2013-08-15 20:50:27.764843 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.764737, event: throttled, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-60> 2013-08-15 20:50:27.764851 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.764793, event: all_read, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-59> 2013-08-15 20:50:27.764860 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.764828, event: dispatched, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-58> 2013-08-15 20:50:27.764881 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.764881, event: waiting_for_osdmap, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-57> 2013-08-15 20:50:27.764887 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.764887, event: wait for new map, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-56> 2013-08-15 20:50:27.764908 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.4 9.4.196.1:6813/8739 5 ==== pg_notify(3.d2(6) epoch 87) v4 ==== 826+0+0 (3806222033 0 0) 0x343d6c0 con 0x3651080
|
|
-55> 2013-08-15 20:50:27.764919 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.764833, event: header_read, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-54> 2013-08-15 20:50:27.764940 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.764834, event: throttled, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-53> 2013-08-15 20:50:27.764946 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.764897, event: all_read, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-52> 2013-08-15 20:50:27.764953 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.764918, event: dispatched, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-51> 2013-08-15 20:50:27.764961 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.764961, event: waiting_for_osdmap, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-50> 2013-08-15 20:50:27.764968 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.764968, event: wait for new map, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-49> 2013-08-15 20:50:27.764988 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.4 9.4.196.1:6813/8739 6 ==== pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2 ==== 1012+0+0 (3098372375 0 0) 0x36ce3c0 con 0x3651080
|
|
-48> 2013-08-15 20:50:27.765001 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.764908, event: header_read, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-47> 2013-08-15 20:50:27.765008 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.764909, event: throttled, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-46> 2013-08-15 20:50:27.765015 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.764975, event: all_read, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-45> 2013-08-15 20:50:27.765022 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.764999, event: dispatched, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-44> 2013-08-15 20:50:27.765029 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.765029, event: waiting_for_osdmap, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-43> 2013-08-15 20:50:27.765037 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.765037, event: wait for new map, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-42> 2013-08-15 20:50:27.840126 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 11 ==== osd_map(86..87 src has 1..87) v3 ==== 5858+0+0 (2545451241 0 0) 0x3663200 con 0x3667e40
|
|
-41> 2013-08-15 20:50:27.840157 7f2b34038700 3 osd.8 63 handle_osd_map epochs [86,87], i have 63, src has [1,87]
|
|
-40> 2013-08-15 20:50:27.840164 7f2b34038700 10 monclient: renew_subs
|
|
-39> 2013-08-15 20:50:27.840167 7f2b34038700 10 monclient: _send_mon_message to mon.ceph-dmon2 at 9.4.196.5:6789/0
|
|
-38> 2013-08-15 20:50:27.840169 7f2b34038700 1 -- 9.4.196.3:6809/8539 --> 9.4.196.5:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343c700 con 0x3667e40
|
|
-37> 2013-08-15 20:50:27.840827 7f2b34038700 1 -- 9.4.196.3:6809/8539 <== mon.2 9.4.196.5:6789/0 12 ==== osd_map(64..87 src has 1..87) v3 ==== 24205+0+0 (1916785775 0 0) 0x36b06c0 con 0x3667e40
|
|
-36> 2013-08-15 20:50:27.840849 7f2b34038700 3 osd.8 63 handle_osd_map epochs [64,87], i have 63, src has [1,87]
|
|
-35> 2013-08-15 20:50:27.841653 7f2b2a514700 1 -- 9.4.196.3:6811/8539 >> :/0 pipe(0x368cf00 sd=37 :6811 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.3:57926/0
|
|
-34> 2013-08-15 20:50:27.842258 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.5 9.4.196.3:6819/4129 1 ==== pg_notify(3.18d(7) epoch 87) v4 ==== 869+0+0 (2428627904 0 0) 0x343ce00 con 0x3666b00
|
|
-33> 2013-08-15 20:50:27.844951 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-32> 2013-08-15 20:50:27.844965 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-31> 2013-08-15 20:50:27.844972 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-30> 2013-08-15 20:50:27.844989 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-29> 2013-08-15 20:50:27.845002 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-28> 2013-08-15 20:50:27.845007 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-27> 2013-08-15 20:50:27.845031 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-26> 2013-08-15 20:50:27.845037 7f2b34038700 1 -- 9.4.196.3:6811/8539 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-25> 2013-08-15 20:50:27.845047 7f2b34038700 1 osd.8 87 state: booting -> active
|
|
-24> 2013-08-15 20:50:27.845348 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.845348, event: started, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-23> 2013-08-15 20:50:27.845376 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:27.845376, event: done, request: pg_query(3.160,3.17f epoch 87) v2
|
|
-22> 2013-08-15 20:50:27.845389 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.845389, event: started, request: pg_query(2.b,3.a epoch 87) v2
|
|
-21> 2013-08-15 20:50:27.845404 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:27.845403, event: done, request: pg_query(2.b,3.a epoch 87) v2
|
|
-20> 2013-08-15 20:50:27.845412 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.845412, event: started, request: pg_query(3.14d epoch 87) v2
|
|
-19> 2013-08-15 20:50:27.845521 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6807/6887 -- pg_notify(3.14d epoch 87) v4 -- ?+0 0x343c380 con 0x36662c0
|
|
-18> 2013-08-15 20:50:27.845537 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:27.845537, event: done, request: pg_query(3.14d epoch 87) v2
|
|
-17> 2013-08-15 20:50:27.845539 7f2b3a044700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-16> 2013-08-15 20:50:27.845556 7f2b3a044700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
-15> 2013-08-15 20:50:27.845553 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.845553, event: started, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-14> 2013-08-15 20:50:27.845582 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:27.845582, event: done, request: pg_notify(3.18d(6) epoch 87) v4
|
|
-13> 2013-08-15 20:50:27.845593 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.845593, event: started, request: pg_notify(3.86(6) epoch 87) v4
|
|
-12> 2013-08-15 20:50:27.845780 7f2b34038700 1 osd.8 pg_epoch: 87 pg[3.86( empty local-les=0 n=0 ec=56 les/c 74/74 87/87/87) [8,1] r=0 lpr=87 pi=73-86/6 mlcod 0'0 inactive] state<Start>: transitioning to Primary
|
|
-11> 2013-08-15 20:50:27.845819 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6801/5083 -- pg_query(3.86 epoch 87) v2 -- ?+0 0x347c3c0 con 0x3666c60
|
|
-10> 2013-08-15 20:50:27.845854 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 4, time: 2013-08-15 20:50:27.845854, event: done, request: pg_notify(3.86 epoch 87) v4
|
|
-9> 2013-08-15 20:50:27.845866 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.845866, event: started, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-8> 2013-08-15 20:50:27.845882 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 5, time: 2013-08-15 20:50:27.845882, event: done, request: pg_query(3.80,3.8f epoch 87) v2
|
|
-7> 2013-08-15 20:50:27.845894 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.845894, event: started, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-6> 2013-08-15 20:50:27.845908 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 6, time: 2013-08-15 20:50:27.845908, event: done, request: pg_notify(3.1a0(6) epoch 87) v4
|
|
-5> 2013-08-15 20:50:27.845917 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.845917, event: started, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-4> 2013-08-15 20:50:27.845925 7f2b2e725700 1 osd.8 pg_epoch: 87 pg[3.69( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 87/87/56) [1,8] r=1 lpr=87 pi=56-86/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:27.846026 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 7, time: 2013-08-15 20:50:27.846025, event: done, request: pg_query(3.69,3.1a1,3.1a2,3.1a8 epoch 87) v2
|
|
-2> 2013-08-15 20:50:27.846043 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.846043, event: started, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-1> 2013-08-15 20:50:27.846441 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.12b( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 87/87/56) [3,8] r=1 lpr=87 pi=56-86/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:27.846501 7f2b3a044700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f2b3a044700 time 2013-08-15 20:50:27.845561
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f2b4104cf8e]
|
|
5: (clone()+0x6d) [0x7f2b3f2fee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:27.850762 7f2b3a044700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f2b3a044700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f2b41054bd0]
|
|
3: (gsignal()+0x37) [0x7f2b3f23c037]
|
|
4: (abort()+0x148) [0x7f2b3f23f698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f2b3fb48e8d]
|
|
6: (()+0x5ef76) [0x7f2b3fb46f76]
|
|
7: (()+0x5efa3) [0x7f2b3fb46fa3]
|
|
8: (()+0x5f1de) [0x7f2b3fb471de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f2b4104cf8e]
|
|
14: (clone()+0x6d) [0x7f2b3f2fee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-81> 2013-08-15 20:50:27.846527 7f2b2ef26700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6810/7799 -- pg_notify(3.12b(11) epoch 87) v4 -- ?+0 0x343da40 con 0x3651340
|
|
-80> 2013-08-15 20:50:27.846615 7f2b2e725700 1 osd.8 pg_epoch: 87 pg[3.1ea( v 63'13 (0'0,63'13] local-les=62 n=9 ec=56 les/c 62/62 87/87/84) [8,2]/[2,0] r=-1 lpr=87 pi=61-86/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-79> 2013-08-15 20:50:27.846679 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6801/5083 -- pg_notify(3.69(11) epoch 87) v4 -- ?+0 0x343dc00 con 0x3666c60
|
|
-78> 2013-08-15 20:50:27.846698 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ea(11) epoch 87) v4 -- ?+0 0x343ddc0 con 0x36662c0
|
|
-77> 2013-08-15 20:50:27.846770 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6807/6887 -- pg_notify(3.108,3.22a epoch 87) v4 -- ?+0 0x37d81c0 con 0x36662c0
|
|
-76> 2013-08-15 20:50:27.846788 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 8, time: 2013-08-15 20:50:27.846788, event: done, request: pg_query(0.37,1.36,2.35,3.34,3.70,3.bd,3.ef,3.108,3.10d,3.13c,3.141,3.194,3.19e,3.1c9,3.1ea,3.1fc,3.200,3.22a,3.23b epoch 87) v2
|
|
-75> 2013-08-15 20:50:27.846803 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.846803, event: started, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-74> 2013-08-15 20:50:27.846824 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 9, time: 2013-08-15 20:50:27.846824, event: done, request: pg_query(0.9,1.8,2.7,3.6 epoch 87) v2
|
|
-73> 2013-08-15 20:50:27.846829 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6801/5083 -- pg_notify(3.69(11) epoch 87) v4 -- ?+0 0x37d8700 con 0x3666c60
|
|
-72> 2013-08-15 20:50:27.846832 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.846832, event: started, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-71> 2013-08-15 20:50:27.846842 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6807/6887 -- pg_notify(3.1ea(11) epoch 87) v4 -- ?+0 0x37d8540 con 0x36662c0
|
|
-70> 2013-08-15 20:50:27.847111 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6804/5992 -- pg_notify(3.126,3.12f,3.1d5,3.233,3.238 epoch 87) v4 -- ?+0 0x37d8000 con 0x3650000
|
|
-69> 2013-08-15 20:50:27.847125 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 10, time: 2013-08-15 20:50:27.847125, event: done, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a,3.66,3.d8,3.e2,3.f5,3.f6,3.11d,3.126,3.12f,3.131,3.1d5,3.1f9,3.233,3.238,3.23f epoch 87) v2
|
|
-68> 2013-08-15 20:50:27.847139 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.847139, event: started, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-67> 2013-08-15 20:50:27.847241 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.125( v 63'23 (0'0,63'23] local-les=62 n=7 ec=56 les/c 62/62 87/87/56) [1,8] r=1 lpr=87 pi=56-86/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-66> 2013-08-15 20:50:27.847355 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6801/5083 -- pg_notify(3.14c,3.224 epoch 87) v4 -- ?+0 0x37d8c40 con 0x3666c60
|
|
-65> 2013-08-15 20:50:27.847367 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 11, time: 2013-08-15 20:50:27.847367, event: done, request: pg_query(3.8e,3.d3,3.ec,3.125,3.14c,3.182,3.214,3.224 epoch 87) v2
|
|
-64> 2013-08-15 20:50:27.847377 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.847377, event: started, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-63> 2013-08-15 20:50:27.847393 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 12, time: 2013-08-15 20:50:27.847393, event: done, request: pg_notify(3.23e(6) epoch 87) v4
|
|
-62> 2013-08-15 20:50:27.847399 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.847399, event: started, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-61> 2013-08-15 20:50:27.847473 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6810/7799 -- pg_notify(3.af,3.10c epoch 87) v4 -- ?+0 0x37d8a80 con 0x3651340
|
|
-60> 2013-08-15 20:50:27.847485 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 13, time: 2013-08-15 20:50:27.847485, event: done, request: pg_query(3.af,3.b9,3.10c,3.118,3.11c,3.12b epoch 87) v2
|
|
-59> 2013-08-15 20:50:27.847495 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.847495, event: started, request: pg_query(3.178 epoch 87) v2
|
|
-58> 2013-08-15 20:50:27.847504 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 14, time: 2013-08-15 20:50:27.847504, event: done, request: pg_query(3.178 epoch 87) v2
|
|
-57> 2013-08-15 20:50:27.847511 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.847511, event: started, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-56> 2013-08-15 20:50:27.847544 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 15, time: 2013-08-15 20:50:27.847544, event: done, request: pg_query(0.24,1.23,2.22,3.21,3.41,3.58,3.8b,3.1b8 epoch 87) v2
|
|
-55> 2013-08-15 20:50:27.847537 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6810/7799 -- pg_notify(3.12b(11) epoch 87) v4 -- ?+0 0x37d8380 con 0x3651340
|
|
-54> 2013-08-15 20:50:27.847552 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.847552, event: started, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-53> 2013-08-15 20:50:27.847631 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 16, time: 2013-08-15 20:50:27.847631, event: done, request: pg_query(0.1d,0.26,1.1c,1.25,2.1b,2.24,3.1a,3.23 epoch 87) v2
|
|
-52> 2013-08-15 20:50:27.847640 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.847640, event: started, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-51> 2013-08-15 20:50:27.847655 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 17, time: 2013-08-15 20:50:27.847655, event: done, request: pg_notify(3.d2(6) epoch 87) v4
|
|
-50> 2013-08-15 20:50:27.847661 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.847661, event: started, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-49> 2013-08-15 20:50:27.847761 7f2b34038700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6813/8739 -- pg_notify(3.e5 epoch 87) v4 -- ?+0 0x37d88c0 con 0x3651080
|
|
-48> 2013-08-15 20:50:27.847773 7f2b34038700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 18, time: 2013-08-15 20:50:27.847773, event: done, request: pg_query(0.34,1.33,2.32,3.31,3.a1,3.aa,3.c7,3.e5 epoch 87) v2
|
|
-47> 2013-08-15 20:50:27.847792 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.842180, event: header_read, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-46> 2013-08-15 20:50:27.847802 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.842182, event: throttled, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-45> 2013-08-15 20:50:27.847806 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.842233, event: all_read, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-44> 2013-08-15 20:50:27.847810 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.842279, event: dispatched, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-43> 2013-08-15 20:50:27.847814 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.847814, event: waiting_for_osdmap, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-42> 2013-08-15 20:50:27.847820 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.847820, event: started, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-41> 2013-08-15 20:50:27.847839 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 19, time: 2013-08-15 20:50:27.847839, event: done, request: pg_notify(3.18d(7) epoch 87) v4
|
|
-40> 2013-08-15 20:50:27.847849 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.5 9.4.196.3:6819/4129 2 ==== pg_notify(3.86(6) epoch 87) v4 ==== 826+0+0 (1311178567 0 0) 0x343cc40 con 0x3666b00
|
|
-39> 2013-08-15 20:50:27.847860 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.844608, event: header_read, request: pg_notify(3.86(6) epoch 87) v4
|
|
-38> 2013-08-15 20:50:27.847865 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.844610, event: throttled, request: pg_notify(3.86(6) epoch 87) v4
|
|
-37> 2013-08-15 20:50:27.847868 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.844656, event: all_read, request: pg_notify(3.86(6) epoch 87) v4
|
|
-36> 2013-08-15 20:50:27.847874 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.847858, event: dispatched, request: pg_notify(3.86(6) epoch 87) v4
|
|
-35> 2013-08-15 20:50:27.847877 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.847877, event: waiting_for_osdmap, request: pg_notify(3.86(6) epoch 87) v4
|
|
-34> 2013-08-15 20:50:27.847882 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.847882, event: started, request: pg_notify(3.86(6) epoch 87) v4
|
|
-33> 2013-08-15 20:50:27.847896 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 20, time: 2013-08-15 20:50:27.847896, event: done, request: pg_notify(3.86(6) epoch 87) v4
|
|
-32> 2013-08-15 20:50:27.847908 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 7 ==== pg_notify(3.86(6) epoch 87) v4 ==== 826+0+0 (2360900112 0 0) 0x343cfc0 con 0x3666c60
|
|
-31> 2013-08-15 20:50:27.847916 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.846372, event: header_read, request: pg_notify(3.86(6) epoch 87) v4
|
|
-30> 2013-08-15 20:50:27.847923 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.846373, event: throttled, request: pg_notify(3.86(6) epoch 87) v4
|
|
-29> 2013-08-15 20:50:27.847927 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.846421, event: all_read, request: pg_notify(3.86(6) epoch 87) v4
|
|
-28> 2013-08-15 20:50:27.847930 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.847915, event: dispatched, request: pg_notify(3.86(6) epoch 87) v4
|
|
-27> 2013-08-15 20:50:27.847935 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.847935, event: waiting_for_osdmap, request: pg_notify(3.86(6) epoch 87) v4
|
|
-26> 2013-08-15 20:50:27.847943 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.847942, event: started, request: pg_notify(3.86(6) epoch 87) v4
|
|
-25> 2013-08-15 20:50:27.847958 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 21, time: 2013-08-15 20:50:27.847958, event: done, request: pg_notify(3.86(6) epoch 87) v4
|
|
-24> 2013-08-15 20:50:27.847969 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.3 9.4.196.1:6810/7799 5 ==== pg_query(3.12b epoch 87) v2 ==== 137+0+0 (1646483409 0 0) 0x347dc20 con 0x3651340
|
|
-23> 2013-08-15 20:50:27.847977 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.847096, event: header_read, request: pg_query(3.12b epoch 87) v2
|
|
-22> 2013-08-15 20:50:27.847981 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.847097, event: throttled, request: pg_query(3.12b epoch 87) v2
|
|
-21> 2013-08-15 20:50:27.847984 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.847134, event: all_read, request: pg_query(3.12b epoch 87) v2
|
|
-20> 2013-08-15 20:50:27.847988 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.847976, event: dispatched, request: pg_query(3.12b epoch 87) v2
|
|
-19> 2013-08-15 20:50:27.847992 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.847992, event: waiting_for_osdmap, request: pg_query(3.12b epoch 87) v2
|
|
-18> 2013-08-15 20:50:27.847996 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.847996, event: started, request: pg_query(3.12b epoch 87) v2
|
|
-17> 2013-08-15 20:50:27.848009 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 22, time: 2013-08-15 20:50:27.848008, event: done, request: pg_query(3.12b epoch 87) v2
|
|
-16> 2013-08-15 20:50:27.848016 7f2b33036700 1 -- 9.4.196.3:6811/8539 <== osd.1 9.4.196.1:6801/5083 8 ==== pg_query(3.69 epoch 87) v2 ==== 137+0+0 (2206772679 0 0) 0x347c3c0 con 0x3666c60
|
|
-15> 2013-08-15 20:50:27.848024 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.847213, event: header_read, request: pg_query(3.69 epoch 87) v2
|
|
-14> 2013-08-15 20:50:27.848027 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.847215, event: throttled, request: pg_query(3.69 epoch 87) v2
|
|
-13> 2013-08-15 20:50:27.848032 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.847255, event: all_read, request: pg_query(3.69 epoch 87) v2
|
|
-12> 2013-08-15 20:50:27.848027 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6810/7799 -- pg_log(3.12b epoch 87 query_epoch 87) v3 -- ?+0 0x3433600 con 0x3651340
|
|
-11> 2013-08-15 20:50:27.848035 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.848023, event: dispatched, request: pg_query(3.69 epoch 87) v2
|
|
-10> 2013-08-15 20:50:27.848039 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.848039, event: waiting_for_osdmap, request: pg_query(3.69 epoch 87) v2
|
|
-9> 2013-08-15 20:50:27.848043 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.848043, event: started, request: pg_query(3.69 epoch 87) v2
|
|
-8> 2013-08-15 20:50:27.848054 7f2b33036700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 23, time: 2013-08-15 20:50:27.848053, event: done, request: pg_query(3.69 epoch 87) v2
|
|
-7> 2013-08-15 20:50:27.848064 7f2b2e725700 1 -- 9.4.196.3:6811/8539 --> 9.4.196.1:6801/5083 -- pg_log(3.69 epoch 87 query_epoch 87) v3 -- ?+0 0x3434680 con 0x3666c60
|
|
-6> 2013-08-15 20:50:27.848165 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.66( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 87/87/84) [8,0]/[0,3] r=-1 lpr=87 pi=61-86/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:27.848779 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.11c( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 87/87/56) [3,8] r=1 lpr=87 pi=56-86/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:27.849289 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.11d( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 87/87/84) [8,0]/[0,1] r=-1 lpr=87 pi=61-86/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:27.849823 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.58( v 63'21 (0'0,63'21] local-les=62 n=5 ec=56 les/c 62/62 87/87/56) [3,8] r=1 lpr=87 pi=56-86/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:27.850228 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.118( v 63'17 (0'0,63'17] local-les=62 n=7 ec=56 les/c 62/62 87/87/84) [8,3]/[3,1] r=-1 lpr=87 pi=61-86/11 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:27.850715 7f2b2ef26700 1 osd.8 pg_epoch: 87 pg[3.10d( v 63'24 (0'0,63'24] local-les=62 n=4 ec=56 les/c 62/62 87/87/56) [2,8] r=1 lpr=87 pi=56-86/11 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:27.850762 7f2b3a044700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f2b3a044700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f2b41054bd0]
|
|
3: (gsignal()+0x37) [0x7f2b3f23c037]
|
|
4: (abort()+0x148) [0x7f2b3f23f698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f2b3fb48e8d]
|
|
6: (()+0x5ef76) [0x7f2b3fb46f76]
|
|
7: (()+0x5efa3) [0x7f2b3fb46fa3]
|
|
8: (()+0x5f1de) [0x7f2b3fb471de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f2b4104cf8e]
|
|
14: (clone()+0x6d) [0x7f2b3f2fee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:28.004802 7f88f687f7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8914
|
|
2013-08-15 20:50:28.005324 7f88f687f7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:28.062230 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:28.062243 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:28.062503 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:28.080199 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:28.080283 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:28.116345 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:28.117196 7f88f687f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:28.117213 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:28.176885 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:28.177382 7f88f687f7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
2013-08-15 20:50:28.217812 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:28.217823 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:28.218109 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:28.247712 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:28.247771 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:28.271873 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:28.272430 7f88f687f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:28.272443 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:28.329757 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:29.769612 7f88ef253700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:29.769629 7f88ef253700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
2013-08-15 20:50:29.770730 7f88ef253700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f88ef253700 time 2013-08-15 20:50:29.769635
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f88f625bf8e]
|
|
5: (clone()+0x6d) [0x7f88f450de1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-217> 2013-08-15 20:50:28.003393 7f88f687f7c0 5 asok(0x33f80e0) register_command perfcounters_dump hook 0x33f2010
|
|
-216> 2013-08-15 20:50:28.003436 7f88f687f7c0 5 asok(0x33f80e0) register_command 1 hook 0x33f2010
|
|
-215> 2013-08-15 20:50:28.003442 7f88f687f7c0 5 asok(0x33f80e0) register_command perf dump hook 0x33f2010
|
|
-214> 2013-08-15 20:50:28.003450 7f88f687f7c0 5 asok(0x33f80e0) register_command perfcounters_schema hook 0x33f2010
|
|
-213> 2013-08-15 20:50:28.003454 7f88f687f7c0 5 asok(0x33f80e0) register_command 2 hook 0x33f2010
|
|
-212> 2013-08-15 20:50:28.003456 7f88f687f7c0 5 asok(0x33f80e0) register_command perf schema hook 0x33f2010
|
|
-211> 2013-08-15 20:50:28.003474 7f88f687f7c0 5 asok(0x33f80e0) register_command config show hook 0x33f2010
|
|
-210> 2013-08-15 20:50:28.003479 7f88f687f7c0 5 asok(0x33f80e0) register_command config set hook 0x33f2010
|
|
-209> 2013-08-15 20:50:28.003482 7f88f687f7c0 5 asok(0x33f80e0) register_command log flush hook 0x33f2010
|
|
-208> 2013-08-15 20:50:28.003485 7f88f687f7c0 5 asok(0x33f80e0) register_command log dump hook 0x33f2010
|
|
-207> 2013-08-15 20:50:28.003490 7f88f687f7c0 5 asok(0x33f80e0) register_command log reopen hook 0x33f2010
|
|
-206> 2013-08-15 20:50:28.004802 7f88f687f7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 8914
|
|
-205> 2013-08-15 20:50:28.005324 7f88f687f7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-204> 2013-08-15 20:50:28.005430 7f88f687f7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6809/8914 need_addr=1
|
|
-203> 2013-08-15 20:50:28.005461 7f88f687f7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-202> 2013-08-15 20:50:28.005469 7f88f687f7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6828/8914 need_addr=0
|
|
-201> 2013-08-15 20:50:28.005489 7f88f687f7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-200> 2013-08-15 20:50:28.005494 7f88f687f7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6829/8914 need_addr=0
|
|
-199> 2013-08-15 20:50:28.007220 7f88f687f7c0 5 asok(0x33f80e0) init /var/run/ceph/ceph-osd.8.asok
|
|
-198> 2013-08-15 20:50:28.007234 7f88f687f7c0 5 asok(0x33f80e0) bind_and_listen /var/run/ceph/ceph-osd.8.asok
|
|
-197> 2013-08-15 20:50:28.007264 7f88f687f7c0 5 asok(0x33f80e0) register_command 0 hook 0x33f00b8
|
|
-196> 2013-08-15 20:50:28.007270 7f88f687f7c0 5 asok(0x33f80e0) register_command version hook 0x33f00b8
|
|
-195> 2013-08-15 20:50:28.007274 7f88f687f7c0 5 asok(0x33f80e0) register_command git_version hook 0x33f00b8
|
|
-194> 2013-08-15 20:50:28.007279 7f88f687f7c0 5 asok(0x33f80e0) register_command help hook 0x33f20d0
|
|
-193> 2013-08-15 20:50:28.007304 7f88f2259700 5 asok(0x33f80e0) entry start
|
|
-192> 2013-08-15 20:50:28.062230 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-191> 2013-08-15 20:50:28.062243 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-190> 2013-08-15 20:50:28.062503 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-189> 2013-08-15 20:50:28.080199 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-188> 2013-08-15 20:50:28.080283 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-187> 2013-08-15 20:50:28.116345 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-186> 2013-08-15 20:50:28.117162 7f88f687f7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-185> 2013-08-15 20:50:28.117196 7f88f687f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-184> 2013-08-15 20:50:28.117213 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-183> 2013-08-15 20:50:28.122658 7f88f687f7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-182> 2013-08-15 20:50:28.126797 7f88f687f7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-181> 2013-08-15 20:50:28.130686 7f88f687f7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-180> 2013-08-15 20:50:28.134601 7f88f687f7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-179> 2013-08-15 20:50:28.138423 7f88f687f7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-178> 2013-08-15 20:50:28.142317 7f88f687f7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-177> 2013-08-15 20:50:28.146155 7f88f687f7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-176> 2013-08-15 20:50:28.149967 7f88f687f7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-175> 2013-08-15 20:50:28.153788 7f88f687f7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-174> 2013-08-15 20:50:28.157600 7f88f687f7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-173> 2013-08-15 20:50:28.161422 7f88f687f7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-172> 2013-08-15 20:50:28.165200 7f88f687f7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-171> 2013-08-15 20:50:28.168988 7f88f687f7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-170> 2013-08-15 20:50:28.172935 7f88f687f7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-169> 2013-08-15 20:50:28.176802 7f88f687f7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-168> 2013-08-15 20:50:28.176833 7f88f687f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-167> 2013-08-15 20:50:28.176838 7f88f687f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-166> 2013-08-15 20:50:28.176844 7f88f687f7c0 3 journal journal_replay: end of journal, done.
|
|
-165> 2013-08-15 20:50:28.176885 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-164> 2013-08-15 20:50:28.177241 7f88ef253700 1 FileStore::op_tp worker finish
|
|
-163> 2013-08-15 20:50:28.177271 7f88efa54700 1 FileStore::op_tp worker finish
|
|
-162> 2013-08-15 20:50:28.177382 7f88f687f7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
-161> 2013-08-15 20:50:28.177621 7f88f687f7c0 10 monclient(hunting): build_initial_monmap
|
|
-160> 2013-08-15 20:50:28.177669 7f88f687f7c0 5 adding auth protocol: cephx
|
|
-159> 2013-08-15 20:50:28.177676 7f88f687f7c0 5 adding auth protocol: cephx
|
|
-158> 2013-08-15 20:50:28.177756 7f88f687f7c0 1 -- 0.0.0.0:6809/8914 messenger.start
|
|
-157> 2013-08-15 20:50:28.177782 7f88f687f7c0 1 -- :/0 messenger.start
|
|
-156> 2013-08-15 20:50:28.177792 7f88f687f7c0 1 -- 9.4.196.3:6829/8914 messenger.start
|
|
-155> 2013-08-15 20:50:28.177802 7f88f687f7c0 1 -- 9.4.196.3:6828/8914 messenger.start
|
|
-154> 2013-08-15 20:50:28.177881 7f88f687f7c0 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
|
|
-153> 2013-08-15 20:50:28.217812 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-152> 2013-08-15 20:50:28.217823 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-151> 2013-08-15 20:50:28.218109 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-150> 2013-08-15 20:50:28.247712 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-149> 2013-08-15 20:50:28.247771 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-148> 2013-08-15 20:50:28.271873 7f88f687f7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-147> 2013-08-15 20:50:28.272400 7f88f687f7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-146> 2013-08-15 20:50:28.272430 7f88f687f7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-145> 2013-08-15 20:50:28.272443 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-144> 2013-08-15 20:50:28.276391 7f88f687f7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-143> 2013-08-15 20:50:28.280230 7f88f687f7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-142> 2013-08-15 20:50:28.284064 7f88f687f7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-141> 2013-08-15 20:50:28.287945 7f88f687f7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-140> 2013-08-15 20:50:28.291762 7f88f687f7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-139> 2013-08-15 20:50:28.295549 7f88f687f7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-138> 2013-08-15 20:50:28.299339 7f88f687f7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-137> 2013-08-15 20:50:28.303142 7f88f687f7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-136> 2013-08-15 20:50:28.306936 7f88f687f7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-135> 2013-08-15 20:50:28.310718 7f88f687f7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-134> 2013-08-15 20:50:28.314492 7f88f687f7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-133> 2013-08-15 20:50:28.318286 7f88f687f7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-132> 2013-08-15 20:50:28.322108 7f88f687f7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-131> 2013-08-15 20:50:28.325885 7f88f687f7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-130> 2013-08-15 20:50:28.329691 7f88f687f7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-129> 2013-08-15 20:50:28.329707 7f88f687f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-128> 2013-08-15 20:50:28.329712 7f88f687f7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-127> 2013-08-15 20:50:28.329713 7f88f687f7c0 3 journal journal_replay: end of journal, done.
|
|
-126> 2013-08-15 20:50:28.329757 7f88f687f7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-125> 2013-08-15 20:50:28.329952 7f88f687f7c0 2 osd.8 0 boot
|
|
-124> 2013-08-15 20:50:28.355659 7f88f687f7c0 2 osd.8 63 superblock: i am osd.8
|
|
-123> 2013-08-15 20:50:28.355700 7f88f687f7c0 1 accepter.accepter.start
|
|
-122> 2013-08-15 20:50:28.355751 7f88f687f7c0 1 accepter.accepter.start
|
|
-121> 2013-08-15 20:50:28.355805 7f88f687f7c0 1 accepter.accepter.start
|
|
-120> 2013-08-15 20:50:28.355822 7f88f687f7c0 10 monclient(hunting): init
|
|
-119> 2013-08-15 20:50:28.355829 7f88f687f7c0 5 adding auth protocol: cephx
|
|
-118> 2013-08-15 20:50:28.355836 7f88f687f7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-117> 2013-08-15 20:50:28.355977 7f88f687f7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-8/keyring
|
|
-116> 2013-08-15 20:50:28.356150 7f88f687f7c0 5 asok(0x33f80e0) register_command dump_ops_in_flight hook 0x33f2750
|
|
-115> 2013-08-15 20:50:28.356160 7f88f687f7c0 5 asok(0x33f80e0) register_command dump_historic_ops hook 0x33f2750
|
|
-114> 2013-08-15 20:50:28.356165 7f88f687f7c0 5 asok(0x33f80e0) register_command dump_op_pq_state hook 0x33f2750
|
|
-113> 2013-08-15 20:50:28.356171 7f88f687f7c0 5 asok(0x33f80e0) register_command setomapval hook 0x35e6e80
|
|
-112> 2013-08-15 20:50:28.356177 7f88f687f7c0 5 asok(0x33f80e0) register_command rmomapkey hook 0x35e6e80
|
|
-111> 2013-08-15 20:50:28.356182 7f88f687f7c0 5 asok(0x33f80e0) register_command setomapheader hook 0x35e6e80
|
|
-110> 2013-08-15 20:50:28.356185 7f88f687f7c0 5 asok(0x33f80e0) register_command getomap hook 0x35e6e80
|
|
-109> 2013-08-15 20:50:28.356189 7f88f687f7c0 5 asok(0x33f80e0) register_command truncobj hook 0x35e6e80
|
|
-108> 2013-08-15 20:50:28.356192 7f88f687f7c0 5 asok(0x33f80e0) register_command injectdataerr hook 0x35e6e80
|
|
-107> 2013-08-15 20:50:28.356207 7f88f687f7c0 5 asok(0x33f80e0) register_command injectmdataerr hook 0x35e6e80
|
|
-106> 2013-08-15 20:50:28.356217 7f88e2239700 5 osd.8 63 heartbeat: osd_stat(1022 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-105> 2013-08-15 20:50:28.356253 7f88f687f7c0 10 monclient(hunting): _reopen_session
|
|
-104> 2013-08-15 20:50:28.356315 7f88f687f7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x3650840 addr 9.4.196.1:6789/0
|
|
-103> 2013-08-15 20:50:28.356342 7f88f687f7c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-102> 2013-08-15 20:50:28.356347 7f88f687f7c0 1 -- 0.0.0.0:6809/8914 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x3404b40 con 0x3650840
|
|
-101> 2013-08-15 20:50:28.356360 7f88f687f7c0 10 monclient(hunting): renew_subs
|
|
-100> 2013-08-15 20:50:28.356781 7f88f6811700 1 -- 9.4.196.3:6809/8914 learned my addr 9.4.196.3:6809/8914
|
|
-99> 2013-08-15 20:50:28.356987 7f88e9247700 10 monclient(hunting): renew_subs
|
|
-98> 2013-08-15 20:50:28.357531 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x347a1e0 con 0x3650840
|
|
-97> 2013-08-15 20:50:28.357552 7f88e9247700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-96> 2013-08-15 20:50:28.357570 7f88e9247700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-95> 2013-08-15 20:50:28.357586 7f88e9247700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-94> 2013-08-15 20:50:28.357609 7f88e9247700 1 monclient(hunting): found mon.zc2store
|
|
-93> 2013-08-15 20:50:28.357618 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2621558902 0 0) 0x365a200 con 0x3650840
|
|
-92> 2013-08-15 20:50:28.357644 7f88e9247700 10 monclient: my global_id is 4383
|
|
-91> 2013-08-15 20:50:28.357766 7f88e9247700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-90> 2013-08-15 20:50:28.357775 7f88e9247700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x3405d40 con 0x3650840
|
|
-89> 2013-08-15 20:50:28.358450 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (2591561153 0 0) 0x365a000 con 0x3650840
|
|
-88> 2013-08-15 20:50:28.358518 7f88e9247700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-87> 2013-08-15 20:50:28.358526 7f88e9247700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x365e240 con 0x3650840
|
|
-86> 2013-08-15 20:50:28.359223 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (2376247279 0 0) 0x365a400 con 0x3650840
|
|
-85> 2013-08-15 20:50:28.359303 7f88e9247700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-84> 2013-08-15 20:50:28.359311 7f88e9247700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x343a380 con 0x3650840
|
|
-83> 2013-08-15 20:50:28.359336 7f88e9247700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-82> 2013-08-15 20:50:28.359356 7f88e9247700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x343a540 con 0x3650840
|
|
-81> 2013-08-15 20:50:28.359373 7f88e9247700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:49:58.359372)
|
|
-80> 2013-08-15 20:50:28.359383 7f88e9247700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-79> 2013-08-15 20:50:28.359390 7f88e9247700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x365e000 con 0x3650840
|
|
-78> 2013-08-15 20:50:28.359460 7f88f687f7c0 5 monclient: authenticate success, global_id 4383
|
|
-77> 2013-08-15 20:50:28.359475 7f88f687f7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:50:58.359474)
|
|
-76> 2013-08-15 20:50:28.359702 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x347a5a0 con 0x3650840
|
|
-75> 2013-08-15 20:50:28.359715 7f88e9247700 10 monclient: handle_monmap mon_map v1
|
|
-74> 2013-08-15 20:50:28.359729 7f88e9247700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-73> 2013-08-15 20:50:28.359734 7f88e9247700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-72> 2013-08-15 20:50:28.359758 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x343a380 con 0x3650840
|
|
-71> 2013-08-15 20:50:28.359768 7f88e9247700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:28.356362 renew after 2013-08-15 20:52:58.356362
|
|
-70> 2013-08-15 20:50:28.359793 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x347a3c0 con 0x3650840
|
|
-69> 2013-08-15 20:50:28.359802 7f88e9247700 10 monclient: handle_monmap mon_map v1
|
|
-68> 2013-08-15 20:50:28.359811 7f88e9247700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-67> 2013-08-15 20:50:28.359815 7f88e9247700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-66> 2013-08-15 20:50:28.359832 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x343a700 con 0x3650840
|
|
-65> 2013-08-15 20:50:28.359842 7f88e9247700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-64> 2013-08-15 20:50:28.359849 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (2740294759 0 0) 0x365a600 con 0x3650840
|
|
-63> 2013-08-15 20:50:28.359890 7f88e9247700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:49:58.359890)
|
|
-62> 2013-08-15 20:50:28.359930 7f88f687f7c0 10 monclient: wait_auth_rotating done
|
|
-61> 2013-08-15 20:50:28.360500 7f88f687f7c0 10 monclient: get_version osdmap
|
|
-60> 2013-08-15 20:50:28.360511 7f88f687f7c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-59> 2013-08-15 20:50:28.360520 7f88f687f7c0 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x34609c0 con 0x3650840
|
|
-58> 2013-08-15 20:50:28.360848 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=87) v2 ==== 24+0+0 (1748105714 0 0) 0x343a540 con 0x3650840
|
|
-57> 2013-08-15 20:50:28.360899 7f88e523f700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-56> 2013-08-15 20:50:28.360911 7f88e523f700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- osd_boot(osd.8 booted 0 v63) v3 -- ?+0 0x367e000 con 0x3650840
|
|
-55> 2013-08-15 20:50:29.759148 7f88dff2b700 1 -- 9.4.196.3:6828/8914 >> :/0 pipe(0x3655180 sd=33 :6828 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:56101/0
|
|
-54> 2013-08-15 20:50:29.761071 7f88e8245700 1 -- 9.4.196.3:6828/8914 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(88..89 src has 1..89) v3 ==== 1979+0+0 (1011610323 0 0) 0x365f440 con 0x3666160
|
|
-53> 2013-08-15 20:50:29.761105 7f88e8245700 3 osd.8 63 handle_osd_map epochs [88,89], i have 63, src has [1,89]
|
|
-52> 2013-08-15 20:50:29.761116 7f88e8245700 10 monclient: renew_subs
|
|
-51> 2013-08-15 20:50:29.761120 7f88e8245700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-50> 2013-08-15 20:50:29.761130 7f88e8245700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343a8c0 con 0x3650840
|
|
-49> 2013-08-15 20:50:29.761178 7f88e8245700 1 -- 9.4.196.3:6828/8914 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(0.9 epoch 89) v2 ==== 137+0+0 (3268372894 0 0) 0x347a1e0 con 0x3666160
|
|
-48> 2013-08-15 20:50:29.761199 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.761070, event: header_read, request: pg_query(0.9 epoch 89) v2
|
|
-47> 2013-08-15 20:50:29.761210 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.761071, event: throttled, request: pg_query(0.9 epoch 89) v2
|
|
-46> 2013-08-15 20:50:29.761216 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.761139, event: all_read, request: pg_query(0.9 epoch 89) v2
|
|
-45> 2013-08-15 20:50:29.761221 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.761193, event: dispatched, request: pg_query(0.9 epoch 89) v2
|
|
-44> 2013-08-15 20:50:29.761228 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.761228, event: waiting_for_osdmap, request: pg_query(0.9 epoch 89) v2
|
|
-43> 2013-08-15 20:50:29.761239 7f88e8245700 10 monclient: renew_subs
|
|
-42> 2013-08-15 20:50:29.761255 7f88e8245700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-41> 2013-08-15 20:50:29.761265 7f88e8245700 1 -- 9.4.196.3:6809/8914 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x343ac40 con 0x3650840
|
|
-40> 2013-08-15 20:50:29.761281 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:29.761281, event: wait for new map, request: pg_query(0.9 epoch 89) v2
|
|
-39> 2013-08-15 20:50:29.761319 7f88e8245700 1 -- 9.4.196.3:6828/8914 <== osd.4 9.4.196.1:6813/8739 3 ==== pg_query(1.8 epoch 89) v2 ==== 137+0+0 (678238060 0 0) 0x347a780 con 0x3666160
|
|
-38> 2013-08-15 20:50:29.761343 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.761177, event: header_read, request: pg_query(1.8 epoch 89) v2
|
|
-37> 2013-08-15 20:50:29.761357 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.761178, event: throttled, request: pg_query(1.8 epoch 89) v2
|
|
-36> 2013-08-15 20:50:29.761365 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.761300, event: all_read, request: pg_query(1.8 epoch 89) v2
|
|
-35> 2013-08-15 20:50:29.761373 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.761339, event: dispatched, request: pg_query(1.8 epoch 89) v2
|
|
-34> 2013-08-15 20:50:29.761377 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.761377, event: waiting_for_osdmap, request: pg_query(1.8 epoch 89) v2
|
|
-33> 2013-08-15 20:50:29.761382 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:29.761382, event: wait for new map, request: pg_query(1.8 epoch 89) v2
|
|
-32> 2013-08-15 20:50:29.761580 7f88dfd29700 1 -- 9.4.196.3:6828/8914 >> :/0 pipe(0x3655680 sd=34 :6828 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:56102/0
|
|
-31> 2013-08-15 20:50:29.762178 7f88e8245700 1 -- 9.4.196.3:6828/8914 <== osd.4 9.4.196.1:6813/8739 4 ==== pg_query(2.7 epoch 89) v2 ==== 137+0+0 (865832026 0 0) 0x347a5a0 con 0x3666160
|
|
-30> 2013-08-15 20:50:29.762203 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.762126, event: header_read, request: pg_query(2.7 epoch 89) v2
|
|
-29> 2013-08-15 20:50:29.762216 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.762127, event: throttled, request: pg_query(2.7 epoch 89) v2
|
|
-28> 2013-08-15 20:50:29.762225 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.762162, event: all_read, request: pg_query(2.7 epoch 89) v2
|
|
-27> 2013-08-15 20:50:29.762235 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.762200, event: dispatched, request: pg_query(2.7 epoch 89) v2
|
|
-26> 2013-08-15 20:50:29.762244 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.762244, event: waiting_for_osdmap, request: pg_query(2.7 epoch 89) v2
|
|
-25> 2013-08-15 20:50:29.762253 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:29.762253, event: wait for new map, request: pg_query(2.7 epoch 89) v2
|
|
-24> 2013-08-15 20:50:29.762263 7f88e8245700 1 -- 9.4.196.3:6828/8914 <== osd.4 9.4.196.1:6813/8739 5 ==== pg_query(3.6 epoch 89) v2 ==== 137+0+0 (1022327187 0 0) 0x347ad20 con 0x3666160
|
|
-23> 2013-08-15 20:50:29.762273 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.762182, event: header_read, request: pg_query(3.6 epoch 89) v2
|
|
-22> 2013-08-15 20:50:29.762277 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.762183, event: throttled, request: pg_query(3.6 epoch 89) v2
|
|
-21> 2013-08-15 20:50:29.762280 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.762241, event: all_read, request: pg_query(3.6 epoch 89) v2
|
|
-20> 2013-08-15 20:50:29.762284 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.762271, event: dispatched, request: pg_query(3.6 epoch 89) v2
|
|
-19> 2013-08-15 20:50:29.762287 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.762287, event: waiting_for_osdmap, request: pg_query(3.6 epoch 89) v2
|
|
-18> 2013-08-15 20:50:29.762294 7f88e8245700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:29.762294, event: wait for new map, request: pg_query(3.6 epoch 89) v2
|
|
-17> 2013-08-15 20:50:29.764732 7f88e9247700 1 -- 9.4.196.3:6809/8914 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..88 src has 1..88) v3 ==== 24497+0+0 (52318603 0 0) 0x365fd40 con 0x3650840
|
|
-16> 2013-08-15 20:50:29.764751 7f88e9247700 3 osd.8 63 handle_osd_map epochs [64,88], i have 63, src has [1,88]
|
|
-15> 2013-08-15 20:50:29.765343 7f88e8245700 1 -- 9.4.196.3:6828/8914 <== osd.4 9.4.196.1:6813/8739 6 ==== pg_query(3.178 epoch 89) v2 ==== 137+0+0 (1525815570 0 0) 0x347ab40 con 0x3666160
|
|
-14> 2013-08-15 20:50:29.768990 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-13> 2013-08-15 20:50:29.769003 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-12> 2013-08-15 20:50:29.769009 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-11> 2013-08-15 20:50:29.769026 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-10> 2013-08-15 20:50:29.769038 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-9> 2013-08-15 20:50:29.769044 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-8> 2013-08-15 20:50:29.769068 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-7> 2013-08-15 20:50:29.769073 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-6> 2013-08-15 20:50:29.769084 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-5> 2013-08-15 20:50:29.769090 7f88e9247700 1 -- 9.4.196.3:6828/8914 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-4> 2013-08-15 20:50:29.769612 7f88ef253700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-3> 2013-08-15 20:50:29.769629 7f88ef253700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
-2> 2013-08-15 20:50:29.769616 7f88dfb27700 1 -- 9.4.196.3:6829/8914 >> :/0 pipe(0x3655900 sd=35 :6829 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:56550/0
|
|
-1> 2013-08-15 20:50:29.770063 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.69( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 88/88/56) [1] r=-1 lpr=88 pi=56-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:29.770730 7f88ef253700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f88ef253700 time 2013-08-15 20:50:29.769635
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f88f625bf8e]
|
|
5: (clone()+0x6d) [0x7f88f450de1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:29.774850 7f88ef253700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f88ef253700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f88f6263bd0]
|
|
3: (gsignal()+0x37) [0x7f88f444b037]
|
|
4: (abort()+0x148) [0x7f88f444e698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f88f4d57e8d]
|
|
6: (()+0x5ef76) [0x7f88f4d55f76]
|
|
7: (()+0x5efa3) [0x7f88f4d55fa3]
|
|
8: (()+0x5f1de) [0x7f88f4d561de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f88f625bf8e]
|
|
14: (clone()+0x6d) [0x7f88f450de1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-12> 2013-08-15 20:50:29.770794 7f88e4a3e700 1 osd.8 pg_epoch: 88 pg[3.1ea( v 63'13 (0'0,63'13] local-les=62 n=9 ec=56 les/c 62/62 88/88/84) [2] r=-1 lpr=88 pi=61-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-11> 2013-08-15 20:50:29.770926 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.125( v 63'23 (0'0,63'23] local-les=62 n=7 ec=56 les/c 62/62 88/88/56) [1] r=-1 lpr=88 pi=56-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-10> 2013-08-15 20:50:29.771171 7f88df925700 1 -- 9.4.196.3:6828/8914 >> :/0 pipe(0x3655400 sd=36 :6828 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.3:56106/0
|
|
-9> 2013-08-15 20:50:29.771585 7f88e4a3e700 1 osd.8 pg_epoch: 88 pg[3.12b( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 88/88/56) [3] r=-1 lpr=88 pi=56-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-8> 2013-08-15 20:50:29.772237 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.66( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 88/88/84) [0]/[0,3] r=-1 lpr=88 pi=61-87/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-7> 2013-08-15 20:50:29.772460 7f88df723700 1 -- 9.4.196.3:6829/8914 >> :/0 pipe(0x379c780 sd=37 :6829 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.1:56553/0
|
|
-6> 2013-08-15 20:50:29.772743 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.11c( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 88/88/56) [3] r=-1 lpr=88 pi=56-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-5> 2013-08-15 20:50:29.772977 7f88df521700 1 -- 9.4.196.3:6829/8914 >> :/0 pipe(0x379c500 sd=38 :6829 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:56554/0
|
|
-4> 2013-08-15 20:50:29.773288 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.11d( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 88/88/84) [0]/[0,1] r=-1 lpr=88 pi=61-87/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-3> 2013-08-15 20:50:29.773842 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.58( v 63'21 (0'0,63'21] local-les=62 n=5 ec=56 les/c 62/62 88/88/56) [3] r=-1 lpr=88 pi=56-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:29.774265 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.118( v 63'17 (0'0,63'17] local-les=62 n=7 ec=56 les/c 62/62 88/88/84) [3]/[3,1] r=-1 lpr=88 pi=61-87/12 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:29.774767 7f88e423d700 1 osd.8 pg_epoch: 88 pg[3.10d( v 63'24 (0'0,63'24] local-les=62 n=4 ec=56 les/c 62/62 88/88/56) [2] r=-1 lpr=88 pi=56-87/12 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:29.774850 7f88ef253700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f88ef253700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f88f6263bd0]
|
|
3: (gsignal()+0x37) [0x7f88f444b037]
|
|
4: (abort()+0x148) [0x7f88f444e698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f88f4d57e8d]
|
|
6: (()+0x5ef76) [0x7f88f4d55f76]
|
|
7: (()+0x5efa3) [0x7f88f4d55fa3]
|
|
8: (()+0x5f1de) [0x7f88f4d561de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f88f625bf8e]
|
|
14: (clone()+0x6d) [0x7f88f450de1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:29.999829 7fcae41a07c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9101
|
|
2013-08-15 20:50:30.000365 7fcae41a07c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:30.059787 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:30.059805 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:30.060111 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:30.077726 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:30.077853 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:30.114123 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:30.115068 7fcae41a07c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:30.115091 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.179778 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.180214 7fcae41a07c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
2013-08-15 20:50:30.215412 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:30.215434 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:30.215670 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:30.239356 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:30.239470 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:30.263707 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:30.264282 7fcae41a07c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:30.264303 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:30.327493 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:31.795865 7fcadcb74700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:31.795885 7fcadcb74700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
2013-08-15 20:50:31.797003 7fcadcb74700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fcadcb74700 time 2013-08-15 20:50:31.795891
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fcae3b7cf8e]
|
|
5: (clone()+0x6d) [0x7fcae1e2ee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-218> 2013-08-15 20:50:29.998528 7fcae41a07c0 5 asok(0x28840e0) register_command perfcounters_dump hook 0x287e010
|
|
-217> 2013-08-15 20:50:29.998557 7fcae41a07c0 5 asok(0x28840e0) register_command 1 hook 0x287e010
|
|
-216> 2013-08-15 20:50:29.998564 7fcae41a07c0 5 asok(0x28840e0) register_command perf dump hook 0x287e010
|
|
-215> 2013-08-15 20:50:29.998570 7fcae41a07c0 5 asok(0x28840e0) register_command perfcounters_schema hook 0x287e010
|
|
-214> 2013-08-15 20:50:29.998574 7fcae41a07c0 5 asok(0x28840e0) register_command 2 hook 0x287e010
|
|
-213> 2013-08-15 20:50:29.998577 7fcae41a07c0 5 asok(0x28840e0) register_command perf schema hook 0x287e010
|
|
-212> 2013-08-15 20:50:29.998582 7fcae41a07c0 5 asok(0x28840e0) register_command config show hook 0x287e010
|
|
-211> 2013-08-15 20:50:29.998587 7fcae41a07c0 5 asok(0x28840e0) register_command config set hook 0x287e010
|
|
-210> 2013-08-15 20:50:29.998591 7fcae41a07c0 5 asok(0x28840e0) register_command log flush hook 0x287e010
|
|
-209> 2013-08-15 20:50:29.998595 7fcae41a07c0 5 asok(0x28840e0) register_command log dump hook 0x287e010
|
|
-208> 2013-08-15 20:50:29.998599 7fcae41a07c0 5 asok(0x28840e0) register_command log reopen hook 0x287e010
|
|
-207> 2013-08-15 20:50:29.999829 7fcae41a07c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9101
|
|
-206> 2013-08-15 20:50:30.000365 7fcae41a07c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-205> 2013-08-15 20:50:30.000527 7fcae41a07c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6825/9101 need_addr=1
|
|
-204> 2013-08-15 20:50:30.000557 7fcae41a07c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-203> 2013-08-15 20:50:30.000565 7fcae41a07c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6830/9101 need_addr=0
|
|
-202> 2013-08-15 20:50:30.000583 7fcae41a07c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-201> 2013-08-15 20:50:30.000588 7fcae41a07c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6831/9101 need_addr=0
|
|
-200> 2013-08-15 20:50:30.002517 7fcae41a07c0 5 asok(0x28840e0) init /var/run/ceph/ceph-osd.8.asok
|
|
-199> 2013-08-15 20:50:30.002533 7fcae41a07c0 5 asok(0x28840e0) bind_and_listen /var/run/ceph/ceph-osd.8.asok
|
|
-198> 2013-08-15 20:50:30.002563 7fcae41a07c0 5 asok(0x28840e0) register_command 0 hook 0x287c0b8
|
|
-197> 2013-08-15 20:50:30.002569 7fcae41a07c0 5 asok(0x28840e0) register_command version hook 0x287c0b8
|
|
-196> 2013-08-15 20:50:30.002573 7fcae41a07c0 5 asok(0x28840e0) register_command git_version hook 0x287c0b8
|
|
-195> 2013-08-15 20:50:30.002578 7fcae41a07c0 5 asok(0x28840e0) register_command help hook 0x287e0d0
|
|
-194> 2013-08-15 20:50:30.002681 7fcadfb7a700 5 asok(0x28840e0) entry start
|
|
-193> 2013-08-15 20:50:30.059787 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-192> 2013-08-15 20:50:30.059805 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-191> 2013-08-15 20:50:30.060111 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-190> 2013-08-15 20:50:30.077726 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-189> 2013-08-15 20:50:30.077853 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-188> 2013-08-15 20:50:30.114123 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-187> 2013-08-15 20:50:30.115028 7fcae41a07c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-186> 2013-08-15 20:50:30.115068 7fcae41a07c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-185> 2013-08-15 20:50:30.115091 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-184> 2013-08-15 20:50:30.120655 7fcae41a07c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-183> 2013-08-15 20:50:30.125005 7fcae41a07c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-182> 2013-08-15 20:50:30.129338 7fcae41a07c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-181> 2013-08-15 20:50:30.133633 7fcae41a07c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-180> 2013-08-15 20:50:30.137925 7fcae41a07c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-179> 2013-08-15 20:50:30.142239 7fcae41a07c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-178> 2013-08-15 20:50:30.146535 7fcae41a07c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-177> 2013-08-15 20:50:30.151021 7fcae41a07c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-176> 2013-08-15 20:50:30.155228 7fcae41a07c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-175> 2013-08-15 20:50:30.159370 7fcae41a07c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-174> 2013-08-15 20:50:30.163486 7fcae41a07c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-173> 2013-08-15 20:50:30.167590 7fcae41a07c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-172> 2013-08-15 20:50:30.171600 7fcae41a07c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-171> 2013-08-15 20:50:30.175666 7fcae41a07c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-170> 2013-08-15 20:50:30.179675 7fcae41a07c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-169> 2013-08-15 20:50:30.179710 7fcae41a07c0 2 journal No further valid entries found, journal is most likely valid
|
|
-168> 2013-08-15 20:50:30.179718 7fcae41a07c0 2 journal No further valid entries found, journal is most likely valid
|
|
-167> 2013-08-15 20:50:30.179725 7fcae41a07c0 3 journal journal_replay: end of journal, done.
|
|
-166> 2013-08-15 20:50:30.179778 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-165> 2013-08-15 20:50:30.180111 7fcadcb74700 1 FileStore::op_tp worker finish
|
|
-164> 2013-08-15 20:50:30.180129 7fcadd375700 1 FileStore::op_tp worker finish
|
|
-163> 2013-08-15 20:50:30.180214 7fcae41a07c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
-162> 2013-08-15 20:50:30.180516 7fcae41a07c0 10 monclient(hunting): build_initial_monmap
|
|
-161> 2013-08-15 20:50:30.180567 7fcae41a07c0 5 adding auth protocol: cephx
|
|
-160> 2013-08-15 20:50:30.180574 7fcae41a07c0 5 adding auth protocol: cephx
|
|
-159> 2013-08-15 20:50:30.180655 7fcae41a07c0 1 -- 0.0.0.0:6825/9101 messenger.start
|
|
-158> 2013-08-15 20:50:30.180695 7fcae41a07c0 1 -- :/0 messenger.start
|
|
-157> 2013-08-15 20:50:30.180706 7fcae41a07c0 1 -- 9.4.196.3:6831/9101 messenger.start
|
|
-156> 2013-08-15 20:50:30.180716 7fcae41a07c0 1 -- 9.4.196.3:6830/9101 messenger.start
|
|
-155> 2013-08-15 20:50:30.180791 7fcae41a07c0 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
|
|
-154> 2013-08-15 20:50:30.215412 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-153> 2013-08-15 20:50:30.215434 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-152> 2013-08-15 20:50:30.215670 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-151> 2013-08-15 20:50:30.239356 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-150> 2013-08-15 20:50:30.239470 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-149> 2013-08-15 20:50:30.263707 7fcae41a07c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-148> 2013-08-15 20:50:30.264262 7fcae41a07c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-147> 2013-08-15 20:50:30.264282 7fcae41a07c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-146> 2013-08-15 20:50:30.264303 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-145> 2013-08-15 20:50:30.268610 7fcae41a07c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-144> 2013-08-15 20:50:30.272800 7fcae41a07c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-143> 2013-08-15 20:50:30.277010 7fcae41a07c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-142> 2013-08-15 20:50:30.281308 7fcae41a07c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-141> 2013-08-15 20:50:30.285575 7fcae41a07c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-140> 2013-08-15 20:50:30.289841 7fcae41a07c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-139> 2013-08-15 20:50:30.294165 7fcae41a07c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-138> 2013-08-15 20:50:30.298430 7fcae41a07c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-137> 2013-08-15 20:50:30.302646 7fcae41a07c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-136> 2013-08-15 20:50:30.306862 7fcae41a07c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-135> 2013-08-15 20:50:30.311046 7fcae41a07c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-134> 2013-08-15 20:50:30.315209 7fcae41a07c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-133> 2013-08-15 20:50:30.319301 7fcae41a07c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-132> 2013-08-15 20:50:30.323371 7fcae41a07c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-131> 2013-08-15 20:50:30.327434 7fcae41a07c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-130> 2013-08-15 20:50:30.327453 7fcae41a07c0 2 journal No further valid entries found, journal is most likely valid
|
|
-129> 2013-08-15 20:50:30.327458 7fcae41a07c0 2 journal No further valid entries found, journal is most likely valid
|
|
-128> 2013-08-15 20:50:30.327459 7fcae41a07c0 3 journal journal_replay: end of journal, done.
|
|
-127> 2013-08-15 20:50:30.327493 7fcae41a07c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-126> 2013-08-15 20:50:30.327683 7fcae41a07c0 2 osd.8 0 boot
|
|
-125> 2013-08-15 20:50:30.354775 7fcae41a07c0 2 osd.8 63 superblock: i am osd.8
|
|
-124> 2013-08-15 20:50:30.354833 7fcae41a07c0 1 accepter.accepter.start
|
|
-123> 2013-08-15 20:50:30.354898 7fcae41a07c0 1 accepter.accepter.start
|
|
-122> 2013-08-15 20:50:30.354942 7fcae41a07c0 1 accepter.accepter.start
|
|
-121> 2013-08-15 20:50:30.354958 7fcae41a07c0 10 monclient(hunting): init
|
|
-120> 2013-08-15 20:50:30.354965 7fcae41a07c0 5 adding auth protocol: cephx
|
|
-119> 2013-08-15 20:50:30.354969 7fcae41a07c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-118> 2013-08-15 20:50:30.355080 7fcae41a07c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-8/keyring
|
|
-117> 2013-08-15 20:50:30.355231 7fcae41a07c0 5 asok(0x28840e0) register_command dump_ops_in_flight hook 0x287e750
|
|
-116> 2013-08-15 20:50:30.355239 7fcae41a07c0 5 asok(0x28840e0) register_command dump_historic_ops hook 0x287e750
|
|
-115> 2013-08-15 20:50:30.355245 7fcae41a07c0 5 asok(0x28840e0) register_command dump_op_pq_state hook 0x287e750
|
|
-114> 2013-08-15 20:50:30.355251 7fcae41a07c0 5 asok(0x28840e0) register_command setomapval hook 0x2a72e80
|
|
-113> 2013-08-15 20:50:30.355257 7fcae41a07c0 5 asok(0x28840e0) register_command rmomapkey hook 0x2a72e80
|
|
-112> 2013-08-15 20:50:30.355262 7fcae41a07c0 5 asok(0x28840e0) register_command setomapheader hook 0x2a72e80
|
|
-111> 2013-08-15 20:50:30.355267 7fcae41a07c0 5 asok(0x28840e0) register_command getomap hook 0x2a72e80
|
|
-110> 2013-08-15 20:50:30.355271 7fcae41a07c0 5 asok(0x28840e0) register_command truncobj hook 0x2a72e80
|
|
-109> 2013-08-15 20:50:30.355276 7fcae41a07c0 5 asok(0x28840e0) register_command injectdataerr hook 0x2a72e80
|
|
-108> 2013-08-15 20:50:30.355280 7fcae41a07c0 5 asok(0x28840e0) register_command injectmdataerr hook 0x2a72e80
|
|
-107> 2013-08-15 20:50:30.355273 7fcacfb5a700 5 osd.8 63 heartbeat: osd_stat(1022 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-106> 2013-08-15 20:50:30.355348 7fcae41a07c0 10 monclient(hunting): _reopen_session
|
|
-105> 2013-08-15 20:50:30.355418 7fcae41a07c0 10 monclient(hunting): _pick_new_mon picked mon.noname-c con 0x2adc840 addr 9.4.196.1:6789/0
|
|
-104> 2013-08-15 20:50:30.355432 7fcae41a07c0 10 monclient(hunting): _send_mon_message to mon.noname-c at 9.4.196.1:6789/0
|
|
-103> 2013-08-15 20:50:30.355436 7fcae41a07c0 1 -- 0.0.0.0:6825/9101 --> 9.4.196.1:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x2890b40 con 0x2adc840
|
|
-102> 2013-08-15 20:50:30.355447 7fcae41a07c0 10 monclient(hunting): renew_subs
|
|
-101> 2013-08-15 20:50:30.355852 7fcae4132700 1 -- 9.4.196.3:6825/9101 learned my addr 9.4.196.3:6825/9101
|
|
-100> 2013-08-15 20:50:30.356015 7fcad6b68700 10 monclient(hunting): renew_subs
|
|
-99> 2013-08-15 20:50:30.356428 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 1 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x29061e0 con 0x2adc840
|
|
-98> 2013-08-15 20:50:30.356458 7fcad6b68700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-97> 2013-08-15 20:50:30.356473 7fcad6b68700 10 monclient(hunting): got monmap 1, mon.noname-c is now rank -1
|
|
-96> 2013-08-15 20:50:30.356478 7fcad6b68700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-95> 2013-08-15 20:50:30.356500 7fcad6b68700 1 monclient(hunting): found mon.zc2store
|
|
-94> 2013-08-15 20:50:30.356521 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (2237974531 0 0) 0x2ae6200 con 0x2adc840
|
|
-93> 2013-08-15 20:50:30.356543 7fcad6b68700 10 monclient: my global_id is 4392
|
|
-92> 2013-08-15 20:50:30.356641 7fcad6b68700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-91> 2013-08-15 20:50:30.356650 7fcad6b68700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x2891d40 con 0x2adc840
|
|
-90> 2013-08-15 20:50:30.357193 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1923290671 0 0) 0x2ae6000 con 0x2adc840
|
|
-89> 2013-08-15 20:50:30.357273 7fcad6b68700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-88> 2013-08-15 20:50:30.357281 7fcad6b68700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x2aea240 con 0x2adc840
|
|
-87> 2013-08-15 20:50:30.357799 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (2169053487 0 0) 0x2ae6400 con 0x2adc840
|
|
-86> 2013-08-15 20:50:30.357848 7fcad6b68700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-85> 2013-08-15 20:50:30.357855 7fcad6b68700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x28c6380 con 0x2adc840
|
|
-84> 2013-08-15 20:50:30.357876 7fcad6b68700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-83> 2013-08-15 20:50:30.357885 7fcad6b68700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x28c6540 con 0x2adc840
|
|
-82> 2013-08-15 20:50:30.357904 7fcad6b68700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:00.357904)
|
|
-81> 2013-08-15 20:50:30.357914 7fcad6b68700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-80> 2013-08-15 20:50:30.357921 7fcad6b68700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x2aea000 con 0x2adc840
|
|
-79> 2013-08-15 20:50:30.357944 7fcae41a07c0 5 monclient: authenticate success, global_id 4392
|
|
-78> 2013-08-15 20:50:30.357957 7fcae41a07c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:00.357956)
|
|
-77> 2013-08-15 20:50:30.358219 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 5 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x29065a0 con 0x2adc840
|
|
-76> 2013-08-15 20:50:30.358235 7fcad6b68700 10 monclient: handle_monmap mon_map v1
|
|
-75> 2013-08-15 20:50:30.358248 7fcad6b68700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-74> 2013-08-15 20:50:30.358255 7fcad6b68700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-73> 2013-08-15 20:50:30.358278 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x28c6380 con 0x2adc840
|
|
-72> 2013-08-15 20:50:30.358287 7fcad6b68700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:30.355450 renew after 2013-08-15 20:53:00.355450
|
|
-71> 2013-08-15 20:50:30.358298 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 7 ==== mon_map v1 ==== 498+0+0 (2134428392 0 0) 0x29063c0 con 0x2adc840
|
|
-70> 2013-08-15 20:50:30.358306 7fcad6b68700 10 monclient: handle_monmap mon_map v1
|
|
-69> 2013-08-15 20:50:30.358313 7fcad6b68700 10 monclient: got monmap 1, mon.zc2store is now rank 0
|
|
-68> 2013-08-15 20:50:30.358318 7fcad6b68700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 9.4.196.5:6789/0 mon.ceph-dmon2
|
|
|
|
-67> 2013-08-15 20:50:30.358337 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x28c6700 con 0x2adc840
|
|
-66> 2013-08-15 20:50:30.358346 7fcad6b68700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-65> 2013-08-15 20:50:30.358349 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (35315552 0 0) 0x2ae6600 con 0x2adc840
|
|
-64> 2013-08-15 20:50:30.358387 7fcad6b68700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:00.358387)
|
|
-63> 2013-08-15 20:50:30.358429 7fcae41a07c0 10 monclient: wait_auth_rotating done
|
|
-62> 2013-08-15 20:50:30.358919 7fcae41a07c0 10 monclient: get_version osdmap
|
|
-61> 2013-08-15 20:50:30.358931 7fcae41a07c0 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-60> 2013-08-15 20:50:30.358942 7fcae41a07c0 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x28ec9c0 con 0x2adc840
|
|
-59> 2013-08-15 20:50:30.359344 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 10 ==== mon_check_map_ack(handle=1 version=89) v2 ==== 24+0+0 (3324254003 0 0) 0x28c6540 con 0x2adc840
|
|
-58> 2013-08-15 20:50:30.359385 7fcad2b60700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-57> 2013-08-15 20:50:30.359400 7fcad2b60700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- osd_boot(osd.8 booted 0 v63) v3 -- ?+0 0x2b0a000 con 0x2adc840
|
|
-56> 2013-08-15 20:50:31.784895 7fcacd84c700 1 -- 9.4.196.3:6830/9101 >> :/0 pipe(0x2ae1180 sd=33 :6830 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.3:38898/0
|
|
-55> 2013-08-15 20:50:31.785563 7fcacd64a700 1 -- 9.4.196.3:6831/9101 >> :/0 pipe(0x2ae1400 sd=34 :6831 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.1:54234/0
|
|
-54> 2013-08-15 20:50:31.786993 7fcad5b66700 1 -- 9.4.196.3:6830/9101 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x2aeb440 con 0x2add340
|
|
-53> 2013-08-15 20:50:31.787027 7fcad5b66700 3 osd.8 63 handle_osd_map epochs [90,91], i have 63, src has [1,91]
|
|
-52> 2013-08-15 20:50:31.787038 7fcad5b66700 10 monclient: renew_subs
|
|
-51> 2013-08-15 20:50:31.787045 7fcad5b66700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-50> 2013-08-15 20:50:31.787055 7fcad5b66700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x28c6a80 con 0x2adc840
|
|
-49> 2013-08-15 20:50:31.787099 7fcad5b66700 1 -- 9.4.196.3:6830/9101 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_notify(3.18d(10) epoch 91) v4 ==== 982+0+0 (1473456836 0 0) 0x28c68c0 con 0x2add340
|
|
-48> 2013-08-15 20:50:31.787123 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.787012, event: header_read, request: pg_notify(3.18d(10) epoch 91) v4
|
|
-47> 2013-08-15 20:50:31.787136 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.787014, event: throttled, request: pg_notify(3.18d(10) epoch 91) v4
|
|
-46> 2013-08-15 20:50:31.787143 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.787081, event: all_read, request: pg_notify(3.18d(10) epoch 91) v4
|
|
-45> 2013-08-15 20:50:31.787149 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.787116, event: dispatched, request: pg_notify(3.18d(10) epoch 91) v4
|
|
-44> 2013-08-15 20:50:31.787155 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.787155, event: waiting_for_osdmap, request: pg_notify(3.18d(10) epoch 91) v4
|
|
-43> 2013-08-15 20:50:31.787166 7fcad5b66700 10 monclient: renew_subs
|
|
-42> 2013-08-15 20:50:31.787170 7fcad5b66700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-41> 2013-08-15 20:50:31.787176 7fcad5b66700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x28c6e00 con 0x2adc840
|
|
-40> 2013-08-15 20:50:31.787195 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:31.787194, event: wait for new map, request: pg_notify(3.18d(10) epoch 91) v4
|
|
-39> 2013-08-15 20:50:31.787336 7fcad5b66700 1 -- 9.4.196.3:6830/9101 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x2aebb00 con 0x2add8c0
|
|
-38> 2013-08-15 20:50:31.787356 7fcad5b66700 3 osd.8 63 handle_osd_map epochs [90,91], i have 63, src has [1,91]
|
|
-37> 2013-08-15 20:50:31.787366 7fcad5b66700 10 monclient: renew_subs
|
|
-36> 2013-08-15 20:50:31.787371 7fcad5b66700 10 monclient: _send_mon_message to mon.zc2store at 9.4.196.1:6789/0
|
|
-35> 2013-08-15 20:50:31.787376 7fcad5b66700 1 -- 9.4.196.3:6825/9101 --> 9.4.196.1:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x28c6c40 con 0x2adc840
|
|
-34> 2013-08-15 20:50:31.787915 7fcad5b66700 1 -- 9.4.196.3:6830/9101 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(0.9 epoch 91) v2 ==== 137+0+0 (4255530542 0 0) 0x29061e0 con 0x2add8c0
|
|
-33> 2013-08-15 20:50:31.787933 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787855, event: header_read, request: pg_query(0.9 epoch 91) v2
|
|
-32> 2013-08-15 20:50:31.787941 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787857, event: throttled, request: pg_query(0.9 epoch 91) v2
|
|
-31> 2013-08-15 20:50:31.787947 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787900, event: all_read, request: pg_query(0.9 epoch 91) v2
|
|
-30> 2013-08-15 20:50:31.787952 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787930, event: dispatched, request: pg_query(0.9 epoch 91) v2
|
|
-29> 2013-08-15 20:50:31.787959 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787959, event: waiting_for_osdmap, request: pg_query(0.9 epoch 91) v2
|
|
-28> 2013-08-15 20:50:31.787969 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:31.787969, event: wait for new map, request: pg_query(0.9 epoch 91) v2
|
|
-27> 2013-08-15 20:50:31.788020 7fcacd448700 1 -- 9.4.196.3:6831/9101 >> :/0 pipe(0x2ae1900 sd=35 :6831 s=0 pgs=0 cs=0 l=0).accept sd=35 9.4.196.1:54235/0
|
|
-26> 2013-08-15 20:50:31.788614 7fcad5b66700 1 -- 9.4.196.3:6830/9101 <== osd.1 9.4.196.1:6801/5083 3 ==== pg_query(3.182 epoch 91) v2 ==== 137+0+0 (1756850204 0 0) 0x29065a0 con 0x2add340
|
|
-25> 2013-08-15 20:50:31.788632 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.788559, event: header_read, request: pg_query(3.182 epoch 91) v2
|
|
-24> 2013-08-15 20:50:31.788642 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.788560, event: throttled, request: pg_query(3.182 epoch 91) v2
|
|
-23> 2013-08-15 20:50:31.788651 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.788598, event: all_read, request: pg_query(3.182 epoch 91) v2
|
|
-22> 2013-08-15 20:50:31.788659 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.788630, event: dispatched, request: pg_query(3.182 epoch 91) v2
|
|
-21> 2013-08-15 20:50:31.788666 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.788666, event: waiting_for_osdmap, request: pg_query(3.182 epoch 91) v2
|
|
-20> 2013-08-15 20:50:31.788673 7fcad5b66700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:31.788673, event: wait for new map, request: pg_query(3.182 epoch 91) v2
|
|
-19> 2013-08-15 20:50:31.788765 7fcad6b68700 1 -- 9.4.196.3:6825/9101 <== mon.0 9.4.196.1:6789/0 11 ==== osd_map(64..90 src has 1..90) v3 ==== 26633+0+0 (225616071 0 0) 0x2aeb440 con 0x2adc840
|
|
-18> 2013-08-15 20:50:31.788791 7fcad6b68700 3 osd.8 63 handle_osd_map epochs [64,90], i have 63, src has [1,90]
|
|
-17> 2013-08-15 20:50:31.790692 7fcacd246700 1 -- 9.4.196.3:6830/9101 >> :/0 pipe(0x2ae1b80 sd=36 :6830 s=0 pgs=0 cs=0 l=0).accept sd=36 9.4.196.3:38899/0
|
|
-16> 2013-08-15 20:50:31.794793 7fcad5b66700 1 -- 9.4.196.3:6830/9101 <== osd.3 9.4.196.1:6810/7799 1 ==== osd_map(90..91 src has 1..91) v3 ==== 2429+0+0 (3244034295 0 0) 0x2b74900 con 0x2add4a0
|
|
-15> 2013-08-15 20:50:31.795054 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-14> 2013-08-15 20:50:31.795071 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-13> 2013-08-15 20:50:31.795081 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-12> 2013-08-15 20:50:31.795109 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-11> 2013-08-15 20:50:31.795130 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-10> 2013-08-15 20:50:31.795139 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-9> 2013-08-15 20:50:31.795180 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-8> 2013-08-15 20:50:31.795190 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-7> 2013-08-15 20:50:31.795207 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-6> 2013-08-15 20:50:31.795216 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-5> 2013-08-15 20:50:31.795231 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6826/8751 -- pipe dne
|
|
-4> 2013-08-15 20:50:31.795240 7fcad6b68700 1 -- 9.4.196.3:6830/9101 mark_down 9.4.196.3:6813/8749 -- pipe dne
|
|
-3> 2013-08-15 20:50:31.795865 7fcadcb74700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-2> 2013-08-15 20:50:31.795885 7fcadcb74700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
-1> 2013-08-15 20:50:31.796234 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.69( v 63'14 (0'0,63'14] local-les=62 n=4 ec=56 les/c 62/62 90/90/56) [1] r=-1 lpr=90 pi=56-89/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:31.797003 7fcadcb74700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7fcadcb74700 time 2013-08-15 20:50:31.795891
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7fcae3b7cf8e]
|
|
5: (clone()+0x6d) [0x7fcae1e2ee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:31.802343 7fcadcb74700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fcadcb74700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fcae3b84bd0]
|
|
3: (gsignal()+0x37) [0x7fcae1d6c037]
|
|
4: (abort()+0x148) [0x7fcae1d6f698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fcae2678e8d]
|
|
6: (()+0x5ef76) [0x7fcae2676f76]
|
|
7: (()+0x5efa3) [0x7fcae2676fa3]
|
|
8: (()+0x5f1de) [0x7fcae26771de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fcae3b7cf8e]
|
|
14: (clone()+0x6d) [0x7fcae1e2ee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-8> 2013-08-15 20:50:31.797046 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.1ea( v 63'13 (0'0,63'13] local-les=62 n=9 ec=56 les/c 62/62 90/90/84) [2]/[2,0] r=-1 lpr=90 pi=61-89/14 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-7> 2013-08-15 20:50:31.798138 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.12b( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 90/90/56) [3] r=-1 lpr=90 pi=56-89/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-6> 2013-08-15 20:50:31.798397 7fcacd044700 1 -- 9.4.196.3:6830/9101 >> :/0 pipe(0x2c40f00 sd=37 :6830 s=0 pgs=0 cs=0 l=0).accept sd=37 9.4.196.3:38903/0
|
|
-5> 2013-08-15 20:50:31.798895 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.125( v 63'23 (0'0,63'23] local-les=62 n=7 ec=56 les/c 62/62 90/90/56) [1] r=-1 lpr=90 pi=56-89/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-4> 2013-08-15 20:50:31.799460 7fcacce42700 1 -- 9.4.196.3:6831/9101 >> :/0 pipe(0x2ae1680 sd=38 :6831 s=0 pgs=0 cs=0 l=0).accept sd=38 9.4.196.1:54238/0
|
|
-3> 2013-08-15 20:50:31.800232 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.66( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 90/90/84) [0]/[0,3] r=-1 lpr=90 pi=61-89/14 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:31.801000 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.11c( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 90/90/56) [3] r=-1 lpr=90 pi=56-89/14 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-1> 2013-08-15 20:50:31.801984 7fcad1b5e700 1 osd.8 pg_epoch: 90 pg[3.11d( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 90/90/84) [0]/[0,1] r=-1 lpr=90 pi=61-89/14 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:31.802343 7fcadcb74700 -1 *** Caught signal (Aborted) **
|
|
in thread 7fcadcb74700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7fcae3b84bd0]
|
|
3: (gsignal()+0x37) [0x7fcae1d6c037]
|
|
4: (abort()+0x148) [0x7fcae1d6f698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fcae2678e8d]
|
|
6: (()+0x5ef76) [0x7fcae2676f76]
|
|
7: (()+0x5efa3) [0x7fcae2676fa3]
|
|
8: (()+0x5f1de) [0x7fcae26771de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7fcae3b7cf8e]
|
|
14: (clone()+0x6d) [0x7fcae1e2ee1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:32.112492 7f9dfd3fb7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9433
|
|
2013-08-15 20:50:32.113046 7f9dfd3fb7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
2013-08-15 20:50:32.171005 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:32.171031 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:32.171305 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:32.188915 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:32.188986 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:32.213327 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:32.214215 7f9dfd3fb7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:32.214243 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.278825 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.279206 7f9dfd3fb7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
2013-08-15 20:50:32.314589 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
2013-08-15 20:50:32.314603 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
2013-08-15 20:50:32.314858 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
2013-08-15 20:50:32.338646 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
2013-08-15 20:50:32.338690 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
2013-08-15 20:50:32.362800 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
2013-08-15 20:50:32.363375 7f9dfd3fb7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
2013-08-15 20:50:32.363388 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:32.427139 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
2013-08-15 20:50:33.763293 7f9df5dcf700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
2013-08-15 20:50:33.763305 7f9df5dcf700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
2013-08-15 20:50:33.764339 7f9df5dcf700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f9df5dcf700 time 2013-08-15 20:50:33.763311
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f9dfcdd7f8e]
|
|
5: (clone()+0x6d) [0x7f9dfb089e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-226> 2013-08-15 20:50:32.111121 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command perfcounters_dump hook 0x1b94010
|
|
-225> 2013-08-15 20:50:32.111151 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command 1 hook 0x1b94010
|
|
-224> 2013-08-15 20:50:32.111156 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command perf dump hook 0x1b94010
|
|
-223> 2013-08-15 20:50:32.111164 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command perfcounters_schema hook 0x1b94010
|
|
-222> 2013-08-15 20:50:32.111168 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command 2 hook 0x1b94010
|
|
-221> 2013-08-15 20:50:32.111171 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command perf schema hook 0x1b94010
|
|
-220> 2013-08-15 20:50:32.111177 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command config show hook 0x1b94010
|
|
-219> 2013-08-15 20:50:32.111181 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command config set hook 0x1b94010
|
|
-218> 2013-08-15 20:50:32.111185 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command log flush hook 0x1b94010
|
|
-217> 2013-08-15 20:50:32.111189 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command log dump hook 0x1b94010
|
|
-216> 2013-08-15 20:50:32.111193 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command log reopen hook 0x1b94010
|
|
-215> 2013-08-15 20:50:32.112492 7f9dfd3fb7c0 0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff), process ceph-osd, pid 9433
|
|
-214> 2013-08-15 20:50:32.113046 7f9dfd3fb7c0 -1 [0;33m ** WARNING: specified cluster addr but not public addr; we recommend **
|
|
** you specify neither or both. **[0m
|
|
-213> 2013-08-15 20:50:32.113168 7f9dfd3fb7c0 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6805/9433 need_addr=1
|
|
-212> 2013-08-15 20:50:32.113204 7f9dfd3fb7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-211> 2013-08-15 20:50:32.113221 7f9dfd3fb7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6825/9433 need_addr=0
|
|
-210> 2013-08-15 20:50:32.113245 7f9dfd3fb7c0 1 -- 9.4.196.3:0/0 learned my addr 9.4.196.3:0/0
|
|
-209> 2013-08-15 20:50:32.113250 7f9dfd3fb7c0 1 accepter.accepter.bind my_inst.addr is 9.4.196.3:6832/9433 need_addr=0
|
|
-208> 2013-08-15 20:50:32.115325 7f9dfd3fb7c0 5 asok(0x1b9a0e0) init /var/run/ceph/ceph-osd.8.asok
|
|
-207> 2013-08-15 20:50:32.115341 7f9dfd3fb7c0 5 asok(0x1b9a0e0) bind_and_listen /var/run/ceph/ceph-osd.8.asok
|
|
-206> 2013-08-15 20:50:32.115373 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command 0 hook 0x1b920b8
|
|
-205> 2013-08-15 20:50:32.115379 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command version hook 0x1b920b8
|
|
-204> 2013-08-15 20:50:32.115385 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command git_version hook 0x1b920b8
|
|
-203> 2013-08-15 20:50:32.115391 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command help hook 0x1b940d0
|
|
-202> 2013-08-15 20:50:32.115434 7f9df8dd5700 5 asok(0x1b9a0e0) entry start
|
|
-201> 2013-08-15 20:50:32.171005 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-200> 2013-08-15 20:50:32.171031 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-199> 2013-08-15 20:50:32.171305 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-198> 2013-08-15 20:50:32.188915 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-197> 2013-08-15 20:50:32.188986 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-196> 2013-08-15 20:50:32.213327 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-195> 2013-08-15 20:50:32.214188 7f9dfd3fb7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-194> 2013-08-15 20:50:32.214215 7f9dfd3fb7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-193> 2013-08-15 20:50:32.214243 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-192> 2013-08-15 20:50:32.219814 7f9dfd3fb7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-191> 2013-08-15 20:50:32.224088 7f9dfd3fb7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-190> 2013-08-15 20:50:32.228348 7f9dfd3fb7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-189> 2013-08-15 20:50:32.232588 7f9dfd3fb7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-188> 2013-08-15 20:50:32.236787 7f9dfd3fb7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-187> 2013-08-15 20:50:32.241026 7f9dfd3fb7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-186> 2013-08-15 20:50:32.245239 7f9dfd3fb7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-185> 2013-08-15 20:50:32.249434 7f9dfd3fb7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-184> 2013-08-15 20:50:32.253625 7f9dfd3fb7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-183> 2013-08-15 20:50:32.257817 7f9dfd3fb7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-182> 2013-08-15 20:50:32.262049 7f9dfd3fb7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-181> 2013-08-15 20:50:32.266244 7f9dfd3fb7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-180> 2013-08-15 20:50:32.270516 7f9dfd3fb7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-179> 2013-08-15 20:50:32.274644 7f9dfd3fb7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-178> 2013-08-15 20:50:32.278716 7f9dfd3fb7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-177> 2013-08-15 20:50:32.278760 7f9dfd3fb7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-176> 2013-08-15 20:50:32.278766 7f9dfd3fb7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-175> 2013-08-15 20:50:32.278778 7f9dfd3fb7c0 3 journal journal_replay: end of journal, done.
|
|
-174> 2013-08-15 20:50:32.278825 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 18: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-173> 2013-08-15 20:50:32.279134 7f9df5dcf700 1 FileStore::op_tp worker finish
|
|
-172> 2013-08-15 20:50:32.279147 7f9df65d0700 1 FileStore::op_tp worker finish
|
|
-171> 2013-08-15 20:50:32.279206 7f9dfd3fb7c0 1 journal close /var/lib/ceph/osd/ceph-8/journal
|
|
-170> 2013-08-15 20:50:32.279505 7f9dfd3fb7c0 10 monclient(hunting): build_initial_monmap
|
|
-169> 2013-08-15 20:50:32.279559 7f9dfd3fb7c0 5 adding auth protocol: cephx
|
|
-168> 2013-08-15 20:50:32.279566 7f9dfd3fb7c0 5 adding auth protocol: cephx
|
|
-167> 2013-08-15 20:50:32.279655 7f9dfd3fb7c0 1 -- 0.0.0.0:6805/9433 messenger.start
|
|
-166> 2013-08-15 20:50:32.279686 7f9dfd3fb7c0 1 -- :/0 messenger.start
|
|
-165> 2013-08-15 20:50:32.279697 7f9dfd3fb7c0 1 -- 9.4.196.3:6832/9433 messenger.start
|
|
-164> 2013-08-15 20:50:32.279708 7f9dfd3fb7c0 1 -- 9.4.196.3:6825/9433 messenger.start
|
|
-163> 2013-08-15 20:50:32.279802 7f9dfd3fb7c0 2 osd.8 0 mounting /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal
|
|
-162> 2013-08-15 20:50:32.314589 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is supported and appears to work
|
|
-161> 2013-08-15 20:50:32.314603 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
|
|
-160> 2013-08-15 20:50:32.314858 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount did NOT detect btrfs
|
|
-159> 2013-08-15 20:50:32.338646 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount syncfs(2) syscall fully supported (by glibc and kernel)
|
|
-158> 2013-08-15 20:50:32.338690 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount found snaps <>
|
|
-157> 2013-08-15 20:50:32.362800 7f9dfd3fb7c0 0 filestore(/var/lib/ceph/osd/ceph-8) mount: enabling WRITEAHEAD journal mode: btrfs not detected
|
|
-156> 2013-08-15 20:50:32.363355 7f9dfd3fb7c0 2 journal open /var/lib/ceph/osd/ceph-8/journal fsid 3ce6bfee-6cb2-40e9-a25d-4913b345bca1 fs_op_seq 2428
|
|
-155> 2013-08-15 20:50:32.363375 7f9dfd3fb7c0 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
|
|
-154> 2013-08-15 20:50:32.363388 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-153> 2013-08-15 20:50:32.367676 7f9dfd3fb7c0 2 journal read_entry 3612495872 : seq 2414 4195925 bytes
|
|
-152> 2013-08-15 20:50:32.371778 7f9dfd3fb7c0 2 journal read_entry 3616698368 : seq 2415 4195919 bytes
|
|
-151> 2013-08-15 20:50:32.375802 7f9dfd3fb7c0 2 journal read_entry 3620900864 : seq 2416 4195919 bytes
|
|
-150> 2013-08-15 20:50:32.379847 7f9dfd3fb7c0 2 journal read_entry 3625103360 : seq 2417 4195919 bytes
|
|
-149> 2013-08-15 20:50:32.383852 7f9dfd3fb7c0 2 journal read_entry 3629305856 : seq 2418 4195925 bytes
|
|
-148> 2013-08-15 20:50:32.387928 7f9dfd3fb7c0 2 journal read_entry 3633508352 : seq 2419 4195919 bytes
|
|
-147> 2013-08-15 20:50:32.392349 7f9dfd3fb7c0 2 journal read_entry 3637710848 : seq 2420 4195919 bytes
|
|
-146> 2013-08-15 20:50:32.396766 7f9dfd3fb7c0 2 journal read_entry 3641913344 : seq 2421 4195913 bytes
|
|
-145> 2013-08-15 20:50:32.401142 7f9dfd3fb7c0 2 journal read_entry 3646115840 : seq 2422 4195919 bytes
|
|
-144> 2013-08-15 20:50:32.405508 7f9dfd3fb7c0 2 journal read_entry 3650318336 : seq 2423 4195925 bytes
|
|
-143> 2013-08-15 20:50:32.409862 7f9dfd3fb7c0 2 journal read_entry 3654520832 : seq 2424 4195925 bytes
|
|
-142> 2013-08-15 20:50:32.414192 7f9dfd3fb7c0 2 journal read_entry 3658723328 : seq 2425 4195925 bytes
|
|
-141> 2013-08-15 20:50:32.418515 7f9dfd3fb7c0 2 journal read_entry 3662925824 : seq 2426 4195925 bytes
|
|
-140> 2013-08-15 20:50:32.422815 7f9dfd3fb7c0 2 journal read_entry 3667128320 : seq 2427 4195925 bytes
|
|
-139> 2013-08-15 20:50:32.427084 7f9dfd3fb7c0 2 journal read_entry 3671330816 : seq 2428 4195925 bytes
|
|
-138> 2013-08-15 20:50:32.427104 7f9dfd3fb7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-137> 2013-08-15 20:50:32.427110 7f9dfd3fb7c0 2 journal No further valid entries found, journal is most likely valid
|
|
-136> 2013-08-15 20:50:32.427113 7f9dfd3fb7c0 3 journal journal_replay: end of journal, done.
|
|
-135> 2013-08-15 20:50:32.427139 7f9dfd3fb7c0 1 journal _open /var/lib/ceph/osd/ceph-8/journal fd 26: 5368709120 bytes, block size 4096 bytes, directio = 1, aio = 0
|
|
-134> 2013-08-15 20:50:32.427315 7f9dfd3fb7c0 2 osd.8 0 boot
|
|
-133> 2013-08-15 20:50:32.453967 7f9dfd3fb7c0 2 osd.8 63 superblock: i am osd.8
|
|
-132> 2013-08-15 20:50:32.454020 7f9dfd3fb7c0 1 accepter.accepter.start
|
|
-131> 2013-08-15 20:50:32.454054 7f9dfd3fb7c0 1 accepter.accepter.start
|
|
-130> 2013-08-15 20:50:32.454092 7f9dfd3fb7c0 1 accepter.accepter.start
|
|
-129> 2013-08-15 20:50:32.454112 7f9dfd3fb7c0 10 monclient(hunting): init
|
|
-128> 2013-08-15 20:50:32.454118 7f9dfd3fb7c0 5 adding auth protocol: cephx
|
|
-127> 2013-08-15 20:50:32.454120 7f9dfd3fb7c0 10 monclient(hunting): auth_supported 2 method cephx
|
|
-126> 2013-08-15 20:50:32.454244 7f9dfd3fb7c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-8/keyring
|
|
-125> 2013-08-15 20:50:32.454431 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command dump_ops_in_flight hook 0x1b94750
|
|
-124> 2013-08-15 20:50:32.454441 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command dump_historic_ops hook 0x1b94750
|
|
-123> 2013-08-15 20:50:32.454449 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command dump_op_pq_state hook 0x1b94750
|
|
-122> 2013-08-15 20:50:32.454452 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command setomapval hook 0x1d88e80
|
|
-121> 2013-08-15 20:50:32.454459 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command rmomapkey hook 0x1d88e80
|
|
-120> 2013-08-15 20:50:32.454465 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command setomapheader hook 0x1d88e80
|
|
-119> 2013-08-15 20:50:32.454470 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command getomap hook 0x1d88e80
|
|
-118> 2013-08-15 20:50:32.454475 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command truncobj hook 0x1d88e80
|
|
-117> 2013-08-15 20:50:32.454479 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command injectdataerr hook 0x1d88e80
|
|
-116> 2013-08-15 20:50:32.454481 7f9dfd3fb7c0 5 asok(0x1b9a0e0) register_command injectmdataerr hook 0x1d88e80
|
|
-115> 2013-08-15 20:50:32.454473 7f9de8db5700 5 osd.8 63 heartbeat: osd_stat(1022 MB used, 278 GB avail, 279 GB total, peers []/[])
|
|
-114> 2013-08-15 20:50:32.454535 7f9dfd3fb7c0 10 monclient(hunting): _reopen_session
|
|
-113> 2013-08-15 20:50:32.454584 7f9dfd3fb7c0 10 monclient(hunting): _pick_new_mon picked mon.noname-a con 0x1df2840 addr 9.4.196.4:6789/0
|
|
-112> 2013-08-15 20:50:32.454599 7f9dfd3fb7c0 10 monclient(hunting): _send_mon_message to mon.noname-a at 9.4.196.4:6789/0
|
|
-111> 2013-08-15 20:50:32.454604 7f9dfd3fb7c0 1 -- 0.0.0.0:6805/9433 --> 9.4.196.4:6789/0 -- auth(proto 0 26 bytes epoch 0) v1 -- ?+0 0x1ba6b40 con 0x1df2840
|
|
-110> 2013-08-15 20:50:32.454615 7f9dfd3fb7c0 10 monclient(hunting): renew_subs
|
|
-109> 2013-08-15 20:50:32.454994 7f9dfd38d700 1 -- 9.4.196.3:6805/9433 learned my addr 9.4.196.3:6805/9433
|
|
-108> 2013-08-15 20:50:32.455253 7f9defdc3700 10 monclient(hunting): renew_subs
|
|
-107> 2013-08-15 20:50:32.455701 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 1 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x1c1c1e0 con 0x1df2840
|
|
-106> 2013-08-15 20:50:32.455721 7f9defdc3700 10 monclient(hunting): handle_monmap mon_map v1
|
|
-105> 2013-08-15 20:50:32.455736 7f9defdc3700 10 monclient(hunting): got monmap 1, mon.noname-a is now rank -1
|
|
-104> 2013-08-15 20:50:32.455741 7f9defdc3700 10 monclient(hunting): dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-103> 2013-08-15 20:50:32.455760 7f9defdc3700 1 monclient(hunting): found mon.ceph-dmon1
|
|
-102> 2013-08-15 20:50:32.455767 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 2 ==== auth_reply(proto 2 0 Success) v1 ==== 33+0+0 (1105302022 0 0) 0x1dfc200 con 0x1df2840
|
|
-101> 2013-08-15 20:50:32.455788 7f9defdc3700 10 monclient: my global_id is 4327
|
|
-100> 2013-08-15 20:50:32.455896 7f9defdc3700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-99> 2013-08-15 20:50:32.455906 7f9defdc3700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x1ba7d40 con 0x1df2840
|
|
-98> 2013-08-15 20:50:32.456456 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 3 ==== auth_reply(proto 2 0 Success) v1 ==== 206+0+0 (1998789652 0 0) 0x1dfc000 con 0x1df2840
|
|
-97> 2013-08-15 20:50:32.456527 7f9defdc3700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-96> 2013-08-15 20:50:32.456534 7f9defdc3700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- ?+0 0x1e00240 con 0x1df2840
|
|
-95> 2013-08-15 20:50:32.457159 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 4 ==== auth_reply(proto 2 0 Success) v1 ==== 393+0+0 (1894879301 0 0) 0x1dfc400 con 0x1df2840
|
|
-94> 2013-08-15 20:50:32.457209 7f9defdc3700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-93> 2013-08-15 20:50:32.457215 7f9defdc3700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x1bdc380 con 0x1df2840
|
|
-92> 2013-08-15 20:50:32.457230 7f9defdc3700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-91> 2013-08-15 20:50:32.457236 7f9defdc3700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=0+,osd_pg_creates=0}) v2 -- ?+0 0x1bdc540 con 0x1df2840
|
|
-90> 2013-08-15 20:50:32.457245 7f9defdc3700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2013-08-15 20:50:02.457245)
|
|
-89> 2013-08-15 20:50:32.457255 7f9defdc3700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-88> 2013-08-15 20:50:32.457258 7f9defdc3700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- ?+0 0x1e00000 con 0x1df2840
|
|
-87> 2013-08-15 20:50:32.457271 7f9dfd3fb7c0 5 monclient: authenticate success, global_id 4327
|
|
-86> 2013-08-15 20:50:32.457283 7f9dfd3fb7c0 10 monclient: wait_auth_rotating waiting (until 2013-08-15 20:51:02.457282)
|
|
-85> 2013-08-15 20:50:32.457623 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 5 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x1c1c5a0 con 0x1df2840
|
|
-84> 2013-08-15 20:50:32.457636 7f9defdc3700 10 monclient: handle_monmap mon_map v1
|
|
-83> 2013-08-15 20:50:32.457645 7f9defdc3700 10 monclient: got monmap 1, mon.ceph-dmon1 is now rank 1
|
|
-82> 2013-08-15 20:50:32.457649 7f9defdc3700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-81> 2013-08-15 20:50:32.457667 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1bdc380 con 0x1df2840
|
|
-80> 2013-08-15 20:50:32.457676 7f9defdc3700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:32.454618 renew after 2013-08-15 20:53:02.454618
|
|
-79> 2013-08-15 20:50:32.457687 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 7 ==== mon_map v1 ==== 498+0+0 (3470509687 0 0) 0x1c1c3c0 con 0x1df2840
|
|
-78> 2013-08-15 20:50:32.457695 7f9defdc3700 10 monclient: handle_monmap mon_map v1
|
|
-77> 2013-08-15 20:50:32.457700 7f9defdc3700 10 monclient: got monmap 1, mon.ceph-dmon1 is now rank 1
|
|
-76> 2013-08-15 20:50:32.457703 7f9defdc3700 10 monclient: dump:
|
|
epoch 1
|
|
fsid 6d277a5d-0fb1-4d8e-8e78-732b672464e8
|
|
last_changed 0.000000
|
|
created 0.000000
|
|
0: 9.4.196.1:6789/0 mon.zc2store
|
|
1: 9.4.196.4:6789/0 mon.ceph-dmon1
|
|
2: 0.0.0.0:0/2 mon.ceph-dmon2
|
|
|
|
-75> 2013-08-15 20:50:32.457717 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1bdc700 con 0x1df2840
|
|
-74> 2013-08-15 20:50:32.457725 7f9defdc3700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-73> 2013-08-15 20:50:32.457823 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 9 ==== auth_reply(proto 2 0 Success) v1 ==== 194+0+0 (3908324304 0 0) 0x1dfc600 con 0x1df2840
|
|
-72> 2013-08-15 20:50:32.457862 7f9defdc3700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2013-08-15 20:50:02.457861)
|
|
-71> 2013-08-15 20:50:32.457888 7f9dfd3fb7c0 10 monclient: wait_auth_rotating done
|
|
-70> 2013-08-15 20:50:32.458500 7f9dfd3fb7c0 10 monclient: get_version osdmap
|
|
-69> 2013-08-15 20:50:32.458508 7f9dfd3fb7c0 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-68> 2013-08-15 20:50:32.458516 7f9dfd3fb7c0 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x1c029c0 con 0x1df2840
|
|
-67> 2013-08-15 20:50:32.458918 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 10 ==== mon_check_map_ack(handle=1 version=91) v2 ==== 24+0+0 (667974206 0 0) 0x1bdc540 con 0x1df2840
|
|
-66> 2013-08-15 20:50:32.458974 7f9debdbb700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-65> 2013-08-15 20:50:32.459003 7f9debdbb700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- osd_boot(osd.8 booted 0 v63) v3 -- ?+0 0x1e20000 con 0x1df2840
|
|
-64> 2013-08-15 20:50:33.755396 7f9de6aa7700 1 -- 9.4.196.3:6832/9433 >> :/0 pipe(0x1df7180 sd=33 :6832 s=0 pgs=0 cs=0 l=0).accept sd=33 9.4.196.1:44553/0
|
|
-63> 2013-08-15 20:50:33.755554 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.0 9.4.196.1:6804/5992 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x1e01200 con 0x1df3ce0
|
|
-62> 2013-08-15 20:50:33.755593 7f9deedc1700 3 osd.8 63 handle_osd_map epochs [92,93], i have 63, src has [1,93]
|
|
-61> 2013-08-15 20:50:33.755601 7f9deedc1700 10 monclient: renew_subs
|
|
-60> 2013-08-15 20:50:33.755604 7f9deedc1700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-59> 2013-08-15 20:50:33.755627 7f9deedc1700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x1bdc8c0 con 0x1df2840
|
|
-58> 2013-08-15 20:50:33.755646 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.0 9.4.196.1:6804/5992 2 ==== pg_query(2.b,3.a epoch 93) v2 ==== 262+0+0 (3195657227 0 0) 0x1c1c1e0 con 0x1df3ce0
|
|
-57> 2013-08-15 20:50:33.755668 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.755495, event: header_read, request: pg_query(2.b,3.a epoch 93) v2
|
|
-56> 2013-08-15 20:50:33.755696 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.755496, event: throttled, request: pg_query(2.b,3.a epoch 93) v2
|
|
-55> 2013-08-15 20:50:33.755705 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.755525, event: all_read, request: pg_query(2.b,3.a epoch 93) v2
|
|
-54> 2013-08-15 20:50:33.755710 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.755660, event: dispatched, request: pg_query(2.b,3.a epoch 93) v2
|
|
-53> 2013-08-15 20:50:33.755714 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.755714, event: waiting_for_osdmap, request: pg_query(2.b,3.a epoch 93) v2
|
|
-52> 2013-08-15 20:50:33.755723 7f9deedc1700 10 monclient: renew_subs
|
|
-51> 2013-08-15 20:50:33.755728 7f9deedc1700 10 monclient: _send_mon_message to mon.ceph-dmon1 at 9.4.196.4:6789/0
|
|
-50> 2013-08-15 20:50:33.755734 7f9deedc1700 1 -- 9.4.196.3:6805/9433 --> 9.4.196.4:6789/0 -- mon_subscribe({monmap=2+,osd_pg_creates=0,osdmap=64}) v2 -- ?+0 0x1bdcc40 con 0x1df2840
|
|
-49> 2013-08-15 20:50:33.755745 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.755745, event: wait for new map, request: pg_query(2.b,3.a epoch 93) v2
|
|
-48> 2013-08-15 20:50:33.757363 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 11 ==== osd_map(64..93 src has 1..93) v3 ==== 30396+0+0 (1597789786 0 0) 0x1e018c0 con 0x1df2840
|
|
-47> 2013-08-15 20:50:33.757385 7f9defdc3700 3 osd.8 63 handle_osd_map epochs [64,93], i have 63, src has [1,93]
|
|
-46> 2013-08-15 20:50:33.761637 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.0 9.4.196.1:6804/5992 3 ==== pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2 ==== 762+0+0 (3511946927 0 0) 0x1c1c780 con 0x1df3ce0
|
|
-45> 2013-08-15 20:50:33.762510 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6804/4365 -- pipe dne
|
|
-44> 2013-08-15 20:50:33.762524 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6806/4361 -- pipe dne
|
|
-43> 2013-08-15 20:50:33.762540 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6805/4369 -- pipe dne
|
|
-42> 2013-08-15 20:50:33.762566 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6819/4359 -- pipe dne
|
|
-41> 2013-08-15 20:50:33.762584 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6807/4371 -- pipe dne
|
|
-40> 2013-08-15 20:50:33.762592 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6814/4363 -- pipe dne
|
|
-39> 2013-08-15 20:50:33.762626 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6813/4125 -- pipe dne
|
|
-38> 2013-08-15 20:50:33.762633 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6807/4117 -- pipe dne
|
|
-37> 2013-08-15 20:50:33.762651 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6822/8361 -- pipe dne
|
|
-36> 2013-08-15 20:50:33.762660 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6807/8356 -- pipe dne
|
|
-35> 2013-08-15 20:50:33.762675 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6826/8751 -- pipe dne
|
|
-34> 2013-08-15 20:50:33.762682 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6813/8749 -- pipe dne
|
|
-33> 2013-08-15 20:50:33.762698 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6833/9115 -- pipe dne
|
|
-32> 2013-08-15 20:50:33.762705 7f9defdc3700 1 -- 9.4.196.3:6825/9433 mark_down 9.4.196.3:6809/9079 -- pipe dne
|
|
-31> 2013-08-15 20:50:33.762719 7f9defdc3700 1 osd.8 93 state: booting -> active
|
|
-30> 2013-08-15 20:50:33.763047 7f9defdc3700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.763046, event: started, request: pg_query(2.b,3.a epoch 93) v2
|
|
-29> 2013-08-15 20:50:33.763081 7f9defdc3700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 0, time: 2013-08-15 20:50:33.763081, event: done, request: pg_query(2.b,3.a epoch 93) v2
|
|
-28> 2013-08-15 20:50:33.763117 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 12 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1bdd180 con 0x1df2840
|
|
-27> 2013-08-15 20:50:33.763123 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.761560, event: header_read, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-26> 2013-08-15 20:50:33.763137 7f9defdc3700 10 monclient: handle_subscribe_ack sent 2013-08-15 20:50:33.755602 renew after 2013-08-15 20:53:03.755602
|
|
-25> 2013-08-15 20:50:33.763157 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.761561, event: throttled, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-24> 2013-08-15 20:50:33.763163 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 13 ==== osd_map(64..93 src has 1..93) v3 ==== 30396+0+0 (1597789786 0 0) 0x1e01d40 con 0x1df2840
|
|
-23> 2013-08-15 20:50:33.763168 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.761618, event: all_read, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-22> 2013-08-15 20:50:33.763178 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.761662, event: dispatched, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-21> 2013-08-15 20:50:33.763187 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.763187, event: waiting_for_osdmap, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-20> 2013-08-15 20:50:33.763199 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.763199, event: started, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-19> 2013-08-15 20:50:33.763242 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 1, time: 2013-08-15 20:50:33.763241, event: done, request: pg_query(0.d,0.2d,1.c,1.2c,2.2b,3.2a epoch 93) v2
|
|
-18> 2013-08-15 20:50:33.763265 7f9defdc3700 3 osd.8 93 handle_osd_map epochs [64,93], i have 93, src has [1,93]
|
|
-17> 2013-08-15 20:50:33.763263 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.1 9.4.196.1:6801/5083 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x1efe000 con 0x1df3340
|
|
-16> 2013-08-15 20:50:33.763281 7f9deedc1700 3 osd.8 93 handle_osd_map epochs [92,93], i have 93, src has [1,93]
|
|
-15> 2013-08-15 20:50:33.763280 7f9defdc3700 1 -- 9.4.196.3:6805/9433 <== mon.1 9.4.196.4:6789/0 14 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (4014499548 0 0) 0x1bdcfc0 con 0x1df2840
|
|
-14> 2013-08-15 20:50:33.763290 7f9defdc3700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
|
|
-13> 2013-08-15 20:50:33.763292 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.1 9.4.196.1:6801/5083 2 ==== pg_notify(3.18d(12) epoch 93) v4 ==== 1060+0+0 (2760967449 0 0) 0x1bdd340 con 0x1df3340
|
|
-12> 2013-08-15 20:50:33.763293 7f9df5dcf700 -1 journal FileJournal::write_bl : write_fd failed: (28) No space left on device
|
|
-11> 2013-08-15 20:50:33.763305 7f9df5dcf700 -1 journal FileJournal::do_write: write_bl(pos=3671330816) failed
|
|
-10> 2013-08-15 20:50:33.763304 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.762531, event: header_read, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-9> 2013-08-15 20:50:33.763313 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.762532, event: throttled, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-8> 2013-08-15 20:50:33.763319 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.762598, event: all_read, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-7> 2013-08-15 20:50:33.763327 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.763302, event: dispatched, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-6> 2013-08-15 20:50:33.763334 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.763334, event: waiting_for_osdmap, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-5> 2013-08-15 20:50:33.763346 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.763346, event: started, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-4> 2013-08-15 20:50:33.763373 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 2, time: 2013-08-15 20:50:33.763373, event: done, request: pg_notify(3.18d(12) epoch 93) v4
|
|
-3> 2013-08-15 20:50:33.763776 7f9deadb9700 1 osd.8 pg_epoch: 93 pg[3.12b( v 63'16 (0'0,63'16] local-les=62 n=4 ec=56 les/c 62/62 93/93/56) [3,8] r=1 lpr=93 pi=56-92/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-2> 2013-08-15 20:50:33.763870 7f9deadb9700 1 -- 9.4.196.3:6825/9433 --> 9.4.196.1:6810/7799 -- pg_notify(3.12b(17) epoch 93) v4 -- ?+0 0x1bdc380 con 0x1df34a0
|
|
-1> 2013-08-15 20:50:33.764069 7f9de68a5700 1 -- 9.4.196.3:6825/9433 >> :/0 pipe(0x1df7400 sd=34 :6825 s=0 pgs=0 cs=0 l=0).accept sd=34 9.4.196.3:44267/0
|
|
0> 2013-08-15 20:50:33.764339 7f9df5dcf700 -1 os/FileJournal.cc: In function 'void FileJournal::do_write(ceph::bufferlist&)' thread 7f9df5dcf700 time 2013-08-15 20:50:33.763311
|
|
os/FileJournal.cc: 1048: FAILED assert(0)
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
2: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
3: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
4: (()+0x7f8e) [0x7f9dfcdd7f8e]
|
|
5: (clone()+0x6d) [0x7f9dfb089e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|
|
2013-08-15 20:50:33.767786 7f9df5dcf700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f9df5dcf700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f9dfcddfbd0]
|
|
3: (gsignal()+0x37) [0x7f9dfafc7037]
|
|
4: (abort()+0x148) [0x7f9dfafca698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9dfb8d3e8d]
|
|
6: (()+0x5ef76) [0x7f9dfb8d1f76]
|
|
7: (()+0x5efa3) [0x7f9dfb8d1fa3]
|
|
8: (()+0x5f1de) [0x7f9dfb8d21de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f9dfcdd7f8e]
|
|
14: (clone()+0x6d) [0x7f9dfb089e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- begin dump of recent events ---
|
|
-15> 2013-08-15 20:50:33.764692 7f9deadb9700 1 osd.8 pg_epoch: 93 pg[3.125( v 63'23 (0'0,63'23] local-les=62 n=7 ec=56 les/c 62/62 93/93/56) [1,8] r=1 lpr=93 pi=56-92/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-14> 2013-08-15 20:50:33.765534 7f9deadb9700 1 osd.8 pg_epoch: 93 pg[3.66( v 63'15 (0'0,63'15] local-les=62 n=5 ec=56 les/c 62/62 93/93/84) [8,0]/[0,3] r=-1 lpr=93 pi=61-92/17 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-13> 2013-08-15 20:50:33.766156 7f9deadb9700 1 osd.8 pg_epoch: 93 pg[3.11c( v 63'20 (0'0,63'20] local-les=62 n=4 ec=56 les/c 62/62 93/93/56) [3,8] r=1 lpr=93 pi=56-92/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
-12> 2013-08-15 20:50:33.766725 7f9deadb9700 1 osd.8 pg_epoch: 93 pg[3.11d( v 63'11 (0'0,63'11] local-les=62 n=1 ec=56 les/c 62/62 93/93/84) [8,0]/[0,1] r=-1 lpr=93 pi=61-92/17 lcod 0'0 remapped NOTIFY] state<Start>: transitioning to Stray
|
|
-11> 2013-08-15 20:50:33.767119 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.4 9.4.196.1:6813/8739 1 ==== osd_map(92..93 src has 1..93) v3 ==== 1879+0+0 (2962968938 0 0) 0x1efe000 con 0x1df38c0
|
|
-10> 2013-08-15 20:50:33.767144 7f9deedc1700 3 osd.8 93 handle_osd_map epochs [92,93], i have 93, src has [1,93]
|
|
-9> 2013-08-15 20:50:33.767197 7f9deedc1700 1 -- 9.4.196.3:6825/9433 <== osd.4 9.4.196.1:6813/8739 2 ==== pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2 ==== 637+0+0 (200914757 0 0) 0x1c1c960 con 0x1df38c0
|
|
-8> 2013-08-15 20:50:33.767220 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767134, event: header_read, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-7> 2013-08-15 20:50:33.767233 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767135, event: throttled, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-6> 2013-08-15 20:50:33.767244 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767184, event: all_read, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-5> 2013-08-15 20:50:33.767254 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767215, event: dispatched, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-4> 2013-08-15 20:50:33.767264 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767264, event: waiting_for_osdmap, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-3> 2013-08-15 20:50:33.767277 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767277, event: started, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-2> 2013-08-15 20:50:33.767320 7f9deedc1700 5 --OSD::tracker-- reqid: unknown.0.0:0, seq: 3, time: 2013-08-15 20:50:33.767320, event: done, request: pg_query(0.9,1.8,2.7,3.6,3.178 epoch 93) v2
|
|
-1> 2013-08-15 20:50:33.767637 7f9deadb9700 1 osd.8 pg_epoch: 93 pg[3.58( v 63'21 (0'0,63'21] local-les=62 n=5 ec=56 les/c 62/62 93/93/56) [3,8] r=1 lpr=93 pi=56-92/17 lcod 0'0 inactive NOTIFY] state<Start>: transitioning to Stray
|
|
0> 2013-08-15 20:50:33.767786 7f9df5dcf700 -1 *** Caught signal (Aborted) **
|
|
in thread 7f9df5dcf700
|
|
|
|
ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
|
|
1: /usr/bin/ceph-osd() [0x842bb0]
|
|
2: (()+0xfbd0) [0x7f9dfcddfbd0]
|
|
3: (gsignal()+0x37) [0x7f9dfafc7037]
|
|
4: (abort()+0x148) [0x7f9dfafca698]
|
|
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f9dfb8d3e8d]
|
|
6: (()+0x5ef76) [0x7f9dfb8d1f76]
|
|
7: (()+0x5efa3) [0x7f9dfb8d1fa3]
|
|
8: (()+0x5f1de) [0x7f9dfb8d21de]
|
|
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x43d) [0x8ef5bd]
|
|
10: (FileJournal::do_write(ceph::buffer::list&)+0xb19) [0x7ca5a9]
|
|
11: (FileJournal::write_thread_entry()+0xb70) [0x7ccf70]
|
|
12: (FileJournal::Writer::entry()+0xd) [0x6f88ed]
|
|
13: (()+0x7f8e) [0x7f9dfcdd7f8e]
|
|
14: (clone()+0x6d) [0x7f9dfb089e1d]
|
|
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
|
|
|
|
--- logging levels ---
|
|
0/ 5 none
|
|
0/ 1 lockdep
|
|
0/ 1 context
|
|
1/ 1 crush
|
|
1/ 5 mds
|
|
1/ 5 mds_balancer
|
|
1/ 5 mds_locker
|
|
1/ 5 mds_log
|
|
1/ 5 mds_log_expire
|
|
1/ 5 mds_migrator
|
|
0/ 1 buffer
|
|
0/ 1 timer
|
|
0/ 1 filer
|
|
0/ 1 striper
|
|
0/ 1 objecter
|
|
0/ 5 rados
|
|
0/ 5 rbd
|
|
0/ 5 journaler
|
|
0/ 5 objectcacher
|
|
0/ 5 client
|
|
0/ 5 osd
|
|
0/ 5 optracker
|
|
0/ 5 objclass
|
|
1/ 3 filestore
|
|
1/ 3 journal
|
|
0/ 5 ms
|
|
1/ 5 mon
|
|
0/10 monc
|
|
0/ 5 paxos
|
|
0/ 5 tp
|
|
1/ 5 auth
|
|
1/ 5 crypto
|
|
1/ 1 finisher
|
|
1/ 5 heartbeatmap
|
|
1/ 5 perfcounter
|
|
1/ 5 rgw
|
|
1/ 5 hadoop
|
|
1/ 5 javaclient
|
|
1/ 5 asok
|
|
1/ 1 throttle
|
|
-2/-2 (syslog threshold)
|
|
-1/-1 (stderr threshold)
|
|
max_recent 10000
|
|
max_new 1000
|
|
log_file /var/log/ceph/ceph-osd.8.log
|
|
--- end dump of recent events ---
|