Project

General

Profile

Bug #19299 » ceph-osd.log.txt

Ben Meekhof, 03/17/2017 07:45 PM

 
2017-03-17 15:18:35.252775 7fc390b16700 1 RefCountedObject::get 0x7fc3b3cc4000 2 -> 3
2017-03-17 15:18:35.252777 7fc390b16700 1 RefCountedObject::get 0x7fc3b3cc4000 3 -> 4
2017-03-17 15:18:35.252780 7fc390b16700 1 RefCountedObject::put 0x7fc3b3cc4000 4 -> 3
2017-03-17 15:18:35.252781 7fc390b16700 1 RefCountedObject::put 0x7fc3b3cc4000 3 -> 2
2017-03-17 15:18:35.252783 7fc390b16700 20 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1).process prev state is STATE_OPEN_MESSAGE_READ_FRONT
2017-03-17 15:18:35.252788 7fc390b16700 25 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1).read_until len is 1 state_offset is 0
2017-03-17 15:18:35.252803 7fc390b16700 25 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1).read_until read_bulk recv_end is 0 left is 1 got 0
2017-03-17 15:18:35.252809 7fc390b16700 25 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1).read_until need len 1 remaining 1 bytes
2017-03-17 15:18:35.252815 7fc390b16700 20 Event(0x7fc3a05cc680 nevent=5000 time_id=2).dispatch_event_external 0x7fc3a543d600 pending 1
2017-03-17 15:18:35.252817 7fc390b16700 20 Event(0x7fc3a05cc680 nevent=5000 time_id=2).process_events event_wq process is 110 mask is 1
2017-03-17 15:18:35.252820 7fc390b16700 20 Event(0x7fc3a05cc680 nevent=5000 time_id=2).process_events do 0x7fc3a543d600
2017-03-17 15:18:35.252821 7fc390b16700 10 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1).handle_write
2017-03-17 15:18:35.252828 7fc390b16700 10 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1).handle_write try send msg ack, acked 1 messages
2017-03-17 15:18:35.252853 7fc390b16700 10 -- 204.39.195.26:6862/1531376 >> 141.211.169.13:6789/0 conn(0x7fc3a1203000 :-1 s=STATE_OPEN pgs=235722 cs=1 l=1)._try_send sent bytes 9 remaining bytes 0
2017-03-17 15:18:35.252860 7fc390b16700 30 stack operator() calling event process
2017-03-17 15:18:35.252862 7fc390b16700 10 Event(0x7fc3a05cc680 nevent=5000 time_id=2).process_events wait second 30 usec 0
2017-03-17 15:18:35.280407 7fc374687700 30 osd.455 234319 heartbeat_entry woke up
2017-03-17 15:18:35.280424 7fc374687700 30 osd.455 234319 heartbeat
2017-03-17 15:18:35.280485 7fc374687700 30 osd.455 234319 heartbeat: daily_loadavg 67.6974
2017-03-17 15:18:35.280501 7fc374687700 30 osd.455 234319 heartbeat checking stats
2017-03-17 15:18:35.280528 7fc374687700 20 osd.455 234319 update_osd_stat osd_stat(130 GB used, 7319 GB avail, 7450 GB total, peers [] op hist [])
2017-03-17 15:18:35.280533 7fc374687700 5 osd.455 234319 heartbeat: osd_stat(130 GB used, 7319 GB avail, 7450 GB total, peers [] op hist [])
2017-03-17 15:18:35.280537 7fc374687700 30 osd.455 234319 heartbeat lonely?
2017-03-17 15:18:35.280539 7fc374687700 30 osd.455 234319 heartbeat done
2017-03-17 15:18:35.280541 7fc374687700 30 osd.455 234319 heartbeat_entry sleeping for 4.1
2017-03-17 15:18:35.339558 7fc389db8700 20 filestore(/var/lib/ceph/osd/ceph-455) sync_entry woke after 5.003081
2017-03-17 15:18:35.339574 7fc389db8700 10 FileStore::op_tp pause
2017-03-17 15:18:35.339577 7fc389db8700 15 FileStore::op_tp paused
2017-03-17 15:18:35.339579 7fc389db8700 10 journal commit_start max_applied_seq 834321, open_ops 0
2017-03-17 15:18:35.339582 7fc389db8700 10 journal commit_start blocked, all open_ops have completed
2017-03-17 15:18:35.339584 7fc389db8700 10 journal commit_start nothing to do
2017-03-17 15:18:35.339586 7fc389db8700 10 journal commit_start
2017-03-17 15:18:35.339588 7fc389db8700 10 FileStore::op_tp unpause
2017-03-17 15:18:35.339594 7fc389db8700 20 filestore(/var/lib/ceph/osd/ceph-455) sync_entry waiting for max_interval 5.000000
2017-03-17 15:18:35.340411 7fc3875b3700 20 tp_fstore_op worker waiting
2017-03-17 15:18:35.340417 7fc3875b3700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:35.350407 7fc387db4700 20 tp_fstore_op worker waiting
2017-03-17 15:18:35.350417 7fc387db4700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:36.092424 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread awake
2017-03-17 15:18:36.092447 7fc38c3d8700 10 timer(0x7fc3a0970190).timer_thread executing 0x7fc3a657c3c0
2017-03-17 15:18:36.092452 7fc38c3d8700 10 osd.455 234319 tick_without_osd_lock
2017-03-17 15:18:36.092465 7fc38c3d8700 10 timer(0x7fc3a0970190).add_event_at 2017-03-17 15:18:37.092464 -> 0x7fc3a543d5e0
2017-03-17 15:18:36.092475 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread going to sleep
2017-03-17 15:18:36.279530 7fc37b695700 20 tp_osd worker waiting
2017-03-17 15:18:36.279556 7fc37b695700 20 heartbeat_map reset_timeout 'tp_osd thread tp_osd' grace 60 suicide 0
2017-03-17 15:18:36.337432 7fc37ae94700 20 tp_osd worker waiting
2017-03-17 15:18:36.337458 7fc37ae94700 20 heartbeat_map reset_timeout 'tp_osd thread tp_osd' grace 60 suicide 0
2017-03-17 15:18:36.671435 7fc379e92700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.671459 7fc379e92700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.697602 7fc376e8c700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.697623 7fc376e8c700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.718410 7fc374e88700 20 tp_osd_cmd worker waiting
2017-03-17 15:18:36.718431 7fc374e88700 20 heartbeat_map reset_timeout 'tp_osd_cmd thread tp_osd_cmd' grace 60 suicide 0
2017-03-17 15:18:36.735425 7fc37768d700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.735442 7fc37768d700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.740415 7fc375e8a700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.740436 7fc375e8a700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.765418 7fc375689700 20 tp_osd_disk worker waiting
2017-03-17 15:18:36.765431 7fc375689700 20 heartbeat_map reset_timeout 'tp_osd_disk thread tp_osd_disk' grace 60 suicide 0
2017-03-17 15:18:36.848512 7fc377e8e700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.848529 7fc377e8e700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.864444 7fc37868f700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.864459 7fc37868f700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.868475 7fc37668b700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.868487 7fc37668b700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.876421 7fc379691700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.876433 7fc379691700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.927415 7fc378e90700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.927433 7fc378e90700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:36.928419 7fc37a693700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:36.928428 7fc37a693700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:37.093611 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread awake
2017-03-17 15:18:37.093639 7fc38c3d8700 10 timer(0x7fc3a0970190).timer_thread executing 0x7fc3a543d5e0
2017-03-17 15:18:37.093643 7fc38c3d8700 10 osd.455 234319 tick_without_osd_lock
2017-03-17 15:18:37.093657 7fc38c3d8700 10 timer(0x7fc3a0970190).add_event_at 2017-03-17 15:18:38.093657 -> 0x7fc3a657c3c0
2017-03-17 15:18:37.093668 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread going to sleep
2017-03-17 15:18:37.341410 7fc3875b3700 20 tp_fstore_op worker waiting
2017-03-17 15:18:37.341427 7fc3875b3700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:37.350482 7fc387db4700 20 tp_fstore_op worker waiting
2017-03-17 15:18:37.350494 7fc387db4700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:37.442416 7fc38d3da700 10 osd.455.objecter tick
2017-03-17 15:18:37.480421 7fc382ea4700 20 osd.455 234319 got_full_map 234413, nothing requested
2017-03-17 15:18:37.482301 7fc382ea4700 10 osd.455 234319 handle_osd_map got inc map for epoch 234414
2017-03-17 15:18:38.094412 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread awake
2017-03-17 15:18:38.094439 7fc38c3d8700 10 timer(0x7fc3a0970190).timer_thread executing 0x7fc3a657c3c0
2017-03-17 15:18:38.094445 7fc38c3d8700 10 osd.455 234319 tick_without_osd_lock
2017-03-17 15:18:38.094458 7fc38c3d8700 10 timer(0x7fc3a0970190).add_event_at 2017-03-17 15:18:39.094457 -> 0x7fc3a543d5e0
2017-03-17 15:18:38.094467 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread going to sleep
2017-03-17 15:18:38.283471 7fc37b695700 20 tp_osd worker waiting
2017-03-17 15:18:38.283492 7fc37b695700 20 heartbeat_map reset_timeout 'tp_osd thread tp_osd' grace 60 suicide 0
2017-03-17 15:18:38.338421 7fc37ae94700 20 tp_osd worker waiting
2017-03-17 15:18:38.338439 7fc37ae94700 20 heartbeat_map reset_timeout 'tp_osd thread tp_osd' grace 60 suicide 0
2017-03-17 15:18:38.672531 7fc379e92700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.672559 7fc379e92700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.700418 7fc376e8c700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.700438 7fc376e8c700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.719477 7fc374e88700 20 tp_osd_cmd worker waiting
2017-03-17 15:18:38.719500 7fc374e88700 20 heartbeat_map reset_timeout 'tp_osd_cmd thread tp_osd_cmd' grace 60 suicide 0
2017-03-17 15:18:38.736515 7fc37768d700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.736533 7fc37768d700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.741441 7fc375e8a700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.741465 7fc375e8a700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.766488 7fc375689700 20 tp_osd_disk worker waiting
2017-03-17 15:18:38.766506 7fc375689700 20 heartbeat_map reset_timeout 'tp_osd_disk thread tp_osd_disk' grace 60 suicide 0
2017-03-17 15:18:38.848695 7fc377e8e700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.848713 7fc377e8e700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.865612 7fc37868f700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.865629 7fc37868f700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.869417 7fc37668b700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.869434 7fc37668b700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.877465 7fc379691700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.877477 7fc379691700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.930429 7fc378e90700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.930447 7fc378e90700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:38.936457 7fc37a693700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:38.936475 7fc37a693700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:39.095413 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread awake
2017-03-17 15:18:39.095441 7fc38c3d8700 10 timer(0x7fc3a0970190).timer_thread executing 0x7fc3a543d5e0
2017-03-17 15:18:39.095446 7fc38c3d8700 10 osd.455 234319 tick_without_osd_lock
2017-03-17 15:18:39.095459 7fc38c3d8700 10 timer(0x7fc3a0970190).add_event_at 2017-03-17 15:18:40.095459 -> 0x7fc3a657c3c0
2017-03-17 15:18:39.095469 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread going to sleep
2017-03-17 15:18:39.342407 7fc3875b3700 20 tp_fstore_op worker waiting
2017-03-17 15:18:39.342424 7fc3875b3700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:39.351407 7fc387db4700 20 tp_fstore_op worker waiting
2017-03-17 15:18:39.351422 7fc387db4700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:39.383409 7fc374687700 30 osd.455 234319 heartbeat_entry woke up
2017-03-17 15:18:39.383424 7fc374687700 30 osd.455 234319 heartbeat
2017-03-17 15:18:39.383483 7fc374687700 30 osd.455 234319 heartbeat: daily_loadavg 67.6976
2017-03-17 15:18:39.383498 7fc374687700 30 osd.455 234319 heartbeat checking stats
2017-03-17 15:18:39.383523 7fc374687700 20 osd.455 234319 update_osd_stat osd_stat(130 GB used, 7319 GB avail, 7450 GB total, peers [] op hist [])
2017-03-17 15:18:39.383529 7fc374687700 5 osd.455 234319 heartbeat: osd_stat(130 GB used, 7319 GB avail, 7450 GB total, peers [] op hist [])
2017-03-17 15:18:39.383533 7fc374687700 30 osd.455 234319 heartbeat lonely?
2017-03-17 15:18:39.383534 7fc374687700 30 osd.455 234319 heartbeat done
2017-03-17 15:18:39.383536 7fc374687700 30 osd.455 234319 heartbeat_entry sleeping for 0.5
2017-03-17 15:18:39.818532 7fc38eba1700 20 heartbeat_map is_healthy = healthy, total workers: 16, number of unhealthy: 0
2017-03-17 15:18:39.884425 7fc374687700 30 osd.455 234319 heartbeat_entry woke up
2017-03-17 15:18:39.884444 7fc374687700 30 osd.455 234319 heartbeat
2017-03-17 15:18:39.884506 7fc374687700 30 osd.455 234319 heartbeat: daily_loadavg 67.6977
2017-03-17 15:18:39.884520 7fc374687700 30 osd.455 234319 heartbeat checking stats
2017-03-17 15:18:39.884546 7fc374687700 20 osd.455 234319 update_osd_stat osd_stat(130 GB used, 7319 GB avail, 7450 GB total, peers [] op hist [])
2017-03-17 15:18:39.884551 7fc374687700 5 osd.455 234319 heartbeat: osd_stat(130 GB used, 7319 GB avail, 7450 GB total, peers [] op hist [])
2017-03-17 15:18:39.884555 7fc374687700 30 osd.455 234319 heartbeat lonely?
2017-03-17 15:18:39.884557 7fc374687700 30 osd.455 234319 heartbeat done
2017-03-17 15:18:39.884559 7fc374687700 30 osd.455 234319 heartbeat_entry sleeping for 4.1
2017-03-17 15:18:40.096422 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread awake
2017-03-17 15:18:40.096449 7fc38c3d8700 10 timer(0x7fc3a0970190).timer_thread executing 0x7fc3a657c3c0
2017-03-17 15:18:40.096453 7fc38c3d8700 10 osd.455 234319 tick_without_osd_lock
2017-03-17 15:18:40.096466 7fc38c3d8700 10 timer(0x7fc3a0970190).add_event_at 2017-03-17 15:18:41.096465 -> 0x7fc3a543d5e0
2017-03-17 15:18:40.096476 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread going to sleep
2017-03-17 15:18:40.284420 7fc37b695700 20 tp_osd worker waiting
2017-03-17 15:18:40.284444 7fc37b695700 20 heartbeat_map reset_timeout 'tp_osd thread tp_osd' grace 60 suicide 0
2017-03-17 15:18:40.339432 7fc37ae94700 20 tp_osd worker waiting
2017-03-17 15:18:40.339455 7fc37ae94700 20 heartbeat_map reset_timeout 'tp_osd thread tp_osd' grace 60 suicide 0
2017-03-17 15:18:40.339663 7fc389db8700 20 filestore(/var/lib/ceph/osd/ceph-455) sync_entry woke after 5.000068
2017-03-17 15:18:40.339682 7fc389db8700 10 FileStore::op_tp pause
2017-03-17 15:18:40.339684 7fc389db8700 15 FileStore::op_tp paused
2017-03-17 15:18:40.339686 7fc389db8700 10 journal commit_start max_applied_seq 834321, open_ops 0
2017-03-17 15:18:40.339690 7fc389db8700 10 journal commit_start blocked, all open_ops have completed
2017-03-17 15:18:40.339691 7fc389db8700 10 journal commit_start nothing to do
2017-03-17 15:18:40.339693 7fc389db8700 10 journal commit_start
2017-03-17 15:18:40.339695 7fc389db8700 10 FileStore::op_tp unpause
2017-03-17 15:18:40.339701 7fc389db8700 20 filestore(/var/lib/ceph/osd/ceph-455) sync_entry waiting for max_interval 5.000000
2017-03-17 15:18:40.348421 7fc3875b3700 20 tp_fstore_op worker waiting
2017-03-17 15:18:40.348437 7fc3875b3700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:40.350419 7fc387db4700 20 tp_fstore_op worker waiting
2017-03-17 15:18:40.350425 7fc387db4700 20 heartbeat_map reset_timeout 'tp_fstore_op thread tp_fstore_op' grace 60 suicide 0
2017-03-17 15:18:40.673420 7fc379e92700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.673450 7fc379e92700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.702419 7fc376e8c700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.702447 7fc376e8c700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.720417 7fc374e88700 20 tp_osd_cmd worker waiting
2017-03-17 15:18:40.720451 7fc374e88700 20 heartbeat_map reset_timeout 'tp_osd_cmd thread tp_osd_cmd' grace 60 suicide 0
2017-03-17 15:18:40.737417 7fc37768d700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.737440 7fc37768d700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.742425 7fc375e8a700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.742445 7fc375e8a700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.767416 7fc375689700 20 tp_osd_disk worker waiting
2017-03-17 15:18:40.767432 7fc375689700 20 heartbeat_map reset_timeout 'tp_osd_disk thread tp_osd_disk' grace 60 suicide 0
2017-03-17 15:18:40.849416 7fc377e8e700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.849435 7fc377e8e700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.866411 7fc37868f700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.866431 7fc37868f700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.871416 7fc37668b700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.871432 7fc37668b700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.878505 7fc379691700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.878520 7fc379691700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.931412 7fc378e90700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.931436 7fc378e90700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:40.937470 7fc37a693700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 15 suicide 150
2017-03-17 15:18:40.937485 7fc37a693700 20 heartbeat_map reset_timeout 'tp_osd_tp thread tp_osd_tp' grace 60 suicide 0
2017-03-17 15:18:41.097412 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread awake
2017-03-17 15:18:41.097439 7fc38c3d8700 10 timer(0x7fc3a0970190).timer_thread executing 0x7fc3a543d5e0
2017-03-17 15:18:41.097443 7fc38c3d8700 10 osd.455 234319 tick_without_osd_lock
2017-03-17 15:18:41.097466 7fc38c3d8700 10 timer(0x7fc3a0970190).add_event_at 2017-03-17 15:18:42.097456 -> 0x7fc3a657c3c0
2017-03-17 15:18:41.097476 7fc38c3d8700 20 timer(0x7fc3a0970190).timer_thread going to sleep
2017-03-17 15:18:41.651999 7fc38e3a0700 30 asok(0x7fc3a06ac000) AdminSocket: calling accept
2017-03-17 15:18:41.652024 7fc38e3a0700 30 asok(0x7fc3a06ac000) AdminSocket: finished accept
2017-03-17 15:18:41.652086 7fc38e3a0700 1 do_command '1' '
2017-03-17 15:18:41.652509 7fc38e3a0700 1 do_command '1' 'result is 18440 bytes
2017-03-17 15:18:41.652513 7fc38e3a0700 5 asok(0x7fc3a06ac000) AdminSocket: request '1' '' to 0x7fc3a058c030 returned 18440 bytes
(1-1/5)