|
2016-11-25 12:13:10.996632 7fc2bcef9700 0 log_channel(cluster) log [WRN] : 10676 slow requests, 5 included below; oldest blocked for > 21763.059689 secs
|
|
2016-11-25 12:13:10.996638 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 794.261923 seconds old, received at 2016-11-25 11:59:56.733672: client_request(client.3857990:27862456 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:56.733794) currently failed to rdlock, waiting
|
|
2016-11-25 12:13:10.996642 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 793.676423 seconds old, received at 2016-11-25 11:59:57.319172: client_request(client.3939857:2951921 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:57.318619) currently failed to rdlock, waiting
|
|
2016-11-25 12:13:10.996645 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 793.380468 seconds old, received at 2016-11-25 11:59:57.615128: client_request(client.3857990:27862656 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:57.615780) currently failed to rdlock, waiting
|
|
2016-11-25 12:13:10.996648 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 793.042619 seconds old, received at 2016-11-25 11:59:57.952977: client_request(client.3939857:2951969 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:57.951624) currently failed to rdlock, waiting
|
|
2016-11-25 12:13:10.996650 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 792.293800 seconds old, received at 2016-11-25 11:59:58.701795: client_request(client.3857990:27862845 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:58.702762) currently failed to rdlock, waiting
|
|
2016-11-25 12:13:13.366923 7fc2c1215700 1 mds.v6.us-imm-cephmon2 asok_command: dump_blocked_ops (starting...)
|
|
2016-11-25 12:13:35.211847 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:35.211859 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:13:37.364531 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:39.211903 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:39.211920 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:13:42.364603 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:43.211958 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:43.211968 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:13:46.457885 7fc2b84e7700 0 -- [2a02:4780:bad:c0de::17]:6803/18126 >> [2a02:4780:bad:c0de::2]:0/3523720559 pipe(0x7fc2da1a6800 sd=43 :6803 s=2 pgs=284 cs=1 l=0 c=0x7fc2d0d92180).fault with nothing to send, going to standby
|
|
2016-11-25 12:13:46.458414 7fc2b86e9700 0 -- [2a02:4780:bad:c0de::17]:6803/18126 >> [2a02:4780:bad:c0de::3]:0/4069917085 pipe(0x7fc2da1a5400 sd=42 :6803 s=2 pgs=2656 cs=1 l=0 c=0x7fc2d0d92000).fault with nothing to send, going to standby
|
|
2016-11-25 12:13:46.458441 7fc2b82e5700 0 -- [2a02:4780:bad:c0de::17]:6803/18126 >> [2a02:4780:bad:c0de::10]:0/1406418688 pipe(0x7fc2e03e0000 sd=44 :6803 s=2 pgs=22 cs=1 l=0 c=0x7fc2d0dfa000).fault with nothing to send, going to standby
|
|
2016-11-25 12:13:46.459154 7fc2b80e3700 0 -- [2a02:4780:bad:c0de::17]:6803/18126 >> [2a02:4780:bad:f00d::3]:0/2232148986 pipe(0x7fc2e03e1400 sd=45 :6803 s=2 pgs=9 cs=1 l=0 c=0x7fc2d0dfa180).fault with nothing to send, going to standby
|
|
2016-11-25 12:13:46.459171 7fc2b88eb700 0 -- [2a02:4780:bad:c0de::17]:6803/18126 >> [2a02:4780:bad:f00d::4]:0/319192201 pipe(0x7fc2da1a4000 sd=41 :6803 s=2 pgs=6 cs=1 l=0 c=0x7fc2d0f2aa80).fault with nothing to send, going to standby
|
|
2016-11-25 12:13:46.459223 7fc2b7ee1700 0 -- [2a02:4780:bad:c0de::17]:6803/18126 >> [2a02:4780:bad:f00d::2]:0/3199769431 pipe(0x7fc2e03e2800 sd=46 :6803 s=2 pgs=2 cs=1 l=0 c=0x7fc2d0dfa300).fault with nothing to send, going to standby
|
|
2016-11-25 12:13:47.212022 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:47.212032 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:13:47.364665 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:51.212075 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:51.212082 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:13:52.364726 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:55.212138 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:55.212148 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:13:57.364795 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:59.212194 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:13:59.212200 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:02.364870 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:03.212254 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:03.212264 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:07.212313 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:07.212322 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:07.364945 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:11.212375 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:11.212384 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:12.365016 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:15.221902 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:15.221924 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:17.365090 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:19.221960 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:19.221968 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:22.365160 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:23.222019 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:23.222026 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:27.222079 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:27.222086 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:27.365230 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:31.222143 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:31.222150 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:32.365296 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:35.222200 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:35.222208 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:37.365818 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:39.230402 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:39.230412 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:42.366312 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:43.230462 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:43.230469 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:47.230524 7fc2bc6f8700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:47.230531 7fc2bc6f8700 1 mds.beacon.v6.us-imm-cephmon2 _send skipping beacon, heartbeat map not healthy
|
|
2016-11-25 12:14:47.366379 7fc2c1a16700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:48.543766 7fc2bcef9700 0 log_channel(cluster) log [WRN] : 10689 slow requests, 5 included below; oldest blocked for > 21771.760103 secs
|
|
2016-11-25 12:14:48.543778 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 802.376836 seconds old, received at 2016-11-25 11:59:57.319172: client_request(client.3939857:2951921 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:57.318619) currently failed to rdlock, waiting
|
|
2016-11-25 12:14:48.543781 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 802.080881 seconds old, received at 2016-11-25 11:59:57.615128: client_request(client.3857990:27862656 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:57.615780) currently failed to rdlock, waiting
|
|
2016-11-25 12:14:48.543783 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 801.743032 seconds old, received at 2016-11-25 11:59:57.952977: client_request(client.3939857:2951969 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:57.951624) currently failed to rdlock, waiting
|
|
2016-11-25 12:14:48.543788 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 800.994213 seconds old, received at 2016-11-25 11:59:58.701795: client_request(client.3857990:27862845 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:58.702762) currently failed to rdlock, waiting
|
|
2016-11-25 12:14:48.543793 7fc2bcef9700 0 log_channel(cluster) log [WRN] : slow request 800.805076 seconds old, received at 2016-11-25 11:59:58.890933: client_request(client.3935504:3671877 getattr pAsLsXsFs #1000bc4fa29 2016-11-25 11:59:58.892010) currently failed to rdlock, waiting
|
|
2016-11-25 12:14:50.160040 7fc2bcef9700 1 heartbeat_map reset_timeout 'MDSRank' had timed out after 15
|
|
2016-11-25 12:14:50.160053 7fc2baef5700 -1 mds.0.journaler(rw) _finish_write_head got (108) Cannot send after transport endpoint shutdown
|
|
2016-11-25 12:14:50.160071 7fc2baef5700 -1 mds.0.journaler(rw) handle_write_error (108) Cannot send after transport endpoint shutdown
|
|
2016-11-25 12:14:50.196760 7fc2baef5700 -1 MDSIOContextBase: blacklisted! Restarting...
|
|
2016-11-25 12:14:50.196766 7fc2baef5700 1 mds.v6.us-imm-cephmon2 respawn
|
|
2016-11-25 12:14:50.196770 7fc2baef5700 1 mds.v6.us-imm-cephmon2 e: '/usr/bin/ceph-mds'
|
|
2016-11-25 12:14:50.196772 7fc2baef5700 1 mds.v6.us-imm-cephmon2 0: '/usr/bin/ceph-mds'
|
|
2016-11-25 12:14:50.196774 7fc2baef5700 1 mds.v6.us-imm-cephmon2 1: '-f'
|
|
2016-11-25 12:14:50.196775 7fc2baef5700 1 mds.v6.us-imm-cephmon2 2: '--cluster'
|
|
2016-11-25 12:14:50.196776 7fc2baef5700 1 mds.v6.us-imm-cephmon2 3: 'ceph'
|
|
2016-11-25 12:14:50.196777 7fc2baef5700 1 mds.v6.us-imm-cephmon2 4: '--id'
|
|
2016-11-25 12:14:50.196778 7fc2baef5700 1 mds.v6.us-imm-cephmon2 5: 'v6.us-imm-cephmon2'
|
|
2016-11-25 12:14:50.196779 7fc2baef5700 1 mds.v6.us-imm-cephmon2 6: '--setuser'
|
|
2016-11-25 12:14:50.196780 7fc2baef5700 1 mds.v6.us-imm-cephmon2 7: 'ceph'
|
|
2016-11-25 12:14:50.196781 7fc2baef5700 1 mds.v6.us-imm-cephmon2 8: '--setgroup'
|
|
2016-11-25 12:14:50.196783 7fc2baef5700 1 mds.v6.us-imm-cephmon2 9: 'ceph'
|
|
2016-11-25 12:14:50.196803 7fc2baef5700 1 mds.v6.us-imm-cephmon2 exe_path /usr/bin/ceph-mds
|
|
2016-11-25 12:14:52.856985 7ff7048f3180 0 ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b), process ceph-mds, pid 18126
|
|
2016-11-25 12:14:52.857357 7ff7048f3180 0 pidfile_write: ignore empty --pid-file
|
|
2016-11-25 12:14:53.929412 7ff6fe8ce700 1 mds.v6.us-imm-cephmon2 handle_mds_map standby
|
