Project

General

Profile

Bug #12567

OSD:ENOENT on clone suggests osd bug

Added by science luo over 8 years ago. Updated over 7 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This occurred in my osd.49 when 1 mon down.
Ceph version:0.87
Cluster:5 nodes(4 mons , each node has 11 OSDS)

Here is osd.49 log:
15:33:10.801590) v2 -- ?+0 0x11aaf2a0 con 0xdbad9c0
209> 2015-07-29 15:33:10.810109 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.12 111.111.111.241:6829/3005260 4110 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0x105a5fe0 con 0xdbbf4a0
208> 2015-07-29 15:33:10.810147 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.6 111.111.111.241:6824/4008016 4109 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0x11582780 con 0x103f5700
207> 2015-07-29 15:33:10.810166 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.3 111.111.111.241:6807/5005501 4110 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0x11435e00 con 0x103ca6e0
206> 2015-07-29 15:33:10.810183 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.15 111.111.111.241:6830/3005967 4110 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0x11435a40 con 0xdba47e0
205> 2015-07-29 15:33:10.810199 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.21 111.111.111.241:6812/6009073 4109 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0x126ee3a0 con 0x1052d340
204> 2015-07-29 15:33:10.810306 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.9 111.111.111.241:6822/3007510 4110 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0x16ad3c20 con 0xdba5de0
203> 2015-07-29 15:33:10.811134 7fd61ea8e700 1 - 100.100.100.242:0/11168 <== osd.24 111.111.111.241:6826/4007752 4110 ==== osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.361747) v2 ==== 47+0+0 (1484665921 0 0) 0xfceb680 con 0xdbbffa0
202> 2015-07-29 15:33:10.828820 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.38 100.100.100.244:0/7777 4086 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.824588) v2 ==== 47+0+0 (361394971 0 0) 0x125e2ee0 con 0xdbe19c0
201> 2015-07-29 15:33:10.828851 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.244:0/7777 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.824588) v2 -- ?+0 0x11ab5860 con 0xdbe19c0
200> 2015-07-29 15:33:10.828861 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.38 100.100.100.244:0/7777 4086 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.824588) v2 ==== 47+0+0 (361394971 0 0) 0x17be5c00 con 0xdbfeb00
199> 2015-07-29 15:33:10.828910 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.244:0/7777 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.824588) v2 -- ?+0 0x12658000 con 0xdbfeb00
198> 2015-07-29 15:33:10.896808 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.36 111.111.111.244:0/5867 4109 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.893404) v2 ==== 47+0+0 (83847468 0 0) 0x183650e0 con 0x590a2c0
197> 2015-07-29 15:33:10.896836 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.36 111.111.111.244:0/5867 4109 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.893404) v2 ==== 47+0+0 (83847468 0 0) 0x116f70e0 con 0xdbc6840
196> 2015-07-29 15:33:10.896856 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.244:0/5867 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.893404) v2 -- ?+0 0x125e2ee0 con 0x590a2c0
195> 2015-07-29 15:33:10.896908 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.244:0/5867 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.893404) v2 -- ?+0 0x17be5c00 con 0xdbc6840
194> 2015-07-29 15:33:10.901834 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.16 111.111.111.243:0/8273 4120 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.899167) v2 ==== 47+0+0 (2547709976 0 0) 0x115f3a40 con 0x590adc0
193> 2015-07-29 15:33:10.901859 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.243:0/8273 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.899167) v2 -- ?+0 0x183650e0 con 0x590adc0
192> 2015-07-29 15:33:10.901854 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.16 111.111.111.243:0/8273 4120 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.899167) v2 ==== 47+0+0 (2547709976 0 0) 0x120ef0c0 con 0xdbe0d60
191> 2015-07-29 15:33:10.901894 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.243:0/8273 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.899167) v2 -- ?+0 0x116f70e0 con 0xdbe0d60
190> 2015-07-29 15:33:10.940274 7fd6345f0700 2 - 100.100.100.242:6827/11168 >> 100.100.100.240:6789/0 pipe(0x56aa840 sd=27 :64608 s=2 pgs=550846 cs=1 l=1 c=0xc5c8840).reader couldn't read tag, (0) Success
189> 2015-07-29 15:33:10.940328 7fd6345f0700 2 - 100.100.100.242:6827/11168 >> 100.100.100.240:6789/0 pipe(0x56aa840 sd=27 :64608 s=2 pgs=550846 cs=1 l=1 c=0xc5c8840).fault (0) Success
188> 2015-07-29 15:33:10.940432 7fd621a94700 10 monclient: ms_handle_reset current mon 100.100.100.240:6789/0
-187> 2015-07-29 15:33:10.940449 7fd621a94700 0 monclient: hunting for new mon
-186> 2015-07-29 15:33:10.940452 7fd621a94700 10 monclient: _reopen_session rank -1 name
-185> 2015-07-29 15:33:10.940456 7fd621a94700 1 -
100.100.100.242:6827/11168 mark_down 0xc5c8840 -- pipe dne
184> 2015-07-29 15:33:10.940547 7fd621a94700 10 monclient: picked mon.ceph242 con 0xac2d700 addr 100.100.100.242:6789/0
-183> 2015-07-29 15:33:10.940562 7fd621a94700 10 monclient(hunting): _send_mon_message to mon.ceph242 at 100.100.100.242:6789/0
-182> 2015-07-29 15:33:10.940576 7fd621a94700 1 -
100.100.100.242:6827/11168 --> 100.100.100.242:6789/0 -- auth(proto 0 27 bytes epoch 8) v1 -- ?+0 0x18efe1c0 con 0xac2d700
181> 2015-07-29 15:33:10.940594 7fd621a94700 10 monclient(hunting): renew_subs
-180> 2015-07-29 15:33:10.940606 7fd621a94700 1 osd.49 83012 ms_handle_reset con 0xc5c8840 session 0
-179> 2015-07-29 15:33:10.940840 7fd621a94700 10 monclient(hunting): renew_subs
-178> 2015-07-29 15:33:10.941364 7fd621a94700 1 -
100.100.100.242:6827/11168 <== mon.2 100.100.100.242:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3965078172 0 0) 0x19520d00 con 0xac2d700
177> 2015-07-29 15:33:10.941490 7fd621a94700 10 monclient(hunting): _send_mon_message to mon.ceph242 at 100.100.100.242:6789/0
-176> 2015-07-29 15:33:10.941502 7fd621a94700 1 -
100.100.100.242:6827/11168 --> 100.100.100.242:6789/0 -- auth(proto 2 128 bytes epoch 0) v1 -- ?+0 0x18efdf80 con 0xac2d700
175> 2015-07-29 15:33:10.942469 7fd621a94700 1 - 100.100.100.242:6827/11168 <== mon.2 100.100.100.242:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 225+0+0 (3518583229 0 0) 0x18efe1c0 con 0xac2d700
174> 2015-07-29 15:33:10.942535 7fd621a94700 1 monclient(hunting): found mon.ceph242
-173> 2015-07-29 15:33:10.942540 7fd621a94700 10 monclient: _send_mon_message to mon.ceph242 at 100.100.100.242:6789/0
-172> 2015-07-29 15:33:10.942544 7fd621a94700 1 -
100.100.100.242:6827/11168 --> 100.100.100.242:6789/0 -- mon_subscribe({monmap=9+,osd_pg_creates=0}) v2 -- ?+0 0x125d6ee0 con 0xac2d700
171> 2015-07-29 15:33:10.942555 7fd621a94700 10 monclient: _send_mon_message to mon.ceph242 at 100.100.100.242:6789/0
-170> 2015-07-29 15:33:10.942560 7fd621a94700 1 -
100.100.100.242:6827/11168 --> 100.100.100.242:6789/0 -- mon_subscribe({monmap=9+,osd_pg_creates=0,osdmap=83012}) v2 -- ?+0 0x11b37480 con 0xac2d700
169> 2015-07-29 15:33:10.942578 7fd621a94700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2015-07-29 15:32:40.942577)
-168> 2015-07-29 15:33:10.946661 7fd62ba02700 5 osd.49 83012 tick
-167> 2015-07-29 15:33:10.953091 7fd61c289700 1 -
111.111.111.242:6819/11168 <== osd.8 111.111.111.240:0/12559 4061 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.947432) v2 ==== 47+0+0 (2176729210 0 0) 0x11de5c20 con 0xdc00260
166> 2015-07-29 15:33:10.953121 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.240:0/12559 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.947432) v2 -- ?+0 0x115f3a40 con 0xdc00260
165> 2015-07-29 15:33:10.953115 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.8 111.111.111.240:0/12559 4061 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.947432) v2 ==== 47+0+0 (2176729210 0 0) 0x16ad0940 con 0xdbff600
164> 2015-07-29 15:33:10.953147 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.240:0/12559 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.947432) v2 -- ?+0 0x120ef0c0 con 0xdbff600
163> 2015-07-29 15:33:10.957973 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.19 111.111.111.243:0/8837 4103 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.955356) v2 ==== 47+0+0 (790159330 0 0) 0xff41660 con 0x590a840
162> 2015-07-29 15:33:10.957988 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.19 111.111.111.243:0/8837 4103 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.955356) v2 ==== 47+0+0 (790159330 0 0) 0x11e63660 con 0xdbc6dc0
161> 2015-07-29 15:33:10.958015 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.243:0/8837 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.955356) v2 -- ?+0 0x11de5c20 con 0x590a840
160> 2015-07-29 15:33:10.958044 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.243:0/8837 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.955356) v2 -- ?+0 0x16ad0940 con 0xdbc6dc0
159> 2015-07-29 15:33:10.965966 7fd621a94700 1 - 100.100.100.242:6827/11168 <== mon.2 100.100.100.242:6789/0 3 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (216012157 0 0) 0x16a2fc00 con 0xac2d700
158> 2015-07-29 15:33:10.965985 7fd621a94700 10 monclient: handle_subscribe_ack sent 2015-07-29 15:33:10.940599 renew after 2015-07-29 15:35:40.940599
-157> 2015-07-29 15:33:10.966882 7fd621a94700 1 -
100.100.100.242:6827/11168 <== mon.2 100.100.100.242:6789/0 4 ==== osd_map(83012..83012 src has 75918..83012) v3 ==== 453+0+0 (3237935167 0 0) 0x18efdf80 con 0xac2d700
156> 2015-07-29 15:33:10.966909 7fd621a94700 3 osd.49 83012 handle_osd_map epochs [83012,83012], i have 83012, src has [75918,83012]
-155> 2015-07-29 15:33:10.966924 7fd621a94700 1 -
100.100.100.242:6827/11168 <== mon.2 100.100.100.242:6789/0 5 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (216012157 0 0) 0x16a2fa40 con 0xac2d700
154> 2015-07-29 15:33:10.966934 7fd621a94700 10 monclient: handle_subscribe_ack sent 0.000000, ignoring
-153> 2015-07-29 15:33:10.999612 7fd61c289700 1 -
111.111.111.242:6819/11168 <== osd.15 111.111.111.241:0/5967 4103 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.994757) v2 ==== 47+0+0 (917036899 0 0) 0x123f94a0 con 0xdbde9a0
152> 2015-07-29 15:33:10.999647 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.241:0/5967 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.994757) v2 -- ?+0 0xff41660 con 0xdbde9a0
151> 2015-07-29 15:33:10.999645 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.15 111.111.111.241:0/5967 4103 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:10.994757) v2 ==== 47+0+0 (917036899 0 0) 0xf619680 con 0xdbe1020
150> 2015-07-29 15:33:10.999677 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.241:0/5967 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:10.994757) v2 -- ?+0 0x11e63660 con 0xdbe1020
149> 2015-07-29 15:33:11.026641 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.3 111.111.111.241:0/5501 4122 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.021995) v2 ==== 47+0+0 (3309541758 0 0) 0x11d77840 con 0xdbdf080
148> 2015-07-29 15:33:11.026671 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.241:0/5501 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.021995) v2 -- ?+0 0x123f94a0 con 0xdbdf080
147> 2015-07-29 15:33:11.026675 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.3 111.111.111.241:0/5501 4122 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.021995) v2 ==== 47+0+0 (3309541758 0 0) 0x11ab5680 con 0xf41c000
146> 2015-07-29 15:33:11.026712 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.241:0/5501 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.021995) v2 -- ?+0 0xf619680 con 0xf41c000
145> 2015-07-29 15:33:11.042567 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.7 100.100.100.243:0/9316 4126 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.039823) v2 ==== 47+0+0 (3603736655 0 0) 0x11fb1fe0 con 0xf4a5760
144> 2015-07-29 15:33:11.042608 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.243:0/9316 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.039823) v2 -- ?+0 0x11d77840 con 0xf4a5760
143> 2015-07-29 15:33:11.042694 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.7 100.100.100.243:0/9316 4126 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.039823) v2 ==== 47+0+0 (3603736655 0 0) 0x11ae10e0 con 0xf4a6260
142> 2015-07-29 15:33:11.042725 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.243:0/9316 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.039823) v2 -- ?+0 0x11ab5680 con 0xf4a6260
141> 2015-07-29 15:33:11.059834 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.32 100.100.100.244:0/6584 4164 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.056700) v2 ==== 47+0+0 (1562959942 0 0) 0x125a92c0 con 0xdbc15a0
140> 2015-07-29 15:33:11.059885 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.244:0/6584 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.056700) v2 -- ?+0 0x11ae10e0 con 0xdbc15a0
139> 2015-07-29 15:33:11.059907 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.32 100.100.100.244:0/6584 4164 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.056700) v2 ==== 47+0+0 (1562959942 0 0) 0x112a45a0 con 0xdbad700
138> 2015-07-29 15:33:11.059940 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.244:0/6584 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.056700) v2 -- ?+0 0x11fb1fe0 con 0xdbad700
137> 2015-07-29 15:33:11.079109 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.37 111.111.111.244:0/7305 4093 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.075413) v2 ==== 47+0+0 (1461858650 0 0) 0x122ce940 con 0xdbffce0
136> 2015-07-29 15:33:11.079120 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.37 111.111.111.244:0/7305 4093 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.075413) v2 ==== 47+0+0 (1461858650 0 0) 0x12841a40 con 0xdbff080
135> 2015-07-29 15:33:11.079143 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.244:0/7305 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.075413) v2 -- ?+0 0x112a45a0 con 0xdbffce0
134> 2015-07-29 15:33:11.079187 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.244:0/7305 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.075413) v2 -- ?+0 0x125a92c0 con 0xdbff080
133> 2015-07-29 15:33:11.180221 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.14 100.100.100.240:0/6000 4069 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.175389) v2 ==== 47+0+0 (757796643 0 0) 0x11d52ee0 con 0x1045a000
132> 2015-07-29 15:33:11.180272 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.240:0/6000 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.175389) v2 -- ?+0 0x12841a40 con 0x1045a000
131> 2015-07-29 15:33:11.180278 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.14 100.100.100.240:0/6000 4069 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.175389) v2 ==== 47+0+0 (757796643 0 0) 0x117c6940 con 0x1021e580
130> 2015-07-29 15:33:11.180343 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.240:0/6000 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.175389) v2 -- ?+0 0x122ce940 con 0x1021e580
129> 2015-07-29 15:33:11.207310 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.9 100.100.100.241:0/7510 4111 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.202336) v2 ==== 47+0+0 (518348525 0 0) 0x1342d660 con 0x8974dc0
128> 2015-07-29 15:33:11.207326 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.9 100.100.100.241:0/7510 4111 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.202336) v2 ==== 47+0+0 (518348525 0 0) 0x11fe6b20 con 0x1045b760
127> 2015-07-29 15:33:11.207345 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.241:0/7510 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.202336) v2 -- ?+0 0x117c6940 con 0x8974dc0
126> 2015-07-29 15:33:11.207419 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.241:0/7510 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.202336) v2 -- ?+0 0x11d52ee0 con 0x1045b760
125> 2015-07-29 15:33:11.287409 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.33 111.111.111.244:0/7536 4161 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.284159) v2 ==== 47+0+0 (2128597635 0 0) 0x8e57a20 con 0xdbff8c0
124> 2015-07-29 15:33:11.287449 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.244:0/7536 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.284159) v2 -- ?+0 0x11fe6b20 con 0xdbff8c0
123> 2015-07-29 15:33:11.287597 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.33 111.111.111.244:0/7536 4161 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.284159) v2 ==== 47+0+0 (2128597635 0 0) 0x11faf480 con 0xdc00680
122> 2015-07-29 15:33:11.287628 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.244:0/7536 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.284159) v2 -- ?+0 0x1342d660 con 0xdc00680
121> 2015-07-29 15:33:11.388434 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.6 111.111.111.241:0/8016 4122 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.383671) v2 ==== 47+0+0 (971255437 0 0) 0x11cb7660 con 0xdbc6160
120> 2015-07-29 15:33:11.388426 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.6 111.111.111.241:0/8016 4122 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.383671) v2 ==== 47+0+0 (971255437 0 0) 0x118832a0 con 0xdc00520
119> 2015-07-29 15:33:11.388477 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.241:0/8016 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.383671) v2 -- ?+0 0x8e57a20 con 0xdbc6160
118> 2015-07-29 15:33:11.388516 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.241:0/8016 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.383671) v2 -- ?+0 0x11faf480 con 0xdc00520
117> 2015-07-29 15:33:11.528438 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.3 111.111.111.241:0/5501 4123 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.523710) v2 ==== 47+0+0 (2760192665 0 0) 0x11ab6580 con 0xf41c000
116> 2015-07-29 15:33:11.528484 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.241:0/5501 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.523710) v2 -- ?+0 0x11cb7660 con 0xf41c000
115> 2015-07-29 15:33:11.528519 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.3 111.111.111.241:0/5501 4123 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.523710) v2 ==== 47+0+0 (2760192665 0 0) 0x11d76940 con 0xdbdf080
114> 2015-07-29 15:33:11.528580 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.241:0/5501 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.523710) v2 -- ?+0 0x118832a0 con 0xdbdf080
113> 2015-07-29 15:33:11.561220 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.12 100.100.100.241:0/5260 4106 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.556569) v2 ==== 47+0+0 (3049844282 0 0) 0x11d4e940 con 0xdbc6580
112> 2015-07-29 15:33:11.561223 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.12 100.100.100.241:0/5260 4106 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.556569) v2 ==== 47+0+0 (3049844282 0 0) 0x11992580 con 0xdbaac60
111> 2015-07-29 15:33:11.561254 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.241:0/5260 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.556569) v2 -- ?+0 0x11ab6580 con 0xdbc6580
110> 2015-07-29 15:33:11.561295 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.241:0/5260 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.556569) v2 -- ?+0 0x11d76940 con 0xdbaac60
109> 2015-07-29 15:33:11.621902 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.29 111.111.111.243:0/9094 4101 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.619150) v2 ==== 47+0+0 (1296534809 0 0) 0x124943c0 con 0x590ab00
108> 2015-07-29 15:33:11.621919 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.29 111.111.111.243:0/9094 4101 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.619150) v2 ==== 47+0+0 (1296534809 0 0) 0x133334a0 con 0xdbde580
107> 2015-07-29 15:33:11.621946 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.243:0/9094 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.619150) v2 -- ?+0 0x11992580 con 0x590ab00
106> 2015-07-29 15:33:11.621989 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.243:0/9094 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.619150) v2 -- ?+0 0x11d4e940 con 0xdbde580
105> 2015-07-29 15:33:11.709242 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.9 100.100.100.241:0/7510 4112 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.704311) v2 ==== 47+0+0 (3792342017 0 0) 0x11845680 con 0x8974dc0
104> 2015-07-29 15:33:11.709317 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.241:0/7510 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.704311) v2 -- ?+0 0x124943c0 con 0x8974dc0
103> 2015-07-29 15:33:11.709325 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.9 100.100.100.241:0/7510 4112 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.704311) v2 ==== 47+0+0 (3792342017 0 0) 0x117aeee0 con 0x1045b760
102> 2015-07-29 15:33:11.709368 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.241:0/7510 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.704311) v2 -- ?+0 0x133334a0 con 0x1045b760
101> 2015-07-29 15:33:11.803291 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.27 111.111.111.241:0/6811 1591 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.798272) v2 ==== 47+0+0 (259112101 0 0) 0x120f5860 con 0xf7d98c0
100> 2015-07-29 15:33:11.803332 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.241:0/6811 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.798272) v2 -- ?+0 0x117aeee0 con 0xf7d98c0
99> 2015-07-29 15:33:11.804217 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.27 111.111.111.241:0/6811 1591 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.798272) v2 ==== 47+0+0 (259112101 0 0) 0x11aac780 con 0xf7d8840
98> 2015-07-29 15:33:11.804278 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.241:0/6811 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.798272) v2 -- ?+0 0x11845680 con 0xf7d8840
97> 2015-07-29 15:33:11.833486 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.22 111.111.111.240:0/7591 4115 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.828435) v2 ==== 47+0+0 (3056061894 0 0) 0x11f12580 con 0xf4a6520
96> 2015-07-29 15:33:11.833517 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.240:0/7591 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.828435) v2 -- ?+0 0x120f5860 con 0xf4a6520
95> 2015-07-29 15:33:11.834497 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.22 111.111.111.240:0/7591 4115 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:11.828435) v2 ==== 47+0+0 (3056061894 0 0) 0x117fd4a0 con 0x1021e2c0
94> 2015-07-29 15:33:11.834536 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.240:0/7591 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:11.828435) v2 -- ?+0 0x11aac780 con 0x1021e2c0
93> 2015-07-29 15:33:11.946769 7fd62ba02700 5 osd.49 83012 tick
-92> 2015-07-29 15:33:12.012957 7fd61da8c700 1 -
100.100.100.242:6828/11168 <== osd.11 100.100.100.240:0/7143 4074 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.008182) v2 ==== 47+0+0 (4049433826 0 0) 0x11ed5680 con 0x1045a580
91> 2015-07-29 15:33:12.012964 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.11 100.100.100.240:0/7143 4074 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.008182) v2 ==== 47+0+0 (4049433826 0 0) 0x1249b2a0 con 0x1045ac60
90> 2015-07-29 15:33:12.013018 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.240:0/7143 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.008182) v2 -- ?+0 0x11f12580 con 0x1045a580
89> 2015-07-29 15:33:12.013060 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.240:0/7143 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.008182) v2 -- ?+0 0x117fd4a0 con 0x1045ac60
88> 2015-07-29 15:33:12.050933 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.2 100.100.100.240:0/20887 213 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.045851) v2 ==== 47+0+0 (27849096 0 0) 0x121c1e00 con 0xf7d8f20
87> 2015-07-29 15:33:12.050967 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.240:0/20887 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.045851) v2 -- ?+0 0x1249b2a0 con 0xf7d8f20
86> 2015-07-29 15:33:12.050966 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.2 100.100.100.240:0/20887 213 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.045851) v2 ==== 47+0+0 (27849096 0 0) 0x1283e3a0 con 0xf7d9fa0
85> 2015-07-29 15:33:12.050999 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.240:0/20887 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.045851) v2 -- ?+0 0x11ed5680 con 0xf7d9fa0
84> 2015-07-29 15:33:12.106082 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.34 100.100.100.244:0/6968 4118 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.102776) v2 ==== 47+0+0 (3374910409 0 0) 0x123263a0 con 0xbcfe160
83> 2015-07-29 15:33:12.106080 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.34 100.100.100.244:0/6968 4118 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.102776) v2 ==== 47+0+0 (3374910409 0 0) 0x12044000 con 0xbe7e840
82> 2015-07-29 15:33:12.106117 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.244:0/6968 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.102776) v2 -- ?+0 0x1283e3a0 con 0xbcfe160
81> 2015-07-29 15:33:12.106153 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.244:0/6968 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.102776) v2 -- ?+0 0x121c1e00 con 0xbe7e840
80> 2015-07-29 15:33:12.159303 7fd60cf61700 1 - 111.111.111.242:6818/11168 <== osd.0 111.111.111.241:6809/7006182 278 ==== osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[]) v11 ==== 1603+0+1694 (3034535879 0 3039739132) 0x13725600 con 0xc5df860
79> 2015-07-29 15:33:12.159343 7fd60cf61700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159177, event: header_read, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
78> 2015-07-29 15:33:12.159361 7fd60cf61700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159180, event: throttled, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
77> 2015-07-29 15:33:12.159375 7fd60cf61700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159296, event: all_read, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
76> 2015-07-29 15:33:12.159391 7fd60cf61700 5 - op tracker -- seq: 8860, time: 0.000000, event: dispatched, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
75> 2015-07-29 15:33:12.159450 7fd614d76700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159450, event: reached_pg, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
74> 2015-07-29 15:33:12.159500 7fd614d76700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159500, event: started, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
73> 2015-07-29 15:33:12.159854 7fd614d76700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159854, event: started, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
72> 2015-07-29 15:33:12.159893 7fd614d76700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159893, event: commit_queued_for_journal_write, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
71> 2015-07-29 15:33:12.159984 7fd6281fb700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.159984, event: write_thread_in_journal_buffer, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
70> 2015-07-29 15:33:12.160172 7fd6279fa700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.160171, event: journaled_completion_queued, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
69> 2015-07-29 15:33:12.160247 7fd6251f5700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.160247, event: commit_sent, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
68> 2015-07-29 15:33:12.160278 7fd6251f5700 1 - 111.111.111.242:6818/11168 --> 111.111.111.241:6809/7006182 -- osd_sub_op_reply(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] ondisk, result = 0) v2 -- ?+0 0x164b5340 con 0xc5df860
67> 2015-07-29 15:33:12.162037 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.1 100.100.100.243:0/10013 4122 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.159186) v2 ==== 47+0+0 (1945120148 0 0) 0x11d545a0 con 0xbd57180
66> 2015-07-29 15:33:12.162081 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.243:0/10013 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.159186) v2 -- ?+0 0x123263a0 con 0xbd57180
65> 2015-07-29 15:33:12.162244 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.1 100.100.100.243:0/10013 4122 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.159186) v2 ==== 47+0+0 (1945120148 0 0) 0x113781e0 con 0x8899e40
64> 2015-07-29 15:33:12.162274 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.243:0/10013 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.159186) v2 -- ?+0 0x12044000 con 0x8899e40
63> 2015-07-29 15:33:12.167659 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.4 100.100.100.243:0/7173 4101 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.164928) v2 ==== 47+0+0 (3758003415 0 0) 0x16ad4760 con 0x1045af20
62> 2015-07-29 15:33:12.167676 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.4 100.100.100.243:0/7173 4101 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.164928) v2 ==== 47+0+0 (3758003415 0 0) 0x125b92c0 con 0x1045b4a0
61> 2015-07-29 15:33:12.167701 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.243:0/7173 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.164928) v2 -- ?+0 0x113781e0 con 0x1045af20
60> 2015-07-29 15:33:12.167761 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.243:0/7173 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.164928) v2 -- ?+0 0x11d545a0 con 0x1045b4a0
59> 2015-07-29 15:33:12.182436 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.37 111.111.111.244:0/7305 4094 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.178038) v2 ==== 47+0+0 (300687293 0 0) 0x128405a0 con 0xdbff080
58> 2015-07-29 15:33:12.182387 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.37 111.111.111.244:0/7305 4094 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.178038) v2 ==== 47+0+0 (300687293 0 0) 0x18cd94a0 con 0xdbffce0
57> 2015-07-29 15:33:12.182471 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.244:0/7305 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.178038) v2 -- ?+0 0x125b92c0 con 0xdbff080
56> 2015-07-29 15:33:12.182502 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.244:0/7305 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.178038) v2 -- ?+0 0x16ad4760 con 0xdbffce0
55> 2015-07-29 15:33:12.282079 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.14 100.100.100.240:0/6000 4070 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.277369) v2 ==== 47+0+0 (3974314133 0 0) 0x123bc1e0 con 0x1045a000
54> 2015-07-29 15:33:12.282082 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.14 100.100.100.240:0/6000 4070 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.277369) v2 ==== 47+0+0 (3974314133 0 0) 0x11ce5c20 con 0x1021e580
53> 2015-07-29 15:33:12.282121 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.240:0/6000 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.277369) v2 -- ?+0 0x128405a0 con 0x1045a000
52> 2015-07-29 15:33:12.282157 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.240:0/6000 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.277369) v2 -- ?+0 0x18cd94a0 con 0x1021e580
51> 2015-07-29 15:33:12.332377 7fd60c456700 1 - 111.111.111.242:6818/11168 <== osd.21 111.111.111.241:6801/6009073 213 ==== osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[]) v11 ==== 1603+0+1694 (626930942 0 755294468) 0x13a4f600 con 0xdb1d600
50> 2015-07-29 15:33:12.332411 7fd60c456700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.332234, event: header_read, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
49> 2015-07-29 15:33:12.332428 7fd60c456700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.332237, event: throttled, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
48> 2015-07-29 15:33:12.332438 7fd60c456700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.332369, event: all_read, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
47> 2015-07-29 15:33:12.332451 7fd60c456700 5 - op tracker -- seq: 8861, time: 0.000000, event: dispatched, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
46> 2015-07-29 15:33:12.332503 7fd616579700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.332503, event: reached_pg, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
45> 2015-07-29 15:33:12.332546 7fd616579700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.332546, event: started, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
44> 2015-07-29 15:33:12.333216 7fd616579700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.333216, event: started, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
43> 2015-07-29 15:33:12.333273 7fd616579700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.333273, event: commit_queued_for_journal_write, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
42> 2015-07-29 15:33:12.333338 7fd6281fb700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.333337, event: write_thread_in_journal_buffer, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
41> 2015-07-29 15:33:12.333502 7fd6279fa700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.333502, event: journaled_completion_queued, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
40> 2015-07-29 15:33:12.333544 7fd6251f5700 5 - op tracker -- seq: 8861, time: 2015-07-29 15:33:12.333544, event: commit_sent, op: osd_sub_op(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] v 83012'124 snapset=0=[]:[] snapc=0=[])
39> 2015-07-29 15:33:12.333573 7fd6251f5700 1 - 111.111.111.242:6818/11168 --> 111.111.111.241:6801/6009073 -- osd_sub_op_reply(client.8458935.0:26 37.15 19185a15/rbd_data.402f432bc84e87.0000000000000011/head//37 [] ondisk, result = 0) v2 -- ?+0 0x164b6680 con 0xdb1d600
38> 2015-07-29 15:33:12.437046 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.26 100.100.100.243:0/5963 4107 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.433671) v2 ==== 47+0+0 (582745337 0 0) 0x18c39c00 con 0x1045a2c0
37> 2015-07-29 15:33:12.437051 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.26 100.100.100.243:0/5963 4107 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.433671) v2 ==== 47+0+0 (582745337 0 0) 0x16c40940 con 0x1045a9a0
36> 2015-07-29 15:33:12.437088 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.243:0/5963 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.433671) v2 -- ?+0 0x123bc1e0 con 0x1045a2c0
35> 2015-07-29 15:33:12.437126 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.243:0/5963 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.433671) v2 -- ?+0 0x11ce5c20 con 0x1045a9a0
34> 2015-07-29 15:33:12.464960 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.40 111.111.111.244:0/8930 4126 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.460918) v2 ==== 47+0+0 (2275200791 0 0) 0xf296ee0 con 0xdbdef20
33> 2015-07-29 15:33:12.464979 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.40 111.111.111.244:0/8930 4126 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.460918) v2 ==== 47+0+0 (2275200791 0 0) 0x10d01a20 con 0xdbe15a0
32> 2015-07-29 15:33:12.465014 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.244:0/8930 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.460918) v2 -- ?+0 0x16c40940 con 0xdbdef20
31> 2015-07-29 15:33:12.465060 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.244:0/8930 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.460918) v2 -- ?+0 0x18c39c00 con 0xdbe15a0
30> 2015-07-29 15:33:12.540937 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.5 100.100.100.240:0/12334 4087 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.536091) v2 ==== 47+0+0 (2485156309 0 0) 0x124fda40 con 0xbe15020
29> 2015-07-29 15:33:12.540985 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.5 100.100.100.240:0/12334 4087 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.536091) v2 ==== 47+0+0 (2485156309 0 0) 0x1283e940 con 0xbc9e7e0
28> 2015-07-29 15:33:12.540999 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.240:0/12334 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.536091) v2 -- ?+0 0xf296ee0 con 0xbe15020
27> 2015-07-29 15:33:12.541092 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.240:0/12334 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.536091) v2 -- ?+0 0x10d01a20 con 0xbc9e7e0
26> 2015-07-29 15:33:12.629701 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.23 100.100.100.243:0/9546 4168 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.626463) v2 ==== 47+0+0 (995941527 0 0) 0x127110e0 con 0xf4a51e0
25> 2015-07-29 15:33:12.629747 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 100.100.100.243:0/9546 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.626463) v2 -- ?+0 0x124fda40 con 0xf4a51e0
24> 2015-07-29 15:33:12.629746 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.23 100.100.100.243:0/9546 4168 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.626463) v2 ==== 47+0+0 (995941527 0 0) 0x11548760 con 0xf4a4160
23> 2015-07-29 15:33:12.629786 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 100.100.100.243:0/9546 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.626463) v2 -- ?+0 0x1283e940 con 0xf4a4160
22> 2015-07-29 15:33:12.648241 7fd61da8c700 1 - 100.100.100.242:6828/11168 <== osd.39 111.111.111.244:0/8246 4074 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.644770) v2 ==== 47+0+0 (572865005 0 0) 0x16ad3860 con 0x1045b8c0
21> 2015-07-29 15:33:12.648250 7fd61c289700 1 - 111.111.111.242:6819/11168 <== osd.39 111.111.111.244:0/8246 4074 ==== osd_ping(ping e83012 stamp 2015-07-29 15:33:12.644770) v2 ==== 47+0+0 (572865005 0 0) 0x133bb0e0 con 0x891eaa0
20> 2015-07-29 15:33:12.648273 7fd61da8c700 1 - 100.100.100.242:6828/11168 --> 111.111.111.244:0/8246 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.644770) v2 -- ?+0 0x11548760 con 0x1045b8c0
19> 2015-07-29 15:33:12.648309 7fd61c289700 1 - 111.111.111.242:6819/11168 --> 111.111.111.244:0/8246 -- osd_ping(ping_reply e83012 stamp 2015-07-29 15:33:12.644770) v2 -- ?+0 0x127110e0 con 0x891eaa0
18> 2015-07-29 15:33:12.676524 7fd60c456700 1 - 111.111.111.242:6818/11168 <== osd.21 111.111.111.241:6801/6009073 214 ==== osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[]) v11 ==== 1603+0+1694 (699979673 0 1828030878) 0x13a4eb00 con 0xdb1d600
17> 2015-07-29 15:33:12.676557 7fd60c456700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.676404, event: header_read, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
16> 2015-07-29 15:33:12.676573 7fd60c456700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.676406, event: throttled, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
15> 2015-07-29 15:33:12.676594 7fd60c456700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.676517, event: all_read, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
14> 2015-07-29 15:33:12.676606 7fd60c456700 5 - op tracker -- seq: 8862, time: 0.000000, event: dispatched, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
13> 2015-07-29 15:33:12.676676 7fd613d74700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.676676, event: reached_pg, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
12> 2015-07-29 15:33:12.676714 7fd613d74700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.676714, event: started, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
11> 2015-07-29 15:33:12.677195 7fd613d74700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.677195, event: started, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
10> 2015-07-29 15:33:12.677226 7fd613d74700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.677226, event: commit_queued_for_journal_write, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
9> 2015-07-29 15:33:12.677302 7fd6281fb700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.677302, event: write_thread_in_journal_buffer, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
8> 2015-07-29 15:33:12.677517 7fd6279fa700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.677517, event: journaled_completion_queued, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
7> 2015-07-29 15:33:12.677577 7fd6251f5700 5 - op tracker -- seq: 8862, time: 2015-07-29 15:33:12.677577, event: commit_sent, op: osd_sub_op(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] v 83012'126 snapset=0=[]:[] snapc=0=[])
6> 2015-07-29 15:33:12.677607 7fd6251f5700 1 - 111.111.111.242:6818/11168 --> 111.111.111.241:6801/6009073 -- osd_sub_op_reply(client.8458935.0:33 37.15 41a05d95/rbd_data.402f432bc84e87.0000000000000018/head//37 [] ondisk, result = 0) v2 -- ?+0 0x164b63c0 con 0xdb1d600
5> 2015-07-29 15:33:12.690998 7fd6259f6700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.690998, event: sub_op_applied, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
4> 2015-07-29 15:33:12.691056 7fd6259f6700 5 - op tracker -- seq: 8860, time: 2015-07-29 15:33:12.691056, event: done, op: osd_sub_op(client.8458935.0:12 37.54 ed9da8d4/rbd_data.402f432bc84e87.0000000000000003/head//37 [] v 83012'132 snapset=0=[]:[] snapc=0=[])
-3> 2015-07-29 15:33:12.725760 7fd6269f8700 0 filestore(/var/lib/ceph/osd/ceph-49) error (2) No such file or directory not handled on operation 17 (2162103.0.10, or op 10, counting from 0)
-2> 2015-07-29 15:33:12.725783 7fd6269f8700 0 filestore(/var/lib/ceph/osd/ceph-49) ENOENT on clone suggests osd bug
-1> 2015-07-29 15:33:12.725784 7fd6269f8700 0 filestore(/var/lib/ceph/osd/ceph-49) transaction dump: { "ops": [ { "op_num": 0,
"op_name": "omap_setkeys",
"collection": "meta",
"oid": "a468ec03\/snapmapper\/0\/\/-1",
"attr_lens": { "OBJ_0000000000000025.51A58191.34.rbd%udata%e402f432bc84e87%e0000000000000011..": 105}}, { "op_num": 1,
"op_name": "omap_setkeys",
"collection": "meta",
"oid": "a468ec03\/snapmapper\/0\/\/-1",
"attr_lens": { "MAP_000000000000002C_0000000000000025.51A58191.34.rbd%udata%e402f432bc84e87%e0000000000000011..": 93,
"MAP_0000000000000034_0000000000000025.51A58191.34.rbd%udata%e402f432bc84e87%e0000000000000011..": 93}}, { "op_num": 2,
"op_name": "omap_setkeys",
"collection": "meta",
"oid": "36948\/pglog_37.15\/0\/\/-1",
"attr_lens": { "0000083012.00000000000000000123": 207,
"0000083012.00000000000000000124": 180}}, { "op_num": 3,
"op_name": "omap_setkeys",
"collection": "meta",
"oid": "16ef7597\/infos\/head\/\/-1",
"attr_lens": { "37.15_epoch": 4,
"37.15_info": 729}}, { "op_num": 4,
"op_name": "omap_rmkeys",
"collection": "meta",
"oid": "36948\/pglog_37.15\/0\/\/-1"}, { "op_num": 5,
"op_name": "omap_setkeys",
"collection": "meta",
"oid": "36948\/pglog_37.15\/0\/\/-1",
"attr_lens": { "0000083012.00000000000000000123": 207,
"0000083012.00000000000000000124": 180,
"can_rollback_to": 12}}, { "op_num": 6,
"op_name": "clone",
"collection": "37.15_head",
"src_oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/head\/\/37",
"dst_oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/34\/\/37"}, { "op_num": 7,
"op_name": "setattr",
"collection": "37.15_head",
"oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/34\/\/37",
"name": "_",
"length": 260}, { "op_num": 8,
"op_name": "rmattr",
"collection": "37.15_head",
"oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/34\/\/37",
"name": "snapset"}, { "op_num": 9,
"op_name": "remove",
"collection": "37.15_head",
"oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/head\/\/37"}, { "op_num": 10,
"op_name": "clone",
"collection": "37.15_head",
"src_oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/f\/\/37",
"dst_oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/head\/\/37"}, { "op_num": 11,
"op_name": "setattr",
"collection": "37.15_head",
"oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/head\/\/37",
"name": "_",
"length": 267}, { "op_num": 12,
"op_name": "setattr",
"collection": "37.15_head",
"oid": "19185a15\/rbd_data.402f432bc84e87.0000000000000011\/head\/\/37",
"name": "snapset",
"length": 159}]}
0> 2015-07-29 15:33:12.729939 7fd6269f8700 -1 os/FileStore.cc: In function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int, ThreadPool::TPHandle*)' thread 7fd6269f8700 time 2015-07-29 15:33:12.728175
os/FileStore.cc: 2715: FAILED assert(0 == "unexpected error")

ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb7d925]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x888) [0x8f7098]
3: (FileStore::_do_transactions(std::list&lt;ObjectStore::Transaction*, std::allocator&lt;ObjectStore::Transaction*&gt; >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x8fb654]
4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x8fb7f8]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa66) [0xb6e636]
6: (ThreadPool::WorkThread::entry()+0x10) [0xb6f6c0]
7: (()+0x7df3) [0x7fd633027df3]
8: (clone()+0x6d) [0x7fd631b0a54d]
NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.49.log
--
end dump of recent events ---
2015-07-29 15:33:12.819093 7fd6269f8700 -1 ** Caught signal (Aborted) *
in thread 7fd6269f8700

ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
1: /usr/bin/ceph-osd() [0xa8b002]
2: (()+0xf130) [0x7fd63302f130]
3: (gsignal()+0x39) [0x7fd631a495f9]
4: (abort()+0x148) [0x7fd631a4b068]
5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fd63234d9d5]
6: (()+0x5e946) [0x7fd63234b946]
7: (()+0x5e973) [0x7fd63234b973]
8: (()+0x5eb9f) [0x7fd63234bb9f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xb7db1a]
10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x888) [0x8f7098]
11: (FileStore::_do_transactions(std::list&lt;ObjectStore::Transaction*, std::allocator&lt;ObjectStore::Transaction*&gt; >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x8fb654]
12: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x8fb7f8]
13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa66) [0xb6e636]
14: (ThreadPool::WorkThread::entry()+0x10) [0xb6f6c0]
15: (()+0x7df3) [0x7fd633027df3]
16: (clone()+0x6d) [0x7fd631b0a54d]
NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- begin dump of recent events ---
0> 2015-07-29 15:33:12.819093 7fd6269f8700 -1 ** Caught signal (Aborted) *
in thread 7fd6269f8700

ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)
1: /usr/bin/ceph-osd() [0xa8b002]
2: (()+0xf130) [0x7fd63302f130]
3: (gsignal()+0x39) [0x7fd631a495f9]
4: (abort()+0x148) [0x7fd631a4b068]
5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fd63234d9d5]
6: (()+0x5e946) [0x7fd63234b946]
7: (()+0x5e973) [0x7fd63234b973]
8: (()+0x5eb9f) [0x7fd63234bb9f]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27a) [0xb7db1a]
10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0x888) [0x8f7098]
11: (FileStore::_do_transactions(std::list&lt;ObjectStore::Transaction*, std::allocator&lt;ObjectStore::Transaction*&gt; >&, unsigned long, ThreadPool::TPHandle*)+0x64) [0x8fb654]
12: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x188) [0x8fb7f8]
13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa66) [0xb6e636]
14: (ThreadPool::WorkThread::entry()+0x10) [0xb6f6c0]
15: (()+0x7df3) [0x7fd633027df3]
16: (clone()+0x6d) [0x7fd631b0a54d]
NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.49.log
--
end dump of recent events ---

History

#1 Updated by Samuel Just over 7 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF