Bug #22617: ceph rgw can't delete some object - rgw - Ceph

Actions

Copy link

Bug #22617

open

ceph rgw can't delete some object

Added by fpq f over 6 years ago. Updated over 6 years ago.

Status:

Need More Info

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

I put some files on ceph and when i delete files "s3cmd rm s3://fpq/fpq.mp3",and i check on ceph culster “rados -p default.rgw.buckets.data ls”,i found some object can't delete like

93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.GFG5Wj7Pjh2F1Ykpz8fN0Ve8OpBZYGh_1
93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.GFG5Wj7Pjh2F1Ykpz8fN0Ve8OpBZYGh_2

and i see rgw-node
[root@ceph-rgw ceph]# radosgw-admin gc list --include-all --cluster ceph --name client.radosgw.gateway
2018-01-09 01:11:02.716499 7ff5d1ca59c0 0 RGWZoneParams::create(): error creating default zone params: (17) File exists
[ {
"tag": "93541ddd-3a56-49fe-a109-83e09bcd7483.211795.12\u0000",
"time": "2018-01-09 03:10:41.0.910238s",
"objs": [ {
"pool": "default.rgw.buckets.data",
"oid": "93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.GFG5Wj7Pjh2F1Ykpz8fN0Ve8OpBZYGh_1",
"key": "",
"instance": ""
}, {
"pool": "default.rgw.buckets.data",
"oid": "93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.GFG5Wj7Pjh2F1Ykpz8fN0Ve8OpBZYGh_2",
"key": "",
"instance": ""
}
]
}
]

and less /var/log/ceph/client.radosgw.gateway.log

2018-01-09 00:02:44.935987 7f4f337fe700 0 gc::process: removing default.rgw.buckets.data:93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.US9QzR0IwEIjQqT1hOeT4z3LGOUQvOB_1
2018-01-09 00:02:44.952399 7f4f337fe700 0 gc::process: removing default.rgw.buckets.data:93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.US9QzR0IwEIjQqT1hOeT4z3LGOUQvOB_2
2018-01-09 01:02:46.033928 7f4f337fe700 0 gc::process: removing default.rgw.buckets.data:93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.Ft7LpRIVKhT1BAkIkP-qgdLTt8GYWJR_1
2018-01-09 01:02:46.049797 7f4f337fe700 0 gc::process: removing default.rgw.buckets.data:93541ddd-3a56-49fe-a109-83e09bcd7483.44424.2__shadow_.Ft7LpRIVKhT1BAkIkP-qgdLTt8GYWJR_2
2018-01-09 01:08:49.354710 7f30d0d699c0 0 RGWZoneParams::create(): error creating default zone params: (17) File exists
2018-01-09 01:10:38.716822 7f4ef3fbf700 1 ====== starting new request req=0x7f4ef3fb9710 =====
2018-01-09 01:10:39.183636 7f4ef3fbf700 1 ====== req done req=0x7f4ef3fb9710 op status=0 http_status=200 ======
2018-01-09 01:10:39.183750 7f4ef3fbf700 1 civetweb: 0x7f4f5c001f60: 10.134.238.153 - - [09/Jan/2018:01:10:38 +0800] "PUT /Test1/fpq.mp3 HTTP/1.1" 200 0 - -
2018-01-09 01:10:52.173880 7f4f087e8700 1 ====== starting new request req=0x7f4f087e2710 =====
2018-01-09 01:10:52.222616 7f4f087e8700 1 ====== req done req=0x7f4f087e2710 op status=0 http_status=204 ======
2018-01-09 01:10:52.222686 7f4f087e8700 1 civetweb: 0x7f4f60000f90: 10.134.238.153 - - [09/Jan/2018:01:10:52 +0800] "DELETE /Test1/fpq.mp3 HTTP/1.1" 204 0 - -
2018-01-09 01:11:02.716499 7ff5d1ca59c0 0 RGWZoneParams::create(): error creating default zone params: (17) File exists

Files

Download all files

client.radosgw.gateway.log.txt (69.8 KB) client.radosgw.gateway.log.txt		fpq f, 01/12/2018 04:38 PM
test.txt (38.6 KB) test.txt		fpq f, 01/16/2018 01:02 PM

Actions

Copy link

Updated by Matt Benjamin over 6 years ago

Status changed from New to Need More Info

@fpq f, this looks like something we fixed; could you attach a log snippet with --debug-rgw=18 --debug-ms=1 ?

the latter will show osd_ops as well as RGW events.

thanks!

Matt

Actions

Copy link

Updated by fpq f over 6 years ago

@Matt Li Benjamin thank you and I look forward to your reply
[root@ceph-rgw ~]# radosgw-admin gc list --include-all --cluster ceph --name client.radosgw.gateway --debug-rgw=18 --debug-ms=1
2018-01-12 23:39:46.207088 7f6195c1c9c0 1 -- :/0 messenger.start
2018-01-12 23:39:46.208343 7f6195c1c9c0 1 -- :/1325339148 --> 10.134.238.157:6789/0 -- auth(proto 0 40 bytes epoch 0) v1 -- ?+0 0x559e7aa5d0c0 con 0x559e7aa5cae0
2018-01-12 23:39:46.208860 7f6195c12700 1 -- 10.134.238.152:0/1325339148 learned my addr 10.134.238.152:0/1325339148
2018-01-12 23:39:46.209590 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 1 ==== mon_map magic: 0 v1 ==== 500+0+0 (3223744036 0 0) 0x7f6168000b90 con 0x559e7aa5cae0
2018-01-12 23:39:46.209660 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (3247362194 0 0) 0x7f6168001050 con 0x559e7aa5cae0
2018-01-12 23:39:46.209770 7f617994b700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7f6164001880 con 0x559e7aa5cae0
2018-01-12 23:39:46.210213 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (3070268181 0 0) 0x7f6168001050 con 0x559e7aa5cae0
2018-01-12 23:39:46.210317 7f617994b700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x7f6164003390 con 0x559e7aa5cae0
2018-01-12 23:39:46.210770 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 425+0+0 (2410636779 0 0) 0x7f61680008c0 con 0x559e7aa5cae0
2018-01-12 23:39:46.210845 7f617994b700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x559e7aa60ec0 con 0x559e7aa5cae0
2018-01-12 23:39:46.210914 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x559e7aa60ec0 con 0x559e7aa5cae0
2018-01-12 23:39:46.211086 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 5 ==== mon_map magic: 0 v1 ==== 500+0+0 (3223744036 0 0) 0x7f61680012b0 con 0x559e7aa5cae0
2018-01-12 23:39:46.211221 7f6171ffb700 2 RGWDataChangesLog::ChangesRenewThread: start
2018-01-12 23:39:46.211438 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 6 ==== osd_map(265..265 src has 1..265) v3 ==== 8305+0+0 (896830133 0 0) 0x7f61680008c0 con 0x559e7aa5cae0
2018-01-12 23:39:46.211712 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:1 3.85fca992 default.realm [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa66860 con 0x559e7aa65370
2018-01-12 23:39:46.214735 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 1 ==== osd_op_reply(1 default.realm [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 175+0+0 (2283260449 0 0) 0x7f6154000a50 con 0x559e7aa65370
2018-01-12 23:39:46.214874 7f6195c1c9c0 1 - 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:2 3.85fca992 default.realm [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa66f40 con 0x559e7aa65370
2018-01-12 23:39:46.215898 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 2 ==== osd_op_reply(2 default.realm [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 175+0+0 (2283260449 0 0) 0x7f6154000a50 con 0x559e7aa65370
2018-01-12 23:39:46.215970 7f6195c1c9c0 10 could not read realm id: (2) No such file or directory
2018-01-12 23:39:46.216082 7f6195c1c9c0 1 - 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:3 3.0 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6ac00 con 0x559e7aa66ac0
2018-01-12 23:39:46.219417 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 1 ==== osd_op_reply(3 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (73087745 0 3248547820) 0x7f614c000b50 con 0x559e7aa66ac0
2018-01-12 23:39:46.219463 7f61714f7700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:4 3.1 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f614c002580 con 0x559e7aa65370
2018-01-12 23:39:46.220554 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 3 ==== osd_op_reply(4 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2239183445 0 3248547820) 0x7f6154000a50 con 0x559e7aa65370
2018-01-12 23:39:46.220595 7f61716f9700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:5 3.2 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f61540029e0 con 0x559e7aa65370
2018-01-12 23:39:46.221589 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 4 ==== osd_op_reply(5 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (66015064 0 3248547820) 0x7f6154000a50 con 0x559e7aa65370
2018-01-12 23:39:46.221669 7f61716f9700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:6 3.3 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f6154005d70 con 0x7f6154004930
2018-01-12 23:39:46.223072 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 1 ==== osd_op_reply(6 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+103 (2535676660 0 4170302097) 0x7f6144000b80 con 0x7f6154004930
2018-01-12 23:39:46.223140 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:7 3.4 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6b040 con 0x7f6154004930
2018-01-12 23:39:46.223615 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 2 ==== osd_op_reply(7 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (187945907 0 3248547820) 0x7f6144000b80 con 0x7f6154004930
2018-01-12 23:39:46.223654 7f61712f5700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:8 3.5 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f61440027f0 con 0x559e7aa65370
2018-01-12 23:39:46.224419 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 5 ==== osd_op_reply(8 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2317341415 0 3248547820) 0x7f6154000a50 con 0x559e7aa65370
2018-01-12 23:39:46.224454 7f61716f9700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:9 3.6 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f61540062a0 con 0x559e7aa65370
2018-01-12 23:39:46.225100 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 6 ==== osd_op_reply(9 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+98 (777405040 0 661761295) 0x7f61540062a0 con 0x559e7aa65370
2018-01-12 23:39:46.225152 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:10 3.7 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6b040 con 0x559e7aa65370
2018-01-12 23:39:46.225856 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 7 ==== osd_op_reply(10 [pgls start_epoch 0] v75'2 uv2 ondisk = 1) v7 ==== 120+0+102 (3625854734 0 37397930) 0x7f61540062a0 con 0x559e7aa65370
2018-01-12 23:39:46.225930 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:11 3.bd31b503 region_map [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa66f40 con 0x7f6154004930
2018-01-12 23:39:46.226424 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 3 ==== osd_op_reply(11 region_map [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 172+0+0 (832903909 0 0) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.226484 7f6195c1c9c0 10 cannot find current period zonegroup using local zonegroup
2018-01-12 23:39:46.226532 7f6195c1c9c0 1 - 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:12 3.85fca992 default.realm [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa66f40 con 0x559e7aa65370
2018-01-12 23:39:46.227685 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 8 ==== osd_op_reply(12 default.realm [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 175+0+0 (2283260449 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.227741 7f6195c1c9c0 10 could not read realm id: (2) No such file or directory
2018-01-12 23:39:46.227749 7f6195c1c9c0 10 Creating default zonegroup
2018-01-12 23:39:46.227785 7f6195c1c9c0 1 - 10.134.238.152:0/1325339148 --> 10.134.238.157:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x559e7aa66860 con 0x559e7aa5cae0
2018-01-12 23:39:46.228059 7f617994b700 1 -- 10.134.238.152:0/1325339148 <== mon.0 10.134.238.157:6789/0 7 ==== mon_get_version_reply(handle=1 version=265) v2 ==== 24+0+0 (4011090033 0 0) 0x7f61680008c0 con 0x559e7aa5cae0
2018-01-12 23:39:46.228148 7f6195c1c9c0 10 couldn't find old data placement pools config, setting up new ones for the zone
2018-01-12 23:39:46.228188 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:13 3.0 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa679b0 con 0x559e7aa66ac0
2018-01-12 23:39:46.228901 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 2 ==== osd_op_reply(13 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (73087745 0 3248547820) 0x7f614c000b50 con 0x559e7aa66ac0
2018-01-12 23:39:46.228938 7f61714f7700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:14 3.1 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f614c002b90 con 0x559e7aa65370
2018-01-12 23:39:46.229581 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 9 ==== osd_op_reply(14 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2239183445 0 3248547820) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.229639 7f61716f9700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:15 3.2 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f6154001e70 con 0x559e7aa65370
2018-01-12 23:39:46.230093 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 10 ==== osd_op_reply(15 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (66015064 0 3248547820) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.230129 7f61716f9700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:16 3.3 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f61540023f0 con 0x7f6154004930
2018-01-12 23:39:46.230591 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 4 ==== osd_op_reply(16 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+103 (2535676660 0 4170302097) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.230661 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:17 3.4 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa679b0 con 0x7f6154004930
2018-01-12 23:39:46.231103 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 5 ==== osd_op_reply(17 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (187945907 0 3248547820) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.231139 7f61712f5700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:18 3.5 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f6144002040 con 0x559e7aa65370
2018-01-12 23:39:46.231679 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 11 ==== osd_op_reply(18 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2317341415 0 3248547820) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.231715 7f61716f9700 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:19 3.6 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7f61540023c0 con 0x559e7aa65370
2018-01-12 23:39:46.232340 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 12 ==== osd_op_reply(19 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+98 (777405040 0 661761295) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.232401 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:20 3.7 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa679b0 con 0x559e7aa65370
2018-01-12 23:39:46.233469 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 13 ==== osd_op_reply(20 [pgls start_epoch 0] v75'2 uv2 ondisk = 1) v7 ==== 120+0+102 (3625854734 0 37397930) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.233546 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:21 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6b480 con 0x559e7aa65370
2018-01-12 23:39:46.234149 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 14 ==== osd_op_reply(21 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.234239 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:22 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6b480 con 0x559e7aa65370
2018-01-12 23:39:46.235557 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 15 ==== osd_op_reply(22 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.235640 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:23 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6bc90 con 0x559e7aa65370
2018-01-12 23:39:46.236707 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 16 ==== osd_op_reply(23 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.236764 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:24 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa63350 con 0x559e7aa65370
2018-01-12 23:39:46.237811 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 17 ==== osd_op_reply(24 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.237952 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:25 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6c600 con 0x559e7aa65370
2018-01-12 23:39:46.239030 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 18 ==== osd_op_reply(25 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.239084 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:26 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6c600 con 0x559e7aa65370
2018-01-12 23:39:46.240176 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 19 ==== osd_op_reply(26 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.240215 7f6195c1c9c0 10 ERROR: name default already in use for obj id 93541ddd-3a56-49fe-a109-83e09bcd7483
2018-01-12 23:39:46.240224 7f6195c1c9c0 0 RGWZoneParams::create(): error creating default zone params: (17) File exists
2018-01-12 23:39:46.240231 7f6195c1c9c0 10 zone_params::create_default() returned EEXIST, we raced with another default zone_params creation
2018-01-12 23:39:46.240257 7f6195c1c9c0 1 - 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:27 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x559e7aa65370
2018-01-12 23:39:46.240691 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 20 ==== osd_op_reply(27 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.240746 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:28 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x559e7aa65370
2018-01-12 23:39:46.241455 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 21 ==== osd_op_reply(28 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.241529 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:29 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.242567 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 22 ==== osd_op_reply(29 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.242619 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:30 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.243185 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 23 ==== osd_op_reply(30 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.243268 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:31 3.1eaca32f zonegroups_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d4f0 con 0x559e7aa65370
2018-01-12 23:39:46.243814 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 24 ==== osd_op_reply(31 zonegroups_names.default [getxattrs,stat] v0'0 uv2 ondisk = 0) v7 ==== 186+0+20 (3398099016 0 769767392) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.243868 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:32 3.1eaca32f zonegroups_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x559e7aa65370
2018-01-12 23:39:46.244396 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 25 ==== osd_op_reply(32 zonegroups_names.default [read 0~46] v0'0 uv2 ondisk = 0) v7 ==== 144+0+46 (520762147 0 994641691) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.244436 7f6195c1c9c0 10 ERROR: name default already in use for obj id 0356828d-f71f-4a85-8c71-8fb11aeb5683
2018-01-12 23:39:46.244439 7f6195c1c9c0 10 create_default() returned EEXIST, we raced with another zonegroup creation
2018-01-12 23:39:46.244464 7f6195c1c9c0 1 - 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:33 3.1eaca32f zonegroups_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.245019 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 26 ==== osd_op_reply(33 zonegroups_names.default [getxattrs,stat] v0'0 uv2 ondisk = 0) v7 ==== 186+0+20 (3398099016 0 769767392) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.245073 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:34 3.1eaca32f zonegroups_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.245709 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 27 ==== osd_op_reply(34 zonegroups_names.default [read 0~46] v0'0 uv2 ondisk = 0) v7 ==== 144+0+46 (520762147 0 994641691) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.245776 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:35 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x7f6154004930
2018-01-12 23:39:46.246349 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 6 ==== osd_op_reply(35 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 213+0+20 (4083675718 0 178235766) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.246402 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:36 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x7f6154004930
2018-01-12 23:39:46.246858 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 7 ==== osd_op_reply(36 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~324] v0'0 uv1 ondisk = 0) v7 ==== 171+0+324 (2897459508 0 2519818751) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.246934 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:37 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.247650 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 28 ==== osd_op_reply(37 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.247704 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:38 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x559e7aa65370
2018-01-12 23:39:46.248266 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 29 ==== osd_op_reply(38 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.248347 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:39 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x7f6154004930
2018-01-12 23:39:46.248835 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 8 ==== osd_op_reply(39 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 213+0+20 (4083675718 0 178235766) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.248888 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:40 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x7f6154004930
2018-01-12 23:39:46.249306 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 9 ==== osd_op_reply(40 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~324] v0'0 uv1 ondisk = 0) v7 ==== 171+0+324 (2897459508 0 2519818751) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.249356 7f6195c1c9c0 10 Cannot find current period zone using local zone
2018-01-12 23:39:46.249365 7f6195c1c9c0 10 Using default name default
2018-01-12 23:39:46.249391 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:41 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6cc30 con 0x559e7aa65370
2018-01-12 23:39:46.250025 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 30 ==== osd_op_reply(41 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.250078 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:42 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.250723 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 31 ==== osd_op_reply(42 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.250785 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:43 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.251348 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 32 ==== osd_op_reply(43 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.251403 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:44 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa6d090 con 0x559e7aa65370
2018-01-12 23:39:46.252129 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 33 ==== osd_op_reply(44 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.252266 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:45 4.4322fa9f notify.0 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa6efa0 con 0x559e7aa66ac0
2018-01-12 23:39:46.253248 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 3 ==== osd_op_reply(45 notify.0 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (2679103017 0 0) 0x7f614c002b90 con 0x559e7aa66ac0
2018-01-12 23:39:46.253313 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:46 4.4322fa9f notify.0 [watch watch cookie 94139150955520] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa6f750 con 0x559e7aa66ac0
2018-01-12 23:39:46.281524 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 4 ==== osd_op_reply(46 notify.0 [watch watch cookie 94139150955520] v265'35 uv1 ondisk = 0) v7 ==== 128+0+0 (4294242809 0 0) 0x7f614c002b90 con 0x559e7aa66ac0
2018-01-12 23:39:46.281583 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:47 4.16dafda0 notify.1 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa6f750 con 0x7f6154004930
2018-01-12 23:39:46.282094 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 10 ==== osd_op_reply(47 notify.1 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (2954800741 0 0) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.282152 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:48 4.16dafda0 notify.1 [watch watch cookie 94139150957904] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa700a0 con 0x7f6154004930
2018-01-12 23:39:46.293036 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 11 ==== osd_op_reply(48 notify.1 [watch watch cookie 94139150957904] v265'35 uv1 ondisk = 0) v7 ==== 128+0+0 (1840946043 0 0) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.293089 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:49 4.88aa5c95 notify.2 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa700a0 con 0x559e7aa65370
2018-01-12 23:39:46.293714 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 34 ==== osd_op_reply(49 notify.2 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3617160630 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.293764 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:50 4.88aa5c95 notify.2 [watch watch cookie 94139150960288] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa709f0 con 0x559e7aa65370
2018-01-12 23:39:46.308097 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 35 ==== osd_op_reply(50 notify.2 [watch watch cookie 94139150960288] v265'69 uv1 ondisk = 0) v7 ==== 128+0+0 (2833913002 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.308146 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:51 4.f8c99aee notify.3 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa709f0 con 0x559e7aa66ac0
2018-01-12 23:39:46.308728 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 5 ==== osd_op_reply(51 notify.3 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3896087742 0 0) 0x7f614c002b90 con 0x559e7aa66ac0
2018-01-12 23:39:46.308778 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:52 4.f8c99aee notify.3 [watch watch cookie 94139150962672] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa71340 con 0x559e7aa66ac0
2018-01-12 23:39:46.342290 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 6 ==== osd_op_reply(52 notify.3 [watch watch cookie 94139150962672] v265'35 uv1 ondisk = 0) v7 ==== 128+0+0 (3967860736 0 0) 0x7f614c002b90 con 0x559e7aa66ac0
2018-01-12 23:39:46.342401 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:53 4.a204812d notify.4 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa71340 con 0x559e7aa65370
2018-01-12 23:39:46.343332 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 36 ==== osd_op_reply(53 notify.4 [create 0~0] v0'0 uv3 ondisk = 0) v7 ==== 128+0+0 (838607157 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.343493 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:54 4.a204812d notify.4 [watch watch cookie 94139150965056] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa71c90 con 0x559e7aa65370
2018-01-12 23:39:46.365322 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 37 ==== osd_op_reply(54 notify.4 [watch watch cookie 94139150965056] v265'70 uv3 ondisk = 0) v7 ==== 128+0+0 (4231051436 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.365393 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:55 4.31099063 notify.5 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa71c90 con 0x559e7aa65370
2018-01-12 23:39:46.366320 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 38 ==== osd_op_reply(55 notify.5 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (105356080 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.366446 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:56 4.31099063 notify.5 [watch watch cookie 94139150967024] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa72440 con 0x559e7aa65370
2018-01-12 23:39:46.386050 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 39 ==== osd_op_reply(56 notify.5 [watch watch cookie 94139150967024] v265'36 uv1 ondisk = 0) v7 ==== 128+0+0 (1267730040 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.386124 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:57 4.97c520d4 notify.6 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa72440 con 0x559e7aa65370
2018-01-12 23:39:46.386683 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 40 ==== osd_op_reply(57 notify.6 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3036545333 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.386749 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:58 4.97c520d4 notify.6 [watch watch cookie 94139150969392] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa72440 con 0x559e7aa65370
2018-01-12 23:39:46.405297 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 41 ==== osd_op_reply(58 notify.6 [watch watch cookie 94139150969392] v265'35 uv1 ondisk = 0) v7 ==== 128+0+0 (4098374148 0 0) 0x7f61540016a0 con 0x559e7aa65370
2018-01-12 23:39:46.405370 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:59 4.84ada7c9 notify.7 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa72440 con 0x7f6154004930
2018-01-12 23:39:46.405878 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 12 ==== osd_op_reply(59 notify.7 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3944968614 0 0) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.405955 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:60 4.84ada7c9 notify.7 [watch watch cookie 94139150971392] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa73550 con 0x7f6154004930
2018-01-12 23:39:46.418732 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 13 ==== osd_op_reply(60 notify.7 [watch watch cookie 94139150971392] v265'35 uv1 ondisk = 0) v7 ==== 128+0+0 (2424101214 0 0) 0x7f61440015a0 con 0x7f6154004930
2018-01-12 23:39:46.418779 7f6195c1c9c0 2 all 8 watchers are set, enabling cache
2018-01-12 23:39:46.420277 7f6195c1c9c0 10 find_oldest_log_period returning empty cursor
2018-01-12 23:39:46.421520 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:61 6.990e66d8 gc.0 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x7f6154004930
2018-01-12 23:39:46.422076 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 14 ==== osd_op_reply(61 gc.0 [call] v0'0 uv503 ondisk = 0) v7 ==== 124+0+11 (1883365143 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.422130 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:62 6.6830fb87 gc.1 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x559e7aa66ac0
2018-01-12 23:39:46.423640 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 7 ==== osd_op_reply(62 gc.1 [call] v0'0 uv664 ondisk = 0) v7 ==== 124+0+11 (4029700444 0 1993775135) 0x7f614c001830 con 0x559e7aa66ac0
2018-01-12 23:39:46.423700 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:63 6.3b1b8c5b gc.2 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x7f6154004930
2018-01-12 23:39:46.424206 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 15 ==== osd_op_reply(63 gc.2 [call] v0'0 uv1014 ondisk = 0) v7 ==== 124+0+11 (4009701049 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.424254 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:64 6.7820b530 gc.3 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x7f6154004930
2018-01-12 23:39:46.424785 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 16 ==== osd_op_reply(64 gc.3 [call] v0'0 uv505 ondisk = 0) v7 ==== 124+0+11 (1158933749 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.424838 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:65 6.370bf669 gc.4 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x7f6154004930
2018-01-12 23:39:46.425362 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 17 ==== osd_op_reply(65 gc.4 [call] v0'0 uv173 ondisk = 0) v7 ==== 124+0+11 (29911682 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.425410 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:66 6.15a298d6 gc.5 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x559e7aa65370
2018-01-12 23:39:46.426783 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 42 ==== osd_op_reply(66 gc.5 [call] v0'0 uv1166 ondisk = 0) v7 ==== 124+0+11 (2246524045 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.426847 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:67 6.170a8acf gc.6 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x559e7aa66ac0
2018-01-12 23:39:46.428038 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 8 ==== osd_op_reply(67 gc.6 [call] v0'0 uv666 ondisk = 0) v7 ==== 124+0+11 (1884759338 0 1993775135) 0x7f614c001830 con 0x559e7aa66ac0
2018-01-12 23:39:46.428085 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:68 6.21d2251d gc.7 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x559e7aa65370
2018-01-12 23:39:46.429247 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 43 ==== osd_op_reply(68 gc.7 [call] v0'0 uv665 ondisk = 0) v7 ==== 124+0+11 (1517249890 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.429318 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:69 6.61b8e3df gc.8 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c4c0 con 0x559e7aa66ac0
2018-01-12 23:39:46.430698 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 9 ==== osd_op_reply(69 gc.8 [call] v0'0 uv671 ondisk = 0) v7 ==== 124+0+317 (513930980 0 2422894778) 0x7f614c001e50 con 0x559e7aa66ac0
2018-01-12 23:39:46.430758 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:70 6.3768e783 gc.9 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9d720 con 0x7f6154004930
2018-01-12 23:39:46.431324 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 18 ==== osd_op_reply(70 gc.9 [call] v0'0 uv1016 ondisk = 0) v7 ==== 124+0+11 (4264334908 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.431373 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:71 6.eaec33ee gc.10 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.432330 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 44 ==== osd_op_reply(71 gc.10 [call] v0'0 uv1168 ondisk = 0) v7 ==== 125+0+11 (2443685706 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.432377 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:72 6.6a0e9a15 gc.11 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.433174 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 45 ==== osd_op_reply(72 gc.11 [call] v0'0 uv667 ondisk = 0) v7 ==== 125+0+11 (3040810550 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.433230 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:73 6.a45046b8 gc.12 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x7f6154004930
2018-01-12 23:39:46.433713 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 19 ==== osd_op_reply(73 gc.12 [call] v0'0 uv507 ondisk = 0) v7 ==== 125+0+11 (948183447 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.433759 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:74 6.48504074 gc.13 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.434464 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 46 ==== osd_op_reply(74 gc.13 [call] v0'0 uv834 ondisk = 0) v7 ==== 125+0+11 (1194630701 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.434511 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:75 6.94de7ebe gc.14 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.435375 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 47 ==== osd_op_reply(75 gc.14 [call] v0'0 uv1170 ondisk = 0) v7 ==== 125+0+11 (2482818027 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.435439 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:76 6.685c6f7 gc.15 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa66ac0
2018-01-12 23:39:46.436060 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 10 ==== osd_op_reply(76 gc.15 [call] v0'0 uv670 ondisk = 0) v7 ==== 125+0+11 (1898269746 0 1993775135) 0x7f614c001e50 con 0x559e7aa66ac0
2018-01-12 23:39:46.436127 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:77 6.c2ff39f3 gc.16 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x7f6154004930
2018-01-12 23:39:46.436645 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 20 ==== osd_op_reply(77 gc.16 [call] v0'0 uv1018 ondisk = 0) v7 ==== 125+0+11 (2761537126 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.436690 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:78 6.fe80bd8c gc.17 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.437820 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 48 ==== osd_op_reply(78 gc.17 [call] v0'0 uv836 ondisk = 0) v7 ==== 125+0+11 (2472420855 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.437864 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:79 6.90752cb4 gc.18 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.438915 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 49 ==== osd_op_reply(79 gc.18 [call] v0'0 uv838 ondisk = 0) v7 ==== 125+0+11 (3386800825 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.438972 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:80 6.3abc5326 gc.19 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.439885 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 50 ==== osd_op_reply(80 gc.19 [call] v0'0 uv1172 ondisk = 0) v7 ==== 125+0+11 (1876763931 0 1993775135) 0x7f6154009690 con 0x559e7aa65370
2018-01-12 23:39:46.439930 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:81 6.9955331c gc.20 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.441561 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 51 ==== osd_op_reply(81 gc.20 [call] v0'0 uv841 ondisk = 0) v7 ==== 125+0+317 (2712409680 0 2861782348) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.441614 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:82 6.6dc01772 gc.21 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9ec30 con 0x7f6154004930
2018-01-12 23:39:46.442092 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 21 ==== osd_op_reply(82 gc.21 [call] v0'0 uv332 ondisk = 0) v7 ==== 125+0+11 (3833541100 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.442136 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:83 6.97748d0d gc.22 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.443458 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 52 ==== osd_op_reply(83 gc.22 [call] v0'0 uv669 ondisk = 0) v7 ==== 125+0+11 (1992283848 0 1993775135) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.443503 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:84 6.64f84676 gc.23 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.444549 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 53 ==== osd_op_reply(84 gc.23 [call] v0'0 uv1174 ondisk = 0) v7 ==== 125+0+11 (1475893854 0 1993775135) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.444593 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:85 6.8bdc9d gc.24 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.445149 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 54 ==== osd_op_reply(85 gc.24 [call] v0'0 uv671 ondisk = 0) v7 ==== 125+0+11 (481360272 0 1993775135) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.445193 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:86 6.51baec8a gc.25 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x7f6154004930
2018-01-12 23:39:46.445696 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 22 ==== osd_op_reply(86 gc.25 [call] v0'0 uv334 ondisk = 0) v7 ==== 125+0+11 (883504252 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.445743 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:87 6.386217e gc.26 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.446332 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 55 ==== osd_op_reply(87 gc.26 [call] v0'0 uv1176 ondisk = 0) v7 ==== 125+0+11 (643081722 0 1993775135) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.446377 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:88 6.eba9ff7b gc.27 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x7f6154004930
2018-01-12 23:39:46.446881 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 23 ==== osd_op_reply(88 gc.27 [call] v0'0 uv1020 ondisk = 0) v7 ==== 125+0+11 (1785052404 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.446926 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:89 6.4070ac6b gc.28 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x7f6154004930
2018-01-12 23:39:46.447419 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 24 ==== osd_op_reply(89 gc.28 [call] v0'0 uv1022 ondisk = 0) v7 ==== 125+0+11 (2282598246 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.447464 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:90 6.65639084 gc.29 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.448071 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 56 ==== osd_op_reply(90 gc.29 [call] v0'0 uv832 ondisk = 0) v7 ==== 125+0+11 (3062454349 0 1993775135) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.448115 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:91 6.36574d2e gc.30 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x559e7aa65370
2018-01-12 23:39:46.448808 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 57 ==== osd_op_reply(91 gc.30 [call] v0'0 uv1164 ondisk = 0) v7 ==== 125+0+11 (1311581852 0 1993775135) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.448853 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:92 6.1040a0a3 gc.31 [call rgw.gc_list] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x559e7aa9c540 con 0x7f6154004930
2018-01-12 23:39:46.449299 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 25 ==== osd_op_reply(92 gc.31 [call] v0'0 uv1012 ondisk = 0) v7 ==== 125+0+11 (2022937194 0 1993775135) 0x7f6144002d20 con 0x7f6154004930
[ {
"tag": "93541ddd-3a56-49fe-a109-83e09bcd7483.44309.2911\u0000",
"time": "2018-01-13 01:39:11.0.525349s",
"objs": [ {
"pool": "default.rgw.buckets.data",
"oid": "93541ddd-3a56-49fe-a109-83e09bcd7483.24794.1__shadow_.C02M8szXNkOcXnukPmq_dHE92_Fz1mu_1",
"key": "",
"instance": ""
}
]
}, {
"tag": "93541ddd-3a56-49fe-a109-83e09bcd7483.44309.2877\u0000",
"time": "2018-01-13 01:38:42.0.119527s",
"objs": [ {
"pool": "default.rgw.buckets.data",
"oid": "93541ddd-3a56-49fe-a109-83e09bcd7483.24794.1__shadow_.C02M8szXNkOcXnukPmq_dHE92_Fz1mu_1",
"key": "",
"instance": ""
}
]
}
]
2018-01-12 23:39:46.450336 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:93 4.4322fa9f notify.0 [watch unwatch cookie 94139150955520] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa74ff0 con 0x559e7aa66ac0
2018-01-12 23:39:46.475336 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 11 ==== osd_op_reply(93 notify.0 [watch unwatch cookie 94139150955520] v265'36 uv1 ondisk = 0) v7 ==== 128+0+0 (1869868457 0 0) 0x7f614c001e50 con 0x559e7aa66ac0
2018-01-12 23:39:46.475416 7f6195c1c9c0 2 removed watcher, disabling cache
2018-01-12 23:39:46.475471 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:94 4.16dafda0 notify.1 [watch unwatch cookie 94139150957904] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa75650 con 0x7f6154004930
2018-01-12 23:39:46.496157 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 26 ==== osd_op_reply(94 notify.1 [watch unwatch cookie 94139150957904] v265'36 uv1 ondisk = 0) v7 ==== 128+0+0 (4248667947 0 0) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.496221 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:95 4.88aa5c95 notify.2 [watch unwatch cookie 94139150960288] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa75980 con 0x559e7aa65370
2018-01-12 23:39:46.513793 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 58 ==== osd_op_reply(95 notify.2 [watch unwatch cookie 94139150960288] v265'71 uv1 ondisk = 0) v7 ==== 128+0+0 (21922467 0 0) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.513848 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.158:6804/3586 -- osd_op(client.94113.0:96 4.f8c99aee notify.3 [watch unwatch cookie 94139150962672] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa75fe0 con 0x559e7aa66ac0
2018-01-12 23:39:46.538716 7f61714f7700 1 -- 10.134.238.152:0/1325339148 <== osd.4 10.134.238.158:6804/3586 12 ==== osd_op_reply(96 notify.3 [watch unwatch cookie 94139150962672] v265'36 uv1 ondisk = 0) v7 ==== 128+0+0 (2080889936 0 0) 0x7f614c001e50 con 0x559e7aa66ac0
2018-01-12 23:39:46.538773 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:97 4.a204812d notify.4 [watch unwatch cookie 94139150965056] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa76640 con 0x559e7aa65370
2018-01-12 23:39:46.562775 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 59 ==== osd_op_reply(97 notify.4 [watch unwatch cookie 94139150965056] v265'72 uv3 ondisk = 0) v7 ==== 128+0+0 (166606457 0 0) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.562835 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:98 4.31099063 notify.5 [watch unwatch cookie 94139150967024] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa76ca0 con 0x559e7aa65370
2018-01-12 23:39:46.582550 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 60 ==== osd_op_reply(98 notify.5 [watch unwatch cookie 94139150967024] v265'37 uv1 ondisk = 0) v7 ==== 128+0+0 (4113176134 0 0) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.582605 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.159:6800/11592 -- osd_op(client.94113.0:99 4.97c520d4 notify.6 [watch unwatch cookie 94139150969392] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa77300 con 0x559e7aa65370
2018-01-12 23:39:46.608002 7f61716f9700 1 -- 10.134.238.152:0/1325339148 <== osd.5 10.134.238.159:6800/11592 61 ==== osd_op_reply(99 notify.6 [watch unwatch cookie 94139150969392] v265'36 uv1 ondisk = 0) v7 ==== 128+0+0 (1691297364 0 0) 0x7f615400a6d0 con 0x559e7aa65370
2018-01-12 23:39:46.608058 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 --> 10.134.238.157:6804/17168 -- osd_op(client.94113.0:100 4.84ada7c9 notify.7 [watch unwatch cookie 94139150971392] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x559e7aa77960 con 0x7f6154004930
2018-01-12 23:39:46.624277 7f61712f5700 1 -- 10.134.238.152:0/1325339148 <== osd.3 10.134.238.157:6804/17168 27 ==== osd_op_reply(100 notify.7 [watch unwatch cookie 94139150971392] v265'36 uv1 ondisk = 0) v7 ==== 128+0+0 (16499982 0 0) 0x7f6144002d20 con 0x7f6154004930
2018-01-12 23:39:46.625103 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 mark_down 0x7f6154004930 -- 0x7f6154003670
2018-01-12 23:39:46.625148 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 mark_down 0x559e7aa66ac0 -- 0x559e7aa68670
2018-01-12 23:39:46.625177 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 mark_down 0x559e7aa65370 -- 0x559e7aa64060
2018-01-12 23:39:46.625263 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 mark_down 0x559e7aa5cae0 -- 0x559e7aa58710
2018-01-12 23:39:46.625653 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 mark_down_all
2018-01-12 23:39:46.625825 7f6195c1c9c0 1 -- 10.134.238.152:0/1325339148 shutdown complete.
[root@ceph-rgw ~]#

Actions

Copy link

Updated by Matt Benjamin over 6 years ago

@fpq f, I don't see any http_status=xxx lines in the new log output (though you do in your original text); we need to see the log output that corresponds to an attempt to remove the object that can't be deleted.

Actions

Copy link

Updated by fpq f over 6 years ago

File client.radosgw.gateway.log.txt client.radosgw.gateway.log.txt added

@Matt Li Benjamin

Actions

Copy link

Updated by Matt Benjamin over 6 years ago

@fpq f, sorry, this is CLOSE--what I'm looking for is that you run radosgw with "-d --debug-rgw=18 --debug-ms=1" and capture the log, doing the delete op which fails;

Matt

Actions

Copy link

Updated by fpq f over 6 years ago

File test.txt test.txt added

@Matt Li Benjamin like this？

Actions

Copy link

Updated by fpq f over 6 years ago

@Matt Li Benjamin cloud you help me
[root@ceph-rgw ~]# radosgw d --debug-rgw=18 --debug-ms=1 --name client.radosgw.gateway
2018-01-16 21:00:33.481282 7fbd034e09c0 0 ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374), process radosgw, pid 9406
2018-01-16 21:00:33.485362 7fbd034e09c0 -1 asok(0x55aa4d6b9e40) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/ceph-client.radosgw.gateway.asok': (17) File exists
2018-01-16 21:00:33.487024 7fbd034e09c0 1 - :/0 messenger.start
2018-01-16 21:00:33.487452 7fbd034e09c0 1 -- :/3412370064 --> 10.134.238.159:6789/0 -- auth(proto 0 40 bytes epoch 0) v1 -- ?+0 0x55aa4d71b970 con 0x55aa4d71b390
2018-01-16 21:00:33.489297 7fbd034d6700 1 -- 10.134.238.152:0/3412370064 learned my addr 10.134.238.152:0/3412370064
2018-01-16 21:00:33.490520 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 1 ==== mon_map magic: 0 v1 ==== 500+0+0 (3223744036 0 0) 0x7fbcd8000b90 con 0x55aa4d71b390
2018-01-16 21:00:33.490614 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (2969333790 0 0) 0x7fbcd80008c0 con 0x55aa4d71b390
2018-01-16 21:00:33.490733 7fbce2887700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6789/0 -- auth(proto 2 32 bytes epoch 0) v1 -- ?+0 0x7fbcd4001880 con 0x55aa4d71b390
2018-01-16 21:00:33.491562 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 3 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 222+0+0 (4237805047 0 0) 0x7fbcd8000a50 con 0x55aa4d71b390
2018-01-16 21:00:33.491667 7fbce2887700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6789/0 -- auth(proto 2 181 bytes epoch 0) v1 -- ?+0 0x7fbcd4003390 con 0x55aa4d71b390
2018-01-16 21:00:33.492756 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 425+0+0 (217033238 0 0) 0x7fbcd8000e90 con 0x55aa4d71b390
2018-01-16 21:00:33.492839 7fbce2887700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0x55aa4d755800 con 0x55aa4d71b390
2018-01-16 21:00:33.492912 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6789/0 -- mon_subscribe({osdmap=0}) v2 -- ?+0 0x55aa4d754700 con 0x55aa4d71b390
2018-01-16 21:00:33.493380 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 5 ==== mon_map magic: 0 v1 ==== 500+0+0 (3223744036 0 0) 0x7fbcd8000e90 con 0x55aa4d71b390
2018-01-16 21:00:33.493464 7fbcd2ffd700 2 RGWDataChangesLog::ChangesRenewThread: start
2018-01-16 21:00:33.494284 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 6 ==== osd_map(265..265 src has 1..265) v3 ==== 8305+0+0 (896830133 0 0) 0x7fbcd8002f80 con 0x55aa4d71b390
2018-01-16 21:00:33.494566 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:1 3.85fca992 default.realm [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75b150 con 0x55aa4d759c30
2018-01-16 21:00:33.497039 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 1 ==== osd_op_reply(1 default.realm [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 175+0+0 (2283260449 0 0) 0x7fbcc8000a50 con 0x55aa4d759c30
2018-01-16 21:00:33.497182 7fbd034e09c0 1 - 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:2 3.85fca992 default.realm [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75b870 con 0x55aa4d759c30
2018-01-16 21:00:33.498337 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 2 ==== osd_op_reply(2 default.realm [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 175+0+0 (2283260449 0 0) 0x7fbcc8000a50 con 0x55aa4d759c30
2018-01-16 21:00:33.498407 7fbd034e09c0 10 could not read realm id: (2) No such file or directory
2018-01-16 21:00:33.498513 7fbd034e09c0 1 - 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:3 3.0 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75f490 con 0x55aa4d75e050
2018-01-16 21:00:33.501468 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 1 ==== osd_op_reply(3 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (73087745 0 3248547820) 0x7fbcc0000b50 con 0x55aa4d75e050
2018-01-16 21:00:33.501513 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:4 3.1 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc0002580 con 0x55aa4d759c30
2018-01-16 21:00:33.502596 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 3 ==== osd_op_reply(4 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2239183445 0 3248547820) 0x7fbcc8000a50 con 0x55aa4d759c30
2018-01-16 21:00:33.502634 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:5 3.2 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc80029e0 con 0x55aa4d759c30
2018-01-16 21:00:33.503629 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 4 ==== osd_op_reply(5 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (66015064 0 3248547820) 0x7fbcc8000a50 con 0x55aa4d759c30
2018-01-16 21:00:33.503712 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:6 3.3 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc8005d70 con 0x7fbcc8004930
2018-01-16 21:00:33.505142 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 1 ==== osd_op_reply(6 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+103 (2535676660 0 4170302097) 0x7fbcb8000b80 con 0x7fbcc8004930
2018-01-16 21:00:33.505231 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:7 3.4 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75f8d0 con 0x7fbcc8004930
2018-01-16 21:00:33.505701 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 2 ==== osd_op_reply(7 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (187945907 0 3248547820) 0x7fbcb8000b80 con 0x7fbcc8004930
2018-01-16 21:00:33.505740 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:8 3.5 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcb80027f0 con 0x55aa4d759c30
2018-01-16 21:00:33.507468 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 5 ==== osd_op_reply(8 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2317341415 0 3248547820) 0x7fbcc8000a50 con 0x55aa4d759c30
2018-01-16 21:00:33.507507 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:9 3.6 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc80062a0 con 0x55aa4d759c30
2018-01-16 21:00:33.508924 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 6 ==== osd_op_reply(9 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+98 (777405040 0 661761295) 0x7fbcc80062a0 con 0x55aa4d759c30
2018-01-16 21:00:33.508994 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:10 3.7 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75f8d0 con 0x55aa4d759c30
2018-01-16 21:00:33.510002 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 7 ==== osd_op_reply(10 [pgls start_epoch 0] v75'2 uv2 ondisk = 1) v7 ==== 120+0+102 (3625854734 0 37397930) 0x7fbcc80062a0 con 0x55aa4d759c30
2018-01-16 21:00:33.510108 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:11 3.bd31b503 region_map [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75b870 con 0x7fbcc8004930
2018-01-16 21:00:33.510528 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 3 ==== osd_op_reply(11 region_map [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 172+0+0 (832903909 0 0) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.510597 7fbd034e09c0 10 cannot find current period zonegroup using local zonegroup
2018-01-16 21:00:33.510637 7fbd034e09c0 1 - 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:12 3.85fca992 default.realm [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75b870 con 0x55aa4d759c30
2018-01-16 21:00:33.511600 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 8 ==== osd_op_reply(12 default.realm [getxattrs,stat] v0'0 uv0 ack = 2 ((2) No such file or directory)) v7 ==== 175+0+0 (2283260449 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.511664 7fbd034e09c0 10 could not read realm id: (2) No such file or directory
2018-01-16 21:00:33.511673 7fbd034e09c0 10 Creating default zonegroup
2018-01-16 21:00:33.511723 7fbd034e09c0 1 - 10.134.238.152:0/3412370064 --> 10.134.238.159:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- ?+0 0x55aa4d75f8d0 con 0x55aa4d71b390
2018-01-16 21:00:33.512503 7fbce2887700 1 -- 10.134.238.152:0/3412370064 <== mon.2 10.134.238.159:6789/0 7 ==== mon_get_version_reply(handle=1 version=265) v2 ==== 24+0+0 (4011090033 0 0) 0x7fbcd8000f00 con 0x55aa4d71b390
2018-01-16 21:00:33.512596 7fbd034e09c0 10 couldn't find old data placement pools config, setting up new ones for the zone
2018-01-16 21:00:33.512645 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:13 3.0 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75c040 con 0x55aa4d75e050
2018-01-16 21:00:33.513657 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 2 ==== osd_op_reply(13 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (73087745 0 3248547820) 0x7fbcc0000b50 con 0x55aa4d75e050
2018-01-16 21:00:33.513693 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:14 3.1 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc0002b90 con 0x55aa4d759c30
2018-01-16 21:00:33.514807 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 9 ==== osd_op_reply(14 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2239183445 0 3248547820) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.514844 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:15 3.2 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc8001e70 con 0x55aa4d759c30
2018-01-16 21:00:33.515923 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 10 ==== osd_op_reply(15 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (66015064 0 3248547820) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.515955 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:16 3.3 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc80023f0 con 0x7fbcc8004930
2018-01-16 21:00:33.516494 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 4 ==== osd_op_reply(16 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+103 (2535676660 0 4170302097) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.516562 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:17 3.4 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75c040 con 0x7fbcc8004930
2018-01-16 21:00:33.517065 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 5 ==== osd_op_reply(17 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (187945907 0 3248547820) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.517104 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:18 3.5 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcb8002040 con 0x55aa4d759c30
2018-01-16 21:00:33.518088 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 11 ==== osd_op_reply(18 [pgls start_epoch 0] v0'0 uv0 ondisk = 1) v7 ==== 120+0+44 (2317341415 0 3248547820) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.518124 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:19 3.6 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x7fbcc80023c0 con 0x55aa4d759c30
2018-01-16 21:00:33.519182 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 12 ==== osd_op_reply(19 [pgls start_epoch 0] v75'1 uv1 ondisk = 1) v7 ==== 120+0+98 (777405040 0 661761295) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.519250 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:20 3.7 [pgls start_epoch 0] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75c040 con 0x55aa4d759c30
2018-01-16 21:00:33.520264 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 13 ==== osd_op_reply(20 [pgls start_epoch 0] v75'2 uv2 ondisk = 1) v7 ==== 120+0+102 (3625854734 0 37397930) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.520366 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:21 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d75c040 con 0x55aa4d759c30
2018-01-16 21:00:33.521343 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 14 ==== osd_op_reply(21 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.521433 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:22 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d7578f0 con 0x55aa4d759c30
2018-01-16 21:00:33.522479 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 15 ==== osd_op_reply(22 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.522576 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:23 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d757890 con 0x55aa4d759c30
2018-01-16 21:00:33.523596 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 16 ==== osd_op_reply(23 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.523671 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:24 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d7601d0 con 0x55aa4d759c30
2018-01-16 21:00:33.524642 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 17 ==== osd_op_reply(24 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.524803 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:25 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d760e90 con 0x55aa4d759c30
2018-01-16 21:00:33.525854 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 18 ==== osd_op_reply(25 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.525926 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:26 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d760e90 con 0x55aa4d759c30
2018-01-16 21:00:33.526977 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 19 ==== osd_op_reply(26 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.527038 7fbd034e09c0 10 ERROR: name default already in use for obj id 93541ddd-3a56-49fe-a109-83e09bcd7483
2018-01-16 21:00:33.527044 7fbd034e09c0 0 RGWZoneParams::create(): error creating default zone params: (17) File exists
2018-01-16 21:00:33.527050 7fbd034e09c0 10 zone_params::create_default() returned EEXIST, we raced with another default zone_params creation
2018-01-16 21:00:33.527076 7fbd034e09c0 1 - 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:27 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d7612f0 con 0x55aa4d759c30
2018-01-16 21:00:33.528118 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 20 ==== osd_op_reply(27 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.528184 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:28 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d760e90 con 0x55aa4d759c30
2018-01-16 21:00:33.529167 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 21 ==== osd_op_reply(28 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.529248 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:29 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d760e90 con 0x55aa4d759c30
2018-01-16 21:00:33.530317 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 22 ==== osd_op_reply(29 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.530387 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:30 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d760e90 con 0x55aa4d759c30
2018-01-16 21:00:33.531389 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 23 ==== osd_op_reply(30 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.531496 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:31 3.1eaca32f zonegroups_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761c90 con 0x55aa4d759c30
2018-01-16 21:00:33.532620 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 24 ==== osd_op_reply(31 zonegroups_names.default [getxattrs,stat] v0'0 uv2 ondisk = 0) v7 ==== 186+0+20 (3398099016 0 769767392) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.532726 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:32 3.1eaca32f zonegroups_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761c90 con 0x55aa4d759c30
2018-01-16 21:00:33.533679 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 25 ==== osd_op_reply(32 zonegroups_names.default [read 0~46] v0'0 uv2 ondisk = 0) v7 ==== 144+0+46 (520762147 0 994641691) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.533760 7fbd034e09c0 10 ERROR: name default already in use for obj id 0356828d-f71f-4a85-8c71-8fb11aeb5683
2018-01-16 21:00:33.533765 7fbd034e09c0 10 create_default() returned EEXIST, we raced with another zonegroup creation
2018-01-16 21:00:33.533791 7fbd034e09c0 1 - 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:33 3.1eaca32f zonegroups_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d762460 con 0x55aa4d759c30
2018-01-16 21:00:33.534684 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 26 ==== osd_op_reply(33 zonegroups_names.default [getxattrs,stat] v0'0 uv2 ondisk = 0) v7 ==== 186+0+20 (3398099016 0 769767392) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.534766 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:34 3.1eaca32f zonegroups_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761c90 con 0x55aa4d759c30
2018-01-16 21:00:33.535829 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 27 ==== osd_op_reply(34 zonegroups_names.default [read 0~46] v0'0 uv2 ondisk = 0) v7 ==== 144+0+46 (520762147 0 994641691) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.535931 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:35 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d762a60 con 0x7fbcc8004930
2018-01-16 21:00:33.536753 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 6 ==== osd_op_reply(35 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 213+0+20 (4083675718 0 178235766) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.536850 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:36 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d762460 con 0x7fbcc8004930
2018-01-16 21:00:33.537526 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 7 ==== osd_op_reply(36 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~324] v0'0 uv1 ondisk = 0) v7 ==== 171+0+324 (2897459508 0 2519818751) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.537646 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:37 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x55aa4d759c30
2018-01-16 21:00:33.538784 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 28 ==== osd_op_reply(37 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.538884 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:38 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x55aa4d759c30
2018-01-16 21:00:33.539991 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 29 ==== osd_op_reply(38 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.540137 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:39 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x7fbcc8004930
2018-01-16 21:00:33.540907 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 8 ==== osd_op_reply(39 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 213+0+20 (4083675718 0 178235766) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.541022 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:40 3.e7017213 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x7fbcc8004930
2018-01-16 21:00:33.541673 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 9 ==== osd_op_reply(40 zonegroup_info.0356828d-f71f-4a85-8c71-8fb11aeb5683 [read 0~324] v0'0 uv1 ondisk = 0) v7 ==== 171+0+324 (2897459508 0 2519818751) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.541773 7fbd034e09c0 10 Cannot find current period zone using local zone
2018-01-16 21:00:33.541779 7fbd034e09c0 10 Using default name default
2018-01-16 21:00:33.541804 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:41 3.7376dc2f zone_names.default [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x55aa4d759c30
2018-01-16 21:00:33.542801 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 30 ==== osd_op_reply(41 zone_names.default [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 180+0+20 (2828502694 0 2350394807) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.542899 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:42 3.7376dc2f zone_names.default [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x55aa4d759c30
2018-01-16 21:00:33.543896 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 31 ==== osd_op_reply(42 zone_names.default [read 0~46] v0'0 uv1 ondisk = 0) v7 ==== 138+0+46 (2493670163 0 3868786191) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.544001 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:43 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x55aa4d759c30
2018-01-16 21:00:33.544994 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 32 ==== osd_op_reply(43 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [getxattrs,stat] v0'0 uv1 ondisk = 0) v7 ==== 208+0+20 (1890529474 0 2877186962) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.545102 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:44 3.b3a6a33e zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~524288] snapc 0=[] ack+read+known_if_redirected e265) v7 -- ?+0 0x55aa4d761e40 con 0x55aa4d759c30
2018-01-16 21:00:33.546142 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 33 ==== osd_op_reply(44 zone_info.93541ddd-3a56-49fe-a109-83e09bcd7483 [read 0~1220] v0'0 uv1 ondisk = 0) v7 ==== 166+0+1220 (3743396430 0 3852199067) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.546341 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:45 4.4322fa9f notify.0 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d763930 con 0x55aa4d75e050
2018-01-16 21:00:33.547508 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 3 ==== osd_op_reply(45 notify.0 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (2679103017 0 0) 0x7fbcc0002b90 con 0x55aa4d75e050
2018-01-16 21:00:33.547645 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:46 4.4322fa9f notify.0 [watch watch cookie 94189932382016] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d763fd0 con 0x55aa4d75e050
2018-01-16 21:00:33.577397 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 4 ==== osd_op_reply(46 notify.0 [watch watch cookie 94189932382016] v265'43 uv1 ondisk = 0) v7 ==== 128+0+0 (3703752419 0 0) 0x7fbcc0002b90 con 0x55aa4d75e050
2018-01-16 21:00:33.577506 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:47 4.16dafda0 notify.1 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d763fd0 con 0x7fbcc8004930
2018-01-16 21:00:33.578014 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 10 ==== osd_op_reply(47 notify.1 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (2954800741 0 0) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.578069 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:48 4.16dafda0 notify.1 [watch watch cookie 94189932395920] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d764900 con 0x7fbcc8004930
2018-01-16 21:00:33.596286 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 11 ==== osd_op_reply(48 notify.1 [watch watch cookie 94189932395920] v265'43 uv1 ondisk = 0) v7 ==== 128+0+0 (2922030704 0 0) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.596416 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:49 4.88aa5c95 notify.2 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d764900 con 0x55aa4d759c30
2018-01-16 21:00:33.597506 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 34 ==== osd_op_reply(49 notify.2 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3617160630 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.597600 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:50 4.88aa5c95 notify.2 [watch watch cookie 94189932398272] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d765140 con 0x55aa4d759c30
2018-01-16 21:00:33.613392 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 35 ==== osd_op_reply(50 notify.2 [watch watch cookie 94189932398272] v265'85 uv1 ondisk = 0) v7 ==== 128+0+0 (716529099 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.613474 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:51 4.f8c99aee notify.3 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d765140 con 0x55aa4d75e050
2018-01-16 21:00:33.614516 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 5 ==== osd_op_reply(51 notify.3 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3896087742 0 0) 0x7fbcc0002b90 con 0x55aa4d75e050
2018-01-16 21:00:33.614595 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:52 4.f8c99aee notify.3 [watch watch cookie 94189932400384] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d7659f0 con 0x55aa4d75e050
2018-01-16 21:00:33.628666 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 6 ==== osd_op_reply(52 notify.3 [watch watch cookie 94189932400384] v265'43 uv1 ondisk = 0) v7 ==== 128+0+0 (3679361280 0 0) 0x7fbcc0002b90 con 0x55aa4d75e050
2018-01-16 21:00:33.628777 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:53 4.a204812d notify.4 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d7659f0 con 0x55aa4d759c30
2018-01-16 21:00:33.629405 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 36 ==== osd_op_reply(53 notify.4 [create 0~0] v0'0 uv3 ondisk = 0) v7 ==== 128+0+0 (838607157 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.629492 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:54 4.a204812d notify.4 [watch watch cookie 94189932402608] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d766320 con 0x55aa4d759c30
2018-01-16 21:00:33.642070 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 37 ==== osd_op_reply(54 notify.4 [watch watch cookie 94189932402608] v265'86 uv3 ondisk = 0) v7 ==== 128+0+0 (3902195444 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.642155 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:55 4.31099063 notify.5 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d766320 con 0x55aa4d759c30
2018-01-16 21:00:33.643219 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 38 ==== osd_op_reply(55 notify.5 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (105356080 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.643297 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:56 4.31099063 notify.5 [watch watch cookie 94189932404960] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d766c50 con 0x55aa4d759c30
2018-01-16 21:00:33.659017 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 39 ==== osd_op_reply(56 notify.5 [watch watch cookie 94189932404960] v265'44 uv1 ondisk = 0) v7 ==== 128+0+0 (1978113558 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.659106 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:57 4.97c520d4 notify.6 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d766c50 con 0x55aa4d759c30
2018-01-16 21:00:33.660148 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 40 ==== osd_op_reply(57 notify.6 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3036545333 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.660227 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:58 4.97c520d4 notify.6 [watch watch cookie 94189932407312] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d767570 con 0x55aa4d759c30
2018-01-16 21:00:33.687585 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 41 ==== osd_op_reply(58 notify.6 [watch watch cookie 94189932407312] v265'43 uv1 ondisk = 0) v7 ==== 128+0+0 (268471567 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.687665 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:59 4.84ada7c9 notify.7 [create 0~0] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d767570 con 0x7fbcc8004930
2018-01-16 21:00:33.688499 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 12 ==== osd_op_reply(59 notify.7 [create 0~0] v0'0 uv1 ondisk = 0) v7 ==== 128+0+0 (3944968614 0 0) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.688590 7fbd034e09c0 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:60 4.84ada7c9 notify.7 [watch watch cookie 94189932409648] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x55aa4d767e80 con 0x7fbcc8004930
2018-01-16 21:00:33.704814 7fbcd22f7700 1 -- 10.134.238.152:0/3412370064 <== osd.3 10.134.238.157:6804/17168 13 ==== osd_op_reply(60 notify.7 [watch watch cookie 94189932409648] v265'43 uv1 ondisk = 0) v7 ==== 128+0+0 (3032547970 0 0) 0x7fbcb80015a0 con 0x7fbcc8004930
2018-01-16 21:00:33.704887 7fbd034e09c0 2 all 8 watchers are set, enabling cache
2018-01-16 21:00:33.705023 7fbcd19f5700 2 garbage collection: start
2018-01-16 21:00:33.705435 7fbcd19f5700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:61 6.c2ff39f3 gc.16 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x7fbcb0002fb0 con 0x7fbcc8004930
2018-01-16 21:00:33.705514 7fbcd11f4700 2 object expiration: start
2018-01-16 21:00:33.705664 7fbcd11f4700 1 -- 10.134.238.152:0/3412370064 --> 10.134.238.159:6800/11592 -- osd_op(client.74204.0:62 7.9a32ce58 obj_delete_at_hint.0000000000 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x7fbca4003000 con 0x55aa4d759c30
2018-01-16 21:00:33.706537 7fbd034e09c0 10 find_oldest_log_period returning empty cursor
2018-01-16 21:00:33.706937 7fbcd26fb700 1 -- 10.134.238.152:0/3412370064 <== osd.5 10.134.238.159:6800/11592 42 ==== osd_op_reply(62 obj_delete_at_hint.0000000000 [call] v0'0 uv0 ondisk = 16 ((16) Device or resource busy)) v7 ==== 149+0+0 (2015788343 0 0) 0x7fbcc80016a0 con 0x55aa4d759c30
2018-01-16 21:00:33.707007 7fbcd11f4700 5 process_single_shard(): failed to acquire lock on obj_delete_at_hint.0000000000
2018-01-16 21:00:33.707034 7fbcd11f4700 1 - 10.134.238.152:0/3412370064 --> 10.134.238.158:6804/3586 -- osd_op(client.74204.0:63 7.5dc198f obj_delete_at_hint.0000000001 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x7fbca40035e0 con 0x55aa4d75e050
2018-01-16 21:00:33.707301 7fbc8ffff700 10 ERROR: can't get key: ret=-2
2018-01-16 21:00:33.707310 7fbc8ffff700 5 ERROR: sync_all_users() returned ret=-2
2018-01-16 21:00:33.708042 7fbd034e09c0 0 starting handler: civetweb
2018-01-16 21:00:33.708443 7fbcd24f9700 1 -- 10.134.238.152:0/3412370064 <== osd.4 10.134.238.158:6804/3586 7 ==== osd_op_reply(63 obj_delete_at_hint.0000000001 [call] v0'0 uv0 ondisk = 16 ((16) Device or resource busy)) v7 ==== 149+0+0 (3998785031 0 0) 0x7fbcc0001680 con 0x55aa4d75e050
2018-01-16 21:00:33.708483 7fbcd11f4700 5 process_single_shard(): failed to acquire lock on obj_delete_at_hint.0000000001
2018-01-16 21:00:33.708499 7fbd034e09c0 0 civetweb: 0x55aa4d2dcbe0: set_ports_option: cannot bind to 7480: 98 (Address already in use)
2018-01-16 21:00:33.708517 7fbd034e09c0 -1 ERROR: failed run
2018-01-16 21:00:33.708509 7fbcd11f4700 1 - 10.134.238.152:0/3412370064 --> 10.134.238.157:6804/17168 -- osd_op(client.74204.0:64 7.68795c0e obj_delete_at_hint.0000000002 [call lock.lock] snapc 0=[] ondisk+write+known_if_redirected e265) v7 -- ?+0 0x7fbca40035e0 con 0x7fbcc8004930
[root@ceph-rgw ~]#

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » rgw

Custom queries

Bug #22617

ceph rgw can't delete some object

Updated by Matt Benjamin over 6 years ago

Updated by fpq f over 6 years ago

Updated by Matt Benjamin over 6 years ago

Updated by fpq f over 6 years ago

Updated by Matt Benjamin over 6 years ago

Updated by fpq f over 6 years ago

Updated by fpq f over 6 years ago