Ceph » rgw

We have two Ceph clusters within the same zonegroup: siteA and siteB. Both running v14.2.2.

Each site is configured with a pubsub zone: siteApubsub and siteBpubsub.

We've configured a notification, topic, and subscription, and are seeing events appear in the subscription list, but often hours/days after the S3 PUT request that triggered them. We are currently not acking any events, so we've built up a long list, but events are expiring after 7 days as configured.

e.g.

[ceph-deploy@ceph-walesa-node-0 ~]$ sudo radosgw-admin pubsub sub pull --sub-name walessub --rgw-zone siteApubsub --uid wales
... {
"id": "1567544050.658641.b6c40bf0",
"event": "OBJECT_CREATE",
"timestamp": "2019-09-03 20:54:10.658641Z",
"info": {
"attrs": {
"mtime": "2019-09-03 16:53:27.196689Z"
},
"bucket": {
"bucket_id": "57e997aa-b6c1-493f-9990-08e65fbb9b12.4980.1",
"name": "wales-scsdata",
"tenant": ""
},
"key": {
"instance": "KEmn2ViD5j24858xI5rX3WILEzE-v3x",
"name": "95/44/2019090316532779E95146780000-Subscriber"
}
}
},

See the four hour delay between info.attrs.mtime (when the S3 request occurred) and timestamp (when the event appeared on the subscription list). We've seen delays of multiple days for some events. Events seem to be sent (i.e. added to the subscription list) in batches; pulling out just the timestamps/mtime for events in the list shows short bursts of adding events relating to a backlog of S3 requests.

e.g.

timestamp mtime
03/09/19 17:17:45 29/08/19 15:50:12
03/09/19 17:17:45 30/08/19 09:38:51
03/09/19 17:18:25 30/08/19 09:29:53
03/09/19 17:18:25 30/08/19 23:38:45
03/09/19 20:54:10 30/08/19 23:38:45
03/09/19 20:54:10 30/08/19 23:38:45
03/09/19 20:54:10 30/08/19 23:38:45
03/09/19 20:54:10 30/08/19 23:38:45
03/09/19 20:54:10 30/08/19 23:38:44
03/09/19 20:54:10 30/08/19 23:07:02
03/09/19 20:54:10 30/08/19 23:07:00
03/09/19 20:54:10 30/08/19 23:07:00
03/09/19 20:54:11 30/08/19 23:06:59
03/09/19 20:54:11 30/08/19 23:06:59
03/09/19 20:54:11 30/08/19 23:06:58
03/09/19 20:54:11 03/09/19 17:09:28
03/09/19 20:54:11 03/09/19 16:53:27
03/09/19 20:55:24 03/09/19 17:16:09
03/09/19 20:55:24 03/09/19 17:18:10
03/09/19 20:55:24 03/09/19 20:53:50
03/09/19 22:18:38 03/09/19 22:18:36
03/09/19 22:18:38 03/09/19 22:18:37

Pubsub config:

[ceph-deploy@ceph-walesa-node-0 ~]$ sudo radosgw-admin user info --uid wales {
"user_id": "wales",
"display_name": "Wales EAS S3 user",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"subusers": [],
"keys": [ {
"user": "wales",
"access_key": "RZVTHZE7CH1RA2ZTOBCM",
"secret_key": "eO1silaIm5p8gNaDIcVANvu3csklU3VbCCVJtESX"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"default_storage_class": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw",
"mfa_ids": []
}

[ceph-deploy@ceph-walesa-node-0 ~]$ sudo radosgw-admin pubsub topics list --uid wales --rgw-zone siteApubsub {
"topics": [ {
"topic": {
"user": "wales",
"name": "walestopic"
},
"subs": [
"walessub"
]
}
]
}

[ceph-deploy@ceph-walesa-node-0 ~]$ sudo radosgw-admin pubsub sub get --sub-name walessub --uid wales --rgw-zone siteApubsub {
"user": "wales",
"name": "walessub",
"topic": "walestopic",
"dest": {
"bucket_name": "pubsub-wales-walestopic",
"oid_prefix": "",
"push_endpoint": "http://172.21.6.3:8103/s3mgr/"
}
}