Actions
Bug #59577
opensync notifications has object size of zero
Status:
Pending Backport
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
multisite, notification backport_processed
Backport:
reef
Regression:
No
Severity:
3 - minor
Reviewed:
Description
how to reproduce:
run an http POST server on port 10900
run the following commands:
MON=1 OSD=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 2 export AWS_ACCESS_KEY_ID=1234567890 export AWS_SECRET_ACCESS_KEY=pencil aws --region=zg1 --endpoint-url http://localhost:8201 sns create-topic --name=fishtopic --attributes='{"push-endpoint": "http://localhost:10900"}' aws --region=zg1 --endpoint-url http://localhost:8101 s3 mb s3://fish aws --region=zg1 --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket fish --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": ["s3:ObjectSynced:Create"]}]}' head -c 512 </dev/urandom > myfile aws --region=zg1 --endpoint-url http://localhost:8101 s3 cp myfile s3://fish
Updated by Yuval Lifshitz about 1 year ago
comparing sync notifications vs. regular notifications.
regular:
{ [40/57] "Records": [ { "eventVersion": "2.2", "eventSource": "ceph:s3", "awsRegion": "zg1", "eventTime": "2023-04-27T15:40:42.321565Z", "eventName": "ObjectCreated:Put", "userIdentity": { "principalId": "zone.user" }, "requestParameters": { "sourceIPAddress": "" }, "responseElements": { "x-amz-request-id": "9f96b2e9-955a-405d-a6c5-0c8d4075f403.4158.14554084588647464963", "x-amz-id-2": "103e-zg1-1-zg1" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "notif1", "bucket": { "name": "fish", "ownerIdentity": { "principalId": "zone.user" }, "arn": "arn:aws:s3:zg1::fish", "id": "9f96b2e9-955a-405d-a6c5-0c8d4075f403.4160.1" }, "object": { "key": "myfile1", "size": 512, "eTag": "79b473733f93105648e822db4b742331", "versionId": "", "sequencer": "7A974A6446D3B915", "metadata": [ { "key": "x-amz-content-sha256", "val": "cfd26bce46b835f777b02458b301f4d2ef25d65f78247954c52ee3e0f2412b62" }, { "key": "x-amz-date", "val": "20230427T154042Z" } ], "tags": [] } }, "eventId": "1682610042.364499.79b473733f93105648e822db4b742331", "opaqueData": "" } ] }
{ "Records": [ { "eventVersion": "2.2", "eventSource": "ceph:s3", "awsRegion": "zg1", "eventTime": "2023-04-27T15:40:49.106938Z", "eventName": "ObjectSynced:Create", "userIdentity": { "principalId": "rgw sync" }, "requestParameters": { "sourceIPAddress": "" }, "responseElements": { "x-amz-request-id": "0", "x-amz-id-2": "102a-zg1-2-zg1" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "notif1", "bucket": { "name": "fish", "ownerIdentity": { "principalId": "" }, "arn": "arn:aws:s3:zg1::fish", "id": "9f96b2e9-955a-405d-a6c5-0c8d4075f403.4160.1" }, "object": { "key": "myfile1", "size": 0, "eTag": "79b473733f93105648e822db4b742331", "versionId": "", "sequencer": "81974A648CCF5F06", "metadata": [], "tags": [] } }, "eventId": "1682610049.106942.79b473733f93105648e822db4b742331", "opaqueData": "" } ] }
Updated by Jane Zhu almost 1 year ago
https://github.com/ceph/ceph/pull/51295
Verified by the following test.
$ head -c 512 </dev/urandom > myfile $ aws --region=zg1 --endpoint-url http://localhost:8101 s3 cp myfile s3://fish
Regular notification:
{ "Records": [ { "eventVersion": "2.2", "eventSource": "ceph:s3", "awsRegion": "zg1", "eventTime": "2023-05-01T01:43:38.908732Z", "eventName": "ObjectCreated:Put", "userIdentity": { "principalId": "zone.user" }, "requestParameters": { "sourceIPAddress": "" }, "responseElements": { "x-amz-request-id": "20f69b95-f1a5-4b7a-acfd-8eb19a58a3ac.4158.8547944457124373088", "x-amz-id-2": "103e-zg1-1-zg1" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "notif2", "bucket": { "name": "fish", "ownerIdentity": { "principalId": "zone.user" }, "arn": "arn:aws:s3:zg1::fish", "id": "20f69b95-f1a5-4b7a-acfd-8eb19a58a3ac.4160.1" }, "object": { "key": "myfile", "size": 512, "eTag": "ec39de9f1753775cb1be3aa18d83e65c", "versionId": "d06CtFxY3hX4nHKfDTOVRgVzmGvbmJG", "sequencer": "4A194F64355D923A", "metadata": [ { "key": "x-amz-content-sha256", "val": "3363c72a2edd4e813094a9d9ae4333b21571e5226a021fcdd32e71a1736c139e" }, { "key": "x-amz-date", "val": "20230501T014337Z" } ], "tags": [] } }, "eventId": "1682905418.982670.ec39de9f1753775cb1be3aa18d83e65c", "opaqueData": "" } ] }
Sync notification:
{ "Records": [ { "eventVersion": "2.2", "eventSource": "ceph:s3", "awsRegion": "zg1", "eventTime": "2023-05-01T01:43:52.584781Z", "eventName": "ObjectSynced:Create", "userIdentity": { "principalId": "rgw sync" }, "requestParameters": { "sourceIPAddress": "" }, "responseElements": { "x-amz-request-id": "0", "x-amz-id-2": "102a-zg1-2-zg1" }, "s3": { "s3SchemaVersion": "1.0", "configurationId": "notif1", "bucket": { "name": "fish", "ownerIdentity": { "principalId": "" }, "arn": "arn:aws:s3:zg1::fish", "id": "20f69b95-f1a5-4b7a-acfd-8eb19a58a3ac.4160.1" }, "object": { "key": "myfile", "size": 512, "eTag": "ec39de9f1753775cb1be3aa18d83e65c", "versionId": "d06CtFxY3hX4nHKfDTOVRgVzmGvbmJG", "sequencer": "58194F64242DDB22", "metadata": [], "tags": [] } }, "eventId": "1682905432.584789.ec39de9f1753775cb1be3aa18d83e65c", "opaqueData": "" } ] }
Updated by Yuval Lifshitz 12 months ago
- Status changed from New to Fix Under Review
- Pull request ID set to 51295
Updated by Yuval Lifshitz 12 months ago
looks like bucket owner is also missing. should be fixed by changing the following line to "populate_event()" in "rgw_notify.cc":
event.bucket_ownerIdentity = res.bucket->get_owner() ? res.bucket->get_owner()->get_id().id : res.bucket->get_info().owner.id;
btw, a more complete test could be found here: https://gist.github.com/yuvalif/527a5064a1748d447129c594838448b0
Updated by Casey Bodley 12 months ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to reef
is this needed on earlier releases?
Updated by Backport Bot 12 months ago
- Copied to Backport #59646: reef: sync notifications has object size of zero added
Updated by Backport Bot 12 months ago
- Tags changed from multisite, notification to multisite, notification backport_processed
Actions