Actions
Bug #59577
opensync notifications has object size of zero
Added by Yuval Lifshitz about 1 year ago. Updated about 1 year ago.
Status:
Pending Backport
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
multisite, notification backport_processed
Backport:
reef
Regression:
No
Severity:
3 - minor
Reviewed:
Description
how to reproduce:
run an http POST server on port 10900
run the following commands:
MON=1 OSD=1 MDS=0 MGR=0 ../src/test/rgw/test-rgw-multisite.sh 2
export AWS_ACCESS_KEY_ID=1234567890
export AWS_SECRET_ACCESS_KEY=pencil
aws --region=zg1 --endpoint-url http://localhost:8201 sns create-topic --name=fishtopic --attributes='{"push-endpoint": "http://localhost:10900"}'
aws --region=zg1 --endpoint-url http://localhost:8101 s3 mb s3://fish
aws --region=zg1 --endpoint-url http://localhost:8201 s3api put-bucket-notification-configuration --bucket fish --notification-configuration='{"TopicConfigurations": [{"Id": "notif1", "TopicArn": "arn:aws:sns:default::fishtopic", "Events": ["s3:ObjectSynced:Create"]}]}'
head -c 512 </dev/urandom > myfile
aws --region=zg1 --endpoint-url http://localhost:8101 s3 cp myfile s3://fish
Updated by Yuval Lifshitz about 1 year ago
comparing sync notifications vs. regular notifications.
regular:
{ [40/57]
"Records": [
{
"eventVersion": "2.2",
"eventSource": "ceph:s3",
"awsRegion": "zg1",
"eventTime": "2023-04-27T15:40:42.321565Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "zone.user"
},
"requestParameters": {
"sourceIPAddress": ""
},
"responseElements": {
"x-amz-request-id": "9f96b2e9-955a-405d-a6c5-0c8d4075f403.4158.14554084588647464963",
"x-amz-id-2": "103e-zg1-1-zg1"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "notif1",
"bucket": {
"name": "fish",
"ownerIdentity": {
"principalId": "zone.user"
},
"arn": "arn:aws:s3:zg1::fish",
"id": "9f96b2e9-955a-405d-a6c5-0c8d4075f403.4160.1"
},
"object": {
"key": "myfile1",
"size": 512,
"eTag": "79b473733f93105648e822db4b742331",
"versionId": "",
"sequencer": "7A974A6446D3B915",
"metadata": [
{
"key": "x-amz-content-sha256",
"val": "cfd26bce46b835f777b02458b301f4d2ef25d65f78247954c52ee3e0f2412b62"
},
{
"key": "x-amz-date",
"val": "20230427T154042Z"
}
],
"tags": []
}
},
"eventId": "1682610042.364499.79b473733f93105648e822db4b742331",
"opaqueData": ""
}
]
}
{
"Records": [
{
"eventVersion": "2.2",
"eventSource": "ceph:s3",
"awsRegion": "zg1",
"eventTime": "2023-04-27T15:40:49.106938Z",
"eventName": "ObjectSynced:Create",
"userIdentity": {
"principalId": "rgw sync"
},
"requestParameters": {
"sourceIPAddress": ""
},
"responseElements": {
"x-amz-request-id": "0",
"x-amz-id-2": "102a-zg1-2-zg1"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "notif1",
"bucket": {
"name": "fish",
"ownerIdentity": {
"principalId": ""
},
"arn": "arn:aws:s3:zg1::fish",
"id": "9f96b2e9-955a-405d-a6c5-0c8d4075f403.4160.1"
},
"object": {
"key": "myfile1",
"size": 0,
"eTag": "79b473733f93105648e822db4b742331",
"versionId": "",
"sequencer": "81974A648CCF5F06",
"metadata": [],
"tags": []
}
},
"eventId": "1682610049.106942.79b473733f93105648e822db4b742331",
"opaqueData": ""
}
]
}
Updated by Jane Zhu about 1 year ago
https://github.com/ceph/ceph/pull/51295
Verified by the following test.
$ head -c 512 </dev/urandom > myfile $ aws --region=zg1 --endpoint-url http://localhost:8101 s3 cp myfile s3://fish
Regular notification:
{
"Records": [
{
"eventVersion": "2.2",
"eventSource": "ceph:s3",
"awsRegion": "zg1",
"eventTime": "2023-05-01T01:43:38.908732Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "zone.user"
},
"requestParameters": {
"sourceIPAddress": ""
},
"responseElements": {
"x-amz-request-id": "20f69b95-f1a5-4b7a-acfd-8eb19a58a3ac.4158.8547944457124373088",
"x-amz-id-2": "103e-zg1-1-zg1"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "notif2",
"bucket": {
"name": "fish",
"ownerIdentity": {
"principalId": "zone.user"
},
"arn": "arn:aws:s3:zg1::fish",
"id": "20f69b95-f1a5-4b7a-acfd-8eb19a58a3ac.4160.1"
},
"object": {
"key": "myfile",
"size": 512,
"eTag": "ec39de9f1753775cb1be3aa18d83e65c",
"versionId": "d06CtFxY3hX4nHKfDTOVRgVzmGvbmJG",
"sequencer": "4A194F64355D923A",
"metadata": [
{
"key": "x-amz-content-sha256",
"val": "3363c72a2edd4e813094a9d9ae4333b21571e5226a021fcdd32e71a1736c139e"
},
{
"key": "x-amz-date",
"val": "20230501T014337Z"
}
],
"tags": []
}
},
"eventId": "1682905418.982670.ec39de9f1753775cb1be3aa18d83e65c",
"opaqueData": ""
}
]
}
Sync notification:
{
"Records": [
{
"eventVersion": "2.2",
"eventSource": "ceph:s3",
"awsRegion": "zg1",
"eventTime": "2023-05-01T01:43:52.584781Z",
"eventName": "ObjectSynced:Create",
"userIdentity": {
"principalId": "rgw sync"
},
"requestParameters": {
"sourceIPAddress": ""
},
"responseElements": {
"x-amz-request-id": "0",
"x-amz-id-2": "102a-zg1-2-zg1"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "notif1",
"bucket": {
"name": "fish",
"ownerIdentity": {
"principalId": ""
},
"arn": "arn:aws:s3:zg1::fish",
"id": "20f69b95-f1a5-4b7a-acfd-8eb19a58a3ac.4160.1"
},
"object": {
"key": "myfile",
"size": 512,
"eTag": "ec39de9f1753775cb1be3aa18d83e65c",
"versionId": "d06CtFxY3hX4nHKfDTOVRgVzmGvbmJG",
"sequencer": "58194F64242DDB22",
"metadata": [],
"tags": []
}
},
"eventId": "1682905432.584789.ec39de9f1753775cb1be3aa18d83e65c",
"opaqueData": ""
}
]
}
Updated by Yuval Lifshitz about 1 year ago
- Status changed from New to Fix Under Review
- Pull request ID set to 51295
Updated by Yuval Lifshitz about 1 year ago
looks like bucket owner is also missing. should be fixed by changing the following line to "populate_event()" in "rgw_notify.cc":
event.bucket_ownerIdentity = res.bucket->get_owner() ? res.bucket->get_owner()->get_id().id : res.bucket->get_info().owner.id;
btw, a more complete test could be found here: https://gist.github.com/yuvalif/527a5064a1748d447129c594838448b0
Updated by Jane Zhu about 1 year ago
Yuval Lifshitz wrote:
looks like bucket owner is also missing. should be fixed by changing the following line to "populate_event()" in "rgw_notify.cc":
[...]
Ah, didn't notice. Will add this in.
Updated by Casey Bodley about 1 year ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to reef
is this needed on earlier releases?
Updated by Backport Bot about 1 year ago
- Copied to Backport #59646: reef: sync notifications has object size of zero added
Updated by Backport Bot about 1 year ago
- Tags changed from multisite, notification to multisite, notification backport_processed
Actions