Project

General

Profile

Bug #37801

s3 compatibility for listing object

Added by Bálint Nagy about 5 years ago. Updated about 5 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi Guys!

I've been developing a Node.js application using Ceph as a distributed storage for my application. The application is using the aws-sdk.

The application is running in docker on a windows 10 host with the following setup commands:
docker stop ceph
docker rm ceph

docker network rm cephnet
docker network create --subnet=175.20.0.0/16 cephnet

docker volume rm --force varlibceph
docker volume create --name varlibceph

docker volume rm --force etcceph
docker volume create --name etcceph

docker run --name ceph --net cephnet --privileged -p 127.0.0.1:8000:8000 --mount type=volume,source=etcceph,target=/etc/ceph --mount type=volume,source=varlibceph,target=/var/lib/ceph --restart always --detach --ip 175.20.0.12 --memory 2g --env RGW_CIVETWEB_PORT=8000 --env CEPH_DEMO_UID=asdf --env CEPH_DEMO_ACCESS_KEY=X5XX1XXXX69XXXXXXX7J --env CEPH_DEMO_SECRET_KEY=0XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --env CEPH_PUBLIC_NETWORK=175.20.0.0/16 --env IP=175.20.0.12 --env MON_IP=175.20.0.12 ceph/daemon demo -xc "[[ ! -e /etc/ceph/I_AM_A_DEMO ]] && find /etc/ceph -mindepth 1 -delete ; . /opt/ceph-container/bin/entrypoint.sh demo"

Used version is:
ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable)

Upload / download works as expected but when I'm listing the buckets I'm only getting 1000 items but I'm not getting any Marker for listObjects() or NextContinuationToken in listObjectsV2()

This makes hard to get/list all the items already uploaded into Ceph.

If you have any info whether this is a bug or I'm just making something wrong I'd appriciate your help.

Thanks,
Bálint

History

#1 Updated by Matt Benjamin about 5 years ago

  • Status changed from New to In Progress

albin antony from our team is finishing up a listobjects v2 implementation

old: https://github.com/ceph/ceph/pull/26361

#2 Updated by Albin Antony about 5 years ago

Matt Benjamin wrote:

albin antony from our team is finishing up a listobjects v2 implementation

old: https://github.com/ceph/ceph/pull/26361

new: https://github.com/ceph/ceph/pull/26512

#3 Updated by Matt Benjamin about 5 years ago

  • Assignee set to Albin Antony

#4 Updated by Albin Antony about 5 years ago

Hi Bálint Nagy,

The ceph version 13.2.2 mimic doesn't support listobjectsv2(). That's why you are not getting the nextContinuationToken. But the listobjects() will be working fine. Could you share me your listobjects() implementation? I will look into it.

Thanks,
Albin

#5 Updated by Bálint Nagy about 5 years ago

Hi Albin!

Thanks for reaching out. Originaly I've used listobjectsv2() and then realized the problem. Then tried to use the listobjects() function but I've made a mistake that I've used the Marker instead of the given NextMarker. --noob mistake--

All in all it was my bug in that case.

My original head ache came from that v2() wasn't fully supported, though it gives result even if it wasn't fully implemented and give me an impression that it should work.

Thanks,
Bálint

Albin Antony wrote:

Hi Bálint Nagy,

The ceph version 13.2.2 mimic doesn't support listobjectsv2(). That's why you are not getting the nextContinuationToken. But the listobjects() will be working fine. Could you share me your listobjects() implementation? I will look into it.

Thanks,
Albin

#6 Updated by Albin Antony about 5 years ago

Hi Bálint Nagy,

I hope the next release of ceph will support v2 also. Feel free to reach out to me if you have doubts regarding the implementations :)

Thanks,
Albin

Also available in: Atom PDF