Project

General

Profile

Bug #43180

rgw: ordered listing of bucket with many incomplete multipart uploads fails

Added by Eric Ivancich 3 months ago. Updated 9 days ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Target version:
% Done:

0%

Source:
Tags:
Backport:
nautilus,mimic
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

This seems to be a result of https://tracker.ceph.com/issues/42358 .

Please see Beom-Seok Park's reproducer:

https://tracker.ceph.com/issues/42358#note-4

Related issues

Follows rgw - Bug #42358: rgw: reduce per-shard entry count during ordered bucket listing Resolved
Copied to rgw - Backport #44142: mimic: rgw: ordered listing of bucket with many incomplete multipart uploads fails New
Copied to rgw - Backport #44143: nautilus: rgw: ordered listing of bucket with many incomplete multipart uploads fails New

History

#1 Updated by Eric Ivancich 3 months ago

The following script, based on Beom-Seok Park's description, is able to reproduce the issue:

#!/bin/bash

bkt=test-bkt
bigobj=big-obj
smallobj=small-obj

filebig=/tmp/filebig
filesmall=/tmp/small.$$

fifo=/tmp/fifo.$$

export minsize=2684354560
export bs=4096

makebig() {
echo creating large local file $filebig
count=$(expr $minsize / $bs)
dd if=/dev/urandom of=$filebig bs=$bs count=$count
}

beep() {
/bin/echo -e -n "\a"
/bin/echo -e -n "\a"
}

echo "hello" >$filesmall

if [ ! -f $filebig ] ;then
makebig
elif [ $(stat -c %s $filebig) -lt $minsize ] ;then
makebig
fi

echo stopping ceph
../src/stop.sh

echo starting ceph
OSD=1 RGW=1 MON=1 MGR=0 MDS=0 FS=0 ../src/vstart.sh -n -l -d

echo sleeping....
sleep 5

echo making bucket $bkt
s3cmd mb s3://${bkt}

echo resharding bucket
bin/radosgw-admin bucket reshard --bucket=${bkt} --num-shards=97

echo uploading small file
s3cmd put $filesmall s3://${bkt}/${smallobj}

mkfifo $fifo

echo uploading large file with chunk size of 5MB
s3cmd put $filebig s3://${bkt}/${bigobj} --multipart-chunk-size-mb=5 >$fifo &

beep
stoppart=490
while read line ;do
echo "$line" | grep --quiet "part $stoppart "
if [ $? -eq 0 ] ;then
echo "found part $stoppart"
break
fi
done <$fifo
beep

echo killing upload
kill %1

echo listing bucket
s3cmd ls s3://${bkt}

echo done
beep

#2 Updated by Eric Ivancich 3 months ago

  • Severity changed from 3 - minor to 2 - major

#3 Updated by Matt Benjamin 3 months ago

Hi Eric,

Thanks solely to the reproducer (which I hadn't arrived at in testing with the provided instructions alone, due to small dynamic shard counts I think), I was able to dig into this and write a workaround for review. It handles the simple case, but there are some obvious questions as to whether it's actually a correct fix:

https://github.com/ceph/ceph/pull/32110

thanks!

Matt

#4 Updated by Nathan Cutler 2 months ago

  • Follows Bug #42358: rgw: reduce per-shard entry count during ordered bucket listing added

#5 Updated by Nathan Cutler 2 months ago

BACKPORTING NOTE: backport #42358 and this one as a single unit.

Backport issues at #42358 have been deleted.

#6 Updated by Daniel Gryniewicz 2 months ago

  • Pull request ID set to 32110

#7 Updated by Eric Ivancich about 1 month ago

Proposing PR https://github.com/ceph/ceph/pull/32636 as an alternative to address issue.

#8 Updated by Eric Ivancich about 1 month ago

  • Pull request ID changed from 32110 to 32636

#9 Updated by Eric Ivancich about 1 month ago

  • Status changed from In Progress to Fix Under Review

#10 Updated by Eric Ivancich 10 days ago

  • Status changed from Fix Under Review to Pending Backport

#11 Updated by Eric Ivancich 10 days ago

  • Backport changed from nautilus,mimic to octopus,nautilus,mimic

#12 Updated by Nathan Cutler 9 days ago

  • Backport changed from octopus,nautilus,mimic to nautilus,mimic

#13 Updated by Nathan Cutler 9 days ago

  • Copied to Backport #44142: mimic: rgw: ordered listing of bucket with many incomplete multipart uploads fails added

#14 Updated by Nathan Cutler 9 days ago

  • Copied to Backport #44143: nautilus: rgw: ordered listing of bucket with many incomplete multipart uploads fails added

Also available in: Atom PDF