Ceph » rgw

The following script, based on Beom-Seok Park's description, is able to reproduce the issue:

#!/bin/bash

bkt=test-bkt
bigobj=big-obj
smallobj=small-obj

filebig=/tmp/filebig
filesmall=/tmp/small.$$

fifo=/tmp/fifo.$$

export minsize=2684354560
export bs=4096

makebig() {
echo creating large local file $filebig
count=$(expr $minsize / $bs)
dd if=/dev/urandom of=$filebig bs=$bs count=$count
}

beep() {
/bin/echo -e -n "\a"
/bin/echo -e -n "\a"
}

echo "hello" >$filesmall

if [ ! -f $filebig ] ;then
makebig
elif [ $(stat -c %s $filebig) -lt $minsize ] ;then
makebig
fi

echo stopping ceph
../src/stop.sh

echo starting ceph
OSD=1 RGW=1 MON=1 MGR=0 MDS=0 FS=0 ../src/vstart.sh -n -l -d

echo sleeping....
sleep 5

echo making bucket $bkt
s3cmd mb s3://${bkt}

echo resharding bucket
bin/radosgw-admin bucket reshard --bucket=${bkt} --num-shards=97

echo uploading small file
s3cmd put $filesmall s3://${bkt}/${smallobj}

mkfifo $fifo

echo uploading large file with chunk size of 5MB
s3cmd put $filebig s3://${bkt}/${bigobj} --multipart-chunk-size-mb=5 >$fifo &

beep
stoppart=490
while read line ;do
echo "$line" | grep --quiet "part $stoppart "
if [ $? -eq 0 ] ;then
echo "found part $stoppart"
break
fi
done <$fifo
beep

echo killing upload
kill %1

echo listing bucket
s3cmd ls s3://${bkt}

echo done
beep

Hi Eric,

Thanks solely to the reproducer (which I hadn't arrived at in testing with the provided instructions alone, due to small dynamic shard counts I think), I was able to dig into this and write a workaround for review. It handles the simple case, but there are some obvious questions as to whether it's actually a correct fix:

https://github.com/ceph/ceph/pull/32110

thanks!

Matt

BACKPORTING NOTE: backport #42358 and this one as a single unit.

Backport issues at #42358 have been deleted.

Proposing PR https://github.com/ceph/ceph/pull/32636 as an alternative to address issue.