Ceph » Linux kernel client

Hi Alex,

While deleting a pool with mapped rbd images is just asking for trouble, we shouldn't panic. Can you provide some details on the panic (backtrace, etc) and your steps to reproduce in a form of a shell script? I tried but it didn't crash here.

Hello Ilya,

I've managed to re-create the issue, to make life easier I'll just explain our setup a bit.

We have three monitor servers, and three additional servers which contain 28 OSDs each. Each of the servers containing OSDs are called pyrum, prashadi, and pricei.

The script is in two sections.

If I run this on one of the OSD servers:

ceph osd pool create temp 4096
rbd create temp/test_rbd --size 1024
rbd map temp/test_rbd
parted -a optimal /dev/rbd0 mklabel gpt mkpart primary 0% 100%
mkfs -t xfs /dev/rbd0p1
mkdir /mnt/rbd
mount /dev/rbd0p1 /mnt/rbd
dd if=/dev/zero of=/mnt/rbd/data bs=512 count=2048

Then this on one of the monitors:

ceph osd pool delete temp temp --yes-i-really-really-mean-it
for host in pyrum prashadi pricei; do ssh $host stop ceph-all; done

Doing this reliably results in a panic on the server with the image mapped. I've attached two screenshots also. The first is just after deleting the pool (after_removal.jpg) the second is the actual panic (panic.jpg).

Let me know if you'd like any more info.

Regards,
Alex.

Your panic.jpg shows only a tail of a panic splat, which contains virtually no useful information, but I'm pretty sure that you are hitting a BUG_ON(!list_empty(&req->r_req_lru_item)); in __kick_osd_requests(). In newer kernels this BUG_ON has been changed to a WARN_ON so it'll stamp out a warning instead of panicing, but the actual issue that causes this BUG_ON/WARN_ON to fire needs fixing - I'll bump #9779 (which is what's going to fix this) and start working on it.

I'd encourage you to not delete pools with mapped rbd images. It's OK to delete a pool to quickly remove unmapped images, but, like I said, doing this with mapped images while I/O from/to those images is in progress is just asking for trouble - at the very least you are leaving behind a useless block device with a bumped refcount and process(es) stuck in D state waiting for I/O. Use something like

while read DEV; do
    rbd unmap $DEV
done < <(rbd --format=json showmapped | python -c " 
import sys, json;
for i in json.load(sys.stdin).itervalues():
    if i['pool'] == '$POOL':
        print i['device']
")

to unmap all mapped images from a given pool. That said, we won't panic and will print something like "pool foo existed but now does not" to dmesg after #9779 is resolved.