Ceph

Please be advised that this issue appears to be a regression in Python 2.7.8 (see details in the Debian bug report).

http://bugs.python.org/issue21963

Fascinating info so far, Dmitry, thanks for your work on this. Anxious to see what the Python team thinks of the assertion that it's a Python issue.

I believe that we should attempt to replicate the problem first as I know the Python ticket will get ignored unless there is a way to reproduce (other than
installing ceph with that Python version and call `ceph -s`)

Currently looking at `rados.py` and the `run_in_thread` function looks like a good candidate to start with:

def run_in_thread(target, args, timeout=0):
    interrupt = False

    countdown = timeout
    t = RadosThread(target, args)

    # allow the main thread to exit (presumably, avoid a join() on this
    # subthread) before this thread terminates.  This allows SIGINT
    # exit of a blocked call.  See below.
    t.daemon = True

    t.start()
    try:
        # poll for thread exit
        while t.is_alive():
            t.join(POLL_TIME_INCR)
            if timeout and t.is_alive():
                countdown = countdown - POLL_TIME_INCR
                if countdown <= 0:
                    raise KeyboardInterrupt

        t.join()        # in case t exits before reaching the join() above
    except KeyboardInterrupt:
        # ..but allow SIGINT to terminate the waiting.  Note: this
        # relies on the Linux kernel behavior of delivering the signal
        # to the main thread in preference to any subthread (all that's
        # strictly guaranteed is that *some* thread that has the signal
        # unblocked will receive it).  But there doesn't seem to be
        # any interface to create t with SIGINT blocked.
        interrupt = True

    if interrupt:
        t.retval = -errno.EINTR
    return t.retval

For a moment Python maintainer in Debian kindly fixed this issue for us by adding patch to revert problematic change in Python.
However this is a time bomb as it is (potentially) affecting Ceph on all architectures and distributions outside Debian.
Please follow-up with Python developers or make changes for compatibility with Python-2.7.8.
This is a very serious issue because no cluster components can be started with vanilla Python-2.7.8.

Just a note that people are hitting this in fedora 21, now:

https://bugzilla.redhat.com/show_bug.cgi?id=1155335

This works around the problem, while also destroying the exit code from the ceph program, so if you rely on that, this won't help, but it will at least let the command exit:

change the last line of ceph, 'sys.exit(main())', to

    main()
    os.kill(os.getpid(), 15)

In order to get the exit code, I tried this:

    result = main()
    del cluster_handle
    sys.exit(result)

Which resulted in a core dump:

Illegal instruction (core dumped)

Which, of course, was the same result if I tried cluster_handle.shutdown() (since Rados.__del__ does the same call).

I was able to work around this by removing the Rados.__del__ function from rados.py. This allowed the thread to be at least abandoned with the above code and sys.exit to conclude.

The SIGILL was cured in master with the application of 92615ea and cf2104d. I've tested backporting these to firefly which allowed shutdown to be called without crashing.

Still need to del cluster_handle to avoid the hang which I think is because it's not guaranteed that threads will garbage collect with sys.exit.

PR #3053

I think the right fix for this is to remove Rados.__del__. I'll come up with a pull request unless you want to, Joe.

• merged in master with https://github.com/ceph/ceph/pull/3119

• merged in giant by https://github.com/ceph/ceph/pull/3168

e00270b rados.py: remove Rados.__del__(); it just causes problems (in firefly), ed8c9af rados.py: remove Rados.__del__(); it just causes problems (in giant),