Ceph

After some work this week-end, I tried quite some things to keep the test locked in the cluster (to avoid my user entering it again), but without success. The crash finally appeared on another OSD (osd.0), on the second node, while letting the cluster recover. It was under btrfs too so I cleaned up the osd.1 and formatted with XFS. Letting the cluster recover ended in the same crash under XFS.

I tried an infinite loop to restart the OSD, hoping to let the "bad area" behind at some time. I seemed to work a bit but finally it ended crashing after a few milliseconds of service. So I had to rebuild the whole cluster to give the service back to the developers.

IMHO, Ceph shouldn't crash when getting corrupted data from the disk, so I filled a bug. The fact that the error seemed to be transferred between the OSDs, crashing the other ones, is very wrong to me. I'll keep the logs a few weeks, if someone decides to open this issue again I'll upload them.

52f622a2/rbd_data.22d4c74b0dc51.0000000000002853/c4//2 seems to be the missing object.

Can you locate that object and its other snaps on each node (probably only exists on a few nodes) and attach a dump of the xattrs from each one?

You should be able to find each clone of that object with find . -name "*52f622a2*" in each osd's current/2.22_head/ directory.

If you already killed the cluster, any other crash logs would be good.

Samuel Just wrote:

What kernel version are you running?

The debian wheezy's kernel package so its a 3.2 (linux-image-3.2.0-4-amd64_3.2.41-2+deb7u2_amd64.deb).

Today I had contact with a cluster that suffers the same problem.

ceph version 0.67.5 (a60ac9194718083a4b6a225fc17cad6096c69bd1)

1: /usr/bin/ceph-osd() [0x7f9bba]
2: (()+0xfcb0) [0x7fb29b121cb0]
3: (gsignal()+0x35) [0x7fb299892425]
4: (abort()+0x17b) [0x7fb299895b8b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7fb29a1e569d]
6: (()+0xb5846) [0x7fb29a1e3846]
7: (()+0xb5873) [0x7fb29a1e3873]
8: (()+0xb596e) [0x7fb29a1e396e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x1df) [0x8bf3df]
10: (ReplicatedPG::trim_object(hobject_t const&)+0x150) [0x5e9400]
11: (ReplicatedPG::TrimmingObjects::react(ReplicatedPG::SnapTrim
const&)+0x423) [0x618613]
12: (boost::statechart::simple_state<ReplicatedPG::TrimmingObjects,
ReplicatedPG::SnapTrimmer, boost::mpl::list<mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na,
mpl_::na, mpl_::na, mpl_::na, mpl_::na>,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0xbc) [0x646eec]
13: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer,
ReplicatedPG::NotTrimming, std::allocator<void>,
boost::statechart::null_exception_translator>::process_queued_events()+0xfb)
[0x62d8ab]
14: (boost::statechart::state_machine<ReplicatedPG::SnapTrimmer,
ReplicatedPG::NotTrimming, std::allocator<void>,
boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
const&)+0x1e) [0x62da3e]
15: (ReplicatedPG::snap_trimmer()+0x516) [0x5cb1f6]
16: (OSD::SnapTrimWQ::_process(PG*)+0x14) [0x69a3b4]
17: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0x8b24f6]
18: (ThreadPool::WorkThread::entry()+0x10) [0x8b4300]
19: (()+0x7e9a) [0x7fb29b119e9a]
20: (clone()+0x6d) [0x7fb2999503fd]

One OSD is constantly offline. After the node joins back the cluster, one corrupted pg causes the assert in trim_object() to trigger:

  int r = find_object_context(coid, &obc, false, NULL);
  if (r == -ENOENT || coid.snap != obc->obs.oi.soid.snap) {
    derr << __func__ << "could not find coid " << coid << dendl;
    assert(0);
  }

gdb> print coid

$2 = (const hobject_t &) @0x214dd848: {oid = {name = {static npos = <optimized out>,
      _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>},
        _M_p = 0x20ea6608 "rbd_data.5d5c0d2ae8944a.", '0' <repeats 13 times>, "c89"}}}, snap = {val = 4947}, hash = 1936754598, max = false,
  pool = 2, nspace = {static npos = <optimized out>,
    _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xd27b38 ""}}, key = {
    static npos = <optimized out>,
    _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xd27b38 ""}}}

We tracked the object down to a block device in rados that contained a windows VM and was a clone of a template image.

While debugging it seemed like find_object_context(coid, &obc, false, NULL); was called from other places without a problem, only while beeing called from trim_object() it crashes the server.

Prior the problem started, there was a problem with the router that connects the ceph cluster, leading to a network downtime of ~2 minutes. I don't know of only one or many nodes where affected by the network outage. Each node runs ~7 ceph-osd instances with 4 nodes in total.

Please mark this bug more urgent, because this is a case where data corruption occured and that causes permanent node failure.

Our customer will now try to remove the rados image to see if this solves the problem.

The nodes use xfs, so I don't think that the original assumption that filesystem corruption may be the cause is the only way to trigger this problem.

find_object_context() in trim_object returned -ENOENT.

But I couldn't track down why, optimized builds were driving me nuts and every time the debugger stopped, the node gets into a strange state when paused for like 5 secondes. constanlty reconnecting errors...

The initial problem was thought to be related to snapshots, therefore they cleaned all snapshots on the cluster but this did not fix it.

the identified vm was a clone of a template machine.

while we were debugging the problem I set a breakpoint to the find_object_context() first with a filter to the pg that caused the crash. The function seemed to return valid results from other contexts of the state machine, only when the trim_object is called, it returns not found.

We deleted the rados block that was in the stack trace, but deleting it did not solve the problem.
The node still crashes when syncing the cluster.

Daniel Poelzleithner wrote:

We deleted the rados block that was in the stack trace, but deleting it did not solve the problem.
The node still crashes when syncing the cluster.

Just a hunch:

You know what the offending object is, right? Map it with ceph osd map <poolname> <objname>. Then look into the OSDs that have it, find the file, and run getfattr -d <filename> on all your copies. Let me know if any of your copies returns -E2BIG (i.e. bash telling you "argument list too long").

We applied the fix last week the crashing node.
The osd node now runs again, but see that the missing objects gets trim events ~ every 1-2 seconds that fail of course, but do not seem to harm anymore.
Any idea what causes this and how it can be fixed ?

Hey there,

we're currently using Nathan's patch to work around this issue that we encountered on v0.94.10. The patch still applies cleanly there.

Looking at the trunk I can see that the trim code seems to have evolved to not assert on missing objects any longer.

However, looking to upgrade towards Jewel this seems to become an issue again: the trim code isn't robust in Jewel yet (v.10.2.10, see ReplicatedPG.cc:3375).

Any chance to either forward-port Nathan's patch or backport the evolution from trunk?

Quick update: the patch applied cleanly for 0.94.10 but needed a change in the logging close.

Christian Theune wrote:

Quick update: the patch applied cleanly for 0.94.10 but needed a change in the logging close.

s/close/code/ obviously.

@Christian: For details see https://github.com/ceph/ceph/blob/master/SubmittingPatches.rst but, to summarize, to get the patch into the git repo, the standard procedure is to open a GitHub PR targeting the branch "master", and then ask for the patch to be backported to the stable branches (currently luminous and jewel).

If it is confirmed that the bug is already fixed in master and the master fix is not backportable (e.g. because the backport would be too complicated), then it is possible to consider a special fix for the stable branch(es) in question.

This policy exists to avoid exactly the situation you describe - an issue is fixed in a stable branch but not in master, so as soon as the next release comes out "a regression is born".

Thanks for that input. I guess this is a bit of a pickle in any case: the patch from Hammer likely won't apply at all and the change in Jewel is so substantial that it doesn't map backwards either. I guess it was fixed "along the way", but I'll try to find out.

@Christian: I just noticed this is fixing a FileStore-on-BTRFS issue. As of luminous BTRFS is no longer recommended for FileStore - maybe you could consider migrating your OSDs to XFS when you upgrade to Jewel and take care of the issue that way?

Maybe it's intended to fix something on BTRFS, but this has been happening to us on on XFS and the patch I'm talking about isn't really backend specific at all but touches an area where Ceph-internal data structures can become corrupted and then do not have any chance to recovery in Hammer and Jewel (as I said, Luminous seems fine from looking at the code).

Christian Theune wrote:

Maybe it's intended to fix something on BTRFS, but this has been happening to us on on XFS and the patch I'm talking about isn't really backend specific at all but touches an area where Ceph-internal data structures can become corrupted and then do not have any chance to recovery in Hammer and Jewel (as I said, Luminous seems fine from looking at the code).

Hammer is EOL, but perhaps you could open a PR targeting the jewel branch, then? It would be fine if the commit message contained a sentence or two explaining why the fix cannot be cherry-picked from master/luminous. In a pinch, this could be just a copy-paste of what you already wrote, above.

FYI, we're finally moving to Jewel (and hopefully Luminous after that) so I took a look at this patch again. I think I managed to create viable version of the hammer patch, mixed with some inspiration from the newer code in luminous. Note that luminous seems to have fixed the issue by now and porting the Hammer patch to Jewel is sufficient without further work on the newer branches. I guess the Kraken branch doesn't matter anyway.

Thanks for taking a look at https://github.com/ceph/ceph/pull/24423, even if Jewel might already be EOL as well ... (Hopefully we can catch up to Luminous this time without running into an already EOLed version yet again ... ;) )

https://github.com/ceph/ceph/pull/24423