Project

General

Profile

Actions
Copy link

Bug #2353

closed

osd: current/ snap check problem

Added by Sage Weil almost 12 years ago. Updated almost 12 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2012-04-27 12:18:08.015605 7f281c7c17a0 0 filestore(/var/lib/ceph/osd/ceph-153) mount found snaps <42996,42997,42998>
2012-04-27 12:18:08.033759 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) ERROR: current/ volume data version is not equal to snapshotted version.
2012-04-27 12:18:08.033782 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) Current version 42998, last snap 42998
2012-04-27 12:18:08.033785 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) Force rollback to snapshotted version with 'osd use stale snap = true'
2012-04-27 12:18:08.033787 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) config option for --osd-use-stale-snap startup argument.

Actions
Copy link
 #1

Updated by Yehuda Sadeh almost 12 years ago

I'd say the error message is bad:

    // if current/ is marked as non-snapshotted, refuse to roll
    // back (without clear direction) to avoid throwing out new
    // data.
    struct stat st;
    if (::stat(nosnapfn, &st) == 0) {
      if (!m_osd_use_stale_snap) {
        derr << "ERROR: current/ volume data version is not equal to snapshotted version." << dendl;
        derr << "Current version " << curr_seq << ", last snap " << cp << dendl;
        derr << "Force rollback to snapshotted version with 'osd use stale snap = true'" << dendl;
        derr << "config option for --osd-use-stale-snap startup argument." << dendl;
        ret = -ENOTSUP;
        goto close_basedir_fd;
      }

Actions
Copy link
 #2

Updated by Sage Weil almost 12 years ago 


2012-04-26 12:43:59.940705 7fa14c0a97a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work
2012-04-26 12:43:59.940715 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work
2012-04-26 12:43:59.940813 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs
2012-04-26 12:43:59.940819 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported
2012-04-26 12:43:59.965695 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work
2012-04-26 12:44:00.214556 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supported
2012-04-26 12:44:00.214594 7fa14c0a97a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs
2012-04-26 12:44:00.214654 7fa14c0a97a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported
2012-04-26 12:44:00.290255 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY is supported
2012-04-26 12:44:00.290717 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs
2012-04-26 12:44:00.290728 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 841)
2012-04-26 12:44:00.290779 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported
2012-04-26 12:44:00.363595 7fa14c0a97a0  0 2012-04-26 12:44:00.363605 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supportedfilestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supported

2012-04-26 12:44:00.363948 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported
2012-04-26 12:44:00.434227 7fa14c0a97a0  0 2012-04-26 12:44:00.434250 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY is supportedfilestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY failed: (17) File exists

2012-04-26 12:44:00.434363 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs snaps enabled, but no SNAP_DESTROY ioctl (from kernel 2.6.32+); DISABLING
2012-04-26 12:44:00.434578 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 842)
2012-04-26 12:44:00.434644 7fa14c0a97a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 842)
2012-04-26 12:44:00.434772 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported
2012-04-26 12:44:00.513976 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported
2012-04-26 12:44:00.514103 7fa14c0a97a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported
2012-04-26 12:44:00.526348 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported
2012-04-26 12:44:00.582310 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount found snaps <130795,131005>
2012-04-26 12:44:00.606505 7fa14c0a97a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported
2012-04-26 12:44:00.662228 7fc54c8ab7a0  0 filestore(/var/lib/ceph/osd/ceph-149) lock_fsid failed to lock /var/lib/ceph/osd/ceph-149/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
2012-04-26 12:44:00.662274 7fc54c8ab7a0 -1 filestore(/var/lib/ceph/osd/ceph-149) FileStore::mount: lock_fsid failed
2012-04-26 12:44:00.662332 7fc54c8ab7a0 -1 ESC[0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-149: (16) Device or resource busyESC[0m2012-04-26 12:44:00.662288 7fa14c0a97a0  0 
filestore(/var/lib/ceph/osd/ceph-149) lock_fsid failed to lock /var/lib/ceph/osd/ceph-149/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable
2012-04-26 12:44:00.662328 7fa14c0a97a0 -1 filestore(/var/lib/ceph/osd/ceph-149) FileStore::mount: lock_fsid failed
2012-04-26 12:44:00.662385 7fa14c0a97a0 -1 ESC[0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-149: (16) Device or resource busyESC[0m
2012-04-26 12:44:00.732750 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount: enabling WRITEAHEAD journal mode: 'filestore btrfs snap' mode is not enabled
2012-04-26 12:44:00.752473 7fbac99757a0  1 journal _open /var/lib/ceph/osd/ceph-149/journal fd 23: 524288000 bytes, block size 4096 bytes, directio = 1, aio = 0
2012-04-26 12:44:00.757500 7fbac99757a0  1 journal _open /var/lib/ceph/osd/ceph-149/journal fd 23: 524288000 bytes, block size 4096 bytes, directio = 1, aio = 0
2012-04-26 12:44:00.757865 7fbac99757a0  1 journal close /var/lib/ceph/osd/ceph-149/journal
2012-04-26 12:44:00.758354 7fbac99757a0  1 -- 0.0.0.0:6821/43305 messenger.start
2012-04-26 12:44:00.758366 7fbac99757a0  1 -- 0.0.0.0:6821/43305 accepter.start
2012-04-26 12:44:00.758396 7fbac99757a0  1 -- :/0 messenger.start
2012-04-26 12:44:00.758407 7fbac99757a0  1 -- 0.0.0.0:6823/43305 messenger.start
2012-04-26 12:44:00.758412 7fbac99757a0  1 -- 0.0.0.0:6823/43305 accepter.start
2012-04-26 12:44:00.758443 7fbac99757a0  1 -- 0.0.0.0:6822/43305 messenger.start
2012-04-26 12:44:00.758449 7fbac99757a0  1 -- 0.0.0.0:6822/43305 accepter.start
2012-04-26 12:44:00.758575 7fbac99757a0  2 osd.149 0 mounting /var/lib/ceph/osd/ceph-149 /var/lib/ceph/osd/ceph-149/journal
2012-04-26 12:44:00.832330 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work
2012-04-26 12:44:00.832564 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs
2012-04-26 12:44:00.832570 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported
2012-04-26 12:44:01.021927 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supported
2012-04-26 12:44:01.106220 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY is supported
2012-04-26 12:44:01.106605 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 846)
2012-04-26 12:44:01.188546 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported
2012-04-26 12:44:01.190686 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported
2012-04-26 12:44:01.270320 7fbac99757a0  0 filestore(/var/lib/ceph/osd/ceph-149) mount found snaps <130795,131005>
2012-04-26 12:44:01.270349 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) ERROR: current/ volume data version is not equal to snapshotted version.
2012-04-26 12:44:01.270353 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) Current version 131005, last snap 131005
2012-04-26 12:44:01.270354 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) Force rollback to snapshotted version with 'osd use stale snap = true'
2012-04-26 12:44:01.270358 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) config option for --osd-use-stale-snap startup argument.
2012-04-26 12:44:01.270362 7fbac99757a0 -1 osd.149 0 OSD:init: unable to mount object store
Actions
Copy link
 #3

Updated by Yehuda Sadeh almost 12 years ago

Problem was that more than one daemon was brought up for the same osd, racing for the fs type check before mounting it, which made a specific btrfs ioctl to fail where it shouldn't have.

Actions
Copy link
 #4

Updated by Sage Weil almost 12 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF