Actions
Bug #2353
closedosd: current/ snap check problem
Added by Sage Weil about 12 years ago. Updated about 12 years ago.
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2012-04-27 12:18:08.015605 7f281c7c17a0 0 filestore(/var/lib/ceph/osd/ceph-153) mount found snaps <42996,42997,42998>
2012-04-27 12:18:08.033759 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) ERROR: current/ volume data version is not equal to snapshotted version.
2012-04-27 12:18:08.033782 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) Current version 42998, last snap 42998
2012-04-27 12:18:08.033785 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) Force rollback to snapshotted version with 'osd use stale snap = true'
2012-04-27 12:18:08.033787 7f281c7c17a0 -1 filestore(/var/lib/ceph/osd/ceph-153) config option for --osd-use-stale-snap startup argument.
Updated by Yehuda Sadeh about 12 years ago
I'd say the error message is bad:
// if current/ is marked as non-snapshotted, refuse to roll
// back (without clear direction) to avoid throwing out new
// data.
struct stat st;
if (::stat(nosnapfn, &st) == 0) {
if (!m_osd_use_stale_snap) {
derr << "ERROR: current/ volume data version is not equal to snapshotted version." << dendl;
derr << "Current version " << curr_seq << ", last snap " << cp << dendl;
derr << "Force rollback to snapshotted version with 'osd use stale snap = true'" << dendl;
derr << "config option for --osd-use-stale-snap startup argument." << dendl;
ret = -ENOTSUP;
goto close_basedir_fd;
}
Updated by Sage Weil about 12 years ago
2012-04-26 12:43:59.940705 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work 2012-04-26 12:43:59.940715 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work 2012-04-26 12:43:59.940813 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs 2012-04-26 12:43:59.940819 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported 2012-04-26 12:43:59.965695 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work 2012-04-26 12:44:00.214556 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supported 2012-04-26 12:44:00.214594 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs 2012-04-26 12:44:00.214654 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported 2012-04-26 12:44:00.290255 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY is supported 2012-04-26 12:44:00.290717 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs 2012-04-26 12:44:00.290728 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 841) 2012-04-26 12:44:00.290779 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported 2012-04-26 12:44:00.363595 7fa14c0a97a0 0 2012-04-26 12:44:00.363605 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supportedfilestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supported 2012-04-26 12:44:00.363948 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported 2012-04-26 12:44:00.434227 7fa14c0a97a0 0 2012-04-26 12:44:00.434250 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY is supportedfilestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY failed: (17) File exists 2012-04-26 12:44:00.434363 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs snaps enabled, but no SNAP_DESTROY ioctl (from kernel 2.6.32+); DISABLING 2012-04-26 12:44:00.434578 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 842) 2012-04-26 12:44:00.434644 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 842) 2012-04-26 12:44:00.434772 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported 2012-04-26 12:44:00.513976 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported 2012-04-26 12:44:00.514103 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported 2012-04-26 12:44:00.526348 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported 2012-04-26 12:44:00.582310 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount found snaps <130795,131005> 2012-04-26 12:44:00.606505 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported 2012-04-26 12:44:00.662228 7fc54c8ab7a0 0 filestore(/var/lib/ceph/osd/ceph-149) lock_fsid failed to lock /var/lib/ceph/osd/ceph-149/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable 2012-04-26 12:44:00.662274 7fc54c8ab7a0 -1 filestore(/var/lib/ceph/osd/ceph-149) FileStore::mount: lock_fsid failed 2012-04-26 12:44:00.662332 7fc54c8ab7a0 -1 ESC[0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-149: (16) Device or resource busyESC[0m2012-04-26 12:44:00.662288 7fa14c0a97a0 0 filestore(/var/lib/ceph/osd/ceph-149) lock_fsid failed to lock /var/lib/ceph/osd/ceph-149/fsid, is another ceph-osd still running? (11) Resource temporarily unavailable 2012-04-26 12:44:00.662328 7fa14c0a97a0 -1 filestore(/var/lib/ceph/osd/ceph-149) FileStore::mount: lock_fsid failed 2012-04-26 12:44:00.662385 7fa14c0a97a0 -1 ESC[0;31m ** ERROR: error converting store /var/lib/ceph/osd/ceph-149: (16) Device or resource busyESC[0m 2012-04-26 12:44:00.732750 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount: enabling WRITEAHEAD journal mode: 'filestore btrfs snap' mode is not enabled 2012-04-26 12:44:00.752473 7fbac99757a0 1 journal _open /var/lib/ceph/osd/ceph-149/journal fd 23: 524288000 bytes, block size 4096 bytes, directio = 1, aio = 0 2012-04-26 12:44:00.757500 7fbac99757a0 1 journal _open /var/lib/ceph/osd/ceph-149/journal fd 23: 524288000 bytes, block size 4096 bytes, directio = 1, aio = 0 2012-04-26 12:44:00.757865 7fbac99757a0 1 journal close /var/lib/ceph/osd/ceph-149/journal 2012-04-26 12:44:00.758354 7fbac99757a0 1 -- 0.0.0.0:6821/43305 messenger.start 2012-04-26 12:44:00.758366 7fbac99757a0 1 -- 0.0.0.0:6821/43305 accepter.start 2012-04-26 12:44:00.758396 7fbac99757a0 1 -- :/0 messenger.start 2012-04-26 12:44:00.758407 7fbac99757a0 1 -- 0.0.0.0:6823/43305 messenger.start 2012-04-26 12:44:00.758412 7fbac99757a0 1 -- 0.0.0.0:6823/43305 accepter.start 2012-04-26 12:44:00.758443 7fbac99757a0 1 -- 0.0.0.0:6822/43305 messenger.start 2012-04-26 12:44:00.758449 7fbac99757a0 1 -- 0.0.0.0:6822/43305 accepter.start 2012-04-26 12:44:00.758575 7fbac99757a0 2 osd.149 0 mounting /var/lib/ceph/osd/ceph-149 /var/lib/ceph/osd/ceph-149/journal 2012-04-26 12:44:00.832330 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount FIEMAP ioctl is supported and appears to work 2012-04-26 12:44:00.832564 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount detected btrfs 2012-04-26 12:44:00.832570 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs CLONE_RANGE ioctl is supported 2012-04-26 12:44:01.021927 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE is supported 2012-04-26 12:44:01.106220 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_DESTROY is supported 2012-04-26 12:44:01.106605 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs START_SYNC is supported (transid 846) 2012-04-26 12:44:01.188546 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs WAIT_SYNC is supported 2012-04-26 12:44:01.190686 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount btrfs SNAP_CREATE_V2 is supported 2012-04-26 12:44:01.270320 7fbac99757a0 0 filestore(/var/lib/ceph/osd/ceph-149) mount found snaps <130795,131005> 2012-04-26 12:44:01.270349 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) ERROR: current/ volume data version is not equal to snapshotted version. 2012-04-26 12:44:01.270353 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) Current version 131005, last snap 131005 2012-04-26 12:44:01.270354 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) Force rollback to snapshotted version with 'osd use stale snap = true' 2012-04-26 12:44:01.270358 7fbac99757a0 -1 filestore(/var/lib/ceph/osd/ceph-149) config option for --osd-use-stale-snap startup argument. 2012-04-26 12:44:01.270362 7fbac99757a0 -1 osd.149 0 OSD:init: unable to mount object store
Updated by Yehuda Sadeh about 12 years ago
Problem was that more than one daemon was brought up for the same osd, racing for the fs type check before mounting it, which made a specific btrfs ioctl to fail where it shouldn't have.
Actions