Bug #8241
closedXfsFileStoreBackend tries to set extsize but may get EINVAL
Added by Dan Mick about 10 years ago. Updated almost 10 years ago.
0%
Description
IRC user pinguini reports a problem with FSSETXATTR:
(09:59:06 PM) pinguini: hello everyone. need help. a hase osd on XFS and got in logs some error reporting : 2014-04-29 11:41:59.199100 7fa991a09700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-13) set_extsize: FSSETXATTR: (22) Invalid argument (09:59:10 PM) pinguini: many times in 4 of 10 identical osds...on 2 of 5 machines
I looked at the xfs kernel code and saw that it could return EINVAL if the file already had allocated extents, or if it's not a multiple of the block size, or if it's too large. I don't know which of these might be the case on this system.
Updated by Dan Mick about 10 years ago
- Category set to OSD
- Source changed from other to Community (user)
Updated by Volker Voigt about 10 years ago
Today pinguini (with a different nick "anton" this time) gave more information on IRC regarding this issue:
all nodes in cluster have ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
some nodes upgraded from 0.72
kernel is: Linux hw06 3.15.0-031500rc2-generic #201404201435 SMP Sun Apr 20 18:36:18 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
some nodes was upgraded from ubuntu 13.10 to 14.04
newly created osd:
root@hw03:~# tail -f /var/log/ceph/ceph-osd.7.log
2014-05-02 18:50:44.426218 7f9aaeb80700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:44.442945 7f9aae37f700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:45.184573 7f9aaeb80700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:45.442877 7f9aae37f700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:45.451180 7f9aaeb80700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:48.234559 7f9aae37f700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:48.259494 7f9aaeb80700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:53.450641 7f9aae37f700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:53.467471 7f9aaeb80700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:53.475828 7f9aae37f700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
Linux hw03 3.15.0-031500rc2-generic #201404201435 SMP Sun Apr 20 18:36:18 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root@hw03:~# xfs_info /dev/sdd4
meta-data=/dev/sdd4 isize=2048 agcount=32, agsize=22587448 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=722798336, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=352928, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
/dev/sdd4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,noquota 0 0
distr upgraded osd:
root@hw06:~# tail -f /var/log/ceph/ceph-osd.9.log
2014-05-02 18:51:48.385939 7fdb7a03a700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.444236 7fdb7a83b700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.586052 7fdb7a03a700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.627782 7fdb7a83b700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.636085 7fdb7a03a700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.652831 7fdb7a83b700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.686273 7fdb7a03a700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.752952 7fdb7a83b700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:49.361170 7fdb7a03a700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:49.744531 7fdb7a83b700 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
root@hw06:~# xfs_info /dev/sda4
meta-data=/dev/sda4 isize=2048 agcount=32, agsize=22587512 blks
= sectsz=4096 attr=2
data = bsize=4096 blocks=722800384, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal bsize=4096 blocks=352929, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
cat /proc/mounts | grep ceph-9
/dev/sda4 /var/lib/ceph/osd/ceph-9 xfs rw,noatime,attr2,inode64,noquota 0 0
(Unfortunately pinguini/anton could not add this information him/herself, as account registration is still broken. See http://tracker.ceph.com/issues/7609 )
Updated by Mark Nelson almost 10 years ago
Seeing this with master during RBD tiering tests only on the cache tier OSDs. Eventually during heavy RBD random 4MB write workloads the OSDs crash with heartbeat timeouts which may or may not be related.
Updated by Dan Mick almost 10 years ago
- Assignee set to Ilya Dryomov
- Priority changed from Normal to High
Bumped pri, assigned to Ilya for investigation
Updated by Michael Potter almost 10 years ago
I'm also seeing this error. Will try removing my RBD cache tier and see if it stops happening.
--- end dump of recent events --- 2014-05-21 16:46:47.368473 7fe104f007c0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 26032 2014-05-21 16:46:47.370012 7fe104f007c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount detected xfs (libxfs) 2014-05-21 16:46:47.429592 7fe104f007c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is supported and appears to work 2014-05-21 16:46:47.429609 7fe104f007c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config optio n 2014-05-21 16:46:47.802252 7fe104f007c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-05-21 16:46:47.802323 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_feature: extsize is supported 2014-05-21 16:46:47.971222 7fe104f007c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2014-05-21 16:46:52.308030 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:52.309383 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:52.310689 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:52.312772 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
...Snip repeat...
2014-05-21 16:46:52.595155 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:52.665014 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:52.665312 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:52.665601 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument 2014-05-21 16:46:53.708731 7fe104f007c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount detected xfs (libxfs) 2014-05-21 16:46:53.764853 7fe104f007c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is supported and appears to work 2014-05-21 16:46:53.764861 7fe104f007c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2014-05-21 16:46:53.814570 7fe104f007c0 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2014-05-21 16:46:53.814627 7fe104f007c0 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_feature: extsize is supported 2014-05-21 16:46:53.965018 7fe104f007c0 0 filestore(/var/lib/ceph/osd/ceph-7) mount: WRITEAHEAD journal mode explicitly enabled in conf
Updated by Pierre Grandin almost 10 years ago
I am probably facing the same issue. Initially reported as #8439
Did any of you fix the issue on your own cluster? 75% of my cluster is down right now.
Updated by Ilya Dryomov almost 10 years ago
Hi Pierre,
I don't think your crash (#8439) has anything to do with this. set_extsize is simply an allocation hint, EINVAL could be the result of an upgrade from older OSDs. On xfs, which is the only filesystem set_extsize supports, the hint can only be set if the file is empty. Now, rbd doesn't maintain any bitmap or suchlike, hence the hint op is sent unconditionally, which will trigger EINVAL if the underlying object happens to exist. That EINVAL is explicitly ignored higher on the stack though, so it's unlikely to be the problem.
I'll take a look at what is going on with tiering. Also, we should probably lower the log priority of this message.
Updated by Ilya Dryomov almost 10 years ago
- Status changed from New to Fix Under Review
With the fix applied we shouldn't see this message anymore, so I've
left the priority intact. The other known causes are a hint which is
bigger than half the size of the XFS AG or a misaligned hint, both of
which are something we'd want to notice.
Mark's problem with tiering may have been caused by the fact that hint
isn't propagated on cache flushes - its value isn't stored anywhere and
consequently it isn't reapplied during the flush.
Updated by Ilya Dryomov almost 10 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Sage Weil almost 10 years ago
- Status changed from Pending Backport to Resolved