Project

General

Profile

Actions
Copy link

Bug #8241

closed

XfsFileStoreBackend tries to set extsize but may get EINVAL

Added by Dan Mick almost 10 years ago. Updated almost 10 years ago.

Status:
Resolved
Priority:
High
Assignee:
Ilya Dryomov
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

IRC user pinguini reports a problem with FSSETXATTR:

(09:59:06 PM) pinguini: hello everyone. need help. a hase osd on XFS and got in logs some error reporting : 2014-04-29 11:41:59.199100 7fa991a09700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-13) set_extsize: FSSETXATTR: (22) Invalid argument
(09:59:10 PM) pinguini: many times in 4 of 10 identical osds...on 2 of 5 machines

I looked at the xfs kernel code and saw that it could return EINVAL if the file already had allocated extents, or if it's not a multiple of the block size, or if it's too large. I don't know which of these might be the case on this system.

Actions
Copy link
 #1

Updated by Dan Mick almost 10 years ago

  • Category set to OSD
  • Source changed from other to Community (user)
Actions
Copy link
 #2

Updated by Volker Voigt almost 10 years ago

Today pinguini (with a different nick "anton" this time) gave more information on IRC regarding this issue:


all nodes in cluster  have ceph version 0.79 (4c2d73a5095f527c3a2168deb5fa54b3c8991a6e)
some nodes upgraded from 0.72 
kernel is: Linux hw06 3.15.0-031500rc2-generic #201404201435 SMP Sun Apr 20 18:36:18 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

some nodes was upgraded from ubuntu 13.10 to 14.04

newly created osd:
root@hw03:~# tail -f /var/log/ceph/ceph-osd.7.log
2014-05-02 18:50:44.426218 7f9aaeb80700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:44.442945 7f9aae37f700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:45.184573 7f9aaeb80700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:45.442877 7f9aae37f700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:45.451180 7f9aaeb80700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:48.234559 7f9aae37f700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:48.259494 7f9aaeb80700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:53.450641 7f9aae37f700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:53.467471 7f9aaeb80700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:50:53.475828 7f9aae37f700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
Linux hw03 3.15.0-031500rc2-generic #201404201435 SMP Sun Apr 20 18:36:18 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root@hw03:~# xfs_info /dev/sdd4
meta-data=/dev/sdd4              isize=2048   agcount=32, agsize=22587448 blks
         =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=722798336, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=352928, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

/dev/sdd4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,noquota 0 0

distr upgraded osd:
root@hw06:~# tail -f /var/log/ceph/ceph-osd.9.log
2014-05-02 18:51:48.385939 7fdb7a03a700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.444236 7fdb7a83b700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.586052 7fdb7a03a700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.627782 7fdb7a83b700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.636085 7fdb7a03a700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.652831 7fdb7a83b700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.686273 7fdb7a03a700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:48.752952 7fdb7a83b700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:49.361170 7fdb7a03a700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-02 18:51:49.744531 7fdb7a83b700  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-9) set_extsize: FSSETXATTR: (22) Invalid argument

root@hw06:~# xfs_info /dev/sda4
meta-data=/dev/sda4              isize=2048   agcount=32, agsize=22587512 blks
         =                       sectsz=4096  attr=2
data     =                       bsize=4096   blocks=722800384, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=352929, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

 cat /proc/mounts | grep ceph-9
/dev/sda4 /var/lib/ceph/osd/ceph-9 xfs rw,noatime,attr2,inode64,noquota 0 0

(Unfortunately pinguini/anton could not add this information him/herself, as account registration is still broken. See http://tracker.ceph.com/issues/7609 )

Actions
Copy link
 #3

Updated by Mark Nelson almost 10 years ago

Seeing this with master during RBD tiering tests only on the cache tier OSDs. Eventually during heavy RBD random 4MB write workloads the OSDs crash with heartbeat timeouts which may or may not be related.

Actions
Copy link
 #4

Updated by Dan Mick almost 10 years ago

  • Assignee set to Ilya Dryomov
  • Priority changed from Normal to High

Bumped pri, assigned to Ilya for investigation

Actions
Copy link
 #5

Updated by Michael Potter almost 10 years ago

I'm also seeing this error. Will try removing my RBD cache tier and see if it stops happening.

--- end dump of recent events ---
2014-05-21 16:46:47.368473 7fe104f007c0  0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-osd, pid 26032
2014-05-21 16:46:47.370012 7fe104f007c0  0 filestore(/var/lib/ceph/osd/ceph-7) mount detected xfs (libxfs)
2014-05-21 16:46:47.429592 7fe104f007c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is supported and appears to work
2014-05-21 16:46:47.429609 7fe104f007c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config optio
n
2014-05-21 16:46:47.802252 7fe104f007c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-05-21 16:46:47.802323 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_feature: extsize is supported
2014-05-21 16:46:47.971222 7fe104f007c0  0 filestore(/var/lib/ceph/osd/ceph-7) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2014-05-21 16:46:52.308030 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:52.309383 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:52.310689 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:52.312772 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument

...Snip repeat...
2014-05-21 16:46:52.595155 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:52.665014 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:52.665312 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:52.665601 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) set_extsize: FSSETXATTR: (22) Invalid argument
2014-05-21 16:46:53.708731 7fe104f007c0  0 filestore(/var/lib/ceph/osd/ceph-7) mount detected xfs (libxfs)
2014-05-21 16:46:53.764853 7fe104f007c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is supported and appears to work
2014-05-21 16:46:53.764861 7fe104f007c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-05-21 16:46:53.814570 7fe104f007c0  0 genericfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_features: syncfs(2) syscall fully supported (by glibc and kernel)
2014-05-21 16:46:53.814627 7fe104f007c0  0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-7) detect_feature: extsize is supported
2014-05-21 16:46:53.965018 7fe104f007c0  0 filestore(/var/lib/ceph/osd/ceph-7) mount: WRITEAHEAD journal mode explicitly enabled in conf
Actions
Copy link
 #6

Updated by Pierre Grandin almost 10 years ago

I am probably facing the same issue. Initially reported as #8439

Did any of you fix the issue on your own cluster? 75% of my cluster is down right now.

Actions
Copy link
 #7

Updated by Ilya Dryomov almost 10 years ago

Hi Pierre,

I don't think your crash (#8439) has anything to do with this. set_extsize is simply an allocation hint, EINVAL could be the result of an upgrade from older OSDs. On xfs, which is the only filesystem set_extsize supports, the hint can only be set if the file is empty. Now, rbd doesn't maintain any bitmap or suchlike, hence the hint op is sent unconditionally, which will trigger EINVAL if the underlying object happens to exist. That EINVAL is explicitly ignored higher on the stack though, so it's unlikely to be the problem.

I'll take a look at what is going on with tiering. Also, we should probably lower the log priority of this message.

Actions
Copy link
 #8

Updated by Ilya Dryomov almost 10 years ago

  • Status changed from New to Fix Under Review

With the fix applied we shouldn't see this message anymore, so I've
left the priority intact. The other known causes are a hint which is
bigger than half the size of the XFS AG or a misaligned hint, both of
which are something we'd want to notice.

Mark's problem with tiering may have been caused by the fact that hint
isn't propagated on cache flushes - its value isn't stored anywhere and
consequently it isn't reapplied during the flush.

Actions
Copy link
 #9

Updated by Ilya Dryomov almost 10 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #10

Updated by Sage Weil almost 10 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF