Project

General

Profile

Bug #49936

ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024)

Added by Yanhu Cao 30 days ago. Updated 12 days ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

   -30> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12239: bh[ 0x7f060422df50 12239~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12239~1
   -29> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12240: bh[ 0x7f05fc4fcdb0 12240~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12240~5
   -28> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12245: bh[ 0x7f060422e110 12245~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12245~1
   -27> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12246: bh[ 0x7f05fc4fdf50 12246~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12246~5
   -26> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12251: bh[ 0x7f060422e2d0 12251~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12251~1
   -25> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12252: bh[ 0x7f05fc4ff0f0 12252~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12252~5
   -24> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12257: bh[ 0x7f060422e490 12257~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12257~1
   -23> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12258: bh[ 0x7f05fc500290 12258~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12258~5
   -22> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12263: bh[ 0x7f060422e650 12263~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12263~1
   -21> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12264: bh[ 0x7f05fc501430 12264~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12264~5
   -20> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12269: bh[ 0x7f060422e810 12269~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12269~1
   -19> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12270: bh[ 0x7f05fc5025d0 12270~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12270~5
   -18> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12275: bh[ 0x7f060422e9d0 12275~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12275~1
   -17> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12276: bh[ 0x7f05fc503770 12276~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12276~5
   -16> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12281: bh[ 0x7f060422eb90 12281~1 0x7f05fc024df0 (1) v 0 clean firstbyte=0] waiters = {} +0 frag 0~12287 +12281~1
   -15> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx rmap opos 12282: bh[ 0x7f060414ecf0 12282~5 0x7f05fc024df0 (5) v 0 dirty firstbyte=97] waiters = {} +0 frag 0~12287 +12282~5
   -14> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx has all buffers
   -13> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx  adding buffer len 12287 at 0
   -12> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher readx  result is 12287
   -11> 2021-03-22T17:16:07.053+0800 7f06257fa700 20 objectcacher readx done 0x7f060405b8c0 12287
   -10> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher trim  start: bytes: max 209715200  clean 2047, objects: max 1000 current 1
    -9> 2021-03-22T17:16:07.053+0800 7f06257fa700 10 objectcacher trim finish:  max 209715200  clean 2047, objects: max 1000 current 1
    -8> 2021-03-22T17:16:07.053+0800 7f0611ffb700  3 client.24187 ll_read 0x7f05fc023a70 0~12288 = 12287
    -7> 2021-03-22T17:16:07.053+0800 7f061bfff700  1 client.24187.objecter ms_handle_reset 0x7f05fc025650 session 0x7f05fc021e10 osd.1
    -6> 2021-03-22T17:16:07.053+0800 7f061bfff700 10 monclient: _renew_subs
    -5> 2021-03-22T17:16:07.053+0800 7f061bfff700 10 monclient: _send_mon_message to mon.b at v2:10.202.42.175:40417/0
    -4> 2021-03-22T17:16:07.053+0800 7f063485f700 10 monclient: get_auth_request con 0x7f061c009b00 auth_method 0
    -3> 2021-03-22T17:16:07.077+0800 7f0611ffb700 -1 /home/caoyanhu/ceph/src/include/buffer.h: In function 'void ceph::buffer::v15_2_0::list::prepare_iov(VectorT*) const [with VectorT = std::vector<iovec>]' thread 7f0611ffb700 time 2021-03-22T17:16:07.056229+0800
/home/caoyanhu/ceph/src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024)

 ceph version 16.0.0-11573-ge4cff916f2 (e4cff916f20752cc03931d6c4da847993da0272e) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f063aa3fd7e]
 2: /home/caoyanhu/ceph/build/lib/libceph-common.so.2(+0x169c000) [0x7f063aa40000]
 3: (void ceph::buffer::v15_2_0::list::prepare_iov<std::vector<iovec, std::allocator<iovec> > >(std::vector<iovec, std::allocator<iovec> >*) const+0x3a) [0x5559b4806dd4]
 4: bin/ceph-fuse(+0x488f52) [0x5559b4801f52]
 5: /lib/x86_64-linux-gnu/libfuse3.so.3(+0x15626) [0x7f063bab4626]
 6: /lib/x86_64-linux-gnu/libfuse3.so.3(+0x17626) [0x7f063bab6626]
 7: /lib/x86_64-linux-gnu/libfuse3.so.3(+0x13091) [0x7f063bab2091]
 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f063904afa3]
 9: clone()

    -2> 2021-03-22T17:16:07.077+0800 7f061bfff700  1 client.24187.objecter ms_handle_reset 0x7f061c009b00 session 0x7f05fc021e10 osd.1
    -1> 2021-03-22T17:16:07.077+0800 7f063485f700 10 monclient: get_auth_request con 0x7f05fc025650 auth_method 0
     0> 2021-03-22T17:16:07.085+0800 7f0611ffb700 -1 *** Caught signal (Aborted) **
 in thread 7f0611ffb700 thread_name:ceph-fuse
(gdb) bt
#0  raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00005559b4998671 in reraise_fatal (signum=6) at /home/caoyanhu/ceph/src/global/signal_handler.cc:87
#2  0x00005559b4999aab in handle_fatal_signal (signum=6) at /home/caoyanhu/ceph/src/global/signal_handler.cc:332
#3  <signal handler called>
#4  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#5  0x00007f0638905535 in __GI_abort () at abort.c:79
#6  0x00007f063aa3ff4d in ceph::__ceph_assert_fail (assertion=0x5559b4aa9837 "_num <= 1024", file=0x5559b4aa9848 "/home/caoyanhu/ceph/src/include/buffer.h", line=1187,
    func=0x5559b4aabac0 <ceph::buffer::v15_2_0::list::prepare_iov<std::vector<iovec, std::allocator<iovec> > >(std::vector<iovec, std::allocator<iovec> >*) const::__PRETTY_FUNCTION__> "void ceph::buffer::v15_2_0::list::prepare_iov(VectorT*) const [with VectorT = std::vector<iovec>]") at /home/caoyanhu/ceph/src/common/assert.cc:75
#7  0x00007f063aa40000 in ceph::__ceph_assert_fail (ctx=...) at /home/caoyanhu/ceph/src/common/assert.cc:80
#8  0x00005559b4806dd4 in ceph::buffer::v15_2_0::list::prepare_iov<std::vector<iovec, std::allocator<iovec> > > (this=0x7f0611ff9320, piov=0x7f0611ff9300) at /home/caoyanhu/ceph/src/include/buffer.h:1187
#9  0x00005559b4801f52 in fuse_ll_read (req=0x7f0604026b00, ino=1099511627776, size=12288, off=0, fi=0x7f0611ff9370) at /home/caoyanhu/ceph/src/client/fuse_ll.cc:682
#10 0x00007f063bab4626 in do_read (req=<optimized out>, nodeid=<optimized out>, inarg=<optimized out>) at ../lib/fuse_lowlevel.c:1305
#11 0x00007f063bab6626 in fuse_session_process_buf_int (se=0x5559b5290370, buf=buf@entry=0x7f05f032a430, ch=<optimized out>) at ../lib/fuse_lowlevel.c:2579
#12 0x00007f063bab2091 in fuse_do_work (data=0x7f05f032a410) at ../lib/fuse_loop_mt.c:163
#13 0x00007f063904afa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#14 0x00007f06389dc4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) f 8
#8  0x00005559b4806dd4 in ceph::buffer::v15_2_0::list::prepare_iov<std::vector<iovec, std::allocator<iovec> > > (this=0x7f0611ff9320, piov=0x7f0611ff9300) at /home/caoyanhu/ceph/src/include/buffer.h:1187
1187          ceph_assert(_num <= IOV_MAX);
(gdb) p _buffers
$1 = {_root = {next = 0x7f05fc0220c0}, _tail = 0x7f05fcd49530}
(gdb) p _num
$2 = 4095
(gdb)

reproduce source code(rw.c)

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main(int argc, char *argv[])
{
        char path[] = "/mnt/fuse/test.txt";
        int fd;
        int i;
        ssize_t ret;
        char arr[] = "asdf";
        char buf[1024];

        fd = open(path, O_CREAT | O_RDWR, 0644);
        if (fd < 0) {
                return -1;
        }

        for (i = 0; i < 2048; i++) {
                ret = write(fd, arr, sizeof(arr));

                lseek(fd, 1, SEEK_CUR);
        }

        lseek(fd, 0, SEEK_SET);

        ret = read(fd, buf, sizeof(buf));

        close(fd);

        return 0;
}


Related issues

Copied to CephFS - Backport #50022: pacific: ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024) Resolved
Copied to CephFS - Backport #50023: nautilus: ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024) Rejected
Copied to CephFS - Backport #50024: octopus: ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024) Need More Info

History

#1 Updated by Patrick Donnelly 29 days ago

  • Status changed from New to Fix Under Review
  • Assignee set to Yanhu Cao
  • Target version set to v17.0.0
  • Source set to Community (dev)
  • Backport set to pacific,octopus,nautilus
  • Pull request ID set to 40326
  • Component(FS) Client added

#2 Updated by Patrick Donnelly 26 days ago

  • Status changed from Fix Under Review to Pending Backport

#3 Updated by Backport Bot 26 days ago

  • Copied to Backport #50022: pacific: ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024) added

#4 Updated by Backport Bot 26 days ago

  • Copied to Backport #50023: nautilus: ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024) added

#5 Updated by Backport Bot 26 days ago

  • Copied to Backport #50024: octopus: ceph-fuse: src/include/buffer.h: 1187: FAILED ceph_assert(_num <= 1024) added

#6 Updated by Nathan Cutler 12 days ago

@Patrick, you asked for this bugfix to be backported all the way back to nautilus, but AFAICT the code containing the bugfix does not exist in octopus or nautilus. Can you take a look?

Also available in: Atom PDF