Bug #59335: Found coredumps on smithi related to sqlite3 - cephsqlite - Ceph

Actions

Copy link

Bug #59335

open

Found coredumps on smithi related to sqlite3

Added by Laura Flores about 1 year ago. Updated 3 months ago.

Status:

New

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7223995

2023-03-29T04:48:18.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo
2023-03-29T04:48:18.361 INFO:teuthology.task.internal:Restoring /etc/sudoers...
2023-03-29T04:48:18.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
2023-03-29T04:48:18.373 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers
2023-03-29T04:48:18.384 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
2023-03-29T04:48:18.403 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2023-03-29T04:48:18.415 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2023-03-29T04:48:18.423 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core
2023-03-29T04:48:18.436 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core
2023-03-29T04:48:18.467 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump
2023-03-29T04:48:18.490 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi052.front.sepia.ceph.com, flagging run as failed
2023-03-29T04:48:18.490 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump
2023-03-29T04:48:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-03-29T04:48:18.512 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2023-03-29T04:48:18.524 INFO:teuthology.task.internal:Transferring archived files...
2023-03-29T04:48:18.524 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7223995/remote/smithi052
2023-03-29T04:48:18.525 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .
2023-03-29T04:48:25.303 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2023-03-29T04:48:25.513 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7223995/remote/smithi052/coredump/1680065090.39304.core.gz: gzip compressed data, was "1680065090.39304.core", last modified: Wed Mar 29 04:48:18 2023, max compression

2023-03-29T04:48:25.514 INFO:teuthology.task.internal:core is compressed, try accessing gzip file ...
2023-03-29T04:48:26.160 INFO:teuthology.task.internal: core looks like: /tmp/tmpsr59t_lu: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'sqlite3 -cmd .output /dev/null -cmd .load libcephsqlite.so -cmd pragma journal_', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/sqlite3', platform: 'x86_64'

2023-03-29T04:48:26.190 INFO:teuthology.task.internal: dump_program: sqlite3 -cmd .output /dev/null -cmd .load libcephsqlite.so -cmd pragma journal_
2023-03-29T04:48:26.191 DEBUG:teuthology.orchestra.run.smithi052:> which 'sqlite3 -cmd .output /dev/null -cmd .load libcephsqlite.so -cmd pragma journal_'
2023-03-29T04:48:26.199 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-03-29T04:48:26.200 ERROR:teuthology.run_tasks:Manager failed: internal.archive

There is a core file located at /a/yuriw-2023-03-28_22:43:59-rados-wip-yuri11-testing-2023-03-28-0950-distro-default-smithi/7223995/remote/smithi052/coredump/1680065090.39304.core.gz.

The coredump was not generated by any daemon, and there was no executable. So, I checked the details with `file`. Looks like it has something to do with sqlite3. This is also reflected in the above teuthology log.

~$ file 1680065090.39304.core
1680065090.39304.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'sqlite3 -cmd .output /dev/null -cmd .load libcephsqlite.so -cmd pragma journal_', real uid: 1000, effective uid: 1000, real gid: 1271, effective gid: 1271, execfn: '/usr/bin/sqlite3', platform: 'x86_64'

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Laura Flores 12 months ago

/a/yuriw-2023-04-25_21:30:50-rados-wip-yuri3-testing-2023-04-25-1147-distro-default-smithi/7253347

Actions

Copy link

Updated by Laura Flores 12 months ago

Assignee set to Patrick Donnelly

Patrick would you be the person to go to for this?

Actions

Copy link

Updated by Patrick Donnelly 8 months ago

The common theme here is that atexit cleanup of sqlite3 is causing segmentation faults or:

2023-08-21T20:21:24.873 INFO:tasks.workunit.client.0.smithi141.stderr:Error: near line 3: stepping, disk I/O error (10)                                                                       
2023-08-21T20:21:24.885 INFO:tasks.workunit.client.0.smithi141.stderr:terminate called after throwing an instance of 'std::runtime_error'                                                     
2023-08-21T20:21:24.885 INFO:tasks.workunit.client.0.smithi141.stderr:  what():  EVP_DecryptInit_ex failed                                                                                    
2023-08-21T20:21:34.850 INFO:tasks.workunit.client.0.smithi141.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_libcephsqlite.sh: line 129: 40508 Aborted                 (core dumped) sqlite3 -cmd '.output /dev/null' -cmd '.load libcephsqlite.so' -cmd 'pragma journal_mode = PERSIST' -cmd ".open file:///$pool:$ns/baz.db?vfs=ceph" -cmd '.output stdout' <<< "$a"

/teuthology/pdonnell-2023-08-21_19:21:49-rados:basic-main-distro-default-smithi/7374185/teuthology.log

I've not isolated the cause yet and it's probably some kind of memory corruption :/

It's decidedly just an annoyance because this only happens with atexit cleanup.

Actions

Copy link