Project

General

Profile

Actions
Copy link

Bug #11142

closed

"Exception: ceph-objectstore-tool: import failure" in upgrade:firefly-x-hammer-distro-basic-multi run (TCMALLOC)

Added by David Zafman about 9 years ago. Updated over 8 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
David Zafman
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/firefly, upgrade/firefly-x
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-03-16_17:00:02-upgrade:firefly-firefly-distro-basic-vps/
Job: 806521
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-16_17:00:02-upgrade:firefly-firefly-distro-basic-vps/806521/teuthology.log

2015-03-16T19:46:46.363 INFO:teuthology.orchestra.run.vpm037.stdout:remove_coll FORREMOVAL_0_2.15
2015-03-16T19:46:46.364 INFO:teuthology.orchestra.run.vpm037.stdout:Remove successful
2015-03-16T19:46:46.370 INFO:teuthology.orchestra.run.vpm037:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op import --file /home/ubuntu/cephtest/data/exp.2.15.1'
2015-03-16T19:46:46.412 INFO:teuthology.orchestra.run.vpm037.stdout:Importing pgid 2.15
2015-03-16T19:46:46.417 INFO:teuthology.orchestra.run.vpm037.stdout:Import successful
2015-03-16T19:46:46.421 INFO:teuthology.orchestra.run.vpm037.stderr:*** Caught signal (Segmentation fault) **
2015-03-16T19:46:46.422 INFO:teuthology.orchestra.run.vpm037.stderr: in thread 7fb4dedf3780
2015-03-16T19:46:46.447 INFO:teuthology.orchestra.run.vpm037.stderr: ceph version 0.80.9-183-g493d285 (493d285508914769cba3639b601ae6c20303af0d)
2015-03-16T19:46:46.447 INFO:teuthology.orchestra.run.vpm037.stderr: 1: ceph-objectstore-tool() [0x9c0e7a]
2015-03-16T19:46:46.447 INFO:teuthology.orchestra.run.vpm037.stderr: 2: (()+0xfcb0) [0x7fb4de320cb0]
2015-03-16T19:46:46.448 INFO:teuthology.orchestra.run.vpm037.stderr: 3: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)+0x93) [0x7fb4de99ede3]
2015-03-16T19:46:46.448 INFO:teuthology.orchestra.run.vpm037.stderr: 4: (tcmalloc::ThreadCache::Scavenge()+0x44) [0x7fb4de99f0b4]
2015-03-16T19:46:46.448 INFO:teuthology.orchestra.run.vpm037.stderr: 5: (tc_free()+0x24b) [0x7fb4de9aae8b]
2015-03-16T19:46:46.449 INFO:teuthology.orchestra.run.vpm037.stderr: 6: (()+0x3b618) [0x7fb4dba9f618]
2015-03-16T19:46:46.449 INFO:teuthology.orchestra.run.vpm037.stderr: 7: (()+0x3b635) [0x7fb4dba9f635]
2015-03-16T19:46:46.449 INFO:teuthology.orchestra.run.vpm037.stderr: 8: (__libc_start_main()+0xf4) [0x7fb4dba85774]
2015-03-16T19:46:46.450 INFO:teuthology.orchestra.run.vpm037.stderr: 9: ceph-objectstore-tool() [0x616699]
2015-03-16T19:53:30.847 INFO:tasks.workunit.client.0.vpm071.stdout:stopping iogen
2015-03-16T19:53:31.293 INFO:tasks.workunit.client.0.vpm071.stdout:OK
2015-03-16T19:53:35.707 INFO:teuthology.orchestra.run.vpm071:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2015-03-16T19:53:36.930 INFO:tasks.workunit:Stopping suites/iogen.sh on client.0...
2015-03-16T19:53:36.930 INFO:teuthology.orchestra.run.vpm071:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2015-03-16T19:53:37.028 DEBUG:teuthology.parallel:result is None
2015-03-16T19:53:37.028 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients...
2015-03-16T19:53:37.028 INFO:teuthology.orchestra.run.vpm071:Running: 'sudo fusermount -u /home/ubuntu/cephtest/mnt.0'
2015-03-16T19:53:37.197 INFO:tasks.ceph_fuse.ceph-fuse.0.vpm071.stderr:ceph-fuse[22581]: fuse finished with error 0
2015-03-16T19:53:46.142 INFO:teuthology.orchestra.run.vpm071:Running: 'rmdir -- /home/ubuntu/cephtest/mnt.0'
2015-03-16T19:53:46.156 INFO:tasks.thrashosds:joining thrashosds
2015-03-16T19:53:46.157 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/sequential.py", line 55, in task
    mgr.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/thrashosds.py", line 183, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/ceph_manager.py", line 353, in do_join
    self.thread.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
    raise self._exception
Exception: ceph-objectstore-tool: import failure with status 139
2015-03-16T19:53:46.252 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/search?q=7a641f494d7046c6ab74780edbf191a2
Exception: ceph-objectstore-tool: import failure with status 139
2015-03-16T19:53:46.252 DEBUG:teuthology.run_tasks:Unwinding manager install.upgrade

Related issues 2 (0 open — 2 closed)

Has duplicate Ceph - Bug #11513: "ceph-objectstore-tool: import failure with status 139" in upgrade:firefly-x-hammer-distro-basic-vps runDuplicate04/30/2015

Actions
Copied from Ceph - Bug #11139: "Exception: ceph-objectstore-tool: import failure" in upgrade:firefly-x-hammer-distro-basic-multi runResolvedDavid Zafman03/17/2015

Actions
Actions
Copy link
 #1

Updated by David Zafman about 9 years ago

  • Subject changed from "Exception: ceph-objectstore-tool: import failure" in upgrade:firefly-x-hammer-distro-basic-multi run to "Exception: ceph-objectstore-tool: import failure" in upgrade:firefly-x-hammer-distro-basic-multi run (TCMALLOC)
Actions
Copy link
 #2

Updated by David Zafman about 9 years ago

  • Assignee set to David Zafman
Actions
Copy link
 #3

Updated by Sage Weil about 9 years ago

  • Priority changed from Urgent to High
Actions
Copy link
 #5

Updated by Loïc Dachary almost 9 years ago

description: rados/thrash/{clusters/fixed-2.yaml fs/xfs.yaml msgr-failures/osd-delay.yaml thrashers/morepggrow.yaml workloads/ec-readwrite.yaml}

The workload completed successfully, the error happens after, as shown in the log snippet below.

2015-05-15T03:49:55.925 INFO:tasks.rados.rados.0.plana74.stdout:3999:  finishing write tid 2 to plana746502-300
2015-05-15T03:49:55.925 INFO:tasks.rados.rados.0.plana74.stdout:3999:  finishing write tid 4 to plana746502-300
2015-05-15T03:49:55.929 INFO:tasks.rados.rados.0.plana74.stdout:3999:  finishing write tid 5 to plana746502-300
2015-05-15T03:49:55.929 INFO:tasks.rados.rados.0.plana74.stdout:update_object_version oid 300 v 152 (ObjNum 2415 snap 0 seq_num 2415) dirty exists
2015-05-15T03:49:55.930 INFO:tasks.rados.rados.0.plana74.stdout:3999: done (0 left)
2015-05-15T03:49:55.931 INFO:tasks.rados.rados.0.plana74.stderr:0 errors.
2015-05-15T03:49:55.942 INFO:tasks.rados.rados.0.plana74.stderr:
2015-05-15T03:49:55.946 INFO:tasks.thrashosds.ceph_manager:removing pool_name unique_pool_0
2015-05-15T03:49:55.946 INFO:teuthology.orchestra.run.plana74:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados rmpool unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it'
2015-05-15T03:49:56.189 INFO:teuthology.orchestra.run.plana74.stdout:successfully deleted pool unique_pool_0
2015-05-15T03:49:56.191 INFO:tasks.thrashosds:joining thrashosds
2015-05-15T03:49:56.191 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 125, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/thrashosds.py", line 183, in task
    thrash_proc.do_join()
  File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/ceph_manager.py", line 356, in do_join
    self.thread.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
    raise self._exception
Exception: ceph-objectstore-tool: import failure with status 139
Actions
Copy link
 #6

Updated by Nathan Cutler almost 9 years ago

We have started running integration tests on the newly-reset firefly-backports branch (containing what will become 0.80.11) and one of the jobs in the rados suite threw a very similar error (see first Traceback in the log).

Actions
Copy link
 #7

Updated by Yuri Weinstein over 8 years ago

Run: http://pulpito.ceph.com/teuthology-2015-09-29_17:00:08-upgrade:firefly-firefly-distro-basic-vps/
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-09-29_17:00:08-upgrade:firefly-firefly-distro-basic-vps/1076618/teuthology.log

2015-09-29T18:12:34.195 INFO:teuthology.orchestra.run.vpm142.stdout:Write f407769/10000000fd6.00000000/head//1
2015-09-29T18:12:34.198 INFO:teuthology.orchestra.run.vpm142.stdout:Write 309111a9/200.00000047/head//1
2015-09-29T18:12:34.233 INFO:teuthology.orchestra.run.vpm142.stdout:Write 553955e9/10000000e37.00000000/head//1
2015-09-29T18:12:35.308 INFO:teuthology.orchestra.run.vpm142.stdout:Import successful
2015-09-29T18:12:35.499 INFO:teuthology.orchestra.run.vpm142.stderr:*** Caught signal (Segmentation fault) **
2015-09-29T18:12:35.499 INFO:teuthology.orchestra.run.vpm142.stderr: in thread 7f71fa17f780
2015-09-29T18:19:53.082 INFO:tasks.workunit.client.0.vpm121.stdout:stopping iogen
2015-09-29T18:19:53.252 INFO:tasks.workunit.client.0.vpm121.stdout:OK
2015-09-29T18:19:56.821 INFO:teuthology.orchestra.run.vpm121:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp'
2015-09-29T18:19:57.649 INFO:tasks.workunit:Stopping suites/iogen.sh on client.0...
2015-09-29T18:19:57.650 INFO:teuthology.orchestra.run.vpm121:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
Actions
Copy link
 #9

Updated by Yuri Weinstein over 8 years ago

  • Release set to firefly
Actions
Copy link
 #10

Updated by Samuel Just over 8 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF