Project

General

Profile

Actions
Copy link

Bug #4545

closed

error creating empty object store. Invalid argument.

Added by Anonymous about 11 years ago. Updated over 9 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

While running a teuthology test, mkcephfs failed with this error:
INFO:teuthology.task.ceph:Running mkfs on osd nodes...
INFO:teuthology.task.ceph:{}
INFO:teuthology.task.ceph:0
INFO:teuthology.orchestra.run.err:2013-03-25 11:57:49.200303 7f3ce27cc780 1 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs - filestore_xattr_use_omap enabled
INFO:teuthology.orchestra.run.err:2013-03-25 11:57:49.967481 7f3ce27cc780 -1 provided osd id 0 != superblock's -1
INFO:teuthology.orchestra.run.err:2013-03-25 11:57:49.968781 7f3ce27cc780 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0: (22) Invalid argument
ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/mnt/ssd/git/teuthology/teuthology/contextutil.py", line 25, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/mnt/ssd/git/teuthology/teuthology/task/ceph.py", line 607, in cluster
'--monmap', '{tdir}/monmap'.format(tdir=testdir),
File "/mnt/ssd/git/teuthology/teuthology/orchestra/remote.py", line 42, in run
r = self._runner(client=self.ssh, **kwargs)
File "/mnt/ssd/git/teuthology/teuthology/orchestra/run.py", line 266, in run
r.exitstatus = check_status(r.exitstatus)
File "/mnt/ssd/git/teuthology/teuthology/orchestra/run.py", line 262, in _check_status
raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.133.20 with status 1: 'MALLOC_CHECK=3 /tmp/cephtest/buck@bender-2013-03-25_11-56-36/en

I'll upload the YAML file I was using and the entire output from the run in a minute.

Files

Download all files
found_bug.yaml (391 Bytes) found_bug.yaml Anonymous, 03/25/2013 12:02 PM
out.txt (90.4 KB) out.txt Anonymous, 03/25/2013 12:02 PM
Actions
Copy link Download all files
 #1

Updated by Anonymous about 11 years ago

Added the yaml file I was using (needs 3 locked hosts) and the teuthology output as attachments.

Actions
Copy link
 #2

Updated by Anonymous about 11 years ago

  • Assignee set to Anonymous

This may be failing due to a package signing issue that I thought had been resolved. I'll hold onto this ticket until I can tell either way.

Actions
Copy link
 #3

Updated by Anonymous about 11 years ago

  • Assignee deleted (Anonymous)

Alright, I no longer think the apache2 signature is related. This seems like a proper bug in its own right.

Actions
Copy link
 #4

Updated by Sage Weil about 11 years ago

  • Priority changed from Normal to Urgent

i've seen this regularly in the qa runs over the last week or so

Actions
Copy link
 #5

Updated by Sage Weil about 11 years ago

  • Assignee set to Sage Weil
Actions
Copy link
 #6

Updated by Sage Weil about 11 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link
 #7

Updated by tod chen over 9 years ago

i maybe found the problem.
before you mkcephfs,you should ensure the dir(/var/lib/ceph/osd/ceph-0) empty.
once i wrote a script:
#mount /dev/sd6 /var/lib/ceph/osd/ceph-2
#mount /dev/sd6 /var/lib/ceph/osd/ceph-3

you can see that i mount the one same device to two paths.

then everytime i execute the order, the path /var/lib/ceph/osd/ceph-3 will not be empty.

got the error:

2014-08-22 17:17:02.869346 7f99e9898780 -1 provided osd id 3 != superblock's -1
2014-08-22 17:17:02.870036 7f99e9898780 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3: (22) Invalid argument
what we need to do is:

  1. mount /dev/sda6 /var/lib/ceph/osd/ceph-2
  2. mount /dev/sda7 /var/lib/ceph/osd/ceph-3

Joe Buck wrote:

While running a teuthology test, mkcephfs failed with this error:
INFO:teuthology.task.ceph:Running mkfs on osd nodes...
INFO:teuthology.task.ceph:{}
INFO:teuthology.task.ceph:0
INFO:teuthology.orchestra.run.err:2013-03-25 11:57:49.200303 7f3ce27cc780 1 filestore(/var/lib/ceph/osd/ceph-0) limited size xattrs - filestore_xattr_use_omap enabled
INFO:teuthology.orchestra.run.err:2013-03-25 11:57:49.967481 7f3ce27cc780 -1 provided osd id 0 != superblock's -1
INFO:teuthology.orchestra.run.err:2013-03-25 11:57:49.968781 7f3ce27cc780 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0: (22) Invalid argument
ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/mnt/ssd/git/teuthology/teuthology/contextutil.py", line 25, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/mnt/ssd/git/teuthology/teuthology/task/ceph.py", line 607, in cluster
'--monmap', '{tdir}/monmap'.format(tdir=testdir),
File "/mnt/ssd/git/teuthology/teuthology/orchestra/remote.py", line 42, in run
r = self._runner(client=self.ssh, **kwargs)
File "/mnt/ssd/git/teuthology/teuthology/orchestra/run.py", line 266, in run
r.exitstatus = check_status(r.exitstatus)
File "/mnt/ssd/git/teuthology/teuthology/orchestra/run.py", line 262, in _check_status
raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.133.20 with status 1: 'MALLOC_CHECK=3 /tmp/cephtest/buck@bender-2013-03-25_11-56-36/en

I'll upload the YAML file I was using and the entire output from the run in a minute.

Actions
Copy link

Also available in: Atom PDF