Project

General

Profile

Actions
Copy link

Bug #13492

closed

adjust-ulimits ceph-coverage command fails when try to kill process

Added by bo cai over 8 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
High
Assignee:
bo cai
Category:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

I have try many times, teuthology always failed when execute command "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3"

In fact , my command is ". ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3"

I just want to load the user's environment , such as HTTP proxy.

Files

Download all files
teuthology.log (447 KB) teuthology.log Complete_log bo cai, 10/15/2015 12:08 AM
teuthology-2015-10-16.log (526 KB) teuthology-2015-10-16.log bo cai, 10/16/2015 02:14 AM
teuthology-2015-10-16-1113.log (539 KB) teuthology-2015-10-16-1113.log other log bo cai, 10/16/2015 03:18 AM
Actions
Copy link
 #1

Updated by bo cai over 8 years ago

2015-10-13T23:36:55.306 INFO:teuthology.orchestra.run.plana020:Running: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f -i a'
2015-10-13T23:36:55.309 INFO:tasks.ceph.mon.a:Started
2015-10-13T23:36:55.309 INFO:tasks.ceph:Setting crush tunables to default
2015-10-13T23:36:55.310 INFO:teuthology.orchestra.run.plana019:Running: '. ~/.profile ; sudo ceph osd crush tunables default'
2015-10-13T23:36:55.546 INFO:tasks.ceph.mon.2.plana017.stdout:starting mon.2 rank 0 at 172.16.36.102:6789/0 mon_data /var/lib/ceph/mon/ceph-2 fsid 6332a9b1-fc56-4a2c-9803-1215f438ddd9
2015-10-13T23:36:55.572 INFO:tasks.ceph.mon.a.plana020.stdout:starting mon.a rank 3 at 172.16.36.105:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 6332a9b1-fc56-4a2c-9803-1215f438ddd9
2015-10-13T23:36:55.580 INFO:tasks.ceph.mon.1.plana021.stdout:starting mon.1 rank 4 at 172.16.36.106:6789/0 mon_data /var/lib/ceph/mon/ceph-1 fsid 6332a9b1-fc56-4a2c-9803-1215f438ddd9
2015-10-13T23:36:55.597 INFO:tasks.ceph.mon.b.plana018.stdout:starting mon.b rank 1 at 172.16.36.103:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 6332a9b1-fc56-4a2c-9803-1215f438ddd9
2015-10-13T23:36:55.597 INFO:tasks.ceph.mon.c.plana018.stdout:starting mon.c rank 5 at 172.16.36.103:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 6332a9b1-fc56-4a2c-9803-1215f438ddd9
2015-10-13T23:36:55.691 INFO:tasks.ceph.mon.0.plana019.stdout:starting mon.0 rank 2 at 172.16.36.104:6789/0 mon_data /var/lib/ceph/mon/ceph-0 fsid 6332a9b1-fc56-4a2c-9803-1215f438ddd9
2015-10-13T23:37:06.028 INFO:teuthology.orchestra.run.plana019.stderr:adjusted tunables profile to default
2015-10-13T23:37:06.040 INFO:tasks.ceph:Starting osd daemons...
2015-10-13T23:37:06.040 INFO:tasks.ceph.osd.1:Restarting daemon
2015-10-13T23:37:06.041 INFO:teuthology.orchestra.run.plana021:Running: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1'
2015-10-13T23:37:06.044 INFO:tasks.ceph.osd.1:Started
2015-10-13T23:37:06.044 INFO:tasks.ceph.osd.0:Restarting daemon
2015-10-13T23:37:06.045 INFO:teuthology.orchestra.run.plana019:Running: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 0'
2015-10-13T23:37:06.048 INFO:tasks.ceph.osd.0:Started
2015-10-13T23:37:06.048 INFO:tasks.ceph.osd.2:Restarting daemon
2015-10-13T23:37:06.048 INFO:teuthology.orchestra.run.plana017:Running: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2'
2015-10-13T23:37:06.051 INFO:tasks.ceph.osd.2:Started
2015-10-13T23:37:06.052 DEBUG:tasks.ceph.osd.2:waiting for process to exit
2015-10-13T23:37:06.079 INFO:tasks.ceph.osd.1.plana021.stdout:starting osd.1 at :/0 osd_data /var/lib/ceph/osd/ceph-1 /var/lib/ceph/osd/ceph-1/journal
2015-10-13T23:37:06.088 INFO:tasks.ceph.osd.0.plana019.stdout:starting osd.0 at :/0 osd_data /var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2015-10-13T23:37:06.125 INFO:tasks.ceph.osd.1.plana021.stderr:2015-10-13 11:37:06.124277 7f36ad80a980 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2015-10-13T23:37:06.131 INFO:tasks.ceph.osd.0.plana019.stderr:2015-10-13 11:37:06.131467 7f335790f980 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2015-10-13T23:37:06.261 INFO:tasks.ceph.osd.1.plana021.stderr:2015-10-13 11:37:06.260071 7f36ad80a980 -1 osd.1 0 log_to_monitors {default=true}
2015-10-13T23:37:06.335 INFO:tasks.ceph.osd.0.plana019.stderr:2015-10-13 11:37:06.335769 7f335790f980 -1 osd.0 0 log_to_monitors {default=true}
2015-10-13T23:37:06.357 INFO:tasks.ceph.osd.0.plana019.stderr:2015-10-13 11:37:06.356979 7f335790f980 -1 [0;31m * ERROR: osd init failed: (1) Operation not permitted[0m
2015-10-13T23:37:06.480 INFO:tasks.ceph.osd.0.plana019.stderr:daemon-helper: command failed with exit status 1
2015-10-13T23:37:12.052 INFO:tasks.ceph.osd.2:Stopped
2015-10-13T23:37:12.052 INFO:tasks.ceph.osd.2:Restarting daemon
2015-10-13T23:37:12.053 INFO:teuthology.orchestra.run.plana018:Running: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 2'
2015-10-13T23:37:12.057 INFO:tasks.ceph.osd.2:Started
2015-10-13T23:37:12.057 INFO:tasks.ceph.osd.3:Restarting daemon
2015-10-13T23:37:12.057 INFO:teuthology.orchestra.run.plana018:Running: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 3'
2015-10-13T23:37:12.060 INFO:tasks.ceph.osd.3:Started
2015-10-13T23:37:12.061 DEBUG:tasks.ceph.osd.0:waiting for process to exit
2015-10-13T23:37:12.061 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in enter
return self.gen.next()
File "/home/teuthworker/src/ceph-qa-suite_master/tasks/ceph.py", line 1026, in run_daemon
wait=False,
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/daemon.py", line 163, in add_daemon
self.daemons[role][id_].stop()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/daemon.py", line 45, in stop
run.wait([self.proc], timeout=timeout)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 402, in wait
proc.wait()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 115, in wait
label=self.label)
CommandFailedError: Command failed on plana019 with status 1: '. ~/.profile ; sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 0'
2015-10-13T23:37:12.063 INFO:teuthology.misc:Shutting down mon daemons...
2015-10-13T23:37:12.063 DEBUG:tasks.ceph.mon.a:waiting for process to exit
2015-10-13T23:37:12.099 INFO:tasks.ceph.osd.2.plana018.stdout:starting osd.2 at :/0 osd_data /var/lib/ceph/osd/ceph-2 /var/lib/ceph/osd/ceph-2/journal
2015-10-13T23:37:12.100 INFO:tasks.ceph.osd.3.plana018.stdout:starting osd.3 at :/0 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
2015-10-13T23:37:12.148 INFO:tasks.ceph.osd.3.plana018.stderr:2015-10-13 11:37:12.146833 7f6c6c9d6980 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2015-10-13T23:37:12.149 INFO:tasks.ceph.osd.2.plana018.stderr:2015-10-13 11:37:12.146935 7f24befc3980 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway
2015-10-13T23:37:12.312 INFO:tasks.ceph.osd.3.plana018.stderr:2015-10-13 11:37:12.310574 7f6c6c9d6980 -1 osd.3 0 log_to_monitors {default=true}
2015-10-13T23:37:12.313 INFO:tasks.ceph.osd.2.plana018.stderr:2015-10-13 11:37:12.310506 7f24befc3980 -1 osd.2 0 log_to_monitors {default=true}
2015-10-13T23:37:12.335 INFO:tasks.ceph.osd.2.plana018.stderr:2015-10-13 11:37:12.333853 7f24befc3980 -1 [0;31m * ERROR: osd init failed: (1) Operation not permitted[0m
2015-10-13T23:37:12.507 INFO:tasks.ceph.osd.2.plana018.stderr:daemon-helper: command failed with exit status 1
2015-10-13T23:37:18.061 INFO:tasks.ceph.mon.a:Stopped
2015-10-13T23:37:18.061 DEBUG:tasks.ceph.mon.c:waiting for process to exit
2015-10-13T23:37:24.062 INFO:tasks.ceph.mon.c:Stopped
2015-10-13T23:37:24.063 DEBUG:tasks.ceph.mon.b:waiting for process to exit
2015-10-13T23:37:30.063 INFO:tasks.ceph.mon.b:Stopped
2015-10-13T23:37:30.064 DEBUG:tasks.ceph.mon.1:waiting for process to exit
2015-10-13T23:37:36.064 INFO:tasks.ceph.mon.1:Stopped
2015-10-13T23:37:36.064 DEBUG:tasks.ceph.mon.0:waiting for process to exit
2015-10-13T23:37:42.064 INFO:tasks.ceph.mon.0:Stopped
2015-10-13T23:37:42.065 DEBUG:tasks.ceph.mon.2:waiting for process to exit
2015-10-13T23:37:48.065 INFO:tasks.ceph.mon.2:Stopped

Actions
Copy link
 #2

Updated by bo cai over 8 years ago

Here is the teuthology Complete log, I run it at this morning.

If you need any other info, please let me know.

Actions
Copy link
 #3

Updated by Zack Cerza over 8 years ago

  • Status changed from New to Need More Info
  • Assignee set to bo cai

Looks like you're running a modified teuthology. I'd need to see the diff.

Actions
Copy link
 #4

Updated by bo cai over 8 years ago

https://github.com/caibo2014/teuthology/commit/8ef04fa19428d3441d9edbd5bad3ba04a8223dee

and also, I change node's
/etc/sudoers:
Defaults env_reset --> Defaults !env_rese

/home/ubuntu/.profile
 export http_proxy=my http proxy
Actions
Copy link
 #5

Updated by bo cai over 8 years ago

And at this morning, I run teuthology use https://github.com/ceph/teuthology, some error happened.
I will upload the log later.

Actions
Copy link
 #6

Updated by bo cai over 8 years ago

here is my log that use ceph's teuthology, some error happened.

Actions
Copy link
 #7

Updated by bo cai over 8 years ago

here is some log of other error, it may help you.

what profile should I give to the ubuntu user?
now I just give ubuntu passwordless sudo access.
but it always fail when ubuntu try to use ceph command or linxu filesystem command.

Actions
Copy link
 #8

Updated by bo cai over 8 years ago

Zack Cerza wrote:

Looks like you're running a modified teuthology. I'd need to see the diff.

Hi, I reset my change and run again, also some error happened, you can see it in teuthology-2015-10-16-1113.log

thanks for help.

Actions
Copy link
 #9

Updated by bo cai over 8 years ago

I have solved this problem is due to the result of our test environment , please turn off this questionnaire

Actions
Copy link
 #10

Updated by Zack Cerza about 8 years ago

  • Status changed from Need More Info to Closed
Actions
Copy link

Also available in: Atom PDF