Project

General

Profile

Actions
Copy link

Bug #53220

closed

notification tests failing: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

Added by Casey Bodley over 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Yuval Lifshitz
Target version:
-
% Done:

100%

Source:
Tags:
notification kakfa backport_processed
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
43995
Crash signature (v1):
Crash signature (v2):

Description

2021-11-10T19:01:40.487 INFO:teuthology.task.internal:Tidying up after the test...
2021-11-10T19:01:40.488 DEBUG:teuthology.orchestra.run.smithi129:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2021-11-10T19:01:40.490 DEBUG:teuthology.orchestra.run.smithi188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2021-11-10T19:01:40.501 INFO:teuthology.orchestra.run.smithi129.stdout:   394301      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Nov 10 19:01 /home/ubuntu/cephtest
2021-11-10T19:01:40.501 INFO:teuthology.orchestra.run.smithi129.stdout:  1446708      4 drwxrwxr-x   3  ubuntu   ubuntu       4096 Nov 10 19:01 /home/ubuntu/cephtest/kafka-2.6.0-src
2021-11-10T19:01:40.502 INFO:teuthology.orchestra.run.smithi129.stdout:  1446709      4 drwxrwxr-x   2  ubuntu   ubuntu       4096 Nov 10 19:01 /home/ubuntu/cephtest/kafka-2.6.0-src/logs
2021-11-10T19:01:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-11-10T19:01:40.503 INFO:teuthology.orchestra.run.smithi129.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2021-11-10T19:01:40.504 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_91815aeb94449f1ea4365867a0fd4c60103057e1/teuthology/run_tasks.py", line 176, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_91815aeb94449f1ea4365867a0fd4c60103057e1/teuthology/task/internal/__init__.py", line 58, in base
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_91815aeb94449f1ea4365867a0fd4c60103057e1/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_91815aeb94449f1ea4365867a0fd4c60103057e1/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_91815aeb94449f1ea4365867a0fd4c60103057e1/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi129 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'

from http://qa-proxy.ceph.com/teuthology/cbodley-2021-11-10_18:18:35-rgw-wip-cbodley-testing-distro-basic-smithi/6496484/teuthology.log

Related issues 1 (0 open — 1 closed)

Copied to rgw - Backport #53866: pacific: notification tests failing: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'RejectedYuval LifshitzActions
Actions
Copy link
 #1

Updated by Casey Bodley over 2 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Yuval Lifshitz over 2 years ago

  • Assignee changed from Yuval Lifshitz to Kalpesh Pandya
Actions
Copy link
 #3

Updated by Kalpesh Pandya over 2 years ago

Hello,

I don't see this issue here: https://pulpito.ceph.com/kapandya-2021-11-11_10:31:30-rgw:notifications-wip-rgw-kafka-teuth-cleanup-distro-basic-smithi/

I tried to dig out the reason behind the failure which you saw but couldn't find it.

P.S: I didn't change anything in my code. Just created a branch off master and ran it in teuthology with a log message change.

Thanks,
Kalpesh Pandya

Actions
Copy link
 #4

Updated by Casey Bodley over 2 years ago

another example of the failure: http://qa-proxy.ceph.com/teuthology/teuthology-2021-11-04_03:05:02-rgw-master-distro-default-gibba/6483755/teuthology.log

2021-11-10T23:03:26.683 INFO:tasks.kafka:Removing packaged dependencies of Kafka...
2021-11-10T23:03:26.683 DEBUG:teuthology.orchestra.run.gibba001:> rm -rf /home/ubuntu/cephtest/kafka-2.6.0-src/logs
2021-11-10T23:03:26.735 DEBUG:teuthology.orchestra.run.gibba001:> rm -rf /home/ubuntu/cephtest/kafka-2.6.0-src
2021-11-10T23:03:26.895 DEBUG:teuthology.orchestra.run.gibba001:> rm -rf /home/ubuntu/cephtest/kafka-2.6.0-src.tgz

...

2021-11-10T23:03:54.747 DEBUG:teuthology.orchestra.run.gibba001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2021-11-10T23:03:54.750 DEBUG:teuthology.orchestra.run.gibba017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2021-11-10T23:03:54.759 INFO:teuthology.orchestra.run.gibba001.stdout:   394898      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Nov 10 23:03 /home/ubuntu/cephtest
2021-11-10T23:03:54.759 INFO:teuthology.orchestra.run.gibba001.stdout:  1179649      4 drwxrwxr-x   3  ubuntu   ubuntu       4096 Nov 10 23:03 /home/ubuntu/cephtest/kafka-2.6.0-src
2021-11-10T23:03:54.759 INFO:teuthology.orchestra.run.gibba001.stdout:  1179650      4 drwxrwxr-x   2  ubuntu   ubuntu       4096 Nov 10 23:03 /home/ubuntu/cephtest/kafka-2.6.0-src/logs
2021-11-10T23:03:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-11-10T23:03:54.760 INFO:teuthology.orchestra.run.gibba001.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
Actions
Copy link
 #5

Updated by Casey Bodley over 2 years ago

from that same log:

2021-11-10T23:03:26.598 INFO:tasks.kafka:Stopping Zookeeper and Kafka Services...
2021-11-10T23:03:26.599 DEBUG:teuthology.orchestra.run.gibba001:> cd /home/ubuntu/cephtest/kafka-2.6.0-src/bin && ./kafka-server-stop.sh /home/ubuntu/cephtest/kafka-2.6.0-src/config/kafka.properties
2021-11-10T23:03:26.622 DEBUG:teuthology.orchestra.run.gibba001:> cd /home/ubuntu/cephtest/kafka-2.6.0-src/bin && ./zookeeper-server-stop.sh /home/ubuntu/cephtest/kafka-2.6.0-src/config/zookeeper.properties
2021-11-10T23:03:26.683 INFO:tasks.kafka:Removing packaged dependencies of Kafka...
2021-11-10T23:03:26.683 DEBUG:teuthology.orchestra.run.gibba001:> rm -rf /home/ubuntu/cephtest/kafka-2.6.0-src/logs
2021-11-10T23:03:26.735 DEBUG:teuthology.orchestra.run.gibba001:> rm -rf /home/ubuntu/cephtest/kafka-2.6.0-src
2021-11-10T23:03:26.895 DEBUG:teuthology.orchestra.run.gibba001:> rm -rf /home/ubuntu/cephtest/kafka-2.6.0-src.tgz
2021-11-10T23:03:26.909 DEBUG:teuthology.run_tasks:Unwinding manager notification-tests
2021-11-10T23:03:26.921 DEBUG:teuthology.run_tasks:Unwinding manager rabbitmq
2021-11-10T23:03:26.945 INFO:tasks.rabbitmq:Stopping RabbitMQ Service...

if the ./kafka-server-stop.sh script isn't actually waiting for the kafka process to exit, it may continue to log things under kafka-2.6.0-src/logs after we remove the directory with 'rm -rf'?

Actions
Copy link
 #6

Updated by Casey Bodley over 2 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 43995
Actions
Copy link
 #7

Updated by Casey Bodley over 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to pacific
Actions
Copy link
 #8

Updated by Backport Bot over 2 years ago

  • Copied to Backport #53866: pacific: notification tests failing: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' added
Actions
Copy link
 #9

Updated by Cory Snyder about 2 years ago

Casey, the qa/tasks/kafka.py file that was modified in the fix associated with this ticket does not exist on Pacific. Should I close the backport or look into backporting that teuthology task as well?

Actions
Copy link
 #10

Updated by Casey Bodley about 2 years ago

  • Assignee changed from Kalpesh Pandya to Yuval Lifshitz

^ what do you think Yuval? the kafka stuff exists in pacific, should we backport the tests as well?

Actions
Copy link
 #11

Updated by Yuval Lifshitz about 2 years ago

the tests have to be backported first.
there is a PR for that: https://github.com/ceph/ceph/pull/41850 open now for 10 month

Actions
Copy link
 #12

Updated by Casey Bodley about 2 years ago

oof, thanks. it looks like it never got picked up for testing/merge because it was missing the needs-qa tag

Actions
Copy link
 #13

Updated by Backport Bot over 1 year ago

  • Tags changed from notification kakfa to notification kakfa backport_processed
Actions
Copy link
 #14

Updated by Konstantin Shalygin 5 months ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100
Actions
Copy link

Also available in: Atom PDF