Project

General

Profile

Actions

Bug #44889

closed

workunit does not respect suite_branch when it comes to checkout sha1 on remote host

Added by Kyrylo Shatskyy about 4 years ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

The issue is reproduced when it is used the --suite-branch option and --suite fs:basic_functional.
(teuthology-suite -v
--machine-type ovh
--ceph master
--suite fs:basic_functional
--suite-branch refs/pull/34264/merge
-d centos
--limit 11
--newest 100
--filter-out ubuntu,rhel)

Log example:

2020-03-31T11:38:08.274 INFO:teuthology.run:Config:

archive_path: /home/worker/archive/runner-2020-03-31_11:11:47-smoke-nautilus-distro-basic-ovh/81
branch: nautilus
description: smoke/basic/{clusters/{fixed-3-cephfs.yaml openstack.yaml} objectstore/bluestore-bitmap.yaml
tasks/cfuse_workunit_suites_blogbench.yaml}
email: null
first_in_suite: false
job_id: '81'
kernel:
kdb: true
sha1: distro
last_in_suite: false
log-rotate:
ceph-mds: 10G
ceph-osd: 10G
machine_type: ovh
name: runner-2020-03-31_11:11:47-smoke-nautilus-distro-basic-ovh
nuke-on-error: true
openstack:
- volumes:
count: 2
size: 10
- machine:
cpus: 1
disk: 40
ram: 8000
volumes:
count: 4
size: 10
os_type: centos
overrides:
admin_socket:
branch: nautilus
ceph:
conf:
mgr:
debug mgr: 20
debug ms: 1
mon:
debug mon: 20
debug ms: 1
debug paxos: 20
osd:
bdev async discard: true
bdev enable discard: true
bluestore allocator: bitmap
bluestore block size: 96636764160
bluestore fsck on mount: true
debug bluefs: 20
debug bluestore: 20
debug filestore: 20
debug journal: 20
debug ms: 20
debug osd: 25
debug rocksdb: 10
mon osd backfillfull_ratio: 0.85
mon osd full ratio: 0.9
mon osd nearfull ratio: 0.8
osd failsafe full ratio: 0.95
osd objectstore: bluestore
osd shutdown pgref assert: true
fs: xfs
log-whitelist:
- \(MDS_ALL_DOWN\)
- \(MDS_UP_LESS_THAN_MAX\)
sha1: 62afb915ecd67f352966d3175ac7250633a60134
ceph-deploy:
bluestore: true
conf:
client:
log file: /var/log/ceph/ceph-$name.$pid.log
mon:
osd default pool size: 2
osd:
bdev async discard: true
bdev enable discard: true
bluestore block size: 96636764160
bluestore fsck on mount: true
debug bluefs: 20
debug bluestore: 20
debug rocksdb: 10
mon osd backfillfull_ratio: 0.85
mon osd full ratio: 0.9
mon osd nearfull ratio: 0.8
osd failsafe full ratio: 0.95
osd objectstore: bluestore
fs: xfs
install:
ceph:
sha1: 62afb915ecd67f352966d3175ac7250633a60134
thrashosds:
bdev_inject_crash: 2
bdev_inject_crash_probability: 0.5
workunit:
branch: refs/pull/34171/merge
sha1: a88a277341e7515c183d918be3402ae5880161ba
owner: scheduled_runner@teuth-py3
priority: 1000
repo: https://github.com/ceph/ceph.git
roles:
- - mon.a
- mds.a
- mgr.x
- osd.0
- osd.1
- - mon.b
- mds.b
- mon.c
- mgr.y
- osd.2
- osd.3
- - client.0
sha1: 62afb915ecd67f352966d3175ac7250633a60134
suite: smoke
suite_branch: refs/pull/34171/merge
suite_path: /home/worker/src/github.com_ceph_ceph_PR#34171/qa
suite_relpath: qa
suite_repo: https://github.com/ceph/ceph.git
suite_sha1: a88a277341e7515c183d918be3402ae5880161ba
tasks:
- install: null
- ceph:
fs: xfs
- ceph-fuse: null
- workunit:
clients:
all:
- suites/blogbench.sh
teuthology_branch: master
tube: ovh
verbose: true
worker_log: /home/worker/archive/worker_logs/worker.ovh.15183
.
.
.
2020-03-31T11:53:23.563 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/worker/src/teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/worker/src/teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/worker/src/github.com_ceph_ceph_PR#34171/qa/tasks/workunit.py", line 138, in task
cleanup=cleanup)
File "/home/worker/src/github.com_ceph_ceph_PR#34171/qa/tasks/workunit.py", line 288, in _spawn_on_all_clients
timeout=timeout)
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 87, in exit
for result in self:
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback
reraise(*exc_info)
File "/home/worker/src/teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 24, in capture_traceback
return func(*args, **kwargs)
File "/home/worker/src/github.com_ceph_ceph_PR#34171/qa/tasks/workunit.py", line 335, in _run_tests
args=refspec.clone(git_url, clonedir))
File "/home/worker/src/teuthology_master/teuthology/orchestra/remote.py", line 201, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/worker/src/teuthology_master/teuthology/orchestra/run.py", line 473, in run
r.wait()
File "/home/worker/src/teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/worker/src/teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on target-py3-035 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a88a277341e7515c183d918be3402ae5880161ba'

Also it seems like vim qa/tasks/util/workunit.py does not support `refs/pull/*/*` branches too.

Actions #1

Updated by Laura Flores about 1 year ago

Saw this happen in the rados suite. The SHA (aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8) lines up with the test branch that was used as the suite branch in ci (wip-yuri10-testing-2023-05-16-1243).

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276323

2023-05-17T04:53:17.785 INFO:tasks.workunit.client.0.smithi141.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'...
2023-05-17T04:53:19.685 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:38.151 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.689 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:38.155 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.767 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:38.201 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.769 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:38.236 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.771 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:38.206 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.774 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:38.242 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.781 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:38.247 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.840 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:38.274 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.860 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:38.327 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:21.568 INFO:tasks.workunit.client.0.smithi141.stderr:error: inflate: data stream error (incorrect data check)
2023-05-17T04:53:21.569 INFO:tasks.workunit.client.0.smithi141.stderr:fatal: pack has bad object at offset 5930464: inflate returned -3
2023-05-17T04:53:21.569 INFO:tasks.workunit.client.0.smithi141.stderr:fatal: fetch-pack: invalid index-pack output
2023-05-17T04:53:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-05-17T04:53:21.571 INFO:tasks.workunit:failed to check out 'aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8' from https://github.com/ceph/ceph-ci.git; will also try in https://github.com/ceph/ceph.git
2023-05-17T04:53:21.572 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8
2023-05-17T04:53:21.630 INFO:tasks.workunit.client.0.smithi141.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'...
2023-05-17T04:53:39.747 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:58.212 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.795 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:58.260 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.831 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:58.297 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.842 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:58.277 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.851 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:58.285 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.854 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:58.322 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.872 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:58.335 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.875 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:58.310 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.917 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:58.384 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.792 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:02:18.258 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.813 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:02:18.280 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.870 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:18.304 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.871 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:18.306 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.889 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:02:18.355 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.890 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:02:18.358 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.893 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:18.328 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.932 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:02:18.399 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.972 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:02:18.437 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:54:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-05-17T04:54:03.511 INFO:tasks.workunit.client.0.smithi141.stderr:fatal: reference is not a tree: aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8
2023-05-17T04:54:03.512 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Actions #2

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270447

Actions #3

Updated by Josh Durgin 9 months ago

  • Status changed from New to Closed

It's supposed to match the ceph branch. If you want it to use another branch it can be added explicitly, e.g.:

- workunit:
    branch: mybranch
    clients:
      all:
      - suites/blogbench.sh

or as part of the overrides section.

Actions

Also available in: Atom PDF