Project

General

Profile

Actions

Bug #44889

closed

workunit does not respect suite_branch when it comes to checkout sha1 on remote host

Added by Kyrylo Shatskyy about 4 years ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

The issue is reproduced when it is used the --suite-branch option and --suite fs:basic_functional.
(teuthology-suite -v
--machine-type ovh
--ceph master
--suite fs:basic_functional
--suite-branch refs/pull/34264/merge
-d centos
--limit 11
--newest 100
--filter-out ubuntu,rhel)

Log example:

2020-03-31T11:38:08.274 INFO:teuthology.run:Config:

archive_path: /home/worker/archive/runner-2020-03-31_11:11:47-smoke-nautilus-distro-basic-ovh/81
branch: nautilus
description: smoke/basic/{clusters/{fixed-3-cephfs.yaml openstack.yaml} objectstore/bluestore-bitmap.yaml
tasks/cfuse_workunit_suites_blogbench.yaml}
email: null
first_in_suite: false
job_id: '81'
kernel:
kdb: true
sha1: distro
last_in_suite: false
log-rotate:
ceph-mds: 10G
ceph-osd: 10G
machine_type: ovh
name: runner-2020-03-31_11:11:47-smoke-nautilus-distro-basic-ovh
nuke-on-error: true
openstack:
- volumes:
count: 2
size: 10
- machine:
cpus: 1
disk: 40
ram: 8000
volumes:
count: 4
size: 10
os_type: centos
overrides:
admin_socket:
branch: nautilus
ceph:
conf:
mgr:
debug mgr: 20
debug ms: 1
mon:
debug mon: 20
debug ms: 1
debug paxos: 20
osd:
bdev async discard: true
bdev enable discard: true
bluestore allocator: bitmap
bluestore block size: 96636764160
bluestore fsck on mount: true
debug bluefs: 20
debug bluestore: 20
debug filestore: 20
debug journal: 20
debug ms: 20
debug osd: 25
debug rocksdb: 10
mon osd backfillfull_ratio: 0.85
mon osd full ratio: 0.9
mon osd nearfull ratio: 0.8
osd failsafe full ratio: 0.95
osd objectstore: bluestore
osd shutdown pgref assert: true
fs: xfs
log-whitelist:
- \(MDS_ALL_DOWN\)
- \(MDS_UP_LESS_THAN_MAX\)
sha1: 62afb915ecd67f352966d3175ac7250633a60134
ceph-deploy:
bluestore: true
conf:
client:
log file: /var/log/ceph/ceph-$name.$pid.log
mon:
osd default pool size: 2
osd:
bdev async discard: true
bdev enable discard: true
bluestore block size: 96636764160
bluestore fsck on mount: true
debug bluefs: 20
debug bluestore: 20
debug rocksdb: 10
mon osd backfillfull_ratio: 0.85
mon osd full ratio: 0.9
mon osd nearfull ratio: 0.8
osd failsafe full ratio: 0.95
osd objectstore: bluestore
fs: xfs
install:
ceph:
sha1: 62afb915ecd67f352966d3175ac7250633a60134
thrashosds:
bdev_inject_crash: 2
bdev_inject_crash_probability: 0.5
workunit:
branch: refs/pull/34171/merge
sha1: a88a277341e7515c183d918be3402ae5880161ba
owner: scheduled_runner@teuth-py3
priority: 1000
repo: https://github.com/ceph/ceph.git
roles:
- - mon.a
- mds.a
- mgr.x
- osd.0
- osd.1
- - mon.b
- mds.b
- mon.c
- mgr.y
- osd.2
- osd.3
- - client.0
sha1: 62afb915ecd67f352966d3175ac7250633a60134
suite: smoke
suite_branch: refs/pull/34171/merge
suite_path: /home/worker/src/github.com_ceph_ceph_PR#34171/qa
suite_relpath: qa
suite_repo: https://github.com/ceph/ceph.git
suite_sha1: a88a277341e7515c183d918be3402ae5880161ba
tasks:
- install: null
- ceph:
fs: xfs
- ceph-fuse: null
- workunit:
clients:
all:
- suites/blogbench.sh
teuthology_branch: master
tube: ovh
verbose: true
worker_log: /home/worker/archive/worker_logs/worker.ovh.15183
.
.
.
2020-03-31T11:53:23.563 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/worker/src/teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/worker/src/teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/worker/src/github.com_ceph_ceph_PR#34171/qa/tasks/workunit.py", line 138, in task
cleanup=cleanup)
File "/home/worker/src/github.com_ceph_ceph_PR#34171/qa/tasks/workunit.py", line 288, in _spawn_on_all_clients
timeout=timeout)
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 87, in exit
for result in self:
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback
reraise(*exc_info)
File "/home/worker/src/teuthology_master/virtualenv/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/worker/src/teuthology_master/teuthology/parallel.py", line 24, in capture_traceback
return func(*args, **kwargs)
File "/home/worker/src/github.com_ceph_ceph_PR#34171/qa/tasks/workunit.py", line 335, in _run_tests
args=refspec.clone(git_url, clonedir))
File "/home/worker/src/teuthology_master/teuthology/orchestra/remote.py", line 201, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/worker/src/teuthology_master/teuthology/orchestra/run.py", line 473, in run
r.wait()
File "/home/worker/src/teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/worker/src/teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on target-py3-035 with status 128: 'rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a88a277341e7515c183d918be3402ae5880161ba'

Also it seems like vim qa/tasks/util/workunit.py does not support `refs/pull/*/*` branches too.

Actions #1

Updated by Laura Flores 12 months ago

Saw this happen in the rados suite. The SHA (aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8) lines up with the test branch that was used as the suite branch in ci (wip-yuri10-testing-2023-05-16-1243).

/a/yuriw-2023-05-16_23:44:06-rados-wip-yuri10-testing-2023-05-16-1243-distro-default-smithi/7276323

2023-05-17T04:53:17.785 INFO:tasks.workunit.client.0.smithi141.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'...
2023-05-17T04:53:19.685 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:38.151 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.689 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:38.155 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.767 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:38.201 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.769 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:38.236 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.771 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:38.206 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.774 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:38.242 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.781 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:38.247 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.840 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:38.274 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:19.860 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:38.327 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:21.568 INFO:tasks.workunit.client.0.smithi141.stderr:error: inflate: data stream error (incorrect data check)
2023-05-17T04:53:21.569 INFO:tasks.workunit.client.0.smithi141.stderr:fatal: pack has bad object at offset 5930464: inflate returned -3
2023-05-17T04:53:21.569 INFO:tasks.workunit.client.0.smithi141.stderr:fatal: fetch-pack: invalid index-pack output
2023-05-17T04:53:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-05-17T04:53:21.571 INFO:tasks.workunit:failed to check out 'aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8' from https://github.com/ceph/ceph-ci.git; will also try in https://github.com/ceph/ceph.git
2023-05-17T04:53:21.572 DEBUG:teuthology.orchestra.run.smithi141:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8
2023-05-17T04:53:21.630 INFO:tasks.workunit.client.0.smithi141.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'...
2023-05-17T04:53:39.747 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:58.212 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.795 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:58.260 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.831 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:01:58.297 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.842 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:58.277 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.851 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:58.285 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.854 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:58.322 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.872 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:58.335 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.875 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:01:58.310 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:39.917 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:01:58.384 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.792 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:02:18.258 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.813 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:02:18.280 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.870 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:18.304 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.871 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:18.306 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.889 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:02:18.355 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.890 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:02:18.358 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.893 INFO:tasks.ceph.mon.b.smithi190.stderr:==00:00:02:18.328 106240== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.932 INFO:tasks.ceph.mon.c.smithi141.stderr:==00:00:02:18.399 106565== Warning: unimplemented fcntl command: 1036
2023-05-17T04:53:59.972 INFO:tasks.ceph.mon.a.smithi141.stderr:==00:00:02:18.437 106561== Warning: unimplemented fcntl command: 1036
2023-05-17T04:54:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-05-17T04:54:03.511 INFO:tasks.workunit.client.0.smithi141.stderr:fatal: reference is not a tree: aafe04d3ada9a95bff2e2694c9bb0b9ce25b94f8
2023-05-17T04:54:03.512 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Actions #2

Updated by Laura Flores 12 months ago

/a/yuriw-2023-05-10_18:54:50-rados-wip-yuri-testing-2023-05-10-0917-distro-default-smithi/7270447

Actions #3

Updated by Josh Durgin 9 months ago

  • Status changed from New to Closed

It's supposed to match the ceph branch. If you want it to use another branch it can be added explicitly, e.g.:

- workunit:
    branch: mybranch
    clients:
      all:
      - suites/blogbench.sh

or as part of the overrides section.

Actions

Also available in: Atom PDF