Tools » teuthology

I have seen lately errors, see below during suites scheduling.

Here is the run and a log snippet:

https://pulpito.ceph.com/yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi

2024-05-01 22:22:10,359.359 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/read mon_election/connectivity msgr-failures/osd-dispatch-delay msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} thrashers/morepggrow thrashosds-health workloads/cache-agent-big}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684837
2024-05-01 22:22:11,442.442 INFO:teuthology.suite.run:Scheduling rados/singleton-nomsgr/{all/lazy_omap_stats_output mon_election/classic rados supported-random-distro$/{ubuntu_latest}}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684838
2024-05-01 22:22:12,465.465 INFO:teuthology.suite.run:Scheduling rados/perf/{ceph mon_election/connectivity objectstore/bluestore-basic-min-osd-mem-target openstack scheduler/dmclock_default_shards settings/optimized ubuntu_latest workloads/fio_4M_rand_rw}
Traceback (most recent call last):
  File "/cephfs/home/yuriw/teuthology/virtualenv/bin/teuthology-schedule", line 5, in <module>
    from scripts.schedule import main
  File "/cephfs/home/yuriw/teuthology/scripts/schedule.py", line 4, in <module>
    import teuthology.schedule
  File "/cephfs/home/yuriw/teuthology/teuthology/schedule.py", line 4, in <module>
    import teuthology.beanstalk
  File "/cephfs/home/yuriw/teuthology/teuthology/beanstalk.py", line 9, in <module>
    from teuthology import report
  File "/cephfs/home/yuriw/teuthology/teuthology/report.py", line 12, in <module>
    import teuthology.exporter
  File "/cephfs/home/yuriw/teuthology/teuthology/exporter.py", line 238, in <module>
    BootstrapTime = Summary(
  File "/cephfs/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 150, in __init__
    self._metric_init()
  File "/cephfs/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/prometheus_client/metrics.py", line 492, in _metric_init
    self._count = values.ValueClass(self._type, self._name, self._name + '_count', self._labelnames,
  File "/cephfs/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/prometheus_client/values.py", line 68, in __init__
    self.__reset()
  File "/cephfs/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/prometheus_client/values.py", line 82, in __reset
    files[file_prefix] = MmapedDict(filename)
  File "/cephfs/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/prometheus_client/mmap_dict.py", line 81, in __init__
    for key, _, _, pos in self._read_all_values():
  File "/cephfs/home/yuriw/teuthology/virtualenv/lib/python3.8/site-packages/prometheus_client/mmap_dict.py", line 40, in _read_all_values
    raise RuntimeError('Read beyond file size detected, file is corrupted.')
RuntimeError: Read beyond file size detected, file is corrupted.
2024-05-01 22:22:13,396.396 INFO:teuthology.suite.run:Scheduling rados/singleton/{all/divergent_priors2 mon_election/classic msgr-failures/none msgr/async-v2only objectstore/bluestore-comp-zlib rados supported-random-distro$/{centos_latest}}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684839
2024-05-01 22:22:14,392.392 INFO:teuthology.suite.run:Scheduling rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684840
2024-05-01 22:22:15,383.383 INFO:teuthology.suite.run:Scheduling rados/cephadm/workunits/{0-distro/centos_9.stream_runc agent/off mon_election/classic task/test_cephadm_repos}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684841
2024-05-01 22:22:16,387.387 INFO:teuthology.suite.run:Scheduling rados/basic/{ceph clusters/{fixed-2 openstack} mon_election/classic msgr-failures/many msgr/async-v1only objectstore/bluestore-stupid rados supported-random-distro$/{centos_latest} tasks/libcephsqlite}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684842
2024-05-01 22:22:17,409.409 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/peering ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/upmap-read mon_election/classic msgr-failures/fastclose msgr/async-v2only objectstore/bluestore-bitmap rados supported-random-distro$/{centos_latest} thrashers/none thrashosds-health workloads/cache-agent-small}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684843
2024-05-01 22:22:18,426.426 INFO:teuthology.suite.run:Scheduling rados/monthrash/{ceph clusters/9-mons mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/sync workloads/rados_5925}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684844
2024-05-01 22:22:19,481.481 INFO:teuthology.suite.run:Scheduling rados/singleton-nomsgr/{all/librados_hello_world mon_election/connectivity rados supported-random-distro$/{centos_latest}}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684845
2024-05-01 22:22:20,534.534 INFO:teuthology.suite.run:Scheduling rados/cephadm/osds/{0-distro/centos_9.stream 0-nvme-loop 1-start 2-ops/rm-zap-add}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684846
2024-05-01 22:22:21,513.513 INFO:teuthology.suite.run:Scheduling rados/mgr/{clusters/{2-node-mgr} debug/mgr distro/{centos_latest} mgr_ttl_cache/enable mon_election/classic random-objectstore$/{bluestore-hybrid} tasks/{1-install 2-ceph 3-mgrmodules 4-units/crash}}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684847
2024-05-01 22:22:22,541.541 INFO:teuthology.suite.run:Scheduling rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-zstd rados supported-random-distro$/{centos_latest}}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684848
2024-05-01 22:22:23,551.551 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/3-size-2-min-size 1-pg-log-overrides/short_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-5} backoff/peering_and_degraded ceph clusters/{fixed-4 openstack} crc-failures/default d-balancer/crush-compat mon_election/connectivity msgr-failures/few msgr/async objectstore/bluestore-comp-lz4 rados supported-random-distro$/{centos_latest} thrashers/pggrow thrashosds-health workloads/cache-pool-snaps-readproxy}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684849
2024-05-01 22:22:24,628.628 INFO:teuthology.suite.run:Scheduling rados/multimon/{clusters/6 mon_election/classic msgr-failures/few msgr/async-v2only no_pools objectstore/bluestore-stupid rados supported-random-distro$/{ubuntu_latest} tasks/mon_recovery}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684850
2024-05-01 22:22:25,603.603 INFO:teuthology.suite.run:Scheduling rados/thrash-erasure-code-big/{ceph cluster/{12-osds openstack} mon_election/classic msgr-failures/osd-dispatch-delay objectstore/bluestore-stupid rados recovery-overrides/{more-active-recovery} supported-random-distro$/{centos_latest} thrashers/morepggrow thrashosds-health workloads/ec-rados-plugin=clay-k=4-m=2}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684851
2024-05-01 22:22:26,606.606 INFO:teuthology.suite.run:Scheduling rados/cephadm/workunits/{0-distro/ubuntu_22.04 agent/on mon_election/connectivity task/test_extra_daemon_features}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684852
2024-05-01 22:22:27,638.638 INFO:teuthology.suite.run:Scheduling rados/singleton-nomsgr/{all/msgr mon_election/classic rados supported-random-distro$/{centos_latest}}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684853
2024-05-01 22:22:28,622.622 INFO:teuthology.suite.run:Scheduling rados/thrash/{0-size-min-size-overrides/2-size-2-min-size 1-pg-log-overrides/normal_pg_log 2-recovery-overrides/{more-async-partial-recovery} 3-scrub-overrides/{max-simultaneous-scrubs-1} backoff/normal ceph clusters/{fixed-4 openstack} crc-failures/bad_map_crc_failure d-balancer/on mon_election/classic msgr-failures/osd-delay msgr/async-v1only objectstore/bluestore-comp-snappy rados supported-random-distro$/{ubuntu_latest} thrashers/careful thrashosds-health workloads/cache-pool-snaps}
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684854
2024-05-01 22:22:29,645.645 INFO:teuthology.suite.run:Suite rados in /cephfs/home/yuriw/src/github.com_ceph_ceph-c_wip-yuri3-testing-2024-04-05-0825/qa/suites/rados scheduled 334 jobs.
2024-05-01 22:22:29,646.646 INFO:teuthology.suite.run:0/334 jobs were filtered out.
2024-05-01 22:22:29,646.646 INFO:teuthology.suite.run:Scheduled 334 jobs in total.
Job scheduled with name yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi and ID 7684855
2024-05-01 22:22:30,604.604 INFO:teuthology.suite.run:Test results viewable at https://pulpito.ceph.com/yuriw-2024-05-01_22:15:10-rados-wip-yuri3-testing-2024-04-05-0825-distro-default-smithi/