Sage Weil's activity
From 01/28/2020 to 02/26/2020
02/26/2020
- 11:13 PM Dashboard Feature #44301 (Resolved): mgr/dashboard: Enhance user create CLI command to force password change
- 11:13 PM mgr Cleanup #44306 (Resolved): mgr/modules: outsource certificate creation to `mgr_util.py`
- 08:51 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
- ...
- 08:50 PM Orchestrator Bug #44313: ceph-volume prepare is not idempotent and may get called twice
- similar failure, this time a 'daemon mon add' dup:
/a/sage-2020-02-26_08:10:43-rados-wip-sage2-testing-2020-02-25-... - 08:27 PM Orchestrator Bug #44313: ceph-volume prepare is not idempotent and may get called twice
- One possible fix would be to make ceph-volume itself idempotent, so that calling prepare on an already-prepared devic...
- 08:26 PM Orchestrator Bug #44313 (Resolved): ceph-volume prepare is not idempotent and may get called twice
- symptom is a failure like so:...
- 08:48 PM RADOS Bug #44314 (Resolved): osd-backfill-stats.sh failing intermittently in TEST_backfill_sizeup_out()...
- ...
- 08:24 PM Orchestrator Bug #44312 (Duplicate): ceph-volume prepare is not idempotent and may get called twice
- 07:47 PM RADOS Bug #43914 (Fix Under Review): nautilus: ceph tell command times out
- 06:48 PM RADOS Bug #43914: nautilus: ceph tell command times out
- okay yeah, it's because the command wq uses osd_lock...
- 06:41 PM RADOS Bug #43914: nautilus: ceph tell command times out
- so, this was fixed in nautilus, in the sense that https://github.com/ceph/ceph/pull/27696 went into nautilus.
- 06:37 PM RADOS Bug #43914: nautilus: ceph tell command times out
- The thread (or lock?) is busy with...
- 07:33 PM RADOS Bug #42328: osd/PrimaryLogPG.cc: 3962: ceph_abort_msg("out of order op")
- follow-up fix: https://github.com/ceph/ceph/pull/33559 (typo in original commit)
- 06:19 PM Ceph Bug #44250 (Resolved): Error: unable to find a name and tag match for ceph-ci/daemon-base in repo...
- 06:02 PM mgr Bug #44299 (Need More Info): nautilus: mgr osd communication stalled in mixed cluster
- I can't make heads or tails of this. I think we need to reproduce with debug_ms=20
- 12:09 AM RADOS Bug #44297 (Fix Under Review): mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymo...
- 12:02 AM RADOS Bug #44297: mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous connection")
- The command is passed from a nautilus monitor:...
02/25/2020
- 11:51 PM RADOS Bug #44275 (Resolved): NameError: name 'retval' is not defined
- 11:50 PM RADOS Bug #44248 (Pending Backport): Receiving RemoteBackfillReserved in WaitLocalBackfillReserved can ...
- 11:43 PM RADOS Bug #44297 (Resolved): mon/Monitor.cc: 3924: FAILED ceph_assert(!"send_message on anonymous conne...
- on nautilus->octopus/master upgrade...
- 11:40 PM Ceph Bug #44078: centos 8.1: ceph-mon: assertion "z >= signedspan::zero()" failed in ceph::to_timespan(
- centos 8.1 again
/a/sage-2020-02-25_15:51:04-rados-wip-sage2-testing-2020-02-25-0704-distro-basic-smithi/4801785
- 11:39 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
- ...
- 01:18 PM mgr Bug #44271 (Resolved): dashboard: unhandled exception ('pwdUpdateRequired',)
02/24/2020
- 07:42 PM RADOS Bug #44076 (Pending Backport): mon: update + monmap update triggers spawn loop
- 07:29 PM Orchestrator Bug #41746 (Resolved): mgr/rook: `ceph orchestrator device ls` doesn't set `available`
- this is working now AFAICS
- 07:29 PM Orchestrator Bug #43838: cephadm: Forcefully Remove Services (unresponsive hosts)
- One option is to have them 'ceph orch host rm $hostname'...
- 07:27 PM Orchestrator Bug #44121 (Fix Under Review): calling cephadm shell again looses bash history
- 07:19 PM Orchestrator Bug #44270 (Triaged): Under certain circumstances, "ceph orch apply" returns success even when no...
- i bet the problem is that the drive inventory isn't populated yet immediately after bootstrap.
- 07:17 PM Orchestrator Bug #44273 (Need More Info): Getting "stray daemon osd.3 on host admin not managed by cephadm" on...
- This should have been fixed by 607263224c26... can you reproduce this with debug_mgr = 20 and attach a log?
- 07:13 PM Orchestrator Feature #43867 (Resolved): cephadm: progress item for upgrade
- 03:44 PM mgr Bug #44271 (Fix Under Review): dashboard: unhandled exception ('pwdUpdateRequired',)
- 03:36 PM mgr Bug #44271 (Resolved): dashboard: unhandled exception ('pwdUpdateRequired',)
- ...
- 03:09 PM Ceph Bug #44078: centos 8.1: ceph-mon: assertion "z >= signedspan::zero()" failed in ceph::to_timespan(
- I hit this on centos 8.1...
02/23/2020
- 09:05 PM RADOS Bug #43861: ceph_test_rados_watch_notify hang
- ...
- 09:01 PM Dashboard Bug #44256 (Closed): Test failure: test_create_user_with_pwd_expiration_date (tasks.mgr.dashboard...
- ...
- 08:12 PM Orchestrator Feature #44255 (New): cephadm: scheduler should consider other daemons on each node
- When choosing a home for a daemon, we should prefer nodes that have fewer daemons, and/or fewer daemons of the same t...
- 08:11 PM Orchestrator Bug #44254 (Resolved): scheduler should prefer existing daemon locations
- If we are placing N daemons, then we should select nodes that already have daemons for the service. (Otherwise, an a...
- 08:10 PM Orchestrator Bug #44253 (Resolved): _apply_service should move services, not just expand/contract
- if placement is based on, e.g., labels, then moving a label should cause us to move services too (add first, then rem...
- 07:17 PM Orchestrator Bug #44252 (Resolved): cephadm: mgr,mds scale-down should prefer standby daemons
- There are three types of daemons:
1. active daemons
2. standby daemons
3. unknown daemons that are (not yet) par... - 02:37 PM Ceph Bug #44242 (Resolved): shaman container build fails
- 02:36 PM Orchestrator Bug #44170 (Duplicate): Teuthology is testing unrelated container images
- this was because the wip-swagner-testing branch was reused and because #44242 had not been fixed
02/22/2020
- 09:33 PM Ceph Bug #44250: Error: unable to find a name and tag match for ceph-ci/daemon-base in repotags: no su...
- these failed too: https://shaman.ceph.com/repos/ceph/wip-sage2-testing-2020-02-22-1230/a4ce2efef9f5d4ebcff1f0a05bb662...
- 08:46 PM Ceph Bug #44250 (Resolved): Error: unable to find a name and tag match for ceph-ci/daemon-base in repo...
- ...
- 09:32 PM mgr Bug #44245: nautilus: mgr: connection halt
- My guess is we are leaking a message ref somewhere...
- 09:27 PM mgr Bug #44245: nautilus: mgr: connection halt
- lots of connections are busy looping, also waiting on the same throttle...
- 09:24 PM mgr Bug #44245: nautilus: mgr: connection halt
- with debug_ms=20 we see, on the mgr,...
- 06:06 PM mgr Fix #44002 (Resolved): mgr/devicehealth: set default monitoring to 'on'
- 06:05 PM Ceph Bug #44196 (Pending Backport): selinux setsched denials for 'fn_anonymous'
- 02:16 PM Orchestrator Feature #43675 (Resolved): workflow for using a signed dashboard cert
- 03:03 AM Orchestrator Bug #43949 (Resolved): mgr/cephadm: ceph fs volume create: TypeError: %d format: a number is requ...
- 03:02 AM Orchestrator Bug #44119 (Resolved): installing cephamd on bionic is painful:
- 03:01 AM Orchestrator Bug #44121: calling cephadm shell again looses bash history
- we could bind the root .bash_history file to something like /var/lib/ceph/$fsid/.bash_history ?
- 03:01 AM Orchestrator Bug #44209 (Resolved): qa/workunits/cephadm/test_cephadm.sh: prometheus:latest: Invalid JSON in -
- 03:01 AM Orchestrator Bug #44003 (Resolved): cephadm: multiple mgrs scheduled on same host
- 01:14 AM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
- /a/sage-2020-02-21_21:08:33-rados-wip-sage3-testing-2020-02-21-1218-distro-basic-smithi/4788714...
- 12:41 AM Ceph Bug #44242: shaman container build fails
- motherfucker
wip-sage2-testing-2020-02-21-1459 51.68.95.182 https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x... - 12:30 AM Ceph Bug #44242: shaman container build fails
- AAAHAHHHHHAHHHHHH
wip-sage-testing-2020-02-21-1528
51.68.95.182
https://jenkins.ceph.com/job/ceph-dev-new-build/A... - 12:00 AM Ceph Bug #44242: shaman container build fails
- wip-sage2-testing-2020-02-21-1459 51.68.84.74 https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABL...
02/21/2020
- 11:03 PM CephFS Bug #44244 (Fix Under Review): pybind/mgr/volumes: "handle_command module 'volumes' command handl...
- 10:12 PM Ceph Bug #44242: shaman container build fails
- wip-sage4-testing-2020-02-21-1319 .... 51.68.93.161 ... https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,A...
- 04:42 PM Ceph Bug #44242: shaman container build fails
- Recent instances:
- wip-sage-testing-2020-02-21-0633 ... 51.91.144.35 ... https://jenkins.ceph.com/job/ceph-dev-n... - 04:41 PM Ceph Bug #44242 (Resolved): shaman container build fails
- The failure looks like so:...
- 09:03 PM RADOS Bug #44243 (Can't reproduce): memstore make check test fails
- ...
- 08:10 PM mgr Bug #44241 (Resolved): mgr: deadlock w/ register/unregister_client
- 03:01 PM mgr Bug #44241 (Fix Under Review): mgr: deadlock w/ register/unregister_client
- 02:58 PM mgr Bug #44241 (Resolved): mgr: deadlock w/ register/unregister_client
- ...
- 01:21 PM RADOS Bug #42347: nautilus assert during osd shutdown: FAILED ceph_assert((sharded_in_flight_list.back(...
- Bastian Mäuser wrote:
> This is still an issue on 14.2.6 (at least the one shipped with proxmox)
It will appear i... - 02:02 AM Dashboard Bug #44201 (Resolved): mgr/dashboard: Teuthology and dashboard backend API test failure: setUpCla...
02/20/2020
- 07:32 PM Orchestrator Bug #44231 (Resolved): cephadm: cannot capture core files
- At least, I can't figure it out.
On my test box, i set kernel.core_pattern to both a valid hsot and container path... - 07:02 PM RADOS Bug #39525: lz4 compressor corrupts data when buffers are unaligned
- ok, the first crash isn't becuase we just got bad data.. it's because we just read bad data off of disk. see:...
- 04:09 PM RADOS Bug #39525: lz4 compressor corrupts data when buffers are unaligned
- Notes from CERN incident:
- identical corruption, different OSDmaps on different OSDs:... - 05:40 PM RADOS Bug #44229 (New): monclient: _check_auth_rotating possible clock skew, rotating keys expired way ...
- seems to affect cephadm bootstrap tests
first, the error message doesn't make sense, since the bound 2020-02-20T16...
02/19/2020
- 11:42 PM RADOS Bug #44076 (Fix Under Review): mon: update + monmap update triggers spawn loop
- 10:47 PM Orchestrator Bug #44169 (Resolved): informative exception eaten
- 10:46 PM mgr Bug #44186 (Pending Backport): Module 'pg_autoscaler' has failed: division by zero
- 10:45 PM RADOS Bug #44157 (Resolved): cli throws bad exceptoin on control-c
- 10:38 PM Orchestrator Feature #43670 (Resolved): teuthology: Add new upgrade/downgrade process
- 10:37 PM Orchestrator Feature #43867 (Fix Under Review): cephadm: progress item for upgrade
- 10:36 PM Orchestrator Feature #44031 (Fix Under Review): cephadm: Also cache `device ls`.
- 10:35 PM Orchestrator Feature #43836 (Fix Under Review): cephadm adopt: also adopt Prometheus and Grafana daemons from ...
- 10:35 PM Orchestrator Feature #43695 (Fix Under Review): cephadm: alertmanager
- 10:35 PM Orchestrator Feature #43940 (Resolved): orchestrator mgr add and rm
- 10:35 PM Orchestrator Feature #43685 (Resolved): host prepare
- 07:04 PM mgr Bug #44210 (Pending Backport): Devicehealth scrape fails when smartctl return code is non-zero
- 07:00 PM Orchestrator Bug #44165 (Resolved): test_load_data fails
- 06:59 PM Orchestrator Bug #44188 (Resolved): Module 'cephadm' has failed: dictionary changed size during iteration
- 06:59 PM CephFS Bug #44176 (Resolved): qa: "Error EINVAL: 'Module' object has no attribute 'remove_mds'"
- 12:33 PM Orchestrator Bug #44175 (Resolved): cephadm: adopt does not work with filestore OSDs
02/18/2020
- 11:57 PM mgr Bug #44186 (Fix Under Review): Module 'pg_autoscaler' has failed: division by zero
- 07:51 PM mgr Bug #44186 (Resolved): Module 'pg_autoscaler' has failed: division by zero
- ...
- 10:25 PM Orchestrator Bug #44188 (Resolved): Module 'cephadm' has failed: dictionary changed size during iteration
- ...
- 08:32 PM Orchestrator Bug #44175 (Fix Under Review): cephadm: adopt does not work with filestore OSDs
- 07:55 PM RADOS Bug #43903 (Pending Backport): osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
- 07:52 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
- ...
- 02:30 PM Orchestrator Bug #44168 (Resolved): qa/tasks/cephadm in ceph_bootstrap: AttributeError: 'NoneType' object has ...
02/17/2020
- 11:32 PM Orchestrator Bug #44175 (Resolved): cephadm: adopt does not work with filestore OSDs
- it tries to *copy* the data directory, bad bad bad
- 05:40 PM Orchestrator Feature #43685 (Fix Under Review): host prepare
- 02:13 PM Orchestrator Bug #44169 (Resolved): informative exception eaten
- ...
02/16/2020
02/15/2020
- 03:11 PM RADOS Bug #44157 (Fix Under Review): cli throws bad exceptoin on control-c
- 02:38 PM Ceph Bug #43737 (Resolved): install-dep.sh fails in Centos-7.6
- 02:37 PM RADOS Bug #44041 (Resolved): osd: MLease in stray state -> Crashed
- 02:37 PM RADOS Bug #42328 (Resolved): osd/PrimaryLogPG.cc: 3962: ceph_abort_msg("out of order op")
- 02:36 PM RADOS Bug #44062: LibRadosWatchNotify.WatchNotify failure
- /a/sage-2020-02-15_04:59:38-rados-wip-sage3-testing-2020-02-14-1951-distro-basic-smithi/4765960
02/14/2020
- 05:35 PM RADOS Bug #44157 (Resolved): cli throws bad exceptoin on control-c
- ...
02/13/2020
- 01:09 AM Orchestrator Bug #44077 (Resolved): grafana container doesn't start on 18.04
02/12/2020
- 12:18 AM RADOS Bug #44062 (Triaged): LibRadosWatchNotify.WatchNotify failure
- /a/sage-2020-02-11_20:49:48-rados-wip-sage-testing-2020-02-11-1121-distro-basic-smithi/4755080
02/11/2020
- 08:08 PM devops Bug #44065 (Fix Under Review): ceph.spec.in wrong cephadm dependency
- 04:43 PM Orchestrator Bug #44018 (Resolved): cephadm: down host kills serve() thread
- 02:33 PM Orchestrator Bug #44077 (Resolved): grafana container doesn't start on 18.04
- ...
- 02:28 PM bluestore Bug #43904 (Resolved): bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upg...
- 02:28 PM RADOS Bug #44067 (Resolved): cephtool/test.sh test fails to scrub all pools
- 02:27 PM RADOS Bug #44076 (Resolved): mon: update + monmap update triggers spawn loop
- - upgrade monitors from mimic to octopus
- quorum of 2/3 monitors
- enable msgr2
then
- third monitor probes... - 02:57 AM RADOS Bug #44050 (Resolved): mon tell command args don't work
02/10/2020
- 11:14 PM Orchestrator Bug #43913 (Resolved): test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): Assertio...
- 11:14 PM RADOS Bug #43889 (Pending Backport): expected MON_CLOCK_SKEW but got none
- 02:41 PM RADOS Bug #43889 (Fix Under Review): expected MON_CLOCK_SKEW but got none
- 06:17 PM Orchestrator Bug #43883 (Resolved): cephadm: Found left-over process 15516 (podman) in control group while sta...
- 05:27 PM RADOS Bug #44067 (Fix Under Review): cephtool/test.sh test fails to scrub all pools
- 05:14 PM RADOS Bug #44067 (Resolved): cephtool/test.sh test fails to scrub all pools
- ...
- 03:08 PM Orchestrator Feature #44005 (Resolved): cephadm: associate addrs to hosts
- 02:55 PM RADOS Bug #44052 (Pending Backport): ceph -s does not show >32bit pg states
- 02:42 PM RADOS Bug #44062 (Resolved): LibRadosWatchNotify.WatchNotify failure
- ...
- 02:37 PM RADOS Bug #43048: nautilus: upgrade/mimic-x/stress-split: failed to recover before timeout expired
- /a/sage-2020-02-09_21:18:03-rados-wip-sage2-testing-2020-02-09-1152-distro-basic-smithi/4749175...
02/09/2020
- 09:34 PM Orchestrator Feature #44055 (Closed): cephadm: make 'ls' faster
- For both podman and docker, 'ps' tells you the image name but not its hash.
With podman, you can do:... - 05:40 PM RADOS Bug #43889 (In Progress): expected MON_CLOCK_SKEW but got none
- 12:45 AM RADOS Bug #42328 (Fix Under Review): osd/PrimaryLogPG.cc: 3962: ceph_abort_msg("out of order op")
- 12:25 AM RADOS Bug #43903 (In Progress): osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
- 12:24 AM Orchestrator Bug #43703 (Resolved): selinux vs logrotate
- I'm calling this one "fixed", even though for el 8.0 and 8.1 (pre-z-stream) the error is still there.
https://gith... - 12:22 AM Orchestrator Bug #43883 (Fix Under Review): cephadm: Found left-over process 15516 (podman) in control group w...
02/08/2020
- 07:33 PM Orchestrator Bug #44018 (Fix Under Review): cephadm: down host kills serve() thread
- 06:11 PM RADOS Bug #44052 (Fix Under Review): ceph -s does not show >32bit pg states
- 06:07 PM RADOS Bug #44052 (Resolved): ceph -s does not show >32bit pg states
- ceph -s does not show newer pg states, like repair_failed
- 05:15 PM mgr Bug #44051 (New): crash: archive-all slow with lots of crashes
- I have ~500 crashes and it is taking multiple minutes to do 'ceph crash archive-all'
- 03:26 PM RADOS Bug #44050 (Fix Under Review): mon tell command args don't work
- 02:37 PM RADOS Bug #44050: mon tell command args don't work
- 'ceph tell mon.a help' works, but '-h' does not.
- 02:07 PM RADOS Bug #44050 (Resolved): mon tell command args don't work
- Also, 'ceph tell mon.a force-sync --yes-i-really-mean-it' seems to be broken:...
- 02:11 PM RADOS Feature #42638 (Resolved): Allow specifying pg_autoscale_mode when creating a new pool
- 02:08 PM mgr Bug #43313 (Pending Backport): telemetry module can crash on entity name with multiple '.' separa...
- 01:55 PM Orchestrator Bug #43913: test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): AssertionError: 22 ...
- /a/sage-2020-02-07_23:51:30-rados-wip-sage2-testing-2020-02-07-1439-distro-basic-smithi/4742679
- 01:53 PM RADOS Bug #43889: expected MON_CLOCK_SKEW but got none
- /a/sage-2020-02-07_23:51:30-rados-wip-sage2-testing-2020-02-07-1439-distro-basic-smithi/4742672
- 01:47 PM Dashboard Bug #44049 (Can't reproduce): mgr/dashboard: test_perf_counters_mgr_get failed
- ...
- 01:34 PM RADOS Bug #44024 (Resolved): change in utime_t rendering ('T' separator) conflicts with cache tiering h...
02/07/2020
- 07:54 PM mgr Bug #43946 (Resolved): mimic: EINVAL on 'osd dump'
- 05:50 PM mgr Bug #43946 (Fix Under Review): mimic: EINVAL on 'osd dump'
- 01:55 PM mgr Bug #43946: mimic: EINVAL on 'osd dump'
- /a/sage-2020-02-06_19:01:25-rados-wip-sage2-testing-2020-02-05-1649-distro-basic-smithi/4738937
- 07:38 PM mgr Bug #43963 (Pending Backport): telemetry: crash when posting
- 07:27 PM Orchestrator Bug #44029 (Resolved): orchestrator cli: confusing rgw param order
- 05:10 PM Orchestrator Feature #43671 (Resolved): cephadm: {add,remove}-repo
- 04:34 PM RADOS Bug #44041 (Fix Under Review): osd: MLease in stray state -> Crashed
- 04:03 PM RADOS Bug #44041 (Resolved): osd: MLease in stray state -> Crashed
- ...
02/06/2020
- 11:53 PM RADOS Bug #44024 (Fix Under Review): change in utime_t rendering ('T' separator) conflicts with cache t...
- 11:26 PM RADOS Bug #44024 (Resolved): change in utime_t rendering ('T' separator) conflicts with cache tiering h...
- crash like...
- 10:05 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
- ...
- 08:42 PM Orchestrator Feature #43973 (Resolved): There is a need for a wrapper to fetch the journald logs
- 03:02 PM Orchestrator Bug #44018 (Resolved): cephadm: down host kills serve() thread
- ...
- 03:39 AM Orchestrator Feature #44005 (Fix Under Review): cephadm: associate addrs to hosts
- 03:38 AM Orchestrator Feature #44005 (Resolved): cephadm: associate addrs to hosts
- 03:38 AM Orchestrator Feature #43940 (Fix Under Review): orchestrator mgr add and rm
02/05/2020
- 11:02 PM Orchestrator Bug #44003 (Resolved): cephadm: multiple mgrs scheduled on same host
- By default the mgr binds to a bunch of random ports. We shouldn't put more than one on a single host (unless forced ...
- 10:09 PM Orchestrator Bug #43945 (Resolved): Test failure: test_osd_create (tasks.mgr.test_orchestrator_cli.TestOrchest...
- 06:58 PM mgr Bug #43946: mimic: EINVAL on 'osd dump'
- /a/sage-2020-02-05_03:10:48-rados-wip-sage2-testing-2020-02-04-1448-distro-basic-smithi/4733167
- 05:39 PM RADOS Bug #42347 (Won't Fix): nautilus assert during osd shutdown: FAILED ceph_assert((sharded_in_fligh...
- we've backported the osd fast shutdown ( https://github.com/ceph/ceph/pull/32743 ), so this will effectively go away ...
- 05:00 PM Messengers Bug #43808: segv in MonClient::handle_auth_done
- /a/sage-2020-02-05_03:10:48-rados-wip-sage2-testing-2020-02-04-1448-distro-basic-smithi/4733235
ceph-osd.2.log
... - 02:11 PM Orchestrator Feature #43940 (In Progress): orchestrator mgr add and rm
- 02:03 PM Ceph Wiki edit: CDM_05-FEB-2020 (#7)
- 01:36 PM mgr Bug #43642 (Pending Backport): "ceph telemetry show" shows error: AttributeError: 'NoneType' obje...
02/04/2020
- 07:00 PM mgr Bug #43963 (Fix Under Review): telemetry: crash when posting
- 06:55 PM mgr Bug #43963 (Resolved): telemetry: crash when posting
- ...
- 04:39 PM Ceph Bug #40647 (Can't reproduce): OP_CREATE issued to nautilus OSD
- I think this got fixed.. we haven't seen it in 6 months.
- 04:37 PM mgr Bug #39955 (Resolved): After upgrade to Nautilus 14.2.1 mon DB is growing too fast when state of ...
- I think this is okay now?
- 03:37 PM Ceph Wiki edit: Planning (#119)
- 03:14 PM Orchestrator Feature #43671 (Fix Under Review): cephadm: {add,remove}-repo
- 04:00 AM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
- /a/sage-2020-02-04_03:27:12-rados-wip-sage-testing-2020-02-03-1650-distro-basic-smithi/4730285
centos 7.6. others... - 03:28 AM RADOS Bug #43813 (Pending Backport): objecter doesn't send osd_op
02/03/2020
- 10:53 PM Ceph Wiki edit: CDM_05-FEB-2020 (#2)
- 05:01 PM Orchestrator Feature #43671 (In Progress): cephadm: {add,remove}-repo
02/02/2020
- 03:22 PM mgr Bug #43946 (Triaged): mimic: EINVAL on 'osd dump'
- this is actually on the mimic version. shortly after cluster creation, but hte mgr was active and should have provid...
- 03:18 PM mgr Bug #43946 (Resolved): mimic: EINVAL on 'osd dump'
- ...
- 03:17 PM Orchestrator Bug #43945: Test failure: test_osd_create (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli)
- /a/sage-2020-02-01_20:59:41-rados-wip-sage-testing-2020-02-01-1055-distro-basic-smithi/4725047
02/01/2020
- 04:53 PM RADOS Bug #43861: ceph_test_rados_watch_notify hang
- same?
/a/sage-2020-02-01_03:27:35-rados-wip-sage-testing-2020-01-31-1746-distro-basic-smithi/4723146
ceph_test_wa... - 04:50 PM Orchestrator Bug #43945 (Resolved): Test failure: test_osd_create (tasks.mgr.test_orchestrator_cli.TestOrchest...
- ...
01/31/2020
- 11:31 PM RADOS Bug #43795 (Pending Backport): Ceph tools utilizing "global_[pre_]init" no longer process "early"...
- 11:30 PM mgr Bug #43886 (Pending Backport): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
- 10:55 PM mgr Bug #38614: Disable SSLv3 on ceph-mgr?
- I tried this against current master/octopus (containerized):...
- 10:34 PM RADOS Bug #43365 (Closed): Nautilus: Random mon crashes in failed assertion at ceph::time_detail::signe...
- FWIW teh two clusters reporting this crash via telemetry are both ubuntu 18.04
closing this as not a ceph issue; l... - 07:49 PM bluestore Bug #43904 (Fix Under Review): bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given...
- 06:02 PM RADOS Bug #43813 (Fix Under Review): objecter doesn't send osd_op
- 03:50 AM RADOS Bug #43813 (In Progress): objecter doesn't send osd_op
- 03:46 AM RADOS Bug #43813: objecter doesn't send osd_op
- /a/sage-2020-01-30_22:27:29-rados-wip-sage-testing-2020-01-30-1230-distro-basic-smithi/4719487...
- 05:10 PM Orchestrator Feature #43682 (Resolved): fix mds groupings
- 05:09 PM Orchestrator Feature #43679 (Resolved): run host-check periodically
- 05:04 PM Orchestrator Feature #43940 (Resolved): orchestrator mgr add and rm
- these are useful, and not having them is incongruous with the other services.
- 03:56 AM RADOS Bug #42977 (Resolved): mon/Elector.cc: FAILED ceph_assert(m->epoch == get_epoch())
- 03:55 AM CephFS Bug #40867 (In Progress): mgr: failover during in qa testing causes unresponsive client warnings
- Another one:
/a/sage-2020-01-30_22:27:29-rados-wip-sage-testing-2020-01-30-1230-distro-basic-smithi/4719492
01/30/2020
- 06:46 PM bluestore Bug #43904: bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upgrade)
- I think the root problem is that the alloc unit was 64k (bluefs_shared_alloc_size=64k) but there was a more granular ...
- 03:02 PM bluestore Bug #43904: bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upgrade)
- ...
- 02:56 PM bluestore Bug #43904 (Resolved): bluefs _replay invalid extent 1: 0x9d4a2a000~100000: already given (on upg...
- ...
- 06:40 PM mgr Bug #43886 (Fix Under Review): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
- 04:43 PM RADOS Bug #43602 (Fix Under Review): Core dumps not collected in standalone tests for distros using sys...
- 04:43 PM RADOS Bug #43602 (Resolved): Core dumps not collected in standalone tests for distros using systemd-cor...
- 04:41 PM RADOS Bug #43889: expected MON_CLOCK_SKEW but got none
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718332
- 04:16 PM RADOS Bug #43889: expected MON_CLOCK_SKEW but got none
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718133
- 04:40 PM RADOS Bug #43915 (New): leaked Session (alloc from OSD::ms_handle_authentication)
- ...
- 04:37 PM RADOS Bug #43914 (Need More Info): nautilus: ceph tell command times out
- see https://github.com/ceph/ceph/pull/32989
- 04:35 PM RADOS Bug #43914 (Resolved): nautilus: ceph tell command times out
- ...
- 04:17 PM RADOS Bug #43864: osd/repro_long_log.sh failure
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718221
- 04:17 PM RADOS Bug #43885: failed to reach quorum size 9 before timeout expired
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718154
description: rados/... - 04:16 PM Orchestrator Bug #43913: test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): AssertionError: 22 ...
- https://github.com/ceph/ceph/pull/32881 was part of the test branch... that's maybe the cause?
- 04:15 PM Orchestrator Bug #43913 (Resolved): test_error (tasks.mgr.test_orchestrator_cli.TestOrchestratorCli): Assertio...
- ...
- 04:12 PM Orchestrator Bug #43657: test_host_ls test fails
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4717990
- 04:12 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4717957
- 04:11 PM Dashboard Bug #43912 (Resolved): FAIL: test_safe_to_destroy (tasks.mgr.dashboard.test_osd.OsdTest)
- ...
- 03:40 PM Orchestrator Feature #43911 (Resolved): test cephadm rgw deployment
- 03:34 PM ceph-volume Bug #43899: cephadm: Remove the clutch between Teuthology and ceph-volume
- workaround that we'll need to revert/clean up: https://github.com/ceph/ceph/pull/32972/commits/614c0eb77eb44dd7416561...
- 03:22 PM bluestore Bug #42823 (Duplicate): crash in BlueStore::Onode destructor
- Going to optimistically assume this is a dup of #43147 / #43131
- 03:18 PM bluestore Bug #43068 (Need More Info): on disk size (81292) does not match object info size (81237)
- was this bluestore or filestore?
- 03:17 PM bluestore Bug #43183 (Can't reproduce): Segmentation fault in tcmalloc when create osd
- 03:11 PM bluestore Bug #43370: OSD crash in function bluefs::_flush_range with ceph_abort_msg "bluefs enospc"
- crash sig (for the record):...
- 03:10 PM bluestore Bug #42010 (Can't reproduce): segv in BlueStore::OnodeSpace::lookup during deletions
- 02:51 PM RADOS Bug #43903: osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
- the second time,...
- 02:50 PM RADOS Bug #43903: osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
- if i start the osd manually, i can reproduce the same crash:...
- 02:48 PM RADOS Bug #43903 (Resolved): osd segv in ceph::buffer::v14_2_0::ptr::release (PGTempMap::decode)
- ...
- 01:18 PM Messengers Bug #43808: segv in MonClient::handle_auth_done
- /a/sage-2020-01-29_20:14:58-rados-wip-sage-testing-2020-01-29-1034-distro-basic-smithi/4718136
ceph-osd.4.log
d... - 04:11 AM RADOS Bug #43892 (Pending Backport): mon/PaxosService.cc: 188: FAILED ceph_assert(have_pending) during ...
01/29/2020
- 11:48 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
- ok, a similar problem:...
- 05:16 PM Orchestrator Bug #43883 (Need More Info): cephadm: Found left-over process 15516 (podman) in control group whi...
- 05:16 PM Orchestrator Bug #43883: cephadm: Found left-over process 15516 (podman) in control group while starting unit....
- ...
- 01:09 PM Orchestrator Bug #43883 (Resolved): cephadm: Found left-over process 15516 (podman) in control group while sta...
- ...
- 11:18 PM RADOS Bug #43892 (Fix Under Review): mon/PaxosService.cc: 188: FAILED ceph_assert(have_pending) during ...
- 11:15 PM RADOS Bug #43892 (Resolved): mon/PaxosService.cc: 188: FAILED ceph_assert(have_pending) during n->o upg...
- ...
- 08:54 PM mgr Bug #43886 (In Progress): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
- 08:30 PM mgr Bug #43886: FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
- pass insights report has...
- 02:54 PM mgr Bug #43886: FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
- about a 50% failure rate. os independent.
http://pulpito.ceph.com/sage-2020-01-29_13:33:59-rados-wip-sage-testing... - 01:33 PM mgr Bug #43886 (Resolved): FAIL: test_health_history (tasks.mgr.test_insights.TestInsights)
- ...
- 07:39 PM RADOS Bug #42977: mon/Elector.cc: FAILED ceph_assert(m->epoch == get_epoch())
- on mon.g (3), the epoch is 55 (or looks that way, it just sent these):...
- 03:55 PM RADOS Bug #43882 (Need More Info): osd to mon connection lost, osd stuck down
- adding debug: https://github.com/ceph/ceph/pull/32968
- 01:06 PM RADOS Bug #43882 (Can't reproduce): osd to mon connection lost, osd stuck down
- This is a similar symptom to #43825, but it does not appear to be related to split/merge.
OSD is marked down, but ... - 02:51 PM Ceph Revision dd1fe6ac (ceph): Merge pull request #32957 from tchaikov/wip-mgr-cephadm
- mgr/cephadm: init attrs created by settattr()
- 01:45 PM RADOS Bug #43889 (Resolved): expected MON_CLOCK_SKEW but got none
- description: rados/multimon/{clusters/6.yaml msgr-failures/many.yaml msgr/async.yaml
no_pools.yaml objectstore... - 01:44 PM RADOS Bug #43888: osd/osd-bench.sh 'tell osd.N bench' hang
- https://github.com/ceph/ceph/pull/32961 to debug
- 01:41 PM RADOS Bug #43888 (Resolved): osd/osd-bench.sh 'tell osd.N bench' hang
- ...
- 01:36 PM RADOS Bug #43887 (Resolved): ceph_test_rados_delete_pools_parallel failure
- ...
- 01:24 PM bluestore Bug #43814 (Pending Backport): common/bl: claim_append() corrupts memory when a bl consecutively ...
- 01:23 PM rgw Bug #43661 (Resolved): failure in test_sse_kms_post_object_authenticated_request
- 01:23 PM RADOS Bug #43825 (Pending Backport): osd stuck down
- 01:14 PM RADOS Bug #43885 (Can't reproduce): failed to reach quorum size 9 before timeout expired
- This pops up occasionally. Here is a recent one:...
- 12:05 AM Orchestrator Feature #43867 (Resolved): cephadm: progress item for upgrade
- show a progress item for upgrade based on the number of services and how many are updated to the new container image.
01/28/2020
- 08:26 PM rgw Bug #43661 (Fix Under Review): failure in test_sse_kms_post_object_authenticated_request
- 08:24 PM rgw Bug #43842 (Duplicate): s3-tests failure in rados suite
- 12:30 AM rgw Bug #43842 (Duplicate): s3-tests failure in rados suite
- ...
- 08:09 PM mgr Bug #43837 (Pending Backport): mgr/devicehealth: fix telemetry stops sending device reports after...
- 08:08 PM RADOS Bug #38345 (Pending Backport): mon: segv in MonOpRequest::~MonOpRequest OpHistory::cleanup
- 08:07 PM RADOS Bug #43826 (Resolved): osd: leak of from send_lease
- 08:07 PM RADOS Bug #43865 (Resolved): osd-scrub-test.sh fails date check
- ...
- 08:05 PM CephFS Bug #40867: mgr: failover during in qa testing causes unresponsive client warnings
- another instance of this on master,...
- 08:03 PM RADOS Bug #43864 (Resolved): osd/repro_long_log.sh failure
- ...
- 08:01 PM teuthology Bug #43863 (Can't reproduce): mkdir: cannot create directory ‘/home/ubuntu/cephtest/archive/audit...
- ...
- 07:59 PM RADOS Bug #43862 (Can't reproduce): mkfs fsck found fatal error: (2) No such file or directory during c...
- ...
- 07:45 PM RADOS Bug #43861: ceph_test_rados_watch_notify hang
- /a/sage-2020-01-28_03:52:05-rados-wip-sage2-testing-2020-01-27-1839-distro-basic-smithi/4713217
- 07:43 PM RADOS Bug #43861 (Resolved): ceph_test_rados_watch_notify hang
- ...
- 07:34 PM RADOS Bug #43825 (Fix Under Review): osd stuck down
- 07:27 PM RADOS Bug #43825 (In Progress): osd stuck down
- we are splitting:...
- 06:59 PM RADOS Bug #43825: osd stuck down
- 2020-01-28T14:56:26.155+0000 7fd3ba08d700 20 osd.6 285 identify_splits_and_merges 1.5 e245 to e285 pg_nums {76=28,89=...
- 06:39 PM RADOS Bug #43825: osd stuck down
- ...
- 04:34 PM teuthology Bug #43843 (Resolved): ERROR: paramiko 2.7.1 has requirement cryptography>=2.5, but you'll have c...
- 04:34 PM teuthology Bug #43799 (Resolved): Running ceph-ansible task on older releases no longer works with python2.x
- 04:33 PM teuthology Bug #43798 (Resolved): ceph-ansible teuthology task fails with AttributeError
- 03:17 PM Ceph Bug #40683 (Can't reproduce): selinux allow ceph_t to call sudo
- I'm not able to reproduce this (from a cephadm mon container on rhel 8 or centos 7.6), and we're not seeing this deni...
Also available in: Atom