Activity
From 03/09/2018 to 04/07/2018
04/07/2018
04/06/2018
- 08:39 PM Bug #23017 (Resolved): mgr log spamming about down osds
- 08:39 PM Backport #23224 (Resolved): luminous: mgr log spamming about down osds
- 07:27 PM Backport #23224: luminous: mgr log spamming about down osds
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/21053
merged - 08:38 PM Bug #23037 (Resolved): mgr not reporting when ports conflict
- 08:37 PM Backport #23175 (Resolved): luminous: mgr not reporting when ports conflict
- 07:24 PM Backport #23175: luminous: mgr not reporting when ports conflict
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/20712
merged - 06:03 PM Bug #23167 (Resolved): mgr: prometheus: ceph_pg metrics reported by prometheus plugin inconsisten...
- 03:59 PM Bug #23167: mgr: prometheus: ceph_pg metrics reported by prometheus plugin inconsistent with "cep...
- John Spray wrote:
> https://github.com/ceph/ceph/pull/20642
merged - 02:39 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- Thanks -- looks like we have a deadlock bug in ceph-mgr itself.
This thread has taken the (shared) Objecter lock a... - 02:22 PM Feature #23574 (New): Add a HeartbeatMap to ceph-mgr (die on deadlocks)
- In issues such as https://tracker.ceph.com/issues/23460, a deadlock can manifest as a completely stuck daemon. We sh...
04/05/2018
- 04:58 PM Bug #22457: ceph-mgr dashboard has dependency on python-jinja2
- merged https://github.com/ceph/ceph/pull/20748
- 06:29 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- I've attached to a hanging mgr process (eating 40% CPU) and the output of GDB is attached.
Things to know about th...
04/04/2018
- 03:57 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- Wido: so I guess the monclient messages are more of a symptom of something else getting badly stuck -- I suspect that...
- 08:52 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- And I also found one cpu core, which was allocated to this osd by cgroup, had 100% us time.
And the osd is still ali... - 08:32 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- I ran into the same issue from osd, which leads to slow request. (Luminous 12.2.4)
The primary osd's ops were blocke... - 06:12 AM Bug #22457: ceph-mgr dashboard has dependency on python-jinja2
- follow-on fix backported to luminous via https://github.com/ceph/ceph/pull/21233
- 06:09 AM Bug #22457 (Resolved): ceph-mgr dashboard has dependency on python-jinja2
- 06:08 AM Bug #22457 (Pending Backport): ceph-mgr dashboard has dependency on python-jinja2
03/30/2018
- 05:18 AM Bug #22424 (Resolved): balancer should warn about missing requirements
- 02:27 AM Backport #22983 (Resolved): luminous: balancer should warn about missing requirements
03/29/2018
- 01:22 PM Backport #22983: luminous: balancer should warn about missing requirements
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/20359
merged - 08:18 AM Bug #23083 (Resolved): ceph-mgr fails to start after a system reboot on Ubuntu 16.04
- 08:17 AM Backport #23101 (Resolved): luminous: ceph-mgr fails to start after a system reboot on Ubuntu 16.04
03/28/2018
- 10:28 PM Backport #23101: luminous: ceph-mgr fails to start after a system reboot on Ubuntu 16.04
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/20604
merged - 04:25 PM Bug #23219 (Resolved): Update mgr/restful documentation
- 04:25 PM Backport #23230 (Resolved): luminous: Update mgr/restful documentation
- 03:35 PM Bug #23482: ceph-mgr --help stopped working (regression in master)
- If my regression hypothesis is correct, the "culprit" would be one or more of these:...
- 02:39 PM Bug #23482 (Resolved): ceph-mgr --help stopped working (regression in master)
- (Opening with increased priority since this appears to be a regression)
With a recent master, ceph-mgr --help stop... - 08:31 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- So I set the logging of the active manager to:
- debug_mgr: 20
- debug_auth: 5
After running for over 24 hours... - 02:31 AM Backport #23416 (Need More Info): luminous: mgr sends early beacon with no modules reported
- File: src/mgr/PyModule.h needs to be added to luminous to backport this PR.
03/27/2018
- 10:32 AM Dashboard Bug #23406 (Resolved): Attempt to set dashboard login credentials causes ceph-mgr to crash in Pyt...
- 07:18 AM Backport #23224 (In Progress): luminous: mgr log spamming about down osds
- https://github.com/ceph/ceph/pull/21053
03/26/2018
- 02:21 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- Raising priority since this affects a stable release (Luminous)
- 01:52 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- I just found another mgr in this cluster which is "dead", but is eating 300% CPU on 3 cores:...
- 01:39 PM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- I found another mgr in a cluster which has been down now for 4 days....
- 11:05 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- I don't think the TZ is related. In all the cases I've seen the Mon and Mgr were running on the same node.
I saw i... - 10:50 AM Bug #23460: mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too...
- This one's still a mystery to me :-/
Only initial thought is that we're in daylight-savings territory right now, c... - 08:46 AM Bug #23460 (Resolved): mgr deadlock: _check_auth_rotating possible clock skew, rotating keys expi...
- On Luminous v12.2.2 and v12.2.4 clusters running either CentOS or Ubuntu I've seen many Manager going offline with th...
- 10:00 AM Dashboard Bug #23404 (Resolved): dashboard module does not work in Python 3-only environment
- 08:40 AM Bug #22226 (Rejected): ceph zabbix plugin sends incorrect motinoring info to zabbix server
- I this one still active? Otherwise we can close it I think.
Setting it to Rejected for now as I think it is resolv...
03/25/2018
- 03:19 PM Bug #23205: Blocked requests no longer show details
- We continue to get these with no clues as to where to go next:
2018-03-25 06:58:39.693404 7fa0e9ee2700 0 log_chan...
03/23/2018
- 02:03 AM Dashboard Bug #23326 (Resolved): mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot ...
- Fixed by: https://github.com/ceph/ceph/pull/20986
03/22/2018
- 12:21 PM Dashboard Bug #23404 (Fix Under Review): dashboard module does not work in Python 3-only environment
- PR: https://github.com/ceph/ceph/pull/21006
- 08:30 AM Dashboard Bug #23404 (In Progress): dashboard module does not work in Python 3-only environment
- 12:13 PM Dashboard Bug #23406 (Fix Under Review): Attempt to set dashboard login credentials causes ceph-mgr to cras...
- PR: https://github.com/ceph/ceph/pull/21005
- 12:12 PM Dashboard Bug #23406: Attempt to set dashboard login credentials causes ceph-mgr to crash in Python 3-only ...
- The current `handle_pyerror` function implementation relies in the `traceback.format_exception_only` python function ...
- 08:31 AM Dashboard Bug #23406 (In Progress): Attempt to set dashboard login credentials causes ceph-mgr to crash in ...
03/20/2018
- 11:20 AM Backport #23409 (In Progress): luminous: mgr: fix MSG_MGR_MAP handling
- https://github.com/ceph/ceph/pull/20973
03/19/2018
- 08:29 PM Bug #23418 (Won't Fix): doc: Dashboard account creation
- The documentation at http://docs.ceph.com/docs/master/mgr/dashboard/ refers the the following incorrect command:
... - 04:43 PM Backport #23416 (Rejected): luminous: mgr sends early beacon with no modules reported
- 04:42 PM Backport #23409 (Resolved): luminous: mgr: fix MSG_MGR_MAP handling
- https://github.com/ceph/ceph/pull/20973
- 03:18 PM Dashboard Bug #23406 (Resolved): Attempt to set dashboard login credentials causes ceph-mgr to crash in Pyt...
- Environment: SLE-15 (Python 3-only)
Currently running mimic_dev2 but the issue is presumed to be present on master... - 02:25 PM Dashboard Bug #23404 (Resolved): dashboard module does not work in Python 3-only environment
- Environment: SLE-15 (Python 3-only)
After enabling the dashboard, it is impossible to run "ceph mgr dashboard set-... - 01:14 AM Feature #23400 (New): ceph-mgr should raise health alert if OSDs are up but no data being received
03/17/2018
- 10:46 AM Feature #23397 (Resolved): Central "mgr self-test <module>" command
- Currently, many of the manager modules implement a "self-test" command (validate that they can fetch and parse their ...
- 10:38 AM Bug #21598 (Fix Under Review): Users can do "config-key set" while mgr runs, but it doesn't see s...
- This issue will be fixed implicitly when we merge https://github.com/ceph/ceph/pull/20458
- 10:27 AM Feature #21682 (Resolved): mgr should raise health alert when a module throws exceptions (post-load)
- 10:27 AM Dashboard Feature #22522 (Resolved): dashboard: configuration setting browser
03/16/2018
- 02:37 PM Dashboard Bug #23389 (Resolved): dashboard: OSD throughput sparkline graphic appears to show running total
- ... I suspect it's meant to show historical rates instead.
- 11:14 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- A vstart flag is easier to implement, and we should fix this problem that way in the short term. But we can also thin...
- 03:26 AM Bug #23368 (Pending Backport): mgr: fix MSG_MGR_MAP handling
- 03:24 AM Bug #23378 (Resolved): Test failure: test_perf_counters_mgr_get (tasks.mgr.dashboard_v2.test_perf...
- https://github.com/ceph/ceph/pull/20916
03/15/2018
- 07:15 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- John Spray wrote:
> I think what's going on here is that the dashboard used to fail silently (it always required rbd... - 12:40 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- I think what's going on here is that the dashboard used to fail silently (it always required rbd bindings), because v...
- 11:14 AM Bug #23378 (Resolved): Test failure: test_perf_counters_mgr_get (tasks.mgr.dashboard_v2.test_perf...
- ...
- 06:21 AM Bug #22918 (Pending Backport): mgr sends early beacon with no modules reported
- 02:10 AM Bug #23368 (Fix Under Review): mgr: fix MSG_MGR_MAP handling
- 01:26 AM Bug #23368 (Resolved): mgr: fix MSG_MGR_MAP handling
- ceph config show mgr.x doesn't work
root cause is mgr daemon's mgrc has no chance
to process MSG_MGR_MAP in the mgr...
03/14/2018
- 05:38 AM Backport #23313 (In Progress): luminous: mgr: prometheus: internal server error while new OSDs ar...
- https://github.com/ceph/ceph/pull/20891
03/13/2018
- 04:28 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- I understand the temptation but I'd prefer this be fixed. Taking out RBD compilation has significantly improved compi...
- 04:21 PM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- It's not impossible to put some special handling in the dashboard for disabling rbd functionality when it's missing, ...
- 07:19 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- Patrick Donnelly wrote:
> Omitting RBD must be why. Is there a way I can avoid this mgr error without building RBD... - 12:48 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- Oh, I'm building with:...
- 12:16 AM Dashboard Bug #23326: mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot handle comm...
- Generally that's what you'd get from an incomplete build, perhaps doing the `cephfs_testing` vstart target rather tha...
- 04:28 PM Bug #23300 (Duplicate): ceph-mgr returns internal error
- This was fixed in master recently and is being backported to luminous here: https://github.com/ceph/ceph/pull/20642
- 01:55 PM Bug #23330: mon command "mgr metadata $name" has inconsistent argument naming
- Marking for backport because will want to take it along with fix for 23286 when that's done
- 01:55 PM Bug #23330 (Fix Under Review): mon command "mgr metadata $name" has inconsistent argument naming
- This command was using `id` where all the other metadata commands were using `who`, so anyone passing `who` is gettin...
- 11:33 AM Bug #23330 (Resolved): mon command "mgr metadata $name" has inconsistent argument naming
- When running the following mon command:...
- 11:34 AM Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
- I have a potential fix (updating the metadata in DaemonServer::got_mgr_map()) but encountered http://tracker.ceph.com...
- 12:47 AM Bug #23276: balancer/osd: segfault in calc_pg_upmaps
- Hi Dan,
There is already a pending backport for Luminous, see https://github.com/ceph/ceph/pull/20840
03/12/2018
- 11:43 PM Dashboard Bug #23326 (Resolved): mgr: Error EIO: Module 'dashboard_v2' has experienced an error and cannot ...
- When running vstart against master:...
- 04:08 PM Bug #23276: balancer/osd: segfault in calc_pg_upmaps
- Great thanks.
Could someone please add the backports tag for l? - 06:42 AM Bug #23276: balancer/osd: segfault in calc_pg_upmaps
- https://github.com/ceph/ceph/pull/20655
- 09:14 AM Backport #23313 (Resolved): luminous: mgr: prometheus: internal server error while new OSDs are b...
- https://github.com/ceph/ceph/pull/21492
- 03:10 AM Bug #23205: Blocked requests no longer show details
- in other words, what is the best way to troubleshoot the following situation:
2018-03-11 22:00:00.000132 mon.roc-v...
03/11/2018
- 07:26 PM Bug #23300: ceph-mgr returns internal error
- Found it! We had several osds without a device class attached, because we did not want to use them at the moment.
Ad... - 07:20 PM Bug #23300: ceph-mgr returns internal error
- Fun fact: it used to run fine until we were introducing new crush rules and changing the crush rule for a pool:
<p... - 07:17 PM Bug #23300 (Duplicate): ceph-mgr returns internal error
- Hello,
after some weeks of running a new ceph cluster, we get the following answer from the mgr:
black3.place6:... - 05:10 AM Bug #23205: Blocked requests no longer show details
- the one big advantage of having the few slowest OSDs listed in ceph.log or MON log was the ability to go back to trou...
03/09/2018
- 01:59 PM Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
- This bug is a result of how we populate the mgrs into DaemonState from DaemonServer::got_mgr_map without ever reading...
- 01:08 PM Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
- - Hello, what's your ceph version? -
Oh, sorry. it's "ceph version 13.0.1-2832 .. mimic (dev)".
- 10:50 AM Bug #23286 (Resolved): mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname
- When running a vstart.sh cluster on master at cf52fc5a, @list_servers_python()@ returns:...
- 02:23 AM Dashboard Bug #23265 (Resolved): FAIL: test_get (tasks.mgr.dashboard_v2.test_cluster_configuration.ClusterC...
Also available in: Atom