Ceph » Orchestrator

For a command like ceph-volume if the output is too long, the call will fail. Since ceph-volume command is a part of the host refresh cycle if you get into this scenario you become unable to refresh hosts and are essentially stuck.

Sep 24, 2021, 1:24:30 PM [ERR] cephadm exited with an error code: 1,
stderr:Inferring config
/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/mon.prdrepoceph01/config
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/streams.py", line 540, in readline
    line = await self.readuntil(sep)
  File "/usr/lib/python3.8/asyncio/streams.py", line 635, in readuntil
    raise exceptions.LimitOverrunError(
asyncio.exceptions.LimitOverrunError: Separator is found, but chunk is
longer than limit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 8479, in <module>
    main()
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 8467, in main
    r = ctx.func(ctx)
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1732, in _infer_config
    return func(ctx)
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1673, in _infer_fsid
    return func(ctx)
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1760, in _infer_image
    return func(ctx)
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1660, in _validate_fsid
    return func(ctx)
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 4753, in command_ceph_volume
    out, err, code = call_throws(ctx, c.run_cmd())
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1460, in call_throws
    out, err, ret = call(ctx, command, desc, verbosity, timeout, **kwargs)
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1442, in call
    stdout, stderr, returncode = async_run(run_with_timeout())
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in
run_until_complete
    return future.result()
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1433, in run_with_timeout
    stdout, stderr = await asyncio.gather(tee(process.stdout),
  File
"/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/cephadm.f46dc95b01feeedb28941a48e2f1d0abb51139ca828de11150ea7122a8e3549c",
line 1415, in tee
    async for line in reader:
  File "/usr/lib/python3.8/asyncio/streams.py", line 738, in __anext__
    val = await self.readline()
  File "/usr/lib/python3.8/asyncio/streams.py", line 549, in readline
    raise ValueError(e.args[0])
ValueError: Separator is found, but chunk is longer than limit
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1366, in
_remote_connection
    yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1263, in _run_cephadm
    code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error
code: 1, stderr:Inferring config
/var/lib/ceph/91f80988-1cb1-11ec-867e-59d6b2eb9d9d/mon.prdrepoceph01/config
Traceback (most recent call last):
  File "/usr/lib/python3.8/asyncio/streams.py", line 540, in readline
    line = await self.readuntil(sep)
  File "/usr/lib/python3.8/asyncio/streams.py", line 635, in readuntil
    raise exceptions.LimitOverrunError(
asyncio.exceptions.LimitOverrunError: Separator is found, but chunk is
longer than limit