Ceph » Orchestrator

I tried upgrading a test cluster (built on CentOS 7) from v15.2.13 to v16.2.5 today and ran into this problem:

------------------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection
    yield (conn, connr)
  File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm
    code, '\n'.join(err)))
orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman version --format {{.Client.Version}}
/bin/podman: stderr Error: template: image:1:9: executing "image" at <.Client.Version>: can't evaluate field Client in type define.Version
Traceback (most recent call last):
  File "/var/lib/ceph/170b168e-ec0a-11eb-a273-84a9c4d06c32/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in <module>
    main()
  File "/var/lib/ceph/170b168e-ec0a-11eb-a273-84a9c4d06c32/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8216, in main
    check_container_engine(ctx)
  File "/var/lib/ceph/170b168e-ec0a-11eb-a273-84a9c4d06c32/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1985, in check_container_engine
    engine.get_version(ctx)
  File "/var/lib/ceph/170b168e-ec0a-11eb-a273-84a9c4d06c32/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 188, in get_version
    out, _, _ = call_throws(ctx, [self.path, 'version', '--format', '{{.Client.Version}}'])
  File "/var/lib/ceph/170b168e-ec0a-11eb-a273-84a9c4d06c32/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
RuntimeError: Failed command: /bin/podman version --format {{.Client.Version}}

------------------------------------------------------------------------------

This issue appears to have been introduced with the v16.2.2 release. With some troubleshooting I determined that the version of podman in CentOS 7 doesn't support '{{.Client.Version}}':

1. podman version --format {{.Client.Version}}
   Error: template: image:1:9: executing "image" at <.Client.Version>: can't evaluate field Client in type define.Version

But does support it with just '.Version':

1. podman version --format {{.Version}}
   1.6.4