Project

General

Profile

Actions

Support #49497

closed

Cephadm fails to upgrade from 15.2.8 to 15.2.9

Added by Illya S. about 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Tags:
Reviewed:
Affected Versions:
Pull request ID:

Description

I'm trying to upgrade cluster from 15.2.8 to 15.2.9 by running

ceph orch upgrade start --ceph-version 15.2.9

Some seconds later upgrade status shows failure:

ceph orch upgrade status 
{
    "target_image": "docker.io/ceph/ceph:v15.2.9",
    "in_progress": true,
    "services_complete": [],
    "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 
}

If I try manually pull the image, I receive such error:

podman pull docker.io/ceph/ceph:v15.2.9
Trying to pull docker.io/ceph/ceph:v15.2.9...
Getting image source signatures
Copying blob 008f517fda00 done  
Copying blob 7a0437f04f83 done  
Copying config dfc4830796 done  
Writing manifest to image destination
Storing signatures
  Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:22 for /run/utmp): Check /etc/subuid and /etc/subgid: lchown /run/utmp: invalid argument
Error: Error committing the finished image: error adding layer with blob "sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:22 for /run/utmp): Check /etc/subuid and /etc/subgid: lchown /run/utmp: invalid argument

Upgrade to 15.2.8 worked flawless.

Notes on environment:
  1. mon-s and mgr-s are running on CentOS 8.3
  2. cephadm is configured to use non-root user, as described in doc: https://docs.ceph.com/en/latest/cephadm/operations/#configuring-a-different-ssh-user
  3. in the meantime podman was upgraded from 2.0 to 2.2 (if it's related)
Actions #1

Updated by Sage Weil about 3 years ago

  • Project changed from Ceph to Orchestrator
Actions #2

Updated by Jimmy Spets about 3 years ago

I have the same issue on hosts running CentOS 8.3 with podman 2.2.1, only difference is that I run cephadm as root.

Actions #3

Updated by Sebastian Wagner about 3 years ago

  • Priority changed from Normal to Urgent
Actions #4

Updated by Sebastian Wagner about 3 years ago

podman 2.2.1 is really a bad release for cephadm.

See https://docs.ceph.com/en/latest/cephadm/compatibility/

Actions #5

Updated by Sebastian Wagner about 3 years ago

Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace

you need to pull the image as root!

Actions #6

Updated by Sebastian Wagner about 3 years ago

  • Tracker changed from Bug to Support
Actions #7

Updated by Illya S. about 3 years ago

Sebastian Wagner wrote:

Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace

you need to pull the image as root!

I checked now. Cephadm has pulled image using sudo, so I can see it as root. But upgrade still fails with "failed to pull target image" error

Actions #8

Updated by Illya S. about 3 years ago

The error is still here with 15.2.10

Stuck on 15.2.8

Actions #9

Updated by Illya S. almost 3 years ago

Illya S. wrote:

The error is still here with 15.2.10

Stuck on 15.2.8

15.2.11 -- nothing changed

Actions #10

Updated by Sebastian Wagner almost 3 years ago

I don't think this something we can fix unfortunately. Works for everyone else

Actions #11

Updated by Sebastian Wagner almost 3 years ago

  • Priority changed from Urgent to Normal
Actions #12

Updated by Sebastian Wagner over 2 years ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF