Project

General

Profile

Bug #22558

"Transaction check error" in upgrade:client-upgrade-kraken-luminous

Added by Yuri Weinstein over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
build
Target version:
-
Start date:
01/03/2018
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/client-upgrade
Pull request ID:

Description

Seems centos related
Run: http://pulpito.ceph.com/teuthology-2018-01-03_05:50:02-upgrade:client-upgrade-kraken-luminous-distro-basic-ovh/
Jobs: all centos
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2018-01-03_05:50:02-upgrade:client-upgrade-kraken-luminous-distro-basic-ovh/2019047/teuthology.log

2018-01-03T07:57:59.906 INFO:teuthology.orchestra.run.ovh042.stdout:Install   1 Package  (+12 Dependent packages)
2018-01-03T07:57:59.906 INFO:teuthology.orchestra.run.ovh042.stdout:Upgrade  11 Packages (+12 Dependent packages)
2018-01-03T07:57:59.907 INFO:teuthology.orchestra.run.ovh042.stdout:
2018-01-03T07:57:59.907 INFO:teuthology.orchestra.run.ovh042.stdout:Total download size: 63 M
2018-01-03T07:57:59.919 INFO:teuthology.orchestra.run.ovh042.stdout:Downloading packages:
2018-01-03T07:57:59.919 INFO:teuthology.orchestra.run.ovh042.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
2018-01-03T07:58:06.126 INFO:teuthology.orchestra.run.ovh042.stdout:--------------------------------------------------------------------------------
2018-01-03T07:58:06.126 INFO:teuthology.orchestra.run.ovh042.stdout:Total                                               10 MB/s |  63 MB  00:06
2018-01-03T07:58:06.160 INFO:teuthology.orchestra.run.ovh042.stdout:Running transaction check
2018-01-03T07:58:06.227 INFO:teuthology.orchestra.run.ovh042.stdout:Running transaction test
2018-01-03T07:58:06.299 INFO:teuthology.orchestra.run.ovh042.stderr:
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:Transaction check error:
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:  file /usr/bin/ceph-kvstore-tool from install of ceph-base-2:12.2.2-58.g840cc7b.el7.x86_64 conflicts with file from package ceph-test-1:11.2.1-12.gad30823.el7.x86_64
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:  file /usr/bin/ceph-osdomap-tool from install of ceph-osd-2:12.2.2-58.g840cc7b.el7.x86_64 conflicts with file from package ceph-test-1:11.2.1-12.gad30823.el7.x86_64
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:  file /usr/bin/ceph-monstore-tool from install of ceph-mon-2:12.2.2-58.g840cc7b.el7.x86_64 conflicts with file from package ceph-test-1:11.2.1-12.gad30823.el7.x86_64
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:Error Summary
2018-01-03T07:58:06.300 INFO:teuthology.orchestra.run.ovh042.stderr:-------------

Related issues

Related to Ceph - Feature #21762: Add ceph-monstore-tool in ceph-mon package, ceph-kvstore-tool in ceph-mon and ceph-osd, and ceph-osdomap-tool in ceph-osd package. Resolved 10/11/2017
Copied to Ceph - Backport #22611: luminous: "Transaction check error" in upgrade:client-upgrade-kraken-luminous Resolved

History

#1 Updated by Yuri Weinstein over 1 year ago

  • Project changed from RADOS to Ceph

#2 Updated by Kefu Chai over 1 year ago

  • Related to Feature #21762: Add ceph-monstore-tool in ceph-mon package, ceph-kvstore-tool in ceph-mon and ceph-osd, and ceph-osdomap-tool in ceph-osd package. added

#3 Updated by Kefu Chai over 1 year ago

  • Category set to build

it is a regression caused by #21762 's luminous backport.

#4 Updated by Kefu Chai over 1 year ago

  • Status changed from New to Need Review
  • Assignee set to Kefu Chai
  • Backport set to luminous

#5 Updated by Kefu Chai over 1 year ago

Yuri, Kraken was EOL in August 2017. why were you testing the upgrade path from kraken to luminous?

#6 Updated by Kefu Chai over 1 year ago

  • Status changed from Need Review to Pending Backport

#7 Updated by Kefu Chai over 1 year ago

  • Copied to Backport #22611: luminous: "Transaction check error" in upgrade:client-upgrade-kraken-luminous added

#8 Updated by Nathan Cutler over 1 year ago

  • Status changed from Pending Backport to Resolved

#10 Updated by Yuri Weinstein over 1 year ago

  • Status changed from Resolved to New

Reopening for now so @kefu can take a lool

#11 Updated by Nathan Cutler over 1 year ago

The command causing this error is: 2018-01-03T07:57:45.906 INFO:teuthology.orchestra.run.ovh042:Running: "sudo yum -y install '' libcephfs_jni1 rbd-fuse ceph-radosgw librbd1 ceph-fuse python-ceph f ceph-mgr ceph librados2 cephfs-java libcephfs2 libcephfs-devel"

This is, essentially, the list of packages to be upgraded. If ceph-test is added to the list, the error will go away?

#12 Updated by Nathan Cutler over 1 year ago

Ah, ceph-test is explicitly excluded from the list of packages:

  upgrade_workload:
    sequential:
    - install.upgrade:
        client.0: null
        exclude_packages:
        - ceph-test
        - ceph-test-dbg
        - libcephfs1

#13 Updated by Nathan Cutler over 1 year ago

  sha1: 840cc7b1747fe12e42e7e182afd8298d1f8684b0 <-- luminous
...
  suite_sha1: ad30823c0d7544205669d09d1a6b0279f29ceab4 <-- kraken (!)

So this test is run with

--suite-branch kraken
. . . Do we really need to be running that? (If so, we will have to backport any test fixes to the kraken branch)

#14 Updated by Yuri Weinstein over 1 year ago

@Nathan

We do need to use `--suite-branch` as we agreed to be using clients from the appropriate named branches

https://github.com/ceph/ceph/tree/jewel/qa/suites/upgrade/client-upgrade-jewel/jewel-client-x
https://github.com/ceph/ceph/tree/kraken/qa/suites/upgrade/client-upgrade-kraken/kraken-client-x
https://github.com/ceph/ceph/tree/hammer/qa/suites/upgrade/client-upgrade-hammer/hammer-client-x

there also was a technical reason why we had to do so, but I can't recall ATM (IIRC we wanted to install correct packages on older clients w/o problems)

#15 Updated by Kefu Chai over 1 year ago

we still need to add: ceph-{base,osd,mon} >= 12.2.2.8 conflict with ceph-test < 12.2.2-8

because in kraken, ceph-test requires ceph-common without tying it with any specific version. so if we install, for instance, ceph-base (>= 12.2.2.8) to a system with ceph-test (kraken), we have no way to remove the conflicting old ceph-test.

but we are still facing a dilemma here: the test at https://github.com/ceph/ceph/blob/kraken/qa/suites/upgrade/client-upgrade-kraken/kraken-client-x/basic/1-install/kraken-client-x.yaml#L9 needs to preserve the old ceph-test for verifying that the kraken client still behaves when working with luminous.

to address this issue, probably we can add all the conflicting packages, namely ceph-{base,mon,osd}, to the exclude list of client.0 's install.upgrade task. because client.0 only hosts the old clients in this very test.

in short, two tasks:

  • in master and luminous: ceph-{base,osd,mon} >= 12.2.2.8 conflict with ceph-test < 12.2.2-8
  • in kraken: exclude ceph-{base,mon,osd} also.

#17 Updated by Nathan Cutler over 1 year ago

@Yuri, ok we want to test {hammer,jewel,kraken} clients against luminous clusters but these tests don't actually do that:

https://github.com/ceph/ceph/tree/jewel/qa/suites/upgrade/client-upgrade-jewel/jewel-client-x
https://github.com/ceph/ceph/tree/kraken/qa/suites/upgrade/client-upgrade-kraken/kraken-client-x
https://github.com/ceph/ceph/tree/hammer/qa/suites/upgrade/client-upgrade-hammer/hammer-client-x

What they do is install a {hammer,jewel,kraken} cluster (including client) and then upgrade the client to luminous.

#18 Updated by Yuri Weinstein over 1 year ago

@Nathan

After thinking about it and @jdurgin reminded that since we have workloads running before upgrading the client it seems we have all what we need. Agree?

#19 Updated by Nathan Cutler over 1 year ago

  • Assignee changed from Kefu Chai to Nathan Cutler

#20 Updated by Nathan Cutler over 1 year ago

@Yuri Since the test is fixable, we might as well keep running it :-)

#21 Updated by Nathan Cutler over 1 year ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF