Bug #47838: mon/test_mon_osdmap_prune.sh: first_pinned != trim_to - RADOS - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
Pending backports
Product Backlog Scrub

Actions

Copy link

Bug #47838

open

mon/test_mon_osdmap_prune.sh: first_pinned != trim_to

Added by Neha Ojha over 3 years ago. Updated 9 months ago.

Status:

In Progress

Priority:

Normal

Assignee:

Aishwarya Mathuria

Category:

Target version:

% Done:

Source:

Tags:

Backport:

pacific, octopus, quincy, reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2020-10-12T11:01:32.103 INFO:tasks.workunit.client.0.smithi057.stderr:++ jq .osdmap_first_committed
2020-10-12T11:01:32.104 INFO:tasks.workunit.client.0.smithi057.stderr:+ fc=11
2020-10-12T11:01:32.104 INFO:tasks.workunit.client.0.smithi057.stderr:+ [[ 11 -eq 11 ]]
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ old_first_pinned=1
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ old_last_pinned=921
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:++ jq .osdmap_manifest.first_pinned
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ first_pinned=1
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:++ jq .osdmap_manifest.last_pinned
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ last_pinned=931
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ [[ 1 -eq 11 ]]
2020-10-12T11:01:32.106 INFO:tasks.workunit.client.0.smithi057.stderr:+ return 1
2020-10-12T11:01:32.106 INFO:tasks.workunit.client.0.smithi057.stderr:+ exit 1

  old_first_pinned=$first_pinned
  old_last_pinned=$last_pinned
  first_pinned=$(jq '.osdmap_manifest.first_pinned' <<< $report)
  last_pinned=$(jq '.osdmap_manifest.last_pinned' <<< $report)
  [[ $first_pinned -eq $trim_to ]] || return 1

/a/teuthology-2020-10-12_07:01:02-rados-master-distro-basic-smithi/5519390

Related issues 1 (1 open — 0 closed)

Related to RADOS - Bug #54369: mon/test_mon_osdmap_prune.sh: jq .osdmap_first_committed [[ 11 -eq 20 ]]

New

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Neha Ojha over 3 years ago

Did not reproduce here: https://pulpito.ceph.com/nojha-2020-10-12_19:42:27-rados:monthrash-master-distro-basic-smithi/
Trying with more jobs: https://pulpito.ceph.com/nojha-2020-10-12_21:25:34-rados:monthrash-master-distro-basic-smithi/

Actions

Copy link

Updated by Neha Ojha over 3 years ago

ok, fails 1 out 10 times but seems new, need to look more.

2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq .osdmap_first_committed
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ fc=20
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ 20 -eq 20 ]]
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ old_first_pinned=11
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ old_last_pinned=921
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq .osdmap_manifest.first_pinned
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ first_pinned=11
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq .osdmap_manifest.last_pinned
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ last_pinned=931
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ pinned_maps=($(jq '.osdmap_manifest.pinned_maps[]' <<< $report))
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq '.osdmap_manifest.pinned_maps[]'
2020-10-12T23:55:24.505 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ 11 -eq 20 ]]
2020-10-12T23:55:24.505 INFO:tasks.workunit.client.0.smithi003.stderr:+ return 1

/a/nojha-2020-10-12_21:25:34-rados:monthrash-master-distro-basic-smithi/5520847

Actions

Copy link

Updated by Neha Ojha about 3 years ago

/a/yuriw-2021-03-10_21:08:51-rados-wip-yuri8-testing-2021-03-10-0901-pacific-distro-basic-smithi/5954442 - similar

Actions

Copy link

Updated by Deepika Upadhyay over 2 years ago

Backport set to pacific, octopus

Actions

Copy link

Updated by Deepika Upadhyay over 2 years ago

2021-10-19T05:03:54.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ jq .osdmap_first_committed
2021-10-19T05:03:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-10-19T05:03:54.611 INFO:tasks.workunit.client.0.smithi139.stderr:+ fc=11
2021-10-19T05:03:54.611 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ 11 -eq 11 ]]
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:+ old_first_pinned=1
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:+ old_last_pinned=921
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ jq .osdmap_manifest.first_pinned
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:+ first_pinned=1
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:++ jq .osdmap_manifest.last_pinned
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ last_pinned=921
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ 1 -eq 11 ]]
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ return 1
2021-10-19T05:03:54.614 INFO:tasks.workunit.client.0.smithi139.stderr:+ exit 1
2021-10-19T05:03:54.614 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...

/ceph/teuthology-archive/yuriw-2021-10-18_19:03:43-rados-wip-yuri5-testing-2021-10-18-0906-octopus-distro-basic-smithi/6449226/teuthology.log

Actions

Copy link

Updated by Neha Ojha about 2 years ago

Related to Bug #54369: mon/test_mon_osdmap_prune.sh: jq .osdmap_first_committed [[ 11 -eq 20 ]] added

Actions

Copy link

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714654

Actions

Copy link

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-04-01_17:44:32-rados-wip-yuri3-testing-2022-04-01-0659-distro-default-smithi/6772697

Actions

Copy link

Updated by Neha Ojha about 2 years ago

Assignee set to Aishwarya Mathuria

Aishwarya, can you please take a look at this bug? could be a test issue, but we should find out

Actions

Copy link

#10

Updated by Aishwarya Mathuria about 2 years ago

Neha Ojha wrote:

Aishwarya, can you please take a look at this bug? could be a test issue, but we should find out

Sure!

Actions

Copy link

#11

Updated by Laura Flores about 1 year ago

Translation missing: en.field_tag_list set to test-failure

/a/lflores-2023-02-08_20:25:06-rados-wip-lflores-testing-2023-02-06-1529-distro-default-smithi/7162262

2023-02-09T08:18:28.668 INFO:tasks.workunit.client.0.smithi119.stderr:+ fc=20
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 20 -eq 20 ]]
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ old_first_pinned=11
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ old_last_pinned=921
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq .osdmap_manifest.first_pinned
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ first_pinned=11
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq .osdmap_manifest.last_pinned
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ last_pinned=931
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ pinned_maps=($(jq '.osdmap_manifest.pinned_maps[]' <<< $report))
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.osdmap_manifest.pinned_maps[]'
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 11 -eq 20 ]]
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ return 1
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ exit 1
2023-02-09T08:18:28.669 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...

Actions

Copy link

#12

Updated by Aishwarya Mathuria about 1 year ago

From the monitor logs during OSD map trim:

2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032  min_last_epoch_clean 1030
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 get_trim_to explicit mon_osd_force_trim_to = 20
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 get_trim_to trim_to = 20
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) maybe_trim 11~20
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).paxosservice(osdmap 11..1032) maybe_trim trimming to 20, 9 states
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).paxosservice(osdmap 11..1032) trim from 11 to 20
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 11
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim full_11
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 12
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 13
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 14
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 15
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 16
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 17
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 18
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 19
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032)  compacting prefix osdmap
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 encode_trim_extra including full map for e 20
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 _prune_update_trimmed first 20 last_pinned 931

The trim is happening, so now I am looking into why osdmap_manifest.first_pinned is not getting updated.

Actions

Copy link

#13

Updated by Radoslaw Zarzynski about 1 year ago

Status changed from New to In Progress

Actions

Copy link

#14

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7193126

2023-03-04T00:12:50.125 INFO:tasks.workunit.client.0.smithi112.stderr:+ fc=11
2023-03-04T00:12:50.125 INFO:tasks.workunit.client.0.smithi112.stderr:+ [[ 11 -eq 11 ]]
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ old_first_pinned=1
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ old_last_pinned=921
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:++ jq .osdmap_manifest.first_pinned
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ first_pinned=1
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:++ jq .osdmap_manifest.last_pinned
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ last_pinned=921
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ [[ 1 -eq 11 ]]
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ return 1
2023-03-04T00:12:50.127 INFO:tasks.workunit.client.0.smithi112.stderr:+ exit 1
2023-03-04T00:12:50.127 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...
2023-03-04T00:12:50.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2023-03-04T00:12:50.388 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Actions

Copy link

#15