Project

General

Profile

Actions
Copy link

Bug #47838

open

mon/test_mon_osdmap_prune.sh: first_pinned != trim_to

Added by Neha Ojha over 3 years ago. Updated 8 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Aishwarya Mathuria
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific, octopus, quincy, reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2020-10-12T11:01:32.103 INFO:tasks.workunit.client.0.smithi057.stderr:++ jq .osdmap_first_committed
2020-10-12T11:01:32.104 INFO:tasks.workunit.client.0.smithi057.stderr:+ fc=11
2020-10-12T11:01:32.104 INFO:tasks.workunit.client.0.smithi057.stderr:+ [[ 11 -eq 11 ]]
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ old_first_pinned=1
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ old_last_pinned=921
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:++ jq .osdmap_manifest.first_pinned
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ first_pinned=1
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:++ jq .osdmap_manifest.last_pinned
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ last_pinned=931
2020-10-12T11:01:32.105 INFO:tasks.workunit.client.0.smithi057.stderr:+ [[ 1 -eq 11 ]]
2020-10-12T11:01:32.106 INFO:tasks.workunit.client.0.smithi057.stderr:+ return 1
2020-10-12T11:01:32.106 INFO:tasks.workunit.client.0.smithi057.stderr:+ exit 1

  old_first_pinned=$first_pinned
  old_last_pinned=$last_pinned
  first_pinned=$(jq '.osdmap_manifest.first_pinned' <<< $report)
  last_pinned=$(jq '.osdmap_manifest.last_pinned' <<< $report)
  [[ $first_pinned -eq $trim_to ]] || return 1

/a/teuthology-2020-10-12_07:01:02-rados-master-distro-basic-smithi/5519390

Related issues 1 (1 open — 0 closed)

Related to RADOS - Bug #54369: mon/test_mon_osdmap_prune.sh: jq .osdmap_first_committed [[ 11 -eq 20 ]]New

Actions
Actions
Copy link
 #2

Updated by Neha Ojha over 3 years ago

ok, fails 1 out 10 times but seems new, need to look more.

2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq .osdmap_first_committed
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ fc=20
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ 20 -eq 20 ]]
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ old_first_pinned=11
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ old_last_pinned=921
2020-10-12T23:55:24.502 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq .osdmap_manifest.first_pinned
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ first_pinned=11
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq .osdmap_manifest.last_pinned
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ last_pinned=931
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:+ pinned_maps=($(jq '.osdmap_manifest.pinned_maps[]' <<< $report))
2020-10-12T23:55:24.504 INFO:tasks.workunit.client.0.smithi003.stderr:++ jq '.osdmap_manifest.pinned_maps[]'
2020-10-12T23:55:24.505 INFO:tasks.workunit.client.0.smithi003.stderr:+ [[ 11 -eq 20 ]]
2020-10-12T23:55:24.505 INFO:tasks.workunit.client.0.smithi003.stderr:+ return 1

/a/nojha-2020-10-12_21:25:34-rados:monthrash-master-distro-basic-smithi/5520847

Actions
Copy link
 #3

Updated by Neha Ojha about 3 years ago

/a/yuriw-2021-03-10_21:08:51-rados-wip-yuri8-testing-2021-03-10-0901-pacific-distro-basic-smithi/5954442 - similar

Actions
Copy link
 #4

Updated by Deepika Upadhyay over 2 years ago

  • Backport set to pacific, octopus
Actions
Copy link
 #5

Updated by Deepika Upadhyay over 2 years ago 

2021-10-19T05:03:54.608 INFO:tasks.workunit.client.0.smithi139.stderr:++ jq .osdmap_first_committed
2021-10-19T05:03:54.610 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-10-19T05:03:54.611 INFO:tasks.workunit.client.0.smithi139.stderr:+ fc=11
2021-10-19T05:03:54.611 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ 11 -eq 11 ]]
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:+ old_first_pinned=1
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:+ old_last_pinned=921
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:++ jq .osdmap_manifest.first_pinned
2021-10-19T05:03:54.612 INFO:tasks.workunit.client.0.smithi139.stderr:+ first_pinned=1
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:++ jq .osdmap_manifest.last_pinned
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ last_pinned=921
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ [[ 1 -eq 11 ]]
2021-10-19T05:03:54.613 INFO:tasks.workunit.client.0.smithi139.stderr:+ return 1
2021-10-19T05:03:54.614 INFO:tasks.workunit.client.0.smithi139.stderr:+ exit 1
2021-10-19T05:03:54.614 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...

/ceph/teuthology-archive/yuriw-2021-10-18_19:03:43-rados-wip-yuri5-testing-2021-10-18-0906-octopus-distro-basic-smithi/6449226/teuthology.log

Actions
Copy link
 #6

Updated by Neha Ojha about 2 years ago

  • Related to Bug #54369: mon/test_mon_osdmap_prune.sh: jq .osdmap_first_committed [[ 11 -eq 20 ]] added
Actions
Copy link
 #7

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-03-01_17:45:51-rados-wip-yuri3-testing-2022-02-28-0757-pacific-distro-default-smithi/6714654

Actions
Copy link
 #8

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-04-01_17:44:32-rados-wip-yuri3-testing-2022-04-01-0659-distro-default-smithi/6772697

Actions
Copy link
 #9

Updated by Neha Ojha about 2 years ago

  • Assignee set to Aishwarya Mathuria

Aishwarya, can you please take a look at this bug? could be a test issue, but we should find out

Actions
Copy link
 #10

Updated by Aishwarya Mathuria about 2 years ago

Neha Ojha wrote:

Aishwarya, can you please take a look at this bug? could be a test issue, but we should find out

Sure!

Actions
Copy link
 #11

Updated by Laura Flores about 1 year ago

  • Translation missing: en.field_tag_list set to test-failure

/a/lflores-2023-02-08_20:25:06-rados-wip-lflores-testing-2023-02-06-1529-distro-default-smithi/7162262

2023-02-09T08:18:28.668 INFO:tasks.workunit.client.0.smithi119.stderr:+ fc=20
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 20 -eq 20 ]]
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ old_first_pinned=11
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ old_last_pinned=921
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq .osdmap_manifest.first_pinned
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ first_pinned=11
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq .osdmap_manifest.last_pinned
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ last_pinned=931
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ pinned_maps=($(jq '.osdmap_manifest.pinned_maps[]' <<< $report))
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:++ jq '.osdmap_manifest.pinned_maps[]'
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ 11 -eq 20 ]]
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ return 1
2023-02-09T08:18:28.669 INFO:tasks.workunit.client.0.smithi119.stderr:+ exit 1
2023-02-09T08:18:28.669 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...

Actions
Copy link
 #12

Updated by Aishwarya Mathuria about 1 year ago

From the monitor logs during OSD map trim:

2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032  min_last_epoch_clean 1030
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 get_trim_to explicit mon_osd_force_trim_to = 20
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 get_trim_to trim_to = 20
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) maybe_trim 11~20
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).paxosservice(osdmap 11..1032) maybe_trim trimming to 20, 9 states
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).paxosservice(osdmap 11..1032) trim from 11 to 20
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 11
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim full_11
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 12
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 13
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 14
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 15
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 16
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 17
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 18
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032) trim 19
2023-02-09T08:18:47.488+0000 7fae6951b700 20 mon.c@4(leader).paxosservice(osdmap 11..1032)  compacting prefix osdmap
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 encode_trim_extra including full map for e 20
2023-02-09T08:18:47.488+0000 7fae6951b700 10 mon.c@4(leader).osd e1032 _prune_update_trimmed first 20 last_pinned 931

The trim is happening, so now I am looking into why osdmap_manifest.first_pinned is not getting updated.

Actions
Copy link
 #13

Updated by Radoslaw Zarzynski about 1 year ago

  • Status changed from New to In Progress
Actions
Copy link
 #14

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7193126

2023-03-04T00:12:50.125 INFO:tasks.workunit.client.0.smithi112.stderr:+ fc=11
2023-03-04T00:12:50.125 INFO:tasks.workunit.client.0.smithi112.stderr:+ [[ 11 -eq 11 ]]
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ old_first_pinned=1
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ old_last_pinned=921
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:++ jq .osdmap_manifest.first_pinned
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ first_pinned=1
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:++ jq .osdmap_manifest.last_pinned
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ last_pinned=921
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ [[ 1 -eq 11 ]]
2023-03-04T00:12:50.126 INFO:tasks.workunit.client.0.smithi112.stderr:+ return 1
2023-03-04T00:12:50.127 INFO:tasks.workunit.client.0.smithi112.stderr:+ exit 1
2023-03-04T00:12:50.127 INFO:tasks.workunit:Stopping ['mon/test_mon_osdmap_prune.sh'] on client.0...
2023-03-04T00:12:50.127 DEBUG:teuthology.orchestra.run.smithi112:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2023-03-04T00:12:50.388 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):

Actions
Copy link
 #15

Updated by Laura Flores about 1 year ago

  • Backport changed from pacific, octopus to pacific, octopus, quincy, reef
Actions
Copy link
 #16

Updated by Radoslaw Zarzynski about 1 year ago

Is there any update on that?

Actions
Copy link
 #17

Updated by Laura Flores about 1 year ago

/a/yuriw-2023-03-30_21:53:20-rados-wip-yuri7-testing-2023-03-29-1100-distro-default-smithi/7228118

Actions
Copy link
 #19

Updated by Laura Flores 8 months ago

/a/yuriw-2023-08-17_21:18:20-rados-wip-yuri11-testing-2023-08-17-0823-distro-default-smithi/7372203

Actions
Copy link

Also available in: Atom PDF