Bug #7737: osd: deletes vs backfill makes degraded go negative - Ceph - Ceph

Actions

Copy link

Bug #7737

closed

osd: deletes vs backfill makes degraded go negative

Added by Sage Weil about 10 years ago. Updated about 9 years ago.

Status:

Resolved

Priority:

High

Assignee:

Guang Yang

Category:

OSD

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

this is the tail end of a rados bench run that was outpacing backfill. once it started to delete objects, the degraded counts went way negative.

2014-03-15 09:40:57.395383 mon.0 10.214.131.12:6789/0 781 : [INF] pgmap v662: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 28300 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 34 op/s; 11380/35483 objects degraded (32.072%)
2014-03-15 09:40:59.806507 mon.0 10.214.131.12:6789/0 782 : [INF] pgmap v663: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 28300 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 22 op/s; 11380/35483 objects degraded (32.072%)
2014-03-15 09:41:02.391296 mon.0 10.214.131.12:6789/0 783 : [INF] pgmap v664: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 27572 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 10754/34573 objects degraded (31.105%)
2014-03-15 09:41:07.392978 mon.0 10.214.131.12:6789/0 784 : [INF] pgmap v665: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 26824 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 48 op/s; 10158/33638 objects degraded (30.198%)
2014-03-15 09:41:12.394645 mon.0 10.214.131.12:6789/0 785 : [INF] pgmap v666: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 26128 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 9558/32768 objects degraded (29.169%)
2014-03-15 09:41:17.471403 mon.0 10.214.131.12:6789/0 786 : [INF] pgmap v667: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 25408 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 35 op/s; 8972/31868 objects degraded (28.154%)
2014-03-15 09:41:22.399270 mon.0 10.214.131.12:6789/0 787 : [INF] pgmap v668: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 24696 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 35 op/s; 8386/30978 objects degraded (27.071%)
2014-03-15 09:41:27.392460 mon.0 10.214.131.12:6789/0 788 : [INF] pgmap v669: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 23988 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 35 op/s; 7784/30093 objects degraded (25.866%)
2014-03-15 09:41:32.394200 mon.0 10.214.131.12:6789/0 789 : [INF] pgmap v670: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 23300 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 34 op/s; 7212/29233 objects degraded (24.671%)
2014-03-15 09:41:37.395898 mon.0 10.214.131.12:6789/0 790 : [INF] pgmap v671: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 22536 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 6586/28278 objects degraded (23.290%)
2014-03-15 09:41:42.397533 mon.0 10.214.131.12:6789/0 791 : [INF] pgmap v672: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 21792 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 37 op/s; 5960/27348 objects degraded (21.793%)
2014-03-15 09:41:47.400036 mon.0 10.214.131.12:6789/0 792 : [INF] pgmap v673: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 21056 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 5356/26428 objects degraded (20.266%)
2014-03-15 09:41:52.401937 mon.0 10.214.131.12:6789/0 793 : [INF] pgmap v674: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 20344 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 4762/25538 objects degraded (18.647%)
2014-03-15 09:41:57.662050 mon.0 10.214.131.12:6789/0 794 : [INF] pgmap v675: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 19604 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 35 op/s; 4138/24613 objects degraded (16.812%)
2014-03-15 09:42:02.638739 mon.0 10.214.131.12:6789/0 795 : [INF] pgmap v676: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 18852 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 3516/23673 objects degraded (14.852%)
2014-03-15 09:42:07.640080 mon.0 10.214.131.12:6789/0 796 : [INF] pgmap v677: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 18124 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 2936/22763 objects degraded (12.898%)
2014-03-15 09:42:12.642899 mon.0 10.214.131.12:6789/0 797 : [INF] pgmap v678: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 17424 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 35 op/s; 2396/21888 objects degraded (10.947%)
2014-03-15 09:42:17.777997 mon.0 10.214.131.12:6789/0 798 : [INF] pgmap v679: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 16736 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 34 op/s; 1842/21028 objects degraded (8.760%)
2014-03-15 09:42:22.771149 mon.0 10.214.131.12:6789/0 799 : [INF] pgmap v680: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 15960 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 36 op/s; 1192/20058 objects degraded (5.943%)
2014-03-15 09:42:27.772621 mon.0 10.214.131.12:6789/0 800 : [INF] pgmap v681: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 15248 MB data, 65656 MB used, 1795 GB / 1863 GB avail; 0 B/s wr, 37 op/s; 608/19168 objects degraded (3.172%)
2014-03-15 09:42:31.067889 mon.0 10.214.131.12:6789/0 801 : [INF] pgmap v682: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 15248 MB data, 58469 MB used, 1802 GB / 1863 GB avail; 0 B/s wr, 22 op/s; 608/19168 objects degraded (3.172%)
2014-03-15 09:42:32.134711 mon.0 10.214.131.12:6789/0 802 : [INF] pgmap v683: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 15248 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 608/19168 objects degraded (3.172%)
2014-03-15 09:42:33.328305 mon.0 10.214.131.12:6789/0 803 : [INF] pgmap v684: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 14540 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 63 op/s; 44/18283 objects degraded (0.241%)
2014-03-15 09:42:38.062998 mon.0 10.214.131.12:6789/0 804 : [INF] pgmap v685: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 13824 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 60 op/s; -550/17388 objects degraded (-3.163%)
2014-03-15 09:42:43.047803 mon.0 10.214.131.12:6789/0 805 : [INF] pgmap v686: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 13052 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 38 op/s; -1180/16423 objects degraded (-7.185%)
2014-03-15 09:42:48.057898 mon.0 10.214.131.12:6789/0 806 : [INF] pgmap v687: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 12308 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 37 op/s; -1758/15493 objects degraded (-11.347%)
2014-03-15 09:42:53.052762 mon.0 10.214.131.12:6789/0 807 : [INF] pgmap v688: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 11616 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 35 op/s; -2308/14628 objects degraded (-15.778%)
2014-03-15 09:42:58.054205 mon.0 10.214.131.12:6789/0 808 : [INF] pgmap v689: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 10912 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 34 op/s; -2870/13748 objects degraded (-20.876%)
2014-03-15 09:43:00.188807 mon.0 10.214.131.12:6789/0 809 : [INF] pgmap v690: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 10912 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 24 op/s; -2870/13748 objects degraded (-20.876%)
2014-03-15 09:43:03.056878 mon.0 10.214.131.12:6789/0 810 : [INF] pgmap v691: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 10240 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 33 op/s; -3394/12908 objects degraded (-26.294%)
2014-03-15 09:43:08.058576 mon.0 10.214.131.12:6789/0 811 : [INF] pgmap v692: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 9556 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 43 op/s; -3904/12053 objects degraded (-32.390%)
2014-03-15 09:43:13.061557 mon.0 10.214.131.12:6789/0 812 : [INF] pgmap v693: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 8788 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 36 op/s; -4498/11093 objects degraded (-40.548%)
2014-03-15 09:43:18.046537 mon.0 10.214.131.12:6789/0 813 : [INF] pgmap v694: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 8116 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 35 op/s; -5016/10253 objects degraded (-48.922%)
2014-03-15 09:43:23.056823 mon.0 10.214.131.12:6789/0 814 : [INF] pgmap v695: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 7332 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 36 op/s; -5668/9273 objects degraded (-61.124%)
2014-03-15 09:43:28.058218 mon.0 10.214.131.12:6789/0 815 : [INF] pgmap v696: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 6608 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 37 op/s; -6224/8368 objects degraded (-74.379%)
2014-03-15 09:43:33.052872 mon.0 10.214.131.12:6789/0 816 : [INF] pgmap v697: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 5888 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 36 op/s; -6764/7468 objects degraded (-90.573%)
2014-03-15 09:43:38.054544 mon.0 10.214.131.12:6789/0 817 : [INF] pgmap v698: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 5204 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 35 op/s; -7302/6613 objects degraded (-110.419%)
2014-03-15 09:43:43.064036 mon.0 10.214.131.12:6789/0 818 : [INF] pgmap v699: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 4472 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 35 op/s; -7850/5698 objects degraded (-137.768%)
2014-03-15 09:43:48.049093 mon.0 10.214.131.12:6789/0 819 : [INF] pgmap v700: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 3744 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 36 op/s; -8418/4788 objects degraded (-175.815%)
2014-03-15 09:43:53.067769 mon.0 10.214.131.12:6789/0 820 : [INF] pgmap v701: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 3032 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 35 op/s; -8942/3898 objects degraded (-229.400%)
2014-03-15 09:43:58.095632 mon.0 10.214.131.12:6789/0 821 : [INF] pgmap v702: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 2336 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 35 op/s; -9484/3028 objects degraded (-313.210%)
2014-03-15 09:44:03.105370 mon.0 10.214.131.12:6789/0 822 : [INF] pgmap v703: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 1572 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 36 op/s; -10006/2073 objects degraded (-482.682%)
2014-03-15 09:44:08.098509 mon.0 10.214.131.12:6789/0 823 : [INF] pgmap v704: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 808 MB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 38 op/s; -10500/1118 objects degraded (-939.177%)
2014-03-15 09:44:13.091695 mon.0 10.214.131.12:6789/0 824 : [INF] pgmap v705: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 40964 kB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 38 op/s; -10990/158 objects degraded (-6955.696%)
2014-03-15 09:44:14.341791 mon.0 10.214.131.12:6789/0 825 : [INF] osdmap e91: 6 osds: 6 up, 4 in
2014-03-15 09:44:14.461865 mon.0 10.214.131.12:6789/0 826 : [INF] pgmap v706: 133 pgs: 1 active+remapped+backfilling, 132 active+clean; 40964 kB data, 54027 MB used, 1806 GB / 1863 GB avail; 0 B/s wr, 30 op/s; -10990/158 objects degraded (-6955.696%)

ubuntu@teuthology:/var/lib/teuthworker/archive/teuthology-2014-03-15_02:30:01-rados-firefly-distro-basic-plana/131941

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Guang Yang about 9 years ago

We saw this with Firefly v0.80.4. The number of objects at primary is less than replicas.

-bash-4.1$ sudo ceph pg 3.1d61 query | grep num_objects\" 
              "num_objects": 105659,              // this should be the primary?
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105660,
                  "num_objects": 105659,
                  "num_objects": 105660,
                  "num_objects": 105660,

Actions

Copy link