Project

General

Profile

Bug #54985

pacific -> quincy upgrade bucket stats starts returning -2

Added by Vikhyat Umrao about 2 years ago. Updated about 2 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

root@f13-h26-b04-5039ms:~
# radosgw-admin bucket stats |egrep 'bucket\"|num_objects|num_shards' 
error getting bucket stats bucket=mycontainers3 ret=-2
error getting bucket stats bucket=mycontainers4 ret=-2
error getting bucket stats bucket=mycontainers5 ret=-2
        "bucket": "mycontainers1",
        "num_shards": 1381,
                "num_objects": 5073379
        "bucket": "mycontainers2",
        "num_shards": 1499,
                "num_objects": 5072180

root@f13-h26-b04-5039ms:~
# ceph versions
{
    "mon": {
        "ceph version 17.1.0-56-g60fdd357 (60fdd357d1aeaa5f5a03a623dc666183c6bee5d5) quincy (dev)": 3
    },
    "mgr": {
        "ceph version 17.1.0-56-g60fdd357 (60fdd357d1aeaa5f5a03a623dc666183c6bee5d5) quincy (dev)": 3
    },
    "osd": {
        "ceph version 17.1.0-56-g60fdd357 (60fdd357d1aeaa5f5a03a623dc666183c6bee5d5) quincy (dev)": 288
    },
    "mds": {},
    "rgw": {
        "ceph version 17.1.0-56-g60fdd357 (60fdd357d1aeaa5f5a03a623dc666183c6bee5d5) quincy (dev)": 12
    },
    "overall": {
        "ceph version 17.1.0-56-g60fdd357 (60fdd357d1aeaa5f5a03a623dc666183c6bee5d5) quincy (dev)": 306
    }
}

# radosgw-admin bucket stats --bucket=mycontainers1
{
    "bucket": "mycontainers1",
    "num_shards": 1381,
    "tenant": "",
    "zonegroup": "2b98c1de-086d-4f6f-a042-3110874d69ff",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": "" 
    },
    "id": "32677260-5393-40e0-b228-a1ebd6184bc3.51110.1",
    "marker": "32677260-5393-40e0-b228-a1ebd6184bc3.51110.1",
    "index_type": "Normal",
    "owner": "johndoe",
    "ver": "0#4762,1#4608,2#4617,3#4699,4#4757,5#4648,6#4630,7#4702,8#4646,9#4592,10#4629,11#4708,12#4685,13#4770,14#4604,15#4575,16#4572,17#4773,18#4760,19#4728,20#4635,21#4661,22#4620,23#4669,24#4646,25#4694,26#4580,27#4678,28#4657,29#4670,30#4683,31#4559,32#4550,33#4638,34#4661,35#4768,36#4790,37#4653,38#4646,39#4644,40#4640,41#4662,42#4544,43#4627,44#4738,45#4716,46#4798,47#4649,48#4616,49#4621,50#4646,51#4588,52#4705,53#4695,54#4709,55#4655,56#4695,57#4665,58#4760,59#4694,60#4456,61#4682,62#4637,63#4674,64#4541,65#4638,66#4657,67#4614,68#4619,69#4634,70#4834,71#4739,72#4744,73#4652,74#4880,75#4824,76#4729,77#4704,78#4591,79#4697,80#4724,81#4493,82#4672,83#4716,84#4696,85#4596,86#4744,87#4640,88#4617,89#4678,90#4695,91#4633,92#4724,93#4650,94#4574,95#4703,96#4700,97#4657,98#4675,99#4675,100#4644,101#4809,102#4641,103#4622,104#4769,105#4775,106#4576,107#4672,108#4629,109#4719,110#4626,111#4586,112#4741,113#4716,114#4706,115#4600,116#4725,117#4705,118#4703,119#4733,120#4750,121#4577,122#4683,123#4608,124#4542,125#4715,126#4649,127#4632,128#4722,129#4831,130#4777,131#4542,132#4552,133#4631,134#4739,135#4731,136#4638,137#4714,138#4601,139#4712,140#4667,141#4592,142#4750,143#4705,144#4713,145#4698,146#4581,147#4777,148#4682,149#4622,150#4637,151#4613,152#4654,153#4641,154#4663,155#4680,156#4823,157#4846,158#4701,159#4592,160#4710,161#4635,162#4574,163#4664,164#4679,165#4677,166#4620,167#4688,168#4701,169#4644,170#4531,171#4653,172#4824,173#4631,174#4576,175#4678,176#4526,177#4758,178#4579,179#4683,180#4666,181#4666,182#4631,183#4735,184#4627,185#4616,186#4743,187#4556,188#4721,189#4709,190#4647,191#4766,192#4647,193#4721,194#4552,195#4675,196#4691,197#4739,198#4806,199#4561,200#4619,201#4750,202#4662,203#4530,204#4606,205#4756,206#4647,207#4667,208#4579,209#4632,210#4641,211#4729,212#4661,213#4666,214#4678,215#4588,216#4618,217#4660,218#4699,219#4677,220#4731,221#4637,222#4641,223#4852,224#4722,225#4733,226#4601,227#4702,228#4791,229#4674,230#4628,231#4776,232#4746,233#4688,234#4702,235#4660,236#4689,237#4708,238#4591,239#4669,240#4738,241#4641,242#4686,243#4678,244#4574,245#4654,246#4711,247#4609,248#4618,249#4697,250#4588,251#4665,252#4605,253#4689,254#4558,255#4694,256#4577,257#4682,258#4607,259#4698,260#4688,261#4840,262#4614,263#4639,264#4782,265#4722,266#4618,267#4663,268#4640,269#4579,270#4732,271#4666,272#4734,273#4778,274#4715,275#4589,276#4704,277#4664,278#4776,279#4609,280#4663,281#4762,282#4623,283#4668,284#4659,285#4533,286#4681,287#4581,288#4655,289#4610,290#4706,291#4730,292#4579,293#4535,294#4727,295#4697,296#4557,297#4747,298#4837,299#4788,300#4517,301#4650,302#4594,303#4724,304#4708,305#4675,306#4712,307#4528,308#4697,309#4610,310#4817,311#4667,312#4650,313#4553,314#4628,315#4719,316#4679,317#4542,318#4699,319#4725,320#4671,321#4715,322#4738,323#4795,324#4708,325#4645,326#4668,327#4728,328#4675,329#4720,330#4571,331#4641,332#4593,333#4522,334#4690,335#4640,336#4683,337#4653,338#4706,339#4737,340#4683,341#4686,342#4654,343#4685,344#4608,345#4663,346#4686,347#4765,348#4615,349#4653,350#4684,351#4544,352#4625,353#4634,354#4536,355#4726,356#4659,357#4691,358#4645,359#4684,360#4752,361#4729,362#4624,363#4677,364#4577,365#4707,366#4666,367#4626,368#4661,369#4636,370#4746,371#4689,372#4763,373#4636,374#4608,375#4632,376#4715,377#4577,378#4670,379#4685,380#4562,381#4667,382#4744,383#4631,384#4712,385#4643,386#4631,387#4669,388#4589,389#4683,390#4597,391#4793,392#4610,393#4737,394#4717,395#4726,396#4681,397#4505,398#4591,399#4889,400#4643,401#4704,402#4719,403#4876,404#4801,405#4703,406#4653,407#4675,408#4703,409#4595,410#4607,411#4668,412#4515,413#4703,414#4679,415#4722,416#4671,417#4799,418#4700,419#4678,420#4605,421#4714,422#4669,423#4809,424#4671,425#4654,426#4603,427#4530,428#4685,429#4577,430#4730,431#4600,432#4735,433#4644,434#4675,435#4607,436#4710,437#4721,438#4620,439#4698,440#4623,441#4576,442#4698,443#4690,444#4571,445#4747,446#4654,447#4608,448#4657,449#4582,450#4687,451#4591,452#4536,453#4681,454#4752,455#4687,456#4756,457#4574,458#4644,459#4734,460#4638,461#4709,462#4757,463#4700,464#4695,465#4832,466#4546,467#4672,468#4570,469#4682,470#4606,471#4646,472#4682,473#4684,474#4728,475#4716,476#4651,477#4549,478#4721,479#4675,480#4638,481#4571,482#4577,483#4713,484#4755,485#4628,486#4672,487#4618,488#4703,489#4743,490#4599,491#4636,492#4650,493#4775,494#4651,495#4686,496#4591,497#4653,498#4657,499#4710,500#4653,501#4768,502#4701,503#4731,504#4740,505#4712,506#4615,507#4695,508#4516,509#4693,510#4565,511#4665,512#4577,513#4621,514#4701,515#4546,516#4727,517#4573,518#4645,519#4757,520#4630,521#4564,522#4701,523#4776,524#4616,525#4652,526#4640,527#4595,528#4736,529#4651,530#4510,531#4735,532#4651,533#4542,534#4664,535#4717,536#4568,537#4733,538#4630,539#4645,540#4658,541#4554,542#4831,543#4751,544#4648,545#4725,546#4699,547#4674,548#4773,549#4602,550#4695,551#4685,552#4674,553#4629,554#4687,555#4746,556#4709,557#4670,558#4743,559#4651,560#4624,561#4759,562#4825,563#4727,564#4680,565#4737,566#4773,567#4575,568#4745,569#4660,570#4558,571#4760,572#4617,573#4802,574#4767,575#4669,576#4759,577#4620,578#4687,579#4627,580#4623,581#4668,582#4652,583#4658,584#4839,585#4780,586#4715,587#4732,588#4572,589#4646,590#4637,591#4500,592#4656,593#4593,594#4668,595#4649,596#4797,597#4708,598#4640,599#4528,600#4711,601#4603,602#4701,603#4633,604#4659,605#4688,606#4680,607#4714,608#4613,609#4631,610#4649,611#4829,612#4756,613#4607,614#4718,615#4753,616#4687,617#4697,618#4695,619#4662,620#4723,621#4721,622#4595,623#4703,624#4593,625#4675,626#4790,627#4645,628#4637,629#4717,630#4710,631#4706,632#4600,633#4654,634#4649,635#4600,636#4660,637#4706,638#4504,639#4752,640#4745,641#4603,642#4580,643#4575,644#4774,645#4724,646#4588,647#4669,648#4658,649#4660,650#4690,651#4651,652#4525,653#4673,654#4606,655#4706,656#4629,657#4614,658#4631,659#4622,660#4638,661#4674,662#4587,663#4669,664#4607,665#4787,666#4710,667#4619,668#4617,669#4685,670#4705,671#4652,672#4626,673#4724,674#4792,675#4605,676#4688,677#4594,678#4678,679#4683,680#4671,681#4654,682#4577,683#4701,684#4603,685#4672,686#4537,687#4677,688#4697,689#4591,690#4586,691#4782,692#4668,693#4701,694#4615,695#4631,696#4568,697#4636,698#4672,699#4614,700#4580,701#4629,702#4670,703#4782,704#4602,705#4712,706#4668,707#4730,708#4648,709#4757,710#4583,711#4709,712#4606,713#4663,714#4605,715#4714,716#4761,717#4643,718#4707,719#4756,720#4789,721#4641,722#4724,723#4632,724#4629,725#4618,726#4765,727#4631,728#4638,729#4597,730#4604,731#4509,732#4638,733#4640,734#4661,735#4603,736#4715,737#4637,738#4628,739#4563,740#4460,741#4699,742#4681,743#4659,744#4592,745#4669,746#4701,747#4748,748#4723,749#4597,750#4618,751#4709,752#4579,753#4736,754#4654,755#4681,756#4639,757#4643,758#4667,759#4684,760#4651,761#4679,762#4625,763#4718,764#4672,765#4772,766#4683,767#4614,768#4733,769#4728,770#4633,771#4689,772#4609,773#4664,774#4787,775#4715,776#4702,777#4671,778#4657,779#4502,780#4663,781#4652,782#4773,783#4740,784#4724,785#4566,786#4689,787#4712,788#4693,789#4824,790#4588,791#4711,792#4716,793#4640,794#4617,795#4706,796#4656,797#4684,798#4749,799#4617,800#4603,801#4720,802#4681,803#4657,804#4683,805#4587,806#4564,807#4681,808#4694,809#4681,810#4791,811#4624,812#4565,813#4669,814#4743,815#4653,816#4704,817#4680,818#4614,819#4705,820#4617,821#4705,822#4588,823#4720,824#4615,825#4572,826#4671,827#4743,828#4763,829#4602,830#4606,831#4722,832#4679,833#4642,834#4780,835#4545,836#4624,837#4804,838#4544,839#4608,840#4702,841#4738,842#4686,843#4677,844#4717,845#4555,846#4737,847#4755,848#4598,849#4571,850#4640,851#4741,852#4679,853#4626,854#4608,855#4557,856#4586,857#4596,858#4775,859#4686,860#4522,861#4639,862#4665,863#4748,864#4620,865#4767,866#4746,867#4724,868#4772,869#4648,870#4724,871#4681,872#4702,873#4655,874#4723,875#4673,876#4524,877#4665,878#4722,879#4622,880#4633,881#4665,882#4706,883#4812,884#4721,885#4629,886#4645,887#4724,888#4585,889#4720,890#4648,891#4681,892#4755,893#4735,894#4628,895#4634,896#4654,897#4702,898#4590,899#4753,900#4576,901#4735,902#4661,903#4637,904#4656,905#4660,906#4626,907#4607,908#4750,909#4699,910#4802,911#4676,912#4723,913#4507,914#4702,915#4735,916#4529,917#4661,918#4594,919#4614,920#4643,921#4626,922#4648,923#4697,924#4640,925#4610,926#4589,927#4662,928#4698,929#4642,930#4642,931#4753,932#4584,933#4657,934#4642,935#4645,936#4685,937#4563,938#4718,939#4684,940#4634,941#4669,942#4676,943#4682,944#4663,945#4665,946#4727,947#4647,948#4648,949#4681,950#4730,951#4746,952#4540,953#4748,954#4699,955#4678,956#4708,957#4659,958#4666,959#4631,960#4639,961#4728,962#4620,963#4672,964#4751,965#4737,966#4694,967#4644,968#4711,969#4663,970#4665,971#4681,972#3958,973#3835,974#3862,975#3862,976#3851,977#3870,978#3986,979#3931,980#3837,981#3982,982#3936,983#3884,984#3849,985#3955,986#3894,987#3919,988#3810,989#3958,990#3802,991#3938,992#3786,993#3905,994#3856,995#3859,996#3954,997#3794,998#3928,999#3960,1000#3960,1001#3878,1002#3892,1003#3842,1004#3855,1005#3874,1006#3780,1007#4011,1008#3877,1009#3877,1010#3936,1011#3864,1012#3907,1013#3889,1014#3910,1015#3934,1016#3911,1017#3955,1018#3813,1019#3767,1020#3845,1021#3790,1022#3779,1023#3819,1024#3912,1025#3788,1026#3876,1027#3865,1028#3864,1029#3863,1030#3911,1031#3913,1032#3977,1033#3915,1034#3852,1035#3971,1036#3892,1037#4009,1038#3938,1039#3938,1040#3981,1041#3936,1042#3915,1043#3895,1044#3788,1045#3910,1046#3908,1047#3976,1048#3825,1049#4000,1050#3859,1051#3986,1052#3865,1053#3737,1054#3918,1055#3877,1056#3903,1057#3908,1058#3892,1059#3870,1060#3793,1061#3949,1062#3918,1063#3839,1064#3824,1065#3818,1066#3888,1067#3916,1068#3847,1069#4011,1070#3897,1071#3949,1072#3844,1073#3942,1074#3925,1075#3900,1076#3910,1077#3801,1078#3738,1079#3889,1080#4046,1081#3833,1082#3909,1083#3885,1084#3932,1085#3789,1086#3779,1087#3835,1088#3905,1089#3937,1090#3904,1091#3924,1092#3892,1093#3901,1094#3965,1095#3962,1096#3889,1097#3933,1098#3968,1099#4001,1100#3836,1101#3836,1102#3906,1103#3785,1104#3916,1105#3989,1106#3959,1107#3920,1108#3882,1109#3860,1110#3906,1111#3922,1112#3992,1113#3795,1114#3812,1115#3873,1116#3989,1117#3883,1118#3831,1119#3903,1120#3908,1121#3821,1122#3879,1123#3938,1124#3821,1125#3961,1126#3879,1127#3861,1128#3875,1129#3903,1130#3919,1131#3847,1132#3935,1133#3955,1134#3935,1135#3895,1136#3862,1137#3848,1138#3818,1139#3926,1140#3905,1141#3930,1142#3949,1143#3840,1144#3992,1145#3923,1146#3956,1147#3905,1148#3869,1149#3958,1150#3883,1151#3959,1152#3860,1153#3843,1154#4007,1155#3864,1156#3849,1157#3858,1158#3840,1159#3897,1160#3854,1161#3919,1162#3901,1163#3849,1164#3827,1165#3945,1166#3954,1167#3836,1168#3914,1169#3919,1170#3867,1171#3897,1172#3925,1173#3849,1174#3811,1175#3880,1176#3963,1177#3943,1178#3917,1179#3950,1180#3950,1181#3780,1182#3940,1183#3874,1184#3747,1185#3906,1186#3979,1187#3903,1188#3920,1189#3848,1190#3895,1191#3860,1192#4028,1193#3874,1194#3805,1195#4015,1196#3968,1197#3915,1198#3861,1199#3876,1200#3829,1201#3913,1202#3970,1203#3896,1204#3964,1205#3932,1206#3919,1207#3891,1208#3849,1209#3914,1210#3779,1211#3904,1212#3821,1213#3927,1214#4023,1215#3804,1216#3851,1217#4029,1218#3953,1219#3892,1220#3879,1221#3860,1222#3825,1223#3857,1224#3875,1225#3914,1226#3879,1227#3784,1228#3849,1229#3817,1230#3885,1231#3870,1232#3865,1233#3971,1234#3749,1235#3935,1236#3865,1237#3850,1238#3999,1239#3981,1240#3848,1241#3890,1242#3802,1243#3980,1244#3817,1245#3851,1246#3918,1247#3890,1248#3851,1249#3846,1250#3806,1251#3881,1252#3913,1253#3851,1254#3894,1255#3907,1256#3832,1257#3908,1258#3858,1259#3896,1260#3958,1261#3754,1262#3770,1263#3962,1264#3866,1265#3784,1266#3864,1267#3858,1268#3870,1269#3856,1270#3867,1271#3816,1272#3872,1273#3956,1274#3853,1275#3884,1276#3752,1277#3958,1278#3920,1279#3876,1280#3926,1281#3885,1282#3952,1283#3861,1284#3938,1285#3892,1286#3929,1287#3785,1288#3806,1289#3904,1290#3868,1291#3830,1292#3923,1293#3853,1294#3926,1295#3946,1296#3909,1297#3955,1298#3888,1299#3963,1300#3859,1301#3899,1302#4051,1303#3872,1304#3906,1305#3858,1306#3875,1307#3929,1308#3922,1309#4019,1310#3940,1311#3867,1312#3984,1313#3931,1314#3953,1315#3870,1316#3894,1317#3910,1318#3771,1319#3861,1320#3961,1321#3843,1322#3928,1323#3940,1324#3902,1325#3901,1326#3924,1327#3924,1328#3954,1329#3980,1330#3994,1331#3879,1332#3913,1333#3915,1334#3788,1335#3984,1336#3825,1337#3898,1338#3863,1339#3893,1340#3844,1341#3734,1342#3889,1343#3845,1344#3953,1345#3886,1346#3788,1347#3868,1348#3857,1349#3957,1350#3933,1351#3852,1352#3951,1353#3802,1354#3851,1355#3852,1356#3873,1357#3962,1358#3948,1359#3872,1360#3916,1361#3856,1362#3882,1363#3863,1364#3867,1365#3920,1366#3898,1367#3759,1368#4002,1369#3920,1370#3925,1371#3821,1372#3861,1373#3902,1374#3901,1375#3866,1376#3903,1377#3927,1378#3859,1379#3897,1380#3912",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0,967#0,968#0,969#0,970#0,971#0,972#0,973#0,974#0,975#0,976#0,977#0,978#0,979#0,980#0,981#0,982#0,983#0,984#0,985#0,986#0,987#0,988#0,989#0,990#0,991#0,992#0,993#0,994#0,995#0,996#0,997#0,998#0,999#0,1000#0,1001#0,1002#0,1003#0,1004#0,1005#0,1006#0,1007#0,1008#0,1009#0,1010#0,1011#0,1012#0,1013#0,1014#0,1015#0,1016#0,1017#0,1018#0,1019#0,1020#0,1021#0,1022#0,1023#0,1024#0,1025#0,1026#0,1027#0,1028#0,1029#0,1030#0,1031#0,1032#0,1033#0,1034#0,1035#0,1036#0,1037#0,1038#0,1039#0,1040#0,1041#0,1042#0,1043#0,1044#0,1045#0,1046#0,1047#0,1048#0,1049#0,1050#0,1051#0,1052#0,1053#0,1054#0,1055#0,1056#0,1057#0,1058#0,1059#0,1060#0,1061#0,1062#0,1063#0,1064#0,1065#0,1066#0,1067#0,1068#0,1069#0,1070#0,1071#0,1072#0,1073#0,1074#0,1075#0,1076#0,1077#0,1078#0,1079#0,1080#0,1081#0,1082#0,1083#0,1084#0,1085#0,1086#0,1087#0,1088#0,1089#0,1090#0,1091#0,1092#0,1093#0,1094#0,1095#0,1096#0,1097#0,1098#0,1099#0,1100#0,1101#0,1102#0,1103#0,1104#0,1105#0,1106#0,1107#0,1108#0,1109#0,1110#0,1111#0,1112#0,1113#0,1114#0,1115#0,1116#0,1117#0,1118#0,1119#0,1120#0,1121#0,1122#0,1123#0,1124#0,1125#0,1126#0,1127#0,1128#0,1129#0,1130#0,1131#0,1132#0,1133#0,1134#0,1135#0,1136#0,1137#0,1138#0,1139#0,1140#0,1141#0,1142#0,1143#0,1144#0,1145#0,1146#0,1147#0,1148#0,1149#0,1150#0,1151#0,1152#0,1153#0,1154#0,1155#0,1156#0,1157#0,1158#0,1159#0,1160#0,1161#0,1162#0,1163#0,1164#0,1165#0,1166#0,1167#0,1168#0,1169#0,1170#0,1171#0,1172#0,1173#0,1174#0,1175#0,1176#0,1177#0,1178#0,1179#0,1180#0,1181#0,1182#0,1183#0,1184#0,1185#0,1186#0,1187#0,1188#0,1189#0,1190#0,1191#0,1192#0,1193#0,1194#0,1195#0,1196#0,1197#0,1198#0,1199#0,1200#0,1201#0,1202#0,1203#0,1204#0,1205#0,1206#0,1207#0,1208#0,1209#0,1210#0,1211#0,1212#0,1213#0,1214#0,1215#0,1216#0,1217#0,1218#0,1219#0,1220#0,1221#0,1222#0,1223#0,1224#0,1225#0,1226#0,1227#0,1228#0,1229#0,1230#0,1231#0,1232#0,1233#0,1234#0,1235#0,1236#0,1237#0,1238#0,1239#0,1240#0,1241#0,1242#0,1243#0,1244#0,1245#0,1246#0,1247#0,1248#0,1249#0,1250#0,1251#0,1252#0,1253#0,1254#0,1255#0,1256#0,1257#0,1258#0,1259#0,1260#0,1261#0,1262#0,1263#0,1264#0,1265#0,1266#0,1267#0,1268#0,1269#0,1270#0,1271#0,1272#0,1273#0,1274#0,1275#0,1276#0,1277#0,1278#0,1279#0,1280#0,1281#0,1282#0,1283#0,1284#0,1285#0,1286#0,1287#0,1288#0,1289#0,1290#0,1291#0,1292#0,1293#0,1294#0,1295#0,1296#0,1297#0,1298#0,1299#0,1300#0,1301#0,1302#0,1303#0,1304#0,1305#0,1306#0,1307#0,1308#0,1309#0,1310#0,1311#0,1312#0,1313#0,1314#0,1315#0,1316#0,1317#0,1318#0,1319#0,1320#0,1321#0,1322#0,1323#0,1324#0,1325#0,1326#0,1327#0,1328#0,1329#0,1330#0,1331#0,1332#0,1333#0,1334#0,1335#0,1336#0,1337#0,1338#0,1339#0,1340#0,1341#0,1342#0,1343#0,1344#0,1345#0,1346#0,1347#0,1348#0,1349#0,1350#0,1351#0,1352#0,1353#0,1354#0,1355#0,1356#0,1357#0,1358#0,1359#0,1360#0,1361#0,1362#0,1363#0,1364#0,1365#0,1366#0,1367#0,1368#0,1369#0,1370#0,1371#0,1372#0,1373#0,1374#0,1375#0,1376#0,1377#0,1378#0,1379#0,1380#0",
    "mtime": "0.000000",
    "creation_time": "2022-03-21T07:39:03.073314Z",
    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#,967#,968#,969#,970#,971#,972#,973#,974#,975#,976#,977#,978#,979#,980#,981#,982#,983#,984#,985#,986#,987#,988#,989#,990#,991#,992#,993#,994#,995#,996#,997#,998#,999#,1000#,1001#,1002#,1003#,1004#,1005#,1006#,1007#,1008#,1009#,1010#,1011#,1012#,1013#,1014#,1015#,1016#,1017#,1018#,1019#,1020#,1021#,1022#,1023#,1024#,1025#,1026#,1027#,1028#,1029#,1030#,1031#,1032#,1033#,1034#,1035#,1036#,1037#,1038#,1039#,1040#,1041#,1042#,1043#,1044#,1045#,1046#,1047#,1048#,1049#,1050#,1051#,1052#,1053#,1054#,1055#,1056#,1057#,1058#,1059#,1060#,1061#,1062#,1063#,1064#,1065#,1066#,1067#,1068#,1069#,1070#,1071#,1072#,1073#,1074#,1075#,1076#,1077#,1078#,1079#,1080#,1081#,1082#,1083#,1084#,1085#,1086#,1087#,1088#,1089#,1090#,1091#,1092#,1093#,1094#,1095#,1096#,1097#,1098#,1099#,1100#,1101#,1102#,1103#,1104#,1105#,1106#,1107#,1108#,1109#,1110#,1111#,1112#,1113#,1114#,1115#,1116#,1117#,1118#,1119#,1120#,1121#,1122#,1123#,1124#,1125#,1126#,1127#,1128#,1129#,1130#,1131#,1132#,1133#,1134#,1135#,1136#,1137#,1138#,1139#,1140#,1141#,1142#,1143#,1144#,1145#,1146#,1147#,1148#,1149#,1150#,1151#,1152#,1153#,1154#,1155#,1156#,1157#,1158#,1159#,1160#,1161#,1162#,1163#,1164#,1165#,1166#,1167#,1168#,1169#,1170#,1171#,1172#,1173#,1174#,1175#,1176#,1177#,1178#,1179#,1180#,1181#,1182#,1183#,1184#,1185#,1186#,1187#,1188#,1189#,1190#,1191#,1192#,1193#,1194#,1195#,1196#,1197#,1198#,1199#,1200#,1201#,1202#,1203#,1204#,1205#,1206#,1207#,1208#,1209#,1210#,1211#,1212#,1213#,1214#,1215#,1216#,1217#,1218#,1219#,1220#,1221#,1222#,1223#,1224#,1225#,1226#,1227#,1228#,1229#,1230#,1231#,1232#,1233#,1234#,1235#,1236#,1237#,1238#,1239#,1240#,1241#,1242#,1243#,1244#,1245#,1246#,1247#,1248#,1249#,1250#,1251#,1252#,1253#,1254#,1255#,1256#,1257#,1258#,1259#,1260#,1261#,1262#,1263#,1264#,1265#,1266#,1267#,1268#,1269#,1270#,1271#,1272#,1273#,1274#,1275#,1276#,1277#,1278#,1279#,1280#,1281#,1282#,1283#,1284#,1285#,1286#,1287#,1288#,1289#,1290#,1291#,1292#,1293#,1294#,1295#,1296#,1297#,1298#,1299#,1300#,1301#,1302#,1303#,1304#,1305#,1306#,1307#,1308#,1309#,1310#,1311#,1312#,1313#,1314#,1315#,1316#,1317#,1318#,1319#,1320#,1321#,1322#,1323#,1324#,1325#,1326#,1327#,1328#,1329#,1330#,1331#,1332#,1333#,1334#,1335#,1336#,1337#,1338#,1339#,1340#,1341#,1342#,1343#,1344#,1345#,1346#,1347#,1348#,1349#,1350#,1351#,1352#,1353#,1354#,1355#,1356#,1357#,1358#,1359#,1360#,1361#,1362#,1363#,1364#,1365#,1366#,1367#,1368#,1369#,1370#,1371#,1372#,1373#,1374#,1375#,1376#,1377#,1378#,1379#,1380#",
    "usage": {
        "rgw.main": {
            "size": 77351363000,
            "size_actual": 85883617280,
            "size_utilized": 77351363000,
            "size_kb": 75538441,
            "size_kb_actual": 83870720,
            "size_kb_utilized": 75538441,
            "num_objects": 5196323
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

root@f13-h26-b04-5039ms:~
# radosgw-admin bucket stats --bucket=mycontainers3
error getting bucket stats bucket=mycontainers3 ret=-2
failure: (2) No such file or directory: 

mycontainers3_debug_stats.ENOENT.txt View (756 KB) Vikhyat Umrao, 03/21/2022 07:10 PM

History

#1 Updated by Vikhyat Umrao about 2 years ago

Pacific version - 16.2.7-89.el8cp

#2 Updated by Vikhyat Umrao about 2 years ago

Debug logs:

radosgw-admin bucket stats --bucket mycontainers3 --debug-rgw 20 --debug-ms 1  &> /root/mycontainers3_debug_stats.ENOENT.txt

#3 Updated by Vikhyat Umrao about 2 years ago

This looks like some kind of backward compatibility when the cluster was getting upgraded. Just like we have a pacific `radosgw-admin` backward compatibility issue with the quincy cluster reported here - https://tracker.ceph.com/issues/53956.

#4 Updated by Vikhyat Umrao about 2 years ago

We had a similar issue in the downstream version when RHCS 5 to RHCS 5.1 was reported here1 but looks it got fixed with the latest version - 16.2.7-89.el8cp. This looks to be some kind of bucket inconsistency during reshard.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=2062855

#5 Updated by Vikhyat Umrao about 2 years ago

- Reshard did help to bring the bucket stats command to work but usage is gone.

# radosgw-admin bucket reshard --bucket mycontainers3 --num-shards 1499
tenant: 
bucket name: mycontainers3
old bucket instance id: 32677260-5393-40e0-b228-a1ebd6184bc3.26154.1
new bucket instance id: 32677260-5393-40e0-b228-a1ebd6184bc3.238224.1

972000 74973000 74974000 74975000 74976000 74977000 74978000 74979000 74980000 74981000 74982000 74983000 74984000 74985000 74986000 74987000 74988000 74989000 74990000 74991000 74992000 74993000 74994000 74995000 74996000 74997000 74998000 74999000 74999902
2022-03-21T19:53:34.451+0000 7f0c563b6500  1 execute INFO: reshard of bucket "mycontainers3" from "mycontainers3:32677260-5393-40e0-b228-a1ebd6184bc3.26154.1" to "mycontainers3:32677260-5393-40e0-b228-a1ebd6184bc3.238224.1" completed successfully
root@f13-h26-b04-5039ms:~
# radosgw-admin bucket stats --bucket mycontainers3
{
    "bucket": "mycontainers3",
    "num_shards": 1499,
    "tenant": "",
    "zonegroup": "2b98c1de-086d-4f6f-a042-3110874d69ff",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": "" 
    },
    "id": "32677260-5393-40e0-b228-a1ebd6184bc3.238224.1",
    "marker": "32677260-5393-40e0-b228-a1ebd6184bc3.26154.1",
    "index_type": "Normal",
    "owner": "johndoe",
    "ver": "0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1,11#1,12#1,13#1,14#1,15#1,16#1,17#1,18#1,19#1,20#1,21#1,22#1,23#1,24#1,25#1,26#1,27#1,28#1,29#1,30#1,31#1,32#1,33#1,34#1,35#1,36#1,37#1,38#1,39#1,40#1,41#1,42#1,43#1,44#1,45#1,46#1,47#1,48#1,49#1,50#1,51#1,52#1,53#1,54#1,55#1,56#1,57#1,58#1,59#1,60#1,61#1,62#1,63#1,64#1,65#1,66#1,67#1,68#1,69#1,70#1,71#1,72#1,73#1,74#1,75#1,76#1,77#1,78#1,79#1,80#1,81#1,82#1,83#1,84#1,85#1,86#1,87#1,88#1,89#1,90#1,91#1,92#1,93#1,94#1,95#1,96#1,97#1,98#1,99#1,100#1,101#1,102#1,103#1,104#1,105#1,106#1,107#1,108#1,109#1,110#1,111#1,112#1,113#1,114#1,115#1,116#1,117#1,118#1,119#1,120#1,121#1,122#1,123#1,124#1,125#1,126#1,127#1,128#1,129#1,130#1,131#1,132#1,133#1,134#1,135#1,136#1,137#1,138#1,139#1,140#1,141#1,142#1,143#1,144#1,145#1,146#1,147#1,148#1,149#1,150#1,151#1,152#1,153#1,154#1,155#1,156#1,157#1,158#1,159#1,160#1,161#1,162#1,163#1,164#1,165#1,166#1,167#1,168#1,169#1,170#1,171#1,172#1,173#1,174#1,175#1,176#1,177#1,178#1,179#1,180#1,181#1,182#1,183#1,184#1,185#1,186#1,187#1,188#1,189#1,190#1,191#1,192#1,193#1,194#1,195#1,196#1,197#1,198#1,199#1,200#1,201#1,202#1,203#1,204#1,205#1,206#1,207#1,208#1,209#1,210#1,211#1,212#1,213#1,214#1,215#1,216#1,217#1,218#1,219#1,220#1,221#1,222#1,223#1,224#1,225#1,226#1,227#1,228#1,229#1,230#1,231#1,232#1,233#1,234#1,235#1,236#1,237#1,238#1,239#1,240#1,241#1,242#1,243#1,244#1,245#1,246#1,247#1,248#1,249#1,250#1,251#1,252#1,253#1,254#1,255#1,256#1,257#1,258#1,259#1,260#1,261#1,262#1,263#1,264#1,265#1,266#1,267#1,268#1,269#1,270#1,271#1,272#1,273#1,274#1,275#1,276#1,277#1,278#1,279#1,280#1,281#1,282#1,283#1,284#1,285#1,286#1,287#1,288#1,289#1,290#1,291#1,292#1,293#1,294#1,295#1,296#1,297#1,298#1,299#1,300#1,301#1,302#1,303#1,304#1,305#1,306#1,307#1,308#1,309#1,310#1,311#1,312#1,313#1,314#1,315#1,316#1,317#1,318#1,319#1,320#1,321#1,322#1,323#1,324#1,325#1,326#1,327#1,328#1,329#1,330#1,331#1,332#1,333#1,334#1,335#1,336#1,337#1,338#1,339#1,340#1,341#1,342#1,343#1,344#1,345#1,346#1,347#1,348#1,349#1,350#1,351#1,352#1,353#1,354#1,355#1,356#1,357#1,358#1,359#1,360#1,361#1,362#1,363#1,364#1,365#1,366#1,367#1,368#1,369#1,370#1,371#1,372#1,373#1,374#1,375#1,376#1,377#1,378#1,379#1,380#1,381#1,382#1,383#1,384#1,385#1,386#1,387#1,388#1,389#1,390#1,391#1,392#1,393#1,394#1,395#1,396#1,397#1,398#1,399#1,400#1,401#1,402#1,403#1,404#1,405#1,406#1,407#1,408#1,409#1,410#1,411#1,412#1,413#1,414#1,415#1,416#1,417#1,418#1,419#1,420#1,421#1,422#1,423#1,424#1,425#1,426#1,427#1,428#1,429#1,430#1,431#1,432#1,433#1,434#1,435#1,436#1,437#1,438#1,439#1,440#1,441#1,442#1,443#1,444#1,445#1,446#1,447#1,448#1,449#1,450#1,451#1,452#1,453#1,454#1,455#1,456#1,457#1,458#1,459#1,460#1,461#1,462#1,463#1,464#1,465#1,466#1,467#1,468#1,469#1,470#1,471#1,472#1,473#1,474#1,475#1,476#1,477#1,478#1,479#1,480#1,481#1,482#1,483#1,484#1,485#1,486#1,487#1,488#1,489#1,490#1,491#1,492#1,493#1,494#1,495#1,496#1,497#1,498#1,499#1,500#1,501#1,502#1,503#1,504#1,505#1,506#1,507#1,508#1,509#1,510#1,511#1,512#1,513#1,514#1,515#1,516#1,517#1,518#1,519#1,520#1,521#1,522#1,523#1,524#1,525#1,526#1,527#1,528#1,529#1,530#1,531#1,532#1,533#1,534#1,535#1,536#1,537#1,538#1,539#1,540#1,541#1,542#1,543#1,544#1,545#1,546#1,547#1,548#1,549#1,550#1,551#1,552#1,553#1,554#1,555#1,556#1,557#1,558#1,559#1,560#1,561#1,562#1,563#1,564#1,565#1,566#1,567#1,568#1,569#1,570#1,571#1,572#1,573#1,574#1,575#1,576#1,577#1,578#1,579#1,580#1,581#1,582#1,583#1,584#1,585#1,586#1,587#1,588#1,589#1,590#1,591#1,592#1,593#1,594#1,595#1,596#1,597#1,598#1,599#1,600#1,601#1,602#1,603#1,604#1,605#1,606#1,607#1,608#1,609#1,610#1,611#1,612#1,613#1,614#1,615#1,616#1,617#1,618#1,619#1,620#1,621#1,622#1,623#1,624#1,625#1,626#1,627#1,628#1,629#1,630#1,631#1,632#1,633#1,634#1,635#1,636#1,637#1,638#1,639#1,640#1,641#1,642#1,643#1,644#1,645#1,646#1,647#1,648#1,649#1,650#1,651#1,652#1,653#1,654#1,655#1,656#1,657#1,658#1,659#1,660#1,661#1,662#1,663#1,664#1,665#1,666#1,667#1,668#1,669#1,670#1,671#1,672#1,673#1,674#1,675#1,676#1,677#1,678#1,679#1,680#1,681#1,682#1,683#1,684#1,685#1,686#1,687#1,688#1,689#1,690#1,691#1,692#1,693#1,694#1,695#1,696#1,697#1,698#1,699#1,700#1,701#1,702#1,703#1,704#1,705#1,706#1,707#1,708#1,709#1,710#1,711#1,712#1,713#1,714#1,715#1,716#1,717#1,718#1,719#1,720#1,721#1,722#1,723#1,724#1,725#1,726#1,727#1,728#1,729#1,730#1,731#1,732#1,733#1,734#1,735#1,736#1,737#1,738#1,739#1,740#1,741#1,742#1,743#1,744#1,745#1,746#1,747#1,748#1,749#1,750#1,751#1,752#1,753#1,754#1,755#1,756#1,757#1,758#1,759#1,760#1,761#1,762#1,763#1,764#1,765#1,766#1,767#1,768#1,769#1,770#1,771#1,772#1,773#1,774#1,775#1,776#1,777#1,778#1,779#1,780#1,781#1,782#1,783#1,784#1,785#1,786#1,787#1,788#1,789#1,790#1,791#1,792#1,793#1,794#1,795#1,796#1,797#1,798#1,799#1,800#1,801#1,802#1,803#1,804#1,805#1,806#1,807#1,808#1,809#1,810#1,811#1,812#1,813#1,814#1,815#1,816#1,817#1,818#1,819#1,820#1,821#1,822#1,823#1,824#1,825#1,826#1,827#1,828#1,829#1,830#1,831#1,832#1,833#1,834#1,835#1,836#1,837#1,838#1,839#1,840#1,841#1,842#1,843#1,844#1,845#1,846#1,847#1,848#1,849#1,850#1,851#1,852#1,853#1,854#1,855#1,856#1,857#1,858#1,859#1,860#1,861#1,862#1,863#1,864#1,865#1,866#1,867#1,868#1,869#1,870#1,871#1,872#1,873#1,874#1,875#1,876#1,877#1,878#1,879#1,880#1,881#1,882#1,883#1,884#1,885#1,886#1,887#1,888#1,889#1,890#1,891#1,892#1,893#1,894#1,895#1,896#1,897#1,898#1,899#1,900#1,901#1,902#1,903#1,904#1,905#1,906#1,907#1,908#1,909#1,910#1,911#1,912#1,913#1,914#1,915#1,916#1,917#1,918#1,919#1,920#1,921#1,922#1,923#1,924#1,925#1,926#1,927#1,928#1,929#1,930#1,931#1,932#1,933#1,934#1,935#1,936#1,937#1,938#1,939#1,940#1,941#1,942#1,943#1,944#1,945#1,946#1,947#1,948#1,949#1,950#1,951#1,952#1,953#1,954#1,955#1,956#1,957#1,958#1,959#1,960#1,961#1,962#1,963#1,964#1,965#1,966#1,967#1,968#1,969#1,970#1,971#1,972#1,973#1,974#1,975#1,976#1,977#1,978#1,979#1,980#1,981#1,982#1,983#1,984#1,985#1,986#1,987#1,988#1,989#1,990#1,991#1,992#1,993#1,994#1,995#1,996#1,997#1,998#1,999#1,1000#1,1001#1,1002#1,1003#1,1004#1,1005#1,1006#1,1007#1,1008#1,1009#1,1010#1,1011#1,1012#1,1013#1,1014#1,1015#1,1016#1,1017#1,1018#1,1019#1,1020#1,1021#1,1022#1,1023#1,1024#1,1025#1,1026#1,1027#1,1028#1,1029#1,1030#1,1031#1,1032#1,1033#1,1034#1,1035#1,1036#1,1037#1,1038#1,1039#1,1040#1,1041#1,1042#1,1043#1,1044#1,1045#1,1046#1,1047#1,1048#1,1049#1,1050#1,1051#1,1052#1,1053#1,1054#1,1055#1,1056#1,1057#1,1058#1,1059#1,1060#1,1061#1,1062#1,1063#1,1064#1,1065#1,1066#1,1067#1,1068#1,1069#1,1070#1,1071#1,1072#1,1073#1,1074#1,1075#1,1076#1,1077#1,1078#1,1079#1,1080#1,1081#1,1082#1,1083#1,1084#1,1085#1,1086#1,1087#1,1088#1,1089#1,1090#1,1091#1,1092#1,1093#1,1094#1,1095#1,1096#1,1097#1,1098#1,1099#1,1100#1,1101#1,1102#1,1103#1,1104#1,1105#1,1106#1,1107#1,1108#1,1109#1,1110#1,1111#1,1112#1,1113#1,1114#1,1115#1,1116#1,1117#1,1118#1,1119#1,1120#1,1121#1,1122#1,1123#1,1124#1,1125#1,1126#1,1127#1,1128#1,1129#1,1130#1,1131#1,1132#1,1133#1,1134#1,1135#1,1136#1,1137#1,1138#1,1139#1,1140#1,1141#1,1142#1,1143#1,1144#1,1145#1,1146#1,1147#1,1148#1,1149#1,1150#1,1151#1,1152#1,1153#1,1154#1,1155#1,1156#1,1157#1,1158#1,1159#1,1160#1,1161#1,1162#1,1163#1,1164#1,1165#1,1166#1,1167#1,1168#1,1169#1,1170#1,1171#1,1172#1,1173#1,1174#1,1175#1,1176#1,1177#1,1178#1,1179#1,1180#1,1181#1,1182#1,1183#1,1184#1,1185#1,1186#1,1187#1,1188#1,1189#1,1190#1,1191#1,1192#1,1193#1,1194#1,1195#1,1196#1,1197#1,1198#1,1199#1,1200#1,1201#1,1202#1,1203#1,1204#1,1205#1,1206#1,1207#1,1208#1,1209#1,1210#1,1211#1,1212#1,1213#1,1214#1,1215#1,1216#1,1217#1,1218#1,1219#1,1220#1,1221#1,1222#1,1223#1,1224#1,1225#1,1226#1,1227#1,1228#1,1229#1,1230#1,1231#1,1232#1,1233#1,1234#1,1235#1,1236#1,1237#1,1238#1,1239#1,1240#1,1241#1,1242#1,1243#1,1244#1,1245#1,1246#1,1247#1,1248#1,1249#1,1250#1,1251#1,1252#1,1253#1,1254#1,1255#1,1256#1,1257#1,1258#1,1259#1,1260#1,1261#1,1262#1,1263#1,1264#1,1265#1,1266#1,1267#1,1268#1,1269#1,1270#1,1271#1,1272#1,1273#1,1274#1,1275#1,1276#1,1277#1,1278#1,1279#1,1280#1,1281#1,1282#1,1283#1,1284#1,1285#1,1286#1,1287#1,1288#1,1289#1,1290#1,1291#1,1292#1,1293#1,1294#1,1295#1,1296#1,1297#1,1298#1,1299#1,1300#1,1301#1,1302#1,1303#1,1304#1,1305#1,1306#1,1307#1,1308#1,1309#1,1310#1,1311#1,1312#1,1313#1,1314#1,1315#1,1316#1,1317#1,1318#1,1319#1,1320#1,1321#1,1322#1,1323#1,1324#1,1325#1,1326#1,1327#1,1328#1,1329#1,1330#1,1331#1,1332#1,1333#1,1334#1,1335#1,1336#1,1337#1,1338#1,1339#1,1340#1,1341#1,1342#1,1343#1,1344#1,1345#1,1346#1,1347#1,1348#1,1349#1,1350#1,1351#1,1352#1,1353#1,1354#1,1355#1,1356#1,1357#1,1358#1,1359#1,1360#1,1361#1,1362#1,1363#1,1364#1,1365#1,1366#1,1367#1,1368#1,1369#1,1370#1,1371#1,1372#1,1373#1,1374#1,1375#1,1376#1,1377#1,1378#1,1379#1,1380#1,1381#1,1382#1,1383#1,1384#1,1385#1,1386#1,1387#1,1388#1,1389#1,1390#1,1391#1,1392#1,1393#1,1394#1,1395#1,1396#1,1397#1,1398#1,1399#1,1400#1,1401#1,1402#1,1403#1,1404#1,1405#1,1406#1,1407#1,1408#1,1409#1,1410#1,1411#1,1412#1,1413#1,1414#1,1415#1,1416#1,1417#1,1418#1,1419#1,1420#1,1421#1,1422#1,1423#1,1424#1,1425#1,1426#1,1427#1,1428#1,1429#1,1430#1,1431#1,1432#1,1433#1,1434#1,1435#1,1436#1,1437#1,1438#1,1439#1,1440#1,1441#1,1442#1,1443#1,1444#1,1445#1,1446#1,1447#1,1448#1,1449#1,1450#1,1451#1,1452#1,1453#1,1454#1,1455#1,1456#1,1457#1,1458#1,1459#1,1460#1,1461#1,1462#1,1463#1,1464#1,1465#1,1466#1,1467#1,1468#1,1469#1,1470#1,1471#1,1472#1,1473#1,1474#1,1475#1,1476#1,1477#1,1478#1,1479#1,1480#1,1481#1,1482#1,1483#1,1484#1,1485#1,1486#1,1487#1,1488#1,1489#1,1490#1,1491#1,1492#1,1493#1,1494#1,1495#1,1496#1,1497#1,1498#1",
    "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0,80#0,81#0,82#0,83#0,84#0,85#0,86#0,87#0,88#0,89#0,90#0,91#0,92#0,93#0,94#0,95#0,96#0,97#0,98#0,99#0,100#0,101#0,102#0,103#0,104#0,105#0,106#0,107#0,108#0,109#0,110#0,111#0,112#0,113#0,114#0,115#0,116#0,117#0,118#0,119#0,120#0,121#0,122#0,123#0,124#0,125#0,126#0,127#0,128#0,129#0,130#0,131#0,132#0,133#0,134#0,135#0,136#0,137#0,138#0,139#0,140#0,141#0,142#0,143#0,144#0,145#0,146#0,147#0,148#0,149#0,150#0,151#0,152#0,153#0,154#0,155#0,156#0,157#0,158#0,159#0,160#0,161#0,162#0,163#0,164#0,165#0,166#0,167#0,168#0,169#0,170#0,171#0,172#0,173#0,174#0,175#0,176#0,177#0,178#0,179#0,180#0,181#0,182#0,183#0,184#0,185#0,186#0,187#0,188#0,189#0,190#0,191#0,192#0,193#0,194#0,195#0,196#0,197#0,198#0,199#0,200#0,201#0,202#0,203#0,204#0,205#0,206#0,207#0,208#0,209#0,210#0,211#0,212#0,213#0,214#0,215#0,216#0,217#0,218#0,219#0,220#0,221#0,222#0,223#0,224#0,225#0,226#0,227#0,228#0,229#0,230#0,231#0,232#0,233#0,234#0,235#0,236#0,237#0,238#0,239#0,240#0,241#0,242#0,243#0,244#0,245#0,246#0,247#0,248#0,249#0,250#0,251#0,252#0,253#0,254#0,255#0,256#0,257#0,258#0,259#0,260#0,261#0,262#0,263#0,264#0,265#0,266#0,267#0,268#0,269#0,270#0,271#0,272#0,273#0,274#0,275#0,276#0,277#0,278#0,279#0,280#0,281#0,282#0,283#0,284#0,285#0,286#0,287#0,288#0,289#0,290#0,291#0,292#0,293#0,294#0,295#0,296#0,297#0,298#0,299#0,300#0,301#0,302#0,303#0,304#0,305#0,306#0,307#0,308#0,309#0,310#0,311#0,312#0,313#0,314#0,315#0,316#0,317#0,318#0,319#0,320#0,321#0,322#0,323#0,324#0,325#0,326#0,327#0,328#0,329#0,330#0,331#0,332#0,333#0,334#0,335#0,336#0,337#0,338#0,339#0,340#0,341#0,342#0,343#0,344#0,345#0,346#0,347#0,348#0,349#0,350#0,351#0,352#0,353#0,354#0,355#0,356#0,357#0,358#0,359#0,360#0,361#0,362#0,363#0,364#0,365#0,366#0,367#0,368#0,369#0,370#0,371#0,372#0,373#0,374#0,375#0,376#0,377#0,378#0,379#0,380#0,381#0,382#0,383#0,384#0,385#0,386#0,387#0,388#0,389#0,390#0,391#0,392#0,393#0,394#0,395#0,396#0,397#0,398#0,399#0,400#0,401#0,402#0,403#0,404#0,405#0,406#0,407#0,408#0,409#0,410#0,411#0,412#0,413#0,414#0,415#0,416#0,417#0,418#0,419#0,420#0,421#0,422#0,423#0,424#0,425#0,426#0,427#0,428#0,429#0,430#0,431#0,432#0,433#0,434#0,435#0,436#0,437#0,438#0,439#0,440#0,441#0,442#0,443#0,444#0,445#0,446#0,447#0,448#0,449#0,450#0,451#0,452#0,453#0,454#0,455#0,456#0,457#0,458#0,459#0,460#0,461#0,462#0,463#0,464#0,465#0,466#0,467#0,468#0,469#0,470#0,471#0,472#0,473#0,474#0,475#0,476#0,477#0,478#0,479#0,480#0,481#0,482#0,483#0,484#0,485#0,486#0,487#0,488#0,489#0,490#0,491#0,492#0,493#0,494#0,495#0,496#0,497#0,498#0,499#0,500#0,501#0,502#0,503#0,504#0,505#0,506#0,507#0,508#0,509#0,510#0,511#0,512#0,513#0,514#0,515#0,516#0,517#0,518#0,519#0,520#0,521#0,522#0,523#0,524#0,525#0,526#0,527#0,528#0,529#0,530#0,531#0,532#0,533#0,534#0,535#0,536#0,537#0,538#0,539#0,540#0,541#0,542#0,543#0,544#0,545#0,546#0,547#0,548#0,549#0,550#0,551#0,552#0,553#0,554#0,555#0,556#0,557#0,558#0,559#0,560#0,561#0,562#0,563#0,564#0,565#0,566#0,567#0,568#0,569#0,570#0,571#0,572#0,573#0,574#0,575#0,576#0,577#0,578#0,579#0,580#0,581#0,582#0,583#0,584#0,585#0,586#0,587#0,588#0,589#0,590#0,591#0,592#0,593#0,594#0,595#0,596#0,597#0,598#0,599#0,600#0,601#0,602#0,603#0,604#0,605#0,606#0,607#0,608#0,609#0,610#0,611#0,612#0,613#0,614#0,615#0,616#0,617#0,618#0,619#0,620#0,621#0,622#0,623#0,624#0,625#0,626#0,627#0,628#0,629#0,630#0,631#0,632#0,633#0,634#0,635#0,636#0,637#0,638#0,639#0,640#0,641#0,642#0,643#0,644#0,645#0,646#0,647#0,648#0,649#0,650#0,651#0,652#0,653#0,654#0,655#0,656#0,657#0,658#0,659#0,660#0,661#0,662#0,663#0,664#0,665#0,666#0,667#0,668#0,669#0,670#0,671#0,672#0,673#0,674#0,675#0,676#0,677#0,678#0,679#0,680#0,681#0,682#0,683#0,684#0,685#0,686#0,687#0,688#0,689#0,690#0,691#0,692#0,693#0,694#0,695#0,696#0,697#0,698#0,699#0,700#0,701#0,702#0,703#0,704#0,705#0,706#0,707#0,708#0,709#0,710#0,711#0,712#0,713#0,714#0,715#0,716#0,717#0,718#0,719#0,720#0,721#0,722#0,723#0,724#0,725#0,726#0,727#0,728#0,729#0,730#0,731#0,732#0,733#0,734#0,735#0,736#0,737#0,738#0,739#0,740#0,741#0,742#0,743#0,744#0,745#0,746#0,747#0,748#0,749#0,750#0,751#0,752#0,753#0,754#0,755#0,756#0,757#0,758#0,759#0,760#0,761#0,762#0,763#0,764#0,765#0,766#0,767#0,768#0,769#0,770#0,771#0,772#0,773#0,774#0,775#0,776#0,777#0,778#0,779#0,780#0,781#0,782#0,783#0,784#0,785#0,786#0,787#0,788#0,789#0,790#0,791#0,792#0,793#0,794#0,795#0,796#0,797#0,798#0,799#0,800#0,801#0,802#0,803#0,804#0,805#0,806#0,807#0,808#0,809#0,810#0,811#0,812#0,813#0,814#0,815#0,816#0,817#0,818#0,819#0,820#0,821#0,822#0,823#0,824#0,825#0,826#0,827#0,828#0,829#0,830#0,831#0,832#0,833#0,834#0,835#0,836#0,837#0,838#0,839#0,840#0,841#0,842#0,843#0,844#0,845#0,846#0,847#0,848#0,849#0,850#0,851#0,852#0,853#0,854#0,855#0,856#0,857#0,858#0,859#0,860#0,861#0,862#0,863#0,864#0,865#0,866#0,867#0,868#0,869#0,870#0,871#0,872#0,873#0,874#0,875#0,876#0,877#0,878#0,879#0,880#0,881#0,882#0,883#0,884#0,885#0,886#0,887#0,888#0,889#0,890#0,891#0,892#0,893#0,894#0,895#0,896#0,897#0,898#0,899#0,900#0,901#0,902#0,903#0,904#0,905#0,906#0,907#0,908#0,909#0,910#0,911#0,912#0,913#0,914#0,915#0,916#0,917#0,918#0,919#0,920#0,921#0,922#0,923#0,924#0,925#0,926#0,927#0,928#0,929#0,930#0,931#0,932#0,933#0,934#0,935#0,936#0,937#0,938#0,939#0,940#0,941#0,942#0,943#0,944#0,945#0,946#0,947#0,948#0,949#0,950#0,951#0,952#0,953#0,954#0,955#0,956#0,957#0,958#0,959#0,960#0,961#0,962#0,963#0,964#0,965#0,966#0,967#0,968#0,969#0,970#0,971#0,972#0,973#0,974#0,975#0,976#0,977#0,978#0,979#0,980#0,981#0,982#0,983#0,984#0,985#0,986#0,987#0,988#0,989#0,990#0,991#0,992#0,993#0,994#0,995#0,996#0,997#0,998#0,999#0,1000#0,1001#0,1002#0,1003#0,1004#0,1005#0,1006#0,1007#0,1008#0,1009#0,1010#0,1011#0,1012#0,1013#0,1014#0,1015#0,1016#0,1017#0,1018#0,1019#0,1020#0,1021#0,1022#0,1023#0,1024#0,1025#0,1026#0,1027#0,1028#0,1029#0,1030#0,1031#0,1032#0,1033#0,1034#0,1035#0,1036#0,1037#0,1038#0,1039#0,1040#0,1041#0,1042#0,1043#0,1044#0,1045#0,1046#0,1047#0,1048#0,1049#0,1050#0,1051#0,1052#0,1053#0,1054#0,1055#0,1056#0,1057#0,1058#0,1059#0,1060#0,1061#0,1062#0,1063#0,1064#0,1065#0,1066#0,1067#0,1068#0,1069#0,1070#0,1071#0,1072#0,1073#0,1074#0,1075#0,1076#0,1077#0,1078#0,1079#0,1080#0,1081#0,1082#0,1083#0,1084#0,1085#0,1086#0,1087#0,1088#0,1089#0,1090#0,1091#0,1092#0,1093#0,1094#0,1095#0,1096#0,1097#0,1098#0,1099#0,1100#0,1101#0,1102#0,1103#0,1104#0,1105#0,1106#0,1107#0,1108#0,1109#0,1110#0,1111#0,1112#0,1113#0,1114#0,1115#0,1116#0,1117#0,1118#0,1119#0,1120#0,1121#0,1122#0,1123#0,1124#0,1125#0,1126#0,1127#0,1128#0,1129#0,1130#0,1131#0,1132#0,1133#0,1134#0,1135#0,1136#0,1137#0,1138#0,1139#0,1140#0,1141#0,1142#0,1143#0,1144#0,1145#0,1146#0,1147#0,1148#0,1149#0,1150#0,1151#0,1152#0,1153#0,1154#0,1155#0,1156#0,1157#0,1158#0,1159#0,1160#0,1161#0,1162#0,1163#0,1164#0,1165#0,1166#0,1167#0,1168#0,1169#0,1170#0,1171#0,1172#0,1173#0,1174#0,1175#0,1176#0,1177#0,1178#0,1179#0,1180#0,1181#0,1182#0,1183#0,1184#0,1185#0,1186#0,1187#0,1188#0,1189#0,1190#0,1191#0,1192#0,1193#0,1194#0,1195#0,1196#0,1197#0,1198#0,1199#0,1200#0,1201#0,1202#0,1203#0,1204#0,1205#0,1206#0,1207#0,1208#0,1209#0,1210#0,1211#0,1212#0,1213#0,1214#0,1215#0,1216#0,1217#0,1218#0,1219#0,1220#0,1221#0,1222#0,1223#0,1224#0,1225#0,1226#0,1227#0,1228#0,1229#0,1230#0,1231#0,1232#0,1233#0,1234#0,1235#0,1236#0,1237#0,1238#0,1239#0,1240#0,1241#0,1242#0,1243#0,1244#0,1245#0,1246#0,1247#0,1248#0,1249#0,1250#0,1251#0,1252#0,1253#0,1254#0,1255#0,1256#0,1257#0,1258#0,1259#0,1260#0,1261#0,1262#0,1263#0,1264#0,1265#0,1266#0,1267#0,1268#0,1269#0,1270#0,1271#0,1272#0,1273#0,1274#0,1275#0,1276#0,1277#0,1278#0,1279#0,1280#0,1281#0,1282#0,1283#0,1284#0,1285#0,1286#0,1287#0,1288#0,1289#0,1290#0,1291#0,1292#0,1293#0,1294#0,1295#0,1296#0,1297#0,1298#0,1299#0,1300#0,1301#0,1302#0,1303#0,1304#0,1305#0,1306#0,1307#0,1308#0,1309#0,1310#0,1311#0,1312#0,1313#0,1314#0,1315#0,1316#0,1317#0,1318#0,1319#0,1320#0,1321#0,1322#0,1323#0,1324#0,1325#0,1326#0,1327#0,1328#0,1329#0,1330#0,1331#0,1332#0,1333#0,1334#0,1335#0,1336#0,1337#0,1338#0,1339#0,1340#0,1341#0,1342#0,1343#0,1344#0,1345#0,1346#0,1347#0,1348#0,1349#0,1350#0,1351#0,1352#0,1353#0,1354#0,1355#0,1356#0,1357#0,1358#0,1359#0,1360#0,1361#0,1362#0,1363#0,1364#0,1365#0,1366#0,1367#0,1368#0,1369#0,1370#0,1371#0,1372#0,1373#0,1374#0,1375#0,1376#0,1377#0,1378#0,1379#0,1380#0,1381#0,1382#0,1383#0,1384#0,1385#0,1386#0,1387#0,1388#0,1389#0,1390#0,1391#0,1392#0,1393#0,1394#0,1395#0,1396#0,1397#0,1398#0,1399#0,1400#0,1401#0,1402#0,1403#0,1404#0,1405#0,1406#0,1407#0,1408#0,1409#0,1410#0,1411#0,1412#0,1413#0,1414#0,1415#0,1416#0,1417#0,1418#0,1419#0,1420#0,1421#0,1422#0,1423#0,1424#0,1425#0,1426#0,1427#0,1428#0,1429#0,1430#0,1431#0,1432#0,1433#0,1434#0,1435#0,1436#0,1437#0,1438#0,1439#0,1440#0,1441#0,1442#0,1443#0,1444#0,1445#0,1446#0,1447#0,1448#0,1449#0,1450#0,1451#0,1452#0,1453#0,1454#0,1455#0,1456#0,1457#0,1458#0,1459#0,1460#0,1461#0,1462#0,1463#0,1464#0,1465#0,1466#0,1467#0,1468#0,1469#0,1470#0,1471#0,1472#0,1473#0,1474#0,1475#0,1476#0,1477#0,1478#0,1479#0,1480#0,1481#0,1482#0,1483#0,1484#0,1485#0,1486#0,1487#0,1488#0,1489#0,1490#0,1491#0,1492#0,1493#0,1494#0,1495#0,1496#0,1497#0,1498#0",
    "mtime": "0.000000",
    "creation_time": "2022-03-21T07:39:03.115780Z",
    "max_marker": "0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#,80#,81#,82#,83#,84#,85#,86#,87#,88#,89#,90#,91#,92#,93#,94#,95#,96#,97#,98#,99#,100#,101#,102#,103#,104#,105#,106#,107#,108#,109#,110#,111#,112#,113#,114#,115#,116#,117#,118#,119#,120#,121#,122#,123#,124#,125#,126#,127#,128#,129#,130#,131#,132#,133#,134#,135#,136#,137#,138#,139#,140#,141#,142#,143#,144#,145#,146#,147#,148#,149#,150#,151#,152#,153#,154#,155#,156#,157#,158#,159#,160#,161#,162#,163#,164#,165#,166#,167#,168#,169#,170#,171#,172#,173#,174#,175#,176#,177#,178#,179#,180#,181#,182#,183#,184#,185#,186#,187#,188#,189#,190#,191#,192#,193#,194#,195#,196#,197#,198#,199#,200#,201#,202#,203#,204#,205#,206#,207#,208#,209#,210#,211#,212#,213#,214#,215#,216#,217#,218#,219#,220#,221#,222#,223#,224#,225#,226#,227#,228#,229#,230#,231#,232#,233#,234#,235#,236#,237#,238#,239#,240#,241#,242#,243#,244#,245#,246#,247#,248#,249#,250#,251#,252#,253#,254#,255#,256#,257#,258#,259#,260#,261#,262#,263#,264#,265#,266#,267#,268#,269#,270#,271#,272#,273#,274#,275#,276#,277#,278#,279#,280#,281#,282#,283#,284#,285#,286#,287#,288#,289#,290#,291#,292#,293#,294#,295#,296#,297#,298#,299#,300#,301#,302#,303#,304#,305#,306#,307#,308#,309#,310#,311#,312#,313#,314#,315#,316#,317#,318#,319#,320#,321#,322#,323#,324#,325#,326#,327#,328#,329#,330#,331#,332#,333#,334#,335#,336#,337#,338#,339#,340#,341#,342#,343#,344#,345#,346#,347#,348#,349#,350#,351#,352#,353#,354#,355#,356#,357#,358#,359#,360#,361#,362#,363#,364#,365#,366#,367#,368#,369#,370#,371#,372#,373#,374#,375#,376#,377#,378#,379#,380#,381#,382#,383#,384#,385#,386#,387#,388#,389#,390#,391#,392#,393#,394#,395#,396#,397#,398#,399#,400#,401#,402#,403#,404#,405#,406#,407#,408#,409#,410#,411#,412#,413#,414#,415#,416#,417#,418#,419#,420#,421#,422#,423#,424#,425#,426#,427#,428#,429#,430#,431#,432#,433#,434#,435#,436#,437#,438#,439#,440#,441#,442#,443#,444#,445#,446#,447#,448#,449#,450#,451#,452#,453#,454#,455#,456#,457#,458#,459#,460#,461#,462#,463#,464#,465#,466#,467#,468#,469#,470#,471#,472#,473#,474#,475#,476#,477#,478#,479#,480#,481#,482#,483#,484#,485#,486#,487#,488#,489#,490#,491#,492#,493#,494#,495#,496#,497#,498#,499#,500#,501#,502#,503#,504#,505#,506#,507#,508#,509#,510#,511#,512#,513#,514#,515#,516#,517#,518#,519#,520#,521#,522#,523#,524#,525#,526#,527#,528#,529#,530#,531#,532#,533#,534#,535#,536#,537#,538#,539#,540#,541#,542#,543#,544#,545#,546#,547#,548#,549#,550#,551#,552#,553#,554#,555#,556#,557#,558#,559#,560#,561#,562#,563#,564#,565#,566#,567#,568#,569#,570#,571#,572#,573#,574#,575#,576#,577#,578#,579#,580#,581#,582#,583#,584#,585#,586#,587#,588#,589#,590#,591#,592#,593#,594#,595#,596#,597#,598#,599#,600#,601#,602#,603#,604#,605#,606#,607#,608#,609#,610#,611#,612#,613#,614#,615#,616#,617#,618#,619#,620#,621#,622#,623#,624#,625#,626#,627#,628#,629#,630#,631#,632#,633#,634#,635#,636#,637#,638#,639#,640#,641#,642#,643#,644#,645#,646#,647#,648#,649#,650#,651#,652#,653#,654#,655#,656#,657#,658#,659#,660#,661#,662#,663#,664#,665#,666#,667#,668#,669#,670#,671#,672#,673#,674#,675#,676#,677#,678#,679#,680#,681#,682#,683#,684#,685#,686#,687#,688#,689#,690#,691#,692#,693#,694#,695#,696#,697#,698#,699#,700#,701#,702#,703#,704#,705#,706#,707#,708#,709#,710#,711#,712#,713#,714#,715#,716#,717#,718#,719#,720#,721#,722#,723#,724#,725#,726#,727#,728#,729#,730#,731#,732#,733#,734#,735#,736#,737#,738#,739#,740#,741#,742#,743#,744#,745#,746#,747#,748#,749#,750#,751#,752#,753#,754#,755#,756#,757#,758#,759#,760#,761#,762#,763#,764#,765#,766#,767#,768#,769#,770#,771#,772#,773#,774#,775#,776#,777#,778#,779#,780#,781#,782#,783#,784#,785#,786#,787#,788#,789#,790#,791#,792#,793#,794#,795#,796#,797#,798#,799#,800#,801#,802#,803#,804#,805#,806#,807#,808#,809#,810#,811#,812#,813#,814#,815#,816#,817#,818#,819#,820#,821#,822#,823#,824#,825#,826#,827#,828#,829#,830#,831#,832#,833#,834#,835#,836#,837#,838#,839#,840#,841#,842#,843#,844#,845#,846#,847#,848#,849#,850#,851#,852#,853#,854#,855#,856#,857#,858#,859#,860#,861#,862#,863#,864#,865#,866#,867#,868#,869#,870#,871#,872#,873#,874#,875#,876#,877#,878#,879#,880#,881#,882#,883#,884#,885#,886#,887#,888#,889#,890#,891#,892#,893#,894#,895#,896#,897#,898#,899#,900#,901#,902#,903#,904#,905#,906#,907#,908#,909#,910#,911#,912#,913#,914#,915#,916#,917#,918#,919#,920#,921#,922#,923#,924#,925#,926#,927#,928#,929#,930#,931#,932#,933#,934#,935#,936#,937#,938#,939#,940#,941#,942#,943#,944#,945#,946#,947#,948#,949#,950#,951#,952#,953#,954#,955#,956#,957#,958#,959#,960#,961#,962#,963#,964#,965#,966#,967#,968#,969#,970#,971#,972#,973#,974#,975#,976#,977#,978#,979#,980#,981#,982#,983#,984#,985#,986#,987#,988#,989#,990#,991#,992#,993#,994#,995#,996#,997#,998#,999#,1000#,1001#,1002#,1003#,1004#,1005#,1006#,1007#,1008#,1009#,1010#,1011#,1012#,1013#,1014#,1015#,1016#,1017#,1018#,1019#,1020#,1021#,1022#,1023#,1024#,1025#,1026#,1027#,1028#,1029#,1030#,1031#,1032#,1033#,1034#,1035#,1036#,1037#,1038#,1039#,1040#,1041#,1042#,1043#,1044#,1045#,1046#,1047#,1048#,1049#,1050#,1051#,1052#,1053#,1054#,1055#,1056#,1057#,1058#,1059#,1060#,1061#,1062#,1063#,1064#,1065#,1066#,1067#,1068#,1069#,1070#,1071#,1072#,1073#,1074#,1075#,1076#,1077#,1078#,1079#,1080#,1081#,1082#,1083#,1084#,1085#,1086#,1087#,1088#,1089#,1090#,1091#,1092#,1093#,1094#,1095#,1096#,1097#,1098#,1099#,1100#,1101#,1102#,1103#,1104#,1105#,1106#,1107#,1108#,1109#,1110#,1111#,1112#,1113#,1114#,1115#,1116#,1117#,1118#,1119#,1120#,1121#,1122#,1123#,1124#,1125#,1126#,1127#,1128#,1129#,1130#,1131#,1132#,1133#,1134#,1135#,1136#,1137#,1138#,1139#,1140#,1141#,1142#,1143#,1144#,1145#,1146#,1147#,1148#,1149#,1150#,1151#,1152#,1153#,1154#,1155#,1156#,1157#,1158#,1159#,1160#,1161#,1162#,1163#,1164#,1165#,1166#,1167#,1168#,1169#,1170#,1171#,1172#,1173#,1174#,1175#,1176#,1177#,1178#,1179#,1180#,1181#,1182#,1183#,1184#,1185#,1186#,1187#,1188#,1189#,1190#,1191#,1192#,1193#,1194#,1195#,1196#,1197#,1198#,1199#,1200#,1201#,1202#,1203#,1204#,1205#,1206#,1207#,1208#,1209#,1210#,1211#,1212#,1213#,1214#,1215#,1216#,1217#,1218#,1219#,1220#,1221#,1222#,1223#,1224#,1225#,1226#,1227#,1228#,1229#,1230#,1231#,1232#,1233#,1234#,1235#,1236#,1237#,1238#,1239#,1240#,1241#,1242#,1243#,1244#,1245#,1246#,1247#,1248#,1249#,1250#,1251#,1252#,1253#,1254#,1255#,1256#,1257#,1258#,1259#,1260#,1261#,1262#,1263#,1264#,1265#,1266#,1267#,1268#,1269#,1270#,1271#,1272#,1273#,1274#,1275#,1276#,1277#,1278#,1279#,1280#,1281#,1282#,1283#,1284#,1285#,1286#,1287#,1288#,1289#,1290#,1291#,1292#,1293#,1294#,1295#,1296#,1297#,1298#,1299#,1300#,1301#,1302#,1303#,1304#,1305#,1306#,1307#,1308#,1309#,1310#,1311#,1312#,1313#,1314#,1315#,1316#,1317#,1318#,1319#,1320#,1321#,1322#,1323#,1324#,1325#,1326#,1327#,1328#,1329#,1330#,1331#,1332#,1333#,1334#,1335#,1336#,1337#,1338#,1339#,1340#,1341#,1342#,1343#,1344#,1345#,1346#,1347#,1348#,1349#,1350#,1351#,1352#,1353#,1354#,1355#,1356#,1357#,1358#,1359#,1360#,1361#,1362#,1363#,1364#,1365#,1366#,1367#,1368#,1369#,1370#,1371#,1372#,1373#,1374#,1375#,1376#,1377#,1378#,1379#,1380#,1381#,1382#,1383#,1384#,1385#,1386#,1387#,1388#,1389#,1390#,1391#,1392#,1393#,1394#,1395#,1396#,1397#,1398#,1399#,1400#,1401#,1402#,1403#,1404#,1405#,1406#,1407#,1408#,1409#,1410#,1411#,1412#,1413#,1414#,1415#,1416#,1417#,1418#,1419#,1420#,1421#,1422#,1423#,1424#,1425#,1426#,1427#,1428#,1429#,1430#,1431#,1432#,1433#,1434#,1435#,1436#,1437#,1438#,1439#,1440#,1441#,1442#,1443#,1444#,1445#,1446#,1447#,1448#,1449#,1450#,1451#,1452#,1453#,1454#,1455#,1456#,1457#,1458#,1459#,1460#,1461#,1462#,1463#,1464#,1465#,1466#,1467#,1468#,1469#,1470#,1471#,1472#,1473#,1474#,1475#,1476#,1477#,1478#,1479#,1480#,1481#,1482#,1483#,1484#,1485#,1486#,1487#,1488#,1489#,1490#,1491#,1492#,1493#,1494#,1495#,1496#,1497#,1498#",
    "usage": {},
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

#6 Updated by Vikhyat Umrao about 2 years ago

# radosgw-admin bucket check --bucket mycontainers3 --fix
[]
{}
{
    "existing_header": {
        "usage": {}
    },
    "calculated_header": {
        "usage": {}
    }
}

#7 Updated by Vikhyat Umrao about 2 years ago

Tried with the bucket - mycontainers5 and it ended with the same usage 0. Originally these buckets have ~70 million objects.

#8 Updated by Vikhyat Umrao about 2 years ago

# radosgw-admin bucket stats |egrep 'bucket\"|num_objects|num_shards' 
error getting bucket stats bucket=mycontainers4 ret=-2
        "bucket": "mycontainers1",
        "num_shards": 1381,
                "num_objects": 24014513
        "bucket": "mycontainers3",
        "num_shards": 1499,
        "bucket": "mycontainers2",
        "num_shards": 1499,
                "num_objects": 24012178
        "bucket": "mycontainers5",
        "num_shards": 1499,

#9 Updated by Vikhyat Umrao about 2 years ago

The bucket stats in RHCS 5.1(pacific) before upgrading to quincy.

Site1 buckets (rgw):
2022/03/21-14:21:34:         "bucket": "mycontainers1",
        "num_shards": 1381,
                "num_objects": 73367136
        "bucket": "mycontainers3",
        "num_shards": 1277,
                "num_objects": 74999866
        "bucket": "mycontainers4",
        "num_shards": 1019,
                "num_objects": 74999879
        "bucket": "mycontainers2",
        "num_shards": 1499,
                "num_objects": 73366306
        "bucket": "mycontainers5",
        "num_shards": 1151,
                "num_objects": 74999496

- During Upgrade, corruption came!

Site1 buckets (rgw):
2022/03/21-18:35:16:         "bucket": "mycontainers1",
        "num_shards": 1381,
                "num_objects": 446520
        "bucket": "mycontainers2",
        "num_shards": 1499,
                "num_objects": 446448

#10 Updated by Vikhyat Umrao about 2 years ago

The issue was not reproduced when the upgrade was run with upstream pacific -> upstream quincy builds. Maybe something in the downstream 5.1 builds.

radosgw-admin bucket stats |egrep 'bucket\"|num_objects|num_shards'
ceph version 17.1.0 (c675060073a05d40ef404d5921c81178a52af6e0) quincy (dev)
        "bucket": "mycontainers1",
        "num_shards": 1453,
                "num_objects": 66983459
        "bucket": "mycontainers3",
        "num_shards": 1447,
                "num_objects": 74994511
        "bucket": "mycontainers4",
        "num_shards": 1087,
                "num_objects": 74997844
        "bucket": "mycontainers2",
        "num_shards": 1301,
                "num_objects": 66980944
        "bucket": "mycontainers5",
        "num_shards": 787,
                "num_objects": 74998260

#11 Updated by Casey Bodley about 2 years ago

  • Assignee set to Casey Bodley

#12 Updated by Casey Bodley about 2 years ago

Vikhyat Umrao wrote:

The issue was not reproduced when the upgrade was run with upstream pacific -> upstream quincy builds. Maybe something in the downstream 5.1 builds.

should we close this then?

#13 Updated by Casey Bodley about 2 years ago

  • Status changed from New to Need More Info

#14 Updated by Vikhyat Umrao about 2 years ago

  • Status changed from Need More Info to Can't reproduce

Casey Bodley wrote:

Vikhyat Umrao wrote:

The issue was not reproduced when the upgrade was run with upstream pacific -> upstream quincy builds. Maybe something in the downstream 5.1 builds.

should we close this then?

Sure, thanks, Casey. I think we can close this one for now. I have kept the downstream bug open so when next time, we will test the upgrade from downstream pacific to downstream quincy we confirm the issue in downstream builds.

Also available in: Atom PDF