Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2021-09-29T07:26:44Z
Ceph
Redmine
RADOS - Bug #52756 (Resolved): ceph-kvstore-tool repair segmentfault without bluestore-kv
https://tracker.ceph.com/issues/52756
2021-09-29T07:26:44Z
huang jun
hjwsm1989@gmail.com
<pre><code class="text syntaxhl"><span class="CodeRay">steps to reproduce:
1. create a objectstore db
mkdir /tmp/testkv
ceph-objectstore-tool --data-path /tmp/testkv --id=0 --op mkfs
2. repair it
ceph-kvstore-tool rocksdb /tmp/testkv/db repair
3. the repair cmd crashed with segmentfault
</span></code></pre>
bluestore - Bug #52705 (Resolved): pg scrub stat mismatch with special objects that have hash 'ff...
https://tracker.ceph.com/issues/52705
2021-09-22T15:34:34Z
huang jun
hjwsm1989@gmail.com
<p>Here is the steps to reproduce the pg inconsistents:<br />1. build a cluster with vstart<br />MON=1 OSD=3 MDS=0 MGR=1 ../src/vstart.sh -X -n<br />2. create a pool with only 1 pg<br />./bin/ceph osd pool create rbd 1<br />3. put an object with special name, which will let the object hash as 'ffffffff'<br />./bin/rados -p rbd put 59431f0f-60c1-49be-ab18-9bfbee59869c.1019453.1_obj0344489466 /etc/passwd --debug_objecter=20 --debug_ms=5<br />4. trigger the pg scrub<br />./bin/ceph pg scrub 1.0<br />5. the expect output of 'ceph -s' will include 'active+clean+inconsistent'</p>
<p>6. the problem maybe the objectstore::get_coll_range interface which will not include the object with hash 'ffffffff'</p>
Ceph - Bug #51842 (Resolved): upmap verify failed with pool size decreased
https://tracker.ceph.com/issues/51842
2021-07-25T04:06:19Z
huang jun
hjwsm1989@gmail.com
<p>we have a replicated pool with 3 osds, pool size is 3<br /><pre><code class="text syntaxhl"><span class="CodeRay">ID CLASS WEIGHT TYPE NAME
-5 3.00000 root infra-1706
-4 1.00000 host host-0
0 1.00000 osd.0
-6 1.00000 host host-1
1 1.00000 osd.1
-7 1.00000 host host-2
2 1.00000 osd.2
</span></code></pre></p>
<p>with rule:<br /><pre><code class="text syntaxhl"><span class="CodeRay"> "rule": {
"rule_id": 1,
"rule_name": "infra-1706",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 20,
"steps": [
{
"op": "set_chooseleaf_tries",
"num": 5
},
{
"op": "set_choose_tries",
"num": 100
},
{
"op": "take",
"item": -5,
"item_name": "infra-1706"
},
{
"op": "chooseleaf_firstn",
"num": 0,
"type": "osd"
},
{
"op": "emit"
}
]
}
</span></code></pre></p>
<p>and upmap settings<br /><pre><code class="text syntaxhl"><span class="CodeRay">pg_upmap 1.3 [1,0,2]
pg_upmap 1.4 [1,2,0]
pg_upmap 1.6 [1,2,0]
</span></code></pre></p>
<p>after set the pool size to 1<br />the pg upmap recordings are not cleared,<br />which cause these 3 pgs in active state</p>
RADOS - Bug #42485 (Resolved): verify_upmaps can not cancel invalid upmap_items in some cases
https://tracker.ceph.com/issues/42485
2019-10-25T11:49:06Z
huang jun
hjwsm1989@gmail.com
<p>We can not cancel in verify_upmap if remap an osd to different root bucket,</p>
<p>cluster topology:<br />osd.0 ~ osd.29 belongs to datacenter 1<br />osd30 ~ osd.59 belongs to datacenter 2<br />crush rule:<br />take datacenter 1<br />chooseleaf 2 host<br />emit<br />take datacenter 2<br />chooseleaf 2 host<br />emit<br />The pg's primary osd in datacenter 1.</p>
<p>We should cancel the pgid from upmap_items like below</p>
<p>from [26,12,54,46] to [30,12,54,46]<br />pg_upmap_items [26,30]</p>
<p>from [16,25,53,31] to [40,25,53,0]<br />pg_upmap_items [16,20,31,0]</p>
Ceph - Bug #41177 (Resolved): ceph-objectstore-tool: update-mon-db return EINVAL with missed inc_...
https://tracker.ceph.com/issues/41177
2019-08-09T05:04:32Z
huang jun
hjwsm1989@gmail.com
<pre>
we have a cluster, and test the rebuild mon db from osd,
it return EINVAL when operating the osd.4, as below cmdline showed,
the osd.4 didn't have inc_osdmap.252 bc at epoch 252 the osd.4 not created yet.
[root@node1 ~]# ceph-objectstore-tool --id=4 --data-path=/var/lib/ceph/osd/ceph-4 --op get-osdmap --epoch 252 --file xx
ceph-objectstore-tool --id=4 --data-path=/var/lib/ceph/osd/ceph-4 --op=update-mon-db --mon-store-path=/tmp/sds-tools/rebuild-mon/agentmondbUxYBJS_2019-08-08T18:44:52
ignoring keyring (/var/lib/ceph/osd/ceph-4/keyring): can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory
missing #-1:42a33e63:::inc_osdmap.252:0#
[root@node1 ~]# ceph-objectstore-tool --id=4 --data-path=/var/lib/ceph/osd/ceph-4 --op get-osdmap --epoch 252 --file xx
osdmap#252 exported.
[root@node1 ~]# osdmaptool --print xx
osdmaptool: osdmap file 'xx'
epoch 252
fsid bf47aab9-51bf-4a5e-bc8a-15e508ca79ed
created 2019-08-01 16:13:31.208891
modified 2019-08-01 17:08:22.430813
flags nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs
crush_version 43
full_ratio 0.9
backfillfull_ratio 0.85
nearfull_ratio 0.8
omap_full_ratio 0.9
omap_backfillfull_ratio 0.85
omap_nearfull_ratio 0.8
require_min_compat_client luminous
min_compat_client jewel
require_osd_release luminous
pool 19 'pool-3f06526dc4614faca74209340740f81f' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 256 pgp_num 256 last_change 215 flags hashpspool stripe_width 0 async_recovery_max_updates 60 osd_backfillfull_ratio 0.85 osd_full_ratio 0.9 osd_nearfull_ratio 0.8 osd_omap_backfillfull_ratio 0.85 osd_omap_full_ratio 0.9 osd_omap_nearfull_ratio 0.8
pool 20 'pool-827473ed435f4330b7ea94c08c02e221' replicated size 1 min_size 1 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 last_change 230 flags hashpspool stripe_width 0 async_recovery_max_updates 60 osd_backfillfull_ratio 0.85 osd_full_ratio 0.9 osd_nearfull_ratio 0.8 osd_omap_backfillfull_ratio 0.85 osd_omap_nearfull_ratio 0.8
pool 21 '.sds.rgw' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 237 flags hashpspool stripe_width 0 async_recovery_max_updates 200 osd_backfillfull_ratio 0.85 osd_full_ratio 0.9 osd_nearfull_ratio 0.8 osd_omap_backfillfull_ratio 0.85 osd_omap_nearfull_ratio 0.8
pool 22 '.sds.rgw.root' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 243 flags hashpspool stripe_width 0 async_recovery_max_updates 200 osd_backfillfull_ratio 0.85 osd_full_ratio 0.9 osd_nearfull_ratio 0.8 osd_omap_backfillfull_ratio 0.85 osd_omap_nearfull_ratio 0.8
pool 23 '.sds.rgw.control' replicated size 1 min_size 1 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 250 flags hashpspool stripe_width 0 async_recovery_max_updates 200 osd_backfillfull_ratio 0.85 osd_full_ratio 0.9 osd_nearfull_ratio 0.8 osd_omap_backfillfull_ratio 0.85 osd_omap_nearfull_ratio 0.8
pool 24 '.sds.rgw.gc' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 64 pgp_num 64 last_change 251 flags hashpspool stripe_width 0 async_recovery_max_updates 200 osd_backfillfull_ratio 0.85 osd_full_ratio 0.9 osd_nearfull_ratio 0.8 osd_omap_backfillfull_ratio 0.85 osd_omap_nearfull_ratio 0.8
max_osd 4
osd.0 up in weight 1 up_from 203 up_thru 251 down_at 0 last_clean_interval [0,0) 10.255.101.30:6800/32069 10.255.101.30:6801/32069 10.255.101.30:6803/32069 10.255.101.30:6804/32069 10.255.101.30:6802/32069 exists,up aee33031-c36f-42a1-9fb9-4910a4781343
osd.1 up in weight 1 up_from 217 up_thru 226 down_at 0 last_clean_interval [0,0) 10.255.101.31:6800/29399 10.255.101.31:6801/29399 10.255.101.31:6803/29399 10.255.101.31:6804/29399 10.255.101.31:6802/29399 exists,up d86480ae-34d3-4d41-9e49-9b4f9e03dc9e
it's ok to just ignore the inc_osdmap?
</pre>
Ceph - Bug #39948 (Resolved): auth krb compile error
https://tracker.ceph.com/issues/39948
2019-05-16T00:48:34Z
huang jun
hjwsm1989@gmail.com
<p>The compile error:<br /><pre><code class="c syntaxhl"><span class="CodeRay">/usr/src/ceph/src/auth/krb/KrbAuthorizeHandler.cpp:<span class="integer">22</span>:<span class="integer">6</span>: error: prototype <span class="keyword">for</span> <span class="error">‘</span><span class="predefined-type">bool</span> KrbAuthorizeHandler::verify_authorizer(CephContext*, KeyStore*, <span class="directive">const</span> bufferlist&, size_t, ceph::bufferlist*, EntityName*, uint64_t*, AuthCapsInfo*, CryptoKey*, std::string*, std::unique_ptr<AuthAuthorizerChallenge>*)<span class="error">’</span> does not match any in class <span class="error">‘</span>KrbAuthorizeHandler<span class="error">’</span>
<span class="predefined-type">bool</span> KrbAuthorizeHandler::verify_authorizer(
^~~~~~~~~~~~~~~~~~~
In file included from /usr/src/ceph/src/auth/krb/KrbAuthorizeHandler.cpp:<span class="integer">16</span>:<span class="integer">0</span>:
/usr/src/ceph/src/auth/krb/KrbAuthorizeHandler.hpp:<span class="integer">22</span>:<span class="integer">8</span>: error: candidate is: virtual <span class="predefined-type">bool</span> KrbAuthorizeHandler::verify_authorizer(CephContext*, <span class="directive">const</span> KeyStore&, <span class="directive">const</span> bufferlist&, size_t, ceph::bufferlist*, EntityName*, uint64_t*, AuthCapsInfo*, CryptoKey*, std::string*, std::unique_ptr<AuthAuthorizerChallenge>*)
<span class="predefined-type">bool</span> verify_authorizer(
^~~~~~~~~~~~~~~~~
make[<span class="integer">3</span>]: *** [src/auth/CMakeFiles/common-auth-objs.dir/krb/KrbAuthorizeHandler.cpp.o] Error <span class="integer">1</span>
make[<span class="integer">2</span>]: *** [src/auth/CMakeFiles/common-auth-objs.dir/all] Error <span class="integer">2</span>
make[<span class="integer">2</span>]: *** Waiting <span class="keyword">for</span> unfinished jobs....
make[<span class="integer">1</span>]: *** [src/CMakeFiles/ceph-mon.dir/rule] Error <span class="integer">2</span>
make: *** [ceph-mon] Error <span class="integer">2</span>
</span></code></pre></p>
Ceph - Bug #39625 (Resolved): ceph daemon mon.a config set mon_health_to_clog false cause leader ...
https://tracker.ceph.com/issues/39625
2019-05-08T06:04:12Z
huang jun
hjwsm1989@gmail.com
<p>As the subject discribed, use 'ceph daemon mon.a config set mon_health_to_clog false' <br />will cause leader mon (mon.a) failed assert:<br /><pre><code class="c syntaxhl"><span class="CodeRay"><span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">08</span> <span class="integer">13</span>:<span class="integer">53</span>:<span class="integer">3</span><span class="float">9</span><span class="float">.360</span> <span class="float">7f</span>f36eda3700 <span class="integer">10</span> mon.a<span class="error">@</span><span class="integer">0</span>(leader) e1 handle_conf_change mon_health_to_clog
<span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">08</span> <span class="integer">13</span>:<span class="integer">53</span>:<span class="integer">3</span><span class="float">9</span><span class="float">.389</span> <span class="float">7f</span>f36eda3700 -<span class="integer">1</span> /usr/src/ceph/src/common/Timer.cc: In function <span class="char">'b</span>ool SafeTimer::cancel_event(Context*)<span class="char">' </span>thread <span class="float">7f</span>f36eda3700 time <span class="integer">2019</span>-<span class="octal">05</span>-<span class="integer">08</span> <span class="integer">13</span>:<span class="integer">53</span>:<span class="integer">3</span><span class="float">9</span><span class="float">.360803</span>
/usr/src/ceph/src/common/Timer.cc: <span class="integer">153</span>: FAILED ceph_assert(lock.is_locked())
ceph version <span class="integer">1</span><span class="float">4</span><span class="float">.2</span><span class="float">.0</span>-<span class="integer">84</span>-g63ccc22 (<span class="integer">63</span>ccc22b2f38b56daaae28eb2906914840a714d9) nautilus (stable)
<span class="integer">1</span>: (ceph::__ceph_assert_fail(<span class="predefined-type">char</span> <span class="directive">const</span>*, <span class="predefined-type">char</span> <span class="directive">const</span>*, <span class="predefined-type">int</span>, <span class="predefined-type">char</span> <span class="directive">const</span>*)+<span class="hex">0x1aa</span>) [<span class="hex">0x7ff3787992a6</span>]
<span class="integer">2</span>: (()+<span class="hex">0x1295528</span>) [<span class="hex">0x7ff378799528</span>]
<span class="integer">3</span>: (SafeTimer::cancel_event(Context*)+<span class="hex">0x4c</span>) [<span class="hex">0x7ff37874a836</span>]
<span class="integer">4</span>: (Monitor::health_tick_stop()+<span class="hex">0x10c</span>) [<span class="hex">0x7ff382919ca2</span>]
<span class="integer">5</span>: (Monitor::health_events_cleanup()+<span class="hex">0x18</span>) [<span class="hex">0x7ff38291a3d8</span>]
<span class="integer">6</span>: (Monitor::health_to_clog_update_conf(std::set<std::string, std::less<std::string>, std::allocator<std::string> > const&)+<span class="hex">0x165</span>) [<span class="hex">0x7ff38291a58d</span>]
<span class="integer">7</span>: (Monitor::handle_conf_change(ConfigProxy const&, std::set<std::string, std::less<std::string>, std::allocator<std::string> > const&)+<span class="hex">0x86e</span>) [<span class="hex">0x7ff382902780</span>]
<span class="integer">8</span>: (ConfigProxy::call_observers(std::map<ceph::md_config_obs_impl<ConfigProxy>*, std::set<std::string, std::less<std::string>, std::allocator<std::string> >, std::less<ceph::md_config_obs_impl<ConfigProxy>*>, std::allocator<std::pair<ceph::md_config_obs_impl<ConfigProxy>* <span class="directive">const</span>, std::set<std::string, std::less<std::string>, std::allocator<std::string> > > > >&)+<span class="hex">0xb3</span>) [<span class="hex">0x7ff3787bd17f</span>]
<span class="integer">9</span>: (ConfigProxy::apply_changes(std::ostream*)+<span class="hex">0xad</span>) [<span class="hex">0x7ff3787bebc7</span>]
<span class="integer">10</span>: (CephContext::do_command(std::basic_string_view<<span class="predefined-type">char</span>, std::char_traits<<span class="predefined-type">char</span>> >, std::map<std::string, boost::variant<std::string, <span class="predefined-type">bool</span>, <span class="predefined-type">long</span>, <span class="predefined-type">double</span>, std::vector<std::string, std::allocator<std::string> >, std::vector<<span class="predefined-type">long</span>, std::allocator<<span class="predefined-type">long</span>> >, std::vector<<span class="predefined-type">double</span>, std::allocator<<span class="predefined-type">double</span>> > >, std::less<<span class="directive">void</span>>, std::allocator<std::pair<std::string <span class="directive">const</span>, boost::variant<std::string, <span class="predefined-type">bool</span>, <span class="predefined-type">long</span>, <span class="predefined-type">double</span>, std::vector<std::string, std::allocator<std::string> >, std::vector<<span class="predefined-type">long</span>, std::allocator<<span class="predefined-type">long</span>> >, std::vector<<span class="predefined-type">double</span>, std::allocator<<span class="predefined-type">double</span>> > > > > > const&, std::basic_string_view<<span class="predefined-type">char</span>, std::char_traits<<span class="predefined-type">char</span>> >, ceph::buffer::v14_2_0::list*)+<span class="hex">0x162f</span>) [<span class="hex">0x7ff3787b7175</span>]
<span class="integer">11</span>: (CephContextHook::call(std::basic_string_view<<span class="predefined-type">char</span>, std::char_traits<<span class="predefined-type">char</span>> >, std::map<std::string, boost::variant<std::string, <span class="predefined-type">bool</span>, <span class="predefined-type">long</span>, <span class="predefined-type">double</span>, std::vector<std::string, std::allocator<std::string> >, std::vector<<span class="predefined-type">long</span>, std::allocator<<span class="predefined-type">long</span>> >, std::vector<<span class="predefined-type">double</span>, std::allocator<<span class="predefined-type">double</span>> > >, std::less<<span class="directive">void</span>>, std::allocator<std::pair<std::string <span class="directive">const</span>, boost::variant<std::string, <span class="predefined-type">bool</span>, <span class="predefined-type">long</span>, <span class="predefined-type">double</span>, std::vector<std::string, std::allocator<std::string> >, std::vector<<span class="predefined-type">long</span>, std::allocator<<span class="predefined-type">long</span>> >, std::vector<<span class="predefined-type">double</span>, std::allocator<<span class="predefined-type">double</span>> > > > > > const&, std::basic_string_view<<span class="predefined-type">char</span>, std::char_traits<<span class="predefined-type">char</span>> >, ceph::buffer::v14_2_0::list&)+<span class="hex">0x5b</span>) [<span class="hex">0x7ff3787c0999</span>]
<span class="integer">12</span>: (AdminSocket::do_accept()+<span class="hex">0xfa2</span>) [<span class="hex">0x7ff37878d578</span>]
<span class="integer">13</span>: (AdminSocket::entry()+<span class="hex">0x2d7</span>) [<span class="hex">0x7ff37878bfef</span>]
<span class="integer">14</span>: (_ZSt13__invoke_implIvM11AdminSocketDoFvvEPS0_JEET_St21__invoke_memfun_derefOT0_OT1_DpOT2_()+<span class="hex">0x67</span>) [<span class="hex">0x7ff37879416b</span>]
<span class="integer">15</span>: (_ZSt8__invokeIM11AdminSocketDoFvvEJPS0_EENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_()+<span class="hex">0x3f</span>) [<span class="hex">0x7ff378792ef0</span>]
<span class="integer">16</span>: (_ZNSt6thread8_InvokerISt5tupleIJM11AdminSocketDoFvvEPS2_EEE9_M_invokeIJLm0ELm1EEEEDTcl8__invokespcl10_S_declvalIXT_EEEEESt12_Index_tupleIJXspT_EEE()+<span class="hex">0x43</span>) [<span class="hex">0x7ff378797c21</span>]
<span class="integer">17</span>: (_ZNSt6thread8_InvokerISt5tupleIJM11AdminSocketDoFvvEPS2_EEEclEv()+<span class="hex">0x1d</span>) [<span class="hex">0x7ff378797bd7</span>]
<span class="integer">18</span>: (_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJM11AdminSocketDoFvvEPS3_EEEEE6_M_runEv()+<span class="hex">0x1c</span>) [<span class="hex">0x7ff378797bb6</span>]
<span class="integer">19</span>: (()+<span class="hex">0x1b031ef</span>) [<span class="hex">0x7ff3790071ef</span>]
<span class="integer">20</span>: (()+<span class="hex">0x7e25</span>) [<span class="hex">0x7ff374283e25</span>]
<span class="integer">21</span>: (clone()+<span class="hex">0x6d</span>) [<span class="hex">0x7ff37314cbad</span>]
</span></code></pre></p>
<p>The reason maybe that:<br />It need to hold mon lock when doing handle_conf_change for 'mon_health_to_clog*' and 'mon_scrub_interval' changes,<br />bc those configure items need to reset the timer in leader mon, which need the mon lock.</p>
RADOS - Bug #38826 (Resolved): upmap broken the crush rule
https://tracker.ceph.com/issues/38826
2019-03-20T08:24:19Z
huang jun
hjwsm1989@gmail.com
<p>I setup a cluster and want to specify the primary osds through crush rule.<br />Here is the test script<br /><pre><code class="text syntaxhl"><span class="CodeRay">#!/bin/sh
OSDMAPTOOL=./bin/osdmaptool
CRUSHTOOL=./bin/crushtool
$OSDMAPTOOL --createsimple 240 osdmap --clobber --with-default-pool --mark-up-in --osd_pool_default_size=4
$CRUSHTOOL --build --num_osds 240 host straw2 20 rack straw2 3 datacenter straw2 2 root straw2 2 -o crush
$CRUSHTOOL -d crush -o crush.txt
# set crush rule to choose DC-A first, so primary osd are in DC-A
sed -i 's/step take root0/step take datacenter0/g' crush.txt
sed -i 's/step chooseleaf firstn 0 type host/step chooseleaf firstn 2 type host/g' crush.txt
sed -i 's/step emit/step emit\nstep take datacenter1\n step chooseleaf firstn 2 type host\n step emit/g' crush.txt
$CRUSHTOOL -c crush.txt -o crush
$OSDMAPTOOL osdmap --import-crush crush
# use upmap
$OSDMAPTOOL osdmap --upmap-deviation 0.01 --upmap-max 10000 --upmap-pool rbd --upmap result.sh --debug_osd=20 --debug_crush=20 --upmap-save /dev/null > /tmp/upmap.txt 2>&1
$OSDMAPTOOL osdmap --test-map-pgs-dump-all --pool 1 > dc-a.log
exit 0
</span></code></pre><br />The result should be that no primary pg on osd.120~osd.239, but we got this in dc-a.log<br /><pre><code class="text syntaxhl"><span class="CodeRay">osd.160 257 0 0 1 1
osd.161 256 1 1 1 1
osd.162 256 1 1 1 1
osd.163 256 1 1 1 1
</span></code></pre></p>
<p>In the dc-a.log there are many pgs choose 3 hosts in one DC and 1 host in another, <br />like:<br /><pre><code class="text syntaxhl"><span class="CodeRay">1.1585 raw ([57,115,202,145], p57) up ([161,115,202,145], p161) acting ([161,115,202,145], p161)
</span></code></pre></p>
<p>that is not the expected result, we want every pg got 2 hosts in each DC after upmap.</p>
Ceph - Bug #38492 (Resolved): src/test/cls_hello/test_cls_hello.cc:95: Failure
https://tracker.ceph.com/issues/38492
2019-02-27T01:31:28Z
huang jun
hjwsm1989@gmail.com
<p>The cls_hello test failed:<br />2019-02-11T14:49:43.421 INFO:tasks.workunit.client.0.mira031.stdout:[ RUN ] ClsHello.WriteReturnData <br />2019-02-11T14:49:44.198 INFO:tasks.workunit.client.0.mira031.stdout: src/test/cls_hello/test_cls_hello.cc:95: Failure <br />2019-02-11T14:49:44.199 INFO:tasks.workunit.client.0.mira031.stdout: Expected: std::string("too much input data!") <br />2019-02-11T14:49:44.199 INFO:tasks.workunit.client.0.mira031.stdout: Which is: "too much input data!" <br />2019-02-11T14:49:44.199 INFO:tasks.workunit.client.0.mira031.stdout:To be equal to: std::string(out.c_str(), out.length()) <br />2019-02-11T14:49:44.200 INFO:tasks.workunit.client.0.mira031.stdout: Which is: "" <br />2019-02-11T14:49:44.201 INFO:tasks.workunit.client.0.mira031.stdout:[ FAILED ] ClsHello.WriteReturnData (781 ms)</p>
<p>From the log, the steps are:<br />1. client send cls_hello. writes_dont_return_data op to osd<br />2. when osd handling it,fount the client's osdmap is older one, and then share osdmap to that client, and continue to handle the op<br />3.client resend the cls_hello. writes_dont_return_data op after recevie the newer osdmap<br />4.osd found the RETRY op is a dup op,and put it into waiting_for_ondisk list,after the origin op done, it will reply the ops in waiting_for_ondisk,<br />use reply_op_error,but it clears the ops outdata,so the client get the empty outbl that fail the case.</p>
RADOS - Bug #36412 (Closed): ceph-objectstore-tool import after pg splits which will lost objects
https://tracker.ceph.com/issues/36412
2018-10-12T02:57:34Z
huang jun
hjwsm1989@gmail.com
<p>Hi, i have a test cluster, doing the follow steps, the pool is erasure k:m=3:1<br />step 1: export pg 2.f from osd.2, origin pg 2.f stored on [0,5,2,1]<br />2018-10-09T18:35:49.560 INFO:teuthology.orchestra.run.mira036.stderr:Exporting 2.fs2 info 2.fs2( v 211'52 (32'0,211'52] local-lis/les=77/78 n=9 ec=28/28 lis/c 77/28 les/c/f 78/29/0 77/77/28)<br />2018-10-09T18:35:49.564 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:4d#<br />2018-10-09T18:35:49.569 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:6d#<br />2018-10-09T18:35:49.574 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:head#<br />2018-10-09T18:35:49.583 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:1c#<br />2018-10-09T18:35:49.588 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:47#<br />2018-10-09T18:35:49.592 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:69#<br />2018-10-09T18:35:49.605 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:6d#<br />2018-10-09T18:35:49.610 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:82#<br />2018-10-09T18:35:49.620 INFO:teuthology.orchestra.run.mira036.stderr:Read 2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:head#<br />2018-10-09T18:35:49.625 INFO:teuthology.orchestra.run.mira036.stderr:Export successful</p>
<p>step 2: set pool pg_num from 16 to 36, which will casue pg 2.f split to 2.1f</p>
<p>step 3: stop osd.3 and import is to osd.3<br />2018-10-09T18:36:43.470 INFO:teuthology.orchestra.run.mira036.stdout:Importing pgid 2.fs2<br />2018-10-09T18:36:43.522 INFO:tasks.ceph.osd.6.mira031.stderr:2018-10-09 18:38:11.001 7f6a10212700 -1 received signal: Hangup from PID: 22213 task name: /usr/bin/python /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 UID: 0<br />2018-10-09T18:36:43.566 INFO:teuthology.orchestra.run.mira036.stdout:Write 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:4d#<br />2018-10-09T18:36:43.574 INFO:teuthology.orchestra.run.mira031:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops'<br />2018-10-09T18:36:43.576 INFO:teuthology.orchestra.run.mira036.stdout:Write 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:6d#<br />2018-10-09T18:36:43.584 INFO:teuthology.orchestra.run.mira036.stdout:Write 2#2:f79c6653:::rb.radosmodel.mira036.xsky.com23787-19 ooo:head#<br />2018-10-09T18:36:43.584 INFO:teuthology.orchestra.run.mira036.stdout:snapset 6d=[6d,69,66,64,62,60,58,55,50,4d,4a]:
{4d=[4d,4b,4a,49,47,46,43],6d=[6d,69,66,64,62,60,58,55,50]}<br />2018-10-09T18:36:43.606 INFO:teuthology.orchestra.run.mira036.stdout:write_pg epoch 311 info 2.fs2( v 211'52 (32'0,211'52] local-lis/les=77/78 n=9 ec=28/28 lis/c 77/28 les/c/f 78/29/0 219/295/28)<br />2018-10-09T18:36:43.614 INFO:teuthology.orchestra.run.mira036.stdout:Import successful</p>
<p>2018-10-09T18:36:52.835 INFO:teuthology.orchestra.run.mira036.stderr:WARNING: Split occurred, some objects may be ignored<br />2018-10-09T18:36:52.835 INFO:teuthology.orchestra.run.mira036.stderr:check_new_interval interval(77-218 up [0,5,2,2147483647](0) acting [0,5,2,2147483647](0)) up_thru 209 up_from 15 last_epoch_clean 29 interval(77-218 up [0,5,2,2147483647](0) acting [0,5,2,2147483647](0) maybe_went_rw) : primary up 15-209 includes interval<br />2018-10-09T18:36:52.835 INFO:teuthology.orchestra.run.mira036.stderr:check_new_interval interval(219-294 up [0,5,2147483647,2147483647](0) acting [0,5,2147483647,2147483647](0)) : acting set is too small<br />2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:1c#' which belongs in pg 2.1f<br />2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:47#' which belongs in pg 2.1f<br />2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:69#' which belongs in pg 2.1f<br />2018-10-09T18:36:52.836 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:6d#' which belongs in pg 2.1f<br />2018-10-09T18:36:52.837 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:82#' which belongs in pg 2.1f<br />2018-10-09T18:36:52.837 INFO:teuthology.orchestra.run.mira036.stderr:Skipping object '2#2:f9c3f4f0:::rb.radosmodel.mira036.xsky.com23787-13 ooo:head#' which belongs in pg 2.1f<br />2018-10-09T18:36:52.837 INFO:teuthology.orchestra.run.mira036:Running: 'rm -f /home/ubuntu/cephtest/ceph.data/exp.2.fs2.2'</p>
<p>when pg 2.1f peering, it lost objects prefix with "rb.radosmodel.mira036.xsky.com23787-13 ooo "</p>
<p>I think the problem is, when we import pg 2.f to osd.3, we should also import these objects belongs to 2.1f bc there isn't enough acting set in the interval [219-294].<br />If we just skip these objects, it will not found on osd.3, osd.2, and osd.1.</p>
Ceph - Bug #24005 (Closed): pg-upmap can break the crush rule in some cases
https://tracker.ceph.com/issues/24005
2018-05-03T23:16:10Z
huang jun
hjwsm1989@gmail.com
<p>The test script is<br /><pre>
./bin/init-ceph stop
killall ceph-mon ceph-osd
killall ceph-mon ceph-osd
OSD=9 MON=1 MGR=0 MDS=0 ../src/vstart.sh -X -n
./bin/ceph osd crush add-bucket test root
./bin/ceph osd crush add-bucket huangjun-1 host
./bin/ceph osd crush add-bucket huangjun-2 host
./bin/ceph osd crush add-bucket huangjun-3 host
./bin/ceph osd crush move huangjun-1 root=test
./bin/ceph osd crush move huangjun-2 root=test
./bin/ceph osd crush move huangjun-3 root=test
./bin/ceph osd crush add osd.0 1.0 host=huangjun-1
./bin/ceph osd crush add osd.1 1.0 host=huangjun-1
./bin/ceph osd crush add osd.2 1.0 host=huangjun-1
./bin/ceph osd crush add osd.3 1.0 host=huangjun-2
./bin/ceph osd crush add osd.4 1.0 host=huangjun-2
./bin/ceph osd crush add osd.5 1.0 host=huangjun-2
./bin/ceph osd crush add osd.7 1.0 host=huangjun-3
./bin/ceph osd crush add osd.6 1.0 host=huangjun-3
./bin/ceph osd crush add osd.8 1.0 host=huangjun-3
#unlink all osds from default crush bucket
for i in `seq 0 8`; do
./bin/ceph osd crush unlink osd.$i huangjun
done
./bin/ceph osd erasure-code-profile set test k=4 m=2 crush-failure-domain=osd
./bin/ceph osd getcrushmap -o crush
./bin/crushtool -d crush -o crush.txt
echo "
rule test {
id 1
type erasure
min_size 1
max_size 10
step take huangjun-1
step chooseleaf indep 2 type osd
step emit
step take huangjun-2
step chooseleaf indep 2 type osd
step emit
step take huangjun-3
step chooseleaf indep 2 type osd
step emit
}
" >> crush.txt
./bin/crushtool -c crush.txt -o crush.new
./bin/ceph osd setcrushmap -i crush.new
./bin/ceph osd pool create test 256 256 erasure test test
./bin/ceph osd set-require-min-compat-client luminous
</pre><br />The cluster topology is<br /><pre>
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-5 9.00000 root test
-7 3.00000 host huangjun-1
0 hdd 1.00000 osd.0 up 1.00000 1.00000
1 hdd 1.00000 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
-8 3.00000 host huangjun-2
3 hdd 1.00000 osd.3 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 1.00000 1.00000
-9 3.00000 host huangjun-3
6 hdd 1.00000 osd.6 up 1.00000 1.00000
7 hdd 1.00000 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
-1 0 root default
-2 0 host huangjun
</pre></p>
<p>I choose pg 1.1 to test, the origin map is<br /><pre>
osdmap e50 pg 1.1 (1.1) -> up [2,0,5,4,7,8] acting [2,0,5,4,7,8]
</pre></p>
<p>Then i use 'ceph osd pg-upmap-items' to set upmaps<br /><pre>
./bin/ceph osd pg-upmap-items 1.1 2 1 4 2
</pre><br />and check the pg1.1 map again<br /><pre>
[root@huangjun /usr/src/ceph-int/build]$ ./bin/ceph pg map 1.1
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
osdmap e52 pg 1.1 (1.1) -> up [1,0,5,2,7,8] acting [1,0,5,2,7,8]
</pre></p>
<p>My question is:<br />1. the crush rule said we take 2 osds from each host, but after this upmap, <br />we can see, pg1.1 have 3 osds in host huangjun-1, 1 osd in host huangjun-2, <br />which i think break the crush rule, and if the host huangjun-1 is halt, there<br />can be 3 osds down, that will result pg1.1 in down state.</p>
RADOS - Bug #23921 (Resolved): pg-upmap cannot balance in some case
https://tracker.ceph.com/issues/23921
2018-04-28T04:04:22Z
huang jun
hjwsm1989@gmail.com
<p>I have a cluster with 21 osds, cluster topology is<br /><pre>
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-5 21.00000 root test
-7 11.00000 datacenter dc-1
-9 11.00000 rack rack-1
-11 5.00000 host host-1
5 hdd 1.00000 osd.5 up 0.50000 1.00000
6 hdd 1.00000 osd.6 up 1.00000 1.00000
7 hdd 1.00000 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
9 hdd 1.00000 osd.9 up 1.00000 1.00000
-12 2.00000 host host-2
16 hdd 1.00000 osd.16 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
-13 2.00000 host host-3
15 hdd 1.00000 osd.15 up 1.00000 1.00000
18 hdd 1.00000 osd.18 up 1.00000 1.00000
-14 2.00000 host host-4
19 hdd 1.00000 osd.19 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
-8 10.00000 datacenter dc-2
-10 10.00000 rack rack-2
-15 5.00000 host host-5
10 hdd 1.00000 osd.10 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
-16 5.00000 host host-6
0 hdd 1.00000 osd.0 up 1.00000 1.00000
1 hdd 1.00000 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
-1 21.00000 root default
-2 21.00000 host huangjun
0 hdd 1.00000 osd.0 up 1.00000 1.00000
1 hdd 1.00000 osd.1 up 1.00000 1.00000
2 hdd 1.00000 osd.2 up 1.00000 1.00000
3 hdd 1.00000 osd.3 up 1.00000 1.00000
4 hdd 1.00000 osd.4 up 1.00000 1.00000
5 hdd 1.00000 osd.5 up 0.50000 1.00000
6 hdd 1.00000 osd.6 up 1.00000 1.00000
7 hdd 1.00000 osd.7 up 1.00000 1.00000
8 hdd 1.00000 osd.8 up 1.00000 1.00000
9 hdd 1.00000 osd.9 up 1.00000 1.00000
10 hdd 1.00000 osd.10 up 1.00000 1.00000
11 hdd 1.00000 osd.11 up 1.00000 1.00000
12 hdd 1.00000 osd.12 up 1.00000 1.00000
13 hdd 1.00000 osd.13 up 1.00000 1.00000
14 hdd 1.00000 osd.14 up 1.00000 1.00000
15 hdd 1.00000 osd.15 up 1.00000 1.00000
16 hdd 1.00000 osd.16 up 1.00000 1.00000
17 hdd 1.00000 osd.17 up 1.00000 1.00000
18 hdd 1.00000 osd.18 up 1.00000 1.00000
19 hdd 1.00000 osd.19 up 1.00000 1.00000
20 hdd 1.00000 osd.20 up 1.00000 1.00000
</pre></p>
<p>create a pool with 1024pgs, 2 replicated size.<br />after remap, it shows no change<br />ceph osd df<br /><pre>
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS
5 hdd 1.00000 0.50000 981M 34176k 948M 3.40 1.00 40
6 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 99
7 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 109
8 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 121
9 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 95
16 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 82
17 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 91
15 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 95
18 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 93
19 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 100
20 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 99
10 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 85
11 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 94
12 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 81
13 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 118
14 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 102
0 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 107
1 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 113
2 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 106
3 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 110
4 hdd 1.00000 1.00000 981M 34176k 948M 3.40 1.00 108
</pre></p>
<p>I check the log<br /><pre>
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.0 weight 0.1 pgs 107
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.1 weight 0.1 pgs 113
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.2 weight 0.1 pgs 106
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.3 weight 0.1 pgs 110
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.4 weight 0.1 pgs 108
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.5 weight 0.0454545 pgs 40
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.6 weight 0.0909091 pgs 99
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.7 weight 0.0909091 pgs 109
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.8 weight 0.0909091 pgs 121
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.9 weight 0.0909091 pgs 95
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.10 weight 0.1 pgs 85
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.11 weight 0.1 pgs 94
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.12 weight 0.1 pgs 81
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.13 weight 0.1 pgs 118
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.14 weight 0.1 pgs 102
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.15 weight 0.0909091 pgs 95
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.16 weight 0.0909091 pgs 82
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.17 weight 0.0909091 pgs 91
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.18 weight 0.0909091 pgs 93
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.19 weight 0.0909091 pgs 100
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.20 weight 0.0909091 pgs 99
2018-04-28 11:50:39.661 7f87a8cfd700 10 osd_weight_total 1.95455
2018-04-28 11:50:39.661 7f87a8cfd700 10 pgs_per_weight 1047.81
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.0 pgs 107 target 104.781 deviation 2.21863
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.1 pgs 113 target 104.781 deviation 8.21863
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.2 pgs 106 target 104.781 deviation 1.21863
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.3 pgs 110 target 104.781 deviation 5.21863
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.4 pgs 108 target 104.781 deviation 3.21863
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.5 pgs 40 target 47.6279 deviation -7.6279
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.6 pgs 99 target 95.2558 deviation 3.7442
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.7 pgs 109 target 95.2558 deviation 13.7442
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.8 pgs 121 target 95.2558 deviation 25.7442
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.9 pgs 95 target 95.2558 deviation -0.255798
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.10 pgs 85 target 104.781 deviation -19.7814
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.11 pgs 94 target 104.781 deviation -10.7814
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.12 pgs 81 target 104.781 deviation -23.7814
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.13 pgs 118 target 104.781 deviation 13.2186
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.14 pgs 102 target 104.781 deviation -2.78137
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.15 pgs 95 target 95.2558 deviation -0.255798
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.16 pgs 82 target 95.2558 deviation -13.2558
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.17 pgs 91 target 95.2558 deviation -4.2558
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.18 pgs 93 target 95.2558 deviation -2.2558
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.19 pgs 100 target 95.2558 deviation 4.7442
2018-04-28 11:50:39.661 7f87a8cfd700 20 osd.20 pgs 99 target 95.2558 deviation 3.7442
2018-04-28 11:50:39.661 7f87a8cfd700 10 total_deviation 170.065 overfull 0,1,2,3,4,6,7,8,13,19,20 underfull [12,10,16,11,5,17,14,18]
2018-04-28 11:50:39.661 7f87a8cfd700 10 osd.8 move 25
2018-04-28 11:50:39.661 7f87a8cfd700 10 trying 1.0
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_pg_upmap
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule ruleno 1 numrep 2 overfull 0,1,2,3,4,6,7,8,13,19,20 underfull [12,10,16,11,5,17,14,18] orig [8,13]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule step 0 w []
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule take [-9]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule step 1 w [-9]
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack stack [1,1,0,1] orig [8,13] at 8 pw [-9]
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack cumulative_fanout [1,1]
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 12 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 10 type 1 is -15
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 16 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 11 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 5 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 17 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 14 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 18 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 20 _choose_type_stack underfull_buckets [-15,-2]
2018-04-28 11:50:39.661 7f87a8cfd700 10 level 0: type 1 fanout 1 cumulative 1 w [-9]
2018-04-28 11:50:39.661 7f87a8cfd700 10 from -9
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack from 13 got -2 of type 1 over leaves 8
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack w <- [-2] was [-9]
2018-04-28 11:50:39.661 7f87a8cfd700 10 level 1: type 0 fanout 1 cumulative 1 w [-2]
2018-04-28 11:50:39.661 7f87a8cfd700 10 from -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack pos 0 was 8 considering 12
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack pos 0 replace 8 -> 12
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack w <- [12] was [-2]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule step 2 w [12]
2018-04-28 11:50:39.661 7f87a8cfd700 10 emit [12]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule step 3 w []
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule take [-10]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule step 4 w [-10]
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack stack [1,1,0,1] orig [8,13] at 13 pw [-10]
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack cumulative_fanout [1,1]
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 12 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 10 type 1 is -15
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 16 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 11 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 5 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 17 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 14 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack underfull 18 type 1 is -2
2018-04-28 11:50:39.661 7f87a8cfd700 20 _choose_type_stack underfull_buckets [-15,-2]
2018-04-28 11:50:39.661 7f87a8cfd700 10 level 0: type 1 fanout 1 cumulative 1 w [-10]
2018-04-28 11:50:39.661 7f87a8cfd700 10 from -10
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack from -1142358840 got -2 of type 1 over leaves 13
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack w <- [-2] was [-10]
2018-04-28 11:50:39.661 7f87a8cfd700 10 level 1: type 0 fanout 1 cumulative 1 w [-2]
2018-04-28 11:50:39.661 7f87a8cfd700 10 from -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack pos 0 was 13 considering 12
2018-04-28 11:50:39.661 7f87a8cfd700 20 _choose_type_stack in used 12
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack pos 0 was 13 considering 10
2018-04-28 11:50:39.661 7f87a8cfd700 20 _choose_type_stack not in subtree -2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack pos 0 was 13 considering 16
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack pos 0 replace 13 -> 16
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack end of orig, break 1
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack end of orig, break 2
2018-04-28 11:50:39.661 7f87a8cfd700 10 _choose_type_stack w <- [16] was [-2]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_remap_rule step 5 w [16]
2018-04-28 11:50:39.661 7f87a8cfd700 10 emit [16]
2018-04-28 11:50:39.661 7f87a8cfd700 10 try_pg_upmap orig [8,13], out [12,16]
2018-04-28 11:50:39.661 7f87a8cfd700 10 1.0 [8,13] -> [12,16]
2018-04-28 11:50:39.661 7f87a8cfd700 10 1.0 pg_upmap_items [8,12,13,16]
2018-04-28 11:50:39.716 7f87ab502700 10 maybe_remove_pg_upmaps
2018-04-28 11:50:39.716 7f87ab502700 10 maybe_remove_pg_upmaps pg 1.0 crush-rule-id 1 weight_map {0=0.1,1=0.1,2=0.1,3=0.1,4=0.1,5=0.0909091,6=0.0909091,7=0.0909091,8=0.0909091,9=0
.0909091,10=0.1,11=0.1,12=0.1,13=0.1,14=0.1,15=0.0909091,16=0.0909091,17=0.0909091,18=0.0909091,19=0.0909091,20=0.0909091} failure-domain-type 1
2018-04-28 11:50:39.716 7f87ab502700 10 maybe_remove_pg_upmaps pg 1.0 osd 12 parent -2
2018-04-28 11:50:39.716 7f87ab502700 10 maybe_remove_pg_upmaps pg 1.0 osd 16 parent -2
2018-04-28 11:50:39.717 7f87ab502700 10 maybe_remove_pg_upmaps cancel invalid pending pg_upmap_items entry 1.0->[8,12,13,16]
</pre></p>
<p>PG 1.0 remap from 8,13 to 12,16<br />and in root bucket test, the osd.12 and osd.16 are not in the same host, <br />but get the same parent -2, that it is werid. so it will clear the upmap items.<br />because osd.12 and osd.16 in the same host huangjun, but which is not used for pool 'test'<br /><pre>
pool 1 'test' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 104 lfor 0/102 flags hashpspool stripe_width 0 async_recovery_max_updates 200 osd_full_ratio 0.9
</pre><br />crush rule dump is<br /><pre>
[
{
"rule_id": 0,
"rule_name": "replicated_rule",
"ruleset": 0,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -1,
"item_name": "default"
},
{
"op": "choose_firstn",
"num": 0,
"type": "osd"
},
{
"op": "emit"
}
]
},
{
"rule_id": 1,
"rule_name": "test",
"ruleset": 1,
"type": 1,
"min_size": 1,
"max_size": 10,
"steps": [
{
"op": "take",
"item": -9,
"item_name": "rack-1"
},
{
"op": "chooseleaf_firstn",
"num": 1,
"type": "host"
},
{
"op": "emit"
},
{
"op": "take",
"item": -10,
"item_name": "rack-2"
},
{
"op": "chooseleaf_firstn",
"num": 1,
"type": "host"
},
{
"op": "emit"
}
]
}
]
</pre></p>
Dashboard - Bug #23895 (Can't reproduce): cmake failed in compiling dashboard
https://tracker.ceph.com/issues/23895
2018-04-27T01:05:21Z
huang jun
hjwsm1989@gmail.com
<pre>
dashboard frontend is being created
> ceph-dashboard@0.0.0 build /usr/src/ceph/src/pybind/mgr/dashboard/frontend
> ng build "--prod"
You seem to not be depending on "@angular/core". This is an error.
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! ceph-dashboard@0.0.0 build: `ng build "--prod"`
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the ceph-dashboard@0.0.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2018-04-27T01_02_15_946Z-debug.log
make[2]: *** [../src/pybind/mgr/dashboard/frontend/dist] Error 2
make[1]: *** [src/pybind/mgr/dashboard/CMakeFiles/mgr-dashboard-frontend-build.dir/all] Error 2
make: *** [all] Error 2
</pre>
RADOS - Bug #23878 (Resolved): assert on pg upmap
https://tracker.ceph.com/issues/23878
2018-04-26T04:31:51Z
huang jun
hjwsm1989@gmail.com
<p>I use the follow script to test upmap<br /><pre><code class="text syntaxhl"><span class="CodeRay">./bin/init-ceph stop
killall ceph-mon ceph-osd
killall ceph-mon ceph-osd
OSD=9 MON=1 MGR=1 MDS=0 ../src/vstart.sh -X -n
./bin/ceph osd crush add-bucket test root
./bin/ceph osd crush add-bucket huangjun-1 host
./bin/ceph osd crush add-bucket huangjun-2 host
./bin/ceph osd crush add-bucket huangjun-3 host
./bin/ceph osd crush move huangjun-1 root=test
./bin/ceph osd crush move huangjun-2 root=test
./bin/ceph osd crush move huangjun-3 root=test
./bin/ceph osd crush add osd.0 1.0 host=huangjun-1
./bin/ceph osd crush add osd.1 1.0 host=huangjun-1
./bin/ceph osd crush add osd.2 1.0 host=huangjun-1
./bin/ceph osd crush add osd.3 1.0 host=huangjun-2
./bin/ceph osd crush add osd.4 1.0 host=huangjun-2
./bin/ceph osd crush add osd.5 1.0 host=huangjun-2
./bin/ceph osd crush add osd.7 1.0 host=huangjun-3
./bin/ceph osd crush add osd.6 1.0 host=huangjun-3
./bin/ceph osd erasure-code-profile set test k=4 m=2 crush-failure-domain=osd
./bin/ceph osd getcrushmap -o crush
./bin/crushtool -d crush -o crush.txt
echo "
rule test {
id 1
type erasure
min_size 1
max_size 10
step take huangjun-1
step chooseleaf indep 2 type osd
step emit
step take huangjun-2
step chooseleaf indep 2 type osd
step emit
step take huangjun-3
step chooseleaf indep 2 type osd
step emit
}
" >> crush.txt
./bin/crushtool -c crush.txt -o crush.new
./bin/ceph osd setcrushmap -i crush.new
./bin/ceph osd pool create test 256 256 erasure test test
max_deviation=0.01
max_pg=256
pool='test'
./bin/ceph osd getmap -o om
./bin/osdmaptool om --upmap-deviation $max_deviation --upmap-max $max_pg --upmap-pool $pool --upmap result.sh
sh result.sh
rm -f result.sh
./bin/ceph osd crush unlink osd.2 huangjun-1
./bin/ceph osd getmap -o om
./bin/osdmaptool om --upmap-deviation $max_deviation --upmap-max $max_pg --upmap-pool $pool --upmap result.sh
sh result.sh
</span></code></pre></p>
<p>the test crashed with<br /><pre>
*** Caught signal (Aborted) **
in thread 7f9e999b0180 thread_name:osdmaptool
ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)
1: (()+0x21321) [0x56420eafb321]
2: (()+0xf5e0) [0x7f9e8f6755e0]
3: (gsignal()+0x37) [0x7f9e8e0671f7]
4: (abort()+0x148) [0x7f9e8e0688e8]
5: (()+0x74f47) [0x7f9e8e0a6f47]
6: (()+0x7c619) [0x7f9e8e0ae619]
7: (std::_Rb_tree<pg_t, std::pair<pg_t const, std::vector<std::pair<int, int>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<int, int> > > >, std::_Select1st<std::pair<pg_t const, std::vector<std::pair<int, int>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<int, int> > > > >, std::less<pg_t>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<pg_t const, std::vector<std::pair<int, int>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<int, int> > > > > >::_M_erase_aux(std::_Rb_tree_const_iterator<std::pair<pg_t const, std::vector<std::pair<int, int>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<int, int> > > > >)+0x76) [0x7f9e90f7a4b6]
8: (OSDMap::calc_pg_upmaps(CephContext*, float, int, std::set<long, std::less<long>, std::allocator<long> > const&, OSDMap::Incremental*)+0x1041) [0x7f9e90f6a651]
9: (main()+0x3925) [0x56420eaec385]
10: (__libc_start_main()+0xf5) [0x7f9e8e053c05]
11: (()+0x12fc0) [0x56420eaecfc0]
2018-04-26 12:32:04.556614 7f9e999b0180 -1 *** Caught signal (Aborted) **
in thread 7f9e999b0180 thread_name:osdmaptool
</pre></p>
Ceph - Bug #23877 (Duplicate): osd/OSDMap.cc: assert(target > 0)
https://tracker.ceph.com/issues/23877
2018-04-26T03:23:18Z
huang jun
hjwsm1989@gmail.com
The origin cluster toplogy is:
<ol>
<li>begin crush map<br />tunable choose_local_tries 0<br />tunable choose_local_fallback_tries 0<br />tunable choose_total_tries 150<br />tunable chooseleaf_descend_once 1<br />tunable chooseleaf_vary_r 1<br />tunable chooseleaf_stable 1<br />tunable straw_calc_version 1<br />tunable allowed_bucket_algs 54</li>
</ol>
<ol>
<li>devices<br />device 0 osd.0 class hdd<br />device 1 osd.1 class hdd<br />device 2 osd.2 class hdd<br />device 3 osd.3 class hdd<br />device 4 osd.4 class hdd<br />device 5 osd.5 class hdd<br />device 6 osd.6 class hdd<br />device 7 osd.7 class hdd<br />device 8 osd.8 class hdd</li>
</ol>
<ol>
<li>types<br />type 0 osd<br />type 1 host<br />type 2 chassis<br />type 3 rack<br />type 4 row<br />type 5 pdu<br />type 6 pod<br />type 7 room<br />type 8 datacenter<br />type 9 region<br />type 10 root</li>
</ol>
<ol>
<li>buckets<br />host huangjun-1 {<br /> id -3 # do not change unnecessarily
# weight 9.000<br /> alg straw2<br /> hash 0 # rjenkins1<br /> item osd.0 weight 1.000<br /> item osd.1 weight 1.000<br /> item osd.7 weight 1.000<br />}<br />host huangjun-2 {<br /> id -4 # do not change unnecessarily
# weight 9.000<br /> alg straw2<br /> hash 0 # rjenkins1<br /> item osd.2 weight 1.000<br /> item osd.3 weight 1.000<br /> item osd.6 weight 1.000<br />}<br />host huangjun-3 {<br /> id -5 # do not change unnecessarily
# weight 9.000<br /> alg straw2<br /> hash 0 # rjenkins1<br /> item osd.4 weight 1.000<br /> item osd.5 weight 1.000<br />}<br />host huangjun {<br /> id -2 # do not change unnecessarily<br /> id -3 class hdd # do not change unnecessarily
# weight 9.000<br /> alg straw2<br /> hash 0 # rjenkins1<br /> item osd.0 weight 1.000<br /> item osd.1 weight 1.000<br /> item osd.2 weight 1.000<br /> item osd.3 weight 1.000<br /> item osd.4 weight 1.000<br /> item osd.5 weight 1.000<br /> item osd.6 weight 1.000<br /> item osd.7 weight 1.000<br /> item osd.8 weight 1.000<br />}<br />root default {<br /> id -1 # do not change unnecessarily<br /> id -4 class hdd # do not change unnecessarily
# weight 9.000<br /> alg straw2<br /> hash 0 # rjenkins1<br /> item huangjun weight 9.000<br />}</li>
</ol>
root test {<br /> id -6 # do not change unnecessarily
# weight 9.000<br /> alg straw2<br /> hash 0 # rjenkins1<br /> item huangjun-1 weight 3.000<br /> item huangjun-2 weight 3.000<br /> item huangjun-3 weight 2.000<br />}
<ol>
<li>rules<br />rule replicated_rule {<br /> id 0<br /> type replicated<br /> min_size 1<br /> max_size 10<br /> step take default<br /> step choose firstn 0 type osd<br /> step emit<br />}</li>
</ol>
rule test_ec {<br /> id 1<br /> type erasure<br /> min_size 1<br /> max_size 10<br /> step take huangjun-1<br /> step chooseleaf indep 2 type osd<br /> step emit<br /> step take huangjun-2<br /> step chooseleaf indep 2 type osd<br /> step emit<br /> step take huangjun-3<br /> step chooseleaf indep 2 type osd<br /> step emit<br />}
<ol>
<li>end crush map</li>
</ol>
<p>we create a k:m=4:2 crush-failure-domain=osd erasure pool with 256 pgs<br />and do the remap<br />then unlink osd.6 from huangjun-2<br />./bin/ceph osd crush unlink osd.6 huangjun-2<br />and then do remap again<br />it crashed with<br /><code> -18> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.0 weight 0.333333 pgs 170<br /> -17> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.1 weight 0.333333 pgs 170<br /> -16> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.2 weight 0.5 pgs 256<br /> -15> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.3 weight 0.5 pgs 250<br /> -14> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.4 weight 0.5 pgs 256<br /> -13> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.5 weight 0.5 pgs 256<br /> -12> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.7 weight 0.333333 pgs 172<br /> -11> 2018-04-26 01:13:10.885 7f0a2e0c8700 10 osd_weight_total 3<br /> -10> 2018-04-26 01:13:10.885 7f0a2e0c8700 10 pgs_per_weight 512<br /> -9> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.0 pgs 170 target 170.667 deviation -0.666672<br /> -8> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.1 pgs 170 target 170.667 deviation -0.666672<br /> -7> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.2 pgs 256 target 256 deviation 0<br /> -6> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.3 pgs 250 target 256 deviation -6<br /> -5> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.4 pgs 256 target 256 deviation 0<br /> -4> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.5 pgs 256 target 256 deviation 0<br /> -3> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.6 pgs 6 target 0 deviation 6<br /> -2> 2018-04-26 01:13:10.885 7f0a2e0c8700 20 osd.7 pgs 172 target 170.667 deviation 1.33333<br /> -1> 2018-04-26 01:13:10.885 7f0a2e0c8700 10 total_deviation 14.6667 overfull 6,7 underfull [3]<br /> 0> 2018-04-26 01:13:10.891 7f0a2e0c8700 -1 /usr/src/ceph-int/src/osd/OSDMap.cc: In function 'int OSDMap::calc_pg_upmaps(CephContext*, float, int, const std::set<long int>&, OSDMap::Incremental*)' thread 7f0a2e0c8700 time 2018-04-26 01:13:10.886506<br />/usr/src/ceph-int/src/osd/OSDMap.cc: 4124: FAILED assert(target > 0) (null)</code></p>