Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2014-11-17T14:00:55Z
Ceph
Redmine
Ceph - Bug #10125 (Resolved): radosgw is being started as root not apache with systemd
https://tracker.ceph.com/issues/10125
2014-11-17T14:00:55Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>On RHEL 7 when radosgw is started with systemd it runs as root not apache which causes problems with the s3gw.fcgi is called by apache user.</p>
<p><a class="external" href="https://github.com/ceph/ceph/blob/master/src/init-radosgw.sysv#L88">https://github.com/ceph/ceph/blob/master/src/init-radosgw.sysv#L88</a></p>
<p>$SYSTEMD_RUN -r bash -c "ulimit -n 32768; $RADOSGW -n $name"</p>
<p>Should probably be something like:</p>
<p>$SYSTEMD_RUN --user="$user" -r bash -c "ulimit -n 32768; $RADOSGW -n $name"</p>
Ceph - Bug #9668 (Rejected): osd killed by ABRT from FAILED assert
https://tracker.ceph.com/issues/9668
2014-10-06T09:00:37Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>-----------<br />[Mon Oct 6 15:09:03 2014] init: ceph-osd (ceph/46) main process (3058) killed by ABRT signal<br />-----------</p>
<p>2014-10-06 14:04:47.314592 7f61663fd700 0 -- 10.60.1.6:6817/3058 >> 10.60.1.5:6812/2941 pipe(0x2dd2e800 sd=899 :38136 s=2 pgs=680 cs=43 l=0 c=0x1e9b6160).fault with nothing to s<br />end, going to standby<br />2014-10-06 14:32:24.417178 7f6142ded700 0 -- 10.60.1.6:6817/3058 >> 10.60.1.2:6825/7210 pipe(0x2dd2cc80 sd=418 :51395 s=2 pgs=96 cs=11 l=0 c=0xdd7d760).fault with nothing to sen<br />d, going to standby<br />2014-10-06 14:51:54.338095 7f61625d5700 0 -- 10.60.1.6:6815/3058 >> 10.60.1.30:0/1017493 pipe(0x22021900 sd=950 :6815 s=0 pgs=0 cs=0 l=1 c=0x6b13440).accept replacing existing (<br />lossy) channel (new one lossy=1)<br />2014-10-06 15:09:02.936722 7f6171339700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f6171339700 time 2014-10-06 15:09:02.918464<br />common/Thread.cc: 128: FAILED assert(ret == 0)</p>
<pre><code>ceph version 0.80.6 (f93610a4421cb670b08e974c6550ee715ac528ae)<br /> 1: (Thread::create(unsigned long)+0x8a) [0xa49d7a]<br /> 2: (Pipe::accept()+0x30ad) [0xb25fed]<br /> 3: (Pipe::reader()+0x17c0) [0xb29f00]<br /> 4: (Pipe::Reader::entry()+0xd) [0xb2cd5d]<br /> 5: (()+0x8182) [0x7f620fd30182]<br /> 6: (clone()+0x6d) [0x7f620e4a2fbd]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
<p>--- begin dump of recent events ---<br /> <del>9999> 2014-10-06 15:08:38.318318 7f61fbf50700 5 -</del> op tracker -- , seq: 5894143, time: 2014-10-06 15:08:38.318245, event: dispatched, request: osd_sub_op_reply(client.34946378<br />.0:2736589 3.3f4 750ef3f4/rbd_data.2153d3679f42670.0000000000000062/head//3 [] ondisk, result = 0) v2<br /> <del>9998> 2014-10-06 15:08:38.318333 7f61fbf50700 5 -</del> op tracker -- , seq: 5894143, time: 2014-10-06 15:08:38.318333, event: waiting_for_osdmap, request: osd_sub_op_reply(client.<br />34946378.0:2736589 3.3f4 750ef3f4/rbd_data.2153d3679f42670.0000000000000062/head//3 [] ondisk, result = 0) v2<br /> <del>9997> 2014-10-06 15:08:38.318419 7f61f5f44700 5 -</del> op tracker -- , seq: 5894143, time: 2014-10-06 15:08:38.318419, event: reached_pg, request: osd_sub_op_reply(client.34946378<br />.0:2736589 3.3f4 750ef3f4/rbd_data.2153d3679f42670.0000000000000062/head//3 [] ondisk, result = 0) v2<br /> <del>9996> 2014-10-06 15:08:38.318461 7f61f5f44700 5 -</del> op tracker -- , seq: 5894143, time: 2014-10-06 15:08:38.318460, event: started, request: osd_sub_op_reply(client.34946378.0:<br />2736589 3.3f4 750ef3f4/rbd_data.2153d3679f42670.0000000000000062/head//3 [] ondisk, result = 0) v2<br /> <del>9995> 2014-10-06 15:08:38.318722 7f61f5f44700 5 -</del> op tracker -- , seq: 5894142, time: 2014-10-06 15:08:38.318722, event: sub_op_commit_rec, request: osd_op(client.34946378.0:<br />2736589 rbd_data.2153d3679f42670.0000000000000062 [stat,set-alloc-hint object_size 8388608 write_size 8388608,write 6988288~4096] 3.750ef3f4 ack+ondisk+write e179755) v4<br /> <del>9994> 2014-10-06 15:08:38.318749 7f61f5f44700 1 -</del> 10.60.1.6:6815/3058 --> 10.60.1.31:0/1021771 -- osd_op_reply(2736589 rbd_data.2153d3679f42670.0000000000000062 [stat,set-all<br />oc-hint object_size 8388608 write_size 8388608,write 6988288~4096] v179755'3279754 uv3279754 ack = 0) v6 -- ?+0 0x2d25a300 con 0x20e79b20<br /> <del>9993> 2014-10-06 15:08:38.318820 7f61f5f44700 1 -</del> 10.60.1.6:6815/3058 --> 10.60.1.31:0/1021771 -- osd_op_reply(2736589 rbd_data.2153d3679f42670.0000000000000062 [stat,set-all<br />oc-hint object_size 8388608 write_size 8388608,write 6988288~4096] v179755'3279754 uv3279754 ondisk = 0) v6 -- ?+0 0x1c35a500 con 0x20e79b20</p>
Ceph - Bug #9570 (Rejected): osd crash in FileJournal::WriteFinisher::entry() aio
https://tracker.ceph.com/issues/9570
2014-09-22T15:55:23Z
Sheldon Mustard
sheldon.mustard@inktank.com
<a name="Workaround"></a>
<h3 >Workaround<a href="#Workaround" class="wiki-anchor">¶</a></h3>
<p>Try with a kernel newer than 3.13 - as new as the environment allows.</p>
<a name="Collect-more-information"></a>
<h3 >Collect more information<a href="#Collect-more-information" class="wiki-anchor">¶</a></h3>
<p>If the problem happens in your environment, increase the in memory logging with:</p>
<pre>
debug osd = 0/30
debug filestore = 0/30
debug ms = 0/1
debug journal = 0/30
</pre>
<p>save the core dump and add it to this issue, with the log, when it happens.</p>
<a name="Original-description"></a>
<h3 >Original description<a href="#Original-description" class="wiki-anchor">¶</a></h3>
<p>On an OSD running dumpling v0.67.10<br /><pre>
-13> 2014-09-22 16:08:54.484825 7feeb1ebd700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.484530, event: header_read, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-12> 2014-09-22 16:08:54.484842 7feeb1ebd700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.484555, event: throttled, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-11> 2014-09-22 16:08:54.484854 7feeb1ebd700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.484722, event: all_read, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-10> 2014-09-22 16:08:54.484866 7feeb1ebd700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.484819, event: dispatched, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-9> 2014-09-22 16:08:54.484878 7feeb1ebd700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.484878, event: waiting_for_osdmap, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-8> 2014-09-22 16:08:54.484966 7feead6b4700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.484966, event: reached_pg, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-7> 2014-09-22 16:08:54.485003 7feead6b4700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.485003, event: started, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-6> 2014-09-22 16:08:54.485121 7feead6b4700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.485121, event: started, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-5> 2014-09-22 16:08:54.485157 7feead6b4700 5 --OSD::tracker-- reqid: client.19451671.0:182034, seq: 81280699, time: 2014-09-22 16:08:54.485157, event: commit_queued_for_journal_write, request: osd_sub_op(client.19451671.0:182034 366.33e f4e4873e/rb.0.113a598.238e1f29.000000000e4e/head//366 [] v 2466'680337 snapset=0=[]:[] snapc=0=[]) v7
-4> 2014-09-22 16:08:54.536191 7feeaf6b8700 1 -- 10.10.10.7:6810/25820 <== osd.9 10.10.10.24:0/16694 156020 ==== osd_ping(ping e2466 stamp 2014-09-22 16:08:54.535761) v2 ==== 47+0+0 (2119288752 0 0) 0x5071a8c0 con 0x4f26d840
-3> 2014-09-22 16:08:54.536226 7feeb06ba700 1 -- 10.10.10.7:6812/25820 <== osd.9 10.10.10.24:0/16694 156020 ==== osd_ping(ping e2466 stamp 2014-09-22 16:08:54.535761) v2 ==== 47+0+0 (2119288752 0 0) 0x516b1e00 con 0x4f33d6e0
-2> 2014-09-22 16:08:54.536258 7feeaf6b8700 1 -- 10.10.10.7:6810/25820 --> 10.10.10.24:0/16694 -- osd_ping(ping_reply e2466 stamp 2014-09-22 16:08:54.535761) v2 -- ?+0 0x38f711c0 con 0x4f26d840
-1> 2014-09-22 16:08:54.536436 7feeb06ba700 1 -- 10.10.10.7:6812/25820 --> 10.10.10.24:0/16694 -- osd_ping(ping_reply e2466 stamp 2014-09-22 16:08:54.535761) v2 -- ?+0 0x4cdcdc40 con 0x4f33d6e0
0> 2014-09-22 16:08:54.558046 7feeca125700 -1 *** Caught signal (Aborted) **
in thread 7feeca125700
ceph version 0.67.10 (9d446bd416c52cd785ccf048ca67737ceafcdd7f)
1: /usr/bin/ceph-osd() [0x7fd50a]
2: (()+0xfcb0) [0x7feed27e7cb0]
3: (gsignal()+0x35) [0x7feed0f3e0d5]
4: (abort()+0x17b) [0x7feed0f4183b]
5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7feed189769d]
6: (()+0xb5846) [0x7feed1895846]
7: (()+0xb5873) [0x7feed1895873]
8: (()+0xb596e) [0x7feed189596e]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1df) [0x8c68df]
10: (FileJournal::write_finish_thread_entry()+0x6d7) [0x766c67]
11: (FileJournal::WriteFinisher::entry()+0xd) [0x69cbad]
12: (()+0x7e9a) [0x7feed27dfe9a]
13: (clone()+0x6d) [0x7feed0ffc31d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.18.log
--- end dump of recent events ---
2014-09-22 16:08:55.656515 7fd6f29d9780 0 ceph version 0.67.10 (9d446bd416c52cd785ccf048ca67737ceafcdd7f), process ceph-osd, pid 8912
2014-09-22 16:08:55.736899 7fd6f29d9780 1 filestore(/var/lib/ceph/osd/ceph-18) mount detected xfs
2014-09-22 16:08:55.736910 7fd6f29d9780 1 filestore(/var/lib/ceph/osd/ceph-18) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs
2014-09-22 16:08:55.745399 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount FIEMAP ioctl is supported and appears to work
2014-09-22 16:08:55.745427 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-09-22 16:08:55.746476 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount did NOT detect btrfs
2014-09-22 16:08:55.775236 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount syncfs(2) syscall fully supported (by glibc and kernel)
2014-09-22 16:08:55.775847 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount found snaps <>
2014-09-22 16:08:56.254811 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2014-09-22 16:09:00.301748 7fd6f29d9780 1 journal _open /var/lib/ceph/osd/ceph-18/journal fd 19: 16382951424 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-09-22 16:09:00.620119 7fd6f29d9780 1 journal _open /var/lib/ceph/osd/ceph-18/journal fd 19: 16382951424 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-09-22 16:09:00.651236 7fd6f29d9780 1 journal close /var/lib/ceph/osd/ceph-18/journal
2014-09-22 16:09:00.700334 7fd6f29d9780 1 filestore(/var/lib/ceph/osd/ceph-18) mount detected xfs
2014-09-22 16:09:00.707685 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount FIEMAP ioctl is supported and appears to work
2014-09-22 16:09:00.707707 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2014-09-22 16:09:00.708656 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount did NOT detect btrfs
2014-09-22 16:09:00.731348 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount syncfs(2) syscall fully supported (by glibc and kernel)
2014-09-22 16:09:00.731493 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount found snaps <>
2014-09-22 16:09:00.744450 7fd6f29d9780 0 filestore(/var/lib/ceph/osd/ceph-18) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2014-09-22 16:09:00.773887 7fd6f29d9780 1 journal _open /var/lib/ceph/osd/ceph-18/journal fd 20: 16382951424 bytes, block size 4096 bytes, directio = 1, aio = 1
2014-09-22 16:09:00.799478 7fd6f29d9780 1 journal _open /var/lib/ceph/osd/ceph-18/journal fd 20: 16382951424 bytes, block size 4096 bytes, directio = 1, aio = 1
</pre></p>
Ceph - Fix #9566 (Need More Info): osd: prioritize recovery of OSDs with most work to do
https://tracker.ceph.com/issues/9566
2014-09-22T08:43:33Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>Assume 72 hours for host replacement/reprovisioning SLA. When host goes down (hardware failure), we expect complete cluster recovery in ~48+ hours. If we lose one more disk anywhere else during this interval, we lose write access (min_size=2) to a subset of 36 million of objects. Hopefully, much smaller subset. If another disk fails, we lose data permanently. Losing a host and another two disks (out of 576 disks in total) within 48+ hours is a non-zero probability. While we understand that this is an inherent risk with any distributed system, we're not very happy about the fact that the most time spent in recovery is when less than 10% of objects are degraded (very long tail). If we maintained a more or less constant repair rate (for simplicity, let's not account for client/recover throttling), we could've reduced the exposure window from 48 to 12 or less hours.</p>
<p>Note: osd_max_backfill is the default (i.e. 10)</p>
Ceph - Feature #9097 (New): request for tools/commands to see hits/misses on cache pools
https://tracker.ceph.com/issues/9097
2014-08-13T09:22:55Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>request for tools/commands to see hits/misses on cache pools</p>
Ceph-deploy - Feature #8790 (Resolved): configurable fsid for ceph-deploy
https://tracker.ceph.com/issues/8790
2014-07-09T14:03:41Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>I'd like the ability to specify the fsid for a cluster in the ceph-deploy script when running "ceph-deploy new monhost". Use case is that we would like to pre-generate all our configs so that we are not dependent on the ceph cluster when we roll out a new site. This should speed up our general openstack deployments. I realize you can do this if you do a manual deployment, but the goal is to speed up our overall deployments of a new site.</p>
Ceph - Bug #8174 (Resolved): rados put of a long object name crashes the OSD process
https://tracker.ceph.com/issues/8174
2014-04-21T14:54:47Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>rados -p testpool put <br />foo.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000000000000000000000 /etc/group</p>
<p>will crash the OSD process. He confirmed this on 0.67.7 and 0.77 server versions. The OSD filesystem is ext4 mounted with user_xattr. (We didn't try this on an xfs OSD).</p>
<p>The OSD log from 0.77 is attached, and I think the relevant error is this one:</p>
<p>-4> 2014-03-20 09:02:03.319609 7f9a40d1a7a0 0 filestore(/srv/castor/01/ceph-1) ENOSPC on setxattr on 546.7_head/8a9e21df/foo.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000.0000000000000000/head//546 name snapset size 31 <br />-3> 2014-03-20 09:02:03.319686 7f9a40d1a7a0 0 filestore(/srv/castor/01/ceph-1) error (28) No space left on device not handled on operation 14 (42895.1.3, or op 3, counting from 0) <br />-2> 2014-03-20 09:02:03.319765 7f9a40d1a7a0 0 filestore(/srv/castor/01/ceph-1) ENOSPC handling not implemented</p>
<p>Greg:</p>
<p>I don't think we've seen this explicitly before, but it doesn't surprise me -- it looks like they've given an object a name that's too long to fit into an ext4 xattr. We probably want to fix it one way or another but it will require some fiddling and a (probably inconsequential, but still real) change to the contract we provide users (ie, limiting the length of a RADOS name, or maybe something clever like our "lfn" code inside of the xattrs).</p>
Ceph - Feature #7792 (Closed): leveldb 1.12.0 for rhel
https://tracker.ceph.com/issues/7792
2014-03-20T07:28:08Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>It appears libleveldb1 version 1.12.0 is provided for debian/ubuntu here:</p>
<p><a class="external" href="http://ceph.com/debian-dumpling/pool/main/l/leveldb/">http://ceph.com/debian-dumpling/pool/main/l/leveldb/</a></p>
<p>I expect most ceph dumpling users are running this version of leveldb.</p>
<p>No version is provided for dumpling for rhel:</p>
<p><a class="external" href="http://ceph.com/rpm-dumpling/el6/x86_64/">http://ceph.com/rpm-dumpling/el6/x86_64/</a></p>
<p>It is provided for emperor here, but only 1.7.0.</p>
<p><a class="external" href="http://ceph.com/rpm-emperor/el6/x86_64/">http://ceph.com/rpm-emperor/el6/x86_64/</a></p>
<p>Can/should we get 1.12.0 available for rhel?</p>
Ceph - Bug #7648 (Resolved): ceph-mon corner case denial of service
https://tracker.ceph.com/issues/7648
2014-03-07T10:03:55Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>ceph-mon corner case denial of service</p>
<p>Improperly positioned (in CRUSH map) OSD may cause mon process to stop responding, consume all virtual memory and die by OOM killer. For example:</p>
$ ceph osd tree
<ol>
<li>id weight type name up/down reweight <br />-1 1.08 root default <br />-2 1.08 host hosta <br />1 1.08 osd.1 up 1</li>
</ol>
<p>0 0 osd.0 down 0</p>
<p>^^^ osd.0 is not under root default/host (e.g., due to botched ceph-disk-prepare)</p>
<p>$ ceph -f json-pretty mon_status</p>
<p>{ "name": "2", <br />"rank": 2, <br />"state": "peon", <br />"election_epoch": 48, <br />"quorum": [ <br />0, <br />1, <br />2], <br />"outside_quorum": [], <br />"extra_probe_peers": [], <br />"sync_provider": [], <br />"monmap": { "epoch": 2, <br />"fsid": "XXXXXXXXXX", <br />"modified": "2014-03-06 17:15:40.485887", <br />"created": "2014-03-06 17:13:05.099978", <br />"mons": [
{ "rank": 0, <br />"name": "0", <br />"addr": "10.0.0.6:6789\/0"},
{ "rank": 1, <br />"name": "1", <br />"addr": "10.0.0.10:6789\/0"},
{ "rank": 2, <br />"name": "2", <br />"addr": "10.0.0.11:6789\/0"}]}}</p>
<p>^^^ all 3 monitors are up and healthy</p>
<p>$ ceph osd find 0</p>
<p>^^^ hangs. one of the monitors stops responding, runs at 100% CPU until its VIRT consumes all physical memory + swap and then OOM-killed</p>
<p>$ sudo ceph --admin-daemon=/var/run/ceph/ceph-mon.1.asok mon_status
{ "name": "1", <br />"rank": 1, <br />"state": "leader", <br />"election_epoch": 50, <br />"quorum": [ <br />1, <br />2], <br />"outside_quorum": [], <br />"extra_probe_peers": [], <br />"sync_provider": [], <br />"monmap": { "epoch": 2, <br />"fsid": "XXXXXXXXXXXX", <br />"modified": "2014-03-06 17:15:40.485887", <br />"created": "2014-03-06 17:13:05.099978", <br />"mons": [
{ "rank": 0, <br />"name": "0", <br />"addr": "10.0.0.6:6789\/0"},
{ "rank": 1, <br />"name": "1", <br />"addr": "10.0.0.10:6789\/0"},
{ "rank": 2, <br />"name": "2", <br />"addr": "10.0.0.11:6789\/0"}]}}</p>
<p>^^^ <br />monitor id==0 is out, its core dump fills the filesystem (image > 200GB) <br />from mon logs:</p>
<p>0> 2014-03-07 16:19:49.433591 7fb970a19700 -1 *<strong>* Caught signal (Aborted) *</strong> <br />in thread 7fb970a19700</p>
<p>ceph version 0.72.2 (a913ded2ff138aefb8cb84d347d72164099cfd60) <br />1: /usr/bin/ceph-mon() [0x8b0f22] <br />2: (()+0xf030) [0x7fb97609c030] <br />3: (gsignal()+0x35) [0x7fb9749bc475] <br />4: (abort()+0x180) [0x7fb9749bf6f0] <br />5: (_<em>gnu_cxx::</em>_verbose_terminate_handler()+0x11d) [0x7fb97521189d] <br />6: (()+0x63996) [0x7fb97520f996] <br />7: (()+0x639c3) [0x7fb97520f9c3] <br />8: (()+0x63bee) [0x7fb97520fbee] <br />9: (tc_new()+0x48e) [0x7fb9762e2aee] <br />10: (std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > >::_M_insert_aux(_<em>gnu_cxx::</em>_normal_iterator<std::pair<std::string, std::string>*, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > > >, std::pair<std::string, std::string> const&)+0x160) [0x7368c0] <br />11: (CrushWrapper::get_full_location_ordered(int, std::vector<std::pair<std::string, std::string>, std::allocator<std::pair<std::string, std::string> > >&)+0x4ce) [0x73537e] <br />12: (CrushWrapper::get_full_location(int)+0x5f) [0x7356bf] <br />13: (OSDMonitor::preprocess_command(MMonCommand*)+0xb80) [0x678f60] <br />14: (OSDMonitor::preprocess_query(PaxosServiceMessage*)+0x23b) [0x67c15b] <br />15: (PaxosService::dispatch(PaxosServiceMessage*)+0x5cc) [0x650abc] <br />16: (Monitor::handle_command(MMonCommand*)+0x9c8) [0x6147a8] <br />17: (Monitor::dispatch(MonSession*, Message*, bool)+0x3e2) [0x61d6a2] <br />18: (Monitor::_ms_dispatch(Message*)+0x1c6) [0x61db16] <br />19: (Monitor::ms_dispatch(Message*)+0x32) [0x63ba82] <br />20: (DispatchQueue::entry()+0x4eb) [0x88c3db] <br />21: (DispatchQueue::DispatchThread::entry()+0xd) [0x7c469d] <br />22: (()+0x6b50) [0x7fb976093b50] <br />23: (clone()+0x6d) [0x7fb974a64a7d] <br />NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.</p>
<p>if one would repeat `ceph find 0', the next monitor will go down and so on ...</p>
devops - Bug #7598 (Can't reproduce): ceph-disk-activate error with ceph-deploy
https://tracker.ceph.com/issues/7598
2014-03-04T11:42:57Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>ceph-disk-activate throws the following error to ceph-deploy yet it successfully creates the osd, please note this was preceded by a successful "disk zap".</p>
<p>[hosta][DEBUG ] detect platform information from remote host <br />[hosta][DEBUG ] detect machine type <br />[ceph_deploy.osd][INFO ] Distro info: Red Hat Enterprise Linux Server 6.4 Santiago <br />[ceph_deploy.osd][DEBUG ] activating host hosta disk /dev/sdb <br />[ceph_deploy.osd][DEBUG ] will use init type: sysvinit <br />[hosta][INFO ] Running command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb <br />[hosta][WARNIN] ceph-disk: Cannot discover filesystem type: device /dev/sdb: Line is truncated: <br />[hosta][ERROR ] RuntimeError: command returned non-zero exit status: 1 <br />[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk-activate --mark-init sysvinit --mount /dev/sdb</p>
devops - Bug #7486 (Rejected): python-backports needs fixing for rhel
https://tracker.ceph.com/issues/7486
2014-02-20T06:48:59Z
Sheldon Mustard
sheldon.mustard@inktank.com
<p>Getting warnings</p>
<p>/usr/lib/python2.6/site-packages/babel/__init__.py:33: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path from pkg_resources import get_distribution, ResolutionError</p>
<p>like:</p>
<p>glance --version <br />/usr/lib/python2.6/site-packages/babel/__init__.py:33: UserWarning: Module backports was already imported from /usr/lib64/python2.6/site-packages/backports/__init__.pyc, but /usr/lib/python2.6/site-packages is being added to sys.path from pkg_resources import get_distribution, ResolutionError 0.12.0</p>
<p>Installed version Red Hat Version <br />python-backports-ssl_match_hostname-3.4.0.2-1.el6.noarc python-backports-ssl_match_hostname-3.2-0.2.1.a3.el6.noarch <br />python-six-1.4.1-1.el6.noarch python-six-1.1.0-2.1.el6.noarch</p>
<p>we add these: <br />python-backports.x86_64 1.0-3.el6 @ceph-emperor <br />python-backports-ssl_match_hostname.noarch 3.4.0.2-1.el6 @ceph-rpms <br />python-six.noarch 1.4.1-1.el6 @ceph-rpms</p>
<p>--<br />this is a known issue in the version of python libs, and has been updated upstream, we've just not compiled new packages with the fixes of the upstream, as well as the updates we put in. all that needs to happen, is whoever maintains the python RPMs should snag the new upstream, reapply our fixes and package it back up.<br />--</p>