Project

General

Profile

Activity

From 01/31/2013 to 03/01/2013

03/01/2013

05:41 PM Subtask #4127: kernel osd client: clearly separate read and write data buffers
I have put together a series of patches that implement
this clear separation. It required work in both the
osd cli...
Alex Elder
05:36 PM Bug #4284: rbd: bio getting set more than once in a message?
I think I'm going to check in the functions that
isolate/abstract message data operations without
assertions enable...
Alex Elder
05:30 PM Subtask #4263: libceph: clearly abstract message data operations
This is partially done. These two have been committed:
ba1dd81 libceph: set page alignment in start_request()
46...
Alex Elder
11:48 AM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
ubuntu@teuthology:/a/teuthology-2013-02-28_20:00:04-regression-argonaut-master-basic/14358 Tamilarasi muthamizhan
11:36 AM Feature #3817: librbd: make cache write-through until a flush is encountered
Sage Weil
11:35 AM Cleanup #4262: libceph: drop redundant page info assignments
These have been committed.
5038c43 libceph: don't pass request to calc_layout()
3ed9489 libceph: format target ob...
Alex Elder
11:29 AM Cleanup #4262 (Resolved): libceph: drop redundant page info assignments
Sage Weil
11:16 AM Bug #3664: osdc/ObjectCacher.cc: 517: FAILED assert(!i->size())
Reproduced with client side logging after running in a loop for several days. Josh Durgin

02/28/2013

03:43 PM Tasks #4210 (Fix Under Review): get ceph and all libraries versions+patches in EPEL
Josef said earlier this week that he has submitted the epel branch. I'm not sure if that means the builds are now ha... Sage Weil
03:34 PM Tasks #4210: get ceph and all libraries versions+patches in EPEL
Not libvirt, but librbd, librados, etc. Ian Colle
11:47 AM Bug #4285: rbd copy.sh test failing on counting images
triggered again last night, ubuntu@teuthology:/a/teuthology-2013-02-28_01:00:03-regression-next-testing-basic$ less 1... Sage Weil
11:41 AM Bug #4285: rbd copy.sh test failing on counting images
I haven't been able to reproduce this yet, but I suspect it's the previous remove that failed. Unfortunately there ar... Josh Durgin
11:24 AM Subtask #4125: kernel messenger: support multiple sources of data
> That field has no semantic meaning. All it does is
> control the alignment of the memory the OSD reads
> into, w...
Alex Elder
11:00 AM Subtask #4125: kernel messenger: support multiple sources of data
That field has no semantic meaning. All it does is control the alignment of the memory the OSD reads into, which wil... Sage Weil
10:33 AM Subtask #4125: kernel messenger: support multiple sources of data
Another thing about that field. In ceph_osdc_build_request(),
the 64-bit "off" value is subject to this:
...
Alex Elder
10:30 AM Subtask #4125: kernel messenger: support multiple sources of data
I've been winding my way through code this morning and
I've discovered there's something in the ceph message
protoc...
Alex Elder
07:20 AM Bug #4284: rbd: bio getting set more than once in a message?
My test to verify my explanation seems to have done so.
I had a message print when get_reply() was called. This
...
Alex Elder
06:59 AM Bug #4284: rbd: bio getting set more than once in a message?
After further consideration I don't think this is going to
be good enough. My goal here is to allow multiple blocks...
Alex Elder
06:28 AM Bug #4284: rbd: bio getting set more than once in a message?
Testing once more with key debug messages enabled that will I
think allow me to prove my explanation is correct.
Alex Elder

02/27/2013

08:36 PM Bug #4284: rbd: bio getting set more than once in a message?
I'm too tired to offer a guarantee, but I'm pretty sure
this is what's going on. A connection fault occurs while
r...
Alex Elder
05:18 PM Bug #4284: rbd: bio getting set more than once in a message?
OK, back again after a meeting.
I was thinking about it, and I wonder whether this has to
do with a message that ...
Alex Elder
12:53 PM Bug #4284: rbd: bio getting set more than once in a message?
We are in the path where an osd response message has
been received, and the messenger has called its
alloc_message ...
Alex Elder
12:28 PM Bug #4284: rbd: bio getting set more than once in a message?
I have reproduced the problem--just. I haven't had a chance
to look at it but I think I will soon be able to identi...
Alex Elder
09:17 AM Bug #4284 (Resolved): rbd: bio getting set more than once in a message?
I have to reproduce this again because my target machine
that hit this got rebooted. But a recent proposed change
...
Alex Elder
04:29 PM Feature #3417 (In Progress): krbd: read path (layering)
Ian Colle
04:28 PM Subtask #2855 (Closed): krbd: copy-up on write to clone
Tracking effort under 3419 Ian Colle
04:27 PM Feature #3419 (New): krbd: copy-up on write to clone
Ian Colle
04:25 PM Subtask #2854 (Closed): krbd: write path
Tracking under 3418 Ian Colle
04:25 PM Feature #3418 (In Progress): krbd: write path (layering)
Ian Colle
04:23 PM Subtask #2852 (Closed): krbd: open parent on open
Move this tracking to 3416 Ian Colle
04:21 PM Feature #3416 (In Progress): krbd: open parent on open
Ian Colle
04:21 PM Feature #2851: krbd: RBD layering support
Josh created the new feature tickets so they'd show up in redmine backlog - subtasks don't. Ian Colle
11:02 AM Bug #4285 (Resolved): rbd copy.sh test failing on counting images
... Sage Weil
07:58 AM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
The issue for not completing multiple lingering requests
more than once is http://tracker.ceph.com/issues/3967.
(Ju...
Alex Elder

02/26/2013

06:57 PM Bug #4255: osdc/ObjectCacher.cc: 517: FAILED assert(i->empty())
recent log: ubuntu@teuthology:/a/teuthology-2013-02-25_01:00:05-regression-master-testing-gcov/11554
2013-02-25T03...
Tamilarasi muthamizhan
02:56 PM Bug #4272 (Resolved): rbd snap rollback broken on next
commit:f58601d681d3aee6541119ab830ba5e2a9ec3a13 Sage Weil
12:23 PM Bug #4272 (Resolved): rbd snap rollback broken on next
... Sage Weil
11:12 AM Cleanup #4262 (Fix Under Review): libceph: drop redundant page info assignments
Ian Colle
05:12 AM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
Of course it's not going to be easy... The problem showed up
this time at offset 2199533527040 instead of 219953839...
Alex Elder

02/25/2013

07:12 PM Bug #4237 (Resolved): krbd: ignore zero-length requests
This has been committed.
commit 4dda41d3d76747414586a4bad5615b550e0986b1
Author: Alex Elder <elder@inktank.com>
...
Alex Elder
07:11 PM Bug #4238 (Resolved): krbd: still more barrier fixes
This has been committed.
commit 632b88cadece050ca925d74bda250c4a320c5cc7
Author: Alex Elder <elder@inktank.com>
...
Alex Elder
07:11 PM Bug #4235 (Resolved): krbd: add some dout() trace calls
This has been committed.
commit 37206ee5bede14d59306fea3af4c0105d4712342
Author: Alex Elder <elder@inktank.com>
...
Alex Elder
07:10 PM Cleanup #4234 (Resolved): libceph: encapsulate flag setting
This has been committed.
commit c9ffc77adebf9dfe3026ede6c8b3c61586b485b7
Author: Alex Elder <elder@inktank.com>
...
Alex Elder
07:09 PM Bug #4184 (Resolved): krbd: fix some sparse problems
Josh reviewed these patches, and I have updated, re-tested
and committed them.
cc344fa rbd: eliminate sparse warn...
Alex Elder
07:01 PM Feature #3190 (Resolved): qa: test fencing
commit:ccdafa067afed74b209537ae45ff8b49bb5fab3b Josh Durgin
05:49 PM Bug #4249 (Resolved): librbd: recursive lock of snap_lock during snap_rollback
commit:9096d70642880946b0b477e33f7debabbefec9fa Sage Weil
12:26 PM Bug #4249 (Fix Under Review): librbd: recursive lock of snap_lock during snap_rollback
wip-4249 Josh Durgin
05:41 PM Bug #4252 (Resolved): rbd python bindings: error removing image on list_children
commit:21471c1ef4cf02e068a6e6c49cd41e95efec952a Sage Weil
04:46 PM Bug #4252 (Fix Under Review): rbd python bindings: error removing image on list_children
wip-4252, test broke with the watch-notify changes Josh Durgin
04:02 PM Bug #4252 (In Progress): rbd python bindings: error removing image on list_children
Josh Durgin
05:08 PM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
The output did not indicate that rbd thought it
did something wrong.
That is, the object request came back and in...
Alex Elder
04:08 PM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
Requests to write 4096 bytes came back from the osd with a
written count of 2048 bytes.
Now I'm afraid I have to ...
Alex Elder
01:23 PM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
> BTW fixed the rbd namer thing; pull latest teuthology.git.
Well, "fixed" is generous. It had a bug, which I've ...
Alex Elder
12:09 PM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
BTW fixed the rbd namer thing; pull latest teuthology.git. Sage Weil
11:12 AM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
I was reporting if an error code got returned from
the osd. However the loop device function that
reports the "Wri...
Alex Elder
11:04 AM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
The tests I started running this morning finally hit number 259.
It did have the problem, but I got no messages from...
Alex Elder
09:09 AM Bug #4242 (In Progress): krbd: xfstest 259 failure (FS size near 4TB)
I tried to reproduce this with a smaller subset of tests
(just 258 and then 259 repeatedly) over the weekend but
th...
Alex Elder
04:58 PM Tasks #4210 (Rejected): get ceph and all libraries versions+patches in EPEL
EPEL does not take packages already in RHEL. Josh Durgin
03:43 PM Subtask #4263 (Fix Under Review): libceph: clearly abstract message data operations
These patches have been posted for review.
[PATCH 0/4] libceph: abstract setting message data info
[PATCH 1/4] li...
Alex Elder
03:33 PM Subtask #4263 (Resolved): libceph: clearly abstract message data operations
Currently the fields in a message structure related
what constitutes the data portion of a message are
manipulated ...
Alex Elder
03:15 PM Cleanup #4262: libceph: drop redundant page info assignments
These patches have been posted for review (marking
this "Feedback").
[PATCH 0/3] libceph: focus calc_layout() on ...
Alex Elder
03:03 PM Cleanup #4262: libceph: drop redundant page info assignments
Three patches implement this change. The net result is
that calc_layout() becomes focused only on filling in the
o...
Alex Elder
02:53 PM Cleanup #4262 (Resolved): libceph: drop redundant page info assignments
In calc_layout(), there are some assignments to osd request
fields r_num_pages and r_page_alignment. These assignme...
Alex Elder
03:14 PM Bug #4240 (Fix Under Review): libceph: fix a osd request memory leak
This has been posted for review.
[PATCH] libceph: fix a osd request memory leak
Alex Elder
12:27 PM Bug #4255 (Duplicate): osdc/ObjectCacher.cc: 517: FAILED assert(i->empty())
Josh Durgin

02/23/2013

08:43 PM Bug #4255 (Duplicate): osdc/ObjectCacher.cc: 517: FAILED assert(i->empty())
... Sage Weil
12:08 PM Bug #4252 (Resolved): rbd python bindings: error removing image on list_children
... Sage Weil
08:31 AM Bug #4249 (Resolved): librbd: recursive lock of snap_lock during snap_rollback
... Sage Weil

02/22/2013

06:44 PM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
I"m seeing this fairly consistently now. Alex Elder
01:03 PM Bug #4242: krbd: xfstest 259 failure (FS size near 4TB)
Whoops, it was 4 TB (2^42) not 4 GB.
I'll update what I originally wrote shortly.
Alex Elder
01:00 PM Bug #4242 (Closed): krbd: xfstest 259 failure (FS size near 4TB)
I have now seen this at least twice. I think it's inconsistent
but it might be important. Not marking it high yet,...
Alex Elder
03:33 PM Bug #3958 (Resolved): rbd fsx fails with EBUSY
Sage Weil
03:14 PM Bug #1070 (Rejected): krbd: ^C doesn't work
Yeah, marking rejected. thanks for digging in! Sage Weil
03:08 PM Bug #1070: krbd: ^C doesn't work
I can't relate them but this is involved in at
least some of the code involved:
http://tracker.ceph.com/issues/38...
Alex Elder
02:24 PM Fix #3665 (Resolved): librbd: deadlock during flatten
commit:e4fd70fcec3a9abb7f19517326e46f58031c4196 Sage Weil
02:13 PM Bug #913 (Resolved): krbd: handle race between notify and rbd device shutdown
This is very old. And--provided I understand it--it is resolved
in the current rbd code.
When a mapped rbd image...
Alex Elder
01:50 PM Bug #4244 (Resolved): rbd xfstests: re-enable xfstests 049 in nightly run
Pushed to ceph-qa-suite/master.... Alex Elder
01:38 PM Bug #4244 (Resolved): rbd xfstests: re-enable xfstests 049 in nightly run
Running test 049 repeatedly is passing for me now. I'm not sure
when or how it got fixed but I'm going to re-enable...
Alex Elder
01:40 PM Bug #2572 (Resolved): krbd: writeback errors?
I've run xfstests 49 a bunch of times consecutively
and I am no longer seeing this issue.
I'm about to run it 100...
Alex Elder
01:23 PM Bug #2700: blkdeviotune method at libvirt doesn`t work on RBD volumes
Just checked, problem with blkdeviotune still exists in the 1.0.2. Andrey Korolyov
11:53 AM Bug #2700: blkdeviotune method at libvirt doesn`t work on RBD volumes
This is something that we should fix. Is it still an issue in current libvirt? Josh Durgin
01:20 PM Bug #2654 (Resolved): Stale rbd volume cannot be unmaped
Somewhere along the way, this got fixed.... Alex Elder
01:15 PM Bug #4243 (Resolved): rbd cli: usage confusing for snapshot operations
The correct usage for creating a snapshot seems to be:... Alex Elder
12:05 PM Bug #3939 (Duplicate): krbd: circular locking report in sysfs code
Duplicate of 3925. I *did* write it up before. Alex Elder
12:02 PM Bug #3950 (Can't reproduce): krbd: new assertion failure running concurrent rbd test
Not seeing this in current code.
Ian Colle
11:59 AM Bug #3871 (In Progress): krbd: initial header read may be out of date
Ian Colle
11:58 AM Bug #3743 (Won't Fix): krbd: errors on submitted requests are ignored
This was found in old request code and therefore will not be fixed. Ian Colle
11:38 AM Bug #3936 (Rejected): rbd: Strange dd speed behaviour (server side issue?)
Network issue. Ian Colle
11:37 AM Bug #3664: osdc/ObjectCacher.cc: 517: FAILED assert(!i->size())
Need to reproduce with:... Josh Durgin
11:35 AM Bug #3697 (Duplicate): rbd copy.sh test failing in nightly
#3958 Josh Durgin
11:34 AM Bug #3980 (Won't Fix): rbd image created with size zero on a mixed cluster crashes rbd
Problem not in Bobtail and future releases. Won't fix. Ian Colle
11:32 AM Bug #3524 (Duplicate): test_librbd_fsx: crash after flatten
Same as #3665 Josh Durgin
11:26 AM Bug #3703 (Can't reproduce): osd: crash while encrypting
Ian Colle
11:12 AM Bug #4240 (Resolved): libceph: fix a osd request memory leak
If an invalid layout is provided to ceph_osdc_new_request(), its
call to calc_layout() might return an error. At th...
Alex Elder
09:39 AM Bug #4184: krbd: fix some sparse problems
Forgot to set target version. Alex Elder
09:38 AM Bug #4184 (Fix Under Review): krbd: fix some sparse problems
Re-posted for review. The patches have been re-based on
top of fixes for 4234, 4235, 4237, and 4238.
Alex Elder
09:37 AM Cleanup #4234: libceph: encapsulate flag setting
Posted for review. (Marked this as "Feedback" because
I can't mark a cleanup issue "need review")...
Alex Elder
06:22 AM Cleanup #4234 (In Progress): libceph: encapsulate flag setting
Forgot to indicate this was underway. Alex Elder
06:02 AM Cleanup #4234 (Resolved): libceph: encapsulate flag setting
While working on http://tracker.ceph.com/issues/4079 I created
some helper routines to wrap around the code that tes...
Alex Elder
09:36 AM Bug #4235 (Fix Under Review): krbd: add some dout() trace calls
Posted for review.... Alex Elder
06:23 AM Bug #4235 (Resolved): krbd: add some dout() trace calls
While working on http://tracker.ceph.com/issues/4079 I added
some calls to dout() to some of the new request code. ...
Alex Elder
09:36 AM Bug #4238 (Fix Under Review): krbd: still more barrier fixes
Posted for review.... Alex Elder
07:21 AM Bug #4238 (Resolved): krbd: still more barrier fixes
While looking at http://tracker.ceph.com/issues/4079 I
concluded that the fix I put in for this might still
not be ...
Alex Elder
09:35 AM Bug #4237 (Fix Under Review): krbd: ignore zero-length requests
Posted for review.... Alex Elder
06:40 AM Bug #4237 (Resolved): krbd: ignore zero-length requests
The old request code simply ignored zero-length requests
as soon as they were seen. The new request code currently
...
Alex Elder
06:41 AM Feature #4236: krbd: properly handle flush commands
Note that http://tracker.ceph.com/issues/4237 describes
making the new request code ignore all zero-length commands....
Alex Elder
06:38 AM Feature #4236 (Duplicate): krbd: properly handle flush commands
REQ_FLUSH and REQ_FUA are intended to implement drive
cache flushes and barrier semantics. I don't know
precisely ...
Alex Elder
02:45 AM Feature #4231 (Closed): librbd: Java bindings
As it seems right now the new CloudStack code will require more complex RBD operations which can not be accomplished ... Wido den Hollander

02/21/2013

06:16 PM Fix #3665 (Fix Under Review): librbd: deadlock during flatten
wip-rbd-flatten-deadlock Josh Durgin
03:50 PM Bug #4186: rbd: add another "udevadm settle" in the CLI
awesome, I fixed this without even knowing it existed! :) Dan Mick
02:50 PM Bug #3737: Higher ping-latency observed in qemu with rbd_cache=true during disk-write
I've looked at the logs, and I think #4091 should fix this. The high ping times tend to occur around when the cache f... Josh Durgin

02/20/2013

11:20 PM Bug #3737: Higher ping-latency observed in qemu with rbd_cache=true during disk-write
Sigh. The attachment might help... Chris Dunlop
11:18 PM Bug #3737: Higher ping-latency observed in qemu with rbd_cache=true during disk-write
Confirmed here, with ceph-0.56.3 and qemu-1.3.1.
See attached test output.
A summary is, the average ping time,...
Chris Dunlop
09:15 PM Bug #3958 (In Progress): rbd fsx fails with EBUSY
reproducing with logs, vapre f.yaml.out, to confirm this is just the osd's EBUSY. which we should fix. Sage Weil
12:50 PM Cleanup #4053 (Resolved): ceph: cleanup ceph page vector functions
Committed to the testing branch.
9e0eb85 ceph: remove a few bogus declarations
b324814 libceph: use void pointers...
Alex Elder
12:49 PM Subtask #4007 (Resolved): libceph: support STAT osd operation

commit fbfab53966b279f9cdb36b96ffa1e22f042c96ff
Author: Alex Elder <elder@inktank.com>
Date: Fri Feb 8 09:55:48...
Alex Elder
12:42 PM Fix #3665 (In Progress): librbd: deadlock during flatten
Josh Durgin
11:26 AM Feature #4211 (Rejected): get good qemu, libvirt versions+patches in CentOS+
Sage Weil
11:26 AM Tasks #4210 (Resolved): get ceph and all libraries versions+patches in EPEL
Sage Weil

02/19/2013

02:34 PM Feature #2770 (In Progress): krbd: define tasks to add osd_client compound class op support
Sage Weil
02:23 PM Feature #2770 (Resolved): krbd: define tasks to add osd_client compound class op support
Sage Weil
02:22 PM Feature #4021 (Resolved): rbd: openstack: add ability to copy volume to image for rbd
Sage Weil
02:20 PM Feature #4195 (Duplicate): rbd: migrate an active image between pools
Sage Weil
11:40 AM Subtask #4007: libceph: support STAT osd operation
Josh has reviewed this. Testing again before committing
to be included in the testing branch.
Alex Elder
11:39 AM Cleanup #4053: ceph: cleanup ceph page vector functions
Josh has reviewed these, and I've rebased them onto
the top of the testing branch. I'm going to test some
before u...
Alex Elder
10:36 AM Bug #4184 (In Progress): krbd: fix some sparse problems
I have resolved all of the reports that remain in the
code currently in the testing branch. There are a
few report...
Alex Elder
08:57 AM Bug #4183 (Resolved): rbd: EBUSY on unmap right after io
commit:45a4fe09151bcdeaaa2c3b5dbd089a91deaa055f Sage Weil
08:45 AM Bug #4186 (Resolved): rbd: add another "udevadm settle" in the CLI
commit:8ca2274cc09c6a9925cbe86ccaf68db7898956e8 Sage Weil
05:06 AM Bug #4186: rbd: add another "udevadm settle" in the CLI
The change looks good to me.
Reviewed-by: Alex Elder <elder@inktank.com>
Alex Elder

02/18/2013

08:37 PM Bug #4186 (Fix Under Review): rbd: add another "udevadm settle" in the CLI
see wip-4186 Sage Weil
06:04 PM Bug #4186 (Resolved): rbd: add another "udevadm settle" in the CLI
In http://tracker.ceph.com/issues/4183 it was reported that
unmapping an image was failing because it was busy.
F...
Alex Elder
06:10 PM Bug #4183 (In Progress): rbd: EBUSY on unmap right after io
Reassigning to Sage.
Sage, I see you've been working on ceph qa/rbd/map-snapshot-io.sh
and I don't want to interf...
Alex Elder
06:05 PM Bug #4183: rbd: EBUSY on unmap right after io
... Alex Elder
06:04 PM Bug #4183: rbd: EBUSY on unmap right after io
OK, I think I understand now.
It turns out that "blkid" is running again on the
device after its final close. Th...
Alex Elder
05:51 PM Bug #4183: rbd: EBUSY on unmap right after io
We now are very careful about allowing an image to be unmapped
if it's still open, whereas previously we would allow...
Alex Elder
11:24 AM Bug #4183: rbd: EBUSY on unmap right after io
NOTE: I adjusted the test to add the sleep 1 until this is fixed. Sage Weil
11:23 AM Bug #4183 (Resolved): rbd: EBUSY on unmap right after io
the map-snapshot-io.sh does... Sage Weil
05:13 PM Bug #4184 (Resolved): krbd: fix some sparse problems
The Linux kernel has a tool called "sparse" that scans
the source tree and reports various code problems that
the c...
Alex Elder
05:07 PM Cleanup #4164 (Resolved): libceph: update rados.h and related code
commit 0315a7770983bbe69211efed1aaee08324acd54c
Author: Alex Elder <elder@inktank.com>
Date: Fri Feb 15 11:42:30 ...
Alex Elder
05:05 PM Subtask #4126 (Resolved): kernel osd client: kill off some dead code
87f979d ceph: kill ceph_osdc_writepages() "nofail" parameter
fbf8685 ceph: kill ceph_osdc_writepages() "dosync" para...
Alex Elder
05:01 PM Cleanup #4165 (Resolved): libceph: update ceph_fs.h and related code
commit dd6f5e105d85e02bc41db0891eb07152b1746ad9
Author: Alex Elder <elder@inktank.com>
Date: Fri Feb 15 11:42:30 ...
Alex Elder

02/16/2013

09:11 AM Cleanup #4165: libceph: update ceph_fs.h and related code
The following have been posted for review to resolve
this issue.
[PATCH 0/2] libceph: update ceph_fs.h and relate...
Alex Elder
08:11 AM Cleanup #4165: libceph: update ceph_fs.h and related code
(Another related commit)
libceph: update ceph_mds_state_name() and ceph_mds_op_name()
Update ceph_mds_state_nam...
Alex Elder
08:10 AM Cleanup #4165 (Resolved): libceph: update ceph_fs.h and related code
(This work is done. I'm just documenting it because it's really
distinct from http://tracker.ceph.com/issues/4126.)...
Alex Elder
09:10 AM Subtask #4126 (Fix Under Review): kernel osd client: kill off some dead code
(Oops, I forgot to mark this in progress yesterday.)
The following have been posted for review to resolve this
is...
Alex Elder
08:58 AM Cleanup #4164: libceph: update rados.h and related code
The following have been posted for review to resolve this:
[PATCH 0/3] libceph: update rados.h and related code (4...
Alex Elder
08:07 AM Cleanup #4164: libceph: update rados.h and related code
(Another commit related to this.)
libceph: update ceph_osd_op_name()
Update ceph_osd_op_name() to include the n...
Alex Elder
08:06 AM Cleanup #4164: libceph: update rados.h and related code
(Another commit related to this.)
libceph: add ceph_osd_state_name()
Add the definition of ceph_osd_state_name(...
Alex Elder
08:05 AM Cleanup #4164 (Resolved): libceph: update rados.h and related code
(This work is done, I'm just documenting it because it's really
something distinct from http://tracker.ceph.com/issu...
Alex Elder
08:29 AM Cleanup #4163: libceph: improve packing in struct ceph_osd_req_op
This has been posted for review.
[PATCH] libceph: improve packing in struct ceph_osd_req_op
Alex Elder
08:00 AM Cleanup #4163 (Resolved): libceph: improve packing in struct ceph_osd_req_op
(This is done, just documenting it because it's not directly
related to http://tracker.ceph.com/issues/4126)
The ...
Alex Elder

02/15/2013

01:59 PM Cleanup #4156 (Resolved): qa: don't chown rbd stuff in rbd/* workunits
Once we are doing proper installs, we can call sudo as needed and not worry about $PATH etc. Sage Weil

02/14/2013

05:03 PM Subtask #3272: send/receive rbd snapshots
From Sage - "I think this breaks down into a few different pieces:
1) Decide what output format to use. We want t...
Ian Colle
05:02 PM Subtask #3387: librbd: expose changed objects since a given snapshot
8 points Ian Colle
04:25 PM Feature #4021 (Fix Under Review): rbd: openstack: add ability to copy volume to image for rbd
Ian Colle
04:25 PM Subtask #4019 (In Progress): rbd: openstack: simplify volume booting with new api: add boot optio...
Ian Colle
04:12 PM Subtask #4019: rbd: openstack: simplify volume booting with new api: add boot option to volumes
3 points Ian Colle
04:25 PM Subtask #4020 (In Progress): rbd: openstack: simplify volume booting with new api: make image boo...
Ian Colle
04:12 PM Subtask #4020: rbd: openstack: simplify volume booting with new api: make image boot button add s...
2 points Ian Colle
04:24 PM Subtask #4018 (In Progress): rbd: openstack: simplify volume booting with new api: modify boot pa...
Ian Colle
04:24 PM Feature #4017 (In Progress): rbd: openstack: simplify volume booting with new api
Ian Colle
04:02 PM Bug #3947 (Resolved): krbd: read zeroing freed bio?
This got resolved in version 2 of the "new request code,"
which is now present in the testing branch.
Alex Elder
03:12 PM Feature #4136 (Duplicate): rbd xfstests: update list of tests run
The latest version of xfstests includes tests numbered
up to 296. The last test we run is 278. Some of the
update...
Alex Elder
07:06 AM Subtask #4125: kernel messenger: support multiple sources of data
Here's a short list of smaller tasks I can identify right
now in order to implement this. I may create a new issue
...
Alex Elder
05:57 AM Subtask #4125 (Resolved): kernel messenger: support multiple sources of data
We need to allow the osd client to have an array of osd ops
such that each can supply its own buffer for sending and...
Alex Elder
06:25 AM Subtask #4127 (Resolved): kernel osd client: clearly separate read and write data buffers
The osd client (and messenger) have generally speaking one
field representing *either* the data to be sent *or* the
...
Alex Elder
06:14 AM Subtask #4126 (Resolved): kernel osd client: kill off some dead code
While looking through how to implement multiple ops I have
noticed that there's some dead code in the osd client, or...
Alex Elder
05:40 AM Subtask #2854: krbd: write path
This work actually did begin last week, after getting STAT
calls implemented (see http://tracker.ceph.com/issues/400...
Alex Elder
05:35 AM Tasks #2853: krbd: read path
This work is close to complete, in that I have shown
that a lot of functionality seems to work correctly.
I was n...
Alex Elder
05:24 AM Feature #2770: krbd: define tasks to add osd_client compound class op support
After the walk through described above, I spent a little more
time thinking about the messenger aspect of allowing m...
Alex Elder

02/13/2013

06:10 AM Feature #3761: kernel messenger: need to support multiple ops per request
Here's what I'm prototyping.
I'm going to try to expand the definition of the trail portion of
a message so inste...
Alex Elder
05:12 AM Feature #3761 (In Progress): kernel messenger: need to support multiple ops per request
In order to support multiple ops for osd requests we're
going to have to do some work on the messenger as well.
I'm...
Alex Elder

02/12/2013

05:34 PM Bug #4100 (Resolved): rbd: unprotecting a snapshot in the "UNPROTECTING" state fails with EINVAL
commit:fe283813b44a7c45def6768ea0788a3a0635957e and commit:bfb4482c4596759b464caf45f8f30368898519d8 in bobtail. Josh Durgin
09:37 AM Bug #4100: rbd: unprotecting a snapshot in the "UNPROTECTING" state fails with EINVAL
Dan - please review the wip branch. Ian Colle
03:52 PM Bug #3958: rbd fsx fails with EBUSY
ubuntu@teuthology:/a/teuthology-2013-02-11_20:00:06-regression-bobtail-master-basic/4941 Josh Durgin
02:11 PM Feature #2770: krbd: define tasks to add osd_client compound class op support
The way the osd client handles an object class method right now
assumes that outbound data (headed from the client t...
Alex Elder
10:18 AM Feature #2770: krbd: define tasks to add osd_client compound class op support
... Alex Elder
10:16 AM Feature #2770: krbd: define tasks to add osd_client compound class op support
... Alex Elder
11:42 AM Feature #4104: osd_client: support passing page array as data for CALL op
I guess this needs to be considered an rbd task if it is
to show up as a subtask for 2770.
Alex Elder
11:41 AM Feature #4104 (Resolved): osd_client: support passing page array as data for CALL op
The rbd object "copyup" operation is defined as a class method
operation. Currently when a class method needs to su...
Alex Elder
12:49 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
Journal is 1G in RAM. Since there are no writers at all yet, I don't think that journal is filled and causes device s... Ivan Kudryavtsev
12:47 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
Tried on regular 1G ethernet (other client server). Works well. Speed is stable and no such effect.
I don't under...
Ivan Kudryavtsev

02/11/2013

07:25 PM Bug #4100 (Fix Under Review): rbd: unprotecting a snapshot in the "UNPROTECTING" state fails with...
wip-snap-unprotect Josh Durgin
05:34 PM Bug #4100 (Resolved): rbd: unprotecting a snapshot in the "UNPROTECTING" state fails with EINVAL
As reported on ceph-users, an unprotect earlier was taking too long (possibly due to inactive pgs) and was killed. Th... Josh Durgin
03:35 PM Feature #4095 (Rejected): rbd: 2-phase commit for snapshot creation
To ensure snapshots are created as close as possible to when the user intended, instead of just waiting for a notify,... Josh Durgin
02:33 PM Subtask #4092: rbd: re-read header when watch is re-established
Yes, I'll create corresponding tasks for krbd once I get the rest of the general ones in. Josh Durgin
02:09 PM Subtask #4092: rbd: re-read header when watch is re-established
This same issue would apply to the kernel rbd client also,
right?
Alex Elder
01:32 PM Subtask #4092 (Resolved): rbd: re-read header when watch is re-established
This avoids races that would result in a snapshot not being created correctly, like:... Josh Durgin
01:24 PM Subtask #4091 (Resolved): ObjectCacher: optionally make readx/writex calls never block
The idea is to prevent any aio calls from blocking client (i.e. qemu) threads.
This was what Sage was thinking as ...
Josh Durgin
12:41 PM Subtask #4090 (New): rbd: investigate sources of client-side latency
Some possible sources:
* lock contention
* unnecessary data copying
* contention on queues
* throttling in messen...
Josh Durgin
12:39 PM Feature #4089 (Closed): rbd: improve small I/O performance
Root task for general improvement. Josh Durgin
12:36 PM Feature #4088 (Resolved): rbd: optionally copy-on-read instead of copy-on-write
This can be beneficial in some use cases (such as when there's high latency to the original pool, but not the new pool). Josh Durgin
12:31 PM Feature #4087: rbd: bitmaps for tracking object existence
Also, I wonder at what point it becomes worthwhile to
use something different from bitmaps (such as extents
that de...
Alex Elder
12:29 PM Feature #4087: rbd: bitmaps for tracking object existence
I was thinking this weekend of creating this issue exactly.
Are you envisioning keeping these with the image heade...
Alex Elder
12:27 PM Feature #4087 (Resolved): rbd: bitmaps for tracking object existence
This would improve layered image performance, and enable quick, conservative usage for an image. It should be possibl... Josh Durgin
12:24 PM Feature #4086 (Resolved): rbd: rate-limiting
Enforce policies like max-iops from a client point of view. The objecter throttling is too low-level, especially when... Josh Durgin
12:15 PM Feature #4085 (New): qemu-rbd: allow storing snapshot of ram associated with snapshot of disk
This way the entire state of a VM can be restored, instead of just the disks. Josh Durgin
12:12 PM Feature #4084 (Resolved): rbd: incremental backups
Root task for the feature in general Josh Durgin
12:06 PM Feature #4083 (New): rbd-fuse: expose snapshots (and maybe other pools)
Maybe only do so for an image when asked, but optionally always show all pool and snapshots? Need to think about the ... Josh Durgin
12:03 PM Feature #4082 (Rejected): rbd-fuse: improve performance
Use aio, different IoCtxs per image, and the C++ librbd api for fewer data copies. Josh Durgin
12:02 PM Feature #4081 (New): rbd-fuse: improve usage, make consistent with other ceph tools
Use the common command line/conf file/env var parsing. Re-write internals to c++ instead of C as needed.
Josh Durgin
11:51 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
What size is the journal on your osds? You may just be seeing a slowdown when the journals fill up, and must be flush... Josh Durgin
07:57 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
It may be a network issue as well. Is it easy for you by chance to try with regular ethernet (say, gig instead of 10... Sage Weil
07:43 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
> 3.7.2-ceph
That tells me a lot, and in fact makes me suspect it might
not be rbd that's the cause.
What can ...
Alex Elder
07:41 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
During slow periods iostat shows no operations on OSDs. First of all I've thought about sceduler and iowait problems,... Ivan Kudryavtsev
07:38 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
Linux hosting-cloud1-s1.zzzing.ru 3.7.2-ceph #1 SMP Wed Jan 16 23:25:11 NOVT 2013 x86_64 GNU/Linux
Kernel config: ...
Ivan Kudryavtsev
04:29 AM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
I'm sorry if I missed it, but can you tell me the version
of the kernel you are working with? Running "uname -a"
m...
Alex Elder
11:51 AM Bug #3664: osdc/ObjectCacher.cc: 517: FAILED assert(!i->size())
recent log: ubuntu@teuthology:/a/teuthology-2013-02-09_20:00:03-regression-bobtail-master-basic/3977... Tamilarasi muthamizhan
11:30 AM Feature #2770 (In Progress): krbd: define tasks to add osd_client compound class op support
At our sprint planning meeting we discussed this. The task
was too large and unknown to provide a meaningful estima...
Alex Elder
11:08 AM Bug #4033 (Resolved): krbd: add barriers near done flag operations
I just committed this to the ceph-client/testing branch:
commit 4ad2b189c1a52ce8ae1d6d2528c512021a2f1654
Author: ...
Alex Elder
11:08 AM Bug #4010 (Resolved): krbd: turn off interrupts for open/remove locking
I just committed this to the ceph-client/testing branch.
commit 4cfc31e59fc6521ee0950782a028eccb3f5c9096
Author: ...
Alex Elder

02/10/2013

11:56 PM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
eth1 and ib0 connected to ceph cloud. Ivan Kudryavtsev
11:53 PM Bug #3936: rbd: Strange dd speed behaviour (server side issue?)
Ifstat output looks like:... Ivan Kudryavtsev

02/08/2013

08:36 AM Cleanup #4053: ceph: cleanup ceph page vector functions
Apparently for cleanup there is no "need review" so I'm
marking this "Feedback". I've posted a series of patches
t...
Alex Elder
08:30 AM Cleanup #4053 (Resolved): ceph: cleanup ceph page vector functions
This is just documenting some cleanup activity I've done
that I'm about to post for review.
- delete bogus (re)decl...
Alex Elder
08:21 AM Subtask #4007 (Fix Under Review): libceph: support STAT osd operation
A patch implementing this has been posted to the
ceph-devel mailing list for review.
[PATCH] libceph: allow STAT ...
Alex Elder

02/07/2013

05:08 PM Bug #4047 (Resolved): removing a non-existing rbd image logs error in osd logs
when removing a non-existing rbd image floods osd logs even when the debug is turned off. This can be avoided.
ubu...
Tamilarasi muthamizhan
04:22 PM Bug #4045 (Resolved): snap unprotect on a snapshot that is already unprotected throws inappropria...
ceph version 0.56.2-7-gc3468f7 (c3468f76a5e68a6426f03e508d8ecf26950fca2a)
Trying to unprotect a snapshot, that is ...
Tamilarasi muthamizhan
01:49 PM Bug #4003 (Resolved): rbd: EBUSY errors from rbd unmap
closing this. phew! Sage Weil

02/06/2013

11:49 AM Subtask #4007: libceph: support STAT osd operation
This has turned out to be simple change. It was needed in
rbd as well, and I'll just add support to both under this...
Alex Elder
09:19 AM Subtask #4007: libceph: support STAT osd operation
It wasn't really possible to know this up front but
it looks like this is trivial. I've basically
completed it but...
Alex Elder
10:56 AM Bug #3697 (In Progress): rbd copy.sh test failing in nightly
Tamilarasi muthamizhan
08:56 AM Bug #3958: rbd fsx fails with EBUSY
this is causing several failures on master runs.. something has changed.
latest:
ubuntu@teuthology:/a/sage-2013-...
Sage Weil
08:31 AM Bug #4003: rbd: EBUSY errors from rbd unmap
The testing I've been doing now has shown no problems
now that teuthology has been updated.
The two other issues ...
Alex Elder
06:16 AM Bug #4003: rbd: EBUSY errors from rbd unmap
Seems to have done the trick! The kernel_untar_build.sh
task just finished for me without error, and it failed
rel...
Alex Elder
05:06 AM Bug #4003: rbd: EBUSY errors from rbd unmap
That sounds promising, I hope it works!
This was actually the last thing I was looking at last
night while waitin...
Alex Elder
07:20 AM Bug #4033 (Fix Under Review): krbd: add barriers near done flag operations
A fix for this has been posted for review.
[PATCH] rbd: add barriers near done flag operations
Alex Elder
06:15 AM Bug #4033 (Resolved): krbd: add barriers near done flag operations
I fixed this problem while investigating the rbd hangs
in http://tracker.ceph.com/issues/4003.
Somehow, I missed ...
Alex Elder

02/05/2013

11:43 PM Bug #4003: rbd: EBUSY errors from rbd unmap
This was backing up qa stuff because the rbd.py qa task wasn't unmounting during cleanup. That bit is now fixed. I ... Sage Weil
10:54 PM Bug #4003: rbd: EBUSY errors from rbd unmap
new theory:
the reason umount hangs is because nuke is killing the client and osds at the same time. the umount i...
Sage Weil
10:41 PM Bug #4003: rbd: EBUSY errors from rbd unmap
I found that unmount was hanging too. I think somehow the
completion of the I/O is not getting propagated up when
...
Alex Elder
10:33 PM Bug #4003: rbd: EBUSY errors from rbd unmap
aha:... Sage Weil
10:15 PM Bug #4003: rbd: EBUSY errors from rbd unmap
btw i am able to reproduce the EBUSY with just... Sage Weil
08:28 PM Bug #4003: rbd: EBUSY errors from rbd unmap
I've added some instrumentation and find that the rbd
client is not dropping its watch at the end of the
kernel_unt...
Alex Elder
12:51 PM Bug #4003: rbd: EBUSY errors from rbd unmap
The interrupt issue has been fixed, but the other issue
(rbd device can't be unmapped because EBUSY) remains.
I h...
Alex Elder
11:35 AM Bug #4003: rbd: EBUSY errors from rbd unmap
I ran the kernel_untar_build.sh workunit using the
ceph "master" branch and the ceph-client "testing"
branch and go...
Alex Elder
11:13 AM Bug #4003: rbd: EBUSY errors from rbd unmap
I think I found *a* problem, possibly not *the* problem.
This commit:
bc7a62ee5 rbd: prevent open for image ...
Alex Elder
11:04 AM Bug #4003: rbd: EBUSY errors from rbd unmap
I am able to reproduce this problem by running
the kernel_untar_build.sh workunit.
I ran the test using the ceph ...
Alex Elder
08:53 AM Bug #4003: rbd: EBUSY errors from rbd unmap
> Alex, unless there is another high priority regression, can you
> look at this first?
Yes I will.
Alex Elder
08:52 AM Bug #4003: rbd: EBUSY errors from rbd unmap
Sam Lang wrote:
> I was able to verify that this happens with an older version of teuthology, one without the change...
Sage Weil
08:41 AM Bug #4003: rbd: EBUSY errors from rbd unmap
I was able to verify that this happens with an older version of teuthology, one without the changes I've made recentl... Sam Lang
05:18 AM Bug #4003: rbd: EBUSY errors from rbd unmap
I had the impression this might be a problem that
is holding up completion of the nightly test suite.
But I'm not...
Alex Elder
01:47 PM Feature #4021 (Resolved): rbd: openstack: add ability to copy volume to image for rbd
Ian Colle
01:46 PM Subtask #4020 (Resolved): rbd: openstack: simplify volume booting with new api: make image boot b...
Ian Colle
01:44 PM Subtask #4019 (Resolved): rbd: openstack: simplify volume booting with new api: add boot option t...
Ian Colle
01:44 PM Subtask #4018 (Resolved): rbd: openstack: simplify volume booting with new api: modify boot panel...
Ian Colle
01:42 PM Feature #4017 (Resolved): rbd: openstack: simplify volume booting with new api
Ian Colle
01:42 PM Feature #4013 (In Progress): rbd: openstack: extend nova boot api to support going from image to ...
Ian Colle
01:24 PM Feature #4013 (Resolved): rbd: openstack: extend nova boot api to support going from image to volume
Ian Colle
01:41 PM Subtask #4016 (Resolved): rbd: openstack: extend nova boot api: modify libvirt driver to support ...
Ian Colle
01:40 PM Subtask #4015 (Resolved): rbd: openstack: extend nova boot api: add block_dev_mapping_v2 to nova-...
Ian Colle
01:40 PM Subtask #4014 (Resolved): rbd: openstack: extend nova boot api: add block_dev_mapping_v2 to nova-api
Ian Colle
01:13 PM Bug #4012 (Won't Fix): rbd: image creation behaviour has to be uniform across bobtail and argonau...
rbd allows images to be created with size 0 in bobtail, but it fails in argonaut.
similarly,while in bobtail it do...
Tamilarasi muthamizhan
12:52 PM Bug #4010 (Fix Under Review): krbd: turn off interrupts for open/remove locking
Posted for review.
[PATCH] rbd: turn off interrupts for open/remove locking
Alex Elder
12:49 PM Bug #4010 (Resolved): krbd: turn off interrupts for open/remove locking
This fix is done. The problem was discovered while
investigating http://tracker.ceph.com/issues/4003.
This commi...
Alex Elder
10:12 AM Bug #3697: rbd copy.sh test failing in nightly
recent log : ubuntu@teuthology:/a/teuthology-2013-02-04_20:00:03-regression-bobtail-master-basic/15773 Tamilarasi muthamizhan
07:42 AM Subtask #4007 (Resolved): libceph: support STAT osd operation
In order to do layered writes we need to check whether
an object to be written exists before issuing the write.
Thi...
Alex Elder

02/04/2013

05:53 PM Bug #4003: rbd: EBUSY errors from rbd unmap
I really can't tell who's got a watch on the header
object. It should be getting removed when the object
gets unma...
Alex Elder
05:02 PM Bug #4003: rbd: EBUSY errors from rbd unmap
There is clearly something that is keeping the rbd image
from getting removed. I reproduced this with just running
...
Alex Elder
04:12 PM Bug #4003 (In Progress): rbd: EBUSY errors from rbd unmap
This sounds familiar, but I'm going to look a little
more closely to see if I can learn why it's happening.
Alex Elder
04:03 PM Bug #4003 (Resolved): rbd: EBUSY errors from rbd unmap
From the teuthology kernel untar task on rbd, we get EBUSY trying to unmap. I'm investigating that this isn't someho... Sam Lang

02/01/2013

02:18 PM Subtask #3741 (Resolved): krbd: rework request tracking code
commit 9ac90ea3d8dd6ab82f3665a132ca29e6ada56ad8
Author: Alex Elder <elder@inktank.com>
Date: Thu Nov 22 00:00:08 ...
Alex Elder
02:17 PM Feature #3754 (Closed): krbd: use new request tracking code for notify ack
commit 1c8c3c5c571607a188203142020d80aa58e5e280
Author: Alex Elder <elder@inktank.com>
Date: Fri Nov 30 17:53:04 ...
Alex Elder
02:16 PM Tasks #3755: krbd: use new request tracking code for sync object operations
commit 5d08568324f53368f927cc10927b1b105533c044
Author: Alex Elder <elder@inktank.com>
Date: Thu Jan 17 12:25:27 ...
Alex Elder
01:44 PM Tasks #3755 (Resolved): krbd: use new request tracking code for sync object operations
commit 304819b1a49937753ee01aa7ccf8d66547a0be36
Author: Alex Elder <elder@inktank.com>
Date: Sat Jan 19 00:30:28 ...
Alex Elder
02:11 PM Feature #3877 (Closed): krbd: don't wait for notify ack to complete
commit a8a34efcac7a33e7631fe8bf25530bd4be0417f8
Author: Alex Elder <elder@inktank.com>
Date: Thu Jan 17 12:18:46 ...
Alex Elder
01:38 PM Bug #3940 (Resolved): krbd: decrement obj request count when deleting
commit 150fde1984ec8454c163e4f89a50416cd68edbc4
Author: Alex Elder <elder@inktank.com>
Date: Fri Jan 25 17:08:55 ...
Alex Elder
01:38 PM Bug #3937 (Resolved): krbd: crash in rbd_assert(osd_req == obj_request->osd_req)
commit 8d93192992301f8c3a288c8cf4dc8598ac4b8427
Author: Alex Elder <elder@inktank.com>
Date: Fri Jan 25 17:08:55 ...
Alex Elder
01:37 PM Bug #3427 (Resolved): krbd: unmap does not remove block device properly
commit bc7a62ee52cffc735cb8383b6d26648883f1a01e
Author: Alex Elder <elder@inktank.com>
Date: Mon Jan 14 12:43:31 ...
Alex Elder
12:48 PM Bug #1740 (Resolved): krbd: don't return head data when reading from a non-existent snapshot
This was fixed a while ago. Josh Durgin
11:58 AM Bug #3980 (Won't Fix): rbd image created with size zero on a mixed cluster crashes rbd
creating a rbd image with size 0 is allowed in bobtail but not on argonaut.
on a mixed cluster running argonaut[bu...
Tamilarasi muthamizhan

01/31/2013

08:51 PM Bug #3978 (Resolved): krbd qa: concurrent.sh test leaves something read-only
I don't know what exactly is happening here, but it appears
that after running the "rbd/concurrent.sh" workunit, if
...
Alex Elder
02:21 PM Bug #3975 (Rejected): librbd: xfstests 008 failed inside qemu
This one's not a problem. This test pokes random holes in a
file (or maybe fills random spots). And when done it s...
Alex Elder
02:05 PM Bug #3975 (Rejected): librbd: xfstests 008 failed inside qemu
From xfstests output in ubuntu@teuthology:/a/teuthology-2013-01-29_20:00:04-regression-bobtail-master-basic/7794/remo... Josh Durgin
 

Also available in: Atom