Activity
From 01/20/2016 to 02/18/2016
02/18/2016
- 09:45 PM Cleanup #14809 (Resolved): ImageWatcher unwatch and flush should use async librados methods
- See https://github.com/ceph/ceph/pull/7649 for new API
- 05:55 AM Bug #10773: dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 fails
- http://pulpito.ceph.com/loic-2016-02-16_22:05:19-rbd-hammer-backports---basic-multi/12638/...
02/17/2016
- 04:37 PM Bug #14789 (Resolved): Truncate of non-existent object results in object map flagged as exists
- This won't affect the proper read of data but it will incorrectly affect the disk usage calculations (and it introduc...
- 02:54 AM Cleanup #14781 (New): Reuse ImageCtx::op_work_queue for TaskFinisher's finisher thread
- Convert any remaining tasks to be fully async so that the extra thread can be removed from librbd.
- 01:19 AM Bug #14780 (Fix Under Review): librbd: TaskFinisher lifetime no longer matches ImageWatcher
- https://github.com/ceph/ceph/pull/7667
- 01:13 AM Bug #14780 (Resolved): librbd: TaskFinisher lifetime no longer matches ImageWatcher
- Since notify handling was made async from the librados threads in commit:d898995b0e3ea301b1325f68a0532d57afa3c816 tes...
02/12/2016
- 01:37 PM Bug #14373 (Resolved): Cannot reliably create snapshot after freezing QEMU IO
- 01:37 PM Bug #14030 (Resolved): rbd merge-diff doesn't properly handle >2GB diffs
- 01:35 PM Bug #13491 (Resolved): misdirected op in rbd balance-reads test
- 05:42 AM Backport #14610 (Resolved): infernalis: Flattening an rbd image with a running Qemu instances cau...
- 05:38 AM Backport #14610 (In Progress): infernalis: Flattening an rbd image with a running Qemu instances ...
- 05:40 AM Backport #14062 (Resolved): infernalis: rbd merge-diff doesn't properly handle >2GB diffs
- 05:38 AM Backport #14062 (In Progress): infernalis: rbd merge-diff doesn't properly handle >2GB diffs
- 05:40 AM Backport #14542 (Resolved): infernalis: Cannot reliably create snapshot after freezing QEMU IO
- 05:38 AM Backport #14542 (In Progress): infernalis: Cannot reliably create snapshot after freezing QEMU IO
- 05:36 AM Backport #13733 (Resolved): misdirected op in rbd balance-reads test
02/11/2016
- 04:32 PM Subtask #14739 (Can't reproduce): Support deleting a journal when its being replayed by the RBD m...
- Attempting to delete an image which is being mirrored will not work if the RBD mirroring daemon is watching the journ...
- 04:26 PM Feature #14738 (Resolved): Optionally unregister "laggy" journal clients
- Support an optional configuration setting for the maximum number of object sets a journal client can be behind before...
- 04:37 AM Bug #14280 (Resolved): cls_rbd: object_map_save should enable checksums
- 04:37 AM Bug #14164 (Resolved): [ FAILED ] TestLibRBD.SnapRemoveViaLockOwner
- 02:12 AM Tasks #13959: add python3-specific packages for rbd and rados python bindings
- The conversion of rbd.py to cython makes this a little more complex - we'll need to build separate versions of the rb...
02/10/2016
- 01:18 AM Bug #14147 (Closed): coredumps in rbd-jewel-distro-basic-mira
- no longer happening on jewel
- 01:16 AM Bug #8385: RBD / QEMU Crash: Invalid fastbin entry (free)
- This may be related to http://tracker.ceph.com/issues/14620
02/09/2016
- 03:40 PM Backport #14620: hammer: unsafe use of libnss PK11_GetBestSlot()
- Maybe http://tracker.ceph.com/issues/14367 is the corresponding issue ?
- 06:34 AM Backport #14620 (Fix Under Review): hammer: unsafe use of libnss PK11_GetBestSlot()
- 06:22 AM Backport #14620 (In Progress): hammer: unsafe use of libnss PK11_GetBestSlot()
- 06:21 AM Backport #14620: hammer: unsafe use of libnss PK11_GetBestSlot()
PK11_GetBestSlot() does not appear to be threadsafe, and can be called from multiple threads, leading to a crash si...- 06:14 AM Bug #14476 (Resolved): rbd: TaskFinisher::cancel should remove event from SafeTimer
- 06:14 AM Feature #13633 (Resolved): Verify self-managed snapshot functionality on image create
- 06:14 AM Bug #14452 (Resolved): rbd-replay does not check for EOF and goes to endless loop
- 05:57 AM Backport #14142 (Resolved): infernalis: Verify self-managed snapshot functionality on image create
- 05:57 AM Backport #14199 (Resolved): infernalis: [ FAILED ] TestLibRBD.SnapRemoveViaLockOwner
- 05:57 AM Backport #14321 (Resolved): infernalis: cls_rbd: object_map_save should enable checksums
02/08/2016
- 08:43 PM Feature #14700 (Rejected): rbd map should optionally support the advisory locking and blacklisting
- Original proposal:...
- 03:26 PM Backport #14465 (Resolved): infernalis: rbd-replay does not check for EOF and goes to endless loop
- 03:26 PM Backport #14552 (Resolved): infernalis: rbd: TaskFinisher::cancel should remove event from SafeTimer
02/05/2016
- 04:29 PM Subtask #14673 (Resolved): Asynchronously open and close image replayers
- As new mirrored images are detected and old images are no longer mirrored, asynchronously start / stop the replayer.
- 04:50 AM Subtask #14663 (Resolved): Integrate librbd with journal tag allocation
- Prior to unblocking writes, a new tag should be allocated in the journal. This tag will link to the predecessor tag t...
- 04:46 AM Subtask #13298 (Fix Under Review): Journal registered client entry should store bufferlist payload
- *master PR*: https://github.com/ceph/ceph/pull/7529
- 04:45 AM Subtask #14412 (Fix Under Review): Track epoch and period for mirrored images
- 04:45 AM Subtask #14412: Track epoch and period for mirrored images
- *master PR*: https://github.com/ceph/ceph/pull/7529
02/04/2016
- 08:11 PM Subtask #14419 (In Progress): RBD mirroring settings should be moved to new pool object
- 07:59 AM Bug #14483: Flattening an rbd image with a running Qemu instances causes librbd worker threads to...
- Sure thing, thanks for picking this up. Let me know if you need more data - I can reproduce the issue easily.
02/03/2016
- 04:46 PM Documentation #14629 (Closed): rbd-replay documentation should be updated
- Now that tracing is optionally enabled, document the new configuration option (rbd tracing = true). Also note that i...
- 04:43 PM Documentation #14628 (Closed): libvirt documentation should suggest log path, admin socket, and S...
- Update the libvirt documentation to suggest "log path = xyz" and "admin socket = xyz" under the appropriate [client.<...
- 08:41 AM Backport #14620: hammer: unsafe use of libnss PK11_GetBestSlot()
- @Loic, this appears to be a backport of https://github.com/ceph/ceph/pull/3896 which has no associated tracker issue.
- 01:20 AM Backport #14620 (Resolved): hammer: unsafe use of libnss PK11_GetBestSlot()
- https://github.com/ceph/ceph/pull/7488
- 05:45 AM Bug #14225 (Resolved): rbd: fix bench-write
- 01:24 AM Bug #6480: librbd crashed qemu-system-x86_64
- This resurfaced in a similar form on a hammer cluster. After some analysis of the core file, it appeared that some of...
02/02/2016
- 08:15 PM Backport #14611 (Fix Under Review): hammer: Flattening an rbd image with a running Qemu instances...
- 06:46 PM Backport #14611 (In Progress): hammer: Flattening an rbd image with a running Qemu instances caus...
- 03:57 PM Backport #14611 (Resolved): hammer: Flattening an rbd image with a running Qemu instances causes ...
- https://github.com/ceph/ceph/pull/7485
- 07:56 PM Backport #14610 (Fix Under Review): infernalis: Flattening an rbd image with a running Qemu insta...
- 06:46 PM Backport #14610 (In Progress): infernalis: Flattening an rbd image with a running Qemu instances ...
- 03:57 PM Backport #14610 (Resolved): infernalis: Flattening an rbd image with a running Qemu instances cau...
- https://github.com/ceph/ceph/pull/7484
- 03:56 PM Bug #14483 (Pending Backport): Flattening an rbd image with a running Qemu instances causes librb...
- 03:56 PM Bug #14483: Flattening an rbd image with a running Qemu instances causes librbd worker threads to...
- Christian, thank you for the logs!
- 03:08 PM Bug #14483 (In Progress): Flattening an rbd image with a running Qemu instances causes librbd wor...
- 03:09 PM Bug #14482 (Resolved): rbd:rbd-nbd does not have an argument check for nbds_max
- *PR*: https://github.com/ceph/ceph/pull/7337
02/01/2016
- 06:58 PM Bug #14595 (New): Race between releasing exclusive lock and incoming write ops
- Should verify it's not an issue in master due to the major refactoring of librbd.
- 06:56 PM Bug #14595 (Resolved): Race between releasing exclusive lock and incoming write ops
- A thread releasing the exclusive lock (and flushing the cache) might race with a thread submitting a new write operat...
- 04:42 PM Feature #14593 (New): Support optional alignment of journal entries
- Writing out 4K aligned journal entries will avoid the potential for un-aligned (two block) writes in the OSD.
- 03:42 PM Feature #13274 (Resolved): LTTng-UST should be optionally enabled
- 04:59 AM Bug #14058: Can't build Ceph with --without-rbd and --with-debug
- please note this also applies to hammer. but guess "rgw" is enabled most of the time, so we don't backport the fix..
01/30/2016
- 08:01 PM Backport #14143 (Resolved): hammer: Verify self-managed snapshot functionality on image create
- 08:01 PM Backport #14283 (Resolved): hammer: rbd: fix bench-write
- 08:01 PM Backport #14466 (Resolved): hammer: rbd-replay does not check for EOF and goes to endless loop
- 08:01 PM Backport #14553 (Resolved): hammer: rbd: TaskFinisher::cancel should remove event from SafeTimer
- 08:00 PM Backport #14543 (Resolved): hammer: Cannot reliably create snapshot after freezing QEMU IO
- 08:00 PM Backport #14063 (Resolved): hammer: rbd merge-diff doesn't properly handle >2GB diffs
01/29/2016
- 10:12 AM Backport #14321 (In Progress): infernalis: cls_rbd: object_map_save should enable checksums
- 10:09 AM Backport #14465 (In Progress): infernalis: rbd-replay does not check for EOF and goes to endless ...
- 10:07 AM Backport #14552 (In Progress): infernalis: rbd: TaskFinisher::cancel should remove event from Saf...
- 03:50 AM Backport #14552 (Resolved): infernalis: rbd: TaskFinisher::cancel should remove event from SafeTimer
- https://github.com/ceph/ceph/pull/7426
- 05:23 AM Backport #14553 (In Progress): hammer: rbd: TaskFinisher::cancel should remove event from SafeTimer
- 03:50 AM Backport #14553 (Resolved): hammer: rbd: TaskFinisher::cancel should remove event from SafeTimer
- https://github.com/ceph/ceph/pull/7417
- 05:21 AM Backport #14466 (In Progress): hammer: rbd-replay does not check for EOF and goes to endless loop
- 01:50 AM Bug #14522 (Duplicate): RBD Snapshot locks Libvirt Disk With Active Writes
- Excellent -- thanks for all your help with debugging. Closing this ticket as dup of issue #13755.
- 12:13 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Hello Jason/Josh,
I can confirm both of those changes will fix the issue. Thanks for your assistance! - 01:43 AM Tasks #14551 (Resolved): Update RBD default features
- For Jewel, recommend updating default features to include exclusive lock, object map, fast-diff, and deep-flatten fea...
- 01:41 AM Tasks #14550 (Resolved): Deprecate format v1 RBD images
- For Jewel, recommend that we warn on the creation and opening of v1 RBD images.
01/28/2016
- 11:02 PM Backport #14543 (Fix Under Review): hammer: Cannot reliably create snapshot after freezing QEMU IO
- 07:44 PM Backport #14543 (In Progress): hammer: Cannot reliably create snapshot after freezing QEMU IO
- 07:43 PM Backport #14543 (Resolved): hammer: Cannot reliably create snapshot after freezing QEMU IO
- https://github.com/ceph/ceph/pull/7407
- 10:07 PM Backport #14542 (Fix Under Review): infernalis: Cannot reliably create snapshot after freezing QE...
- 07:43 PM Backport #14542 (In Progress): infernalis: Cannot reliably create snapshot after freezing QEMU IO
- 07:43 PM Backport #14542 (Resolved): infernalis: Cannot reliably create snapshot after freezing QEMU IO
- https://github.com/ceph/ceph/pull/7406
- 07:42 PM Bug #14373 (Pending Backport): Cannot reliably create snapshot after freezing QEMU IO
- 02:31 AM Bug #14373 (In Progress): Cannot reliably create snapshot after freezing QEMU IO
- Sorry for the delay, I somehow missed the email notification for this ticket. The logs are perfect and I can see the...
- 04:11 PM Feature #14540 (Resolved): rbd disk-usage CLI command should support calculating full image usage
- Right now you would either need to dump the full pool or re-run "rbd disk-usage" for each snapshot within an image to...
- 04:10 PM Documentation #14539 (Resolved): rbd CLI man page is missing several commands
- Ensure it is up-to-date with the latest commands in the CLI.
- 12:19 PM Bug #13674 (Resolved): Avoid re-writing old-format image header on resize
- 12:19 PM Bug #13636 (Resolved): rbd: pure virtual method called
- 12:19 PM Bug #13164 (Resolved): librbd: reads larger than cache size hang
- 12:19 PM Bug #12415 (Resolved): init-rbdmap uses distro-specific functions
- 12:17 PM Backport #13752 (Rejected): Avoid re-writing old-format image header on resize
- 12:17 PM Backport #13757 (Rejected): rbd: pure virtual method called
- 12:17 PM Backport #13694 (Rejected): init-rbdmap uses distro-specific functions
- 12:17 PM Backport #13388 (Rejected): librbd: reads larger than cache size hang
- 03:53 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- For OpenStack, to enable rbd caching you'll need to put this in the [libvirt] section of /etc/nova/nova.conf on each ...
- 02:06 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- I might have (semi-)good news for you: I believe this is the same issue as tracked under #13755. The fix is already ...
- 12:24 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- cache='none' is disabling the cache.
- 12:19 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Hello,
RBD Cache is enabled in ceph.conf (rbd_cache = true, rbd_cache_size = 67108864) for the client on all hyper... - 12:11 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Is this only occurring on VMs where the RBD cache is disabled?
- 12:05 AM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Perfect, thanks!
- 03:05 AM Bug #14535 (Resolved): rbd CLI does not validate pool and snapshot name optionals
- The "--pool", "--snap", etc optionals are not validated to ensure they don't contain "/" nor "@".
- 02:11 AM Bug #14510 (Resolved): TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED assert(m...
- 02:10 AM Bug #14434 (Resolved): intermittent errors in make check tests from journaling code
01/27/2016
- 11:46 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Tyler Wilson wrote:
> Hello Jason,
>
> I got the instance to lock once more and ran gdb attach $pid, then "thread... - 11:37 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Hello Jason,
I got the instance to lock once more and ran gdb attach $pid, then "thread apply all bt". Attached is... - 11:17 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Another option would be if you could install the ceph-debuginfo package, attach gdb to the hung qemu process, run "th...
- 11:03 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Correct;...
- 10:46 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Odd, no file named /var/log/qemu/qemu-guest-290624.log was created (based on the PID of your admin socket)? Can you ...
- 09:28 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Updated ceph.conf to that document;...
- 08:32 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Try updating your ceph.conf as described here: http://docs.ceph.com/docs/master/rbd/rbd-openstack/#configuring-nova
- 08:17 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- SELinux is enabled however no denies are in the audit log. I disabled SELinux and retried; same result.
- 08:04 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Hmm -- do you have SElinux enabled? You might be experiencing an SElinux denial preventing qemu from writing to /var/...
- 07:21 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Hello Jason,
Can't seem to get the logs that you are looking for, where can I retrieve that? Here is my current ce... - 07:04 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Thanks -- I was actually hoping to get the librbd logs from the qemu process itself before it locks up due to the sna...
- 06:34 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Hello Jason,
Thanks for taking assignment of this. Here is the output from creating a snapshot with debug enabled;... - 05:35 PM Bug #14522: RBD Snapshot locks Libvirt Disk With Active Writes
- Tyler, would it be possible for you to attach the librbd log from qemu with "debug rbd = 20" from shortly before crea...
- 05:27 PM Bug #14522 (In Progress): RBD Snapshot locks Libvirt Disk With Active Writes
- 12:25 AM Bug #14522 (Duplicate): RBD Snapshot locks Libvirt Disk With Active Writes
- On the latest EL7 release of Libvirt (with RDO OpenStack Kilo) I am getting my RBD disks locked when I issue a simple...
- 07:34 PM Subtask #14530 (Resolved): Journal close should be non-blocking
- The RBD replay daemon will potentially have to close thousands of journals on shutdown. To be more efficient, this p...
- 07:17 PM Bug #14510 (Fix Under Review): TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED ...
- *master PR*: https://github.com/ceph/ceph/pull/7382
- 05:58 PM Bug #14510 (In Progress): TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED asser...
- 05:58 PM Bug #14510: TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED assert(m_state == S...
- Whoops -- I was thinking this was a different issue I just fixed with Journal::handle_op_event_safe.
- 02:23 PM Bug #14476 (Pending Backport): rbd: TaskFinisher::cancel should remove event from SafeTimer
- 01:25 AM Bug #14373: Cannot reliably create snapshot after freezing QEMU IO
- Having similar issue on http://tracker.ceph.com/issues/14522
01/26/2016
- 03:18 PM Bug #14510: TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED assert(m_state == S...
- I see it in today master: https://github.com/ceph/ceph/blob/master/src/librbd/Journal.cc#L376
- 02:57 PM Bug #14510 (Need More Info): TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED as...
- 02:56 PM Bug #14510: TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED assert(m_state == S...
- I think you have an old version of code -- "assert(m_state == STATE_READY)" doesn't exist in the codebase.
- 10:29 AM Bug #14510: TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED assert(m_state == S...
- Below is a debug log for the crash. librbd::SnapshotCreateRequest may be called when the journal has not finished the...
- 09:57 AM Bug #14510 (Resolved): TestLibRBD.ObjectMapConsistentSnap librbd/Journal.cc: 376: FAILED assert(m...
- I saw this in a jenkins test: https://jenkins.ceph.com/job/ceph-pull-requests/545/console
And running the test man... - 02:55 PM Backport #14512 (Resolved): hammer: snapshot, flatten, and rollback of cloned image leads to data...
- https://github.com/ceph/ceph/pull/8535
01/24/2016
- 05:15 PM Bug #14483: Flattening an rbd image with a running Qemu instances causes librbd worker threads to...
- I ran the VM again without dropping privileges and without chroot. As soon as the image was flattened, all IO stalled...
- 05:07 PM Bug #14483: Flattening an rbd image with a running Qemu instances causes librbd worker threads to...
- Something else that I'm absolutely unsure whether it's relevant: we run Qemu with dropping privileges to 'noboyd' and...
- 05:01 PM Bug #14483: Flattening an rbd image with a running Qemu instances causes librbd worker threads to...
- Here's the full log of the last days, with multiple instances, including the ones I mentioned above:
http://shared00... - 04:57 PM Bug #14483 (Resolved): Flattening an rbd image with a running Qemu instances causes librbd worker...
- This Ceph 0.94.5 and Qemu 2.2.1 with some backported live-migration fixes (but Live migration isn't involved here).
...
01/23/2016
- 02:24 PM Bug #14482 (Resolved): rbd:rbd-nbd does not have an argument check for nbds_max
- 1.The value of argument nbds_max can be set to -1, which logically can not be understood. I think 0 is the boundary.
...
01/22/2016
- 07:28 PM Bug #14476 (Fix Under Review): rbd: TaskFinisher::cancel should remove event from SafeTimer
- 07:27 PM Bug #14476: rbd: TaskFinisher::cancel should remove event from SafeTimer
- https://github.com/ceph/ceph/pull/7329
- 07:21 PM Bug #14476 (Resolved): rbd: TaskFinisher::cancel should remove event from SafeTimer
- TaskFinisher needs to call SafeTimer::cancel_event, otherwise canceled events fire anyway.
- 01:31 AM Backport #14466 (Resolved): hammer: rbd-replay does not check for EOF and goes to endless loop
- https://github.com/ceph/ceph/pull/7416
- 01:30 AM Backport #14465 (Resolved): infernalis: rbd-replay does not check for EOF and goes to endless loop
- https://github.com/ceph/ceph/pull/7427
01/21/2016
- 02:12 PM Bug #14452 (Pending Backport): rbd-replay does not check for EOF and goes to endless loop
- 01:58 PM Bug #14452: rbd-replay does not check for EOF and goes to endless loop
- https://github.com/ceph/ceph/pull/7310
- 01:58 PM Bug #14452 (Resolved): rbd-replay does not check for EOF and goes to endless loop
- BufferReader::fetch() goes to endless loop because it does not check for EOF.
01/20/2016
- 09:54 PM Tasks #14442 (New): Create test cases in support of rbd-replay
- 09:39 PM Bug #14434 (Fix Under Review): intermittent errors in make check tests from journaling code
- *master PR*: https://github.com/ceph/ceph/pull/7302
- 07:26 PM Bug #14434 (In Progress): intermittent errors in make check tests from journaling code
- 05:41 AM Bug #14434 (Resolved): intermittent errors in make check tests from journaling code
- One example: http://gitbuilder.sepia.ceph.com/gitbuilder-ceph-tarball-trusty-amd64-basic/log.cgi?log=f3e58a054aed197b...
- 09:02 AM Bug #14373: Cannot reliably create snapshot after freezing QEMU IO
- One more observation: sometimes there are such records in qemu logs when instance starts hanging:
2016-01-20 10:44... - 08:58 AM Bug #14373: Cannot reliably create snapshot after freezing QEMU IO
- I also should say it is a very floating issue, meaning it may suddenly appear on one VM and after some time suddenly ...
- 07:50 AM Bug #14373: Cannot reliably create snapshot after freezing QEMU IO
- Here is it. Hope that will help.
2016-01-20 10:33:47.954001 7fadb64ac700 20 librbd::AioRequest: send_pre 0x56262fd...
Also available in: Atom