Project

General

Profile

Actions
Copy link

Bug #10773

closed

dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 fails

Added by Loïc Dachary about 9 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jason Dillaman
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
jewel, hammer
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://qa-proxy.ceph.com/teuthology/loic-2015-02-03_14:40:23-rbd-dumpling-backports---basic-multi/738732/

2015-02-05T09:34:57.816 INFO:tasks.workunit.client.0.plana62.stderr:+ codevers=trusty
2015-02-05T09:34:57.817 INFO:tasks.workunit.client.0.plana62.stderr:+ iotests=qemu-iotests
2015-02-05T09:34:57.817 INFO:tasks.workunit.client.0.plana62.stderr:+ testlist=001 002 003 004 005 008 009 010 011 021 025
2015-02-05T09:34:57.817 INFO:tasks.workunit.client.0.plana62.stderr:+ [ trusty = trusty ]
2015-02-05T09:34:57.817 INFO:tasks.workunit.client.0.plana62.stderr:+ iotests=qemu/tests/qemu-iotests
2015-02-05T09:34:57.818 INFO:tasks.workunit.client.0.plana62.stderr:+ [ qemu/tests/qemu-iotests = qemu/tests/qemu-iotests ]
2015-02-05T09:34:57.818 INFO:tasks.workunit.client.0.plana62.stderr:+ git clone git://apt-mirror.front.sepia.ceph.com/qemu.git
2015-02-05T09:34:57.819 INFO:tasks.workunit.client.0.plana62.stderr:Cloning into 'qemu'...
2015-02-05T09:35:20.192 INFO:tasks.workunit.client.0.plana62.stderr:+ cd qemu
2015-02-05T09:35:20.192 INFO:tasks.workunit.client.0.plana62.stderr:+ git checkout 2528043f1f299e0e88cb026f1ca7c40bbb4e1f80
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:Note: checking out '2528043f1f299e0e88cb026f1ca7c40bbb4e1f80'.
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:You are in 'detached HEAD' state. You can look around, make experimental
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:changes and commit them, and you can discard any commits you make in this
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:state without impacting any branches by performing another checkout.
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:
2015-02-05T09:35:20.536 INFO:tasks.workunit.client.0.plana62.stderr:If you want to create a new branch to retain commits you create, you may
2015-02-05T09:35:20.537 INFO:tasks.workunit.client.0.plana62.stderr:do so (now or later) by using b with the checkout command again. Example:
2015-02-05T09:35:20.537 INFO:tasks.workunit.client.0.plana62.stderr:
2015-02-05T09:35:20.537 INFO:tasks.workunit.client.0.plana62.stderr: git checkout -b new_branch_name
2015-02-05T09:35:20.537 INFO:tasks.workunit.client.0.plana62.stderr:
2015-02-05T09:35:20.537 INFO:tasks.workunit.client.0.plana62.stderr:HEAD is now at 2528043... Update version for v2.2.0-rc3 release
2015-02-05T09:35:20.539 INFO:tasks.workunit.client.0.plana62.stderr:+ cd ..
2015-02-05T09:35:20.539 INFO:tasks.workunit.client.0.plana62.stderr:+ testlist=001 002 003 004 005 008 009 010 011 021 025 032 033 055 077
2015-02-05T09:35:20.539 INFO:tasks.workunit.client.0.plana62.stderr:+ cd qemu/tests/qemu-iotests
2015-02-05T09:35:20.539 INFO:tasks.workunit.client.0.plana62.stderr:+ mkdir bin
2015-02-05T09:35:20.541 INFO:tasks.workunit.client.0.plana62.stderr:+ which qemu-system-x86_64
2015-02-05T09:35:20.542 INFO:tasks.workunit.client.0.plana62.stderr:+ ln -s /usr/bin/qemu-system-x86_64 bin/qemu
2015-02-05T09:35:20.558 INFO:tasks.workunit.client.0.plana62.stderr:+ touch common.env
2015-02-05T09:35:20.560 INFO:tasks.workunit.client.0.plana62.stderr:+ TEST_DIR=rbd PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/ubuntu/cephtest/mnt.0/client.0/tmp/qemu/tests/qemu-iotests/bin ./check -rbd 001 002 003 004 005 008 009 010 011 021 025 032 033 055 077
2015-02-05T09:35:20.849 INFO:tasks.workunit.client.0.plana62.stdout:QEMU - /home/ubuntu/cephtest/mnt.0/client.0/tmp/qemu/tests/qemu-iotests/bin/qemu
2015-02-05T09:35:20.849 INFO:tasks.workunit.client.0.plana62.stdout:QEMU_IMG -- /usr/bin/qemu-img
2015-02-05T09:35:20.849 INFO:tasks.workunit.client.0.plana62.stdout:QEMU_IO -- /usr/bin/qemu-io
2015-02-05T09:35:20.849 INFO:tasks.workunit.client.0.plana62.stdout:QEMU_NBD -- /usr/bin/qemu-nbd
2015-02-05T09:35:20.849 INFO:tasks.workunit.client.0.plana62.stdout:IMGFMT -- raw
2015-02-05T09:35:20.849 INFO:tasks.workunit.client.0.plana62.stdout:IMGPROTO -- rbd
2015-02-05T09:35:20.850 INFO:tasks.workunit.client.0.plana62.stdout:PLATFORM -- Linux/x86_64 plana62 3.19.0-rc7-ceph-00020-g9e0960f
2015-02-05T09:35:20.850 INFO:tasks.workunit.client.0.plana62.stdout:SOCKET_SCM_HELPER --
2015-02-05T09:35:20.850 INFO:tasks.workunit.client.0.plana62.stdout:
2015-02-05T09:35:29.871 INFO:tasks.workunit.client.0.plana62.stdout:001
2015-02-05T09:35:39.027 INFO:tasks.workunit.client.0.plana62.stdout:002
2015-02-05T09:35:47.008 INFO:tasks.workunit.client.0.plana62.stdout:003
2015-02-05T09:35:48.283 INFO:tasks.workunit.client.0.plana62.stdout:004
2015-02-05T09:49:36.610 INFO:tasks.workunit.client.0.plana62.stdout:005
2015-02-05T09:49:45.221 INFO:tasks.workunit.client.0.plana62.stdout:008
2015-02-05T09:49:46.769 INFO:tasks.workunit.client.0.plana62.stdout:009
2015-02-05T09:49:48.364 INFO:tasks.workunit.client.0.plana62.stdout:010
2015-02-05T09:49:51.836 INFO:tasks.workunit.client.0.plana62.stdout:011
2015-02-05T09:49:55.036 INFO:tasks.workunit.client.0.plana62.stdout:021
2015-02-05T09:50:04.627 INFO:tasks.workunit.client.0.plana62.stdout:025
2015-02-05T09:50:06.316 INFO:tasks.workunit.client.0.plana62.stdout:032
2015-02-05T09:50:07.418 INFO:tasks.workunit.client.0.plana62.stdout:033
2015-02-05T09:50:20.306 INFO:tasks.workunit.client.0.plana62.stdout:055
2015-02-05T09:50:22.804 INFO:tasks.workunit.client.0.plana62.stdout:077 - output mismatch (see 077.out.bad)
2015-02-05T09:50:22.857 INFO:tasks.workunit.client.0.plana62.stdout:--- /home/ubuntu/cephtest/mnt.0/client.0/tmp/qemu/tests/qemu-iotests/077.out 2015-02-05 09:35:20.516152395 -0800
2015-02-05T09:50:22.857 INFO:tasks.workunit.client.0.plana62.stdout:+++ 077.out.bad 2015-02-05 09:50:22.690547507 -0800
2015-02-05T09:50:22.858 INFO:tasks.workunit.client.0.plana62.stdout:@ -42,9 +42,9 @
2015-02-05T09:50:22.858 INFO:tasks.workunit.client.0.plana62.stdout: wrote XXX/XXX bytes at offset XXX
2015-02-05T09:50:22.858 INFO:tasks.workunit.client.0.plana62.stdout: XXX bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2015-02-05T09:50:22.858 INFO:tasks.workunit.client.0.plana62.stdout: blkdebug: Resuming request 'A'
2015-02-05T09:50:22.858 INFO:tasks.workunit.client.0.plana62.stdout:+blkdebug: Resuming request 'B'
2015-02-05T09:50:22.859 INFO:tasks.workunit.client.0.plana62.stdout: wrote XXX/XXX bytes at offset XXX
2015-02-05T09:50:22.861 INFO:tasks.workunit.client.0.plana62.stdout: XXX bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2015-02-05T09:50:22.861 INFO:tasks.workunit.client.0.plana62.stdout:-blkdebug: Resuming request 'B'
2015-02-05T09:50:22.862 INFO:tasks.workunit.client.0.plana62.stdout: wrote XXX/XXX bytes at offset XXX
2015-02-05T09:50:22.862 INFO:tasks.workunit.client.0.plana62.stdout: XXX bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2015-02-05T09:50:22.862 INFO:tasks.workunit.client.0.plana62.stdout: wrote XXX/XXX bytes at offset XXX
2015-02-05T09:50:22.982 INFO:tasks.workunit.client.0.plana62.stdout:Failures: 077
2015-02-05T09:50:22.982 INFO:tasks.workunit.client.0.plana62.stdout:Failed 1 of 15 tests

Related issues 2 (0 open — 2 closed)

Copied to rbd - Backport #18221: jewel: dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 failsResolvedJason DillamanActions
Copied to rbd - Backport #18222: hammer: dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 failsResolvedNathan CutlerActions
Actions
Copy link
 #1

Updated by Loïc Dachary about 9 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Loïc Dachary about 9 years ago

The tests are from a given commit in the qemu git repository, this is stable.

git checkout 2528043f1f299e0e88cb026f1ca7c40bbb4e1f80

Actions
Copy link
 #3

Updated by Loïc Dachary about 9 years ago

qa/workunits/rbd/qemu-iotests.sh checkout 2528043f1f299e0e88cb026f1ca7c40bbb4e1f80 but uses the qemu binary that is found on the machine running the test, which is bound to lead to some variance.

Actions
Copy link
 #4

Updated by Loïc Dachary about 9 years ago 

$ git log --patch 2528043f1f299e0e88cb026f1ca7c40bbb4e1f80..master -- tests/qemu-iotests/077
commit 8f9e835fd2e687d2bfe936819c3494af4343614d
Author: Kevin Wolf <kwolf@redhat.com>
Date:   Thu Nov 20 16:27:07 2014 +0100

    qemu-iotests: Use qemu-io -f $IMGFMT

    This patch changes $QEMU_IO so that all tests by default pass a format
    argument to qemu-io.

    There are a few cases where -f $IMGFMT is not wanted because it selects
    the wrong driver or json: filenames including a driver are used. They
    are changed to use $QEMU_IO_PROG, which doesn't include any options.

    Tests 071 and 081 have output changes because now the actual request
    fails instead of reading the 2k probing buffer.

    Signed-off-by: Kevin Wolf <kwolf@redhat.com>
    Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
    Reviewed-by: Max Reitz <mreitz@redhat.com>
    Message-id: 1416497234-29880-3-git-send-email-kwolf@redhat.com
    Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
    Signed-off-by: Kevin Wolf <kwolf@redhat.com>

diff --git a/tests/qemu-iotests/077 b/tests/qemu-iotests/077
index 4dd1bdd..42a9085 100755
--- a/tests/qemu-iotests/077
+++ b/tests/qemu-iotests/077
@@ -52,7 +52,7 @@ echo "== Some concurrent requests involving RMW ==" 

 function test_io()
 {
-echo "open -o file.align=4k blkdebug::$TEST_IMG" 
+echo "open -o driver=$IMGFMT,file.align=4k blkdebug::$TEST_IMG" 
 # A simple RMW request
 cat  <<EOF
 aio_write -P 10 0x200 0x200
Actions
Copy link
 #5

Updated by Loïc Dachary about 9 years ago 

ubuntu@plana62:~$ /usr/bin/qemu-system-x86_64 --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.10), Copyright (c) 2003-2008 Fabrice Bellard
Actions
Copy link
 #6

Updated by Jason Dillaman about 9 years ago

I wasn't able to reproduce this on my Trusty VM. I appear to have the same qemu installed as plana62:

jdillaman@ubuntu:~/ceph/src$ dpkg --list | grep qemu-
ii  qemu-keymaps                           2.0.0+dfsg-2ubuntu1.10                all          QEMU keyboard maps
ii  qemu-system-common                     2.0.0+dfsg-2ubuntu1.10                amd64        QEMU full system emulation binaries (common files)
ii  qemu-system-x86                        2.0.0+dfsg-2ubuntu1.10                amd64        QEMU full system emulation binaries (x86)
ii  qemu-utils                             2.0.0+dfsg-2ubuntu1.10                amd64        QEMU utilities
Actions
Copy link
 #7

Updated by Loïc Dachary about 9 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link
 #8

Updated by Loïc Dachary about 8 years ago

http://pulpito.ceph.com/loic-2016-02-16_22:05:19-rbd-hammer-backports---basic-multi/12638/

2016-02-17T00:26:00.636 INFO:tasks.workunit.client.0.smithi007.stdout:077         - output mismatch (see 077.out.bad)
2016-02-17T00:26:00.637 INFO:tasks.workunit.client.0.smithi007.stdout:--- /home/ubuntu/cephtest/mnt.0/client.0/tmp/qemu/tests/qemu-iotests/077.out    2016-02-17 08:22:00.396233597 +0000
2016-02-17T00:26:00.637 INFO:tasks.workunit.client.0.smithi007.stdout:+++ 077.out.bad    2016-02-17 08:26:00.491171157 +0000
2016-02-17T00:26:00.638 INFO:tasks.workunit.client.0.smithi007.stdout:@@ -50,9 +50,9 @@
2016-02-17T00:26:00.638 INFO:tasks.workunit.client.0.smithi007.stdout: wrote XXX/XXX bytes at offset XXX
2016-02-17T00:26:00.638 INFO:tasks.workunit.client.0.smithi007.stdout: XXX bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2016-02-17T00:26:00.639 INFO:tasks.workunit.client.0.smithi007.stdout: blkdebug: Resuming request 'B'
2016-02-17T00:26:00.639 INFO:tasks.workunit.client.0.smithi007.stdout:+blkdebug: Resuming request 'A'
2016-02-17T00:26:00.639 INFO:tasks.workunit.client.0.smithi007.stdout: wrote XXX/XXX bytes at offset XXX
2016-02-17T00:26:00.639 INFO:tasks.workunit.client.0.smithi007.stdout: XXX bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2016-02-17T00:26:00.640 INFO:tasks.workunit.client.0.smithi007.stdout:-blkdebug: Resuming request 'A'
2016-02-17T00:26:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: wrote XXX/XXX bytes at offset XXX
2016-02-17T00:26:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: XXX bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2016-02-17T00:26:00.641 INFO:tasks.workunit.client.0.smithi007.stdout: wrote XXX/XXX bytes at offset XXX
2016-02-17T00:26:00.670 INFO:tasks.workunit.client.0.smithi007.stdout:Failures: 077
2016-02-17T00:26:00.670 INFO:tasks.workunit.client.0.smithi007.stdout:Failed 1 of 14 tests
2016-02-17T00:26:00.676 INFO:tasks.workunit:Stopping ['rbd/qemu-iotests.sh'] on client.0...
Actions
Copy link
 #9

Updated by Nathan Cutler almost 8 years ago

This resurfaced in an rbd run during integration testing of the hammer 0.94.8 release:

http://pulpito.ceph.com/smithfarm-2016-06-02_00:35:31-rbd-hammer-backports---basic-smithi/229818/

Actions
Copy link
 #11

Updated by Nathan Cutler over 7 years ago

  • Subject changed from dumpling: qemu/tests/qemu-iotests/077 fails to hammer: qemu/tests/qemu-iotests/077 fails
  • Status changed from Can't reproduce to New
  • Assignee set to Jason Dillaman
  • Priority changed from Normal to High

Based on the above, this bug has been around since dumpling (?) but, at least in the 0.94.10 integration testing branch, it appears to be somewhat reproducible. Jason, could you help debug? See http://tracker.ceph.com/issues/17151#note-12

Actions
Copy link
 #12

Updated by Nathan Cutler over 7 years ago

  • Subject changed from hammer: qemu/tests/qemu-iotests/077 fails to dumpling, hammer: qemu/tests/qemu-iotests/077 fails
Actions
Copy link
 #13

Updated by Samuel Just over 7 years ago

  • Subject changed from dumpling, hammer: qemu/tests/qemu-iotests/077 fails to dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 fails
Actions
Copy link
 #14

Updated by Samuel Just over 7 years ago

  • Project changed from Ceph to rbd
  • Category deleted (qemu)
Actions
Copy link
 #15

Updated by Jason Dillaman over 7 years ago

  • Priority changed from High to Normal
Actions
Copy link
 #16

Updated by Jason Dillaman over 7 years ago

  • Status changed from New to Fix Under Review
  • Backport set to jewel, hammer
Actions
Copy link
 #17

Updated by Mykola Golub over 7 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #18

Updated by Nathan Cutler over 7 years ago

  • Copied to Backport #18221: jewel: dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 fails added
Actions
Copy link
 #19

Updated by Nathan Cutler over 7 years ago

  • Copied to Backport #18222: hammer: dumpling, hammer, jewel: qemu/tests/qemu-iotests/077 fails added
Actions
Copy link
 #20

Updated by Nathan Cutler about 7 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF