Infrastructure » sepia

On power cycle

mira035 login: [   45.584519] EXT4-fs error (device sda1): ext4_mb_generate_buddy:758: group 3969, block bitmap and bg descriptor inconsistent: 30542 vs 30539 free clusters
[   45.598613] BUG: sleeping function called from invalid context at ./include/linux/buffer_head.h:358
[   45.607691] in_atomic(): 1, irqs_disabled(): 0, pid: 1444, name: pmdalinux
[   45.614589] 5 locks held by pmdalinux/1444:
[   45.618789]  #0:  (&sbi->s_journal_flag_rwsem){.+.+.+}, at: [<ffffffff811eceb1>] do_writepages+0x21/0x40
[   45.628513]  #1:  (jbd2_handle){++++..}, at: [<ffffffff81364311>] start_this_handle+0x351/0x440
[   45.637565]  #2:  (&ei->i_data_sem){++++..}, at: [<ffffffff81315067>] ext4_map_blocks+0x227/0x5f0
[   45.646769]  #3:  (&lg->lg_mutex){+.+...}, at: [<ffffffff8134b44a>] ext4_mb_initialize_context+0x17a/0x1a0
[   45.656777]  #4:  (&(&bgl->locks[i].lock)->rlock){+.+...}, at: [<ffffffff8134d59d>] ext4_mb_init_cache+0x43d/0x890
[   45.667457] CPU: 6 PID: 1444 Comm: pmdalinux Tainted: G        W       4.9.0-ceph-g297eca11495e #1
[   45.676406] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.2a       06/27/2012
[   45.683545]  ffffc9000264b610 ffffffff8141a261 ffff880429be2580 ffffffff81c44d3c
[   45.691031]  ffffc9000264b638 ffffffff810db3c1 ffffffff81c44d3c 0000000000000166
[   45.698605]  0000000000000000 ffffc9000264b660 ffffffff810db452 ffff88042837b1a0
[   45.706120] Call Trace:
[   45.708578]  [<ffffffff8141a261>] dump_stack+0x85/0xc4
[   45.713715]  [<ffffffff810db3c1>] ___might_sleep+0x201/0x220
[   45.719373]  [<ffffffff810db452>] __might_sleep+0x72/0x90
[   45.724840]  [<ffffffff81335322>] ext4_commit_super+0x1b2/0x280
[   45.730759]  [<ffffffff81337856>] __ext4_grp_locked_error+0x136/0x230
[   45.737202]  [<ffffffff810db452>] ? __might_sleep+0x72/0x90
[   45.742790]  [<ffffffff8134cf3f>] ext4_mb_generate_buddy+0x21f/0x2f0
[   45.749140]  [<ffffffff8134d65d>] ext4_mb_init_cache+0x4fd/0x890
[   45.755146]  [<ffffffff8134db85>] ext4_mb_init_group+0x195/0x260
[   45.761151]  [<ffffffff8134dcfe>] ext4_mb_good_group+0xae/0x170
[   45.767067]  [<ffffffff8135087b>] ext4_mb_regular_allocator+0x16b/0x420
[   45.773752]  [<ffffffff8134c54a>] ? ext4_mb_use_preallocated+0x3ca/0x420
[   45.780449]  [<ffffffff813522f6>] ext4_mb_new_blocks+0x2a6/0xdb0
[   45.786454]  [<ffffffff8124c0bb>] ? __kmalloc+0x22b/0x2d0
[   45.791854]  [<ffffffff8134148e>] ? ext4_find_extent+0x29e/0x2c0
[   45.797858]  [<ffffffff8134148e>] ? ext4_find_extent+0x29e/0x2c0
[   45.803925]  [<ffffffff81346b1d>] ext4_ext_map_blocks+0x1b5d/0x21d0
[   45.810191]  [<ffffffff811ef14e>] ? release_pages+0x31e/0x3a0
[   45.815935]  [<ffffffff8135b013>] ? ext4_es_lookup_extent+0x1c3/0x350
[   45.822402]  [<ffffffff81315087>] ext4_map_blocks+0x247/0x5f0
[   45.828180]  [<ffffffff81107a7e>] ? __lock_is_held+0x3e/0x60
[   45.833841]  [<ffffffff81318d3c>] ext4_writepages+0x86c/0x1050
[   45.839671]  [<ffffffff811eceb1>] ? do_writepages+0x21/0x40
[   45.845247]  [<ffffffff81826747>] ? _raw_spin_unlock+0x27/0x40
[   45.851125]  [<ffffffff811eceb1>] do_writepages+0x21/0x40
[   45.856528]  [<ffffffff811df2d8>] __filemap_fdatawrite_range+0xc8/0x100
[   45.863183]  [<ffffffff811df3bc>] filemap_flush+0x1c/0x20
[   45.868606]  [<ffffffff81316165>] ext4_alloc_da_blocks+0x105/0x130
[   45.874786]  [<ffffffff8130eeeb>] ext4_release_file+0x2b/0xc0
[   45.880533]  [<ffffffff8127981c>] __fput+0x10c/0x1e0
[   45.885496]  [<ffffffff8127992e>] ____fput+0xe/0x10
[   45.890429]  [<ffffffff810d121a>] task_work_run+0x7a/0xb0
[   45.895829]  [<ffffffff810a6fc9>] exit_to_usermode_loop+0x6b/0x9c
[   45.901920]  [<ffffffff81003c07>] do_syscall_64+0x137/0x1d0
[   45.907491]  [<ffffffff81827089>] entry_SYSCALL64_slow_path+0x25/0x25
[   45.914919] JBD2: Spotted dirty metadata buffer (dev = sda1, blocknr = 0). There's a risk of filesystem corruption in case of system crash.

for owner in $(teuthology-lock --summary | grep -i scheduled | awk '{ print $4 }' | sort -u); do echo $owner; echo "========"; teuthology-nuke --stale --owner $owner --unlock; echo "======== "; done
scheduled_cbodley@monet
========
2017-01-04 23:16:02,571.571 INFO:teuthology.nuke:targets:
  mira035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7IFwzMI1N332ITvtcFLW9aUlKCcJkDJAHuHCyKCDOrASz+v+DjnGR79ZnveQovW5QV6FZKhBA/VjgDh8ZYCBFPkNScxnc0tT2YHTZLGRdCKxH+hOdF8aBu77uVyaeDFU5meBFH85YjzJwBOFUCEVD/AbLie1oeRwEYLtSaEKkp8eNg2ZXGuxmMyFgDEqsNajfBoVTdXN1XYexuFy5FDHav0RsqeVuTZO2ptbU41MNqx5OOc03PzneJlg/0TNJFWiS3ytN5VzIkbZ9ygKiA2UmJ0olxhRMYrUs46mVSpQoXirl3gpMyukYhc2we4RJiLM9AHPZ5ckOvwson/WH+mPl
2017-01-04 23:16:02,587.587 INFO:teuthology.task.internal.check_lock:Checking locks...
2017-01-04 23:16:02,609.609 INFO:teuthology.nuke.actions:checking console status of mira035.ipmi.sepia.ceph.com
2017-01-04 23:16:02,988.988 INFO:teuthology.nuke.actions:console ready on mira035.ipmi.sepia.ceph.com
2017-01-04 23:16:02,989.989 INFO:teuthology.task.internal:Opening connections...
2017-01-04 23:16:03,370.370 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules...
2017-01-04 23:16:03,370.370 INFO:teuthology.orchestra.run.mira035:Running: "sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'" 
2017-01-04 23:16:03,753.753 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules.
2017-01-04 23:16:03,753.753 INFO:teuthology.nuke.actions:Unmounting ceph-fuse and killing daemons...
2017-01-04 23:16:03,753.753 INFO:teuthology.orchestra.run:Running command with timeout 180
2017-01-04 23:16:03,753.753 INFO:teuthology.orchestra.run.mira035:Running: 'sudo stop ceph-all || sudo service ceph stop || sudo systemctl stop ceph.target'
2017-01-04 23:16:03,978.978 INFO:teuthology.orchestra.run.mira035.stderr:sudo: stop: command not found
2017-01-04 23:16:04,043.043 INFO:teuthology.orchestra.run.mira035.stderr:Redirecting to /bin/systemctl stop  ceph.service
2017-01-04 23:16:04,043.043 INFO:teuthology.orchestra.run.mira035.stderr:Failed to stop ceph.service: Unit ceph.service not loaded.
2017-01-04 23:16:04,076.076 INFO:teuthology.orchestra.run.mira035.stderr:Failed to stop ceph.target: Unit ceph.target not loaded.
2017-01-04 23:16:04,079.079 INFO:teuthology.orchestra.run:Running command with timeout 120
2017-01-04 23:16:04,080.080 INFO:teuthology.orchestra.run.mira035:Running: "if grep -q ceph-fuse /etc/mtab ; then grep ceph-fuse /etc/mtab | grep -o ' /.* fuse' | grep -o '/.* ' | xargs -n 1 sudo fusermount -u ; fi ; if grep -q rbd-fuse /etc/mtab ; then grep rbd-fuse /etc/mtab | grep -o ' /.* fuse' | grep -o '/.* ' | xargs -n 1 sudo fusermount -u ; fi ; sudo killall --quiet ceph-mon ceph-osd ceph-mds ceph-mgr ceph-fuse ceph-disk radosgw ceph_test_rados rados rbd-fuse apache2 || true" 
2017-01-04 23:16:04,355.355 INFO:teuthology.nuke.actions:All daemons killed.
2017-01-04 23:16:04,355.355 INFO:teuthology.orchestra.run:Running command with timeout 20
2017-01-04 23:16:04,355.355 INFO:teuthology.orchestra.run.mira035:Running: 'sudo pkill -f -9 valgrind.bin'
2017-01-04 23:16:04,580.580 INFO:teuthology.orchestra.run.mira035:Running: "python -c 'import platform; print platform.linux_distribution()'" 
2017-01-04 23:16:04,806.806 INFO:teuthology.orchestra.run.mira035.stdout:('CentOS Linux', '7.3.1611', 'Core')
2017-01-04 23:16:04,814.814 INFO:teuthology.orchestra.run.mira035:Running: 'sudo lsb_release -is'
2017-01-04 23:16:05,099.099 INFO:teuthology.orchestra.run.mira035.stdout:CentOS
2017-01-04 23:16:05,100.100 INFO:teuthology.task.install.rpm:Removing packages: ceph-radosgw, ceph-test, ceph, ceph-fuse, cephfs-java, libcephfs_jni1, libcephfs1, librados2, librbd1, python-ceph, rbd-fuse, ceph-debuginfo, salt-common, salt-minion, calamari-server, multipath-tools, device-mapper-multipath on rpm system.
2017-01-04 23:16:05,100.100 INFO:teuthology.orchestra.run.mira035:Running: 'uname -m'
2017-01-04 23:16:05,276.276 INFO:teuthology.orchestra.run.mira035.stdout:x86_64
2017-01-04 23:16:05,277.277 INFO:teuthology.orchestra.run.mira035:Running: 'sudo lsb_release -is'
2017-01-04 23:16:05,559.559 INFO:teuthology.orchestra.run.mira035.stdout:CentOS
2017-01-04 23:16:05,560.560 INFO:teuthology.orchestra.run.mira035:Running: "for d in ceph-radosgw ceph-test ceph ceph-fuse cephfs-java libcephfs_jni1 libcephfs1 librados2 librbd1 python-ceph rbd-fuse ceph-debuginfo salt-common salt-minion calamari-server multipath-tools device-mapper-multipath ; do sudo yum -y remove '' $d || true ; done" 
2017-01-04 23:16:06,001.001 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:06,001.001 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:06,283.283 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:06,284.284 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:06,565.565 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:06,565.565 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:06,760.760 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:06,760.760 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:07,040.040 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:07,040.040 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:07,312.312 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:07,312.312 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:07,589.589 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:07,589.589 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:07,795.795 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:07,795.795 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:07,987.987 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:07,987.987 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:08,266.266 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:08,266.266 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:08,535.535 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:08,535.535 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:08,815.815 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:08,815.815 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:09,095.095 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:09,095.095 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:09,281.281 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:09,281.281 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:09,471.471 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:09,471.471 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:09,709.709 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:09,710.710 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:09,978.978 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:09,978.978 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:09,995.995 INFO:teuthology.orchestra.run.mira035:Running: 'sudo yum clean all'
2017-01-04 23:16:10,345.345 INFO:teuthology.orchestra.run.mira035.stderr:Config error: Couldn't parse /etc/yum/pluginconf.d/priorities.conf: File contains parsing errors: file:///etc/yum/pluginconf.d/priorities.conf
2017-01-04 23:16:10,345.345 INFO:teuthology.orchestra.run.mira035.stderr:       [line  3]:
2017-01-04 23:16:10,362.362 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/yuriw/teuthology/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/yuriw/teuthology/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/yuriw/teuthology/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/yuriw/teuthology/teuthology/task/install/rpm.py", line 53, in _remove
    'sudo', pkg_mng_cmd, 'clean', pkg_mng_opts,
  File "/home/yuriw/teuthology/teuthology/orchestra/remote.py", line 192, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/yuriw/teuthology/teuthology/orchestra/run.py", line 403, in run
    r.wait()
  File "/home/yuriw/teuthology/teuthology/orchestra/run.py", line 166, in wait
    label=self.label)
CommandFailedError: Command failed on mira035 with status 1: 'sudo yum clean all'
2017-01-04 23:16:10,363.363 ERROR:teuthology.nuke:Could not nuke {u'mira035.front.sepia.ceph.com': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7IFwzMI1N332ITvtcFLW9aUlKCcJkDJAHuHCyKCDOrASz+v+DjnGR79ZnveQovW5QV6FZKhBA/VjgDh8ZYCBFPkNScxnc0tT2YHTZLGRdCKxH+hOdF8aBu77uVyaeDFU5meBFH85YjzJwBOFUCEVD/AbLie1oeRwEYLtSaEKkp8eNg2ZXGuxmMyFgDEqsNajfBoVTdXN1XYexuFy5FDHav0RsqeVuTZO2ptbU41MNqx5OOc03PzneJlg/0TNJFWiS3ytN5VzIkbZ9ygKiA2UmJ0olxhRMYrUs46mVSpQoXirl3gpMyukYhc2we4RJiLM9AHPZ5ckOvwson/WH+mPl'}
Traceback (most recent call last):
  File "/home/yuriw/teuthology/teuthology/nuke/__init__.py", line 281, in nuke_one
    nuke_helper(ctx, should_unlock)
  File "/home/yuriw/teuthology/teuthology/nuke/__init__.py", line 324, in nuke_helper
    remove_installed_packages(ctx)
  File "/home/yuriw/teuthology/teuthology/nuke/actions.py", line 319, in remove_installed_packages
    rpm=rpms,
  File "/home/yuriw/teuthology/teuthology/task/install/__init__.py", line 137, in remove_packages
    system_type], ctx, config, remote, pkgs[system_type])
  File "/home/yuriw/teuthology/teuthology/parallel.py", line 83, in __exit__
    for result in self:
  File "/home/yuriw/teuthology/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/yuriw/teuthology/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/yuriw/teuthology/teuthology/task/install/rpm.py", line 53, in _remove
    'sudo', pkg_mng_cmd, 'clean', pkg_mng_opts,
  File "/home/yuriw/teuthology/teuthology/orchestra/remote.py", line 192, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/yuriw/teuthology/teuthology/orchestra/run.py", line 403, in run
    r.wait()
  File "/home/yuriw/teuthology/teuthology/orchestra/run.py", line 166, in wait
    label=self.label)
CommandFailedError: Command failed on mira035 with status 1: 'sudo yum clean all'
2017-01-04 23:16:10,364.364 ERROR:teuthology.nuke:Could not nuke the following targets:
targets:
  mira035.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7IFwzMI1N332ITvtcFLW9aUlKCcJkDJAHuHCyKCDOrASz+v+DjnGR79ZnveQovW5QV6FZKhBA/VjgDh8ZYCBFPkNScxnc0tT2YHTZLGRdCKxH+hOdF8aBu77uVyaeDFU5meBFH85YjzJwBOFUCEVD/AbLie1oeRwEYLtSaEKkp8eNg2ZXGuxmMyFgDEqsNajfBoVTdXN1XYexuFy5FDHav0RsqeVuTZO2ptbU41MNqx5OOc03PzneJlg/0TNJFWiS3ytN5VzIkbZ9ygKiA2UmJ0olxhRMYrUs46mVSpQoXirl3gpMyukYhc2we4RJiLM9AHPZ5ckOvwson/WH+mPl