Tools » teuthology

I have tried the following fixes to get_file:

1. Use io.StringIO() instead of cStringIO()

args.extend([
        'cat',
        '--',
        path,
        ])
    proc = remote.run(
        args=args,
        stdout=io.StringIO()
        )
    data = proc.stdout.getvalue()

which gives:

INFO:teuthology.run:Summary data:
{duration: 121.92297601699829, failure_reason: '''utf8'' codec can''t decode byte
    0xcf in position 2: invalid continuation byte', flavor: basic, owner: wusui@aardvark,
  success: false}

2. Next I tried using BytesIO:

   args.extend([
        'cat',
        '--',
        path,
        ])
    proc = remote.run(
        args=args,
        stdout=io.BytesIO(),
        )
    data = proc.stdout.getvalue()

which resulted in:

2014-04-15T19:14:12.651 INFO:teuthology.run:Summary data:
{duration: 129.0001790523529, failure_reason: '''unicode'' does not have the buffer
    interface', flavor: basic, owner: wusui@aardvark, success: false}

3. Next I tried to pipe it to stdout and read it:

args.extend([
        'cat',
        '--',
        path,
        ])
    proc = remote.run(
        args=args,
        stdout=run.PIPE,
        wait=False,
        )
    data = proc.stdout.read()

and I got

2014-04-15T19:26:44.583 INFO:teuthology.run:Summary data:
{duration: 122.22275114059448, failure_reason: '''utf8'' codec can''t decode byte
    0xcf in position 2: invalid continuation byte', flavor: basic, owner: wusui@aardvark,
  success: false}

4. I tried reading the read buffer

 args.extend([
        'cat',
        '--',
        path,
        ])
    proc = remote.run(
        args=args,
        stdout=run.PIPE,
        wait=False,
        )
    data = proc.stdout.buffer.read()

and I got

2014-04-15T19:32:08.281 INFO:teuthology.run:Summary data:
{duration: 119.98835802078247, failure_reason: '''ChannelFile'' object has no attribute
    ''buffer''', flavor: basic, owner: wusui@aardvark, success: false}

Conclusions.

It appears to me that the Paramiko remote stdout interface does not make it easy to copy the binary data as binary data. We've been getting lucky in that the bytes going across in the past were valid unicode text. I managed to run into a case where this is not the case.

I looked into trying to copy the data using SFTPClient in paramiko, but since no other code that I saw in teuthology does this, I have not tried writing anything yet. There may be something better than going down that rabbit hole. (Not sure if this will end up in the 'this way leads to madness' category).

I think that converting to base64 is the way to go here.

I have changed the code to encode the data in base64 notation.

This data may also be causing a problem with a tar done later in the test that I am running.

I brought up a separate set of vms (Ubuntu precise) and this problem happened again. So it looks like you could generate this condition by
running teuthology using the yaml below:

roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2, client.0]
- [mon.b, mds.a, osd.3, osd.4, osd.5, client.1]
overrides:
  ceph:
    fs: xfs
    conf:
      osd:
        osd sloppy crc: true
      global:
        ms inject socket failures: 5000
tasks:
- install:
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
- workunit:
    clients:
      client.0:
        - rados/test.sh
targets:
  ubuntu@vpm024.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD3DXqbVavWnm30mD/C2E6WldvAPb5FH+l2MHOzRAGYeJ5oXlEtL6aJY+uV3gEgiNTuBscCixuTIKX+NCtFxG9F0VkGZS/84reQ7psRLitLJ0q2tXS7tRVpdHOSknu0Q+Xym9ujS3Laoy2E9VMuH0In+3iE0QzJqYTC/QGx4M6QbcqLw+1tZCEzH1mTXep7XSsh3r0P96vS+Q7Ov8XlfVeWrDWxc2FJlVWNcX1RZZT9X8RryoTDgCzoS4dfKKOQoXm+GhHlj8uC4hDEp5kvXmAVoqQpYN8MV2wrUrUx/sa1w4jtQ8BRouTWMN56oczn9rGdstB/fG7n5IBqRMmiBmiT
  ubuntu@vpm025.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCosTKvkC8O3OIZ3HpqilyLmqzud2CGMutSApdEwBvf07UZdTcimp1MRB1ORb1ZEl0o7FMtDUWa0ECzJJsmMc5y7JFyO7HJxTMVfVfaNjOl/4ss/8P583bLyDoMbiOmqdF7DZuTrvw8gLU8hW+rYKaPo+byF9slouDjsZxaxxfYanJ270deufPGDNs7L89Haml2oAQH38mYhZWABoDWYx25Z5lkei3z+0JiVYN36PJiQRMJTfs7omEARiPA/H+i5PUx7JlNQ/MKMY/v5dVnhxLoKsJi4TB8LDekgKr8DoihV3sM+unpgNqdAqt08oOaQeePx4R3zccnRwJKVNcb7Wx5

Just noticing this. I created a minimal test case here:
https://gist.github.com/zmc/11016140

Running my test case:

(virtualenv)ubuntu@teuthology:~/zack/teuthology$ ./utest.py 
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 390, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/ubuntu/zack/teuthology/teuthology/orchestra/run.py", line 144, in copy_file_to
    return handler(f, dst, host)
  File "/home/ubuntu/zack/teuthology/teuthology/orchestra/run.py", line 119, in copy_to_log
    for line in f.xreadlines():
  File "/home/ubuntu/zack/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 100, in next
    line = self.readline()
  File "/home/ubuntu/zack/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 251, in readline
    return line if self._flags & self.FLAG_BINARY else u(line)
  File "/home/ubuntu/zack/teuthology/virtualenv/local/lib/python2.7/site-packages/paramiko/py3compat.py", line 49, in u
    return s.decode(encoding)
  File "/home/ubuntu/zack/teuthology/virtualenv/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcf in position 2: invalid continuation byte
<Greenlet at 0x25a48d0: copy_file_to(<paramiko.ChannelFile from <paramiko.Channel 1 (cl, <logging.Logger object at 0x2670610>, '127.0.1.1')> failed with UnicodeDecodeError

Traceback (most recent call last):
  File "./utest.py", line 26, in <module>
    run(conn, ['cat', '--', bad_file_name])
  File "/home/ubuntu/zack/teuthology/teuthology/orchestra/run.py", line 330, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/ubuntu/zack/teuthology/teuthology/orchestra/run.py", line 307, in _check_status
    g_out.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
    raise self._exception
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcf in position 2: invalid continuation byte

If we can tell paramiko that we're trying to read in binary mode, we might be OK.

However -- why are we trying to cat a binary file to stdout?

I figured something out. We should be using paramiko.SFTPClient to transfer files, as opposed to catting them and reading stdout:
https://gist.github.com/zmc/11017829

For forensic interest, I've discovered that the bytes in question are the length of the monmap struct, which varies depending on the number and name of the mons. As far as I can tell, any monmap with 3 mons with single-character names (like,
a, b, and c) will have an encoded monmap that starts with 03 03 CF 01; the CF 01 is the offending sequence to UTF-8 decoding.

The puzzling part is that I can't find anything about the monmap encoding or the handling inside teuthology that would have changed recently, so I've no idea why this just started happening.

Ah! Paramiko was updated to 1.13.0 for teuthworker/teuthology-master, last written today at 1:45PM. Paramiko's ChannelFile objects now respect mode, and if "b" isn't set in the mode, try to decode the output before providing it.

https://github.com/paramiko/paramiko/commit/45e65b6e1eb47944a26e4349d41998844c155df5#diff-39fa2cc98c01f94653c725bae23bd1ceR143

I bet rolling paramiko back to 1.12 will immediately fix this, and we can get the desired behavior if we can set mode param on the ChannelFile constructor.

Indeed, 1.12 does not have the new behavior, and it looks like paramiko.SSHClient.exec_command() offers no interface for controlling the file modes:

stdin = chan.makefile('wb', bufsize)
        stdout = chan.makefile('r', bufsize)
        stderr = chan.makefile_stderr('r', bufsize)

I've cobbled a version of teuthology together that uses SFTPClient and it seems to work on one testcase that I ran. This version changed the behavior of get_file, pull_directory, and pull_directory_tarball. I am not sure that all of the fixes have been made yet.

I have been talking to Dan and I know about the other solutions and work-arounds, but we should probably pursue this change too.

I have pushed a preliminary version of teuthology that uses SFTPClient for
the raw copies. The wip branch on master is wip-usesftpclient-wusui. I am
running this overnight to see how this does, but I am making a preliminary version available if anyone wants to try it / look at it.

I pushed a new version of wip-8116-wusui