Bug #5346: rgw: invalid read from RGWFormatter_Plain::write_data - rgw - Ceph

Actions

Copy link

Bug #5346

closed

rgw: invalid read from RGWFormatter_Plain::write_data

Added by Sage Weil almost 11 years ago. Updated almost 11 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Yehuda Sadeh

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

ubuntu@teuthology:/a/teuthology-2013-06-14_01:00:36-rgw-master-testing-basic/35856$ zless ./remote/ubuntu@plana63.front.sepia.ceph.com/log/valgrind/client.0.log.gz

  <unique>0x2</unique>
  <tid>38</tid>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 4</what>
  <stack>
    <frame>
      <ip>0x64B714</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWFormatter_Plain::write_data(char const*, ...)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_formats.cc</file>
      <line>217</line>
    </frame>
    <frame>
      <ip>0x64B9F8</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWFormatter_Plain::dump_format(char const*, char const*, ...)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_formats.cc</file>
      <line>150</line>
    </frame>
    <frame>
      <ip>0x4F968B</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListBuckets_ObjStore_SWIFT::send_response_data(RGWUserBuckets&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_rest_swift.cc</file>
      <line>76</line>
    </frame>
    <frame>
      <ip>0x6334E4</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListBuckets::execute()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_op.cc</file>
      <line>705</line>
    </frame>
...

job was

ubuntu@teuthology:/a/teuthology-2013-06-14_01:00:36-rgw-master-testing-basic/35856$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 6012c98c90e1d58949d029c221872d98746c2b17
machine_type: plana
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
        lockdep: true
      osd:
        lockdep: true
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: bcfbd0a3ffae6947464d930f636c8b35d1331e9d
  install:
    ceph:
      sha1: bcfbd0a3ffae6947464d930f636c8b35d1331e9d
  s3tests:
    branch: master
  workunit:
    sha1: bcfbd0a3ffae6947464d930f636c8b35d1331e9d
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock.check: null
- install:
    ceph:
      flavor: notcmalloc
- ceph: null
- rgw:
    client.0:
      valgrind:
      - --tool=memcheck
- swift:
    client.0:
      rgw_server: client.0

Actions

Copy link

Updated by Sage Weil almost 11 years ago

Project changed from Ceph to rgw

Actions

Copy link

Updated by Ian Colle almost 11 years ago

Assignee set to Yehuda Sadeh

Actions

Copy link

Updated by Sage Weil almost 11 years ago

Status changed from New to 12

this appears to be triggered by the swift test.. doesn't happen with s3tests or readwrite etc

also present on cuttlefish.

Actions

Copy link

Updated by Yehuda Sadeh almost 11 years ago

well, swift is the only user of the plain formatter I guess.

Actions

Copy link

Updated by Sage Weil almost 11 years ago

using a trivial implemention of strlen avoids this. unfortunately we can't whitelist the glibc strlen call because it is fully inline and not part of the stack seen by valgrind.

we could go with the trivial strlen() reimplementation, or whitelist the entire method (which means we won't catch any other bugs in this method, or callers passing in bad data)

Actions

Copy link