Project

General

Profile

Bug #5346

rgw: invalid read from RGWFormatter_Plain::write_data

Added by Sage Weil over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ubuntu@teuthology:/a/teuthology-2013-06-14_01:00:36-rgw-master-testing-basic/35856$ zless /log/valgrind/client.0.log.gz

  <unique>0x2</unique>
  <tid>38</tid>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 4</what>
  <stack>
    <frame>
      <ip>0x64B714</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWFormatter_Plain::write_data(char const*, ...)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_formats.cc</file>
      <line>217</line>
    </frame>
    <frame>
      <ip>0x64B9F8</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWFormatter_Plain::dump_format(char const*, char const*, ...)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_formats.cc</file>
      <line>150</line>
    </frame>
    <frame>
      <ip>0x4F968B</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListBuckets_ObjStore_SWIFT::send_response_data(RGWUserBuckets&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_rest_swift.cc</file>
      <line>76</line>
    </frame>
    <frame>
      <ip>0x6334E4</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWListBuckets::execute()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.64-428-gbcfbd0a/src/rgw</dir>
      <file>rgw_op.cc</file>
      <line>705</line>
    </frame>
...

job was
ubuntu@teuthology:/a/teuthology-2013-06-14_01:00:36-rgw-master-testing-basic/35856$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 6012c98c90e1d58949d029c221872d98746c2b17
machine_type: plana
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
        lockdep: true
      osd:
        lockdep: true
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: bcfbd0a3ffae6947464d930f636c8b35d1331e9d
  install:
    ceph:
      sha1: bcfbd0a3ffae6947464d930f636c8b35d1331e9d
  s3tests:
    branch: master
  workunit:
    sha1: bcfbd0a3ffae6947464d930f636c8b35d1331e9d
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock.check: null
- install:
    ceph:
      flavor: notcmalloc
- ceph: null
- rgw:
    client.0:
      valgrind:
      - --tool=memcheck
- swift:
    client.0:
      rgw_server: client.0

Associated revisions

Revision 49ff63b1 (diff)
Added by Sage Weil over 10 years ago

rgw: add RGWFormatter_Plain allocation to sidestep cranky strlen()

Valgrind complains about an invalid read when we don't pad the allocation,
and because it is inlined we can't whitelist it for valgrind. Workaround
the warning by just padding our allocations a bit.

Fixes: #5346
Backport: cuttlefish
Signed-off-by: Sage Weil <>

Revision 7e878bcc (diff)
Added by Sage Weil over 10 years ago

rgw: add RGWFormatter_Plain allocation to sidestep cranky strlen()

Valgrind complains about an invalid read when we don't pad the allocation,
and because it is inlined we can't whitelist it for valgrind. Workaround
the warning by just padding our allocations a bit.

Fixes: #5346
Backport: cuttlefish
Signed-off-by: Sage Weil <>
(cherry picked from commit 49ff63b1750789070a8c6fef830c9526ae0f6d9f)

History

#1 Updated by Sage Weil over 10 years ago

  • Project changed from Ceph to rgw

#2 Updated by Ian Colle over 10 years ago

  • Assignee set to Yehuda Sadeh

#3 Updated by Sage Weil over 10 years ago

  • Status changed from New to 12

this appears to be triggered by the swift test.. doesn't happen with s3tests or readwrite etc

also present on cuttlefish.

#4 Updated by Yehuda Sadeh over 10 years ago

well, swift is the only user of the plain formatter I guess.

#5 Updated by Sage Weil over 10 years ago

using a trivial implemention of strlen avoids this. unfortunately we can't whitelist the glibc strlen call because it is fully inline and not part of the stack seen by valgrind.

we could go with the trivial strlen() reimplementation, or whitelist the entire method (which means we won't catch any other bugs in this method, or callers passing in bad data)

#6 Updated by Sage Weil over 10 years ago

  • Status changed from 12 to Fix Under Review

#7 Updated by Yehuda Sadeh over 10 years ago

  • Status changed from Fix Under Review to Resolved

Sage pushed a fix at commit:49ff63b1750789070a8c6fef830c9526ae0f6d9f

Also available in: Atom PDF