Bug #9117: tasks/s3readwrite causes teuthology process to consume too much CPU - teuthology - Ceph

Actions

Copy link

Bug #9117

open

tasks/s3readwrite causes teuthology process to consume too much CPU

Added by Zack Cerza over 9 years ago. Updated over 4 years ago.

Status:

New

Priority:

Normal

Assignee:

Zack Cerza

Category:

% Done:

Source:

other

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

The s3tests-test-readwrite test produces so much log output at such a high rate that a single job will peg teuthology.front's CPU usage at 100% for hours.

The s3tests-test-readwrite should ideally have its output directed to a file that is fetched during cleanup.

Actions

Copy link

Updated by Sage Weil over 9 years ago

Priority changed from Normal to High

Actions

Copy link

Updated by Sage Weil over 9 years ago

Assignee set to Tamilarasi muthamizhan

Actions

Copy link

Updated by Tamilarasi muthamizhan over 9 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Tamilarasi muthamizhan over 9 years ago

Assignee changed from Tamilarasi muthamizhan to Anonymous

Actions

Copy link

Updated by Anonymous over 9 years ago

I have a change that I am testing. It's taking a long time to run (the test takes a while normally) so I probably will leave it running tonight.

Actions

Copy link

Updated by Anonymous over 9 years ago

    raise ConnectionLostError(command=quote(args))
ConnectionLostError: SSH connection was lost: 'rmdir -- /home/ubuntu/cephtest'
2014-10-21 21:56:37,518.518 INFO:teuthology.run:Summary data:
{duration: 1657.4453999996185, failure_reason: 'SSH connection was lost: ''rm -rf
    /home/ubuntu/cephtest/s3-tests''', flavor: basic, owner: wusui@aardvark, status: dead,
  success: false}

2014-10-21 21:56:37,519.519 WARNING:teuthology.report:No job_id found; not reporting results
2014-10-21 21:56:37,520.520 INFO:teuthology.run:DEAD

Latest attempts at diverting the output have not been completely successful. This last run did not complete, but the output to the terminal was not horrendous and the log file was huge with tons of data, so I suspect that the I am heading in the right direction. I just need to be able to survive.

Actions

Copy link

Updated by Anonymous over 9 years ago

Status changed from In Progress to Fix Under Review
Assignee changed from Anonymous to Zack Cerza

A version is now ready for review.

SHA1 -- f3a66d1
PR -- 212
WIP -- wip-9117-wusui

Actions

Copy link

Updated by Zack Cerza over 9 years ago

This is the pull request:
https://github.com/ceph/ceph-qa-suite/pull/212

I think it's important to provide links.

Actions

Copy link

Updated by Anonymous over 9 years ago

I have a new fix for this. It is currently another checkin on top of the old one. I can squash if you prefer to review it as one change.

Actions

Copy link

#10

Updated by Zack Cerza over 9 years ago

Status changed from Fix Under Review to 4
Assignee changed from Zack Cerza to Anonymous

Commented

Actions

Copy link

#11

Updated by Anonymous over 9 years ago

Status changed from 4 to Fix Under Review
Assignee changed from Anonymous to Zack Cerza

I pushed a new version of this to be reviewed. It's PR #212 in ceph-qa-suite

The new version has been rebased into one checkin because the old comments are now inaccurate (the changes were not that many lines so I figure that this is okay).

Actions

Copy link

#12