Project

General

Profile

Actions

Bug #10801

closed

s3tests failed in upgrade:dumpling-dumpling-distro-basic-magna run

Added by Yuri Weinstein about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
dumpling
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.redhat.com/teuthology-2015-02-08_10:00:02-upgrade:dumpling-dumpling-distro-basic-magna/
Jobs: ['42275', '42276', '42277', '42278', '42279', '42280']
Log for one: http://magna002.ceph.redhat.com/teuthology-2015-02-08_10:00:02-upgrade:dumpling-dumpling-distro-basic-magna/42276/

2015-02-08T13:27:11.985 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_get ... FAIL
2015-02-08T13:27:12.044 INFO:tasks.ceph.mon.b.magna073.stdout:starting mon.b rank 0 at 10.8.128.73:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid 0ece04bb-5e28-4f8b-ab6d-ba7d6a9645e2
2015-02-08T13:27:12.435 DEBUG:teuthology.misc:6 of 6 OSDs are up
2015-02-08T13:27:12.436 INFO:teuthology.orchestra.run.magna076:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph health'
2015-02-08T13:27:12.760 DEBUG:teuthology.misc:Ceph health: HEALTH_OK
2015-02-08T13:27:12.761 INFO:teuthology.task.sequential:In sequential, running task sleep...
2015-02-08T13:27:12.762 INFO:teuthology.task.sleep:Sleeping for 60
2015-02-08T13:27:20.489 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_get_bucket_gone ... FAIL
2015-02-08T13:27:28.132 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_get_object_gone ... FAIL
2015-02-08T13:27:35.841 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_bucket_head ... FAIL
2015-02-08T13:27:44.161 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_bucket_head_extended ... FAIL
2015-02-08T13:27:52.317 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_get_bucket_acl ... FAIL
2015-02-08T13:28:00.378 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_get_object_acl ... FAIL
2015-02-08T13:28:08.753 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_authenticated ... FAIL
2015-02-08T13:28:12.759 INFO:teuthology.task.sequential:In sequential, running task ceph.restart...
2015-02-08T13:28:16.439 INFO:teuthology.orchestra.run.magna091.stderr:s3tests.functional.test_s3.test_object_raw_authenticated_bucket_acl ... FAIL

and so on....

Actions #1

Updated by Samuel Just about 9 years ago

  • Project changed from Ceph to rgw
Actions #2

Updated by Yehuda Sadeh about 9 years ago

From what I can tell there are communication issues between the client and apache. We see both client tcp connections that fail to establish, and we also see gateway failing to send back responses since client has closed the connection. This might be environment issue.

Actions #3

Updated by Yehuda Sadeh about 9 years ago

thinking more about it, it might be another incarnation of the wrong apache mpm module that we've seen before.

Actions #4

Updated by Yehuda Sadeh about 9 years ago

This is issue #9169, apache is being set up with the wrong mpn module.

Actions #5

Updated by Yuri Weinstein about 9 years ago

  • Backport set to dumpling
Actions #6

Updated by Yuri Weinstein about 9 years ago

  • Assignee set to Yehuda Sadeh

Yehuda, pls resolve if it's done.

Actions #7

Updated by Yuri Weinstein about 9 years ago

Re-ran suite
Run: http://pulpito.ceph.redhat.com/teuthology-2015-02-18_17:24:07-upgrade:dumpling-dumpling-distro-basic-magna/
Jobs: ['50430', '50431', '50433', '50435']
Logs: http://magna002.ceph.redhat.com/teuthology-2015-02-18_17:24:07-upgrade:dumpling-dumpling-distro-basic-magna/50430/

teuthology@magna002:~$ grep "... ERROR" /a/teuthology-2015-02-18_17:24:07-upgrade:dumpling-dumpling-distro-basic-magna/50430/teuthology.log
2015-02-18T18:41:37.037 ERROR:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices...
2015-02-18T18:41:38.147 ERROR:teuthology.misc:Failed to get wwn devices! Using /dev/sd* devices...
2015-02-18T19:03:42.060 INFO:teuthology.orchestra.run.magna105.stderr:s3tests.functional.test_s3.test_100_continue ... ERROR
2015-02-18T19:08:42.015 INFO:teuthology.orchestra.run.magna105.stderr:s3tests.functional.test_s3.test_atomic_read_8mb ... ERROR
2015-02-18T19:31:44.373 INFO:teuthology.orchestra.run.magna105.stderr:s3tests.functional.test_s3.test_atomic_write_4mb ... ERROR
2015-02-18T19:49:31.004 INFO:teuthology.orchestra.run.magna105.stderr:s3tests.functional.test_s3.test_atomic_write_8mb ... ERROR
2015-02-18T20:22:08.724 INFO:teuthology.orchestra.run.magna105.stderr:s3tests.functional.test_s3.test_atomic_dual_write_4mb ... ERROR
2015-02-18T20:41:39.002 INFO:teuthology.orchestra.run.magna105.stderr:s3tests.functional.test_s3.test_atomic_dual_write_8mb ... ERROR
Actions #8

Updated by Yuri Weinstein about 9 years ago

Re-run the suite after merging https://github.com/ceph/ceph-qa-suite/pull/335 and there are 4 jobs failed with configuration errors
Run: http://pulpito.front.sepia.ceph.com/teuthology-2015-02-24_14:59:47-upgrade:dumpling-dumpling-distro-basic-multi/
Jobs: ['777359', '777361', '777363', '777365']

Actions #9

Updated by Loïc Dachary about 9 years ago

http://pulpito.ceph.com/teuthology-2015-02-24_14:59:47-upgrade:dumpling-dumpling-distro-basic-multi/777359 has

2015-02-24T16:18:50.983 INFO:tasks.rgw.client.0.mira087.stderr:AH00534: apache2: Configuration error: More than one MPM loaded.

Actions #10

Updated by Loïc Dachary about 9 years ago

This is because https://github.com/ceph/ceph-qa-suite/commit/3db4d6f93ba3e58b652c3dadefe5c545bd94a2a3 was cherry picked instead of https://github.com/ceph/ceph-qa-suite/commit/35c9cae84c7964ad96381c7455c40e43dd6b4a8d . They are almost identical and partialy overlap and they were commited the same day, hence the confusion.

Actions #11

Updated by Loïc Dachary about 9 years ago

  • Status changed from New to Fix Under Review
  • Assignee changed from Yehuda Sadeh to Loïc Dachary
Actions #13

Updated by Loïc Dachary about 9 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF