Bug #44631: ceph pg dump error code 124 - RADOS - Ceph

Bug #44631

ceph pg dump error code 124

Added by Sage Weil over 3 years ago. Updated over 2 years ago.

Status:

New

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

pacific, octopus, nautilus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2020-02-25T18:25:09.386 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json
2020-02-25T18:27:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 124
2020-02-25T18:27:09.620 ERROR:teuthology.run_tasks:Saw exception from tasks.

/a/sage-2020-02-25_15:51:04-rados-wip-sage2-testing-2020-02-25-0704-distro-basic-smithi/4801824

no useful log. AFAICS prevoius pg dump commands hit the mon (get_command_descriptions) and then mgr (pg dump), but this particular command hit neither.

then again:

2020-03-16T19:06:26.014 INFO:teuthology.orchestra.run.smithi072:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json
2020-03-16T19:08:26.035 DEBUG:teuthology.orchestra.run:got remote process result: 124
2020-03-16T19:08:26.035 ERROR:teuthology.contextutil:Saw exception from nested tasks

/a/sage-2020-03-16_13:50:19-rados-wip-sage-testing-2020-03-15-1802-distro-basic-smithi/4860316

Related issues

History

#1 Updated by Sage Weil over 3 years ago

124 -> process was killed by SIGTERM (according to https://www.howtogeek.com/423286/how-to-use-the-timeout-command-on-linux/)

#2 Updated by Sage Weil over 3 years ago

/a/sage-2020-03-27_13:32:58-rados-wip-sage3-testing-2020-03-26-1757-distro-basic-smithi/4895381

#3 Updated by Neha Ojha over 3 years ago

I think the pg dump command is timing out for some reason. The timestamps between the following log lines indicate that it waited for 120 seconds. Unfortunately on both occasions the job went dead, we don't have any logs.

2020-02-25T18:25:09.386 INFO:teuthology.orchestra.run.smithi156:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --log-early pg dump --format=json
2020-02-25T18:27:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 124
2020-02-25T18:27:09.620 ERROR:teuthology.run_tasks:Saw exception from tasks.