Actions
Bug #59413
closedcephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
Status:
Duplicate
Priority:
Normal
Assignee:
Category:
Correctness/Safety
Target version:
% Done:
100%
Source:
Q/A
Tags:
backport_processed
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2023-03-31T18:32:11.660 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-03-31T18:32:11.660 DEBUG:teuthology.orchestra.run.smithi097:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0b5ec5ec87530bf1437438fb473a9edd387821f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-03-31T18:32:11.719 INFO:tasks.workunit.client.0.smithi097.stderr:+ set -e 2023-03-31T18:32:11.719 INFO:tasks.workunit.client.0.smithi097.stderr:+ retried=false 2023-03-31T18:32:11.719 INFO:tasks.workunit.client.0.smithi097.stderr:+ trap -- retry EXIT 2023-03-31T18:32:11.720 INFO:tasks.workunit.client.0.smithi097.stderr:+ rm -rf ceph 2023-03-31T18:32:11.721 INFO:tasks.workunit.client.0.smithi097.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-03-31T18:32:11.730 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into 'ceph'... ... 2023-03-31T18:34:07.124 INFO:tasks.workunit.client.0.smithi097.stderr:error: RPC failed; curl 18 transfer closed with outstanding read data remaining 2023-03-31T18:34:07.130 INFO:tasks.workunit.client.0.smithi097.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2023-03-31T18:34:07.137 INFO:tasks.workunit.client.0.smithi097.stderr:fatal: early EOF 2023-03-31T18:34:07.139 INFO:tasks.workunit.client.0.smithi097.stderr:fatal: fetch-pack: invalid index-pack output 2023-03-31T18:34:07.353 INFO:tasks.workunit.client.0.smithi097.stderr:+ retry 2023-03-31T18:34:07.353 INFO:tasks.workunit.client.0.smithi097.stderr:+ rm -rf ceph 2023-03-31T18:34:07.355 INFO:tasks.workunit.client.0.smithi097.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2023-03-31T18:34:07.364 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into 'ceph'... ... 2023-03-31T18:40:08.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-03-31T18:40:08.601+0000 7fead17ff700 1 -- 172.21.15.97:0/3357775182 shutdown_connections 2023-03-31T18:40:08.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-03-31T18:40:08.601+0000 7fead17ff700 1 -- 172.21.15.97:0/3357775182 wait complete. 2023-03-31T18:40:08.604 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 47 2023-03-31T18:40:08.622 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-03-31T18:40:08.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-31T18:40:30.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-31T18:40:30.207 DEBUG:teuthology.orchestra.run.smithi101:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-31T18:40:30.624 INFO:tasks.workunit.client.0.smithi097.stderr:Updating files: 2% (286/11742)^MUpdating files: 3% (353/11742)^MUpdating files: 4% (470/11742)^MUpdating files: 4% (507/11742)^MUpdating files: 5% (588/11742)^MUpdating files: 6% (705/11742)^MUpdating files: 6% (732/11742)^MUpdating files: 7% (822/11742)^MUpdating files: 7% (934/11742)^MUpdating files: 8% (940/11742)^MUpdating files: 9% (1057/11742)^MUpdating files: 9% (1114/11742)^MUpdating files: 10% (1175/11742)^MUpdating files: 10% (1282/11742)^MUpdating files: 11% (1292/11742)^MUpdating files: 12% (1410/11742)^MUpdating files: 12% (1491/11742)^MUpdating files: 13% (1527/11742)^MUpdating files: 13% (1642/11742)^MUpdating files: 14% (1644/11742)^MUpdating files: 15% (1762/11742)^MUpdating files: 15% (1790/11742)^MUpdating files: 16% (1879/11742)^MUpdating files: 16% (1957/11742)^MUpdating files: 17% (1997/11742)^MUpdating files: 17% (2038/11742)^MUpdating files: 18% (2114/11742)^MUpdating files: 18% (2175/11742)^MUpdating files: 19% (2231/11742)^MUpdating files: 19% (2317/11742)^MUpdating files: 20% (2349/11742)^MUpdating files: 21% (2466/11742)^MUpdating files: 21% (2470/11742)^MUpdating files: 22% (2584/11742)^MUpdating files: 22% (2630/11742)^MUpdating files: 23% (2701/11742)^MUpdating files: 23% (2797/11742)^MUpdating files: 24% (2819/11742)^MUpdating files: 24% (2905/11742)^MUpdating files: 25% (2936/11742)^MUpdating files: 25% (3032/11742)^MUpdating files: 26% (3053/11742)^MUpdating files: 27% (3171/11742)^MUpdating files: 27% (3221/11742)^MUpdating files: 28% (3288/11742)^MUpdating files: 28% (3400/11742)^MUpdating files: 29% (3406/11742)^MUpdating files: 30% (3523/11742)^MUpdating files: 30% (3568/11742)^MUpdating files: 31% (3641/11742)^MUpdating files: 31% (3724/11742)^MUpdating files: 32% (3758/11742)^MUpdating files: 33% (3875/11742)^MUpdating files: 33% (3908/11742)^MUpdating files: 34% (3993/11742)^MUpdating files: 34% (4081/11742)^MUpdating files: 35% (4110/11742)^MUpdating files: 36% (4228/11742)^MUpdating files: 36% (4241/11742)^MUpdating files: 37% (4345/11742)^MUpdating files: 37% (4444/11742)^MUpdating files: 38% (4462/11742)^MUpdating files: 39% (4580/11742)^MUpdating files: 39% (4651/11742)^MUpdating files: 40% (4697/11742)^MUpdating files: 41% (4815/11742)^MUpdating files: 41% (4891/11742)^MUpdating files: 42% (4932/11742)^MUpdating files: 42% (5044/11742)^MUpdating files: 43% (5050/11742)^MUpdating files: 44% (5167/11742)^MUpdating files: 44% (5183/11742)^MUpdating files: 45% (5284/11742)^MUpdating files: 45% (5390/11742)^MUpdating files: 46% (5402/11742)^MUpdating files: 47% (5519/11742)^MUpdating files: 47% (5596/11742)^MUpdating files: 48% (5637/11742)^MUpdating files: 49% (5754/11742)^MUpdating files: 49% (5815/11742)^MUpdating files: 50% (5871/11742)^MUpdating files: 50% (5917/11742)^MUpdating files: 51% (5989/11742)^MUpdating files: 52% (6106/11742)^MUpdating files: 52% (6138/11742)^MUpdating files: 53% (6224/11742)^MUpdating files: 54% (6341/11742)^MUpdating files: 54% (6347/11742)^MUpdating files: 55% (6459/11742)^MUpdating files: 55% (6546/11742)^MUpdating files: 56% (6576/11742)^MUpdating files: 57% (6693/11742)^MUpdating files: 57% (6762/11742)^MUpdating files: 58% (6811/11742)^MUpdating files: 59% (6928/11742)^MUpdating files: 59% (6983/11742)^MUpdating files: 60% (7046/11742)^MUpdating files: 61% (7163/11742)^MUpdating files: 61% (7179/11742)^MUpdating files: 62% (7281/11742)^MUpdating files: 62% (7389/11742)^MUpdating files: 63% (7398/11742)^MUpdating files: 64% (7515/11742)^MUpdating files: 64% (7584/11742)^MUpdating files: 64% (7597/11742)^MUpdating files: 65% (7633/11742)^MUpdating files: 65% (7733/11742)^MUpdating files: 66% (7750/11742)^MUpdating files: 67% (7868/11742)^MUpdating files: 67% (7904/11742)^MUpdating files: 68% (7985/11742)^MUpdating files: 68% (8022/11742)^MUpdating files: 69% (8102/11742)^MUpdating files: 69% (8124/11742)^MUpdating files: 70% (8220/11742)^MUpdating files: 70% (8226/11742)^MUpdating files: 71% (8337/11742)^MUpdating files: 71% (8449/11742)^MUpdating files: 72% (8455/11742)^MUpdating files: 72% (8556/11742)^MUpdating files: 73% (8572/11742)^MUpdating files: 73% (8689/11742)^MUpdating files: 74% (8690/11742)^MUpdating files: 74% (8799/11742)^MUpdating files: 75% (8807/11742)^MUpdating files: 75% (8914/11742)^MUpdating files: 76% (8924/11742)^MUpdating files: 77% (9042/11742)^MUpdating files: 77% (9061/11742)^MUpdating files: 78% (9159/11742)^MUpdating files: 78% (9212/11742)^MUpdating files: 79% (9277/11742)^MUpdating files: 79% (9318/11742)^MUpdating files: 79% (9331/11742)^MUpdating files: 80% (9394/11742)^MUpdating files: 80% (9473/11742)^MUpdating files: 81% (9512/11742)^MUpdating files: 82% (9629/11742)^MUpdating files: 82% (9645/11742)^MUpdating files: 83% (9746/11742)^MUpdating files: 84% (9864/11742)^MUpdating files: 84% (9876/11742)^MUpdating files: 84% (9887/11742)^MUpdating files: 85% (9981/11742)^MUpdating files: 85% (10062/11742)^MUpdating files: 86% (10099/11742)^MUpdating files: 87% (10216/11742)^MUpdating files: 87% (10271/11742)^MUpdating files: 88% (10333/11742)^MUpdating files: 88% (10438/11742)^MUpdating files: 89% (10451/11742)^MUpdating files: 90% (10568/11742)^MUpdating files: 90% (10589/11742)^MUpdating files: 91% (10686/11742)^MUpdating files: 92% (10803/11742)^MUpdating files: 92% (10826/11742)^MUpdating files: 93% (10921/11742)^MUpdating files: 93% (11011/11742)^MUpdating files: 94% (11038/11742)^MUpdating files: 95% (11155/11742)^MUpdating files: 95% (11211/11742)^MUpdating files: 96% (11273/11742)^MUpdating files: 97% (11390/11742)^MUpdating files: 97% (11396/11742)^MUpdating files: 98% (11508/11742)^MUpdating files: 98% (11613/11742)^MUpdating files: 99% (11625/11742)^MUpdating files: 100% (11742/11742)^MUpdating files: 100% (11742/11742), done. 2023-03-31T18:40:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-03-31T18:40:30.684 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-03-31T18:40:30.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-03-31T18:40:31.567 ERROR:teuthology.run_tasks:Saw exception from tasks.
It's said the command timedout but it only ran around 8 minutes. But the snaptest-git-ceph.sh script timeout is 6h and the git clone timeout is 3600 seconds.
This is on rhel 8 and has no ntpq command:
2023-03-31T18:44:44.582 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-31T18:44:44.593 INFO:teuthology.task.clock:Checking final clock skew... 2023-03-31T18:44:44.593 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-31T18:44:44.598 DEBUG:teuthology.orchestra.run.smithi101:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-31T18:44:44.632 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2023-03-31T18:44:44.635 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found
Not sure whether is the clock skew issue.
Actions