Bug #59413
cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128"
Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Correctness/Safety
Target version:
% Done:
100%
Source:
Q/A
Tags:
backport_processed
Backport:
reef,quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2023-03-31T18:32:11.660 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-03-31T18:32:11.660 DEBUG:teuthology.orchestra.run.smithi097:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0b5ec5ec87530bf1437438fb473a9edd387821f6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-03-31T18:32:11.719 INFO:tasks.workunit.client.0.smithi097.stderr:+ set -e 2023-03-31T18:32:11.719 INFO:tasks.workunit.client.0.smithi097.stderr:+ retried=false 2023-03-31T18:32:11.719 INFO:tasks.workunit.client.0.smithi097.stderr:+ trap -- retry EXIT 2023-03-31T18:32:11.720 INFO:tasks.workunit.client.0.smithi097.stderr:+ rm -rf ceph 2023-03-31T18:32:11.721 INFO:tasks.workunit.client.0.smithi097.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-03-31T18:32:11.730 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into 'ceph'... ... 2023-03-31T18:34:07.124 INFO:tasks.workunit.client.0.smithi097.stderr:error: RPC failed; curl 18 transfer closed with outstanding read data remaining 2023-03-31T18:34:07.130 INFO:tasks.workunit.client.0.smithi097.stderr:fetch-pack: unexpected disconnect while reading sideband packet 2023-03-31T18:34:07.137 INFO:tasks.workunit.client.0.smithi097.stderr:fatal: early EOF 2023-03-31T18:34:07.139 INFO:tasks.workunit.client.0.smithi097.stderr:fatal: fetch-pack: invalid index-pack output 2023-03-31T18:34:07.353 INFO:tasks.workunit.client.0.smithi097.stderr:+ retry 2023-03-31T18:34:07.353 INFO:tasks.workunit.client.0.smithi097.stderr:+ rm -rf ceph 2023-03-31T18:34:07.355 INFO:tasks.workunit.client.0.smithi097.stderr:+ timeout 3600 git clone https://git.ceph.com/ceph.git 2023-03-31T18:34:07.364 INFO:tasks.workunit.client.0.smithi097.stderr:Cloning into 'ceph'... ... 2023-03-31T18:40:08.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-03-31T18:40:08.601+0000 7fead17ff700 1 -- 172.21.15.97:0/3357775182 shutdown_connections 2023-03-31T18:40:08.604 INFO:teuthology.orchestra.run.smithi097.stderr:2023-03-31T18:40:08.601+0000 7fead17ff700 1 -- 172.21.15.97:0/3357775182 wait complete. 2023-03-31T18:40:08.604 INFO:teuthology.orchestra.run.smithi097.stderr:dumped fsmap epoch 47 2023-03-31T18:40:08.622 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-03-31T18:40:08.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2023-03-31T18:40:30.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-31T18:40:30.207 DEBUG:teuthology.orchestra.run.smithi101:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-31T18:40:30.624 INFO:tasks.workunit.client.0.smithi097.stderr:Updating files: 2% (286/11742)^MUpdating files: 3% (353/11742)^MUpdating files: 4% (470/11742)^MUpdating files: 4% (507/11742)^MUpdating files: 5% (588/11742)^MUpdating files: 6% (705/11742)^MUpdating files: 6% (732/11742)^MUpdating files: 7% (822/11742)^MUpdating files: 7% (934/11742)^MUpdating files: 8% (940/11742)^MUpdating files: 9% (1057/11742)^MUpdating files: 9% (1114/11742)^MUpdating files: 10% (1175/11742)^MUpdating files: 10% (1282/11742)^MUpdating files: 11% (1292/11742)^MUpdating files: 12% (1410/11742)^MUpdating files: 12% (1491/11742)^MUpdating files: 13% (1527/11742)^MUpdating files: 13% (1642/11742)^MUpdating files: 14% (1644/11742)^MUpdating files: 15% (1762/11742)^MUpdating files: 15% (1790/11742)^MUpdating files: 16% (1879/11742)^MUpdating files: 16% (1957/11742)^MUpdating files: 17% (1997/11742)^MUpdating files: 17% (2038/11742)^MUpdating files: 18% (2114/11742)^MUpdating files: 18% (2175/11742)^MUpdating files: 19% (2231/11742)^MUpdating files: 19% (2317/11742)^MUpdating files: 20% (2349/11742)^MUpdating files: 21% (2466/11742)^MUpdating files: 21% (2470/11742)^MUpdating files: 22% (2584/11742)^MUpdating files: 22% (2630/11742)^MUpdating files: 23% (2701/11742)^MUpdating files: 23% (2797/11742)^MUpdating files: 24% (2819/11742)^MUpdating files: 24% (2905/11742)^MUpdating files: 25% (2936/11742)^MUpdating files: 25% (3032/11742)^MUpdating files: 26% (3053/11742)^MUpdating files: 27% (3171/11742)^MUpdating files: 27% (3221/11742)^MUpdating files: 28% (3288/11742)^MUpdating files: 28% (3400/11742)^MUpdating files: 29% (3406/11742)^MUpdating files: 30% (3523/11742)^MUpdating files: 30% (3568/11742)^MUpdating files: 31% (3641/11742)^MUpdating files: 31% (3724/11742)^MUpdating files: 32% (3758/11742)^MUpdating files: 33% (3875/11742)^MUpdating files: 33% (3908/11742)^MUpdating files: 34% (3993/11742)^MUpdating files: 34% (4081/11742)^MUpdating files: 35% (4110/11742)^MUpdating files: 36% (4228/11742)^MUpdating files: 36% (4241/11742)^MUpdating files: 37% (4345/11742)^MUpdating files: 37% (4444/11742)^MUpdating files: 38% (4462/11742)^MUpdating files: 39% (4580/11742)^MUpdating files: 39% (4651/11742)^MUpdating files: 40% (4697/11742)^MUpdating files: 41% (4815/11742)^MUpdating files: 41% (4891/11742)^MUpdating files: 42% (4932/11742)^MUpdating files: 42% (5044/11742)^MUpdating files: 43% (5050/11742)^MUpdating files: 44% (5167/11742)^MUpdating files: 44% (5183/11742)^MUpdating files: 45% (5284/11742)^MUpdating files: 45% (5390/11742)^MUpdating files: 46% (5402/11742)^MUpdating files: 47% (5519/11742)^MUpdating files: 47% (5596/11742)^MUpdating files: 48% (5637/11742)^MUpdating files: 49% (5754/11742)^MUpdating files: 49% (5815/11742)^MUpdating files: 50% (5871/11742)^MUpdating files: 50% (5917/11742)^MUpdating files: 51% (5989/11742)^MUpdating files: 52% (6106/11742)^MUpdating files: 52% (6138/11742)^MUpdating files: 53% (6224/11742)^MUpdating files: 54% (6341/11742)^MUpdating files: 54% (6347/11742)^MUpdating files: 55% (6459/11742)^MUpdating files: 55% (6546/11742)^MUpdating files: 56% (6576/11742)^MUpdating files: 57% (6693/11742)^MUpdating files: 57% (6762/11742)^MUpdating files: 58% (6811/11742)^MUpdating files: 59% (6928/11742)^MUpdating files: 59% (6983/11742)^MUpdating files: 60% (7046/11742)^MUpdating files: 61% (7163/11742)^MUpdating files: 61% (7179/11742)^MUpdating files: 62% (7281/11742)^MUpdating files: 62% (7389/11742)^MUpdating files: 63% (7398/11742)^MUpdating files: 64% (7515/11742)^MUpdating files: 64% (7584/11742)^MUpdating files: 64% (7597/11742)^MUpdating files: 65% (7633/11742)^MUpdating files: 65% (7733/11742)^MUpdating files: 66% (7750/11742)^MUpdating files: 67% (7868/11742)^MUpdating files: 67% (7904/11742)^MUpdating files: 68% (7985/11742)^MUpdating files: 68% (8022/11742)^MUpdating files: 69% (8102/11742)^MUpdating files: 69% (8124/11742)^MUpdating files: 70% (8220/11742)^MUpdating files: 70% (8226/11742)^MUpdating files: 71% (8337/11742)^MUpdating files: 71% (8449/11742)^MUpdating files: 72% (8455/11742)^MUpdating files: 72% (8556/11742)^MUpdating files: 73% (8572/11742)^MUpdating files: 73% (8689/11742)^MUpdating files: 74% (8690/11742)^MUpdating files: 74% (8799/11742)^MUpdating files: 75% (8807/11742)^MUpdating files: 75% (8914/11742)^MUpdating files: 76% (8924/11742)^MUpdating files: 77% (9042/11742)^MUpdating files: 77% (9061/11742)^MUpdating files: 78% (9159/11742)^MUpdating files: 78% (9212/11742)^MUpdating files: 79% (9277/11742)^MUpdating files: 79% (9318/11742)^MUpdating files: 79% (9331/11742)^MUpdating files: 80% (9394/11742)^MUpdating files: 80% (9473/11742)^MUpdating files: 81% (9512/11742)^MUpdating files: 82% (9629/11742)^MUpdating files: 82% (9645/11742)^MUpdating files: 83% (9746/11742)^MUpdating files: 84% (9864/11742)^MUpdating files: 84% (9876/11742)^MUpdating files: 84% (9887/11742)^MUpdating files: 85% (9981/11742)^MUpdating files: 85% (10062/11742)^MUpdating files: 86% (10099/11742)^MUpdating files: 87% (10216/11742)^MUpdating files: 87% (10271/11742)^MUpdating files: 88% (10333/11742)^MUpdating files: 88% (10438/11742)^MUpdating files: 89% (10451/11742)^MUpdating files: 90% (10568/11742)^MUpdating files: 90% (10589/11742)^MUpdating files: 91% (10686/11742)^MUpdating files: 92% (10803/11742)^MUpdating files: 92% (10826/11742)^MUpdating files: 93% (10921/11742)^MUpdating files: 93% (11011/11742)^MUpdating files: 94% (11038/11742)^MUpdating files: 95% (11155/11742)^MUpdating files: 95% (11211/11742)^MUpdating files: 96% (11273/11742)^MUpdating files: 97% (11390/11742)^MUpdating files: 97% (11396/11742)^MUpdating files: 98% (11508/11742)^MUpdating files: 98% (11613/11742)^MUpdating files: 99% (11625/11742)^MUpdating files: 100% (11742/11742)^MUpdating files: 100% (11742/11742), done. 2023-03-31T18:40:30.684 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-03-31T18:40:30.684 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-03-31T18:40:30.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-03-31T18:40:31.567 ERROR:teuthology.run_tasks:Saw exception from tasks.
It's said the command timedout but it only ran around 8 minutes. But the snaptest-git-ceph.sh script timeout is 6h and the git clone timeout is 3600 seconds.
This is on rhel 8 and has no ntpq command:
2023-03-31T18:44:44.582 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-03-31T18:44:44.593 INFO:teuthology.task.clock:Checking final clock skew... 2023-03-31T18:44:44.593 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-31T18:44:44.598 DEBUG:teuthology.orchestra.run.smithi101:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-31T18:44:44.632 INFO:teuthology.orchestra.run.smithi097.stderr:bash: ntpq: command not found 2023-03-31T18:44:44.635 INFO:teuthology.orchestra.run.smithi101.stderr:bash: ntpq: command not found
Not sure whether is the clock skew issue.
Subtasks
Related issues
History
#1 Updated by Venky Shankar 8 months ago
- Category set to Correctness/Safety
- Status changed from New to Triaged
- Assignee set to Xiubo Li
- Target version set to v19.0.0
- Backport set to reef,quincy,pacific
- Component(FS) MDS added
#3 Updated by Patrick Donnelly 3 months ago
/teuthology/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/7395153/teuthology.log
interesting got a BUS error
#4 Updated by Xiubo Li 3 months ago
Patrick Donnelly wrote:
/teuthology/pdonnell-2023-09-12_14:07:50-fs-wip-batrick-testing-20230912.122437-distro-default-smithi/7395153/teuthology.log
interesting got a BUS error
There had one crash in MDS:
-6> 2023-09-12T17:28:18.891+0000 7ff336960700 20 mds.4.bal handle_export_pins export_pin_queue size=0
-5> 2023-09-12T17:28:18.891+0000 7ff336960700 10 mds.4.log trim_all: 1/0/0
-4> 2023-09-12T17:28:18.891+0000 7ff336960700 20 mds.4.log _trim_expired_segments: examining LogSegment(15246/0x36f8870 events=1)
-3> 2023-09-12T17:28:18.891+0000 7ff336960700 10 mds.4.log _trim_expired_segments waiting for expiry LogSegment(15246/0x36f8870 events=1)
-2> 2023-09-12T17:28:18.891+0000 7ff336960700 7 mds.4.cache capping the mdlog
-1> 2023-09-12T17:28:18.891+0000 7ff336960700 20 mds.4.log _submit_entry ELid(0)
0> 2023-09-12T17:28:18.892+0000 7ff336960700 -1 *** Caught signal (Segmentation fault) **
in thread 7ff336960700 thread_name:safe_timer
{
"crash_id": "2023-09-12T17:28:18.893322Z_ce36da19-d151-418d-9f28-4b5703821abc",
"timestamp": "2023-09-12T17:28:18.893322Z",
"process_name": "ceph-mds",
"entity_name": "mds.e",
"ceph_version": "18.0.0-6088-g2110e007",
"utsname_hostname": "smithi161",
"utsname_sysname": "Linux",
"utsname_release": "4.18.0-372.9.1.el8.x86_64",
"utsname_version": "#1 SMP Fri Apr 15 22:12:19 EDT 2022",
"utsname_machine": "x86_64",
"os_name": "Red Hat Enterprise Linux",
"os_id": "rhel",
"os_version_id": "8.6",
"os_version": "8.6 (Ootpa)",
"backtrace": [
"/lib64/libpthread.so.0(+0x12ce0) [0x7ff3427cece0]",
"(CInode::get_dirfrags() const+0x26) [0x55adcc11f106]",
"(MDCache::advance_stray()+0x1f0) [0x55adcc0908b0]",
"(MDLog::_start_new_segment(SegmentBoundary*)+0x465) [0x55adcc2d4975]",
"(MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0xba) [0x55adcc2d4b0a]",
"(MDLog::submit_entry(LogEvent*, MDSLogContextBase*)+0xbf) [0x55adcbfac33f]",
"(MDCache::shutdown_pass()+0xe9f) [0x55adcc0ea30f]",
"(MDSRankDispatcher::tick()+0x300) [0x55adcbf51d60]",
"(Context::complete(int)+0xd) [0x55adcbf272cd]",
"(CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x181) [0x7ff343b22af1]",
"(CommonSafeTimerThread<ceph::fair_mutex>::entry()+0x11) [0x7ff343b23e01]",
"/lib64/libpthread.so.0(+0x81cf) [0x7ff3427c41cf]",
"clone()"
]
}
And also selinux report errors :
2023-09-12T17:32:59.170490+00:00 smithi161 kernel: ceph: sync getxattr 0000000095a244cb during filling trace 2023-09-12T17:32:59.170631+00:00 smithi161 kernel: SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099511632881 2023-09-12T17:33:00.299981+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:02.560012+00:00 smithi161 kernel: ceph: sync getxattr 000000001136644f during filling trace 2023-09-12T17:33:02.560142+00:00 smithi161 kernel: SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099511632881 2023-09-12T17:33:05.156432+00:00 smithi161 kernel: ceph: sync getxattr 00000000415a7735 during filling trace 2023-09-12T17:33:05.156560+00:00 smithi161 kernel: SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099511632881 2023-09-12T17:33:05.571994+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:05.893023+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:08.671051+00:00 smithi161 kernel: libceph: osd4 (1)172.21.15.175:6801 socket closed (con state OPEN) 2023-09-12T17:33:08.751001+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:09.030999+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket error on write 2023-09-12T17:33:09.293993+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket error on write 2023-09-12T17:33:09.654006+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:10.122997+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:10.499032+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:11.244003+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:11.638174+00:00 smithi161 kernel: ceph: sync getxattr 00000000a34cbe3a during filling trace 2023-09-12T17:33:11.638317+00:00 smithi161 kernel: SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099511632881 2023-09-12T17:33:11.839994+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:12.695012+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:13.647767+00:00 smithi161 kernel: ceph: sync getxattr 000000008049d6c8 during filling trace 2023-09-12T17:33:13.647887+00:00 smithi161 kernel: SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099511632881 2023-09-12T17:33:13.821011+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:14.188993+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:14.717026+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:14.997999+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:15.647030+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:17.522067+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:24.056987+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:24.085900+00:00 smithi161 kernel: libceph: osd5 (1)172.21.15.175:6809 socket closed (con state OPEN) 2023-09-12T17:33:24.112999+00:00 smithi161 kernel: libceph: osd5 (1)172.21.15.175:6809 socket closed (con state OPEN) 2023-09-12T17:33:26.084035+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:26.361001+00:00 smithi161 kernel: libceph: mds0 (1)172.21.15.161:6836 socket closed (con state OPEN) 2023-09-12T17:33:27.278783+00:00 smithi161 kernel: ceph: sync getxattr 00000000b57919fe during filling trace 2023-09-12T17:33:27.278939+00:00 smithi161 kernel: SELinux: inode_doinit_use_xattr: getxattr returned 16 for dev=ceph ino=1099511632881
This time the failure should from the test itself, not the RPC:
2023-09-12T17:33:33.802 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-09-12T17:33:33.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-12T17:33:33.802 DEBUG:teuthology.orchestra.run.smithi161:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-09-12T17:33:34.285 INFO:teuthology.orchestra.run.smithi161.stderr:umount: /home/ubuntu/cephtest/mnt.0: target is busy. 2023-09-12T17:33:34.286 INFO:tasks.workunit.client.0.smithi161.stderr:Updating files: 18% (25/137)^MUpdating files: 19% (27/137)^MUpdating files: 20% (28/137)^MUpdating files: 21% (29/137)^MUpdating files: 22% (31/137)^MUpdating files: 23% (32/137)^MUpdating files: 24% (33/137)^MUpdating files: 25% (35/137)^MUpdating files: 26% (36/137)^MUpdating files: 27% (37/137)^MUpdating files: 28% (39/137)^MUpdating files: 29% (40/137)^MUpdating files: 30% (42/137)^MUpdating files: 31% (43/137)^MUpdating files: 32% (44/137)^MUpdating files: 33% (46/137)^MUpdating files: 34% (47/137)^MUpdating files: 35% (48/137)^MUpdating files: 36% (50/137)^MUpdating files: 37% (51/137)^MUpdating files: 38% (53/137)^MUpdating files: 39% (54/137)^MUpdating files: 40% (55/137)^MUpdating files: 41% (57/137)^MUpdating files: 42% (58/137)^MUpdating files: 43% (59/137)^MUpdating files: 44% (61/137)^MUpdating files: 45% (62/137)^MUpdating files: 46% (64/137)^MUpdating files: 47% (65/137)^MUpdating files: 48% (66/137)^MUpdating files: 49% (68/137)^MUpdating files: 50% (69/137)^MUpdating files: 51% (70/137)^MUpdating files: 52% (72/137)^MUpdating files: 53% (73/137)^MUpdating files: 54% (74/137)^MUpdating files: 55% (76/137)^MUpdating files: 56% (77/137)^MUpdating files: 57% (79/137)^MUpdating files: 58% (80/137)^MUpdating files: 59% (81/137)^MUpdating files: 60% (83/137)^MUpdating files: 61% (84/137)^MUpdating files: 62% (85/137)^MUpdating files: 63% (87/137)^MUpdating files: 64% (88/137)^MUpdating files: 65% (90/137)^MUpdating files: 66% (91/137)^MUpdating files: 67% (92/137)^MUpdating files: 68% (94/137)^MUpdating files: 69% (95/137)^MUpdating files: 70% (96/137)^MUpdating files: 71% (98/137)^MUpdating files: 72% (99/137)^MUpdating files: 73% (101/137)^MUpdating files: 74% (102/137)^MUpdating files: 75% (103/137)^MUpdating files: 76% (105/137)^MUpdating files: 77% (106/137)^MUpdating files: 78% (107/137)^Merror: unable to unlink old 'src/os/FileStore.cc': Input/output error 2023-09-12T17:33:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-09-12T17:33:34.307 DEBUG:tasks.cephfs.kernel_mount:Killing processes on client.0... 2023-09-12T17:33:34.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-09-12T17:33:34.308 DEBUG:teuthology.orchestra.run.smithi161:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf 2023-09-12T17:33:34.319 INFO:tasks.workunit.client.0.smithi161.stderr:Updating files: 78% (108/137)^M/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh: line 24: 136921 Bus error (core dumped) git reset --hard $ver
Created a new tracker for the above crash issue: https://tracker.ceph.com/issues/62861.
Thanks!
#5 Updated by Patrick Donnelly 3 months ago
- Status changed from Fix Under Review to Pending Backport
- Source set to Q/A
- Labels (FS) qa, qa-failure added
#6 Updated by Backport Bot 3 months ago
- Copied to Backport #62865: pacific: cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" added
#7 Updated by Backport Bot 3 months ago
- Copied to Backport #62866: reef: cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" added
#8 Updated by Backport Bot 3 months ago
- Copied to Backport #62867: quincy: cephfs: qa snaptest-git-ceph.sh failed with "got remote process result: 128" added
#9 Updated by Backport Bot 3 months ago
- Tags set to backport_processed
#11 Updated by Venky Shankar 3 months ago
Milind Changire wrote:
Milind, its a bit more useful if you can point to the exact job in the test run.