Project

General

Profile

Actions
Copy link

Bug #63265

open

qa: fs/snaps/snaptest-git-ceph.sh failed when reseting to tag 'v0.1'

Added by Xiubo Li 7 months ago. Updated about 2 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Xiubo Li
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429579/

Failed with " unable to create file src/mon/MDSMonitor.cc: File exists":

2023-10-17T01:33:09.933 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files:   0% (17/12853)^MUpdating files:   0% (30/12853)^MUpdating files:   0% (54/12853)^MUpdating files:   1% (129/12853)^MUpdating files:   1% (138/12853)^MUpdating files:   2% (258/12853)^MUpdating files:   2% (298/12853)^MUpdating files:   3% (386/12853)^MUpdating files:   3% (417/12853)^MUpdating files:   3% (490/12853)^MUpdating files:   4% (515/12853)^MUpdating files:   4% (561/12853)^MUpdating files:   4% (627/12853)^MUpdating files:   5% (643/12853)^MUpdating files:   5% (679/12853)^MUpdating files:   5% (722/12853)^MUpdating files:   6% (772/12853)^MUpdating files:   6% (786/12853)^MUpdating files:   6% (853/12853)^MUpdating files:   7% (900/12853)^MUpdating files:   7% (910/12853)^MUpdating files:   7% (977/12853)^MUpdating files:   7% (997/12853)^MUpdating files:   8% (1029/12853)^MUpdating files:   8% (1041/12853)^MUpdating files:   8% (1053/12853)^MUpdating files:   8% (1068/12853)^MUpdating files:   8% (1095/12853)^MUpdating files:   8% (1116/12853)^MUpdating files:   8% (1134/12853)^MUpdating files:   8% (1147/12853)^MUpdating files:   9% (1157/12853)^MUpdating files:   9% (1194/12853)^MUpdating files:   9% (1255/12853)^MUpdating files:  10% (1286/12853)^MUpdating files:  10% (1303/12853)^MUpdating files:  10% (1396/12853)^MUpdating files:  11% (1414/12853)^MUpdating files:  11% (1427/12853)^MUpdating files:  11% (1444/12853)^MUpdating files:  11% (1451/12853)^MUpdating files:  11% (1491/12853)^MUpdating files:  11% (1538/12853)^MUpdating files:  12% (1543/12853)^MUpdating files:  12% (1651/12853)^MUpdating files:  12% (1652/12853)^MUpdating files:  13% (1671/12853)^MUpdating files:  13% (1712/12853)^MUpdating files:  13% (1740/12853)^MUpdating files:  13% (1741/12853)^MUpdating files:  14% (1800/12853)^MUpdating files:  14% (1860/12853)^MUpdating files:  14% (1875/12853)^MUpdating files:  15% (1928/12853)^MUpdating files:  15% (1932/12853)^MUpdating files:  15% (2016/12853)^MUpdating files:  15% (2055/12853)^MUpdating files:  16% (2057/12853)^MUpdating files:  16% (2176/12853)^MUpdating files:  17% (2186/12853)^MUpdating files:  17% (2218/12853)^MUpdating files:  17% (2268/12853)^MUpdating files:  18% (2314/12853)^MUpdating files:  18% (2370/12853)^MUpdating files:  18% (2384/12853)^MUpdating files:  18% (2393/12853)^MUpdating files:  18% (2401/12853)^MUpdating files:  18% (2414/12853)^MUpdating files:  18% (2416/12853)^MUpdating files:  19% (2443/12853)^MUpdating files:  19% (2462/12853)^MUpdating files:  19% (2544/12853)^MUpdating files:  20% (2571/12853)^MUpdating files:  20% (2613/12853)^MUpdating files:  20% (2651/12853)^MUpdating files:  21% (2700/12853)^MUpdating files:  21% (2711/12853)^MUpdating files:  21% (2781/12853)^MUpdating files:  22% (2828/12853)^MUpdating files:  22% (2860/12853)^MUpdating files:  22% (2893/12853)^MUpdating files:  22% (2919/12853)^MUpdating files:  22% (2944/12853)^MUpdating files:  23% (2957/12853)^MUpdating files:  24% (3085/12853)^MUpdating files:  24% (3124/12853)^MUpdating files:  25% (3214/12853)^MUpdating files:  25% (3232/12853)^MUpdating files:  26% (3342/12853)^MUpdating files:  26% (3356/12853)^MUpdating files:  27% (3471/12853)^MUpdating files:  27% (3490/12853)^MUpdating files:  27% (3517/12853)^MUpdating files:  28% (3599/12853)^MUpdating files:  28% (3601/12853)^MUpdating files:  28% (3676/12853)^MUpdating files:  29% (3728/12853)^MUpdating files:  29% (3751/12853)^MUpdating files:  30% (3856/12853)^MUpdating files:  30% (3878/12853)^MUpdating files:  30% (3921/12853)^MUpdating files:  30% (3976/12853)^MUpdating files:  31% (3985/12853)^MUpdating files:  32% (4113/12853)^MUpdating files:  32% (4177/12853)^MUpdating files:  32% (4180/12853)^MUpdating files:  32% (4224/12853)^MUpdating files:  33% (4242/12853)^MUpdating files:  33% (4254/12853)^MUpdating files:  33% (4285/12853)^MUpdating files:  33% (4359/12853)^MUpdating files:  34% (4371/12853)^MUpdating files:  34% (4389/12853)^MUpdating files:  34% (4423/12853)^MUpdating files:  35% (4499/12853)^MUpdating files:  35% (4624/12853)^MUpdating files:  36% (4628/12853)^MUpdating files:  36% (4667/12853)^MUpdating files:  36% (4725/12853)^MUpdating files:  37% (4756/12853)^MUpdating files:  38% (4885/12853)^MUpdating files:  38% (4889/12853)^MUpdating files:  38% (4922/12853)^MUpdating files:  38% (4951/12853)^MUpdating files:  38% (5000/12853)^MUpdating files:  39% (5013/12853)^MUpdating files:  39% (5094/12853)^MUpdating files:  40% (5142/12853)^MUpdating files:  40% (5168/12853)^MUpdating files:  40% (5193/12853)^MUpdating files:  41% (5270/12853)^MUpdating files:  41% (5305/12853)^MUpdating files:  41% (5344/12853)^MUpdating files:  42% (5399/12853)^MUpdating files:  42% (5439/12853)^MUpdating files:  42% (5502/12853)^MUpdating files:  43% (5527/12853)^MUpdating files:  43% (5596/12853)^MUpdating files:  44% (5656/12853)^MUpdating files:  44% (5666/12853)^MUpdating files:  44% (5735/12853)^MUpdating files:  45% (5784/12853)^MUpdating files:  45% (5845/12853)^MUpdating files:  45% (5853/12853)^MUpdating files:  45% (5882/12853)^MUpdating files:  46% (5913/12853)^MUpdating files:  46% (6012/12853)^MUpdating files:  47% (6041/12853)^MUpdating files:  47% (6101/12853)^MUpdating files:  47% (6169/12853)^MUpdating files:  48% (6170/12853)^MUpdating files:  48% (6252/12853)^MUpdating files:  48% (6289/12853)^MUpdating files:  49% (6298/12853)^MUpdating files:  49% (6326/12853)^MUpdating files:  49% (6408/12853)^MUpdating files:  50% (6427/12853)^MUpdating files:  51% (6556/12853)^MUpdating files:  51% (6609/12853)^MUpdating files:  51% (6648/12853)^MUpdating files:  52% (6684/12853)^MUpdating files:  53% (6813/12853)^MUpdating files:  53% (6843/12853)^MUpdating files:  54% (6941/12853)^MUpdating files:  54% (7005/12853)^MUpdating files:  54% (7021/12853)^MUpdating files:  54% (7048/12853)^MUpdating files:  54% (7062/12853)^MUpdating files:  55% (7070/12853)^MUpdating files:  56% (7198/12853)^MUpdating files:  56% (7213/12853)^MUpdating files:  57% (7327/12853)^MUpdating files:  57% (7338/12853)^MUpdating files:  57% (7359/12853)^MUpdating files:  58% (7455/12853)^MUpdating files:  58% (7503/12853)^MUpdating files:  58% (7582/12853)^MUpdating files:  59% (7584/12853)^MUpdating files:  59% (7686/12853)^MUpdating files:  60% (7712/12853)^MUpdating files:  61% (7841/12853)^MUpdating files:  61% (7851/12853)^MUpdating files:  61% (7935/12853)^MUpdating files:  62% (7969/12853)^MUpdating files:  62% (8004/12853)^MUpdating files:  62% (8007/12853)^MUpdating files:  62% (8038/12853)^MUpdating files:  62% (8089/12853)^MUpdating files:  63% (8098/12853)^MUpdating files:  63% (8113/12853)^MUpdating files:  63% (8147/12853)^MUpdating files:  64% (8226/12853)^MUpdating files:  64% (8259/12853)^MUpdating files:  64% (8354/12853)^MUpdating files:  65% (8355/12853)^MUpdating files:  65% (8449/12853)^MUpdating files:  66% (8483/12853)^MUpdating files:  66% (8495/12853)^MUpdating files:  66% (8524/12853)^MUpdating files:  66% (8552/12853)^MUpdating files:  66% (8591/12853)^MUpdating files:  67% (8612/12853)^MUpdating files:  67% (8701/12853)^MUpdating files:  68% (8741/12853)^MUpdating files:  68% (8749/12853)^MUpdating files:  68% (8806/12853)^MUpdating files:  69% (8869/12853)^MUpdating files:  69% (8887/12853)^MUpdating files:  69% (8979/12853)^MUpdating files:  69% (8992/12853)^MUpdating files:  70% (8998/12853)^MUpdating files:  70% (9037/12853)^MUpdating files:  70% (9109/12853)^MUpdating files:  71% (9126/12853)^MUpdating files:  71% (9203/12853)^MUpdating files:  72% (9255/12853)^MUpdating files:  72% (9329/12853)^MUpdating files:  72% (9369/12853)^MUpdating files:  73% (9383/12853)^MUpdating files:  73% (9427/12853)^MUpdating files:  74% (9512/12853)^MUpdating files:  74% (9534/12853)^MUpdating files:  74% (9569/12853)^MUpdating files:  75% (9640/12853)^MUpdating files:  75% (9681/12853)^MUpdating files:  75% (9686/12853)^MUpdating files:  76% (9769/12853)^MUpdating files:  76% (9781/12853)^MUpdating files:  77% (9897/12853)^MUpdating files:  77% (9911/12853)^MUpdating files:  77% (10012/12853)^MUpdating files:  77% (10013/12853)^MUpdating files:  78% (10026/12853)^MUpdating files:  78% (10135/12853)^MUpdating files:  79% (10154/12853)^MUpdating files:  79% (10262/12853)^MUpdating files:  80% (10283/12853)^MUpdating files:  80% (10286/12853)^MUpdating files:  80% (10325/12853)^MUpdating files:  80% (10343/12853)^MUpdating files:  80% (10401/12853)^MUpdating files:  81% (10411/12853)^MUpdating files:  81% (10468/12853)^MUpdating files:  81% (10523/12853)^MUpdating files:  82% (10540/12853)^MUpdating files:  82% (10576/12853)^MUpdating files:  82% (10645/12853)^MUpdating files:  83% (10668/12853)^MUpdating files:  83% (10697/12853)^MUpdating files:  83% (10755/12853)^MUpdating files:  83% (10775/12853)^MUpdating files:  84% (10797/12853)^MUpdating files:  84% (10908/12853)^MUpdating files:  85% (10926/12853)^MUpdating files:  85% (10929/12853)^MUpdating files:  85% (10956/12853)^MUpdating files:  85% (10972/12853)^MUpdating files:  85% (10987/12853)^MUpdating files:  85% (11026/12853)^MUpdating files:  85% (11040/12853)^MUpdating files:  86% (11054/12853)^MUpdating files:  86% (11098/12853)^MUpdating files:  86% (11175/12853)^MUpdating files:  86% (11177/12853)^MUpdating files:  87% (11183/12853)^MUpdating files:  87% (11237/12853)^MUpdating files:  87% (11241/12853)^MUpdating files:  87% (11273/12853)^MUpdating files:  88% (11311/12853)^MUpdating files:  88% (11346/12853)^MUpdating files:  88% (11382/12853)^MUpdating files:  89% (11440/12853)^MUpdating files:  89% (11479/12853)^MUpdating files:  89% (11553/12853)^MUpdating files:  90% (11568/12853)^MUpdating files:  90% (11576/12853)^MUpdating files:  90% (11623/12853)^MUpdating files:  91% (11697/12853)^MUpdating files:  91% (11735/12853)^MUpdating files:  91% (11802/12853)^MUpdating files:  92% (11825/12853)^MUpdating files:  92% (11865/12853)^MUpdating files:  93% (11954/12853)^MUpdating files:  93% (12029/12853)^MUpdating files:  94% (12082/12853)^MUpdating files:  94% (12083/12853)^MUpdating files:  95% (12211/12853)^MUpdating files:  95% (12255/12853)^MUpdating files:  95% (12286/12853)^MUpdating files:  95% (12328/12853)^MUpdating files:  96% (12339/12853)^MUpdating files:  96% (12400/12853)^MUpdating files:  97% (12468/12853)^MUpdating files:  98% (12596/12853)^MUpdating files:  98% (12609/12853)^MUpdating files:  98% (12610/12853)^MUpdating files:  98% (12663/12853)^MUpdating files:  99% (12725/12853)^Merror: unable to create file src/mon/MDSMonitor.cc: File exists
...
2023-10-17T01:33:13.535 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files:  99% (12726/12853)^MUpdating files:  99% (12727/12853)^MUpdating files:  99% (12783/12853)^MUpdating files:  99% (12846/12853)^MUpdating files: 100% (12853/12853)^MUpdating files: 100% (12853/12853), done.
2023-10-17T01:33:13.535 INFO:tasks.workunit.client.0.smithi062.stderr:fatal: Could not reset index file to revision 'v0.1'.
2023-10-17T01:33:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-10-17T01:33:13.549 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
2023-10-17T01:33:13.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2023-10-17T01:33:14.545 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 145, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients
    p.spawn(_run_tests, ctx, refspec, role, [unit], env,
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 424, in _run_tests
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi062 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ab9fe43522bc41de59746200b95f65d4fbcccffe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'
Actions
Copy link
 #1

Updated by Xiubo Li 7 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Milind Changire 7 months ago

  • Assignee set to Xiubo Li
Actions
Copy link
 #4

Updated by Xiubo Li 5 months ago

Xiubo Li wrote:

https://pulpito.ceph.com/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429579/

Failed with " unable to create file src/mon/MDSMonitor.cc: File exists":

[...]

From the mds.a log file, the src/mon/MDSMonitor.cc file was successfully created without any issue:

2023-10-17T01:32:43.352+0000 7f28bdbf6700 20 mds.0.server lat 0.001938
2023-10-17T01:32:43.352+0000 7f28b6be8700  5 mds.0.log _submit_thread 354800868~4258 : EUpdate unlink_local [metablob 0x10000002e46.1*, 4 dirs]
2023-10-17T01:32:43.352+0000 7f28bdbf6700  1 -- [v2:172.21.15.62:6832/3723432109,v1:172.21.15.62:6833/3723432109] <== client.4676 172.21.15.62:0/1312196696 21204 ==== client_request(client.4676:65053 create #0x1000000392c/MDSMonitor.cc 2023-10-17T01:32:43.352476+0000 caller_uid=1000, caller_gid=1285{6,36,1000,1285,}) v6 ==== 201+0+62 (crc 0 0 0) 0x5568b2151b00 con 0x5568b2383400
2023-10-17T01:32:43.352+0000 7f28bdbf6700  4 mds.0.server handle_client_request client_request(client.4676:65053 create #0x1000000392c/MDSMonitor.cc 2023-10-17T01:32:43.352476+0000 caller_uid=1000, caller_gid=1285{6,36,1000,1285,}) v6
2023-10-17T01:32:43.352+0000 7f28bdbf6700 20 mds.0.89 get_session have 0x5568ae392800 client.4676 172.21.15.62:0/1312196696 state open
2023-10-17T01:32:43.352+0000 7f28bdbf6700 15 mds.0.server  oldest_client_tid=64732
2023-10-17T01:32:43.352+0000 7f28bdbf6700  7 mds.0.cache request_start request(client.4676:65053 nref=2 cr=0x5568b2151b00)
2023-10-17T01:32:43.352+0000 7f28bdbf6700  7 mds.0.server dispatch_client_request client_request(client.4676:65053 create #0x1000000392c/MDSMonitor.cc 2023-10-17T01:32:43.352476+0000 caller_uid=1000, caller_gid=1285{6,36,1000,1285,}) v6
2023-10-17T01:32:43.352+0000 7f28bdbf6700  7 mds.0.server open w/ O_CREAT on #0x1000000392c/MDSMonitor.cc
2023-10-17T01:32:43.352+0000 7f28bdbf6700 10 mds.0.server rdlock_path_xlock_dentry request(client.4676:65053 nref=2 cr=0x5568b2151b00) #0x1000000392c/MDSMonitor.cc
2023-10-17T01:32:43.352+0000 7f28bdbf6700  7 mds.0.cache traverse: opening base ino 0x1000000392c snap head
2023-10-17T01:32:43.352+0000 7f28bdbf6700 10 mds.0.locker found lock cache for create on [inode 0x1000000392c [...1fa,head] /client.0/tmp/ceph/src/mon/ auth v7336 pv10235 ap=10 DIRTYPARENT f(v0 m2023-10-17T01:31:19.978737+0000 16=16+0)->f(v0 m2023-10-17T01:32:43.350476+0000 17=17+0) n(v0 rc2023-10-17T01:31:19.978737+0000 b1086509 17=16+1)->n(v0 rc2023-10-17T01:32:43.350476+0000 b981500 18=17+1) (iauth sync r=1) (isnap sync r=2) (inest lock w=3) (ipolicy sync r=1) (ifile excl w=2) (iversion lock w=8 last_client=4676) caps={4676=pAsLsXsFsxcr/pAsLsXsFsxcral@106},l=4676 | request=0 lock=6 dirfrag=1 caps=1 dirtyparent=1 dirty=0 waiter=0 authpin=1 0x5568b10dfb80]
2023-10-17T01:32:43.353+0000 7f28bdbf6700 12 mds.0.cache traverse: path seg depth 0 'MDSMonitor.cc' snapid head
2023-10-17T01:32:43.353+0000 7f28bdbf6700 20 mds.0.cache.dir(0x1000000392c) lookup (MDSMonitor.cc, 'head')
2023-10-17T01:32:43.353+0000 7f28bdbf6700 20 mds.0.cache.dir(0x1000000392c)   hit -> (MDSMonitor.cc,head)
...
2023-10-17T01:32:43.353+0000 7f28bdbf6700 10 mds.0.server early_reply 0 ((0) Success) client_request(client.4676:65053 create #0x1000000392c/MDSMonitor.cc 2023-10-17T01:32:43.352476+0000 caller_uid=1000, caller_gid=1285{6,36,1000,1285,}) v6
...
2023-10-17T01:32:43.472+0000 7f28b7bea700 20 mds.0.bal hit_dir 1 pop is 8.99723, frag * size 18 [pop IRD:[C 2.95e+00] IWR:[C 9.00e+00] RDR:[C 0.00e+00] FET:[C 0.00e+00] STR:[C 0.00e+00] *LOAD:20.9]
2023-10-17T01:32:43.472+0000 7f28b7bea700  7 mds.0.server reply_client_request 0 ((0) Success) client_request(client.4676:65053 create #0x1000000392c/MDSMonitor.cc 2023-10-17T01:32:43.352476+0000 caller_uid=1000, caller_gid=1285{6,36,1000,1285,}) v6

Xiubo Li wrote:

reef: https://pulpito.ceph.com/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435313/

And for this test failure, there is not any log about failing to create any file when resetting head to v0.74:

2023-10-24T23:28:47.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-24T23:28:47.483+0000 7f4b16cc1700  1 -- 172.21.15.29:0/989232246 shutdown_connections
2023-10-24T23:28:47.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-24T23:28:47.483+0000 7f4b16cc1700  1 -- 172.21.15.29:0/989232246 wait complete.
2023-10-24T23:28:47.492 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 193
2023-10-24T23:28:47.508 INFO:tasks.mds_thrash.fs.[a]:waiting till mds map indicates mds.c is in active, standby or standby-replay
2023-10-24T23:28:48.300 INFO:tasks.workunit.client.0.smithi029.stderr:Updating files:  46% (90/192)^MUpdating files:  47% (91/192)^MUpdating files:  48% (93/192)^MUpdating files:  49% (95/192)^MUpdating files:  50% (96/192)^MUpdating files:  51% (98/192)^MUpdating files:  52% (100/192)^MUpdating files:  53% (102/192)^MUpdating files:  54% (104/192)^MUpdating files:  55% (106/192)^MUpdating files:  56% (108/192)^MUpdating files:  57% (110/192)^MUpdating files:  58% (112/192)^MUpdating files:  59% (114/192)^MUpdating files:  60% (116/192)^MUpdating files:  61% (118/192)^MUpdating files:  62% (120/192)^MUpdating files:  63% (121/192)^MUpdating files:  64% (123/192)^MUpdating files:  65% (125/192)^MUpdating files:  66% (127/192)^MUpdating files:  67% (129/192)^MUpdating files:  68% (131/192)^MUpdating files:  69% (133/192)^MUpdating files:  70% (135/192)^MUpdating files:  71% (137/192)^MUpdating files:  72% (139/192)^MUpdating files:  73% (141/192)^MUpdating files:  74% (143/192)^MUpdating files:  75% (144/192)^MUpdating files:  76% (146/192)^MUpdating files:  77% (148/192)^MUpdating files:  78% (150/192)^MUpdating files:  79% (152/192)^MUpdating files:  80% (154/192)^MUpdating files:  81% (156/192)^MUpdating files:  82% (158/192)^MUpdating files:  83% (160/192)^MUpdating files:  84% (162/192)^MUpdating files:  85% (164/192)^MUpdating files:  86% (166/192)^MUpdating files:  87% (168/192)^MUpdating files:  88% (169/192)^MUpdating files:  89% (171/192)^MUpdating files:  89% (172/192)^MUpdating files:  90% (173/192)^MUpdating files:  91% (175/192)^MUpdating files:  92% (177/192)^MUpdating files:  93% (179/192)^MUpdating files:  94% (181/192)^MUpdating files:  95% (183/192)^MUpdating files:  96% (185/192)^MUpdating files:  97% (187/192)^MUpdating files:  98% (189/192)^MUpdating files:  99% (191/192)^MUpdating files: 100% (192/192)^MUpdating files: 100% (192/192), done.
2023-10-24T23:28:48.300 INFO:tasks.workunit.client.0.smithi029.stderr:fatal: Could not reset index file to revision 'v0.74'.
2023-10-24T23:28:48.312 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-10-24T23:28:48.313 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...

And also I didn't find any failure from mds logs and kclient logs.

While for both the above two failures I could see the git command itself crashed:

2023-10-17T01:40:07.786 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2023-10-17T01:40:07.826 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429579/remote/smithi183/coredump/1697506436.127268.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'git reset --hard v0.7', real uid: 1000, effective uid: 1000, real gid: 1285, effective gid: 1285, execfn: '/usr/bin/git', platform: 'x86_64'

And

2023-10-24T23:38:23.501 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2023-10-24T23:38:23.780 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/yuriw-2023-10-24_00:06:34-fs-wip-yuri6-testing-2023-10-23-1148-reef-distro-default-smithi/7435313/remote/smithi186/coredump/1698190173.43665.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'git reset --hard v0.86', real uid: 1000, effective uid: 1000, real gid: 1267, effective gid: 1267, execfn: '/usr/bin/git', platform: 'x86_64'

I just suspect this should be a git tool's bug.

Actions
Copy link
 #5

Updated by Xiubo Li 5 months ago

  • Status changed from New to In Progress
Actions
Copy link
 #6

Updated by Venky Shankar 4 months ago

Xiubo, a pretty similar issue got hit here: https://pulpito.ceph.com/vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi/7511467/

However, this instance the errno is EIO and there is no git binary crash dump.

2024-01-11T17:50:09.316 INFO:tasks.workunit.client.0.smithi064.stderr:Updating files:  59% (114/193)^MUpdating files:  60% (116/193)^MUpdating files:  61% (118/193)^MUpdating files:  62% (120/193)^MUpdating files:  63% (122/193)^MUpdating files:  64% (124/193)^MUpdating files:  65% (126/193)^MUpdating files:  66% (128/193)^MUpdating files:  67% (130/193)^MUpdating files:  68% (132/193)^MUpdating files:  69% (134/193)^MUpdating files:  70% (136/193)^MUpdating files:  71% (138/193)^MUpdating files:  72% (139/193)^MUpdating files:  73% (141/193)^MUpdating files:  74% (143/193)^MUpdating files:  75% (145/193)^MUpdating files:  76% (147/193)^MUpdating files:  77% (149/193)^MUpdating files:  78% (151/193)^MUpdating files:  79% (153/193)^MUpdating files:  80% (155/193)^MUpdating files:  81% (157/193)^MUpdating files:  82% (159/193)^MUpdating files:  83% (161/193)^MUpdating files:  84% (163/193)^Merror: unable to create file src/osd/OSD.cc: Input/output error
2024-01-11T17:50:09.317 INFO:tasks.workunit.client.0.smithi064.stderr:error: unable to create file src/osd/OSD.h: Input/output error

Nothing in MDS or the kernel ring buffer.

Actions
Copy link
 #7

Updated by Xiubo Li 4 months ago

Venky Shankar wrote:

Xiubo, a pretty similar issue got hit here: https://pulpito.ceph.com/vshankar-2024-01-10_15:00:23-fs-wip-vshankar-testing-20240103.072409-1-testing-default-smithi/7511467/

However, this instance the errno is EIO and there is no git binary crash dump.

[...]

Nothing in MDS or the kernel ring buffer.

Okay, let me have a look later. Thanks!

Actions
Copy link
 #8

Updated by Venky Shankar 3 months ago

/a/vshankar-2024-01-22_07:03:31-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7525694

Actions
Copy link
 #9

Updated by Rishabh Dave about 2 months ago

https://pulpito.ceph.com/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588250.

Same error occurred on this job but with a different version number.

Following is the log entry - stderr:fatal: Could not reset index file to revision 'v0.64'.

Actions
Copy link
 #10

Updated by Venky Shankar about 2 months ago

/a/yuriw-2024-03-16_15:03:17-fs-wip-yuri10-testing-2024-03-15-1653-reef-distro-default-smithi/7606232

Failure is with a fuse client. Nothing in client logs that point to an Input/output error.

Actions
Copy link

Also available in: Atom PDF