Ceph » CephFS

https://pulpito.ceph.com/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429579/

Failed with " unable to create file src/mon/MDSMonitor.cc: File exists":

2023-10-17T01:33:09.933 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files:   0% (17/12853)^MUpdating files:   0% (30/12853)^MUpdating files:   0% (54/12853)^MUpdating files:   1% (129/12853)^MUpdating files:   1% (138/12853)^MUpdating files:   2% (258/12853)^MUpdating files:   2% (298/12853)^MUpdating files:   3% (386/12853)^MUpdating files:   3% (417/12853)^MUpdating files:   3% (490/12853)^MUpdating files:   4% (515/12853)^MUpdating files:   4% (561/12853)^MUpdating files:   4% (627/12853)^MUpdating files:   5% (643/12853)^MUpdating files:   5% (679/12853)^MUpdating files:   5% (722/12853)^MUpdating files:   6% (772/12853)^MUpdating files:   6% (786/12853)^MUpdating files:   6% (853/12853)^MUpdating files:   7% (900/12853)^MUpdating files:   7% (910/12853)^MUpdating files:   7% (977/12853)^MUpdating files:   7% (997/12853)^MUpdating files:   8% (1029/12853)^MUpdating files:   8% (1041/12853)^MUpdating files:   8% (1053/12853)^MUpdating files:   8% (1068/12853)^MUpdating files:   8% (1095/12853)^MUpdating files:   8% (1116/12853)^MUpdating files:   8% (1134/12853)^MUpdating files:   8% (1147/12853)^MUpdating files:   9% (1157/12853)^MUpdating files:   9% (1194/12853)^MUpdating files:   9% (1255/12853)^MUpdating files:  10% (1286/12853)^MUpdating files:  10% (1303/12853)^MUpdating files:  10% (1396/12853)^MUpdating files:  11% (1414/12853)^MUpdating files:  11% (1427/12853)^MUpdating files:  11% (1444/12853)^MUpdating files:  11% (1451/12853)^MUpdating files:  11% (1491/12853)^MUpdating files:  11% (1538/12853)^MUpdating files:  12% (1543/12853)^MUpdating files:  12% (1651/12853)^MUpdating files:  12% (1652/12853)^MUpdating files:  13% (1671/12853)^MUpdating files:  13% (1712/12853)^MUpdating files:  13% (1740/12853)^MUpdating files:  13% (1741/12853)^MUpdating files:  14% (1800/12853)^MUpdating files:  14% (1860/12853)^MUpdating files:  14% (1875/12853)^MUpdating files:  15% (1928/12853)^MUpdating files:  15% (1932/12853)^MUpdating files:  15% (2016/12853)^MUpdating files:  15% (2055/12853)^MUpdating files:  16% (2057/12853)^MUpdating files:  16% (2176/12853)^MUpdating files:  17% (2186/12853)^MUpdating files:  17% (2218/12853)^MUpdating files:  17% (2268/12853)^MUpdating files:  18% (2314/12853)^MUpdating files:  18% (2370/12853)^MUpdating files:  18% (2384/12853)^MUpdating files:  18% (2393/12853)^MUpdating files:  18% (2401/12853)^MUpdating files:  18% (2414/12853)^MUpdating files:  18% (2416/12853)^MUpdating files:  19% (2443/12853)^MUpdating files:  19% (2462/12853)^MUpdating files:  19% (2544/12853)^MUpdating files:  20% (2571/12853)^MUpdating files:  20% (2613/12853)^MUpdating files:  20% (2651/12853)^MUpdating files:  21% (2700/12853)^MUpdating files:  21% (2711/12853)^MUpdating files:  21% (2781/12853)^MUpdating files:  22% (2828/12853)^MUpdating files:  22% (2860/12853)^MUpdating files:  22% (2893/12853)^MUpdating files:  22% (2919/12853)^MUpdating files:  22% (2944/12853)^MUpdating files:  23% (2957/12853)^MUpdating files:  24% (3085/12853)^MUpdating files:  24% (3124/12853)^MUpdating files:  25% (3214/12853)^MUpdating files:  25% (3232/12853)^MUpdating files:  26% (3342/12853)^MUpdating files:  26% (3356/12853)^MUpdating files:  27% (3471/12853)^MUpdating files:  27% (3490/12853)^MUpdating files:  27% (3517/12853)^MUpdating files:  28% (3599/12853)^MUpdating files:  28% (3601/12853)^MUpdating files:  28% (3676/12853)^MUpdating files:  29% (3728/12853)^MUpdating files:  29% (3751/12853)^MUpdating files:  30% (3856/12853)^MUpdating files:  30% (3878/12853)^MUpdating files:  30% (3921/12853)^MUpdating files:  30% (3976/12853)^MUpdating files:  31% (3985/12853)^MUpdating files:  32% (4113/12853)^MUpdating files:  32% (4177/12853)^MUpdating files:  32% (4180/12853)^MUpdating files:  32% (4224/12853)^MUpdating files:  33% (4242/12853)^MUpdating files:  33% (4254/12853)^MUpdating files:  33% (4285/12853)^MUpdating files:  33% (4359/12853)^MUpdating files:  34% (4371/12853)^MUpdating files:  34% (4389/12853)^MUpdating files:  34% (4423/12853)^MUpdating files:  35% (4499/12853)^MUpdating files:  35% (4624/12853)^MUpdating files:  36% (4628/12853)^MUpdating files:  36% (4667/12853)^MUpdating files:  36% (4725/12853)^MUpdating files:  37% (4756/12853)^MUpdating files:  38% (4885/12853)^MUpdating files:  38% (4889/12853)^MUpdating files:  38% (4922/12853)^MUpdating files:  38% (4951/12853)^MUpdating files:  38% (5000/12853)^MUpdating files:  39% (5013/12853)^MUpdating files:  39% (5094/12853)^MUpdating files:  40% (5142/12853)^MUpdating files:  40% (5168/12853)^MUpdating files:  40% (5193/12853)^MUpdating files:  41% (5270/12853)^MUpdating files:  41% (5305/12853)^MUpdating files:  41% (5344/12853)^MUpdating files:  42% (5399/12853)^MUpdating files:  42% (5439/12853)^MUpdating files:  42% (5502/12853)^MUpdating files:  43% (5527/12853)^MUpdating files:  43% (5596/12853)^MUpdating files:  44% (5656/12853)^MUpdating files:  44% (5666/12853)^MUpdating files:  44% (5735/12853)^MUpdating files:  45% (5784/12853)^MUpdating files:  45% (5845/12853)^MUpdating files:  45% (5853/12853)^MUpdating files:  45% (5882/12853)^MUpdating files:  46% (5913/12853)^MUpdating files:  46% (6012/12853)^MUpdating files:  47% (6041/12853)^MUpdating files:  47% (6101/12853)^MUpdating files:  47% (6169/12853)^MUpdating files:  48% (6170/12853)^MUpdating files:  48% (6252/12853)^MUpdating files:  48% (6289/12853)^MUpdating files:  49% (6298/12853)^MUpdating files:  49% (6326/12853)^MUpdating files:  49% (6408/12853)^MUpdating files:  50% (6427/12853)^MUpdating files:  51% (6556/12853)^MUpdating files:  51% (6609/12853)^MUpdating files:  51% (6648/12853)^MUpdating files:  52% (6684/12853)^MUpdating files:  53% (6813/12853)^MUpdating files:  53% (6843/12853)^MUpdating files:  54% (6941/12853)^MUpdating files:  54% (7005/12853)^MUpdating files:  54% (7021/12853)^MUpdating files:  54% (7048/12853)^MUpdating files:  54% (7062/12853)^MUpdating files:  55% (7070/12853)^MUpdating files:  56% (7198/12853)^MUpdating files:  56% (7213/12853)^MUpdating files:  57% (7327/12853)^MUpdating files:  57% (7338/12853)^MUpdating files:  57% (7359/12853)^MUpdating files:  58% (7455/12853)^MUpdating files:  58% (7503/12853)^MUpdating files:  58% (7582/12853)^MUpdating files:  59% (7584/12853)^MUpdating files:  59% (7686/12853)^MUpdating files:  60% (7712/12853)^MUpdating files:  61% (7841/12853)^MUpdating files:  61% (7851/12853)^MUpdating files:  61% (7935/12853)^MUpdating files:  62% (7969/12853)^MUpdating files:  62% (8004/12853)^MUpdating files:  62% (8007/12853)^MUpdating files:  62% (8038/12853)^MUpdating files:  62% (8089/12853)^MUpdating files:  63% (8098/12853)^MUpdating files:  63% (8113/12853)^MUpdating files:  63% (8147/12853)^MUpdating files:  64% (8226/12853)^MUpdating files:  64% (8259/12853)^MUpdating files:  64% (8354/12853)^MUpdating files:  65% (8355/12853)^MUpdating files:  65% (8449/12853)^MUpdating files:  66% (8483/12853)^MUpdating files:  66% (8495/12853)^MUpdating files:  66% (8524/12853)^MUpdating files:  66% (8552/12853)^MUpdating files:  66% (8591/12853)^MUpdating files:  67% (8612/12853)^MUpdating files:  67% (8701/12853)^MUpdating files:  68% (8741/12853)^MUpdating files:  68% (8749/12853)^MUpdating files:  68% (8806/12853)^MUpdating files:  69% (8869/12853)^MUpdating files:  69% (8887/12853)^MUpdating files:  69% (8979/12853)^MUpdating files:  69% (8992/12853)^MUpdating files:  70% (8998/12853)^MUpdating files:  70% (9037/12853)^MUpdating files:  70% (9109/12853)^MUpdating files:  71% (9126/12853)^MUpdating files:  71% (9203/12853)^MUpdating files:  72% (9255/12853)^MUpdating files:  72% (9329/12853)^MUpdating files:  72% (9369/12853)^MUpdating files:  73% (9383/12853)^MUpdating files:  73% (9427/12853)^MUpdating files:  74% (9512/12853)^MUpdating files:  74% (9534/12853)^MUpdating files:  74% (9569/12853)^MUpdating files:  75% (9640/12853)^MUpdating files:  75% (9681/12853)^MUpdating files:  75% (9686/12853)^MUpdating files:  76% (9769/12853)^MUpdating files:  76% (9781/12853)^MUpdating files:  77% (9897/12853)^MUpdating files:  77% (9911/12853)^MUpdating files:  77% (10012/12853)^MUpdating files:  77% (10013/12853)^MUpdating files:  78% (10026/12853)^MUpdating files:  78% (10135/12853)^MUpdating files:  79% (10154/12853)^MUpdating files:  79% (10262/12853)^MUpdating files:  80% (10283/12853)^MUpdating files:  80% (10286/12853)^MUpdating files:  80% (10325/12853)^MUpdating files:  80% (10343/12853)^MUpdating files:  80% (10401/12853)^MUpdating files:  81% (10411/12853)^MUpdating files:  81% (10468/12853)^MUpdating files:  81% (10523/12853)^MUpdating files:  82% (10540/12853)^MUpdating files:  82% (10576/12853)^MUpdating files:  82% (10645/12853)^MUpdating files:  83% (10668/12853)^MUpdating files:  83% (10697/12853)^MUpdating files:  83% (10755/12853)^MUpdating files:  83% (10775/12853)^MUpdating files:  84% (10797/12853)^MUpdating files:  84% (10908/12853)^MUpdating files:  85% (10926/12853)^MUpdating files:  85% (10929/12853)^MUpdating files:  85% (10956/12853)^MUpdating files:  85% (10972/12853)^MUpdating files:  85% (10987/12853)^MUpdating files:  85% (11026/12853)^MUpdating files:  85% (11040/12853)^MUpdating files:  86% (11054/12853)^MUpdating files:  86% (11098/12853)^MUpdating files:  86% (11175/12853)^MUpdating files:  86% (11177/12853)^MUpdating files:  87% (11183/12853)^MUpdating files:  87% (11237/12853)^MUpdating files:  87% (11241/12853)^MUpdating files:  87% (11273/12853)^MUpdating files:  88% (11311/12853)^MUpdating files:  88% (11346/12853)^MUpdating files:  88% (11382/12853)^MUpdating files:  89% (11440/12853)^MUpdating files:  89% (11479/12853)^MUpdating files:  89% (11553/12853)^MUpdating files:  90% (11568/12853)^MUpdating files:  90% (11576/12853)^MUpdating files:  90% (11623/12853)^MUpdating files:  91% (11697/12853)^MUpdating files:  91% (11735/12853)^MUpdating files:  91% (11802/12853)^MUpdating files:  92% (11825/12853)^MUpdating files:  92% (11865/12853)^MUpdating files:  93% (11954/12853)^MUpdating files:  93% (12029/12853)^MUpdating files:  94% (12082/12853)^MUpdating files:  94% (12083/12853)^MUpdating files:  95% (12211/12853)^MUpdating files:  95% (12255/12853)^MUpdating files:  95% (12286/12853)^MUpdating files:  95% (12328/12853)^MUpdating files:  96% (12339/12853)^MUpdating files:  96% (12400/12853)^MUpdating files:  97% (12468/12853)^MUpdating files:  98% (12596/12853)^MUpdating files:  98% (12609/12853)^MUpdating files:  98% (12610/12853)^MUpdating files:  98% (12663/12853)^MUpdating files:  99% (12725/12853)^Merror: unable to create file src/mon/MDSMonitor.cc: File exists
...
2023-10-17T01:33:13.535 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files:  99% (12726/12853)^MUpdating files:  99% (12727/12853)^MUpdating files:  99% (12783/12853)^MUpdating files:  99% (12846/12853)^MUpdating files: 100% (12853/12853)^MUpdating files: 100% (12853/12853), done.
2023-10-17T01:33:13.535 INFO:tasks.workunit.client.0.smithi062.stderr:fatal: Could not reset index file to revision 'v0.1'.
2023-10-17T01:33:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-10-17T01:33:13.549 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...
2023-10-17T01:33:13.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2023-10-17T01:33:14.545 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 145, in task
    _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients
    p.spawn(_run_tests, ctx, refspec, role, [unit], env,
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 424, in _run_tests
    remote.run(
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi062 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ab9fe43522bc41de59746200b95f65d4fbcccffe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh'