Bug #63265
Updated by Xiubo Li 7 months ago
https://pulpito.ceph.com/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429579/ Failed with " unable to create file src/mon/MDSMonitor.cc: File exists": <pre> 2023-10-17T01:33:09.933 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files: 0% (17/12853)^MUpdating files: 0% (30/12853)^MUpdating files: 0% (54/12853)^MUpdating files: 1% (129/12853)^MUpdating files: 1% (138/12853)^MUpdating files: 2% (258/12853)^MUpdating files: 2% (298/12853)^MUpdating files: 3% (386/12853)^MUpdating files: 3% (417/12853)^MUpdating files: 3% (490/12853)^MUpdating files: 4% (515/12853)^MUpdating files: 4% (561/12853)^MUpdating files: 4% (627/12853)^MUpdating files: 5% (643/12853)^MUpdating files: 5% (679/12853)^MUpdating files: 5% (722/12853)^MUpdating files: 6% (772/12853)^MUpdating files: 6% (786/12853)^MUpdating files: 6% (853/12853)^MUpdating files: 7% (900/12853)^MUpdating files: 7% (910/12853)^MUpdating files: 7% (977/12853)^MUpdating files: 7% (997/12853)^MUpdating files: 8% (1029/12853)^MUpdating files: 8% (1041/12853)^MUpdating files: 8% (1053/12853)^MUpdating files: 8% (1068/12853)^MUpdating files: 8% (1095/12853)^MUpdating files: 8% (1116/12853)^MUpdating files: 8% (1134/12853)^MUpdating files: 8% (1147/12853)^MUpdating files: 9% (1157/12853)^MUpdating files: 9% (1194/12853)^MUpdating files: 9% (1255/12853)^MUpdating files: 10% (1286/12853)^MUpdating files: 10% (1303/12853)^MUpdating files: 10% (1396/12853)^MUpdating files: 11% (1414/12853)^MUpdating files: 11% (1427/12853)^MUpdating files: 11% (1444/12853)^MUpdating files: 11% (1451/12853)^MUpdating files: 11% (1491/12853)^MUpdating files: 11% (1538/12853)^MUpdating files: 12% (1543/12853)^MUpdating files: 12% (1651/12853)^MUpdating files: 12% (1652/12853)^MUpdating files: 13% (1671/12853)^MUpdating files: 13% (1712/12853)^MUpdating files: 13% (1740/12853)^MUpdating files: 13% (1741/12853)^MUpdating files: 14% (1800/12853)^MUpdating files: 14% (1860/12853)^MUpdating files: 14% (1875/12853)^MUpdating files: 15% (1928/12853)^MUpdating files: 15% (1932/12853)^MUpdating files: 15% (2016/12853)^MUpdating files: 15% (2055/12853)^MUpdating files: 16% (2057/12853)^MUpdating files: 16% (2176/12853)^MUpdating files: 17% (2186/12853)^MUpdating files: 17% (2218/12853)^MUpdating files: 17% (2268/12853)^MUpdating files: 18% (2314/12853)^MUpdating files: 18% (2370/12853)^MUpdating files: 18% (2384/12853)^MUpdating files: 18% (2393/12853)^MUpdating files: 18% (2401/12853)^MUpdating files: 18% (2414/12853)^MUpdating files: 18% (2416/12853)^MUpdating files: 19% (2443/12853)^MUpdating files: 19% (2462/12853)^MUpdating files: 19% (2544/12853)^MUpdating files: 20% (2571/12853)^MUpdating files: 20% (2613/12853)^MUpdating files: 20% (2651/12853)^MUpdating files: 21% (2700/12853)^MUpdating files: 21% (2711/12853)^MUpdating files: 21% (2781/12853)^MUpdating files: 22% (2828/12853)^MUpdating files: 22% (2860/12853)^MUpdating files: 22% (2893/12853)^MUpdating files: 22% (2919/12853)^MUpdating files: 22% (2944/12853)^MUpdating files: 23% (2957/12853)^MUpdating files: 24% (3085/12853)^MUpdating files: 24% (3124/12853)^MUpdating files: 25% (3214/12853)^MUpdating files: 25% (3232/12853)^MUpdating files: 26% (3342/12853)^MUpdating files: 26% (3356/12853)^MUpdating files: 27% (3471/12853)^MUpdating files: 27% (3490/12853)^MUpdating files: 27% (3517/12853)^MUpdating files: 28% (3599/12853)^MUpdating files: 28% (3601/12853)^MUpdating files: 28% (3676/12853)^MUpdating files: 29% (3728/12853)^MUpdating files: 29% (3751/12853)^MUpdating files: 30% (3856/12853)^MUpdating files: 30% (3878/12853)^MUpdating files: 30% (3921/12853)^MUpdating files: 30% (3976/12853)^MUpdating files: 31% (3985/12853)^MUpdating files: 32% (4113/12853)^MUpdating files: 32% (4177/12853)^MUpdating files: 32% (4180/12853)^MUpdating files: 32% (4224/12853)^MUpdating files: 33% (4242/12853)^MUpdating files: 33% (4254/12853)^MUpdating files: 33% (4285/12853)^MUpdating files: 33% (4359/12853)^MUpdating files: 34% (4371/12853)^MUpdating files: 34% (4389/12853)^MUpdating files: 34% (4423/12853)^MUpdating files: 35% (4499/12853)^MUpdating files: 35% (4624/12853)^MUpdating files: 36% (4628/12853)^MUpdating files: 36% (4667/12853)^MUpdating files: 36% (4725/12853)^MUpdating files: 37% (4756/12853)^MUpdating files: 38% (4885/12853)^MUpdating files: 38% (4889/12853)^MUpdating files: 38% (4922/12853)^MUpdating files: 38% (4951/12853)^MUpdating files: 38% (5000/12853)^MUpdating files: 39% (5013/12853)^MUpdating files: 39% (5094/12853)^MUpdating files: 40% (5142/12853)^MUpdating files: 40% (5168/12853)^MUpdating files: 40% (5193/12853)^MUpdating files: 41% (5270/12853)^MUpdating files: 41% (5305/12853)^MUpdating files: 41% (5344/12853)^MUpdating files: 42% (5399/12853)^MUpdating files: 42% (5439/12853)^MUpdating files: 42% (5502/12853)^MUpdating files: 43% (5527/12853)^MUpdating files: 43% (5596/12853)^MUpdating files: 44% (5656/12853)^MUpdating files: 44% (5666/12853)^MUpdating files: 44% (5735/12853)^MUpdating files: 45% (5784/12853)^MUpdating files: 45% (5845/12853)^MUpdating files: 45% (5853/12853)^MUpdating files: 45% (5882/12853)^MUpdating files: 46% (5913/12853)^MUpdating files: 46% (6012/12853)^MUpdating files: 47% (6041/12853)^MUpdating files: 47% (6101/12853)^MUpdating files: 47% (6169/12853)^MUpdating files: 48% (6170/12853)^MUpdating files: 48% (6252/12853)^MUpdating files: 48% (6289/12853)^MUpdating files: 49% (6298/12853)^MUpdating files: 49% (6326/12853)^MUpdating files: 49% (6408/12853)^MUpdating files: 50% (6427/12853)^MUpdating files: 51% (6556/12853)^MUpdating files: 51% (6609/12853)^MUpdating files: 51% (6648/12853)^MUpdating files: 52% (6684/12853)^MUpdating files: 53% (6813/12853)^MUpdating files: 53% (6843/12853)^MUpdating files: 54% (6941/12853)^MUpdating files: 54% (7005/12853)^MUpdating files: 54% (7021/12853)^MUpdating files: 54% (7048/12853)^MUpdating files: 54% (7062/12853)^MUpdating files: 55% (7070/12853)^MUpdating files: 56% (7198/12853)^MUpdating files: 56% (7213/12853)^MUpdating files: 57% (7327/12853)^MUpdating files: 57% (7338/12853)^MUpdating files: 57% (7359/12853)^MUpdating files: 58% (7455/12853)^MUpdating files: 58% (7503/12853)^MUpdating files: 58% (7582/12853)^MUpdating files: 59% (7584/12853)^MUpdating files: 59% (7686/12853)^MUpdating files: 60% (7712/12853)^MUpdating files: 61% (7841/12853)^MUpdating files: 61% (7851/12853)^MUpdating files: 61% (7935/12853)^MUpdating files: 62% (7969/12853)^MUpdating files: 62% (8004/12853)^MUpdating files: 62% (8007/12853)^MUpdating files: 62% (8038/12853)^MUpdating files: 62% (8089/12853)^MUpdating files: 63% (8098/12853)^MUpdating files: 63% (8113/12853)^MUpdating files: 63% (8147/12853)^MUpdating files: 64% (8226/12853)^MUpdating files: 64% (8259/12853)^MUpdating files: 64% (8354/12853)^MUpdating files: 65% (8355/12853)^MUpdating files: 65% (8449/12853)^MUpdating files: 66% (8483/12853)^MUpdating files: 66% (8495/12853)^MUpdating files: 66% (8524/12853)^MUpdating files: 66% (8552/12853)^MUpdating files: 66% (8591/12853)^MUpdating files: 67% (8612/12853)^MUpdating files: 67% (8701/12853)^MUpdating files: 68% (8741/12853)^MUpdating files: 68% (8749/12853)^MUpdating files: 68% (8806/12853)^MUpdating files: 69% (8869/12853)^MUpdating files: 69% (8887/12853)^MUpdating files: 69% (8979/12853)^MUpdating files: 69% (8992/12853)^MUpdating files: 70% (8998/12853)^MUpdating files: 70% (9037/12853)^MUpdating files: 70% (9109/12853)^MUpdating files: 71% (9126/12853)^MUpdating files: 71% (9203/12853)^MUpdating files: 72% (9255/12853)^MUpdating files: 72% (9329/12853)^MUpdating files: 72% (9369/12853)^MUpdating files: 73% (9383/12853)^MUpdating files: 73% (9427/12853)^MUpdating files: 74% (9512/12853)^MUpdating files: 74% (9534/12853)^MUpdating files: 74% (9569/12853)^MUpdating files: 75% (9640/12853)^MUpdating files: 75% (9681/12853)^MUpdating files: 75% (9686/12853)^MUpdating files: 76% (9769/12853)^MUpdating files: 76% (9781/12853)^MUpdating files: 77% (9897/12853)^MUpdating files: 77% (9911/12853)^MUpdating files: 77% (10012/12853)^MUpdating files: 77% (10013/12853)^MUpdating files: 78% (10026/12853)^MUpdating files: 78% (10135/12853)^MUpdating files: 79% (10154/12853)^MUpdating files: 79% (10262/12853)^MUpdating files: 80% (10283/12853)^MUpdating files: 80% (10286/12853)^MUpdating files: 80% (10325/12853)^MUpdating files: 80% (10343/12853)^MUpdating files: 80% (10401/12853)^MUpdating files: 81% (10411/12853)^MUpdating files: 81% (10468/12853)^MUpdating files: 81% (10523/12853)^MUpdating files: 82% (10540/12853)^MUpdating files: 82% (10576/12853)^MUpdating files: 82% (10645/12853)^MUpdating files: 83% (10668/12853)^MUpdating files: 83% (10697/12853)^MUpdating files: 83% (10755/12853)^MUpdating files: 83% (10775/12853)^MUpdating files: 84% (10797/12853)^MUpdating files: 84% (10908/12853)^MUpdating files: 85% (10926/12853)^MUpdating files: 85% (10929/12853)^MUpdating files: 85% (10956/12853)^MUpdating files: 85% (10972/12853)^MUpdating files: 85% (10987/12853)^MUpdating files: 85% (11026/12853)^MUpdating files: 85% (11040/12853)^MUpdating files: 86% (11054/12853)^MUpdating files: 86% (11098/12853)^MUpdating files: 86% (11175/12853)^MUpdating files: 86% (11177/12853)^MUpdating files: 87% (11183/12853)^MUpdating files: 87% (11237/12853)^MUpdating files: 87% (11241/12853)^MUpdating files: 87% (11273/12853)^MUpdating files: 88% (11311/12853)^MUpdating files: 88% (11346/12853)^MUpdating files: 88% (11382/12853)^MUpdating files: 89% (11440/12853)^MUpdating files: 89% (11479/12853)^MUpdating files: 89% (11553/12853)^MUpdating files: 90% (11568/12853)^MUpdating files: 90% (11576/12853)^MUpdating files: 90% (11623/12853)^MUpdating files: 91% (11697/12853)^MUpdating files: 91% (11735/12853)^MUpdating files: 91% (11802/12853)^MUpdating files: 92% (11825/12853)^MUpdating files: 92% (11865/12853)^MUpdating files: 93% (11954/12853)^MUpdating files: 93% (12029/12853)^MUpdating files: 94% (12082/12853)^MUpdating files: 94% (12083/12853)^MUpdating files: 95% (12211/12853)^MUpdating files: 95% (12255/12853)^MUpdating files: 95% (12286/12853)^MUpdating files: 95% (12328/12853)^MUpdating files: 96% (12339/12853)^MUpdating files: 96% (12400/12853)^MUpdating files: 97% (12468/12853)^MUpdating files: 98% (12596/12853)^MUpdating files: 98% (12609/12853)^MUpdating files: 98% (12610/12853)^MUpdating files: 98% (12663/12853)^MUpdating files: 99% (12725/12853)^Merror: unable to create file src/mon/MDSMonitor.cc: File exists ... 2023-10-17T01:33:13.535 INFO:tasks.workunit.client.0.smithi062.stderr:Updating files: 99% (12726/12853)^MUpdating files: 99% (12727/12853)^MUpdating files: 99% (12783/12853)^MUpdating files: 99% (12846/12853)^MUpdating files: 100% (12853/12853)^MUpdating files: 100% (12853/12853), done. 2023-10-17T01:33:13.535 INFO:tasks.workunit.client.0.smithi062.stderr:fatal: Could not reset index file to revision 'v0.1'. 2023-10-17T01:33:13.548 DEBUG:teuthology.orchestra.run:got remote process result: 128 2023-10-17T01:33:13.549 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-17T01:33:13.550 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-17T01:33:14.545 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_ab9fe43522bc41de59746200b95f65d4fbcccffe/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_8cdab074dcca9a68965bc5a50e9c30b691949723/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi062 with status 128: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ab9fe43522bc41de59746200b95f65d4fbcccffe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' </pre>