Ceph » CephFS

Hmmm... similar to https://tracker.ceph.com/issues/17172

Xiubo Li wrote:

This should be the same issue with https://tracker.ceph.com/issues/59343.

Which means this can happen with kclient runs.

I'm running into this here: https://pulpito.ceph.com/vshankar-2023-05-17_11:52:03-fs-wip-vshankar-testing-20230509.090020-6-testing-default-smithi/7276651/

But the failure is a bit different

2023-05-17T12:36:49.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-05-17T12:36:49.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-05-17T12:36:54.701 INFO:tasks.workunit.client.0.smithi092.stderr:Updating files:   7% (881/11784)^MUpdating files:   8% (943/11784)^MUpdating files:   9% (1061/11784)^MUpdating files:  10% (1179/11784)^MUpdating files:  11% (1297/11784)^MUpdating files:  12% (1415/11784)^MUpdating files:  13% (1532/11784)^MUpdating files:  14% (1650/11784)^MUpdating files:  15% (1768/11784)^MUpdating files:  16% (1886/11784)^MUpdating files:  17% (2004/11784)^MUpdating files:  18% (2122/11784)^MUpdating files:  19% (2239/11784)^MUpdating files:
20% (2357/11784)^MUpdating files:  20% (2401/11784)^MUpdating files:  21% (2475/11784)^MUpdating files:  22% (2593/11784)^MUpdating files:  23% (2711/11784)^MUpdating files:  24% (2829/11784)^MUpdating files:  25% (2946/11784)^MUpdating files:  26% (3064/11784)^MUpdating files:  26% (3123/11784)^MUpdating files:  27% (3182/11784)^MUpdating files:  28% (3300/11784)^MUpdating files:  29% (3418/11784)^MUpdating files:  30% (3536/11784)^MUpdating files:  31% (3654/11784)^MUpdating files:  32% (3771/11784)^MUpdating files:  33% (3889/11784)^MUpdating files:  33% (3909/11784)^MUpdating files:  34% (4007/11784)^MUpdating files:  35% (4125/11784)^MUpdating files:  36% (4243/11784)^MUpdating files:  37% (4361/11784)^MUpdating files:  38% (4478/11784)^MUpdating files:  39% (4596/11784)^MUpdating files:  39% (4692/11784)^MUpdating files:  40% (4714/11784)^MUpdating files:  41% (4832/11784)^MUpdating files:  42% (4950/11784)^MUpdating files:  43% (5068/11784)^MUpdating files:  44% (5185/11784)^MUpdating files:  45% (5303/11784)^MUpdating files:  46% (5421/11784)^MUpdating files:  46% (5516/11784)^MUpdating files:  46% (5524/11784)^MUpdating files:  47% (5539/11784)^MUpdating files:  47% (5579/11784)^MUpdating files:  48% (5657/11784)^MUpdating files:  49% (5775/11784)^MUpdating files:  50% (5892/11784)^MUpdating files:  51% (6010/11784)^MUpdating files:  52% (6128/11784)^MUpdating files:  53% (6246/11784)^MUpdating files:  54% (6364/11784)^MUpdating files:  55% (6482/11784)^MUpdating files:  55% (6527/11784)^MUpdating files:  56% (6600/11784)^MUpdating files:  57% (6717/11784)^MUpdating files:  58% (6835/11784)^MUpdating files:  59% (6953/11784)^MUpdating files:  60% (7071/11784)^MUpdating files:  61% (7189/11784)^MUpdating files:  62% (7307/11784)^MUpdating files:  62% (7346/11784)^MUpdating files:  63% (7424/11784)^MUpdating files:  64% (7542/11784)^MUpdating files:
65% (7660/11784)^MUpdating files:  66% (7778/11784)^MUpdating files:  67% (7896/11784)^MUpdating files:  68% (8014/11784)^MUpdating files:  69% (8131/11784)^MUpdating files:  69% (8143/11784)^MUpdating files:  70% (8249/11784)^MUpdating files:  71% (8367/11784)^MUpdating files:  72% (8485/11784)^MUpdating files:  73% (8603/11784)^MUpdating files:  74% (8721/11784)^MUpdating files:  75% (8838/11784)^MUpdating files:  75% (8906/11784)^MUpdating files:  76% (8956/11784)^MUpdating files:  77% (9074/11784)^MUpdating files:  78% (9192/11784)^MUpdating files:  79% (9310/11784)^MUpdating files:  80% (9428/11784)^MUpdating files:  81% (9546/11784)^MUpdating files:  81% (9617/11784)^MUpdating files:  82% (9663/11784)^MUpdating files:  83% (9781/11784)^MUpdating files:  84% (9899/11784)^MUpdating files:  84% (9924/11784)^MUpdating files:  85% (10017/11784)^MUpdating files:  86% (10135/11784)^MUpdating files:  86% (10144/11784)^MUpdating files:  87% (10253/11784)^MUpdating files:  88% (10370/11784)^MUpdating files:  89% (10488/11784)^MUpdating files:  89% (10598/11784)^MUpdating files:  90% (10606/11784)^MUpdating files:  91% (10724/11784)^MUpdating files:  91% (10731/11784)^MUpdating files:  92% (10842/11784)^MUpdating files:  93% (10960/11784)^MUpdating files:  94% (11077/11784)^MUpdating files:  95% (11195/11784)^MUpdating files:  96% (11313/11784)^MUpdating files:  97% (11431/11784)^MUpdating files:  98% (11549/11784)^MUpdating files:  98% (11577/11784)^MUpdating files:  99% (11667/11784)^MUpdating files: 100% (11784/11784)^MUpdating files: 100% (11784/11784), done.
2023-05-17T12:36:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-05-17T12:36:54.726 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...

rather than an error that shows up with `git diff`.

should we move to cloning from github.com instead of git.ceph.com for this specific test ?
git.ceph.com network connectivity is unreliable as compared to github.com
all sorts of errors crop up in the test since the input side itself isn't stable
we'd end up debugging 'git' itself instead of 'ceph', and it would be such a waste

Milind Changire wrote:

should we move to cloning from github.com instead of git.ceph.com for this specific test ?
git.ceph.com network connectivity is unreliable as compared to github.com
all sorts of errors crop up in the test since the input side itself isn't stable
we'd end up debugging 'git' itself instead of 'ceph', and it would be such a waste

This issue isn't a repo clone issue or anything related to flakey endpoints, isn't it?

Failure with kclient: https://pulpito.ceph.com/vshankar-2023-06-15_04:58:28-fs-wip-vshankar-testing-20230614.124123-testing-default-smithi/7305154