Project

General

Profile

Bug #58934

Bug #54460: snaptest-multiple-capsnaps.sh test failure

Bug #59343: qa: fs/snaps/snaptest-multiple-capsnaps.sh failed

snaptest-git-ceph.sh failure with ceph-fuse

Added by Venky Shankar 12 months ago. Updated 8 months ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Category:
Correctness/Safety
Target version:
% Done:

0%

Source:
Tags:
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, ceph-fuse
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://pulpito.ceph.com/vshankar-2023-03-07_05:15:12-fs-wip-vshankar-testing-20230307.030510-testing-default-smithi/7195921/

Seems similar to https://tracker.ceph.com/issues/55332 which involved the kclient (async unlink/create bug), but this is with ceph-fuse.

2023-03-07T09:13:26.893 INFO:tasks.ceph.osd.7.smithi042.stderr:2023-03-07T09:13:26.891+0000 7f3b8e81e700 -1 received  signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph
-i 7  (PID: 71679) UID: 0
2023-03-07T09:13:26.917 INFO:tasks.workunit.client.0.smithi031.stdout:diff --git a/src/mds/CInode.h b/src/mds/CInode.h
2023-03-07T09:13:26.917 INFO:tasks.workunit.client.0.smithi031.stdout:deleted file mode 100644
2023-03-07T09:13:26.917 INFO:tasks.workunit.client.0.smithi031.stdout:index 8181be69c66..00000000000
2023-03-07T09:13:26.918 INFO:tasks.workunit.client.0.smithi031.stdout:--- a/src/mds/CInode.h
2023-03-07T09:13:26.918 INFO:tasks.workunit.client.0.smithi031.stdout:+++ /dev/null

`git diff` is seeing a deleted file (in one of the snapshots) which should have been present?

History

#2 Updated by Venky Shankar 11 months ago

  • Assignee set to Xiubo Li

#3 Updated by Xiubo Li 9 months ago

  • Status changed from New to Duplicate
  • Parent task set to #59343

This should be the same issue with https://tracker.ceph.com/issues/59343.

#4 Updated by Venky Shankar 9 months ago

Xiubo Li wrote:

This should be the same issue with https://tracker.ceph.com/issues/59343.

Which means this can happen with kclient runs.

I'm running into this here: https://pulpito.ceph.com/vshankar-2023-05-17_11:52:03-fs-wip-vshankar-testing-20230509.090020-6-testing-default-smithi/7276651/

But the failure is a bit different

2023-05-17T12:36:49.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-05-17T12:36:49.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-05-17T12:36:54.701 INFO:tasks.workunit.client.0.smithi092.stderr:Updating files:   7% (881/11784)^MUpdating files:   8% (943/11784)^MUpdating files:   9% (1061/11784)^MUpdating files:  10% (1179/11784)^MUpdating files:  11% (1297/11784)^MUpdating files:  12% (1415/11784)^MUpdating files:  13% (1532/11784)^MUpdating files:  14% (1650/11784)^MUpdating files:  15% (1768/11784)^MUpdating files:  16% (1886/11784)^MUpdating files:  17% (2004/11784)^MUpdating files:  18% (2122/11784)^MUpdating files:  19% (2239/11784)^MUpdating files:
20% (2357/11784)^MUpdating files:  20% (2401/11784)^MUpdating files:  21% (2475/11784)^MUpdating files:  22% (2593/11784)^MUpdating files:  23% (2711/11784)^MUpdating files:  24% (2829/11784)^MUpdating files:  25% (2946/11784)^MUpdating files:  26% (3064/11784)^MUpdating files:  26% (3123/11784)^MUpdating files:  27% (3182/11784)^MUpdating files:  28% (3300/11784)^MUpdating files:  29% (3418/11784)^MUpdating files:  30% (3536/11784)^MUpdating files:  31% (3654/11784)^MUpdating files:  32% (3771/11784)^MUpdating files:  33% (3889/11784)^MUpdating files:  33% (3909/11784)^MUpdating files:  34% (4007/11784)^MUpdating files:  35% (4125/11784)^MUpdating files:  36% (4243/11784)^MUpdating files:  37% (4361/11784)^MUpdating files:  38% (4478/11784)^MUpdating files:  39% (4596/11784)^MUpdating files:  39% (4692/11784)^MUpdating files:  40% (4714/11784)^MUpdating files:  41% (4832/11784)^MUpdating files:  42% (4950/11784)^MUpdating files:  43% (5068/11784)^MUpdating files:  44% (5185/11784)^MUpdating files:  45% (5303/11784)^MUpdating files:  46% (5421/11784)^MUpdating files:  46% (5516/11784)^MUpdating files:  46% (5524/11784)^MUpdating files:  47% (5539/11784)^MUpdating files:  47% (5579/11784)^MUpdating files:  48% (5657/11784)^MUpdating files:  49% (5775/11784)^MUpdating files:  50% (5892/11784)^MUpdating files:  51% (6010/11784)^MUpdating files:  52% (6128/11784)^MUpdating files:  53% (6246/11784)^MUpdating files:  54% (6364/11784)^MUpdating files:  55% (6482/11784)^MUpdating files:  55% (6527/11784)^MUpdating files:  56% (6600/11784)^MUpdating files:  57% (6717/11784)^MUpdating files:  58% (6835/11784)^MUpdating files:  59% (6953/11784)^MUpdating files:  60% (7071/11784)^MUpdating files:  61% (7189/11784)^MUpdating files:  62% (7307/11784)^MUpdating files:  62% (7346/11784)^MUpdating files:  63% (7424/11784)^MUpdating files:  64% (7542/11784)^MUpdating files:
65% (7660/11784)^MUpdating files:  66% (7778/11784)^MUpdating files:  67% (7896/11784)^MUpdating files:  68% (8014/11784)^MUpdating files:  69% (8131/11784)^MUpdating files:  69% (8143/11784)^MUpdating files:  70% (8249/11784)^MUpdating files:  71% (8367/11784)^MUpdating files:  72% (8485/11784)^MUpdating files:  73% (8603/11784)^MUpdating files:  74% (8721/11784)^MUpdating files:  75% (8838/11784)^MUpdating files:  75% (8906/11784)^MUpdating files:  76% (8956/11784)^MUpdating files:  77% (9074/11784)^MUpdating files:  78% (9192/11784)^MUpdating files:  79% (9310/11784)^MUpdating files:  80% (9428/11784)^MUpdating files:  81% (9546/11784)^MUpdating files:  81% (9617/11784)^MUpdating files:  82% (9663/11784)^MUpdating files:  83% (9781/11784)^MUpdating files:  84% (9899/11784)^MUpdating files:  84% (9924/11784)^MUpdating files:  85% (10017/11784)^MUpdating files:  86% (10135/11784)^MUpdating files:  86% (10144/11784)^MUpdating files:  87% (10253/11784)^MUpdating files:  88% (10370/11784)^MUpdating files:  89% (10488/11784)^MUpdating files:  89% (10598/11784)^MUpdating files:  90% (10606/11784)^MUpdating files:  91% (10724/11784)^MUpdating files:  91% (10731/11784)^MUpdating files:  92% (10842/11784)^MUpdating files:  93% (10960/11784)^MUpdating files:  94% (11077/11784)^MUpdating files:  95% (11195/11784)^MUpdating files:  96% (11313/11784)^MUpdating files:  97% (11431/11784)^MUpdating files:  98% (11549/11784)^MUpdating files:  98% (11577/11784)^MUpdating files:  99% (11667/11784)^MUpdating files: 100% (11784/11784)^MUpdating files: 100% (11784/11784), done.
2023-05-17T12:36:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 128
2023-05-17T12:36:54.726 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0...

rather than an error that shows up with `git diff`.

#5 Updated by Milind Changire 9 months ago

should we move to cloning from github.com instead of git.ceph.com for this specific test ?
git.ceph.com network connectivity is unreliable as compared to github.com
all sorts of errors crop up in the test since the input side itself isn't stable
we'd end up debugging 'git' itself instead of 'ceph', and it would be such a waste

#6 Updated by Venky Shankar 8 months ago

Milind Changire wrote:

should we move to cloning from github.com instead of git.ceph.com for this specific test ?
git.ceph.com network connectivity is unreliable as compared to github.com
all sorts of errors crop up in the test since the input side itself isn't stable
we'd end up debugging 'git' itself instead of 'ceph', and it would be such a waste

This issue isn't a repo clone issue or anything related to flakey endpoints, isn't it?

Also available in: Atom PDF