Project

General

Profile

Bug #20469

Ceph Client can't access file and show '???'

Added by Yunzhi Cheng over 6 years ago. Updated over 6 years ago.

Status:
Need More Info
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Very strange behavior


root@rndcl1:/data/rtt# ll
ls: cannot access V20170317: No such file or directory
total 4
drwxrwxr-x 1 rttbasedm  rttbasedm  2634001570518 May 25 02:43 ./
drwxr-xr-x 6 ceph       ceph                4096 Jun 22 15:11 ../
drwxr-xr-x 1 rttliveman rttliveman    3047913627 Jun 14 17:46 live/
drwxr-xr-x 1 rttbasedm  rttbasedm    38958647808 May 25 02:51 mk_stats/
drwxr-xr-x 1 rttliveman rttliveman    7985963992 May 24 02:13 sim/
drwxr-xr-x 1 rttbasedm  rttbasedm    33823479613 May 24 03:50 static/
drwxr-xr-x 1 rttbasedm  rttbasedm   597468824650 May 24 04:35 V20160218/
drwxr-xr-x 1 rttbasedm  rttbasedm     1592753749 May 25 02:26 V20160426/
drwxr-xr-x 1 rttbasedm  rttbasedm   560830587870 May 24 11:05 V20160926/
drwxr-xr-x 1 rttbasedm  rttbasedm   278192956875 May 24 22:26 V20170113/
drwxr-xr-x 1 rttbasedm  rttbasedm    23025444635 May 24 04:12 V20170116/
drwxr-xr-x 1 rttbasedm  rttbasedm   370160093533 May 24 18:27 V20170118/
d????????? ? ?          ?                      ?            ? V20170317/

System Info:


root@rndcl1:/data/rtt# sudo ceph --cluster=ceph-data -s
    cluster 8126fa27-e3b1-49e6-8a59-9ad337074533
     health HEALTH_OK
     monmap e1: 3 mons at {rndcl1=10.0.0.2:6789/0,rndcl11=10.0.0.12:6789/0,rndcl28=10.0.0.28:6789/0}
            election epoch 136, quorum 0,1,2 rndcl1,rndcl11,rndcl28
      fsmap e1969: 1/1/1 up {0=rndcl9=up:active}
     osdmap e4458: 28 osds: 28 up, 28 in
            flags sortbitwise,require_jewel_osds
      pgmap v2208569: 2112 pgs, 3 pools, 5761 GB data, 31706 kobjects
            17679 GB used, 86457 GB / 101 TB avail
                2108 active+clean
                   3 active+clean+scrubbing
                   1 active+clean+scrubbing+deep
  client io 97355 kB/s rd, 24099 kB/s wr, 72 op/s rd, 14 op/s wr

I use kernel client and have three ceph mount on the same client host


10.0.0.2,10.0.0.12,10.0.0.28:/hft on /data/hft type ceph (name=admin,noshare,key=client.admin)
10.0.0.2,10.0.0.12,10.0.0.28:/rtt on /data/rtt type ceph (name=admin,noshare,key=client.admin)
10.0.0.26,10.0.0.38,10.0.0.62:/rnd on /data/rnd2 type ceph (name=admin,noshare,key=client.admin)
10.0.0.27,10.0.0.39,10.0.0.63:/rnd on /data/rnd type ceph (name=admin,noshare,key=client.admin)

Ceph Version:


root@rndcl1:/data/rtt# ceph --version
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)

Kernel Version:


root@rndcl1:/data/rtt# uname -a
Linux rndcl1 4.4.0-46-generic #67~14.04.1-Ubuntu SMP Fri Oct 21 16:04:40 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

The directory can be access at other client and when I remount client, it can be resolved. It happened randomly.

debug_fs_after_recovery.txt View (33.5 KB) Yunzhi Cheng, 06/30/2017 07:19 AM

debug_fs_before_recovery.txt View (43.6 KB) Yunzhi Cheng, 06/30/2017 07:19 AM

debug_mds.log.gz (285 KB) Yunzhi Cheng, 06/30/2017 07:19 AM

History

#1 Updated by Zheng Yan over 6 years ago

this one and http://tracker.ceph.com/issues/20467 may be caused by the same bug. please try the newest upstream kernel

#2 Updated by Yunzhi Cheng over 6 years ago

Zheng Yan wrote:

this one and http://tracker.ceph.com/issues/20467 may be caused by the same bug. please try the newest upstream kernel

OK, I will try it. By the way, does fuse-client has this bug ?

#3 Updated by Zheng Yan over 6 years ago

Yunzhi Cheng wrote:

Zheng Yan wrote:

this one and http://tracker.ceph.com/issues/20467 may be caused by the same bug. please try the newest upstream kernel

OK, I will try it. By the way, does fuse-client has this bug ?

no, I don't think so.

#4 Updated by Patrick Donnelly over 6 years ago

  • Status changed from New to Need More Info

Also available in: Atom PDF