Bug #63510
open
ceph fs (meta) data inconsistent with python shutil.copy()
Added by Xiubo Li 6 months ago.
Updated 6 months ago.
Description
This was reported by Frank via the ceph-users mail list: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SYKMCUJOFZVIQ2OZGTFN7KOI4CEXKRSC/
"The problem occurs in newer python versions when using the
shutil.copy function. There is also a function shutil.copy2 for
which the problem does not show up. Copy2 behaves a bit like "cp -
p" while copy is like "cp". The only code difference (linux)
between these 2 functions is that copy calls copyfile+copymode
while copy2 calls copyfile+copystat. For now we asked our users to
use copy2 to avoid the issue."
Files
- Description updated (diff)
- Status changed from New to In Progress
Locally I tried it with shutil.copy2 and shutil.copy, but couldn't reproduce it.
The shutil.copy2 script:
# Python program to explain shutil.copy2() method
# importing os module
import os
# importing shutil module
import shutil
# path
path = '/mnt/kcephfs/'
# List files and directories
# in '/home/User/Documents'
print("Before copying file:")
print(os.listdir(path))
# Source path
source = "/mnt/kcephfs/file.txt"
# Print the metadeta
# of source file
metadata = os.stat(source)
print("Metadata:", metadata, "\n")
# Destination path
destination = "/mnt/kcephfs/test/file_copy2.txt"
# Copy the content of
# source to destination
dest = shutil.copy2(source, destination)
# List files and directories
# in "/home / User / Documents"
print("After copying file:")
print(os.listdir(path))
# Print the metadata
# of the destination file
matadata = os.stat(destination)
print("Metadata:", metadata)
# Print path of newly
# created file
print("Destination path:", dest)
The shutil.copy script:
# Python program to explain shutil.copy2() method
# importing os module
import os
# importing shutil module
import shutil
# path
path = '/mnt/kcephfs/'
# List files and directories
# in '/home/User/Documents'
print("Before copying file:")
print(os.listdir(path))
# Source path
source = "/mnt/kcephfs/file.txt"
# Print the metadeta
# of source file
metadata = os.stat(source)
print("Metadata:", metadata, "\n")
# Destination path
destination = "/mnt/kcephfs/test/file_copy.txt"
# Copy the content of
# source to destination
dest = shutil.copy(source, destination)
# List files and directories
# in "/home / User / Documents"
print("After copying file:")
print(os.listdir(path))
# Print the metadata
# of the destination file
matadata = os.stat(destination)
print("Metadata:", metadata)
# Print path of newly
# created file
print("Destination path:", dest)
- Status changed from In Progress to Need More Info
Here is a test run showing the issue.
Script with name test-copy:
#!/bin/bash
# test script for https://tracker.ceph.com/issues/63510
# pass host name of second host as arg1
# both hosts must have the same fs hierarchy mounted with kclients
(( $#==1 )) || { echo "usage: test-copy HOST2_NAME" >&2; exit 1; }
rm myfile*.txt # clear out test files
echo test > myfile.txt
python --version
python -c "
import shutil;
shutil.copy('myfile.txt','myfile-copy.txt');
shutil.copy2('myfile.txt','myfile-copy2.txt');
"
echo "ls -l on localhost ($(hostname)):"
ls -l "${PWD}" | grep myfile
echo "ls -l on remote host ($1):"
ssh "$1" "ls -l '${PWD}' | grep myfile"
Annotated session output:
# Run with OS default python:
$ ./test-copy sn144
Python 2.7.5
ls -l on localhost (sn143.hpc.ait.dtu.dk):
-rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy2.txt
-rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy.txt
-rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile.txt
ls -l on remote host (sn144):
-rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy2.txt
-rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy.txt # <-- size=5 as expected
-rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile.txt
# Run with newer version:
$ module load Python/3.8.2-GCCcore-9.3.0
$ ./test-copy sn144
Python 3.8.2
ls -l on localhost (sn143.hpc.ait.dtu.dk):
-rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy2.txt
-rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy.txt
-rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile.txt
ls -l on remote host (sn144):
-rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy2.txt
-rw-rw-r-- 1 frans frans 0 Dec 1 13:45 myfile-copy.txt # <-- Upsi! size=0!!
-rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile.txt
I collected the straces of the python commands for both runs in #63510#note-5 and attached them here. They are significantly different.
Frank Schilder wrote:
Here is a test run showing the issue.
Script with name test-copy:
[...]
Annotated session output:
[...]
Hi Frank,
Locally I still couldn't reproduce it. I am using both latest ceph and kclient upstream code.
Please enable and upload the following debug logs:
1, mds logs (set 'debug_mds = 20' and 'debug_ms=1')
2, two kclient logs ("module ceph +p" | sudo tee /sys/kernel/debug/dynamic_debug/control)
Thanks
- Xiubo
Also available in: Atom
PDF