Bug #63510
openceph fs (meta) data inconsistent with python shutil.copy()
0%
Description
This was reported by Frank via the ceph-users mail list: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SYKMCUJOFZVIQ2OZGTFN7KOI4CEXKRSC/
"The problem occurs in newer python versions when using the shutil.copy function. There is also a function shutil.copy2 for which the problem does not show up. Copy2 behaves a bit like "cp - p" while copy is like "cp". The only code difference (linux) between these 2 functions is that copy calls copyfile+copymode while copy2 calls copyfile+copystat. For now we asked our users to use copy2 to avoid the issue."
Files
Updated by Xiubo Li 6 months ago
Locally I tried it with shutil.copy2 and shutil.copy, but couldn't reproduce it.
The shutil.copy2 script:
# Python program to explain shutil.copy2() method # importing os module import os # importing shutil module import shutil # path path = '/mnt/kcephfs/' # List files and directories # in '/home/User/Documents' print("Before copying file:") print(os.listdir(path)) # Source path source = "/mnt/kcephfs/file.txt" # Print the metadeta # of source file metadata = os.stat(source) print("Metadata:", metadata, "\n") # Destination path destination = "/mnt/kcephfs/test/file_copy2.txt" # Copy the content of # source to destination dest = shutil.copy2(source, destination) # List files and directories # in "/home / User / Documents" print("After copying file:") print(os.listdir(path)) # Print the metadata # of the destination file matadata = os.stat(destination) print("Metadata:", metadata) # Print path of newly # created file print("Destination path:", dest)
The shutil.copy script:
# Python program to explain shutil.copy2() method # importing os module import os # importing shutil module import shutil # path path = '/mnt/kcephfs/' # List files and directories # in '/home/User/Documents' print("Before copying file:") print(os.listdir(path)) # Source path source = "/mnt/kcephfs/file.txt" # Print the metadeta # of source file metadata = os.stat(source) print("Metadata:", metadata, "\n") # Destination path destination = "/mnt/kcephfs/test/file_copy.txt" # Copy the content of # source to destination dest = shutil.copy(source, destination) # List files and directories # in "/home / User / Documents" print("After copying file:") print(os.listdir(path)) # Print the metadata # of the destination file matadata = os.stat(destination) print("Metadata:", metadata) # Print path of newly # created file print("Destination path:", dest)
Updated by Frank Schilder 5 months ago
Here is a test run showing the issue.
Script with name test-copy
:
#!/bin/bash # test script for https://tracker.ceph.com/issues/63510 # pass host name of second host as arg1 # both hosts must have the same fs hierarchy mounted with kclients (( $#==1 )) || { echo "usage: test-copy HOST2_NAME" >&2; exit 1; } rm myfile*.txt # clear out test files echo test > myfile.txt python --version python -c " import shutil; shutil.copy('myfile.txt','myfile-copy.txt'); shutil.copy2('myfile.txt','myfile-copy2.txt'); " echo "ls -l on localhost ($(hostname)):" ls -l "${PWD}" | grep myfile echo "ls -l on remote host ($1):" ssh "$1" "ls -l '${PWD}' | grep myfile"
Annotated session output:
# Run with OS default python: $ ./test-copy sn144 Python 2.7.5 ls -l on localhost (sn143.hpc.ait.dtu.dk): -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy2.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile.txt ls -l on remote host (sn144): -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy2.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy.txt # <-- size=5 as expected -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile.txt # Run with newer version: $ module load Python/3.8.2-GCCcore-9.3.0 $ ./test-copy sn144 Python 3.8.2 ls -l on localhost (sn143.hpc.ait.dtu.dk): -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy2.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile.txt ls -l on remote host (sn144): -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy2.txt -rw-rw-r-- 1 frans frans 0 Dec 1 13:45 myfile-copy.txt # <-- Upsi! size=0!! -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile.txt
Updated by Frank Schilder 5 months ago
- File strace.Python_3.8.2 strace.Python_3.8.2 added
- File strace.Python_2.7.5 strace.Python_2.7.5 added
I collected the straces of the python commands for both runs in #63510#note-5 and attached them here. They are significantly different.
Updated by Xiubo Li 5 months ago
Frank Schilder wrote:
Here is a test run showing the issue.
Script with name
test-copy
:[...]
Annotated session output:
[...]
Hi Frank,
Locally I still couldn't reproduce it. I am using both latest ceph and kclient upstream code.
Please enable and upload the following debug logs:
1, mds logs (set 'debug_mds = 20' and 'debug_ms=1')
2, two kclient logs ("module ceph +p" | sudo tee /sys/kernel/debug/dynamic_debug/control)
Thanks
- Xiubo