Actions
Bug #63510
openceph fs (meta) data inconsistent with python shutil.copy()
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This was reported by Frank via the ceph-users mail list: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SYKMCUJOFZVIQ2OZGTFN7KOI4CEXKRSC/
"The problem occurs in newer python versions when using the shutil.copy function. There is also a function shutil.copy2 for which the problem does not show up. Copy2 behaves a bit like "cp - p" while copy is like "cp". The only code difference (linux) between these 2 functions is that copy calls copyfile+copymode while copy2 calls copyfile+copystat. For now we asked our users to use copy2 to avoid the issue."
Files
Updated by Xiubo Li 5 months ago
Locally I tried it with shutil.copy2 and shutil.copy, but couldn't reproduce it.
The shutil.copy2 script:
# Python program to explain shutil.copy2() method
# importing os module
import os
# importing shutil module
import shutil
# path
path = '/mnt/kcephfs/'
# List files and directories
# in '/home/User/Documents'
print("Before copying file:")
print(os.listdir(path))
# Source path
source = "/mnt/kcephfs/file.txt"
# Print the metadeta
# of source file
metadata = os.stat(source)
print("Metadata:", metadata, "\n")
# Destination path
destination = "/mnt/kcephfs/test/file_copy2.txt"
# Copy the content of
# source to destination
dest = shutil.copy2(source, destination)
# List files and directories
# in "/home / User / Documents"
print("After copying file:")
print(os.listdir(path))
# Print the metadata
# of the destination file
matadata = os.stat(destination)
print("Metadata:", metadata)
# Print path of newly
# created file
print("Destination path:", dest)
The shutil.copy script:
# Python program to explain shutil.copy2() method
# importing os module
import os
# importing shutil module
import shutil
# path
path = '/mnt/kcephfs/'
# List files and directories
# in '/home/User/Documents'
print("Before copying file:")
print(os.listdir(path))
# Source path
source = "/mnt/kcephfs/file.txt"
# Print the metadeta
# of source file
metadata = os.stat(source)
print("Metadata:", metadata, "\n")
# Destination path
destination = "/mnt/kcephfs/test/file_copy.txt"
# Copy the content of
# source to destination
dest = shutil.copy(source, destination)
# List files and directories
# in "/home / User / Documents"
print("After copying file:")
print(os.listdir(path))
# Print the metadata
# of the destination file
matadata = os.stat(destination)
print("Metadata:", metadata)
# Print path of newly
# created file
print("Destination path:", dest)
Updated by Frank Schilder 5 months ago
Here is a test run showing the issue.
Script with name test-copy:
#!/bin/bash
# test script for https://tracker.ceph.com/issues/63510
# pass host name of second host as arg1
# both hosts must have the same fs hierarchy mounted with kclients
(( $#==1 )) || { echo "usage: test-copy HOST2_NAME" >&2; exit 1; }
rm myfile*.txt # clear out test files
echo test > myfile.txt
python --version
python -c "
import shutil;
shutil.copy('myfile.txt','myfile-copy.txt');
shutil.copy2('myfile.txt','myfile-copy2.txt');
"
echo "ls -l on localhost ($(hostname)):"
ls -l "${PWD}" | grep myfile
echo "ls -l on remote host ($1):"
ssh "$1" "ls -l '${PWD}' | grep myfile"
Annotated session output:
# Run with OS default python: $ ./test-copy sn144 Python 2.7.5 ls -l on localhost (sn143.hpc.ait.dtu.dk): -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy2.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile.txt ls -l on remote host (sn144): -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy2.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile-copy.txt # <-- size=5 as expected -rw-rw-r-- 1 frans frans 5 Dec 1 13:43 myfile.txt # Run with newer version: $ module load Python/3.8.2-GCCcore-9.3.0 $ ./test-copy sn144 Python 3.8.2 ls -l on localhost (sn143.hpc.ait.dtu.dk): -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy2.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy.txt -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile.txt ls -l on remote host (sn144): -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile-copy2.txt -rw-rw-r-- 1 frans frans 0 Dec 1 13:45 myfile-copy.txt # <-- Upsi! size=0!! -rw-rw-r-- 1 frans frans 5 Dec 1 13:45 myfile.txt
Actions
#6
Updated by Frank Schilder 5 months ago
- File strace.Python_3.8.2 strace.Python_3.8.2 added
- File strace.Python_2.7.5 strace.Python_2.7.5 added
I collected the straces of the python commands for both runs in #63510#note-5 and attached them here. They are significantly different.
Updated by Xiubo Li 5 months ago
Frank Schilder wrote:
Here is a test run showing the issue.
Script with name
test-copy:[...]
Annotated session output:
[...]
Hi Frank,
Locally I still couldn't reproduce it. I am using both latest ceph and kclient upstream code.
Please enable and upload the following debug logs:
1, mds logs (set 'debug_mds = 20' and 'debug_ms=1')
2, two kclient logs ("module ceph +p" | sudo tee /sys/kernel/debug/dynamic_debug/control)
Thanks
- Xiubo
Actions