Bug #22986
closedhadoop-s3a jobs failing with EPERM
0%
Description
the task modifies /etc/resolv.conf to set up dnsmasq, but something changed and the write to /etc/resolv.conf is now failing with EPERM:
Command failed on ovh053 with status 1: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv1, "wb"))\' /etc/resolv.conf'
Updated by Vasu Kulkarni about 6 years ago
Running on 7.3 to see why copying to /etc/resolv.conf is failing for 7.4
http://pulpito.ceph.com/vasu-2018-02-22_03:54:33-rgw-master-distro-basic-ovh/
Updated by Yehuda Sadeh about 6 years ago
- Status changed from New to In Progress
Updated by Vasu Kulkarni about 6 years ago
Testing with: https://github.com/ceph/ceph/pull/20678
Updated by Vasu Kulkarni about 6 years ago
Not sure why this started failing..
http://pulpito.ceph.com/vasu-2018-03-02_02:00:23-rgw-luminous-distro-basic-ovh/
Command works fine when run manually: 'sudo radosgw-admin user create --uid s3a --display-name=s3a cephtests --access-key=EGAQRD2ULOIFKFSKCT4F --secret-key=zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb --email=s3a@ceph.com'
I can only doubt the keys on that node could be not correct
and on another run, I hit cp issue which is very weird, since it just worked for couple of runs before:
http://pulpito.ceph.com/vasu-2018-03-02_02:01:27-rgw-luminous-distro-basic-ovh/
'sudo cp /home/ubuntu/cephtest/resolv.bak /etc/resolv.conf'
Still looking..
Updated by Vasu Kulkarni about 6 years ago
CentOS doesn't allow the overwrite of resolve.conf, I guess some network manager changes, I need to check with David Galloway on that, but for Ubuntu its fine.
Updated by Vasu Kulkarni about 6 years ago
- Assignee changed from Vasu Kulkarni to David Galloway
David,
Do you know why overwriting resolv.conf fails for centos but works for ubuntu?
http://pulpito.ceph.com/vasu-2018-03-02_02:01:27-rgw-luminous-distro-basic-ovh/
'sudo cp /home/ubuntu/cephtest/resolv.bak /etc/resolv.conf'
Updated by David Galloway about 6 years ago
- Assignee changed from David Galloway to Vasu Kulkarni
2018-03-02T03:03:30.596 INFO:teuthology.orchestra.run.ovh052.stderr:cp: cannot create regular file ‘/etc/resolv.conf’: Permission denied [dgalloway@ovh009 ~]$ lsattr /etc/resolv.conf ----i----------- /etc/resolv.conf [root@ovh009 ~]# cat /etc/resolv.conf # Immutable copy of resolv.conf for Sepia use only search front.sepia.ceph.com domain front.sepia.ceph.com nameserver 8.8.8.8
We made /etc/resolv.conf immutable on OVH nodes because of all the DNS failures their Openstack instances have when using their own nameservers. Making it was immutable is hacky but easiest instead of fighting with all the daemons that try to update it (cloud-init, NetworkManager, etc.)
You could have the test sudo chattr -i /etc/resolv.conf
before copying the resolv.tmp file into place.
Updated by Casey Bodley about 6 years ago
- Related to Bug #23454: rgw/s3atests failure: fs contract testBlockReadZeroByteFile/testSeekZeroByteFile/testOpenReadZeroByteFile added
Updated by Casey Bodley about 6 years ago
- Related to Bug #23531: s3a/2.8.0 fs.contract failure Seek/Rename/ComplexDirActions/RecursiveRootListing/EmptyRootDirNonRecursive added
Updated by Yuri Weinstein almost 6 years ago
Updated by Vasu Kulkarni almost 6 years ago
- Related to deleted (Bug #23531: s3a/2.8.0 fs.contract failure Seek/Rename/ComplexDirActions/RecursiveRootListing/EmptyRootDirNonRecursive)