Bug #22986: hadoop-s3a jobs failing with EPERM - rgw - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
outstanding non-trivial backports
Pending backports
Product Backlog Scrub
rgw-easy-first-bug
rgw-multisite-backlog

Actions

Copy link

Bug #22986

closed

hadoop-s3a jobs failing with EPERM

Added by Casey Bodley about 6 years ago. Updated almost 6 years ago.

Status:

Resolved

Priority:

High

Assignee:

Vasu Kulkarni

Target version:

% Done:

Source:

Tags:

hadoop s3a

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

the task modifies /etc/resolv.conf to set up dnsmasq, but something changed and the write to /etc/resolv.conf is now failing with EPERM:

Command failed on ovh053 with status 1: 'sudo python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv¹, "wb"))\' /etc/resolv.conf'

http://qa-proxy.ceph.com/teuthology/yuriw-2018-02-10_16:42:19-rgw-luminous-distro-basic-smithi/2180605/teuthology.log

Related issues 1 (0 open — 1 closed)

Related to rgw - Bug #23454: rgw/s3atests failure: fs contract testBlockReadZeroByteFile/testSeekZeroByteFile/testOpenReadZeroByteFile

Won't Fix

Matt Benjamin

03/23/2018

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

Assignee set to Vasu Kulkarni

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

Priority changed from Normal to High

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

Running on 7.3 to see why copying to /etc/resolv.conf is failing for 7.4

http://pulpito.ceph.com/vasu-2018-02-22_03:54:33-rgw-master-distro-basic-ovh/

Actions

Copy link

Updated by Yehuda Sadeh about 6 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

Testing with: https://github.com/ceph/ceph/pull/20678

Actions

Copy link

Updated by Casey Bodley about 6 years ago

Status changed from In Progress to 7

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

Not sure why this started failing..

http://pulpito.ceph.com/vasu-2018-03-02_02:00:23-rgw-luminous-distro-basic-ovh/

Command works fine when run manually: 'sudo radosgw-admin user create --uid s3a --display-name=s3a cephtests --access-key=EGAQRD2ULOIFKFSKCT4F --secret-key=zi816w1vZKfaSM85Cl0BxXTwSLyN7zB4RbTswrGb --email=s3a@ceph.com'

I can only doubt the keys on that node could be not correct

and on another run, I hit cp issue which is very weird, since it just worked for couple of runs before:

http://pulpito.ceph.com/vasu-2018-03-02_02:01:27-rgw-luminous-distro-basic-ovh/
'sudo cp /home/ubuntu/cephtest/resolv.bak /etc/resolv.conf'

Still looking..

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

CentOS doesn't allow the overwrite of resolve.conf, I guess some network manager changes, I need to check with David Galloway on that, but for Ubuntu its fine.

Actions

Copy link

Updated by Vasu Kulkarni about 6 years ago

Assignee changed from Vasu Kulkarni to David Galloway

David,

Do you know why overwriting resolv.conf fails for centos but works for ubuntu?

http://pulpito.ceph.com/vasu-2018-03-02_02:01:27-rgw-luminous-distro-basic-ovh/
'sudo cp /home/ubuntu/cephtest/resolv.bak /etc/resolv.conf'

Actions

Copy link

#10

Updated by David Galloway about 6 years ago

Assignee changed from David Galloway to Vasu Kulkarni

2018-03-02T03:03:30.596 INFO:teuthology.orchestra.run.ovh052.stderr:cp: cannot create regular file â€˜/etc/resolv.confâ€™: Permission denied

[dgalloway@ovh009 ~]$ lsattr /etc/resolv.conf
----i----------- /etc/resolv.conf

[root@ovh009 ~]# cat /etc/resolv.conf 
# Immutable copy of resolv.conf for Sepia use only
search front.sepia.ceph.com
domain front.sepia.ceph.com
nameserver 8.8.8.8

We made /etc/resolv.conf immutable on OVH nodes because of all the DNS failures their Openstack instances have when using their own nameservers. Making it was immutable is hacky but easiest instead of fighting with all the daemons that try to update it (cloud-init, NetworkManager, etc.)

You could have the test sudo chattr -i /etc/resolv.conf before copying the resolv.tmp file into place.

Actions

Copy link

#11