Bug #1663: Hadoop: file ownership/permission not available in hadoop - CephFS - Ceph

Actions

Copy link

Bug #1663

closed

Hadoop: file ownership/permission not available in hadoop

Added by Noah Watkins over 12 years ago. Updated about 5 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Labels (FS):

Java/Hadoop

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Hadoop complains about incorrect file ownership. An 'ls' via Hadoop FS interface reveals no permission information, but may not be transferred through the listStatus call.

=======================

nwatkins@kyoto:~/Projects/hadoop-0.20.205.0$ cat logs/hadoop-nwatkins-jobtracker-kyoto.log
2011-10-28 15:01:40,392 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = kyoto/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.205.1
STARTUP_MSG: build = git://kyoto/ on branch master -r 7d645ed7364f2c38539f3c4ae3ddd3d40e33c16b; compiled by 'nwatkins' on Wed Oct 26 20:24:39 PDT 2011
****************************************************/
2011-10-28 15:01:40,490 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2011-10-28 15:01:40,499 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2011-10-28 15:01:40,500 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2011-10-28 15:01:40,500 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2011-10-28 15:01:40,575 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
2011-10-28 15:01:40,650 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2011-10-28 15:01:40,650 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2011-10-28 15:01:40,651 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-10-28 15:01:40,652 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2011-10-28 15:01:40,652 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-10-28 15:01:40,652 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2011-10-28 15:01:40,652 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2011-10-28 15:01:40,657 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as nwatkins
2011-10-28 15:01:40,671 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2011-10-28 15:01:40,672 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered.
2011-10-28 15:01:40,673 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered.
2011-10-28 15:01:45,728 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-10-28 15:01:45,781 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-10-28 15:01:45,799 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-10-28 15:01:45,800 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2011-10-28 15:01:45,801 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2011-10-28 15:01:45,801 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2011-10-28 15:01:45,801 INFO org.mortbay.log: jetty-6.1.26
2011-10-28 15:01:46,012 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
2011-10-28 15:01:46,016 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2011-10-28 15:01:46,016 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
2011-10-28 15:01:46,017 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2011-10-28 15:01:46,017 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2011-10-28 15:01:46,030 INFO org.apache.hadoop.fs.FileSystem: initialize:Ceph initialization arguments: CephFSInterface -m 127.0.0.1:6789 --client-readahead-max-periods=1
2011-10-28 15:01:46,126 INFO org.apache.hadoop.fs.FileSystem: initialize:Ceph initialized client. Setting cwd to /
2011-10-28 15:01:46,134 WARN org.apache.hadoop.mapred.JobTracker: Failed to operate on mapred.system.dir (ceph://null/tmp/hadoop-nwatkins/mapred/system) because of permissions.
2011-10-28 15:01:46,134 WARN org.apache.hadoop.mapred.JobTracker: Manually delete the mapred.system.dir (ceph://null/tmp/hadoop-nwatkins/mapred/system) and then start the JobTracker.
2011-10-28 15:01:46,135 WARN org.apache.hadoop.mapred.JobTracker: Bailing out ...
org.apache.hadoop.security.AccessControlException: The systemdir ceph://null/tmp/hadoop-nwatkins/mapred/system is not owned by nwatkins
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2379)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)
2011-10-28 15:01:46,135 FATAL org.apache.hadoop.mapred.JobTracker: org.apache.hadoop.security.AccessControlException: The systemdir ceph://null/tmp/hadoop-nwatkins/mapred/system is not owned by nwatkins
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2379)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

2011-10-28 15:01:46,137 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at kyoto/127.0.1.1
****************************************************/

Actions

Copy link

Updated by Noah Watkins over 12 years ago

This is a very simple hack that will make hadoop ignore the permission for the time being:

diff --git a/src/mapred/org/apache/hadoop/mapred/JobTracker.java b/src/mapred/org/apache/hadoop/mapred/JobTracker.java
index e657da1..a4e416e 100644
--- a/src/mapred/org/apache/hadoop/mapred/JobTracker.java
++ b/src/mapred/org/apache/hadoop/mapred/JobTracker.java
@ -2376,13 +2376,13 @ public class JobTracker implements MRConstants, InterTrackerProtocol,
FileStatus systemDirStatus = fs.getFileStatus(systemDir);
if (!systemDirStatus.getOwner().equals(
getMROwner().getShortUserName())) {
- throw new AccessControlException("The systemdir " + systemDir +
- " is not owned by " + getMROwner().getShortUserName());
// throw new AccessControlException("The systemdir " + systemDir
// " is not owned by " + getMROwner().getShortUserName());
}
if (!systemDirStatus.getPermission().equals(SYSTEM_DIR_PERMISSION)) {
- LOG.warn("Incorrect permissions on " + systemDir
- ". Setting it to " + SYSTEM_DIR_PERMISSION);
- fs.setPermission(systemDir,new FsPermission(SYSTEM_DIR_PERMISSION));
// LOG.warn("Incorrect permissions on " + systemDir
// ". Setting it to " + SYSTEM_DIR_PERMISSION);
+// fs.setPermission(systemDir,new FsPermission(SYSTEM_DIR_PERMISSION));
}
} catch (FileNotFoundException fnf) {} //ignore
// Make sure that the backup data is preserved

Actions

Copy link

Updated by Noah Watkins over 12 years ago

Noah Watkins wrote:

This is a very simple hack that will make hadoop ignore the permission for the time being:

diff --git a/src/mapred/org/apache/hadoop/mapred/JobTracker.java b/src/mapred/org/apache/hadoop/mapred/JobTracker.java
index e657da1..a4e416e 100644
--- a/src/mapred/org/apache/hadoop/mapred/JobTracker.java
+++ b/src/mapred/org/apache/hadoop/mapred/JobTracker.java
@@ -2376,13 +2376,13 @@ public class JobTracker implements MRConstants, InterTrackerProtocol,
           FileStatus systemDirStatus = fs.getFileStatus(systemDir);
           if (!systemDirStatus.getOwner().equals(
               getMROwner().getShortUserName())) {
-            throw new AccessControlException("The systemdir " + systemDir +
-                " is not owned by " + getMROwner().getShortUserName());
+//            throw new AccessControlException("The systemdir " + systemDir +
+//                " is not owned by " + getMROwner().getShortUserName());
           }
           if (!systemDirStatus.getPermission().equals(SYSTEM_DIR_PERMISSION)) {
-            LOG.warn("Incorrect permissions on " + systemDir +
-                ". Setting it to " + SYSTEM_DIR_PERMISSION);
-            fs.setPermission(systemDir,new FsPermission(SYSTEM_DIR_PERMISSION));
+//            LOG.warn("Incorrect permissions on " + systemDir +
+//                ". Setting it to " + SYSTEM_DIR_PERMISSION);
+//            fs.setPermission(systemDir,new FsPermission(SYSTEM_DIR_PERMISSION));
           }
         } catch (FileNotFoundException fnf) {} //ignore
         // Make sure that the backup data is preserved

Actions

Copy link

Updated by Noah Watkins over 12 years ago

a79b7e17ebbc70cedae80216986ae5fd52a1c0b7 provides an OK fix for now. Basically it makes any file look like the current user owns it, sort of how S3FileSystem does things. More importantly it avoids any patches to core hadoop for now.

Actions

Copy link