Project

General

Profile

Actions
Copy link

Bug #1661

closed

Hadoop: expected system directories not present

Added by Noah Watkins over 12 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Noah Watkins
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Java/Hadoop
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hadoop complains that directories within the file system that are expected to be present are not present. Hadoop may expect the file system to create these, or a particular Hadoop file system shim layer may need to integrate further.

================================

nwatkins@kyoto:~/Projects/hadoop-0.20.205.0$ cat logs/hadoop-nwatkins-jobtracker-kyoto.log
2011-10-28 14:58:04,506 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = kyoto/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.205.1
STARTUP_MSG: build = git://kyoto/ on branch master -r 7d645ed7364f2c38539f3c4ae3ddd3d40e33c16b; compiled by 'nwatkins' on Wed Oct 26 20:24:39 PDT 2011
****************************************************/
2011-10-28 14:58:04,605 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2011-10-28 14:58:04,614 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2011-10-28 14:58:04,614 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2011-10-28 14:58:04,614 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2011-10-28 14:58:04,690 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
2011-10-28 14:58:04,768 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2011-10-28 14:58:04,768 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2011-10-28 14:58:04,769 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-10-28 14:58:04,770 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2011-10-28 14:58:04,770 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-10-28 14:58:04,770 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2011-10-28 14:58:04,770 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2011-10-28 14:58:04,775 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as nwatkins
2011-10-28 14:58:04,789 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2011-10-28 14:58:04,791 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered.
2011-10-28 14:58:04,791 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered.
2011-10-28 14:58:09,842 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-10-28 14:58:09,887 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-10-28 14:58:09,904 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-10-28 14:58:09,905 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2011-10-28 14:58:09,906 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2011-10-28 14:58:09,906 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2011-10-28 14:58:09,906 INFO org.mortbay.log: jetty-6.1.26
2011-10-28 14:58:10,105 INFO org.mortbay.log: Started :50030
2011-10-28 14:58:10,109 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2011-10-28 14:58:10,110 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
2011-10-28 14:58:10,110 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2011-10-28 14:58:10,110 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2011-10-28 14:58:10,123 INFO org.apache.hadoop.fs.FileSystem: initialize:Ceph initialization arguments: CephFSInterface -m 127.0.0.1:6789 --client-readahead-max-periods=1
2011-10-28 14:58:10,310 INFO org.apache.hadoop.fs.FileSystem: initialize:Ceph initialized client. Setting cwd to /
2011-10-28 14:58:10,311 WARN org.apache.hadoop.fs.FileSystem: listStatus:enter with path ceph://null/tmp/hadoop-nwatkins/mapred/system
2011-10-28 14:58:10,314 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: ceph://null/tmp/hadoop-nwatkins/mapred/system
java.io.FileNotFoundException
at org.apache.hadoop.fs.ceph.CephFileSystem.listStatus(CephFileSystem.java:462)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2389)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)
2011-10-28 14:58:20,316 WARN org.apache.hadoop.fs.FileSystem: listStatus:enter with path ceph://null/tmp/hadoop-nwatkins/mapred/system
2011-10-28 14:58:20,318 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: ceph://null/tmp/hadoop-nwatkins/mapred/system
java.io.FileNotFoundException
at org.apache.hadoop.fs.ceph.CephFileSystem.listStatus(CephFileSystem.java:462)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2389)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)

Actions
Copy link
 #1

Updated by Noah Watkins over 12 years ago

Adding: when this directory is created by hand before map reduce starts the error is gone.

Actions
Copy link
 #2

Updated by Greg Farnum over 12 years ago

  • Category set to 32

I remember running into this issue when developing things and deciding to just paper over it at the time -- I couldn't find a clear answer about whether this is something the FileSystem interface is expected to do, or if there's just some bootstrapping/setup scripts that get run for HDFS that we aren't implementing yet.

(On a sidenote, I just created a Hadoop category to file these sorts of bugs in.)

Actions
Copy link
 #3

Updated by Noah Watkins over 12 years ago

Good to know. I think at this point I need to paper over many things, but want to record all these issues. I'll just keep recording issues in the tracker to revisit later.

I'll look into this pretty soon, otherwise it will become a headache for multi-user setups pretty quickly.

Actions
Copy link
 #4

Updated by Sage Weil over 12 years ago

Sounds to me like CephFileSystem should just create the directory if it doesn't exist..

Actions
Copy link
 #5

Updated by Noah Watkins over 12 years ago

In this particular instance it is a map-reduce specific directory. I suspect that MapReduce is responsible for this, as poking through some of the other file systems in Hadoop i've never come across code that creates this directories.

Actions
Copy link
 #6

Updated by Greg Farnum over 12 years ago

Blindly creating directories is definitely not the proper solution. Somebody will need to take the time to figure out exactly what Hadoop's expectations for this are and implement it correctly.

Actions
Copy link
 #7

Updated by Greg Farnum over 12 years ago

  • Status changed from New to Resolved
  • Assignee set to Noah Watkins
  • Target version set to v0.39

Apparently this was actually the result of an API mismatch. Fixed by Noah's patch in commit:f9b7ecdb5bba1439dc4c13005a6213ddb6ecfcc5.

Actions
Copy link
 #8

Updated by John Spray over 6 years ago

  • Project changed from Ceph to CephFS
  • Category changed from 20 to 48
  • Target version deleted (v0.39)

Bulk move of hadoop category into FS project.

Actions
Copy link
 #9

Updated by Patrick Donnelly about 5 years ago

  • Category deleted (48)
  • Labels (FS) Java/Hadoop added
Actions
Copy link

Also available in: Atom PDF