Ceph » CephFS

Hadoop complains that directories within the file system that are expected to be present are not present. Hadoop may expect the file system to create these, or a particular Hadoop file system shim layer may need to integrate further.

================================

nwatkins@kyoto:~/Projects/hadoop-0.20.205.0$ cat logs/hadoop-nwatkins-jobtracker-kyoto.log
2011-10-28 14:58:04,506 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = kyoto/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.205.1
STARTUP_MSG: build = git://kyoto/ on branch master -r 7d645ed7364f2c38539f3c4ae3ddd3d40e33c16b; compiled by 'nwatkins' on Wed Oct 26 20:24:39 PDT 2011
****************************************************/
2011-10-28 14:58:04,605 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2011-10-28 14:58:04,614 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2011-10-28 14:58:04,614 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2011-10-28 14:58:04,614 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2011-10-28 14:58:04,690 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
2011-10-28 14:58:04,768 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2011-10-28 14:58:04,768 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2011-10-28 14:58:04,769 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-10-28 14:58:04,770 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2011-10-28 14:58:04,770 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2011-10-28 14:58:04,770 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2011-10-28 14:58:04,770 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2011-10-28 14:58:04,775 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as nwatkins
2011-10-28 14:58:04,789 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2011-10-28 14:58:04,791 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered.
2011-10-28 14:58:04,791 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered.
2011-10-28 14:58:09,842 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-10-28 14:58:09,887 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2011-10-28 14:58:09,904 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2011-10-28 14:58:09,905 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2011-10-28 14:58:09,906 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2011-10-28 14:58:09,906 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2011-10-28 14:58:09,906 INFO org.mortbay.log: jetty-6.1.26
2011-10-28 14:58:10,105 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
2011-10-28 14:58:10,109 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2011-10-28 14:58:10,110 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
2011-10-28 14:58:10,110 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2011-10-28 14:58:10,110 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2011-10-28 14:58:10,123 INFO org.apache.hadoop.fs.FileSystem: initialize:Ceph initialization arguments: CephFSInterface -m 127.0.0.1:6789 --client-readahead-max-periods=1
2011-10-28 14:58:10,310 INFO org.apache.hadoop.fs.FileSystem: initialize:Ceph initialized client. Setting cwd to /
2011-10-28 14:58:10,311 WARN org.apache.hadoop.fs.FileSystem: listStatus:enter with path ceph://null/tmp/hadoop-nwatkins/mapred/system
2011-10-28 14:58:10,314 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: ceph://null/tmp/hadoop-nwatkins/mapred/system
java.io.FileNotFoundException
at org.apache.hadoop.fs.ceph.CephFileSystem.listStatus(CephFileSystem.java:462)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2389)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)
2011-10-28 14:58:20,316 WARN org.apache.hadoop.fs.FileSystem: listStatus:enter with path ceph://null/tmp/hadoop-nwatkins/mapred/system
2011-10-28 14:58:20,318 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: ceph://null/tmp/hadoop-nwatkins/mapred/system
java.io.FileNotFoundException
at org.apache.hadoop.fs.ceph.CephFileSystem.listStatus(CephFileSystem.java:462)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2389)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2192)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2186)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:300)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:291)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4978)