|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectorg.apache.hadoop.filecache.TrackerDistributedCacheManager
public class TrackerDistributedCacheManager
Manages a single machine's instance of a cross-job cache. This class would typically be instantiated by a TaskTracker (or something that emulates it, like LocalJobRunner). This class is internal to Hadoop, and should not be treated as a public interface.
| Nested Class Summary | |
|---|---|
protected class |
TrackerDistributedCacheManager.BaseDirManager
This class holds properties of each base directories and is responsible for clean up unused cache files in base directories. |
protected class |
TrackerDistributedCacheManager.CleanupThread
A thread to check and cleanup the unused files periodically |
| Field Summary | |
|---|---|
protected TrackerDistributedCacheManager.BaseDirManager |
baseDirManager
|
protected TrackerDistributedCacheManager.CleanupThread |
cleanupThread
|
| Constructor Summary | |
|---|---|
TrackerDistributedCacheManager(Configuration conf,
TaskController controller)
|
|
| Method Summary | |
|---|---|
static void |
createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
This method create symlinks for all files in a given dir in another directory. |
static void |
determineTimestampsAndCacheVisibilities(Configuration job)
Determines timestamps of files to be cached, and stores those in the configuration. |
static long |
downloadCacheObject(Configuration conf,
URI source,
Path destination,
long desiredTimestamp,
boolean isArchive,
FsPermission permission)
Download a given path to the local file system. |
static boolean[] |
getArchiveVisibilities(Configuration conf)
Get the booleans on whether the archives are public or not. |
static void |
getDelegationTokens(Configuration job,
Credentials credentials)
For each archive or cache file - get the corresponding delegation token |
static boolean[] |
getFileVisibilities(Configuration conf)
Get the booleans on whether the files are public or not. |
protected TaskDistributedCacheManager |
getTaskDistributedCacheManager(JobID jobId)
|
TaskDistributedCacheManager |
newTaskDistributedCacheManager(JobID jobId,
Configuration taskConf)
|
void |
purgeCache()
Clear the entire contents of the cache and delete the backing files. |
void |
removeTaskDistributedCacheManager(JobID jobId)
|
void |
setArchiveSizes(JobID jobId,
long[] sizes)
Set the sizes for any archives, files, or directories in the private distributed cache. |
void |
startCleanupThread()
Start the background thread |
void |
stopCleanupThread()
Stop the background thread |
static void |
validate(Configuration conf)
This is part of the framework API. |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Field Detail |
|---|
protected TrackerDistributedCacheManager.BaseDirManager baseDirManager
protected TrackerDistributedCacheManager.CleanupThread cleanupThread
| Constructor Detail |
|---|
public TrackerDistributedCacheManager(Configuration conf,
TaskController controller)
throws IOException
IOException| Method Detail |
|---|
public static long downloadCacheObject(Configuration conf,
URI source,
Path destination,
long desiredTimestamp,
boolean isArchive,
FsPermission permission)
throws IOException
conf - the job's configurationsource - the source to copy fromdestination - where to copy the file. must be local fsdesiredTimestamp - the required modification timestamp of the sourceisArchive - is this an archive that should be expandedpermission - the desired permissions of the file.
IOException
public static void createAllSymlink(Configuration conf,
File jobCacheDir,
File workDir)
throws IOException
conf - the configurationjobCacheDir - the target directory for creating symlinksworkDir - the directory in which the symlinks are created
IOExceptionpublic void purgeCache()
public TaskDistributedCacheManager newTaskDistributedCacheManager(JobID jobId,
Configuration taskConf)
throws IOException
IOException
public void setArchiveSizes(JobID jobId,
long[] sizes)
throws IOException
IOExceptionpublic void removeTaskDistributedCacheManager(JobID jobId)
protected TaskDistributedCacheManager getTaskDistributedCacheManager(JobID jobId)
public static void determineTimestampsAndCacheVisibilities(Configuration job)
throws IOException
job -
IOExceptionpublic static boolean[] getFileVisibilities(Configuration conf)
conf - The configuration which stored the timestamps
IOExceptionpublic static boolean[] getArchiveVisibilities(Configuration conf)
conf - The configuration which stored the timestamps
public static void getDelegationTokens(Configuration job,
Credentials credentials)
throws IOException
job - credentials -
IOException
public static void validate(Configuration conf)
throws InvalidJobConfException
conf - a Configuration to be cheked for duplication
in cached URIs
InvalidJobConfExceptionpublic void startCleanupThread()
public void stopCleanupThread()
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||