|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||
java.lang.Objectorg.apache.hadoop.io.SequenceFile
public class SequenceFile
SequenceFiles are flat files consisting of binary key/value
pairs.
SequenceFile provides Writer, Reader and
SequenceFile.Sorter classes for writing, reading and sorting respectively.
SequenceFile Writers based on the
SequenceFile.CompressionType used to compress key/value pairs:
Writer : Uncompressed records.
RecordCompressWriter : Record-compressed files, only compress
values.
BlockCompressWriter : Block-compressed files, both keys &
values are collected in 'blocks'
separately and compressed. The size of
the 'block' is configurable.
The actual compression algorithm used to compress key and/or values can be
specified by using the appropriate CompressionCodec.
The recommended way is to use the static createWriter methods
provided by the SequenceFile to chose the preferred format.
The Reader acts as the bridge and can read any of the above
SequenceFile formats.
Essentially there are 3 different formats for SequenceFiles
depending on the CompressionType specified. All of them share a
common header described below.
CompressionCodec class which is used for
compression of keys and/or values (if compression is
enabled).
SequenceFile.Metadata for this file.
100 bytes or so.
100 bytes or so.
100 bytes or so.
The compressed blocks of key lengths and value lengths consist of the actual lengths of individual keys/values encoded in ZeroCompressedInteger format.
CompressionCodec| Nested Class Summary | |
|---|---|
static class |
SequenceFile.CompressionType
The compression type used to compress key/value pairs in the SequenceFile. |
static class |
SequenceFile.Metadata
The class encapsulating with the metadata of a file. |
static class |
SequenceFile.Reader
Reads key/value pairs from a sequence-format file. |
static class |
SequenceFile.Sorter
Sorts key/value pairs in a sequence-format file. |
static interface |
SequenceFile.ValueBytes
The interface to 'raw' values of SequenceFiles. |
static class |
SequenceFile.Writer
Write key/value pairs to a sequence-format file. |
| Field Summary | |
|---|---|
static int |
SYNC_INTERVAL
The number of bytes between sync points. |
| Method Summary | |
|---|---|
static SequenceFile.Writer |
createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of 'raw' SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
Construct the preferred type of 'raw' SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.Writer |
createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
Construct the preferred type of SequenceFile Writer. |
static SequenceFile.CompressionType |
getCompressionType(Configuration job)
Deprecated. Use SequenceFileOutputFormat.getOutputCompressionType(org.apache.hadoop.mapred.JobConf)
to get SequenceFile.CompressionType for job-outputs. |
static void |
setCompressionType(Configuration job,
SequenceFile.CompressionType val)
Deprecated. Use the one of the many SequenceFile.createWriter methods to specify the SequenceFile.CompressionType while creating the SequenceFile or
SequenceFileOutputFormat.setOutputCompressionType(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.io.SequenceFile.CompressionType)
to specify the SequenceFile.CompressionType for job-outputs.
or |
| Methods inherited from class java.lang.Object |
|---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
| Field Detail |
|---|
public static final int SYNC_INTERVAL
| Method Detail |
|---|
@Deprecated public static SequenceFile.CompressionType getCompressionType(Configuration job)
SequenceFileOutputFormat.getOutputCompressionType(org.apache.hadoop.mapred.JobConf)
to get SequenceFile.CompressionType for job-outputs.
job - the job config to look in
@Deprecated
public static void setCompressionType(Configuration job,
SequenceFile.CompressionType val)
SequenceFile.CompressionType while creating the SequenceFile or
SequenceFileOutputFormat.setOutputCompressionType(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.io.SequenceFile.CompressionType)
to specify the SequenceFile.CompressionType for job-outputs.
or
job - the configuration to modifyval - the new compression type (none, block, record)
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
Progressable progress)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.progress - The Progressable object to track progress.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.codec - The compression codec.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.codec - The compression codec.progress - The Progressable object to track progress.metadata - The metadata of the file.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress,
SequenceFile.Metadata metadata)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.bufferSize - buffer size for the underlaying outputstream.replication - replication factor for the file.blockSize - block size for the file.compressionType - The compression type.codec - The compression codec.progress - The Progressable object to track progress.metadata - The metadata of the file.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
int bufferSize,
short replication,
long blockSize,
boolean createParent,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.bufferSize - buffer size for the underlaying outputstream.replication - replication factor for the file.blockSize - block size for the file.createParent - create parent directory if non-existentcompressionType - The compression type.codec - The compression codec.metadata - The metadata of the file.
IOException
public static SequenceFile.Writer createWriter(FileSystem fs,
Configuration conf,
Path name,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
Progressable progress)
throws IOException
fs - The configured filesystem.conf - The configuration.name - The name of the file.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.codec - The compression codec.progress - The Progressable object to track progress.
IOException
public static SequenceFile.Writer createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec,
SequenceFile.Metadata metadata)
throws IOException
conf - The configuration.out - The stream on top which the writer is to be constructed.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.codec - The compression codec.metadata - The metadata of the file.
IOException
public static SequenceFile.Writer createWriter(Configuration conf,
FSDataOutputStream out,
Class keyClass,
Class valClass,
SequenceFile.CompressionType compressionType,
CompressionCodec codec)
throws IOException
conf - The configuration.out - The stream on top which the writer is to be constructed.keyClass - The 'key' type.valClass - The 'value' type.compressionType - The compression type.codec - The compression codec.
IOException
|
||||||||||
| PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
| SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD | |||||||||