public interface HDFSStore
Instances of this interface are created using HDFSStoreFactory.create(java.lang.String)
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_BATCH_INTERVAL_MILLIS |
static int |
DEFAULT_BATCH_SIZE_MB |
static float |
DEFAULT_BLOCK_CACHE_SIZE |
static boolean |
DEFAULT_BUFFER_PERSISTANCE |
static boolean |
DEFAULT_DISK_SYNCHRONOUS |
static int |
DEFAULT_DISPATCHER_THREADS |
static java.lang.String |
DEFAULT_HOME_DIR |
static int |
DEFAULT_INPUT_FILE_COUNT_MAX |
static int |
DEFAULT_INPUT_FILE_COUNT_MIN |
static int |
DEFAULT_INPUT_FILE_SIZE_MAX_MB |
static boolean |
DEFAULT_MAJOR_COMPACTION |
static int |
DEFAULT_MAJOR_COMPACTION_INTERVAL_MINS |
static int |
DEFAULT_MAJOR_COMPACTION_THREADS |
static int |
DEFAULT_MAX_BUFFER_MEMORY |
static boolean |
DEFAULT_MINOR_COMPACTION |
static int |
DEFAULT_MINOR_COMPACTION_THREADS |
static int |
DEFAULT_OLD_FILE_CLEANUP_INTERVAL_MINS |
static int |
DEFAULT_WRITE_ONLY_FILE_ROLLOVER_INTERVAL |
static int |
DEFAULT_WRITE_ONLY_FILE_SIZE_LIMIT |
static boolean |
DEFAULT_WRITEONLY_HDFSSTORE |
Modifier and Type | Method and Description |
---|---|
HDFSStore |
alter(HDFSStoreMutator mutator)
Identifies attributes configured in
HDFSStoreMutator and applies
the new attribute values to this instance of HDFSStore dynamically. |
HDFSStoreMutator |
createHdfsStoreMutator() |
void |
destroy()
Permanently deletes all HDFS files associated with this
HDFSStore . |
int |
getBatchInterval()
HDFSStore buffer data is persisted on HDFS in batches, and the
BatchInterval defines the number of milliseconds that can elapse between
writing batches to HDFS.
|
int |
getBatchSize()
HDFSStore buffer data is persisted on HDFS in batches.
|
float |
getBlockCacheSize() |
boolean |
getBufferPersistent()
Configure if HDFSStore in-memory buffer data, that has not been persisted
on HDFS yet, should be persisted to a local disk to prevent buffer data
loss.
|
java.lang.String |
getDiskStoreName()
The named DiskStore to use for any local disk persistence needs of
HDFSStore, for e.g.
|
int |
getDispatcherThreads()
The maximum number of threads (per region) used to write batches to HDFS.
|
java.lang.String |
getHDFSClientConfigFile()
The full path to the HDFS client configuration file, for e.g.
|
java.lang.String |
getHomeDir()
HomeDir is the HDFS directory path in which HDFSStore stores files.
|
int |
getInputFileCountMax()
The maximum number of files compacted by Minor compactor in a cycle.
|
int |
getInputFileCountMin()
A minimum number of files must exist in a bucket directory on HDFS before
minor compaction will start compaction.
|
int |
getInputFileSizeMax()
A file larger than this size, in megabytes, will not be compacted by minor
compactor.
|
boolean |
getMajorCompaction()
Major compaction removes old values of a key and deleted records from the
HDFS files, which can save space in HDFS and improve performance when
reading from HDFS.
|
int |
getMajorCompactionInterval()
The number of minutes after which HDFSStore performs the next major
compaction cycle.
|
int |
getMajorCompactionThreads()
The maximum number of threads that HDFSStore uses to perform major
compaction.
|
int |
getMaxMemory()
The maximum amount of memory in megabytes to be used by HDFSStore.
|
boolean |
getMinorCompaction()
Minor compaction reorganizes data in files to optimize read performance and
reduce number of files created on HDFS.
|
int |
getMinorCompactionThreads()
The maximum number of threads that HDFSStore uses to perform minor
compaction.
|
java.lang.String |
getName() |
java.lang.String |
getNameNodeURL()
HDFSStore persists data on a HDFS cluster identified by cluster's NameNode
URL or NameNode Service URL.
|
int |
getPurgeInterval()
HDFSStore may create new files as part of periodic maintenance activity.
|
boolean |
getSynchronousDiskWrite()
HDFS buffers can be persisted on local disk.
|
int |
getWriteOnlyFileRolloverInterval()
For HDFS write-only regions, this defines the number of seconds that can
elapse before HDFSStore closes an HDFS file and begins writing to a new
file.
|
int |
getWriteOnlyFileRolloverSize()
For HDFS write-only regions, this defines the maximum size (in megabytes)
that an HDFS log file can reach before HDFSStore closes the file and begins
writing to a new file.
|
static final java.lang.String DEFAULT_HOME_DIR
static final float DEFAULT_BLOCK_CACHE_SIZE
static final int DEFAULT_WRITE_ONLY_FILE_SIZE_LIMIT
static final int DEFAULT_WRITE_ONLY_FILE_ROLLOVER_INTERVAL
static final int DEFAULT_BATCH_SIZE_MB
static final int DEFAULT_BATCH_INTERVAL_MILLIS
static final boolean DEFAULT_WRITEONLY_HDFSSTORE
static final boolean DEFAULT_BUFFER_PERSISTANCE
static final boolean DEFAULT_DISK_SYNCHRONOUS
static final int DEFAULT_MAX_BUFFER_MEMORY
static final int DEFAULT_DISPATCHER_THREADS
static final boolean DEFAULT_MINOR_COMPACTION
static final int DEFAULT_MINOR_COMPACTION_THREADS
static final boolean DEFAULT_MAJOR_COMPACTION
static final int DEFAULT_MAJOR_COMPACTION_THREADS
static final int DEFAULT_INPUT_FILE_SIZE_MAX_MB
static final int DEFAULT_INPUT_FILE_COUNT_MAX
static final int DEFAULT_INPUT_FILE_COUNT_MIN
static final int DEFAULT_MAJOR_COMPACTION_INTERVAL_MINS
static final int DEFAULT_OLD_FILE_CLEANUP_INTERVAL_MINS
java.lang.String getName()
java.lang.String getNameNodeURL()
getHDFSClientConfigFile()
java.lang.String getHomeDir()
java.lang.String getHDFSClientConfigFile()
int getMaxMemory()
float getBlockCacheSize()
int getBatchSize()
int getBatchInterval()
int getDispatcherThreads()
boolean getBufferPersistent()
java.lang.String getDiskStoreName()
boolean getSynchronousDiskWrite()
int getWriteOnlyFileRolloverSize()
int getWriteOnlyFileRolloverInterval()
boolean getMinorCompaction()
int getMinorCompactionThreads()
boolean getMajorCompaction()
int getMajorCompactionInterval()
int getMajorCompactionThreads()
int getPurgeInterval()
void destroy()
HDFSStore
.
This operation will fail if any region is still using this store for
persistence.java.lang.IllegalStateException
- if any region using this hdfsStore still existsHDFSStoreMutator createHdfsStoreMutator()
HDFSStore alter(HDFSStoreMutator mutator)
HDFSStoreMutator
and applies
the new attribute values to this instance of HDFSStore
dynamically.
Any property which is not set in HDFSStoreMutator
remains
unaltered. In most cases altering the attributes does not cause existing
operations to terminate. The altered attributes are used in the next cycle
of the operation they impact.HDFSStore
int getInputFileSizeMax()
int getInputFileCountMin()
int getInputFileCountMax()