org.tranche.flatfile
Class DataBlockUtil

java.lang.Object
  org.tranche.flatfile.DataBlockUtil

public class DataBlockUtil
extends java.lang.Object

Stores config information in memory, and holds lists of available hashes and resources on disk.

Author:

Jayson Falkner - jfalkner@umich.edu, Bryan E. Smith - bryanesmith@gmail.com

Field Summary

DiskBackedBigHashSet	dataHashes The disk-backed list of data chunk hashes.
static boolean	DEFAULT_USE_CACHE The default value of whether the DataBlock's cache should be used.
DiskBackedBigHashSet	metaDataHashes The disk-backed list of meta data chunk hashes.
static int	ONE_MB The default file chunk.
static long	REQUIRED_TIME_BETWEEN_REPAIRS_ON_SAME_DATABLOCK The time, in milliseconds, that must ellapse since a repair to any given data block before will be repaired again.

Constructor Summary

DataBlockUtil()

DataBlockUtil(FlatFileTrancheServer ffts)

Method Summary

void	add(DataDirectoryConfiguration ddc) Add a DataDirectoryConfiguration object to the DataBlockUtil so that its directory can be used by the server.
void	add(DataDirectoryConfiguration ddc, java.lang.String reason) Add a DataDirectoryConfiguration object to the DataBlockUtil so that its directory can be used by the server.
void	addData(BigHash nameHashToAdd, byte[] content) Synchronized access to setting data chunk bytes.
void	addMetaData(BigHash nameHashToAdd, byte[] content) Synchronized access to setting meta data chunk bytes.
boolean	balanceDataDirectoryConfigurations() Balances data directory configuration usage.
void	close() Close off any resources.
void	deleteData(BigHash bh, java.lang.String desc) Delete a data chunk based on hash.
void	deleteMetaData(BigHash bh, java.lang.String desc) Delete a meta data chunk based on hash.
long	getCorruptedDataBlockBodyCount() Return the total count of DataBlock objects that were corrupted in the body.
long	getCorruptedDataBlockCount() Return total count of corrupted DataBlock objects.
long	getCorruptedDataBlockHeaderCount() Return the total count of DataBlock objects that were corrupted in the header.
java.io.File	getCorruptedDataBlocksLog() Get the data deletion log.
byte[]	getData(BigHash bh) Get data chunk bytes based on hash.
DataBlockCache	getDataBlockCache() Get the DataBlockCache object used by this utility.
protected DataBlock	getDataBlockToAddChunk(BigHash hash) Helper method to get the file/dir that matches the given hash.
protected DataBlock	getDataBlockToGetChunk(BigHash bh, boolean isMetaData) Finds or creates the appropriate data block for a particular hash.
java.io.File	getDataDeletionLog() Get the data deletion log.
java.util.Set	getDataDirectoryConfigurations() Returns a copy of all the DataDirectoryConfiguration objs in memory.
BigHash[]	getDataHashes(long offset, long length) Get the hashes associated with the data chunks on the server.
long	getDownloadedChunksFromCorruptedDataBlockCount() Return the total count of chunks that were downloaded from other servers to replace missing chunks in corrupted DataBlock objects.
long	getFailedMergeDataBlock() Return the count of failed merges of DataBlock objects.
static DataBlockUtilLogger	getLogger() Returns singleton log for instance.
long	getLostChunksFromCorruptedDataBlockCount() Return the total count of chunk that were not salvaged from a corrupted DataBlock that was corrupted in the body.
byte[]	getMetaData(BigHash bh) Get meta data chunk bytes based on hash.
java.io.File	getMetaDataDeletionLog() Get the meta data deletion log.
BigHash[]	getMetaDataHashes(long offset, long length) Get the hashes associated with the meta data chunks on the server.
int	getMinSizeAvailableInTargetDataBlockBeforeBalance() When moving, data block must have this much free space before a move is considered.
long	getSalvagedChunksFromCorruptedDataBlockCount() Return the total count of chunks salvaged from corrupted DataBlock objects.
long	getSuccessMergeDataBlock() Return the count of successfully merge DataBlock objects.
long	getTotalDataBlocksMovedWhileBalancing()
long	getTotalMergeDataBlock() Return the count of total merges of DataBlock objects.
boolean	hasData(BigHash bh) Check whether have data chunk bytes exist based on hash.
boolean	hasMetaData(BigHash bh) Check whether meta data chunk bytes exist based on hash.
protected void	incrementCorruptedDataBlockBodyCount() Increment the number of DataBlock objects that were corrupted in the body.
protected void	incrementCorruptedDataBlockCount() Increment the count of corrupted DataBlock objects.
protected void	incrementCorruptedDataBlockHeaderCount() Increment the number of DataBlock objects that were corrupted in the header.
protected void	incrementDownloadedChunksFromCorruptedDataBlockCount() Increment the number of chunks that were downloaded to replace missing chunks in a corrupted DataBlock.
protected void	incrementLostChunksFromCorruptedDataBlockCount() Increment the number of chunks that were lost in corrupted DataBlock objects.
protected void	incrementSalvagedChunksFromCorruptedDataBlockCount() Increment the number of salvaged chunks from a corrupted DataBlock.
boolean	isAllowedToRepairCorruptedDataBlocks() Returns true if allowed to repair corrupted DataBlocks.
static boolean	isLogging() Returns true if is logging in-memory using DataBlockUtilLog.
boolean	isUseCache() Returns true if using cache, false otherwise.
void	mergeOldDataBlock(java.io.File fileToMerge) Merge the data block, including splitting the data block.
void	mergeOldDataBlock(java.io.File fileToMerge, byte[] buf) Merge the data block, including splitting the data block.
boolean	remove(DataDirectoryConfiguration ddc) Remove a DataDirectoryConfiguration from the server's DataBlockUtil, along w/ associated hashes.
boolean	removeAll(java.util.Set<DataDirectoryConfiguration> configsToRemove) Removes set of DataDirectoryConfiguration objects from server's DataBlockUtil, along w/ associated hashes.
boolean	repairCorruptedDataBlock(java.io.File dataBlockFile, int maxTimeToWaitForQueueToFinish, java.lang.String description) If a data block is corrupted, will throw UnexpectedEndOfDataBlockException.
boolean	repairCorruptedDataBlock(java.io.File dataBlockFile, java.lang.String description) If a data block is corrupted, will throw UnexpectedEndOfDataBlockException.
boolean	repairCorruptedDataBlockForChunk(BigHash forChunkWithHash, int maxTimeToWaitForQueueToFinish, java.lang.String description) If a data block is corrupted, will throw UnexpectedEndOfDataBlockException.
static void	setIsLogging(boolean aIsLogging) Set whether to log in-memory using DataBlockUtilLog.
void	setMinSizeAvailableInTargetDataBlockBeforeBalance(int aMinSizeAvailableInTargetDataBlockBeforeBalance) When moving, data block must have this much free space before a move is considered.
void	setUseCache(boolean isUseCache) Set whether should use cache.
void	waitUntilFinished(long maxTimeToWaitMillis) Waits for any tasks that must be completed to ensure all DataBlock's on filesystem are in an expected state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_USE_CACHE

public static final boolean DEFAULT_USE_CACHE

The default value of whether the DataBlock's cache should be used.

See Also:

Constant Field Values

ONE_MB

public static final int ONE_MB

The default file chunk.

See Also:

Constant Field Values

dataHashes

public final DiskBackedBigHashSet dataHashes

The disk-backed list of data chunk hashes. Used when client requesting data chunks.

metaDataHashes

public final DiskBackedBigHashSet metaDataHashes

The disk-backed list of meta data chunk hashes. Used when client requesting meta data chunks.

REQUIRED_TIME_BETWEEN_REPAIRS_ON_SAME_DATABLOCK

public static final long REQUIRED_TIME_BETWEEN_REPAIRS_ON_SAME_DATABLOCK

The time, in milliseconds, that must ellapse since a repair to any given data block before will be repaired again.

See Also:

Constant Field Values

Constructor Detail

DataBlockUtil

public DataBlockUtil()

DataBlockUtil

public DataBlockUtil(FlatFileTrancheServer ffts)

Parameters:

ffts -

Method Detail

close

public void close()

Close off any resources.

waitUntilFinished

public void waitUntilFinished(long maxTimeToWaitMillis)
                       throws java.lang.InterruptedException

Waits for any tasks that must be completed to ensure all DataBlock's on filesystem are in an expected state.

This includes any task such as repaired broken DataBlock's, which can take a while.

Parameters:

maxTimeToWaitMillis - The maximum amount of time to wait before returning

Throws:

java.lang.InterruptedException

add

public final void add(DataDirectoryConfiguration ddc)

Add a DataDirectoryConfiguration object to the DataBlockUtil so that its directory can be used by the server.

Parameters:

ddc -

add

public final void add(DataDirectoryConfiguration ddc,
                      java.lang.String reason)

Add a DataDirectoryConfiguration object to the DataBlockUtil so that its directory can be used by the server.

Parameters:

ddc -

reason - A brief description of why was called

getDataDirectoryConfigurations

public final java.util.Set getDataDirectoryConfigurations()

Returns a copy of all the DataDirectoryConfiguration objs in memory. Modifying this collection does not affect the DataBlockUtil.

Returns:

remove

public final boolean remove(DataDirectoryConfiguration ddc)
                     throws java.lang.Exception

Remove a DataDirectoryConfiguration from the server's DataBlockUtil, along w/ associated hashes. Very expensive operation.

Parameters:

ddc -

Returns:

False if nothing done.

Throws:

java.lang.Exception

removeAll

public final boolean removeAll(java.util.Set<DataDirectoryConfiguration> configsToRemove)
                        throws java.lang.Exception

Removes set of DataDirectoryConfiguration objects from server's DataBlockUtil, along w/ associated hashes. Very expensive operation.

Parameters:

configsToRemove -

Returns:

Throws:

java.lang.Exception

getDataBlockToAddChunk

protected final DataBlock getDataBlockToAddChunk(BigHash hash)

Helper method to get the file/dir that matches the given hash.

Parameters:

hash -

Returns:

getDataBlockToGetChunk

protected final DataBlock getDataBlockToGetChunk(BigHash bh,
                                                 boolean isMetaData)
                                          throws java.lang.Exception

Finds or creates the appropriate data block for a particular hash.

Parameters:

bh -

isMetaData -

Returns:

Throws:

java.lang.Exception

addData

public final void addData(BigHash nameHashToAdd,
                          byte[] content)
                   throws java.lang.Exception

Synchronized access to setting data chunk bytes.

Parameters:

nameHashToAdd -

content -

Throws:

java.lang.Exception

addMetaData

public final void addMetaData(BigHash nameHashToAdd,
                              byte[] content)
                       throws java.lang.Exception

Synchronized access to setting meta data chunk bytes.

Parameters:

nameHashToAdd -

content -

Throws:

java.lang.Exception

getData

public final byte[] getData(BigHash bh)
                     throws java.lang.Exception

Get data chunk bytes based on hash.

Parameters:

bh -

Returns:

Throws:

java.lang.Exception

hasData

public final boolean hasData(BigHash bh)
                      throws java.lang.Exception

Check whether have data chunk bytes exist based on hash.

Parameters:

bh -

Returns:

Throws:

java.lang.Exception

hasMetaData

public final boolean hasMetaData(BigHash bh)
                          throws java.lang.Exception

Check whether meta data chunk bytes exist based on hash.

Parameters:

bh -

Returns:

Throws:

java.lang.Exception

getMetaData

public final byte[] getMetaData(BigHash bh)
                         throws java.lang.Exception

Get meta data chunk bytes based on hash.

Parameters:

bh -

Returns:

Throws:

java.lang.Exception

deleteData

public final void deleteData(BigHash bh,
                             java.lang.String desc)
                      throws java.lang.Exception

Delete a data chunk based on hash.

Parameters:

bh -

desc - A brief and arbitrary description of why deleting. This is logged. If not sure, just explain who the calling method is.

Throws:

java.lang.Exception

deleteMetaData

public final void deleteMetaData(BigHash bh,
                                 java.lang.String desc)
                          throws java.lang.Exception

Delete a meta data chunk based on hash.

Parameters:

bh -

desc - A brief and arbitrary description of why deleting. This is logged. If not sure, just explain who the calling method is.

Throws:

java.lang.Exception

getDataDeletionLog

public java.io.File getDataDeletionLog()

Get the data deletion log. Check for existance before reading; may not exist if not logging, or might exist even if not logging if logged in past.

Returns:

getCorruptedDataBlocksLog

public java.io.File getCorruptedDataBlocksLog()

Get the data deletion log. Check for existance before reading; may not exist if not logging, or might exist even if not logging if logged in past.

Returns:

getMetaDataDeletionLog

public java.io.File getMetaDataDeletionLog()

Get the meta data deletion log. Check for existance before reading; may not exist if not logging, or might exist even if not logging if logged in past.

Returns:

getDataHashes

public final BigHash[] getDataHashes(long offset,
                                     long length)
                              throws java.lang.Exception

Get the hashes associated with the data chunks on the server.

Parameters:

offset -

length -

Returns:

Throws:

java.lang.Exception

getMetaDataHashes

public final BigHash[] getMetaDataHashes(long offset,
                                         long length)
                                  throws java.lang.Exception

Get the hashes associated with the meta data chunks on the server.

Parameters:

offset -

length -

Returns:

Throws:

java.lang.Exception

mergeOldDataBlock

public final void mergeOldDataBlock(java.io.File fileToMerge)
                             throws java.io.FileNotFoundException,
                                    java.io.IOException,
                                    java.lang.Exception

Merge the data block, including splitting the data block.

Parameters:

fileToMerge -

Throws:

java.io.FileNotFoundException

java.io.IOException

java.lang.Exception

mergeOldDataBlock

public final void mergeOldDataBlock(java.io.File fileToMerge,
                                    byte[] buf)
                             throws java.io.FileNotFoundException,
                                    java.io.IOException,
                                    java.lang.Exception

Merge the data block, including splitting the data block.

Parameters:

fileToMerge -

buf -

Throws:

java.io.FileNotFoundException

java.io.IOException

java.lang.Exception

repairCorruptedDataBlockForChunk

public final boolean repairCorruptedDataBlockForChunk(BigHash forChunkWithHash,
                                                      int maxTimeToWaitForQueueToFinish,
                                                      java.lang.String description)
                                               throws java.lang.Exception

If a data block is corrupted, will throw UnexpectedEndOfDataBlockException. This method attempts to repair the block by doing the following:

1. Backs file up to temporary file, and deletes corrupted data block. Server will be able to write data blocks again!
2. Go on a salvaging mission. See which chunks (data or meta) are okay on the disk, and write them. Keep track of all data and meta data that is not in corrupted block.
3. After done salvaging, start off thread to download chunks that are now missing.

Parameters:

forChunkWithHash - Hash for chunk in the corrupted data block. (If catch an UnexpectedEndOfDataBlockException, the DataBlock is corrupted.)

maxTimeToWaitForQueueToFinish - Time in milliseconds to wait for missing chunks to download. If cannot wait, specify 0, which will submit request and return. This basically a way to wait to alleviate server resources.

description -

Returns:

True if repaired, false otherwise. Won't be repaired if repairs turned off or already submitted.

Throws:

java.lang.Exception

repairCorruptedDataBlock

public final boolean repairCorruptedDataBlock(java.io.File dataBlockFile,
                                              java.lang.String description)
                                       throws java.lang.Exception

If a data block is corrupted, will throw UnexpectedEndOfDataBlockException. This method attempts to repair the block by doing the following:

1. Backs file up to temporary file, and deletes corrupted data block. Server will be able to write data blocks again!
2. Go on a salvaging mission. See which chunks (data or meta) are okay on the disk, and write them. Keep track of all data and meta data that is not in corrupted block.
3. After done salvaging, start off thread to download chunks that are now missing.

Note that this method will wait up to five minutes for missing chunks to download before returning. If cannot wait, specify the maximum time to wait in the alternative signature method.

Parameters:

dataBlockFile - The path for the DataBlock's file. If not known, use a hash for a chunk in the DataBlock.

description -

Returns:

True if repaired, false otherwise. Won't be repaired if repairs turned off or already submitted.

Throws:

java.lang.Exception

repairCorruptedDataBlock

public final boolean repairCorruptedDataBlock(java.io.File dataBlockFile,
                                              int maxTimeToWaitForQueueToFinish,
                                              java.lang.String description)
                                       throws java.lang.Exception

If a data block is corrupted, will throw UnexpectedEndOfDataBlockException. This method attempts to repair the block by doing the following:

1. Backs file up to temporary file, and deletes corrupted data block. Server will be able to write data blocks again!
2. Go on a salvaging mission. See which chunks (data or meta) are okay on the disk, and write them. Keep track of all data and meta data that is not in corrupted block.
3. After done salvaging, start off thread to download chunks that are now missing.

Parameters:

dataBlockFile - The path for the DataBlock's file. If not known, use a hash for a chunk in the DataBlock.

maxTimeToWaitForQueueToFinish - Time in milliseconds to wait for missing chunks to download. If cannot wait, specify 0, which will submit request and return. This basically a way to wait to alleviate server resources.

description -

Returns:

True if repaired, false otherwise. Won't be repaired if repairs turned off or already submitted.

Throws:

java.lang.Exception

isLogging

public static final boolean isLogging()

Returns true if is logging in-memory using DataBlockUtilLog.

Returns:

setIsLogging

public static final void setIsLogging(boolean aIsLogging)

Set whether to log in-memory using DataBlockUtilLog.

Parameters:

aIsLogging -

getLogger

public static final DataBlockUtilLogger getLogger()

Returns singleton log for instance.

Returns:

getSuccessMergeDataBlock

public long getSuccessMergeDataBlock()

Return the count of successfully merge DataBlock objects.

Returns:

getFailedMergeDataBlock

public long getFailedMergeDataBlock()

Return the count of failed merges of DataBlock objects.

Returns:

getTotalMergeDataBlock

public long getTotalMergeDataBlock()

Return the count of total merges of DataBlock objects.

Returns:

getCorruptedDataBlockCount

public long getCorruptedDataBlockCount()

Return total count of corrupted DataBlock objects.

Returns:

getSalvagedChunksFromCorruptedDataBlockCount

public long getSalvagedChunksFromCorruptedDataBlockCount()

Return the total count of chunks salvaged from corrupted DataBlock objects.

Returns:

getDownloadedChunksFromCorruptedDataBlockCount

public long getDownloadedChunksFromCorruptedDataBlockCount()

Return the total count of chunks that were downloaded from other servers to replace missing chunks in corrupted DataBlock objects.

Returns:

getLostChunksFromCorruptedDataBlockCount

public long getLostChunksFromCorruptedDataBlockCount()

Return the total count of chunk that were not salvaged from a corrupted DataBlock that was corrupted in the body.

Returns:

getCorruptedDataBlockHeaderCount

public long getCorruptedDataBlockHeaderCount()

Return the total count of DataBlock objects that were corrupted in the header.

Returns:

getCorruptedDataBlockBodyCount

public long getCorruptedDataBlockBodyCount()

Return the total count of DataBlock objects that were corrupted in the body.

Returns:

incrementCorruptedDataBlockCount

protected void incrementCorruptedDataBlockCount()

Increment the count of corrupted DataBlock objects.

incrementSalvagedChunksFromCorruptedDataBlockCount

protected void incrementSalvagedChunksFromCorruptedDataBlockCount()

Increment the number of salvaged chunks from a corrupted DataBlock.

incrementDownloadedChunksFromCorruptedDataBlockCount

protected void incrementDownloadedChunksFromCorruptedDataBlockCount()

Increment the number of chunks that were downloaded to replace missing chunks in a corrupted DataBlock.

incrementLostChunksFromCorruptedDataBlockCount

protected void incrementLostChunksFromCorruptedDataBlockCount()

Increment the number of chunks that were lost in corrupted DataBlock objects.

incrementCorruptedDataBlockHeaderCount

protected void incrementCorruptedDataBlockHeaderCount()

Increment the number of DataBlock objects that were corrupted in the header.

incrementCorruptedDataBlockBodyCount

protected void incrementCorruptedDataBlockBodyCount()

Increment the number of DataBlock objects that were corrupted in the body.

isAllowedToRepairCorruptedDataBlocks

public boolean isAllowedToRepairCorruptedDataBlocks()

Returns true if allowed to repair corrupted DataBlocks. Check FFTS Configuration attribute.

Returns:

balanceDataDirectoryConfigurations

public boolean balanceDataDirectoryConfigurations()
                                           throws java.lang.Exception

Balances data directory configuration usage. Prevents access while running.

Returns:

True if any balancing took place, false otherwise.

Throws:

java.lang.Exception

getTotalDataBlocksMovedWhileBalancing

public long getTotalDataBlocksMovedWhileBalancing()

Returns:

The total number of data blocks moved while balancing. Starts at 0 for each instance of Tranche server.

getDataBlockCache

public DataBlockCache getDataBlockCache()

Get the DataBlockCache object used by this utility.

Returns:

isUseCache

public boolean isUseCache()

Returns true if using cache, false otherwise.

Returns:

setUseCache

public void setUseCache(boolean isUseCache)

Set whether should use cache.

Parameters:

isUseCache -

getMinSizeAvailableInTargetDataBlockBeforeBalance

public int getMinSizeAvailableInTargetDataBlockBeforeBalance()

When moving, data block must have this much free space before a move is considered.

Returns:

setMinSizeAvailableInTargetDataBlockBeforeBalance

public void setMinSizeAvailableInTargetDataBlockBeforeBalance(int aMinSizeAvailableInTargetDataBlockBeforeBalance)

When moving, data block must have this much free space before a move is considered.

Parameters:

aMinSizeAvailableInTargetDataBlockBeforeBalance -