org.tranche.get
Class GetFileTool

java.lang.Object
  org.tranche.commons.Debuggable
      org.tranche.get.GetFileTool

public class GetFileTool
extends org.tranche.commons.Debuggable

A tool for downloading from a Tranche repository.

Author:

Jayson Falkner - jfalkner@umich.edu, James "Augie" Hill - augman85@gmail.com, Bryan E. Smith - bryanesmith@gmail.com

Field Summary

static boolean	DEFAULT_BATCH
static boolean	DEFAULT_CONTINUE_ON_FAILURE
static int	DEFAULT_DATA_QUEUE_SIZE
static boolean	DEFAULT_KEEP_LAST_MODIFIED
static java.lang.String	DEFAULT_REG_EX
static boolean	DEFAULT_SHOW_SUMMARY
static int	DEFAULT_THREADS
static java.lang.String	DEFAULT_UPLOAD_RELATIVE_PATH
static java.lang.Long	DEFAULT_UPLOAD_TIMESTAMP
static java.lang.String	DEFAULT_UPLOADER_NAME
static boolean	DEFAULT_USE_PERFORMANCE_LOG
static boolean	DEFAULT_USE_UNSPECIFIED_SERVERS
static boolean	DEFAULT_VALIDATE

Constructor Summary

GetFileTool()

Method Summary

boolean	addExternalServerURLsToUse(java.util.Collection<java.lang.String> serverURLs) Adds a collection of server urls from external networks to be used during download.
boolean	addExternalServerURLToUse(java.lang.String serverURL) Adds the url of a server from an external network to be used during download.
boolean	addListener(GetFileToolListener l) Adds a listener.
boolean	addServersToUse(java.util.Collection<java.lang.String> serverHostNames) Adds a collection of server host names to be used during download.
boolean	addServerToUse(java.lang.String serverHostName) Adds the host name of a server to be used during download.
void	clearPassphrase() Unsets the decryption passphrase.
void	clearRegEx() Unsets the regular expression.
protected byte[]	downloadData(BigHash fileHash, BigHash chunkHash) Downloads a data chunk.
long	getBytesDownloaded() Gets the number of bytes that have been downloaded.
long	getBytesToDownload() Gets the number of bytes that are being downloaded.
protected java.util.Collection<java.lang.String>	getConnections(BigHash hash) Gets a collection of server host names that should be used to download a chunk with the given hash.
GetFileToolReport	getDirectory() Download the specified directory.
java.util.Collection<java.lang.String>	getExternalServerURLsToUse() Gets the collection of server urls to be used from separate networks.
GetFileToolFailedChunksListener	getFailedChunksListener()
GetFileToolReport	getFile() Download the specified file.
BigHash	getHash() Gets the hash of the data to be downloaded.
java.util.Collection<GetFileToolListener>	getListeners()
MetaData	getMetaData() Gets the meta data, downloading if necessary.
java.lang.String	getPassphrase() Gets the passphrase to be used during decryption.
ProjectFile	getProjectFile() Gets the project file, downloading if necessary.
java.lang.String	getRegEx() Gets the regular expression that is used to download a portion of a data set.
java.io.File	getSaveFile() Gets the location to which the file should be saved.
java.util.Collection<java.lang.String>	getServersToUse() Gets the collection of server host names to be used.
int	getThreadCount() Gets the number of threads to be used.
TimeEstimator	getTimeEstimator() Gets the time to download object.
java.lang.String	getUploaderName() Gets the name of the specified uploading user.
java.lang.String	getUploadRelativePath() Gets the name of the relative path of the file in the data set.
java.lang.Long	getUploadTimestamp() Gets the timestamp of when the data set was uploaded.
boolean	isBatch() Whether chunks will be downloaded in batches.
boolean	isContinueOnFailure() Whether the download should continue even if some part of it has already failed.
boolean	isExecuting()
boolean	isKeepLastModified() If true, uses original last-modified timestamp for original files.
boolean	isLocked()
boolean	isPaused() Gets whether the download is paused.
boolean	isSendPerformanceInfo()
boolean	isStopped()
boolean	isUsingUnspecifiedServers() Whether servers other than the ones specified should be used.
boolean	isValidate() Sets whether the download will be validated.
static void	main(java.lang.String[] args) Command-line interface.
boolean	removeExternalServerURLsToUse(java.util.Collection<java.lang.String> serverURLs) Removes a collection of host names from the list of servers to be used.
boolean	removeExternalServerURLToUse(java.lang.String serverURL) Removes the server url from the list of servers from external networks to be used.
boolean	removeListener(GetFileToolListener l) Removes a listener.
boolean	removeServersToUse(java.util.Collection<java.lang.String> serverHostNames) Removes a collection of host names from the list of servers to be used.
boolean	removeServerToUse(java.lang.String serverHostName) Removes the server host name from the list of servers to be used.
void	setBatch(boolean batch) Sets whether to download chunks in batches.
void	setContinueOnFailure(boolean continueOnFailure) Sets whether the download should continue even if some part of it has already failed.
void	setHash(BigHash hash) Sets the hash of the data to be downloaded.
void	setKeepLastModified(boolean keepLastModified) If true, uses original last-modified timestamp for original files.
void	setPassphrase(java.lang.String passphrase) Sets the passphrase to be used during decryption.
void	setPause(boolean paused) Set whether the download is paused.
void	setRegEx(java.lang.String regEx) Sets the regular expression to be used to download a portion of a data set.
void	setSaveFile(java.io.File saveTo) Sets the location to which the file should be saved.
void	setSendPerformanceInfo(boolean sendPerformanceInfo)
void	setServersToUse(java.util.Collection<java.lang.String> serversHostUseSet)
void	setSuppressFailedChunkOutput(boolean isSuppress) By default, GetFileTool has a listener attached to print out information about failed chunks.
void	setThreadCount(int threadCount) Sets the number of threads to be used.
protected void	setUpConnections() Used internally to establish and lock connections with servers that will be used.
void	setUploaderName(java.lang.String uploaderName) Sets the name of the uploading user.
void	setUploadRelativePath(java.lang.String uploadRelativePath) Sets the name of the relative path of the file in the data set.
void	setUploadTimestamp(long uploadTimestamp) Sets the timestamp of when the data set was uploaded.
void	setUseUnspecifiedServers(boolean useUnspecifiedServers) Sets whether servers other than the ones specified should be used.
void	setValidate(boolean validate) Sets whether the download will be validated.
void	stop() Stops the download.
protected void	tearDownConnections() Used internally to unlock connections made during set-up.
void	throwExceptionIfLocked()
protected boolean	validateChunk(BigHash expectedHash, byte[] bytes) Validates a data chunk
protected void	validateDiskBacked(BigHash expectedHash, MetaData metaData, java.io.File file, byte[] padding) Validates a file.
protected void	validateInMemory(BigHash expectedHash, MetaData metaData, byte[] file, byte[] padding)

Methods inherited from class org.tranche.commons.Debuggable

debugErr, debugErr, debugOut, isDebug, setDebug

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_KEEP_LAST_MODIFIED

public static final boolean DEFAULT_KEEP_LAST_MODIFIED

See Also:

Constant Field Values

DEFAULT_BATCH

public static boolean DEFAULT_BATCH

DEFAULT_VALIDATE

public static boolean DEFAULT_VALIDATE

DEFAULT_REG_EX

public static java.lang.String DEFAULT_REG_EX

DEFAULT_USE_UNSPECIFIED_SERVERS

public static boolean DEFAULT_USE_UNSPECIFIED_SERVERS

DEFAULT_UPLOAD_TIMESTAMP

public static java.lang.Long DEFAULT_UPLOAD_TIMESTAMP

DEFAULT_UPLOADER_NAME

public static java.lang.String DEFAULT_UPLOADER_NAME

DEFAULT_UPLOAD_RELATIVE_PATH

public static java.lang.String DEFAULT_UPLOAD_RELATIVE_PATH

DEFAULT_THREADS

public static int DEFAULT_THREADS

DEFAULT_SHOW_SUMMARY

public static boolean DEFAULT_SHOW_SUMMARY

DEFAULT_CONTINUE_ON_FAILURE

public static boolean DEFAULT_CONTINUE_ON_FAILURE

DEFAULT_DATA_QUEUE_SIZE

public static int DEFAULT_DATA_QUEUE_SIZE

DEFAULT_USE_PERFORMANCE_LOG

public static boolean DEFAULT_USE_PERFORMANCE_LOG

Constructor Detail

GetFileTool

public GetFileTool()

Method Detail

isExecuting

public boolean isExecuting()

Returns:

isLocked

public boolean isLocked()

Returns:

throwExceptionIfLocked

public void throwExceptionIfLocked()

setBatch

public void setBatch(boolean batch)

Sets whether to download chunks in batches.

Parameters:

batch - Whether to download chunks in batches.

isBatch

public boolean isBatch()

Whether chunks will be downloaded in batches.

Returns:

Whether chunks will be downloaded in batches.

setValidate

public void setValidate(boolean validate)

Sets whether the download will be validated.

Validation consists of recreating hashes and signatures then comparing them to the downloaded data.

Parameters:

validate - Whether the download will be validated.

isValidate

public boolean isValidate()

Sets whether the download will be validated.

Validation consists of recreating hashes and signatures then comparing them to the downloaded data.

Returns:

Whether the download will be validated.

setContinueOnFailure

public void setContinueOnFailure(boolean continueOnFailure)

Sets whether the download should continue even if some part of it has already failed.

Parameters:

continueOnFailure - Whether the download should continue even if some part of it has already failed.

isContinueOnFailure

public boolean isContinueOnFailure()

Whether the download should continue even if some part of it has already failed.

Returns:

Whether the download should continue even if some part of it has already failed.

setHash

public void setHash(BigHash hash)

Sets the hash of the data to be downloaded.

Parameters:

hash - The hash of the data to be downloaded.

getHash

public BigHash getHash()

Gets the hash of the data to be downloaded.

Returns:

The hash of the data to be downloaded.

setSaveFile

public void setSaveFile(java.io.File saveTo)

Sets the location to which the file should be saved.

Parameters:

saveTo - The location to which the file should be saved.

getSaveFile

public java.io.File getSaveFile()

Gets the location to which the file should be saved.

Returns:

The location to which the file should be saved.

setUploaderName

public void setUploaderName(java.lang.String uploaderName)

Sets the name of the uploading user. Required to disambiguate between files that have been uploaded multiple times.

Parameters:

uploaderName - The name of the uploading user.

getUploaderName

public java.lang.String getUploaderName()

Gets the name of the specified uploading user.

Returns:

The name of the specified uploading user.

setUploadTimestamp

public void setUploadTimestamp(long uploadTimestamp)

Sets the timestamp of when the data set was uploaded. Required to disambiguate between files that have been uploaded multiple times.

Parameters:

uploadTimestamp - The timestamp of when the data set was uploaded.

getUploadTimestamp

public java.lang.Long getUploadTimestamp()

Gets the timestamp of when the data set was uploaded.

Returns:

The timestamp of when the data set was uploaded.

setUploadRelativePath

public void setUploadRelativePath(java.lang.String uploadRelativePath)

Sets the name of the relative path of the file in the data set. Required to disambiguate between files that have been uploaded multiple times in a data set.

Parameters:

uploadRelativePath - Name of the relative path of the file in the data set.

getUploadRelativePath

public java.lang.String getUploadRelativePath()

Gets the name of the relative path of the file in the data set.

Returns:

The name of the relative path of the file in the data set.

setPassphrase

public void setPassphrase(java.lang.String passphrase)

Sets the passphrase to be used during decryption.

Parameters:

passphrase - The passphrase to be used during decryption.

getPassphrase

public java.lang.String getPassphrase()

Gets the passphrase to be used during decryption.

Returns:

The passphrase to be used during decryption.

clearPassphrase

public void clearPassphrase()

Unsets the decryption passphrase.

setRegEx

public void setRegEx(java.lang.String regEx)

Sets the regular expression to be used to download a portion of a data set.

Parameters:

regEx - The regular expression.

getRegEx

public java.lang.String getRegEx()

Gets the regular expression that is used to download a portion of a data set.

Returns:

The regular expression.

clearRegEx

public void clearRegEx()

Unsets the regular expression.

addServerToUse

public boolean addServerToUse(java.lang.String serverHostName)

Adds the host name of a server to be used during download.

Parameters:

serverHostName - The host name of a server.

Returns:

Whether the host name was added to the list.

addServersToUse

public boolean addServersToUse(java.util.Collection<java.lang.String> serverHostNames)

Adds a collection of server host names to be used during download.

Parameters:

serverHostNames - A collection of host names to be used during download.

Returns:

Whether the host names were added to the list.

removeServerToUse

public boolean removeServerToUse(java.lang.String serverHostName)

Removes the server host name from the list of servers to be used.

Parameters:

serverHostName - The host name of a server.

Returns:

Whether the host name was removed from the list.

removeServersToUse

public boolean removeServersToUse(java.util.Collection<java.lang.String> serverHostNames)

Removes a collection of host names from the list of servers to be used.

Parameters:

serverHostNames - A collection of server host names.

Returns:

Whether the host names were removed from the list.

getServersToUse

public java.util.Collection<java.lang.String> getServersToUse()

Gets the collection of server host names to be used.

Returns:

A collection of server host names.

addExternalServerURLToUse

public boolean addExternalServerURLToUse(java.lang.String serverURL)

Adds the url of a server from an external network to be used during download.

Note that this collection does not effect the collection of server hosts to use with this network; it only permits the use of additional servers from other networks.

Parameters:

serverURL - The url of a server.

Returns:

Whether the url was added to the list.

addExternalServerURLsToUse

public boolean addExternalServerURLsToUse(java.util.Collection<java.lang.String> serverURLs)

Adds a collection of server urls from external networks to be used during download.

Note that this collection does not effect the collection of server hosts to use with this network; it only permits the use of additional servers from other networks.

Parameters:

serverURLs - A collection of urls to be used during download.

Returns:

Whether the urls were added to the list.

removeExternalServerURLToUse

public boolean removeExternalServerURLToUse(java.lang.String serverURL)

Removes the server url from the list of servers from external networks to be used.

Note that this collection does not effect the collection of server hosts to use with this network; it only permits the use of additional servers from other networks.

Parameters:

serverURL - The url of a server.

Returns:

Whether the url was removed from the list.

removeExternalServerURLsToUse

public boolean removeExternalServerURLsToUse(java.util.Collection<java.lang.String> serverURLs)

Removes a collection of host names from the list of servers to be used.

Note that this collection does not effect the collection of server hosts to use with this network; it only permits the use of additional servers from other networks.

Parameters:

serverURLs - A collection of server urls.

Returns:

Whether the urls were removed from the list.

getExternalServerURLsToUse

public java.util.Collection<java.lang.String> getExternalServerURLsToUse()

Gets the collection of server urls to be used from separate networks.

Note that this collection does not effect the collection of server hosts to use with this network; it only permits the use of additional servers from other networks.

Returns:

A collection of server urls.

setServersToUse

public void setServersToUse(java.util.Collection<java.lang.String> serversHostUseSet)

Parameters:

serversHostUseSet -

setUseUnspecifiedServers

public void setUseUnspecifiedServers(boolean useUnspecifiedServers)

Sets whether servers other than the ones specified should be used.

Parameters:

useUnspecifiedServers - Whether servers other than the ones specified should be used.

isUsingUnspecifiedServers

public boolean isUsingUnspecifiedServers()

Whether servers other than the ones specified should be used.

Returns:

Whether servers other than the ones specified should be used.

setSendPerformanceInfo

public void setSendPerformanceInfo(boolean sendPerformanceInfo)

Parameters:

sendPerformanceInfo -

isSendPerformanceInfo

public boolean isSendPerformanceInfo()

Returns:

setPause

public void setPause(boolean paused)

Set whether the download is paused.

Parameters:

paused - Whether the download is paused.

isPaused

public boolean isPaused()

Gets whether the download is paused.

Returns:

Whether the download is paused.

isStopped

public boolean isStopped()

Returns:

stop

public void stop()

Stops the download.

addListener

public boolean addListener(GetFileToolListener l)

Adds a listener.

Parameters:

l - A listener.

Returns:

Whether the listener was added.

removeListener

public boolean removeListener(GetFileToolListener l)

Removes a listener.

Parameters:

l - A listener.

Returns:

Whether the listener was removed.

getListeners

public java.util.Collection<GetFileToolListener> getListeners()

Returns:

getBytesToDownload

public long getBytesToDownload()

Gets the number of bytes that are being downloaded.

Returns:

The number of bytes that are being downloaded.

getBytesDownloaded

public long getBytesDownloaded()

Gets the number of bytes that have been downloaded.

Returns:

The number of bytes that have been downloaded.

getTimeEstimator

public TimeEstimator getTimeEstimator()

Gets the time to download object.

Returns:

The time to download object.

setThreadCount

public void setThreadCount(int threadCount)

Sets the number of threads to be used.

Parameters:

threadCount - The number of threads to be used.

getThreadCount

public int getThreadCount()

Gets the number of threads to be used.

Returns:

The number of threads to be used.

setUpConnections

protected void setUpConnections()
                         throws java.lang.Exception

Used internally to establish and lock connections with servers that will be used.

Throws:

java.lang.Exception

tearDownConnections

protected void tearDownConnections()

Used internally to unlock connections made during set-up.

getConnections

protected java.util.Collection<java.lang.String> getConnections(BigHash hash)

Gets a collection of server host names that should be used to download a chunk with the given hash.

If noted that the tool can use servers that are not specified, will include all matching connected servers.

Servers ordered: (1) connected servers ordered ascending by outstanding requests that are readable, writable, and have the hash in their hash span (2) the same as the previous, but not writable (3) other servers to use that may have sticky data.

Each of the portions of the list are internally randomized.

Parameters:

hash - A hash.

Returns:

An ordered collection of server host names that should be used to download a chunk with the given hash.

getMetaData

public MetaData getMetaData()
                     throws CouldNotFindMetaDataException,
                            java.lang.Exception

Gets the meta data, downloading if necessary.

Returns:

Throws:

java.lang.CouldNotFindMetaDataException

java.lang.Exception

CouldNotFindMetaDataException

getProjectFile

public ProjectFile getProjectFile()
                           throws java.lang.Exception

Gets the project file, downloading if necessary.

Returns:

The project file.

Throws:

java.lang.Exception

getFile

public GetFileToolReport getFile()

Download the specified file.

Returns:

A report describing what occurred during the download.

getDirectory

public GetFileToolReport getDirectory()

Download the specified directory.

Returns:

A report describing what occurred during the download.

downloadData

protected byte[] downloadData(BigHash fileHash,
                              BigHash chunkHash)

Downloads a data chunk.

Parameters:

fileHash - The hash of the file to which the data chunk belongs.

chunkHash - The data chunk hash.

Returns:

Data chunk bytes.

validateChunk

protected boolean validateChunk(BigHash expectedHash,
                                byte[] bytes)

Validates a data chunk

Parameters:

expectedHash - What hash is expected.

bytes - The bytes to validate.

Returns:

Whether the hash of the given bytes matches the expected hash.

validateInMemory

protected void validateInMemory(BigHash expectedHash,
                                MetaData metaData,
                                byte[] file,
                                byte[] padding)
                         throws java.lang.Exception

Parameters:

expectedHash -

metaData -

file -

padding -

Throws:

java.lang.Exception

validateDiskBacked

protected void validateDiskBacked(BigHash expectedHash,
                                  MetaData metaData,
                                  java.io.File file,
                                  byte[] padding)
                           throws java.lang.Exception

Validates a file.

Parameters:

expectedHash - What hash is expected.

metaData - The meta data of the file

file - The file to be validated.

padding - The bytes that were tacked on the end up the uploaded file.

Throws:

java.lang.Exception

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Command-line interface. Use -h or --help for usage information, or if using Java API, use GetFileTool.printUsage() to print to standard out.

Parameters:

args -

Throws:

java.lang.Exception

setSuppressFailedChunkOutput

public void setSuppressFailedChunkOutput(boolean isSuppress)

By default, GetFileTool has a listener attached to print out information about failed chunks.

To suppress, set to true. Can undo later by setting to false.

Parameters:

isSuppress - If true, removes the failed chunk listener. If true, attaches the failed chunk listener unless already attached.

getFailedChunksListener

public GetFileToolFailedChunksListener getFailedChunksListener()

isKeepLastModified

public boolean isKeepLastModified()

If true, uses original last-modified timestamp for original files. Otherwise, the last-modified will be assigned by the OS/filesystem.

Returns:

the keepLastModified

setKeepLastModified

public void setKeepLastModified(boolean keepLastModified)

If true, uses original last-modified timestamp for original files. Otherwise, the last-modified will be assigned by the OS/filesystem.

Parameters:

keepLastModified - the keepLastModified to set