The Archiving process

It is important to remember that UT-STorR is an archive, not a backup system or an HSM (Heirarchical Storage Manager). Please read the Access and Login page first.

UT-StorR replication policies

UT-StorR offers two different replication policies, “onecopy” and “twocopy”. These refer to the number of individual tapes that the data will reside on once migrated. Onecopy is the default selection and should be fine for most use cases. Twocopy is for extra redundancy in case of mechanical tape malfunctions. Good use cases for twocopy is research data that would have substantial costs or be impossible to reacquire.

Curating data for archiving

Preparing your data

UT-StorR is designed for the storage, retrieval, and sharing of primary data. Examples of this would be raw genomic sequences, raw microscope images, survey responses, labeled image sets, etc. Many project groups put shared resources like Python virtual environments, R packages, etc. into their shared directories to process their data. Those shared packages do not need to be archived. Instead, create a a list of required packages or instructions on how to replicate the processing environment to be stored in the archive directory.

Data structure

Tape archives are designed to work with fewer, larger files. Recommended sizes for files on UT-StorR are in the 20-200GB range per file, with an allowable range of 1GB to 1TB. Figure 1 shows the origin of this recommendation, as very large files (>2TB) can cause a tape drive to be occupied for an extended time which will cause congestion issues. If a researcher has files that are more than 1TB, please contact us and we will help develop a solution.

Many small files can be accumulated into a single file via the UNIX “tar” command: “tar -cpzf name.tar.gz <directory>”. We also recommend that the tar files are compressed with gzip, zstd, etc. Before compressing the files to be archived, we suggest creating a checksum file that contains cryptographic checksums of all the files in the particular archive file. This can be accomplished by the UNIX ‘find’ command: “find <directory> -type f -exec sha1sum {} \; > checksums.sha1”

We also highly suggest that research groups create READMEs for each directory of files that they have archived. These files should be descriptive enough that someone at a later date should be able to identify exactly which archived filesets they want to recall without needing to recall a file, un-archive, and peruse the contents. UT-StorR may also fully offline the archived files after a time, so that the only visible files are the READMEs themselves!

An example README template is provided here (UT login required).

The migration to tape is fully automated via DMF policies. UT-StorR will migrate off to tape any file that is idle for more than 24 hours. This gives some leeway in organizing the archived directories, but we recommend that the directory structure and READMEs be prepped before being put to UT-STorR.

Sharing

Sharing of the archived data is performed through Globus. PIs that wish to set up a share of their archived data can do so by submitting a UT-StorR service request here.