Two filesystems are available to Open Enclave users for storing user files: the Network File System (NFS) and Lustre Haven. NFS contains and is used for home directories and Lustre Haven contains and is used for project and scratch directories. Table 1.1 summarizes the available filesystems.
|NFS Home Directory||/nics/[a,b,c,d]/home/<username>||10GB||Not Purged|
|Lustre Scratch Directory||/lustre/haven/user/<username>||No Quota||Purged|
|Lustre Project Directory||/lustre/haven/proj/<project>||1 TB default|
more by request
Please note that while both NFS and Lustre are reliable filesystems, errors and corruptions can still occur. It is each user’s responsibility to back up your data. To learn about data transfer to/from the Open Enclave, please review the Data Transfer document.
For more information on the Lustre file system, please refer to the Lustre User Guide.
On the Open Enclave, fifteen terabytes of storage space is available via NFS for home directories. Home directories on NFS are periodically backed up for disaster recovery. Each new account on the Open Enclave receives a home directory on NFS. This is each account’s personal storage location for a small amount of files. Here you can store job scripts, virtual environments, and other types of files and data. In a Linux environment, you can refer to your home directory with the environment variable $HOME or with the tilde (~) character.
By default, your home directory is limited to 10GB of storage space. It is not intended to store large amounts of project or job-related data. For job-related data, please use your scratch directory. For project data that you do not want to be purged request and use project space.
To determine how much storage space you have consumed in your home directory, execute the
quota -s command. Figure 2.1 shows the possible output of this command. Of interest are the first “space,” “quota,” and “limit” fields. The “space” field shows how much storage space is currently in use. The “quota” field displays the soft quota placed on your home directory. The “limit” field defines the hard quota, which is the absolute maximum storage space you can consume. When you exceed the quota, you will start a grace period that gives you time to reduce your storage space usage. If you do not reduce your storage space usage during this period, the soft limit defined by the “quota” field will be enforced.
Disk quotas for user user-x (uid 00001): Filesystem space quota limit grace files quota limit grace nfs.nics.utk.edu:/nfs/b 3144M 10240M 10752M 58432 0 0
Figure 2.1 – Output of quota -s
Scratch directories are available for all users on the Lustre Haven filesystem. Approximately 2.7 petabytes of Lustre storage space is available on /lustre/haven which is shared with scratch directories and project directories.
Important: Lustre scratch directories are NOT backed up.
Important Purging Notice: Lustre Haven scratch space is purged monthly on approximately the 3rd Monday of each month. Files in Lustre Haven scratch directories are deleted by the purging process if they have not been accessed or modified within 180 days. Over the course of 2020, the purging timeperiod will be gradually reduced to 90 days. See the purging schedule below. In general, users have many temporary files that are no longer needed once a job completes and results are returned. Many times these files and other orphaned and unneeded files are not deleted by end users and they accumulate in scratch directories and can fill the file system which is detrimental to all users. Purging email notices are sent to all active users at least once at the beginning of each month to remind users that purging of scratch space will take place on the 3rd Monday of each month. This notice explains the purging process and how to request a temporary purge exemption or the process to request a project space (project directories are exempt from purging). To request a temporary purge exemption submit a service request with “temporary purge exemption request” in the subject.
|Purge Estimated Date||Path||Files Purge if not accessed or modified in|
|Jul 20, 2020||/lustre/haven/user/<username>||<180 days|
|Aug 17, 2020||/lustre/haven/user/<username>||<180 days|
|Sep 21, 2020||/lustre/haven/user/<username>||<180 days|
|Oct 19, 2020||/lustre/haven/user/<username>||<180 days|
|Nov 16, 2020||/lustre/haven/user/<username>||<180 days|
|Dec 21, 2020||/lustre/haven/user/<username>||<180 days|
To transfer data out of your scratch space see the Open Enclave Data Transfer documentation.
Each user has access to a scratch directory in Lustre which is located at /lustre/haven/user/<username>. For convenience, use the $SCRATCHDIR environment variable to refer to your Lustre scratch directory.
If you wish to determine which files are eligible to be purged from Lustre space, execute the
lfs find $SCRATCHDIR -mtime +180 -type f command. Files that will be purged from Lustre space are those that are not modified or accessed for 180 days. If you wish to view your total usage of Lustre space, execute the
lfs quota -u <user> /lustre/haven command.
Any attempts to circumvent purging, such as using the touch command on all files in a users scratch directory, will be considered a violation of the Open Enclave acceptable use policy. Instead of taking the time to circumvent purging, why not request a project with corresponding project space. As we are all Tennessee Volunteers, our research community will be improved with positive user actions and behaviors, such as cleaning up unneeded files or requesting a project, instead of circumventing Open Enclave file purging policy. This will result in less wasted staff support time.
If you want to see your home directory quota use the quota -s command. The amount of space used will be in the space column, the soft quota value is in the quota column, and the quota hard limit is in the limit column.
$ quota -s Disk quotas for user user1234 (uid 99999): Filesystem space quota limit grace files quota limit grace nfs.nics.utk.edu:/nfs/b 7508M 10240M 10752M 986 0 0 0
To see how much space you have consumed in the lustre files system overall (anywhere on /lustre) use the lfs quota command. See the example below. The output shows the storage space used in the used column and the number of files in the files column. The soft quota is in the quota column and the hard limit is in the limit column
$ lfs quota -h -u victor /lustre/haven Disk quotas for usr user1234 (uid 99999): Filesystem used quota limit grace files quota limit grace /lustre/haven 6.067T 0k 10T - 147401 50000000 50000000 - 0