Skip to content Skip to main navigation Report an accessibility issue
High Performance & Scientific Computing

File Systems


Two filesystems are available to Open Enclave users for storing user files: the Network File System (NFS) and Lustre. NFS contains and is used for home directories and Lustre is used for project and scratch directories. Table 1.1 summarizes the available filesystems.

File SystemPathQuotaPurged
NFS Home Directory/nics/[a,b,c,d]/home/<username>10GBNot Purged
Lustre Scratch Directory/lustre/haven/user/<username>No QuotaPurged
Lustre Project Directory/lustre/haven/proj/<project>1 TB default
more by request
Not Purged

Please note that while both NFS and Lustre are reliable filesystems, errors and corruptions can still occur. It is each user’s responsibility to back up your data. To learn about data transfer to/from the Open Enclave, please review the Data Transfer document.

For more information on the Lustre file system, please refer to the Lustre User Guide.

Home Directories

On the ISAAC Open Enclave cluster, Network File System (NFS) is used for home directories. Home directories on NFS are periodically backed up for disaster recovery. Each new account on the Open Enclave receives a home directory on NFS. This is each account’s storage location for a small amount of files. Home directory is where you can store job scripts, virtual environments, and other types of files and data. In a Linux environment, you can refer to your home directory with the environment variable $HOME or with the tilde (~) character.

By default, your home directory is limited to 10GB of storage space. It is not intended to store large amounts of project or job-related data. For job-related data, please use your scratch directory. For project data that you do not want to be purged request and use project space.

To determine how much storage space you have consumed in your home directory, execute the quota -s command. Figure 2.1 shows the possible output of this command. Of interest are the first “space,” “quota,” and “limit” fields. The “space” field shows how much storage space is currently in use. The “quota” field displays the soft quota placed on your home directory. The “limit” field defines the hard quota, which is the absolute maximum storage space you can consume. When you exceed the quota, you will start a grace period that gives you time to reduce your storage space usage. If you do not reduce your storage space usage during this period, the soft limit defined by the “quota” field will be enforced.

Disk quotas for user user-x (uid 00001): 
     Filesystem   space   quota   limit   grace   files   quota   limit   grace
                  3144M  10240M  10752M           58432       0       0

Figure 2.1 – Output of quota -s

Scratch Directories

Scratch directories in the ISAAC Open Enclave are available for all users on the Lustre filesystem. Approximately 2.7 petabytes of Lustre storage space is available on /lustre/haven which is shared with scratch directories and project directories.

Important: Lustre scratch directories are NOT backed up.

Important Purging Notice: Lustre scratch space can be purged monthly on approximately the 3rd Monday of each month. Files in Lustre scratch directories (only /lustre/haven/user/{username} directories) are deleted by the purging process if they have not been accessed or modified within 180 days. In general, users have many temporary files that are no longer needed once a job completes and results are returned. Many times these files and other orphaned and unneeded files are not deleted by users and they accumulate in scratch directories and can fill the file system which is detrimental to all users. Purging email notices are sent to all active users prior to purging of scratch space which can take place on the 3rd Monday of each month. This email notice about an upcoming purge will explain the purging process and how to request a temporary purge exemption or the process to request a project space (project directories are exempt from purging). To request a temporary purge exemption submit an HPSC service request with “temporary purge exemption request” in the subject.

To transfer data out of your scratch space see the Open Enclave Data Transfer documentation.

Each user has access to a scratch directory in Lustre which is located at /lustre/haven/user/{username};. For convenience, use the $SCRATCHDIR environment variable to refer to your Lustre scratch directory.

If you wish to determine which files are eligible to be purged from Lustre space, execute the lfs find $SCRATCHDIR -mtime +180 -type f command. Files that will be purged from Lustre space are those that are not modified or accessed for 180 days. If you wish to view your total usage of Lustre space, execute the lfs quota -u <user> /lustre/haven command.

Any attempts to circumvent purging, such as using the touch command on all files in a users scratch directory, will be considered a violation of the Open Enclave acceptable use policy. Instead of taking the time to circumvent purging, why not request a project with corresponding project space. As we are all Tennessee Volunteers, our research community will be improved with positive user actions and behaviors, such as cleaning up unneeded files or requesting a project, instead of circumventing Open Enclave file purging policy. This will result in less wasted staff support time.

File System Quotas

If you want to see your home directory quota use the quota -s command.  The amount of space used will be in the space column, the soft quota value is in the quota column, and the quota hard limit is in the limit column.

$ quota -s
 Disk quotas for user user1234 (uid 99999):
      Filesystem   space   quota   limit   grace   files   quota   limit   grace
                   7508M  10240M  10752M             986       0       0      0

To see how much space you have consumed in the lustre files system overall (anywhere on /lustre) use the lfs quota command.  See the example below.  The output shows the storage space used in the used column and the number of files in the files column.  The soft quota is in the quota column and the hard limit is in the limit column.

$ lfs quota -h -u victor /lustre/haven
 Disk quotas for usr user1234 (uid 99999):
      Filesystem    used   quota   limit   grace   files   quota    limit      grace
   /lustre/haven  6.067T      0k      10T       -  147401  50000000 50000000       -