Introduction

At the time of this writing, three file systems run on the Secure Enclave: NFS (Network File System), EncFS, and Lustre SIP. Home directories are stored on NFS. EncFS holds encrypted project directories. Lustre SIP provides high-performance storage for job-related data, data transfers, and project directories. Table 1.1 summarizes the filesystems available to the Secure Enclave.

While all these file systems are reliable, errors and corruptions can still occur. It is your responsibility to back up your data. To learn about this process on the Secure Enclave, please consult the Data Transfer document.

Home Directories

For home directories, the Secure Enclave uses NFS. 500GB of storage space is available to this filesystem. When you receive a new account on the Secure Enclave, a home directory is automatically created for your use. You will always start in your home directory on the SIP. No other users can access your home directory. Here you may store job scripts, virtual environments, and other types of data up to the quota limit. For convenience, refer to your home directory with the tilde (~) character or with the $HOME environment variable.

Users on the Secure Enclave possess 1GB of storage space in their home directories. Due to the limited size of this space, it is not suitable for large amounts of data. For job-related storage, use the Lustre SIP filesystem. To learn how to back up your data from the Secure Enclave to your own storage resources, please refer to the Data Transfer document.

Home directories are not purged and are regularly backed up. Please note that your home directory is not encrypted.

Project Directories

Encrypted project directories on the cluster use the EncFS filesystem for encryption. Unless they are explicitly mounted, they will be unavailable. Users can mount these directories on the cluster with the sipmount command. Figure 3.1 shows the syntax for this command.

sudo /usr/local/bin/sipmount <project-directory>

Figure 3.1 – Usage of the sipmount Command

The sudo command must be used with sipmount because mounting is a privileged action on the cluster. When you use sudo, you will be prompted to supply your NetID password and authenticate with Duo TFA. After you successfully authenticate, you can use the mount command to verify that the project directory was successfully made available to you. Figure 3.2 shows the process for mounting a project directory. You can access the contents of the project directory from the /projects/SIP-<project-ID> path.

[user@sip-login1 ~]$ sudo /usr/local/bin/sipmount SIP-UTK9999
[sudo] password for user: 
Duo two-factor login for user

Enter a passcode or select one of the following options:

 1. Duo Push to XXX-XXX-1234
 2. SMS passcodes to XXX-XXX-1234

Passcode or option (1-2): 1

Pushed a login request to your device...
Success. Logging you in...
[user@sip-login1 ~]$ mount | grep SIP-UTK9999
encfs on /projects/SIP-UTK9999 type fuse.encfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Figure 3.2 – Mounting an Encrypted Project Directory

Mounted project directories will automatically be unmounted after fifteen minutes of inactivity, but if you wish to ensure that the project directory is secured, you can unmount it with the sipunmount command. The syntax and usage of the command is equivalent to the sipmount command.

If you wish to transfer data from an encrypted project directory to another storage resource, please refer to the Data Transfer document.

Scratch Directories

The Lustre SIP file system provides about 15 terabytes (TB) of global high performance scratch space for data sets related to running jobs on the Secure Enclave resources and transferring data in and out of the data transfer nodes. Every user has their own scratch directory created at account creation time located in their lustre project space /lustre/sip/proj/{project}/{username}. The environment variable $SCRATCHDIR points to each user’s scratch directory location. Scratch space on Secure Enclave can be purged weekly, but it has no storage space or quota limit. Scratch space on the Secure Enclave is for storage of data in use and is not required to be encrypted as described in the Secure Enclave Security Plan Data that is at rest (not being used) is required to be encrypted in the project space.

Lustre SIP Scratch directories are NOT backed up.

Important Points for Users Using Lustre SIP Scratch

• The Lustre SIP Scratch file system is scratch space, intended for work related to job setup, running jobs, and job cleanup and post-processing on SIP resources and not for long term data storage. Files in scratch directories are not backed up and data that has not been used for 30 days is subject to being purged. It is the user’s responsibility to back up all important data to another storage resource.The Lustre find command can be used to determine files that are eligible to purge:> lfs find /lustre/sip/{project}/$USER -mtime +30 -type f
• Striping is an important concept with Lustre—. Striping is the ability to break files into chunks and spread them across multiple storage targets (called OSTs). The striping defaults set up for NICS resources are usually sufficient but may need to be altered in certain use cases, like when dealing with very large files. Please see our Lustre Striping Guide for details.
• Beware of using normal Linux commands for inspecting and managing your files and directories in Lustre scratch space. Using ls -l can cause undue load and may hang because it necessitates access to all OSTs holding your files. Make sure that your ls is not aliased to ls -l.
• Use lfs quota to see your total usage on the Lustre system. You must specify your username and the Lustre path with this command, for example:> lfs quota -u <username> /lustre/haven

For more detailed information regarding Lustre usage, see the following pages:

• Lustre Striping Guide
• Lustre Architecture
• Lustre Troubleshooting

NICS will be developing additional storage policies and will notify users about any storage policy changes.