Globus CLI is a tool developed by Globus which provides an interface to interact with different Globus services using the command prompt or terminal. Using CLI, users can orchestrate data transfer between two endpoints, sync directories, or manipulate directory structure in their account on ISAAC cluster without using the Globus web interface. One of the interesting points about Globus CLI is that users do not need to have CLI installed on each of the computers or clusters to use its features.
In ISAAC infrastructure, we have installed Globus CLI on ISAAC Next Gen cluster as a module. After logging in to the cluster, users can load the module for Globus CLI using the command:
module load globusCLI/3.3.0
Important Note: The Globus CLI is written in python and is available as a python package on PyPI. We used anaconda3/2021.05 to load the python environment and install globus CLI. However, the users does not need to load the Anaconda module. It will get loaded automatically while loading the globusCLI module.
Before starting the transfer of data, let us understand and learn how to use Globus CLI and access the endpoints.
Assuming you are already logged in to the ISAAC NG cluster, please follow the below steps to login to Globus using Globus CLI:
module load globusCLI/3.3.0
To find a list of different commands which work in globus CLI, type the below command:
globus endpoint search "ISAAC-NG DTN1"
globus endpoint search --help
Let us say that we want to transfer data from ISAAC Legacy to ISAAC NG. We need to find the endpoint for ISAAC Legacy. We can find that out by using the same command as we used for ISAAC NG.
globus endpoint search "ISAAC Legacy"
To transfer a single file from one endpoint to another, we will use the below command:
globus transfer <source-endpoint>:/path/to/the/file/filename <destination-endpoint>://path/where/file/will/be/saved/filename
To transfer the file from ISAAC Legacy DTN (source) to ISAAC NG DTN (destination), the command will be written as:
globus transfer $ep_legacy:/nics/b/home/<UTNetID>/tesfile $ep_ng:/nfs/home/<UTNetID>/testfile
To copy a directory containing certain files and/or sub-directories from ISAAC Legacy (endpoint ep_legacy) to ISAAC NG, use -r flag after the globus transfer command
globus transfer -r $ep_legacy:/nics/b/home/<UTNetID>/directory_name $ep_ng:/nfs/home/<UTNetID>/directory_name
Here the flag -r stands for copying the files recursively
In order to sync the contents of the directory on the source endpoint (say ep_legacy) to destination endpoint (ep_ng) directory, use the flag -s or –sync-level followed by one of the four values
globus transfer -s exists $ep_legacy:/nics/b/home/<UTNetID>/directory_name $ep_ng:/nfs/home/<UTNetID>/directory_name
The above command will copy only those contents from the source directory to destination directory which are not present on the destination directory.