Investment criteria and standard node configurations are reviewed annually at the beginning of each fiscal year by OIT. This page has been updated for FY24 on on August 15, 2023 and again on December 2023 due to price fluctuations.
If you are working on a proposal and need the ISAAC facilities document it is available here.
(UT authentication required)
Faculty and other University researchers may purchase compute nodes to be placed in the ISAAC HPC cluster using sponsored research funds, startup funds, or other funds for a private condo. These private condo nodes are exclusive to investors, their project(s), and project members. In addition to private condo exclusive use (the default) private condo resources can participate in the responsible node sharing program.
If you wish to investigate purchasing nodes or know what you want and are ready to purchase nodes for use in a private condo the cluster, please submit an HPSC Service Request (ticket) at this link. A staff member will contact you to discuss details regarding the purchase and once you want to commit to a purchase you will work with the Associate CIO for Service Level and Capacity Management to develop a service level agreement (SLA) that will apply for the equipment purchase (equipment, cost, service period, etc.).
For FY23, the standard SLA period for a compute or GPU node investment will be three years. Other periods will be considered on a case-by-case basis including a proposal or award performance period.
Note: Costs below are estimates; final costs will be determined based upon equipment configuration (usually in the form of a vendor quote or an OIT Service Level Agreement document corresponding to the funding source).
Federal research R accounts can only be charged the direct equipment cost since our facility is not a cost center. A vendor quote would be provided for inclusion in a proposal. After an award, the vendor quote would be requested again, as usually vendor quotes are only good for 30 days.
The compute node costs below are estimated one-time costs for a three year period.
Note: All prices below are estimates, and pricing changes monthly (approximately). Dell advised HPSC in December 2023 that prices are fluctuating and lead times for shipment of equipment can be up to six months due to the industry-wide shortage of GPUs, power supplies, the prices of memory, and other specialized parts. UTK has a good relationship with Dell and we have not seen delays to date but other customers have seen delays in shipment. When an order is placed from a quote we usually get estimated ship dates at the time of order.
With the 16th generation standard compute node, investors will obtain a powerful compute node based on Intel Xeon Ice Lake processors or AMD Genoa or Bergamo processors that can handle significant HPC workloads. The pertinent technical specifications are listed below.
Intel Sapphire Rapids based compute node
The estimated cost for a standard Intel processor compute node is approximately $17,137.11 ($267.77 per core). Modifications can be made to the above configuration on a case-by-case basis.
AMD EPYC based compute node
The estimated cost for a Dell PowerEdge R6625 is approximately $33,447 ($149.31 per core). Modifications can be made to the above configuration on a case-by-case basis.
GPU quotes were obtained in 2023 and since then Dell 16th generation technology has been announced. We will get updated GPU server quotes from Dell as soon as we can.
With the standard GPU node, investors will obtain a compute node that can handle standard HPC workloads and has significant GPU performance for GPU-intensive applications. The pertinent technical specifications are listed below.
The estimated cost for a 15th generation GPU node with one NVIDIA 48GB A40 GPU is $23,952. Modifications can be made to this configuration on a case-by-case basis.
The estimated cost for a Dell PowerEdge R6525 with one NVIDIA 48GB A40 GPU is $30,473. Modifications can be made to the above configuration on a case-by-case basis.
Lustre project storage space is available on each cluster managed with quotas. Project space is not purged. There is also Lustre scratch space available on each cluster managed with quotas on all of the ISAAC clusters. As needed Lustre scratch storage will be purged with a minimum of two weeks notice as part of routine cluster maintenance. We have not had to purge space on any cluster’s Lustre since late 2021. OIT HPSC tries as best as possible to maintain an amount of storage to meet demands of the research community to drastically reduce the need to purge. However, we reserve the right to coordinate purging on any Lustre scratch space to keep enough free space on any Lustre file system to maintain a healthy file system.
University of Tennessee projects receive 1 terabyte (TB) of Lustre project space at no additional cost to the research project. Requests for project space up to 10 TBs can be requested at any time by submitting an HPSC Service Request (see Submit HPSC Service Request in the menu to the left of this page). The OIT HPSC staff can approve any reasonable request for additional project space up to 10 TBs. Any project storage between 10 and 100 TBs should be brought to the attention of the HPSC Director for consideration also via a HPSC Service Request. Any request beyond 100 TBs for an existing project or for a sponsored research proposal should request a quote for the purchase of the additional storage using research, department or faculty project funds.
External organizations and University faculty leading sponsored projects or other funded projects can purchase storage resources on the high-performance Lustre storage system for their research projects. Costs would be determined on a case-by-case basis and based on funding source. In many cases, we would obtain a quote for directly buying drives for the storage amount requested that will go into the empty drive slots in the storage subsystem. Also, Lustre file systems have a finite number of files that the entire file system can contain. Any project or researcher that needs more that 10 million files will have to be addressed on a case-by-case basis.
Backups of Lustre storage are the responsibility of the end users. The three Lustre files systems are 1.3 petabytes, 2.9 petabytes, and 3.6 petabytes and we do not have the capability to backup that much space. Lustre file systems are very reliable and backups are not provided on any Lustre space by default. Any request for Lustre storage backups would be addressed on a case-by-case basis.
Note: VM (virtual machine) storage in the Secure Enclave is handled on a case-by-case basis. Please contact HPSC directly via email or help request ticket if your project has questions about VM storage.
On a case-by-case basis, arrangements may be made for some varieties of back-ups of limited size, such as a duplicate copy or snapshot. If these sorts of backups is desired, project PIs should discuss these desired backups with HPSC staff, so such requirements can be included any investment Service Level Agreement. For more information on the available file systems, please refer to the File Systems document.
| Server Type | Server Components | Date/Price/ $ per core or GPU |
| Dell PowerEdge R660 | Intel Xeon Gold 6438M, 64 cores 256GB DDR5-4800 memory Mellanox ConnectX-6 HDR Infiniband Ethernet NIC & cables | Jan-2024 $17,137.11 $267.77 |
| Dell PowerEdge R6625 | AMD EPYC 9734, 224 cores total 1,024 GB DDR5-4800 memory Mellanox ConnectX-6 HDR Infiniband Ethernet NIC & cables | Dec-2023 $33,447 $149.31 |
| Dell PowerEdge R650 | Intel Xeon Gold 6348, 56 cores 256 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Sep-2022 $13,581.59 $242.53 |
| Dell PowerEdge R6525 | AMD EPYC 7713, 128 cores 512 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Sep-2022 $20,999.11 $164.06 |
| Dell PowerEdge R640 | Intel Xeon Gold 6248R, 48 cores 192 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables | May-2021 $9,131.45 $190.24 |
| Dell PowerEdge R7515 | AMD EPYC 7763, 128 cores 256 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables (3000078872607.1) | Mar-2021 $11,861 $92.66 |
| Dell PowerEdge XE8640 | Intel Xeon Platinum 8462Y+, 64 cores 1,024GB DDR5-4800 memory 4x NVIDIA H100 SXM 80GB GPU 51.2 TB NVMe SSD storage Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Jul-2023 $144,900 $36k/GPU |
| Dell PowerEdge R750 | Intel Xeon Gold 6348, 56 cores 256 GB DDR4-3200 memory NVIDIA A40 48GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Sep-2022 $23,835 $23k/GPU |
| Dell PowerEdge R750XA | Intel Xeon Platinum 8358, 64 cores 1,024 GB DDR4-3200 memory 4x NVIDIA A40 48GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Nov-2021 $57,830 $14.5k/GPU |
| Dell PowerEdge R7525 | AMD EPYC 7713, 128 cores 512GB DDR4-3200 memory 2x NVIDIA A100 40GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Sep-2021 $39,093 $19k/GPU |
| Dell PowerEdge R740 | Intel Xeon Gold 6248R, 48 cores 192 GB DDR4-3200 memory 2x NVIDIA V100S 32GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables | Dec-2021 $22,655 $11.3k/GPU Mar-2021 $20,500 $10k/GPU |
Explore
Write
Chat
Call