Cost estimates on this page were updated on April 24, 2025.

If you are working on a proposal and need the ISAAC facilities document it is available here (UT authentication required).

Overview

Faculty and other University researchers may purchase compute nodes to be placed in the ISAAC HPC cluster using sponsored research funds, startup funds, or other funds for a private condo. Most of the other infrastructure components are provided (storage, networking, operating system, software, etc). These private condo nodes are exclusive to investors, their project(s), and project members. In addition to private condo exclusive use, private condo resources participate in the responsible node sharing program to make unused compute resources available to the campus community with very short limits on allowed run times and to take advantage of idle private condo resources.

If you wish to investigate purchasing nodes, know what you want, and are ready to purchase nodes for use in a private condo, please submit an HPSC Service Request (ticket) at this link. The HPSC Director or HPSC Manager will contact you to discuss investment details. Once a researcher wants to commit to a purchase, an OIT Service Level Agreement (SLA) will be developed and signed for OIT by the Associate CIO for Service Level and Capacity Management, along with the signature of the research Principal Investigator (PI), who must be a faculty member.

The SLA will describe the equipment purchase details (equipment, cost, service period, etc.) along with other service details (which cluster the equipment will go into, how to get support, escalation, etc.) and the funding source to fund the equipment purchase. OIT will purchase it under an OIT account and then, once delivered, charge the funding source according to the details in the SLA. IMPORTANT: The SLA describes an equipment purchase. OIT HPSC then provides service as described in the SLA for no additional charge; i.e., OIT HPSC does not sell “computer services”.

For FY25, the standard SLA period for a compute or GPU node investment will be five years. Other periods, including a proposal or award performance period, will be considered on a case-by-case basis. The compute node costs below are estimated one-time costs for a five-year period.

All prices below are estimates, and pricing changes frequently. Final costs will be determined based on equipment configuration, outlined in a vendor quote (usually from Dell) and specified in an SLA. Lead times for shipment of equipment can be two to six months, depending on the equipment, due to the industry-wide shortage of GPUs, power supplies, the prices of memory, and other specialized parts. UTK has a good relationship with Dell, and we have not seen delays to date, but other customers have seen delays in shipment. Once the order is placed, Dell provides an estimated ship date.

Federal research (formerly R) accounts can only be charged the direct equipment cost since our facility is not a cost center. If the equipment is for a proposal, a vendor quote will be provided for inclusion in the proposal. For a proposal after an award is made, the vendor quote would be requested again, as vendor quotes are usually only good for 30 days.

Compute Nodes

With the 16th generation standard compute node, investors will obtain a powerful compute node with an Intel Xeon Ice Lake, AMD Genoa, or AMD Bergamo processor that can handle significant HPC workloads. The technical specifications are listed below.

Modifications can be made to the above configuration on a case-by-case basis.

GPU Nodes from Dell

GPU configurations and quotes can be quite complex. There are currently (as of May 2024) ten different types of GPUs available with three from AMD and seven from NVIDIA. Many of the lower end GPUs L40S, A40, and A16 are only available in smaller server configurations, whereas, the NVIDIA H100 which is the top-of-the-line GPU is really only available in the higher end servers, such as, the Dell PowerEdge XE8640 and XE9640. These smaller GPUs have availability but there performance and size is limited. The Director has come up with a way to purchase the XE8640 nodes with four NVIDIA H100s and get them in a reasonable timeframe. E accounts can be used to purchase a portion of a XE8640 which we will sell in 1/4 increments. R accounts can be used if they are not tied to Federal research funds.

With the Dell PowerEdge XE8640 standard GPU node, investors will obtain a high end GPU node that can provide significant GPU performance for GPU-intensive applications. The pertinent technical specifications are listed below.

This is a very feature and component rich configuration for HPC and AI/ML workloads.

OIT HPSC has already purchased several of these nodes, and researchers can buy a 25% share of one node (i.e., one H100 GPU) at $46,965.

Storage Investments

Lustre project storage space is available on each cluster and is managed with quotas. Project space is not purged. University of Tennessee projects receive 1 terabyte (TB) of Lustre project space on ISAAC clusters at no additional cost to the research project.

Project PIs may request increases to their storage quota. For FY25, the cost of storage is $100 per terabyte (TB). For all storage requests, the project PI must submit an HPSC Service Request (see “Submit HPSC Service Request” in the menu to the left of this page).

• Requests for project storage up to 10 TBs can generally be approved at no additional cost.
• Requests for project storage from 10-100 TB must submit a data management plan and attend training for UT-StoR (archival tape storage) and can generally be approved at no additional cost.
• Any request beyond 100 TBs should request a quote for the purchase of the additional storage. Note that the estimated cost ($100/TB) includes the storage drives using RAID-6; the storage infrastructure, which includes controllers, Infiniband switches and cables, drive drawers, cables, etc.; the metadata storage (SSDs and controllers); and a license and support for the ExaScalar Lustre file system software from Data Direct Networks. RAID-6 means the raw storage can withstand two drive failures without losing data.

Lustre storage is not backed up; users are responsible for backing up their data. Any request for Lustre storage backups would be addressed on a case-by-case basis and would likely involve the researchers getting access to the UT-StorR long-term archival storage to develop and implement a data management plan for their research data.

Note: VM (virtual machine) storage in the Secure Enclave is handled on a case-by-case basis. Please contact HPSC by submitting a support request (ticket) if you have questions about VM storage. For more information on the available file systems, please refer to the File Systems document.

UT-StorR Long Term Archival Storage

The UT-StorR long term archival storage system is now available and can help with the following use cases:

• researchers who use the ISAAC campus clusters and have large amounts of primary and results research data and data collections that are no longer actively used or used very infrequently
• UTK Core Facilities that manage instrument and other data and data collections for a community
• in cooperation and coordination with UTK Library, the data storage by researchers, research projects, or Core Facilities to meet the data management plans of funded research projects.

To meet federally (and other) funded projects that require data management plans you can include the costs of the tapes for long term storage of you research results, papers and other documents described in a proposal data management plan. The cost of the tapes is approximately $100 for each 18 TB LTO-9 tape cartridge. Storage in UT-StorR of one petabyte of data would be stored on 56 tapes which would cost $5,600. Currently, UT-StorR hardware, software, and maintenance is funded through FY26 and no additional costs besides tape costs should be included. We will be working with ORIED to come up with a cost recovery plan for UT-StorR. UT-StorR may become a Core Facility to implement a cost recovery mechanism. When the cost recovery is known and available this paragraph will be updated with those costs that should be included in budgets for research proposals that have a data management plan requirement.

For more information on UT-StorR see the Archival Menu on the OIT HPSC pages (in the menu to the left of this page).

Investment History

Historical Server Pricing for Comparisons

Server Type	Server Components	Date/Price/ $ per core or GPU
Dell PowerEdge R660	Intel Xeon Gold 6438M, 64 cores 256GB DDR5-4800 memory Mellanox ConnectX-6 HDR Infiniband Ethernet NIC & cables	Jan-2024 $17,137.11 $267.77
Dell PowerEdge R6625	AMD EPYC 9734, 224 cores total 1,024 GB DDR5-4800 memory Mellanox ConnectX-6 HDR Infiniband Ethernet NIC & cables	Dec-2023 $33,447 $149.31
Dell PowerEdge R650	Intel Xeon Gold 6348, 56 cores 256 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Sep-2022 $13,581.59 $242.53
Dell PowerEdge R6525	AMD EPYC 7713, 128 cores 512 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Sep-2022 $20,999.11 $164.06
Dell PowerEdge R640	Intel Xeon Gold 6248R, 48 cores 192 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables	May-2021 $9,131.45 $190.24
Dell PowerEdge R7515	AMD EPYC 7763, 128 cores 256 GB DDR4-3200 memory Mellanox ConnectX-6 HDR, Ethernet NIC & cables (3000078872607.1)	Mar-2021 $11,861 $92.66
Dell PowerEdge XE8640	Intel Xeon Platinum 8462Y+, 64 cores 1,024GB DDR5-4800 memory 4x NVIDIA H100 SXM 80GB GPU 51.2 TB NVMe SSD storage Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Jul-2023 $144,900 $36k/GPU
Dell PowerEdge R750	Intel Xeon Gold 6348, 56 cores 256 GB DDR4-3200 memory NVIDIA A40 48GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Sep-2022 $23,835 $23k/GPU
Dell PowerEdge R750XA	Intel Xeon Platinum 8358, 64 cores 1,024 GB DDR4-3200 memory 4x NVIDIA A40 48GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Nov-2021 $57,830 $14.5k/GPU
Dell PowerEdge R7525	AMD EPYC 7713, 128 cores 512GB DDR4-3200 memory 2x NVIDIA A100 40GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Sep-2021 $39,093 $19k/GPU
Dell PowerEdge R740	Intel Xeon Gold 6248R, 48 cores 192 GB DDR4-3200 memory 2x NVIDIA V100S 32GB GPU Mellanox ConnectX-6 HDR, Ethernet NIC & cables	Dec-2021 $22,655 $11.3k/GPU Mar-2021 $20,500 $10k/GPU