Trn3 Interconnect Bandwidth Documentation Clarification

[sikkaaman]

Describe the issue

Would appreciate clarification on a discrepancy between two tables in the docs on Amazon EC2 Trn3 architecture (https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trn3-arch.html).

1. In the "Trn3 Gen1/Gen2 UltraServer specifications" table:
   • Interconnect is listed as: "NeuronLink-v4 bandwidth (GiB/sec/device): 2,048" for both Gen1 and Gen2 UltraServers.

2. In the "Bandwidth summary" table (under Trn3 UltraServer Connectivity and Networking):

• Per-chip bandwidth across PCIe Gen6 x8 links is broken down as:
  • Intra-server (within sled): 256 GB/s (4 × PCIe Gen6 x8 via intra-server switch)
  • Inter-server (within rack): 320 GB/s (5 × PCIe Gen6 x8 via inter-server switch)
  • Inter-rack: 128 GB/s (2 × PCIe Gen6 x8 direct links)

This above totals to 704 GB/s per chip (sum of 11 × PCIe Gen6 x8 links).

How is the 2,048 GiB/sec figure calculated? Is this bidirectional or unidirectional?

Links

https://awsdocs-neuron.readthedocs-hosted.com/en/latest/about-neuron/arch/neuron-hardware/trn3-arch.html

[github-actions]

Hi @sikkaaman, Thank you for filing the issue! We will take a look and get back to you.