®
32-Lane 8-Port PCIe® Gen2
System Interconnect Switch with
Non-Transparent Bridging
◆
89HPES32NT8BG2
Product Brief
The 89HPES32NT8BG2 is a member of the IDT family of PCI
Express® switching solutions. The PES32NT8BG2 is a 32-lane, 8-port
system interconnect switch optimized for PCI Express Gen2 packet
switching in high-performance applications, supporting multiple simulta-
neous peer-to-peer traffic flows. Target applications include multi-host or
intelligent I/O based systems where inter-domain communication is
required, such as servers, storage, communications, and embedded
systems.
With Non-Transparent Bridging functionality and innovative Switch
Partitioning feature, the PES32NT8BG2 allows true multi-host or multi-
processor communications in a single device. Integrated DMA control-
lers enable high-performance system design by off-loading data transfer
operations across memories from the processors. Each lane is capable
of 5 GT/s link speed in both directions and is fully compliant with PCI
Express Base Specification 2.1.
Device Overview
Features
◆
High Performance Non-Blocking Switch Architecture
–
32-lane, 8-port PCIe switch with flexible port configuration
–
Integrated SerDes supports 5.0 GT/s Gen2 and 2.5 GT/s
Gen1 operation
–
Delivers up to 32 GBps (256 Gbps) of switching capacity
–
Supports 128 Bytes to 2 KB maximum payload size
–
Low latency cut-through architecture
–
Supports one virtual channel and eight traffic classes
◆
Port Configurability
–
Eight x4 switch ports
•
Adjacent x4 ports can be merged to achieve x8 port widths
–
Automatic per port link width negotiation
(x8 --> x4 --> x2 --> x1)
–
Crosslink support
–
Automatic lane reversal
–
Per lane SerDes configuration
•
De-emphasis
•
Receive equalization
•
Drive strength
◆
Innovative Switch Partitioning Feature
–
Supports up to 8 fully independent switch partitions
–
Logically independent switches in the same device
–
Configurable downstream port device numbering
–
Supports dynamic reconfiguration of switch partitions
•
Dynamic port reconfiguration — downstream, upstream,
non-transparent bridge
•
Dynamic migration of ports between partitions
•
Movable upstream port within and between switch partitions
Non-Transparent Bridging (NTB) Support
–
Supports up to 8 NT endpoints per switch, each endpoint can
communicate with other switch partitions or external PCIe
domains or CPUs
–
6 BARs per NT Endpoint
•
Bar address translation
•
All BARs support 32/64-bit base and limit address translation
•
Two BARs (BAR2 and BAR4) support look-up table based
address translation
–
32 inbound and outbound doorbell registers
–
4 inbound and outbound message registers
–
Supports up to 64 masters
–
Unlimited number of outstanding transactions
◆
Multicast
–
Compliant with the PCI-SIG multicast
–
Supports 64 multicast groups
–
Supports multicast across non-transparent port
–
Multicast overlay mechanism support
–
ECRC regeneration support
◆
Integrated Direct Memory Access (DMA) Controllers
–
Supports up to 2 DMA upstream ports, each with 2 DMA chan-
nels
–
Supports 32-bit and 64-bit memory-to-memory transfers
•
Fly-by translation provides reduced latency and increased
performance over buffered approach
•
Supports arbitrary source and destination address alignment
•
Supports intra- as well as inter-partition data transfers using
the non-transparent endpoint
–
Supports DMA transfers to multicast groups
–
Linked list descriptor-based operation
–
Flexible addressing modes
•
Linear addressing
•
Constant addressing
◆
Quality of Service (QoS)
–
Port arbitration
•
Round robin
–
Request metering
•
IDT proprietary feature that balances bandwidth among
switch ports for maximum system throughput
–
High performance switch core architecture
•
Combined Input Output Queued (CIOQ) switch architecture
with large buffers
◆
Clocking
–
Supports 100 MHz and 125 MHz reference clock frequencies
–
Flexible port clocking modes
IDT and the IDT logo are registered trademarks of Integrated Device Technology, Inc.
1 of 7
©
2009 Integrated Device Technology, Inc.
September 15, 2009
IDT 89HPES32NT8BG2 Product Brief
•
Common clock
•
Non-common clock
•
Local port clock with SSC (spread spectrum setting) and port
reference clock input
◆
Hot-Plug and Hot Swap
–
Hot-plug controller on all ports
•
Hot-plug supported on all downstream switch ports
–
All ports support hot-plug using low-cost external I
2
C I/O
expanders
–
Configurable presence-detect supports card and cable appli-
cations
–
GPE output pin for hot-plug event notification
•
Enables SCI/SMI generation for legacy operating system
support
–
Hot-swap capable I/O
◆
Power Management
–
Supports D0, D3hot and D3 power management states
–
Active State Power Management (ASPM)
•
Supports L0, L0s, L1, L2/L3 Ready, and L3 link states
•
Configurable L0s and L1 entry timers allow performance/
power-savings tuning
–
SerDes power savings
•
Supports low swing / half-swing SerDes operation
•
SerDes associated with unused ports are turned off
•
SerDes associated with unused lanes are placed in a low
power state
◆
Reliability, Availability, and Serviceability (RAS)
–
ECRC support
–
AER on all ports
–
SECDED ECC protection on all internal RAMs
–
End-to-end data path parity protection
–
Checksum Serial EEPROM content protected
–
Ability to generate an interrupt (INTx or MSI) on link up/down
transitions
◆
Initialization / Configuration
–
Supports Root (BIOS, OS, or driver), Serial EEPROM, or
SMBus switch initialization
–
Common switch configurations are supported with pin strap-
ping (no external components)
–
Supports in-system Serial EEPROM initialization/program-
ming
◆
On-Die Temperature Sensor
–
Range of 0 to 127.5 degrees Celsius
–
Three programmable temperature thresholds with over and
under temperature threshold alarms
–
Automatic recording of maximum high or minimum low
temperature
◆
9 General Purpose I/O
◆
Test and Debug
–
Ability to inject AER errors simplifies in system error handling
software validation
–
On-chip link activity and status outputs available for several
ports
2 of 7
–
Per port link activity and status outputs available using
external I
2
C I/O expander for all remaining ports
–
Supports IEEE 1149.6 AC JTAG and IEEE 1149.1 JTAG
◆
Standards and Compatibility
–
PCI Express Base Specification 2.1 compliant
–
Implements the following optional PCI Express features
•
Advanced Error Reporting (AER) on all ports
•
End-to-End CRC (ECRC)
•
Access Control Services (ACS)
•
Device Serial Number Enhanced Capability
•
Sub-System ID and Sub-System Vendor ID Capability
•
Internal Error Reporting
•
Multicast
•
VGA and ISA enable
•
L0s and L1 ASPM
•
ARI
◆
Power Supplies
–
Requires three power supply voltages (1.0V, 2.5V, and 3.3V)
◆
Packaged in a 23mm x 23mm 484-ball Flip Chip BGA with
1mm ball spacing
September 15, 2009
IDT 89HPES32NT8BG2 Product Brief
Figure 1 illustrates the block architecture of the switch. It contains a flexible and high-performance switch core, 32 full-duplex SerDes capable of
delivering PCI Express Gen 2 speeds, up to 8 ports with movable upstream ports, and a number of innovative and unique features that enables
product differentiation, cost and power saving, and time-to-market improvement.
Block Diagram
8-Port Switch Core / 32 Gen2 PCI Express Lanes
Frame Buffer
Route Table
Port
Arbitration
Scheduler
Transaction Layer
Data Link Layer
Transaction Layer
Data Link Layer
Transaction Layer
Data Link Layer
Multiplexer / Demultiplexer
Phy
Logical
Layer
Phy
Logical
Layer
Phy
Logical
Layer
Phy
Logical
Layer
Multiplexer / Demultiplexer
Phy
Logical
Layer
Phy
Logical
Layer
Phy
Logical
Layer
Phy
Logical
Layer
Multiplexer / Demultiplexer
Phy
Logical
Layer
Phy
Logical
Layer
Phy
Logical
Layer
Phy
Logical
Layer
SerDes
SerDes
SerDes SerDes
SerDes
SerDes
SerDes SerDes
SerDes
SerDes
SerDes SerDes
(Port 0)
(Port 2)
(Ports 4, 6, 8, 12, 16)
(Port 20)
Figure 1 PES32NT8BG2 Block Diagram
3 of 7
September 15, 2009
IDT 89HPES32NT8BG2 Product Brief
Applications
Advanced Failover System
PES32NT8BG2 is optimized for storage, communications control
plane, embedded systems and high-port PCI Express fanout applica-
tions. Applications shown here are generic to IDT’s Gen2 PCIe switch
family.
Storage Applications
CPU
CPU
Dual host control with failover (shown in Figure 4) ensures reliability
and availability of systems in case of CPU failure. NTB across domains
allows active-active dual host topology, where both CPUs can access all
the downstream line cards. If the primary CPU fails, dynamic switch
partitioning capability can allow reallocation of downstream ports to the
secondary CPU and maintain continuity of operation.
Before Failover
After Failover
Active/
Primary
CPU or Card
Upstream
x4
NTB
NTB
P2P
x8
PES32NT8G2
P2P
NTB
N TB
x8
x8
PES32NT8G2
NTB
NTB
P2P
Active/
Primary
CPU or Card
Upstream
x4
Active/
Secondary
CPU or Card
NTB
x4
Active/
Secondary
CPU or Card
NTB
x4
P2P
P2P
P2P
NTB
N TB
P2P
P2P
P2P
P2P
NTB
NTB
P2P
P2P
P2P
…
PES32NT24G2
20 x1
P2P
P2P
P2P
…
PES32NT24G2
I/O
I/O
I/O
Intelligent
I/O
Intelligent
I/O
I/O
I/O
I/O
20 x1
FPGA /
Local
NPU / /
FPGA
Local
NPU
FPGA
ASIC / /
Devices
Devices
Local
ASIC /
NPU
Devices
ASIC
Line Card
Line Card
Line Card
Line Card
Line Card
x4
x4
x4
x4
x4
x4
x4
x4
…
FPGA /
Line Card
Local
NPU / /
FPGA
Local
ASIC / /
Devices
NPU
FPGA
Devices
Local
ASIC /
NPU
Devices
ASIC
…
Figure 2 Storage Redundancy Model
Figure 4 Dual-Host Failover System
Dual-Star Topology
Large RAID storage systems, either direct-attached or SAN/NAS
attached, are generally built with redundancy. PCI Express switches are
used to connect CPUs to I/O controllers and across the redundancy
trunks, as illustrated in Figure 2. I/O controllers can be SAS, SATA, Fibre
Channel, or some combination of these. NTB functions enable commu-
nications across PCI Express domains and allow data exchanges and
synchronization across trunks. Additional NTB ports can also be
connected to intelligent I/O cards hosting embedded processors.
Communications Control Plane Applications
High port count, dynamic switch partitioning configuration, and
multiple NTB ports enable many possible communications control
configurations. Some common configurations are depicted below.
High Fanout
A dual-star topology, shown in Figure 5, ensures that no single node
or connection failure will bring down the system. Each I/O or line card
has connectivity to one primary CPU and also backup connectivity to a
secondary controller via an NTB port. If failure occurs, the NTB port can
be reallocated via dynamic reconfiguration of the line card switch while
the secondary CPU takes over. A high number of ports will allow the
switch to connect to many line cards.
Before Failover
Active/
Primary
CPU or Card
x4
P2P
After Failover
Standby/
Secondary
CPU or Card
Active/
Primary
CPU or Card
x4
P2P
Standby/
Secondary
CPU or Card
PES32NT24G2
x4
NTB
P2P
PES32NT24G2
NTB
NTB
P2P
x4
A straightforward use of the switch is to fan out to many different line
cards as illustrated in Figure 3. Cascading with another Gen 2 PCIe
switch will allow more than twenty-three x1 downstream ports.
NTB
P2P
P2P
P2P
…
…
P2P
P2P
P2P
P2P
P2P
P2P
…
NTB
…
P2P
P2P
P2P
22 x1
NTB
3-port NT
switch
IO /
FPGA /
NPU
NTB
3-port NT
switch
IO /
FPGA /
NPU
NTB
3-port NT
switch
IO /
FPGA /
NPU
22 x1
NTB
NTB
3-port NT
switch
IO /
FPGA /
NPU
NTB
Reallocated
3-port NT
switch
IO /
FPGA /
NPU
3-port NT
switch
IO /
FPGA /
NPU
CPU
x4
x4
…
…
Figure 5 Dual-Star Topology
PES32NT24G2
PES32NT24G2
42 x1
To FPGA or ASIC
Figure 3 High Fanout Usage Model
4 of 7
September 15, 2009
IDT 89HPES32NT8BG2 Product Brief
Feature Descriptions & Benefits
Switch Partitioning
Switch partitioning is an innovative and unique IDT feature that
allows a switch to be statically or dynamically reconfigured into multiple
independent logical switches within a single physical device.
PES32NT8BG2 can support up to 8 partitions. Any port can be an
upstream port or downstream port and any root can have zero, one, or
more downstream ports associated with its partition (see Figure 6). The
partition configuration can be done statically or dynamically by writing
into the switch configuration registers via configuration EEPROM, I
2
C
interface, or one of the roots.
Bandwidth Balancing
Dynamic switch partitioning can be utilized to perform I/O bandwidth
balancing to optimize overall system throughput (Figure 7). A multi-root
system, such as in bladed systems, may have unbalanced traffic density
across its I/O cards. System bandwidth balancing can be performed by
dynamically re-allocating low-traffic or idle I/Os to heavy traffic density
partitions from the software application layer.
ROOT
ROOT
ROOT
ROOT
ROOT
ROOT
ROOT
ROOT
89HPES32NT24G2
64-lane, 16-port PCIe Gen2
System Interco nnect Switch
89HPES32NT24G2
64-lane, 16-port PCIe Gen2
System Interconnect Switch
ROOT
ROOT
ROOT
ROOT
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
Initial System State
Re-allocated Resources
Figure 7 Dynamic Redistribution of I/Os to Optimize System Bandwidth
P2P
P2P
P2P
P2P
Flexible Slot Mapping for Hardware Re-Use
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
P2P
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
The flexibility of port mapping in switch partitioning allows maximum
hardware re-use for multiple variants of product line configurations to
meet the customized needs of your end customers, saving cost and
improving time to market.
Figure 8 below illustrates a 2-socket CPU vs. a 4-socket CPU config-
uration using the same hardware platform with a different switch parti-
tioning setup.
Figure 6 Example of Switch Partitioning Logical View
Switch partitioning enables a number of applications, allowing unique
benefits and differentiating value proposition for your products. See
Table 1 for a partial list of these benefits.
Application
Replacing multiple discrete
switches
Bandwidth balancing in multi-root
multi processor systems
Flexible slot mapping
Benefits
Saves power, space and cost over
multiple discrete PCIe switches
Improved performance through
optimal allocation of system
resources
Saves power, space and cost over
PCIe signal switch solutions
Enables configurations that are
not practical using PCIe signal
switches
Provides greater flexibility than
movable upstream port or
upstream port failover
CPU
CPU
CPU
CPU
CPU
CPU
“Northbridge” /
I/O Controller
“Northbridge” /
I/O Controller
“Northbridge” /
I/O Controller
89HPES32NT24G2
64-lane, 16-port PCIe Gen2
System Interconnect Switch
89HPES32NT24G2
64-lane, 16-port PCIe Gen2
System Interconnect Switch
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/ O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
I/O
4 Socket Server—Base Design
2 Socket Server—Reduced Cost Design
Figure 8 Example of Flexible Slot Mapping
Advanced Failover
Port failover in high availability
systems
Table 1 Switch Partitioning Applications and Benefits
Replacing Multiple Discrete Switches
Multi-root systems with high availability requirement can take advan-
tage of dynamic switch partitioning by re-allocating downstream ports to
a standby/secondary root upon failure (Figure 9). The device provides a
built-in automatic failover mechanism by specifying failover configuration
registers. Failover can be initiated by software, external signal pins, or
by a watchdog timer.
When switch partitioning is configured with multiple independent PCI
Express domains, it can replace multiple discrete PCI Express switches,
providing savings in cost, power, and board space.
5 of 7
September 15, 2009
TI Technology Forum
京公网安备 11010802033920号
EEWORLD
