PCIe is without doubt one of the most latency-sensitive types of serial communication as a result of its address-based semantics imply that processor threads are sometimes ready for the outcomes of a transaction. The arrival of PCIe 4.0, and particularly of PCI 5.0, have pushed the necessity to use retimers in lots of longer-reach PCIe functions.
This text explores the overall latency surroundings for PCIe at six totally different layers and discusses concepts for learn how to optimize every of these layers, together with with retimers.
Latency addition by layer
Utility latency arises from many various sources. The next chart describes six totally different layers, the standard latency added at every layer, the supply of the latency in every layer and strategies that can be utilized to attenuate the latency skilled in every layer. Latency ought to be optimized from the highest of the desk to the underside.
Determine 1 Latency impacts PCIe utility efficiency at six totally different layers. Supply: Kandou
On the high of the desk, there are alternatives to save lots of milliseconds to seconds. In the midst of the desk, it’s attainable to save lots of microseconds to milliseconds. On the backside of the desk, it’s attainable to save lots of tens of nanoseconds. That stated, each ingredient of latency contributes to the general system efficiency, so alternatives for enchancment ought to be taken at every degree the place possible and financial.
Retry and restoration latency
A big alternative for enchancment in lots of techniques is discovered on the knowledge hyperlink layer. It’s because the retransmit “retry” mechanism could be triggered comparatively typically, even in regular operation. With a bit error ratio (BER) on 1E-12 on a 16-lane x 32 Gbps/lane hyperlink (PCIe 5.0), a retransmit will happen about each two seconds.
The retransmitted site visitors is handled as 5th precedence per the PCI-SIG Implementation Guideline Part 3.6.2.1. A serious latency hit is skilled at any time when this occurs. This latency supply additionally degrades the latency consistency of the system and may trigger an utility to have a stall or stutter in its efficiency.
A extra critical latency hit is taken when the hyperlink coaching and standing state machine (LTSSM) is compelled to enter the restoration state on account of poor sign integrity, maybe at a nook situation comparable to excessive or altering working temperature. The operation of the restoration state machine can take a whole bunch of milliseconds or extra to discover a new setting for the transmit equalizers. Within the worst case, PCIe hyperlinks could be compelled to run at a decrease velocity, downshifting from say 32 GT/s to 16 GT/s if the attention just isn’t open sufficient, maybe on account of excessive or altering working temperature.
Utilizing retimers to enhance BER
A technique of lowering latency by avoiding retry and restoration occasions is to make use of retimer gadgets within the excessive BER paths, sometimes the longest paths within the system. Retimers enhance each the attention top (EH) and the attention width (EW) seen by the subsequent receiver within the path. A retimer recovers a clear digital copy of the sign, generates a clear transmit clock, and sends out a buffered copy of the info.
Determine 2 Retimers enhance each the EH and EW in lots of conditions. Supply: Kandou
Retimers enhance the EW by enhancing all sources of jitter. That features knowledge dependent jitter (DDJ) typically on account of inter-symbol interference (ISI), random jitter (RJ) typically on account of thermal noise and clock jitter, and bounded uncorrected jitter (BUJ) typically on account of crosstalk. Retimers “reset” all these jitter sources.
PCIe clock modes
An necessary subject to introduce earlier than discussing retimer latency: the three PCIe clock modes. These are separate reference clock with impartial spread-spectrum (SRIS), separate reference clocks with no spread-spectrum clocking (SRNS), and customary clock. Retimers should generate a brand new transmit clock, so the selection between these modes makes a big distinction within the added latency. The next diagram illustrates the frequent clock mode on the high and the SRIS mode on the backside.
Determine 3 The highest exhibits frequent clock mode, the underside exhibits SRIS mode. Supply: Kandou
PCIe clock modes and retimers
In retimer functions, it’s greatest to make use of the frequent clock mode. On this mode, the CPU, retimer and end-point all share the identical reference clock. All three factors have the identical components per million (PPM) offset for his or her clock and have the identical spread-spectrum profile. The retimer elastic buffer could be set dramatically smaller.
The SRIS and SRNS clock modes can be utilized with retimers. That stated, with these modes, the CPU and retimer are in a single clock area and the end-point is in one other. The retimer then has the job of accounting for this clocking distinction for its ingress site visitors from the end-point. It does this by being conscious of the packet boundaries and adjusting the hole between the packets.
The retimer should account for +/- 300 PPM of clock fee offset plus -5000 PPM for the unfold spectrum clock distinction in SRIS mode. The retimer elastic buffer should be configured to account for this distinction. The latency addition will get larger for bigger packet sizes. One alternative is that if the CPU helps it, its ingress could be set to frequent clock mode though there are totally different clocks because the retimer has already finished the buffering required by the SRIS or SRNS modes.
Lane-to-lane skew matching
Within the case the place the system has not accounted for the extra lane-to-lane skew attributable to the retimer in its price range, the retimer should add extra latency to reset that skew. The PCIe specification limits the Tx skew to be 1.5 ns and the Rx skew to be 5 ns.
Retimer structure
The elastic buffer types the central integrating retailer inside a retimer. It permits the gadget to recreate the transmit clocks whereas not shedding the consumer’s knowledge packet info. The next diagram exhibits the structure of a typical PCIe retimer.
Determine 4 A typical PCIe retimer structure consists of an elastic buffer because the central integrating retailer. Supply: Kandou
The elastic buffer is the place the place the latency from a retimer can add up and the place the wildly totally different clocks present in SRIS clocking mode should be tailored in between. It’s also right here the place typically the lane-to-lane skew should be cleaned up.
PCIe retimer latency specification
The PCIe specification units limits for retimer-added latency. In non-SRIS clocking modes, this restrict is 64 ns for the info charges from 5GT/s to 32 GT/s and 128 ns for two.5 GT/s. In SRIS modes, the restrict additionally is determined by the utmost packet dimension and a big desk of the bounds is offered within the specification. A complicated retimer structure permits a system to satisfy these specs in all circumstances and to considerably beat them in sure circumstances.
Bypass buffer utilization
It’s attainable to make use of a small, low-latency elastic buffer, typically known as a bypass buffer, if all 4 situations are met. This bypass buffer can have a latency on the order of 10 ns and is usually applied as a range-restricted area of the bigger elastic buffer.
These 4 situations are:
- The system operates in frequent clock mode. Observe that it’s nonetheless attainable to make use of spread-spectrum clocks in frequent clock mode.
- The retimer is free of having to reset the lane-to-lane skew as a result of the system designer has fastidiously laid out the PCB to help the end-to-end price range for lane-to-lane skew, together with the skew launched by the retimer.
- The retimer LTSSM continues to be ready to reply to TS1/TS2 instructions; as an example, various protocol or vendor-defined instructions.
- The speed adaptation is disabled.
The next diagram illustrates a bypass buffer in operation.
Determine 5 It’s attainable to make use of a low-latency bypass buffer if sure situations are met. Supply: Kandou
Retimer’s position in latency optimization
Utility latency ought to be optimized from the highest layer all the way down to the underside layer, and each ingredient issues to efficiency. For workloads that require ultra-low latency, another protocol comparable to CXL ought to be thought-about when possible.
A great place to enhance latency and enhance latency consistency is so as to add retimers to the longest hyperlinks with the worst BERs. Retimers can enhance each the EH and the EW. Sign integrity margins ought to be validated on the working corners, particularly excessive and altering temperature. The usage of retimers can enhance latency by maintaining techniques out of retry occasions, which could be routine at excessive charges. The usage of retimers also can assist keep away from expensive restoration occasions.
Retimers can be utilized in a latency-optimized bypass buffer mode by way of frequent clock mode, accounting for lane-to-lane skew on the system degree, sustaining TS1/TS2 communication, and disabling fee adaption.
Editor’s Observe: This text is predicated on a presentation by Jay Li of Kandou through the 2022 PCI-SIG Improvement Convention.
Jay Li is product advertising director at Kandou.
Brian Holden is answerable for the requirements technique at Kandou.
Associated Content material
