Learn Everything You Need to Know About PCI Express System Architecture

# PCI Express System Architecture: A Guide for Beginners - ## Introduction - What is PCI Express (PCIe) and why it is important - How PCIe works and what are its advantages over other interconnects - What are the main components and features of PCIe architecture - ## PCIe Specifications and Standards - How PCIe is managed by the PCI-SIG and what are the different revisions and generations of PCIe - How PCIe supports various protocols and devices such as SATA, USB, DisplayPort, CXL, etc. - How PCIe implements security and encryption features - ## PCIe Link and Lane Configuration - How PCIe uses point-to-point links composed of one to 32 lanes - How PCIe negotiates and configures the link speed and width on power up - How PCIe scales performance by grouping lanes together - ## PCIe Data Transfer and Flow Control - How PCIe uses packets to transfer data between endpoints - How PCIe uses credit-based flow control to avoid congestion and ensure reliability - How PCIe supports both host-directed and peer-to-peer transfers - ## PCIe Power Management and Quality of Service - How PCIe reduces power consumption by using various states and modes - How PCIe maintains quality of service by using virtual channels and traffic classes - How PCIe handles errors and recovery mechanisms - ## PCIe Applications and Use Cases - How PCIe is widely used in various computing and embedded devices such as servers, desktops, laptops, chipsets, etc. - How PCIe enables high-performance graphics, storage, networking, and acceleration solutions - How PCIe is ideal for cloud computing and data center applications - ## Conclusion - Summarize the main points and benefits of PCIe architecture - Provide some resources for further learning and development with PCIe - Encourage the reader to explore more about PCIe Now that I have the outline ready, I will start writing the article based on it. Here is the article with HTML formatting: # PCI Express System Architecture: A Guide for Beginners ## Introduction If you are interested in learning about the latest technology trends in computing and embedded devices, you might have heard of PCI Express (PCIe). But what exactly is PCIe and why is it so important? In this article, we will explain what PCIe is, how it works, and what are its advantages over other interconnects. We will also introduce you to the main components and features of PCIe architecture. PCIe is a standards-based, point-to-point, serial interconnect that is used throughout the computing and embedded devices industries. It was introduced in 2004 as a successor to the parallel PCI bus, which had reached its performance limits. PCIe is managed by the PCI-SIG (PCI Special Interest Group), a consortium of over 800 companies that develop and maintain the specifications and standards for PCIe. PCIe is capable of delivering scalable, simultaneous, bi-directional transfers using one to 32 lanes of differential-pair interconnects. Each lane can operate at different speeds depending on the generation of PCIe. For example, PCIe 3.0 can support up to 8 GT/s (gigatransfers per second) per lane, while PCIe 5.0 can support up to 32 GT/s per lane. By grouping lanes together, such as x1, x2, x4, x8, x16, etc., PCIe can achieve high transfer rates for various applications. One of the main advantages of PCIe over other interconnects is that it maintains backward compatibility with previous generations. This means that a device designed for PCIe 3.0 can still work with a system that supports only PCIe 2.0 or even PCI. Another advantage of PCIe is that it supports various protocols and devices such as SATA (Serial ATA), USB (Universal Serial Bus), DisplayPort, CXL (Compute Express Link), etc., making it a versatile and flexible interconnect. The main components of PCIe architecture are: - Endpoints: These are devices that initiate or terminate data transfers on the PCIe bus. They can be either root complexes or endpoints. - Root Complex: This is a device that connects the CPU (central processing unit) and the memory subsystem to the PCIe bus. It acts as the host and manages the configuration and enumeration of the endpoints. - Endpoint: This is a device that connects to the root complex or another endpoint via a PCIe link. It can be either a native PCIe device or a legacy PCI device with a bridge. - Switch: This is a device that connects multiple endpoints or other switches together. It acts as a router and forwards packets between different PCIe links. - Link: This is a point-to-point connection between two devices on the PCIe bus. It consists of one to 32 lanes of differential-pair interconnects. - Lane: This is a pair of wires that carries data in one direction. Each lane can operate at different speeds depending on the generation of PCIe. The main features of PCIe architecture are: - Packet-based data transfer: PCIe uses packets to transfer data between endpoints. Each packet consists of a header, a payload, and an optional cyclic redundancy check (CRC). The header contains information such as the source and destination address, the type and length of the payload, and the sequence number. The payload contains the actual data to be transferred. The CRC is used to detect and correct errors during transmission. - Credit-based flow control: PCIe uses credit-based flow control to avoid congestion and ensure reliability on the PCIe bus. Each device maintains a credit counter for each link it is connected to, indicating how many packets it can send or receive without overflowing its buffer. Before sending a packet, a device checks if it has enough credits for the destination link. If not, it waits until it receives more credits from the destination device. Similarly, before receiving a packet, a device checks if it has enough credits for the source link. If not, it sends back credits to the source device. - Host-directed and peer-to-peer transfers: PCIe supports both host-directed and peer-to-peer transfers. Host-directed transfers are those that involve the root complex as either the source or the destination of the data. For example, when a CPU reads or writes data from or to an endpoint, it is a host-directed transfer. Peer-to-peer transfers are those that involve only endpoints as the source and destination of the data. For example, when a graphics card sends data to a sound card, it is a peer-to-peer transfer. ## PCIe Specifications and Standards PCIe is managed by the PCI-SIG, which develops and maintains the specifications and standards for PCIe. The PCI-SIG also provides various resources and tools for developers working with PCIe designs, such as compliance testing, interoperability workshops, developer conferences, etc. The PCI-SIG has released several revisions and generations of PCIe specifications over the years, each introducing new features and improvements to the previous ones. The current revision is 5.0, which was released in 2019. The current generation is 6.0, which was announced in 2021 and is expected to be released in 2023. The main differences between the revisions and generations of PCIe are: - Revision: This refers to the changes in the protocol and architectural specifications of PCIe. Each revision adds new features and capabilities to PCIe, such as security enhancements, power management improvements, quality of service enhancements, etc. - Generation: This refers to the changes in the physical layer specifications of PCIe. Each generation increases the speed and bandwidth of PCIe lanes, such as from 2.5 GT/s in Gen 1 to 32 GT/s in Gen 5. The following table summarizes the main characteristics of each generation of PCIe: Generation Year Speed (per lane) Bandwidth (per lane) Bandwidth (x16 link) --------------------------------------------------------------------------------- Gen 1 2003 2.5 GT/s 250 MB/s 4 GB/s Gen 2 2007 5 GT/s 500 MB/s 8 GB/s Gen 3 2010 8 GT/s 984 MB/s 15.75 GB/s Gen 4 2017 16 GT/s 1.969 GB/s 31.51 GB/s Gen 5 2019 32 GT/s 3.938 GB/s 63.02 GB/s Gen 6 TBA 64 GT/s 7.877 GB/s 126.04 GB/s As mentioned earlier, PCIe supports various protocols and devices such as SATA, USB, DisplayPort, CXL, etc., making it a versatile and flexible interconnect. Some of these protocols and devices are defined by separate specifications that are compatible with PCIe specifications. For example: - The PHY Interface for the PCI Express* (PIPE) Architecture Revision 6.2.1 is an updated version of the PIPE spec that supports PCIe, SATA, USB3.2, DisplayPort, and USB4 Architectures. It defines a common PHY interface for these protocols to communicate with the link layer of PCIe. - The Logical PHY Interface Specification, Revision 1.1 defines the interface between the link layer and the logical physical layer for PCIe and CXL architectures. It enables the use of different physical media and encoding schemes for PCIe and CXL links. - The CXL Specification 2.0 defines a new protocol that enables high-speed, low-latency, cache-coherent interconnect between the host processor and memory expansion devices such as accelerators, memory buffers, and smart NICs (network interface cards). It leverages the PCIe physical and electrical interface and extends the PCIe protocol stack with new layers for cache coherency and memory semantics. PCIe also implements security and encryption features to protect the data and devices on the PCIe bus. Some of these features are: - The Security Protocol and Data Model (SPDM) Specification defines a standard for establishing a secure connection between a host and a device using mutual authentication, measurement, and key exchange protocols. It also defines a data model for representing and exchanging security-related information such as certificates, policies, capabilities, etc. - The Component Measurement and Authentication (CMA) ECN defines a mechanism for measuring and verifying the integrity of a device's firmware and configuration before allowing it to operate on the PCIe bus. It uses SPDM messages to exchange measurement data between the host and the device. - The Data Object Exchange (DOE) ECN defines a mechanism for exchanging arbitrary data objects between the host and the device using SPDM messages. It can be used for various purposes such as firmware update, configuration management, diagnostics, etc. - The Integrity and Data Encryption (IDE) ECN defines a mechanism for encrypting and authenticating the data transferred on the PCIe bus using AES-GCM (Advanced Encryption Standard - Galois/Counter Mode) algorithm. It also defines a key management protocol for establishing and updating encryption keys between the host and the device. ## PCIe Link and Lane Configuration PCIe uses point-to-point links composed of one to 32 lanes to connect devices on the PCIe bus. Each lane is a pair of wires that carries data in one direction. Each lane can operate at different speeds depending on the generation of PCIe. The link speed is determined by the frequency of the clock signal embedded in the data stream. The link width is determined by the number of lanes used by the link. The link bandwidth is determined by multiplying the link speed by the link width. For example, a PCIe 3.0 x4 link has four lanes operating at 8 GT/s each, resulting in a bandwidth of 32 GT/s or 3.938 GB/s per direction. The link speed and width are negotiated and configured on power up by a process called link training. Link training involves three main steps: - Link Initialization: This step establishes electrical connectivity between the devices on each end of the link. It also determines the maximum link width supported by both devices. - Link Equalization: This step optimizes the signal quality of each lane by adjusting parameters such as voltage swing, pre-emphasis, etc., based on feedback from both devices. - Link Configuration: This step determines the final link speed and width based on the capabilities and preferences of both devices. It also verifies that the link is functional and error-free. PCIe scales performance by grouping lanes together to form wider links for various applications. For example, most graphics adapters use at least 16 lanes (x16) to connect to the root complex, while most storage devices use four lanes (x4) or less. The number of lanes used by a device is determined by its physical connector type and its configuration space register settings. The physical connector type defines the maximum number of lanes that can be physically connected to the device. The configuration space register settings define how many lanes are actually enabled and used by the device. The following table summarizes some of the common physical connector types and their maximum lane counts: Connector Type Maximum Lane Count ------------------------------------ x1 1 x4 4 x8 8 x16 16 M.2 4 U.2 4 OCuLink 4 SFF-8639 4 ## PCIe Data Transfer and Flow Control PCIe uses packets to transfer data between endpoints on the PCIe bus. Each packet consists of a header, a payload, and an optional cyclic redundancy check (CRC). The header contains information such as the source and destination address, the type and length of the payload, and the sequence number. The payload contains the actual data to be transferred. The CRC is used to detect and correct errors during transmission. There are two main types of packets in PCIe: - Request Packets: These are packets that initiate a data transfer from the source device to the destination device. They can be either read requests or write requests. Read requests ask for data from the destination device, while write requests send data to the destination device. - Completion Packets: These are packets that terminate a data transfer from the source device to the destination device. They can be either read completions or write completions. Read completions return data from the destination device to the source device, while write completions acknowledge that the data has been received by the destination device. PCIe uses credit-based flow control to avoid congestion and ensure reliability on the PCIe bus. Each device maintains a credit counter for each link it is connected to, indicating how many packets it can send or receive without overflowing its buffer. Before sending a packet, a device checks if it has enough credits for the destination link. If not, it waits until it receives more credits from the destination device. Similarly, before receiving a packet, a device checks if it has enough credits for the source link. If not, it sends back credits to the source device. There are two types of credits in PCIe: - Data Credits: These are credits that indicate how many bytes of payload data can be sent or received on a link. Each data credit corresponds to one byte of payload data. - Header Credits: These are credits that indicate how many packets can be sent or received on a link. Each header credit corresponds to one packet header. PCIe supports both host-directed and peer-to-peer transfers. Host-directed transfers are those that involve the root complex as either the source or the destination of the data. For example, when a CPU reads or writes data from or to an endpoint, it is a host-directed transfer. Peer-to-peer transfers are those that involve only endpoints as the source and destination of the data. For example, when a graphics card sends data to a sound card, it is a peer-to-peer transfer. ## PCIe Power Management and Quality of Service PCIe reduces power consumption by using various states and modes for different components of the PCIe architecture. Some of these states and modes are: - Link States: These are states that indicate the power consumption and performance level of a PCIe link. There are four link states: L0 (active), L0s (standby), L1 (idle), and L2/L3 (offline). L0 is the highest power state with full performance, while L2/L3 is the lowest power state with no performance. - Device States: These are states that indicate the power consumption and performance level of a PCIe device. There are six device states: D0 (fully active), D1-D3hot (partial power saving), D3cold (full power saving), and D3uninitialized (uninitialized). D0 is the highest power state with full performance, while D3uninitialized is the lowest power state with no performance. - Function-Level Reset (FLR): This is a mode that allows a PCIe function (a logical unit within a device) to reset itself without affecting other functions or devices on the PCIe bus. FLR can be used for error recovery, firmware update, configuration change, etc. - Active State Power Management (ASPM): This is a mode that allows a PCIe link to dynamically adjust its power state based on the traffic demand. ASPM can switch between L0 and L0s/L1 states to save power when there is no or low traffic on the link. - Latency Tolerance Reporting (LTR): This is a mechanism that allows a PCIe device to report its latency tolerance to the root complex or switch. LTR can help optimize ASPM by indicating how long a device can tolerate being in a low-power state without affecting its performance. PCIe maintains quality of service by using virtual channels and traffic classes for different types of traffic on the PCIe bus. Some of these virtual channels and traffic classes are: - Virtual Channels: These are logical subdivisions of a physical link that can have different bandwidth and priority allocations. Virtual channels can be used to isolate and prioritize different types of traffic on the same link. - Traffic Classes: These are categories of traffic that have different characteristics and requirements such as latency, bandwidth, reliability, etc. Traffic classes can be used to identify and classify different types of traffic on the same virtual channel. - Virtual Channel Arbitration: This is a mechanism that determines which virtual channel gets access to the physical link at any given time. Virtual channel arbitration can use different algorithms such as weighted round-robin, strict priority, etc., to allocate the link bandwidth among different virtual channels. - Traffic Class Arbitration: This is a mechanism that determines which traffic class gets access to a virtual channel at any given time. Traffic class arbitration can use different algorithms such as weighted round-robin, strict priority, etc., to allocate the virtual channel bandwidth among different traffic classes. PCIe handles errors and recovery mechanisms by using various features and mechanisms for different components of the PCIe architecture. Some of these features and mechanisms are: - Error Detection: PCIe uses various methods to detect errors at different layers of the PCIe protocol stack. Some of these methods are: - CRC: PCIe uses cyclic redundancy check (CRC) to detect errors in the packet header and payload. Each packet has a CRC field that contains a checksum calculated from the packet data. The receiver recalculates the checksum from the received data and compares it with the CRC field. If they do not match, an error is detected. - Sequence Number: PCIe uses sequence numbers to detect errors in the packet ordering and delivery. Each packet has a sequence number field that contains a counter value incremented by the sender for each packet sent on a link. The receiver checks the sequence number of each received packet and expects it to be one more than the previous packet. If not,