A typical data center has two separate networks: one for Ethernet and one for Fibre Channel storage. These networks are physically and logically separated from each other, as shown on the left in Figure 9-1. Here, a SAN leverages dual fabric with multipathing failover initiated by the client, and a LAN leverages single fully meshed fabric with higher levels of component redundancy. Both networks have a redundant pair of switches at each layer, which results in increased CAPEX and OPEX as well as reduced business functionality.
The consolidation of I/O traffic in the data center brings the Fibre Channel and Ethernet networks into a single integrated infrastructure, as shown on the right in Figure 9-1. This results in decreased CAPEX and OPEX. An access switch in the consolidated topology is Fibre Channel over Ethernet Forwarder (FCF). Dual fabrics are still deployed for redundancy.
Figure 9-1 Traditional LAN and SAN Infrastructure (Left); Converged LAN and SAN Infrastructure (Right)
Servers used to have three types of NICs: Ethernet NICs for LAN traffic, management traffic, backup traffic, and so on; host channel adapter (HCA) for Inter-Process Communication (IPC) traffic; and HBAs for Fibre Channel/storage traffic, as shown in Figure 9-2. With the development of FCoE, converged network adapters (CNAs) were introduced; they can handle all the previous three types of traffic over single high-speed 10G links, resulting in a lesser number of interfaces for servers.
Figure 9-2 Converged Network Adapter
Fibre Channel over Ethernet (FCoE) allows Fibre Channel traffic to be encapsulated over a physical Ethernet link. FCoE frames use a unique Ethertype so that FCoE traffic and standard Ethernet traffic can be carried on the same link.
Classic Ethernet is a best-effort protocol; in the event of congestion, Ethernet will discard packets, relying on higher-level protocols to provide retransmission and other reliability mechanisms. Fibre Channel traffic requires a lossless transport layer; because it is a data storage protocol, it is unacceptable to lose a single data packet. Native Fibre Channel implements a lossless service at the transport layer using a buffer-to-buffer credit system.
For FCoE traffic, the Ethernet link must provide a lossless service. Ethernet links on Cisco Nexus devices provide two mechanisms to ensure lossless transport for FCoE traffic: link-level flow control (LLFC) and priority-based flow control (PFC). For FCoE, PFC is recommended.
IEEE 802.3x link-level flow control allows a congested receiver to signal the far end to pause the data transmission for a short period of time. The pause functionality is applied to all the traffic on the link.
The priority flow control feature applies pause functionality to specific classes of traffic on the Ethernet link. For example, PFC can provide lossless service for the FCoE traffic and best-effort service for the standard Ethernet traffic. PFC can provide different levels of service to specific classes of Ethernet traffic. We discuss these enhancements to Ethernet protocol in detail in the next section.