FCoE Multi-hop : Why wait?

I saw an interesting discussion within the vSpecialist group here at EMC. The question was when the FCoE protocol would be ratified for multi-hop. I thought I would try and clarify what the obstacles are and why it is very important to wait.

You may or may not know about the Data Center Bridging (DCB) task group at the IEEE. They are working hard on making FCoE a wide reaching standard. A good post to read first (or after) is this one by Scott Lowe.

But I think what the DCB group is working on needs to be explained a bit. Believe it or not, the DCB is not about the fibre-channel frames themselves. It is about the Ethernet layer that encapsulates FC frames becoming robust enough to reliably deliver FC frame in multi-hop topologies.

So here is a quick and simple lesson to get you caught up. This is how the FCP layers are built:

What you want to note above is that all the magic begins at FC-2. Framing, flow control, and classes of service as the beginning of logical management of the FC frame.

Compare that to the OSI standard for a standard TCP/IP frame (in this case Ethernet at Layer 2):

As you can see the magic begins at Layer 2 within the OSI model. Ethernet lives at Layer 2 and is the beginning of addressing, collision detection, and flow control.

Now this is how an FCoE frame is built:

As you can see a couple things have happened here. First, we have replaced FC-0 and FC-1 with the Data Link and Physical layers from OSI (specifically Ethernet). Secondly, we have inserted a new FCoE layer (blue layer) for mapping between Ethernet and FC. Take a second to go back and look over these till you get the gist.

So that is all well and good. But, why does this mean we cannot do multi-hop? It is actually pretty simple: We can, we just do not want to do it yet.

Let me visualize it for you. I want you to picture a FedEx Express truck. It has a simple job. It is given packages (frames) and it delivers them to addresses (FC address). Now, the FedEx Company prides itself on reliable delivery. It has all kinds of processes and methods(Flow Control, Classes of service) for ensuring that the truck reaches the address and delivers the packages on time. These methods have been finely tuned specifically for this job.

The FedEx truck is our fibre-channel protocol (FCP). It is specifically designed to have several classes of service with advanced flow control mechanisms like buffer-to-buffer and end-to-end credit.

Now let’s take a semi-truck. A semi can carry multiple packages and can hook up to loading docks almost everywhere. It is the most common means of transporting cargo. So say we take our FedEx truck and load it into the back of a semi-truck. Now we can fit multiple FedEx trucks in and carry more data. We can continue using the semi-truck for other packages like before but now have a single transport vehicle.

The semi-truck is the Ethernet standard. It is simple, effective, and used everywhere. It can carry more data (9000 bytes w/ jumbo frames) and has a ton of physical media standards.

So the picture here is that the FC frames are fit into the Ethernet semi-truck. This is why the DCB standards are so import. Let me explain it a little more.

In our analogy, when a standard FCP frame has to get somewhere, what is the first vehicle to be given an address to deliver to? The FedEx truck (FCP).

But, in FCoE what is the first vehicle to be given an address? Ahh, the semi-truck (Ethernet frame), right? The FedEx truck inside still has the FC address to deliver to, but first the semi-truck has to deliver to the MAC address (Ethernet addressing).

The semi-truck has to reach its destination, unload our FedEx truck and usher him on his way (FCoE layer above handles the loading/unloading).

So if our FedEx truck has all these mechanisms to make sure that its packages arrive on-time and in order; what happens when the semi-truck is late? Did you have the “AH-HA!” moment yet?

The Ethernet standard is not built to deliver as effectively as FCP. And since we are trusting Ethernet to ship our FC frames around in FCoE we need to overhaul our semi-truck and add similar methods and mechanisms that our FedEx truck uses.

This is where DCB comes in. The IEEE is currently working on several additions to the Ethernet standard to improve in areas it is lacking with FC delivery in mind. Here is a brief overview of what they are working on:

  • Congestion Notification (CN) provides end to end congestion management for protocols that do not already have congestion control mechanisms built in; e.g. Fibre Channel over Ethernet (FCoE). It is also expected to benefit protocols such as TCP that do have native congestion management as it reacts to congestion in a more timely manner.
  • Priority-based Flow Control (PFC) provides a link level flow control mechanism that can be controlled for independently for each priority. The goal of this mechanism is to ensure zero loss due to congestion in DCB networks.
  • Enhanced Transmission Selection (ETS) provides a common management framework for assignment of bandwidth to traffic classes.
  • A discovery and capability exchange protocol that is used for conveying capabilities and configuration of the above features between neighbors to ensure consistent configuration across the network. This protocol is expected to leverage functionality provided by 802.1AB (LLDP).

All of these additions will enable FCoE to be used reliably as a multi-hop protocol. So it is not that it will not work. It is that we want to make sure it works well. Or in other words, shipping a package and making sure it gets there are not the same thing.

And to really open your mind just think about how all these new components of Ethernet will affect existing layers above in the OSI standard. With better flow control and class of service/shaping what happens to the strong reliance on TCP or DSCP in present use? I don’t know but expect a shift in the next few years. DCB will create a low-level robust layer that will change the way we view data networks.

I hope this helps and I apologize to all IEEE guys/girls for any butchering of your standards.

.nick

I saw an interesting discussion within the vSpecialist group here at EMC. The question was when the FCoE protocol would be ratified for multi-hop. I thought I would try and clarify what the obstacles are and why it is very important to wait.
You may or may not know about the Data Center Bridging (DCB) task group at the IEEE. They are working hard on making FCoE a wide reaching standard. A good post to read first (or after) is this one by Scott Lowe.
But I think what the DCB group is working on needs to be explained a bit. Believe it or not, the DCB is not about the fibre-channel frames themselves. It is about the Ethernet layer that encapsulates FC frames becoming robust enough to reliably deliver FC frame in multi-hop topologies.
So here is a quick and simple lesson to get you caught up. This is how the layers are built:

 

What you want to note above is that all the magic begins at FC-2. Framing, flow control, and classes of service as the beginning of logical management of the FC frame.
Compare that to the OSI standard for a standard TCP/IP frame (in this case Ethernet at Layer 2):

As you can see the magic begins at Layer 2 within the OSI model. Ethernet lives at Layer 2 and is the beginning of addressing, collision detection, and flow control.
Now this is how an FCoE frame is built:

As you can see a couple things have happened here. First, we have replaced FC-0 and FC-1 with the Data Link and Physical layers from OSI (specifically Ethernet). Secondly, we have inserted a new FCoE layer for mapping between Ethernet and FC. Take a second to go back and look over these till you get the gist.
So that is all well and good. But, why does this mean we cannot do multi-hop? It is actually pretty simple: We can, we just do not want to do it yet.
Let me visualize it for you. I want you to picture a FedEx Express truck. It has a simple job. It is given packages (frames) and it delivers them to addresses (FC address). Now, the FedEx Company prides itself on reliable delivery. It has all kinds of processes and methods for ensuring that the truck reaches the address (Flow Control, Classes of service). These methods have been finely tuned specifically for this job.

The FedEx truck is our fibre-channel protocol (FCP). It is specifically designed to have several classes of service with advanced flow control mechanisms like buffer-to-buffer and end-to-end credit.
Now let’s take a semi-truck. A semi can carry multiple packages and can hook up to loading docks almost everywhere. It is the most common means of transporting cargo. So say we take our FedEx truck and load it into the back of a semi-truck. Now we can fit multiple FedEx trucks in and carry more data. We can continue using the semi-truck for other packages like before but now have a single transport vehicle.

The semi-truck is the Ethernet standard. It is simple, effective, and used everywhere. It can carry more data (9000 bytes w/ jumbo frames) and has a ton of physical media standards.
So the picture here is that the FC frames are fit into the Ethernet semi-truck. This is why the DCB standards are so import. Let me explain it a little more.
In our analogy, when a standard FCP frame has to get somewhere, what is the first vehicle to be given an address to deliver to? The FedEx truck (FCP).
But, in FCoE what is the first vehicle to be given an address? Ahh, the semi-truck (Ethernet frame), right? The FedEx truck inside still has the FC address to deliver to, but first the semi-truck has to deliver to the MAC address (Ethernet addressing).
The semi-truck has to reach its destination, unload our FedEx truck and usher him on his way (FCoE layer above handles the loading/unloading).
So if our FedEx truck has all these mechanisms to make sure that its packages arrive on-time and in order; what happens when the semi-truck is late? Did you have the “AH-HA!” moment yet?
The Ethernet standard is not built to deliver as effectively as FCP. And since we are trusting Ethernet to ship our FC frames around in FCoE we need to overhaul our semi-truck and add similar methods and mechanisms that our FedEx truck uses.
This is where DCB comes in. The IEEE is currently working on several additions to the Ethernet standard to improve in areas it is lacking with FC delivery in mind. Here is a brief overview of what they are working on:
·    Congestion Notification (CN) provides end to end congestion management for protocols that do not already have congestion control mechanisms built in; e.g. Fibre Channel over Ethernet (FCoE). It is also expected to benefit protocols such as TCP that do have native congestion management as it reacts to congestion in a more timely manner.
·    Priority-based Flow Control (PFC) provides a link level flow control mechanism that can be controlled for independently for each priority. The goal of this mechanism is to ensure zero loss due to congestion in DCB networks.
·    Enhanced Transmission Selection (ETS) provides a common management framework for assignment of bandwidth to traffic classes.
·    A discovery and capability exchange protocol that is used for conveying capabilities and configuration of the above features between neighbors to ensure consistent configuration across the network. This protocol is expected to leverage functionality provided by 802.1AB (LLDP).
All of these additions will enable FCoE to be used reliably as a multi-hop protocol. So it is not that it will not work. It is that we want to make sure it works well. Or in other words, shipping a package and making sure it gets there are not the same thing.
And to really open your mind just think about how all these new components of Ethernet will affect existing layers above in the OSI standard. With better flow control and class of service/shaping what happens to the strong reliance on TCP or DSCP in present use? I don’t know but expect a shift in the next few years. DCB will create a low-level robust layer that will change the way we view data networks.
I hope this helps and I apologize to all IEEE guys/girls for any butchering of your standards.

.nick

 

Commentary

7 Comments Leave a comment

  1. DCB task group at IEEE creates a stronger/better/more reliable than ever TCP/IP protocol to allow convergence and FCoE… We can onkly benefit from this new technology since TCP/IP has been left out with no improvment for the last 10 years (exception with IPv6 but it hasn’t take off yet…)
    Rgds,
    Didier

    • Yup. Good news all around.

      Make sure everyone knows that TCP/IP is shorthand for the IP protocol suite (multiple protocols within). The changes outside of Ethernet will likely soon follow the expansion of Ethernet itself.

%d bloggers like this: