Tuning Buffer Credits on NetApp MetroCluster Fabric Switches

When configuring your NetApp MetroCluster Fabric switches there are two ways to calculate the necessary amount of Buffer Credits on the ISL's:
  • The NetApp and Brocade best practices state the "desired distance" of a ISL port should be set to 1.5 times the actual distance. This will then set the buffer credits to a certain amount.
  • This article from NetApp also describes a second method: calculating the average frame rate to determine the needed amount of buffer credits.
Depending on the average frame size of your ISL traffic the difference between these two methods can be significant.

Use the following  to  calculate if your buffer credit settings might be a problem:

portstats64show <port> - This gives the amount of frames send (stat64_ftx) and amount of BB_credit zero occurrences (tim64_txcrd_z).
stat64_ftx      3           top_int : Frames transmitted
                970578587   bottom_int : Frames transmitted

tim64_txcrd_z   0           top_int : Time BB_credit zero
                46631371    bottom_int : Time BB_credit zero

The total amount of frames sent is (2^32)*<top_int>+<bottom_int>.
The total amount of BB_credit zero is (2^32)*<top_int>+<bottom_int>.
The percentage of BB_credit zero is (<BB_credit zero>/<frames sent>)*100.

According to Brocade some BB_credit zero is allowed, but no more than 15% (which still seems high).  

Use the following to calculate the average frame size:

portstats64show <port> - This gives us the amount of words sent (stat64_wtx), which translates to bytes send.
stat64_wtx      1485        top_int : 4-byte words transmitted
                4108584283  bottom_int : 4-byte words transmitted

total amount of bytes sent = ((2^32)*<top_int>+<bottom_int>)*4. average frame size = <bytes sent>/<frames sent>.

Use the following to calculate the desired distance which is used by the Brocade switch to set the buffer credits:

desired distance = (real distance*2112) / average frame size.

Set this distance on the Brocade switches using:

portCfgLongDistance <port> LS 1 <desired distance>  

To see how the buffer credits are configured for each port and how many your switch still has available: portbuffershow

If you want you can calculate how the switch will translate the desired distance to buffer credits:

buffer credits = <desired distance>*(<link speed>/2) + 6

If QoS is enabled on the port it will allocate another 14 buffer credits. By default QoS is enabled, but no QoS zones are configured. In effect this does nothing except consume an extra 14 buffer credits per port. Unless you are going to really use QoS by configuring QoS zones it is best to disable QoS on all ports. This will give you more buffer credits to use where they are really needed.  

An example comparison of the two methods:

Real distance: 50km
Average frame size: 1016 bytes
Link speed: 4GB

NetApp best practices method desired distance: 50km*1.5 = 75km. This will configure the switch to use 75*4/2+6 = 156 buffer credits.

Average frame size method desired distance: (50*2112)/1016 = 104km. This will configure the switch to use 104*4/2+6 = 214 buffer credits.

In this example 214 buffer credits will be needed, while using best practices only 156 buffer credits would be configured..  

Real world:

Using the average frame size method the BB_Credit zero percentage on one of our ISL’s dropped from 22% to 2%. On another ISL it dropped from 16% to 0.3%!

The differences will not always be this dramatic, and in a lot of cases the best practices method will ensure there are more buffer credits configured then needed. Which is a good thing. There is no advantage to having a lot of buffer credits available, so use them to get the maximum throughput between your switches!


Not Published

0/1000 characters
Go Top