Tuning Buffer Credits on NetApp MetroCluster Fabric SwitchesBrenn Oosterbaan
- The NetApp and Brocade best practices state the "desired distance" of a ISL port should be set to 1.5 times the actual distance. This will then set the buffer credits to a certain amount.
- This article from NetApp also describes a second method: calculating the average frame rate to determine the needed amount of buffer credits.
Use the following to calculate if your buffer credit settings might be a problem:
portstats64show <port> - This gives the amount of frames send (stat64_ftx) and amount of BB_credit zero occurrences (tim64_txcrd_z).
stat64_ftx 3 top_int : Frames transmitted 970578587 bottom_int : Frames transmitted
tim64_txcrd_z 0 top_int : Time BB_credit zero 46631371 bottom_int : Time BB_credit zero
The total amount of frames sent is (2^32)*<top_int>+<bottom_int>.
The total amount of BB_credit zero is (2^32)*<top_int>+<bottom_int>.
The percentage of BB_credit zero is (<BB_credit zero>/<frames sent>)*100.
According to Brocade some BB_credit zero is allowed, but no more than 15% (which still seems high).
Use the following to calculate the average frame size:
portstats64show <port> - This gives us the amount of words sent (stat64_wtx), which translates to bytes send.
stat64_wtx 1485 top_int : 4-byte words transmitted 4108584283 bottom_int : 4-byte words transmitted
total amount of bytes sent = ((2^32)*<top_int>+<bottom_int>)*4. average frame size = <bytes sent>/<frames sent>.
Use the following to calculate the desired distance which is used by the Brocade switch to set the buffer credits:
desired distance = (real distance*2112) / average frame size.
Set this distance on the Brocade switches using:
portCfgLongDistance <port> LS 1 <desired distance>
To see how the buffer credits are configured for each port and how many your switch still has available: portbuffershow
If you want you can calculate how the switch will translate the desired distance to buffer credits:
buffer credits = <desired distance>*(<link speed>/2) + 6
If QoS is enabled on the port it will allocate another 14 buffer credits. By default QoS is enabled, but no QoS zones are configured. In effect this does nothing except consume an extra 14 buffer credits per port. Unless you are going to really use QoS by configuring QoS zones it is best to disable QoS on all ports. This will give you more buffer credits to use where they are really needed.
An example comparison of the two methods:
Real distance: 50km
Average frame size: 1016 bytes
Link speed: 4GB
NetApp best practices method desired distance: 50km*1.5 = 75km. This will configure the switch to use 75*4/2+6 = 156 buffer credits.
Average frame size method desired distance: (50*2112)/1016 = 104km. This will configure the switch to use 104*4/2+6 = 214 buffer credits.
In this example 214 buffer credits will be needed, while using best practices only 156 buffer credits would be configured..
Using the average frame size method the BB_Credit zero percentage on one of our ISL’s dropped from 22% to 2%. On another ISL it dropped from 16% to 0.3%!
The differences will not always be this dramatic, and in a lot of cases the best practices method will ensure there are more buffer credits configured then needed. Which is a good thing. There is no advantage to having a lot of buffer credits available, so use them to get the maximum throughput between your switches!