Switching

Hello clever people,

I have a network refresh in progress and we have a few technology options to choose from.  The network will need to support iSCSI at 10Gig and according to the 'HP StoreVirtual 4000 Storage Network design considerations and best practices' document, it says:

"For sequential workloads with larger block sizes, it is recommended that the switch has sufficient buffer cache. In these
environments, it is recommended to allocate at least 512 KB of buffer cache per port"

I have been searching for quite a while to find out what the buffer sizes are on the EX4200, EX4550 and QFX3500, but that information doesn't seem to be published anywhere (almost like it's a dirty secret).  Even looking at the CLI on a 4200 didn't reveal how big the egress buffers were.

The class of service documentation (http://www.juniper.net/techpubs/en_US/junos13.2/information-products/pathway-pages/ex-series/cos.pdf#search=%22class%20of%20service%20ex%22) mentions each port having its own dedicated buffer and a shared pool of buffer space that ports can dip into when their own buffer runs out, but nowhere does it say how big this pool is, or how big each dedicated buffer is.

So, can anyone shed any light on how big these buffers are on these switches?  Also, does anyone have any real world experience with 10Gig/1Gig iSCSI (particularly on the EX4550)?  Is the 512KB buffer really an issue?

I have found a Juniper document which basically says configure DCBX for iSCSI and you'll be fine, but that assumes that both the storage and HBAs in the host support DCBX for iSCSI (neither of which do in our case - so that's not going to work).

Any help would be much appreciated.

Cheers

Martin

EX4200 has 2.5MB shared buffer

EX4500 has 4MB shared buffer

QFX3500 has 9MB shared buffer

QFX5100 has 12MB shared buffer

None of these have dedicated per-port buffering as far as I am aware.

The values that ronf mentioned are per PFE. EX4200 can have 2 or 3 PFEs and the rest of devices 1 PFE).

But this buffer by default is split equally in shared buffers and dedicated buffers.

This means that each port has some buffers that cannot be used by other ports and if not used, they are wasted.

If a port needs more buffers, it can take buffers from the shared buffers. 

So a port can be using from 0 shared buffers up to all shared buffers.

You can change the size of the shared buffer like this:

{master:1}[edit]
root@EX# set class-of-service shared-buffer percent ?
Possible completions:
<percent> Percentage of shared buffer to be used (5..100)
{master:1}[edit]
root@EX#

If you choose 100 percent, then the ports will not have any dedicated buffers.

If you go to 5 percent, then almost everything will be allocated as dedicated buffers to the ports.

If you intend to change this, carefully analyze all the consequences that comes from this change.

- if you leave the ports without no dedicated buffers, you might prevent some ports to send traffic if there is an aggressive flow that will eat all the shared buffer

-if you leave no shared buffer, then the buffers that goes to ports underutilized will be wasted and flows that need more buffers will encounter packet loss

=====

If this worked for you please flag my post as an "Accepted Solution" so others can benefit. A kudo would be cool if you think I earned it.

If anyone still trying to figure out the buffer size for EX4200. It is described as follows.

24 ports on EX4200= 1PFE (Packet-Forwarding-Engine) 

My EX4200-has 48 ports + 4 uplink ports = 3PFE

Every 1PFE = has 3MB  of buffer size.

Hence, EX4200 with 48ports + 4 uplink ports = 3PFE = 3 x 3MB = 9MB total buffer space available for shared buffer.

(Note: Found above info based on 2010 Junos CoS document, couldn't find any latest. Please also check attached gif, has explaination also for EX2200, EX3200, EX8200 or go http://www.juniper.net/us/en/local/pdf/implementation-guides/8010073-en.pdf )

Hope this Helps. Cheers!

- AJ