[Ltsp-discuss] Beware Ethernet Flow Control

[Jason Maas]

Hello,

Yesterday we figured out a problem with our network which was causing
severe disruption of our LTSP setup at times and I thought I'd share
what I learned in case it's helpful to anyone else.

Our LTSP server has a gigabit NIC and is plugged into a gigabit switch.
Our LTSP clients have 100 megabit NICs. The problem we were seeing is
that a single client displaying anything resembling video (youtube,
screencast demo of software, flash-animated website) would bring all
other terminals on the network to a crawl. Very painful, not cool.

Well, it turned out that a feature of modern ethernet called Flow
Control (802.3x) was the source of our problem. It was enabled on a
switch with mixed speed connections and when a 100 megabit connection
was getting flooded it would tell the sender, our LTSP server, to stop
sending traffic for a bit. All traffic. As you can imagine that's not
helpful in a thin client environment!

Turning off Flow Control (using the web interface on our Netgear smart
switches) made it so one terminal's network traffic doesn't affect the
rest of the terminals. Yay!

Here's a link to a nice blog article which explains it better than I can:

http://virtualthreads.blogspot.com/2006/02/beware-ethernet-flow-control.html

As a side note, turning on SSH compression by adding a line like
"Compression yes" in a place /opt/ltsp/i386/etc/ssh/ssh_config as was
discussed recently dropped the amount of traffic down *considerably* so
that other devices on the same edge switch (without gigabit uplink)
won't get starved either.

Note that we're still running Ubuntu 8.04 and thus not using local apps
yet, which could of course change the amount of network traffic for
videos, etc. We'll get there someday!

HTH,
Jason

[David Burgess]

Post by Jason Maas
Hello,
Yesterday we figured out a problem with our network which was causing
severe disruption of our LTSP setup at times and I thought I'd share
what I learned in case it's helpful to anyone else.
Our LTSP server has a gigabit NIC and is plugged into a gigabit switch.
 Our LTSP clients have 100 megabit NICs.  The problem we were seeing is
that a single client displaying anything resembling video (youtube,
screencast demo of software, flash-animated website) would bring all
other terminals on the network to a crawl.  Very painful, not cool.
Well, it turned out that a feature of modern ethernet called Flow
Control (802.3x) was the source of our problem.  It was enabled on a
switch with mixed speed connections and when a 100 megabit connection
was getting flooded it would tell the sender, our LTSP server, to stop
sending traffic for a bit.  All traffic.  As you can imagine that's not
helpful in a thin client environment!
Turning off Flow Control (using the web interface on our Netgear smart
switches) made it so one terminal's network traffic doesn't affect the
rest of the terminals.  Yay!
http://virtualthreads.blogspot.com/2006/02/beware-ethernet-flow-control.html
As a side note, turning on SSH compression by adding a line like
"Compression yes" in a place /opt/ltsp/i386/etc/ssh/ssh_config as was
discussed recently dropped the amount of traffic down *considerably* so
that other devices on the same edge switch (without gigabit uplink)
won't get starved either.
Note that we're still running Ubuntu 8.04 and thus not using local apps
yet, which could of course change the amount of network traffic for
videos, etc.  We'll get there someday!
HTH,
Jason

Praise be! I've been seeing exactly this on our network and I'm
hopeful that you've nailed the cause. But hey, my Netgear Prosafe
gigabit switch is unmanaged. Will disabling flow control on the server
fix this, or is this happening in the switch? Anybody have any ideas
about that? I guess I won't know for sure until I try it.

Thanks again for posting this!

db

[Άλκης Γεωργόπουλος]

Thanks for posting this!

I've been having the same problem (my gigabit server on my gigabit
switch is able to send at most 100Mbps to my 100Mbps clients) and I was
trying to figure out why!
I had to use 4-NIC bonding to get it to 400Mbps..

My switch is unmanaged, is there any way to disable flow control on the
server?

--
Alkis Georgopoulos

[Wojtek]

Hello,

our installation produced a lot of traffic too (peek was 15-17 Mbytes/s
on each server).
See the page
http://lartc.org/lartc.html#LARTC.QDISC
(Queueing Disciplines for Bandwidth Management)
for infos how to enable traffic control on a linux server. This "howto"
is quite long because linux has A LOT of configuration possibilities. A
realy great stuff!


Greetings,

Wojtek

Post by Άλκης Γεωργόπουλος
Thanks for posting this!
I've been having the same problem (my gigabit server on my gigabit
switch is able to send at most 100Mbps to my 100Mbps clients) and I was
trying to figure out why!
I had to use 4-NIC bonding to get it to 400Mbps..
My switch is unmanaged, is there any way to disable flow control on the
server?
--
Alkis Georgopoulos
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net

[Άλκης Γεωργόπουλος]

Post by Wojtek
See the page
http://lartc.org/lartc.html#LARTC.QDISC
(Queueing Disciplines for Bandwidth Management)
for infos how to enable traffic control on a linux server.

Erm... thanks, but I was looking for a way to *disable flow* control,
not enable traffic control.

I think I found a way:
ethtool -A eth0 tx off

Unfortunately, my Realtec NICs do not support this setting.
Fortunately, my Intel NICs support this, so I'll do a benchmark in
Thursday to see if my network is now 10 times faster! :)

--
Alkis Georgopoulos

[David Burgess]

Post by Άλκης Γεωργόπουλος
ethtool -A eth0 tx off

This command will turn of autonegotiation. Disabling flow control on
the intel e1000e driver and e1000 (and probably other intel boards) is
done when loading the module. Do "modprobe e1000e FlowControl=0" if
you only have one card using the e1000e driver. In my case I have two
and I want to leave it on for eth0, so I do "modprobe e1000e
FlowControl=3,0". If you're using ubuntu you can edit
/etc/modprobe.d/options and add the line "options e1000e
FlowControl=3,0" to make it permanent. If you're using other than
intel nics then you'll have to check the docs for turning off flow
control, but I imagine it will likewise happen when the module is
loaded.

I haven't tried this on my ltsp server yet. I'm hoping it will fix the
very real problem of things like flash and gnome-system-monitor
slowing the whole tc network to a crawl.

db

[David Burgess]

Post by David Burgess
Do "modprobe e1000e FlowControl=0" if
you only have one card using the e1000e driver.

Correcting myself now. The e1000 module recognizes the FlowControl
option but the e1000e apparently does not.

With the e1000e card I have done "ethtool -A eth1 autoneg off tx off"
and can confirm that running heave flash on one tc does no longer
interfere with the interactivity on another. I will have to create a
simple init script to do this at boot. Case closed for me.

Thanks again for posting this!

db

[David Burgess]

Post by David Burgess
"ethtool -A eth1 autoneg off tx off"

I will get this right yet. Turning off tx doesn't actually help. The
following was the magic command for me.

ethtool -A eth1 autoneg off rx off

Sorry for the confusion. Oh, to be able to go back and edit old posts!

db

[Άλκης Γεωργόπουλος]

Στις 01-03-2009, ημέρα Κυρ, και ώρα 12:45 -0700, ο/η David Burgess

Post by David Burgess
ethtool -A eth1 autoneg off rx off

I didn't get a chance to try this myself yet, but why "autoneg off"?
As I understand it, "autoneg on" would tell the switch to not send PAUSE
frames at all,
which (if the switch supports it) would be better than the switch
sending PAUSE frames and the NIC not respecting them (rx off).

Of course, even with "autoneg on", "rx off" should be there as well to
tell the switch that our NIC doesn't accept PAUSE frames.

Did I understand it wrong?


Also, you were right about /etc/modprobe.d/options, but it seems that
not all drivers support this, and not all of them use the same parameter
name.
E.g. the tg3 driver needs rx_flow_control=off.

--
Thanks,
Alkis Georgopoulos

[David Burgess]

Post by Άλκης Γεωργόπουλος
Στις 01-03-2009, ημέρα Κυρ, και ώρα 12:45 -0700, ο/η David Burgess

Post by David Burgess
ethtool -A eth1 autoneg off rx off

I didn't get a chance to try this myself yet, but why "autoneg off"?
As I understand it, "autoneg on" would tell the switch to not send PAUSE
frames at all,
which (if the switch supports it) would be better than the switch
sending PAUSE frames and the NIC not respecting them (rx off).
Of course, even with "autoneg on", "rx off" should be there as well to
tell the switch that our NIC doesn't accept PAUSE frames.
Did I understand it wrong?
Also, you were right about /etc/modprobe.d/options, but it seems that
not all drivers support this, and not all of them use the same parameter
name.
E.g. the tg3 driver needs rx_flow_control=off.
--
Thanks,
Alkis Georgopoulos

Quoting from Intel's linux documentation for the e1000e driver:

"In order to disable receive flow control using ethtool, you must turn
off auto-negotiation on the same command line.

For example:

ethtool -A eth? autoneg off rx off"

I tried this with my server's Pro 1000 PT desktop adapter and it
solved the issue I was having, so I didn't investigate further. You
may be correct in theory that leaving autonegotiation on would be more
desirable, but I didn't test that, and according to intel, as you can
see, it must be turned off to disable receive flow control.

You can get the docs for intel's gigabit nics here:
http://downloadcenter.intel.com/Detail_Desc.aspx?agr=Y&DwnldID=11848

As you pointed out, other adapter drivers will handle flow control
differently. In fact, I have another machine using an intel pro 1000
GT adapter, which uses the e1000 driver (pci instead of pcie) and flow
control is disabled when loading the module, rather than with ethtool,
as with the e1000e adapters. This is in accordance with intel's
documentation.

db

[Jason Maas]

Hi David,

Post by David Burgess
Praise be! I've been seeing exactly this on our network and I'm
hopeful that you've nailed the cause.

Great, I'm glad that my experience is useful to others!

Post by David Burgess
But hey, my Netgear Prosafe gigabit switch is unmanaged. Will
disabling flow control on the server fix this, or is this happening
in the switch?

I had no idea of the answer to your question when you first wrote it
since I had only messed with the settings on the switch. Based on your
emails since then it looks like you've found ways to disable flow
control on the LTSP server. Good job sleuthing!

FYI, I highly recommend getting at least "smart" or "web controllable"
switches if at all possible for your core network switches. They
usually don't cost too much more than a "dumb" or unmanaged switch and
since the network is such an important component of an LTSP installation
it's worth it to invest some money there IMHO.

Post by David Burgess
Thanks again for posting this!

You're very welcome. Thanks for taking it to the next level!

Jason

[Άλκης Γεωργόπουλος]

Just reporting that I did a benchmark with flow control = off.

Before: speed ~= 90 Mbps
After: speed ~= 900 Mbps

So upgrading the clients to gigabit isn't really required! A switch
with a single (or two, if bonding is used) gigabit ports is enough! :)

Thanks Jason, you saved a lot of people a lot of money.

--
Alkis Georgopoulos

[Keith]

This all sounds awesome. We have Planet switches - no place to adjust flow
contorl. How do I do this with software? We are using Ubuntu 8.10 & LTSP 5?

- Keith

----- Original Message -----
From: "Άλκης Γεωργόπουλος" <***@gmail.com>
To: <ltsp-***@lists.sourceforge.net>
Sent: Tuesday, March 03, 2009 10:43 AM
Subject: Re: [Ltsp-discuss] Beware Ethernet Flow Control

Post by Άλκης Γεωργόπουλος
Just reporting that I did a benchmark with flow control = off.
Before: speed ~= 90 Mbps
After: speed ~= 900 Mbps
So upgrading the clients to gigabit isn't really required! A switch
with a single (or two, if bonding is used) gigabit ports is enough! :)
Thanks Jason, you saved a lot of people a lot of money.
--
Alkis Georgopoulos
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,
CA
-OSBC tackles the biggest issue in open source: Open Sourcing the
Enterprise
-Strategies to boost innovation and cut costs with open source
participation
SFAD
http://p.sf.net/sfu/XcvMzF8H
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net

[Άλκης Γεωργόπουλος]

Post by Keith
This all sounds awesome. We have Planet switches - no place to adjust flow
contorl. How do I do this with software? We are using Ubuntu 8.10 & LTSP 5?
- Keith

I've started a page in ubuntu wiki for this:
https://help.ubuntu.com/community/UbuntuLTSP/FlowControl

Normally,
sudo ethtool -A eth0 autoneg off rx off
can be used (any hints on a good place to put this?)

but for drivers that do not support ethtool, the appropriate parameters
may be appended to /etc/modprobe.d/options.

Anyone that has corrections / suggestions / information please do edit
the wiki page and add it... :-)

--
Alkis Georgopoulos

[Jason Maas]

Hi Alkis,

Post by Άλκης Γεωργόπουλος
https://help.ubuntu.com/community/UbuntuLTSP/FlowControl

Fantastic! I was hoping someone would get this written up in a wiki.
You did a really good job summarizing the situation and what to do about
it. Thank you very much to you and others who have helped us discover
more of the specifics of the problem and the various ways to solve it!

Jason

[Peter]

Hi Keith
Play with ethernet flow at your own peril :)
There will be a lot of improvements in the future , wait until there is real
proof.
Andy will pick up on things like that if anyone will in our area.
To change the file size,
sudo su
rezize2fs /dev/md0
no size says it will expand to fill the whole partition.
wait about 5 mins and all should be OK
If not I'll have to go back to the drawing board.
Regards
Peter
----- Original Message -----
From: "Keith" <***@gsps.co.za>
To: <ltsp-***@lists.sourceforge.net>
Sent: Tuesday, March 03, 2009 2:19 PM
Subject: Re: [Ltsp-discuss] Beware Ethernet Flow Control

Post by Keith
This all sounds awesome. We have Planet switches - no place to adjust flow
contorl. How do I do this with software? We are using Ubuntu 8.10 & LTSP 5?
- Keith
----- Original Message -----
Sent: Tuesday, March 03, 2009 10:43 AM
Subject: Re: [Ltsp-discuss] Beware Ethernet Flow Control

Post by Άλκης Γεωργόπουλος
Just reporting that I did a benchmark with flow control = off.
Before: speed ~= 90 Mbps
After: speed ~= 900 Mbps
So upgrading the clients to gigabit isn't really required! A switch
with a single (or two, if bonding is used) gigabit ports is enough! :)
Thanks Jason, you saved a lot of people a lot of money.
--
Alkis Georgopoulos

--------------------------------------------------------------------------

----

Post by Keith

Post by Άλκης Γεωργόπουλος
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,
CA
-OSBC tackles the biggest issue in open source: Open Sourcing the
Enterprise
-Strategies to boost innovation and cut costs with open source
participation
SFAD
http://p.sf.net/sfu/XcvMzF8H
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net

--------------------------------------------------------------------------

----

Post by Keith
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco,

CA

Post by Keith
-OSBC tackles the biggest issue in open source: Open Sourcing the

Enterprise

Post by Keith
-Strategies to boost innovation and cut costs with open source

participation
SFAD

Post by Keith
http://p.sf.net/sfu/XcvMzF8H
_____________________________________________________________________
https://lists.sourceforge.net/lists/listinfo/ltsp-discuss
For additional LTSP help, try #ltsp channel on irc.freenode.net

[Jan Kunder]

Post by Jason Maas
Hello,
Yesterday we figured out a problem with our network which was causing
severe disruption of our LTSP setup at times and I thought I'd share
what I learned in case it's helpful to anyone else.
Our LTSP server has a gigabit NIC and is plugged into a gigabit switch.
Our LTSP clients have 100 megabit NICs. The problem we were seeing is
that a single client displaying anything resembling video (youtube,
screencast demo of software, flash-animated website) would bring all
other terminals on the network to a crawl. Very painful, not cool.
Well, it turned out that a feature of modern ethernet called Flow
Control (802.3x) was the source of our problem. It was enabled on a
switch with mixed speed connections and when a 100 megabit connection
was getting flooded it would tell the sender, our LTSP server, to stop
sending traffic for a bit. All traffic. As you can imagine that's not
helpful in a thin client environment!
Turning off Flow Control (using the web interface on our Netgear smart
switches) made it so one terminal's network traffic doesn't affect the
rest of the terminals. Yay!
http://virtualthreads.blogspot.com/2006/02/beware-ethernet-flow-control.html
As a side note, turning on SSH compression by adding a line like
"Compression yes" in a place /opt/ltsp/i386/etc/ssh/ssh_config as was
discussed recently dropped the amount of traffic down *considerably* so
that other devices on the same edge switch (without gigabit uplink)
won't get starved either.
Note that we're still running Ubuntu 8.04 and thus not using local apps
yet, which could of course change the amount of network traffic for
videos, etc. We'll get there someday!
HTH,
Jason

Many thanks for pointing this out and "bugreporting".

ty
JK

tag tags Beware Ethernet Flow Control overflow un-managed switch hub video
youtube traffic overload highload thin client