Understanding NIC Utilization in VMware ESX

In early December of 2006, I wrote a very popular article on VMware ESX, NIC teaming, and VLAN trunking. In that article, I laid out the configuration for using both NIC teaming and VLAN trunking. In particular, the NIC teaming configuration in that article described the use of Cisco Gigabit EtherChannel for link aggregation, in which both the physical switch and the vSwitch are configured to distribute traffic across all the links between them.

Since that time, the question has come up many times: which method is better, with EtherChannel or without? Many engineers prefer not to use EtherChannel (or its standardized equivalent, static LACP/802.3ad) because of the added complexity involved. It’s easier to just team the NICs at the vSwitch level and leave the physical switches alone. That is true, but what about performance? And what impact does this have on NIC utilization?

There are two ways of handling NIC teaming in VMware ESX:

  1. Without any physical switch configuration
  2. With physical switch configuration (EtherChannel, static LACP/802.3ad, or its equivalent)

In the NIC teaming/VLAN trunking article I referenced above, I noted that there is a corresponding vSwitch configuration that matches each of these types of NIC teaming:

  1. For NIC teaming without physical switch configuration, the vSwitch must be set to either “Route based on originating virtual port ID”, “Route based on source MAC hash”, or “Use explicit failover order”
  2. For NIC teaming with physical switch configuration—EtherChannel, static LACP/802.3ad, or its equivalent—the vSwitch must be set to “Route based on ip hash”

In order to better understand how these settings and different configurations affect NIC utilization, I set out to do some tests in the lab. Most of my tests were centered around IP-based storage from the host (i.e., using NFS or iSCSI for VMDKs), and only tested two basic configurations: using “Route based on originating virtual port ID” and no link aggregation and using “Route based on ip hash” with link aggregation. Although the tests were slanted toward IP-based storage traffic, the underlying principles should be the same for other types of traffic as well. Here’s what I found.

NIC Teaming Without Link Aggregation

First, it’s important to understand the basic behavior in this configuration. Because the vSwitch is set to “Route based on originating virtual port ID”, network traffic will be placed onto a specific uplink and won’t use any other uplinks until that uplink fails. (This is described in more detail in this PDF from VMware.) Every VM and every VMkernel port gets its own virtual port ID. These virtual port IDs are visible using esxtop (launch esxtop, then press “n” to switch to network statistics). That’s simple enough, but what does this mean in practical terms?

  • Each VM will only use a single network uplink, regardless of how many different connections that particular VM may be handling. All traffic to and from that VM will be place on that single uplink, regardless of how many uplinks are configured on the vSwitch.
  • Each VMkernel NIC will only use a single network uplink. This is true both for VMotion as well as IP-based storage traffic, and is true regardless of how many uplinks are configured on the vSwitch.
  • Even when the traffic patterns are such that using multiple uplinks would be helpful—for example, when a VM is copying data to or from two different network locations at the same time, or when a VMkernel NIC is accessing two different iSCSI targets—only a single uplink will be utilized.

This last bullet is particularly important. Consider the implications in a VMware Infrastructure 3 (VI3) environment using the software iSCSI initiator with multiple iSCSI targets. Even though multiple iSCSI targets may be configured, all the iSCSI targets will share one uplink from that vSwitch using this configuration. Obviously, that is not ideal.

Note that this doesn’t really impact VMotion traffic, since VMotion is a point-to-point type of connection. VMotion would only be impacted if placed on a vSwitch with other types of traffic and their virtual port IDs were assigned to the same uplink.

NIC Teaming With Link Aggregation

In this configuration, EtherChannel/static LACP/802.3ad is configured on the physical switch and the ESX vSwitch is configured for “Route based on ip hash.” With this configuration, the behavior changes quite dramatically.

  • Traffic to or from a VM could be placed onto any uplink on the vSwitch, depending upon the source and destination IP addresses. Each pair of source and destination IP addresses could be placed on different uplinks, but any given pair of IP addresses can use only a single uplink. In other words, multiple connections to or from the VM will benefit, but each individual connection can only utilize a single link.
  • Each VMkernel NIC will utilize multiple uplinks only if multiple destination IP addresses are involved. Conceivably, you could also use multiple VMkernel NICs with multiple source IP addresses, but I haven’t tested that configuration.
  • Traffic that is primarily point-to-point won’t see any major benefit from this configuration. A single VM being accessed by another single client won’t see a traffic boost other than that possibly gained by the placement of other traffic onto other uplinks.

This configuration can help improve uplink utilization on a vSwitch because traffic is dynamically placed onto the uplinks based on the source and destination IP addresses. This helps improve overall NIC utilization when there are multiple VMs or when a VMkernel NIC is accessing multiple IP-based storage targets. Note again, though, that individual connections will only ever be able to utilize a single uplink.

Practical Application

I come back again to the question I asked earlier: what does this mean in practical terms?

  • If you want to scale IP-based storage traffic, you’ll have to use link aggregation and multiple targets. Using link aggregation with a single target (destination IP address) won’t use more than a single uplink; similarly, no link aggregation with multiple targets will still result in only a single uplink being used. Only with link aggregation and multiple targets will multiple uplinks get used.
  • Link aggregation will help with better overall uplink utilization for vSwitches hosting VMs. Because there are multiple source/destination address pairs in play, the vSwitch will spread them around the uplinks dynamically.
  • To achieve the best possible uplink utilization as well as provide redundancy, you’ll need physical switches that support cross-stack link aggregation. I believe the Cisco Catalyst 3750 switches do, as do the Catalyst 3120 switches for the HP c-Class blade chassis. I don’t know about other vendors, since I deal primarily with Cisco.

Clearly, this has some implications for efficient and scalable VI3 designs. I’d love to hear everyone’s feedback on this matter. In my humble opinion, extended conversations about this topic can only serve to better educate the community as a whole.

UPDATE: Reader Tim Washburn pointed out that the “Route based on Source MAC hash” actually can’t be used in conjunction with link aggregation; it’s behavior is identical to “Route based on originating virtual port ID”. Thanks for the correction, Tim!

Tags: , , , ,

  1. Duncan’s avatar

    I totally agree with you on this one. The cross stack channel feature is a nice one, but in my opinion most of the time it’s too expensive to justify. Especially considering the fact that when running on iSCSI most customers have to deal with a single destination ip address which indeed results in the same path being taken, always. so In this case, the cheap good old Virtual Port ID will just work fine. And you can easily set it up on two separate switches for redundancy reasons without complications if you don’t forget about STP.

    Anyway, it would be very cool if and when iSCSI multipathing is supported together with round robin load balancing for instance. But we will just have to wait.

  2. Andrew’s avatar

    Thanks for a great article, as always.

    I have a question: I am currently using a single NETAPP as a NFS storage mount for the virtual machine disks. Both the ESX hosts, and the physical switches are configured to use port channeling (or ip hashing in the case of ESX).

    Is is possible to give the NETAPP multiple IPs on the private VLAN and mount the same volume multiple times (e.g., from 1.1.1.1, mount /vol/vol1, from 1.1.1.2, mount /vol/vol1…even though it’s the same NETAPP and same volume)? Then divide the VMs across the two mounts…in my head, since there are different IPs, even though it’s the same two hosts, the port channel will divide the traffic across the multiple interfaces.

    There is some configuration overhead, but would this have the same effect as having multiple storage servers?

    Thanks.

  3. Matt Brown’s avatar

    I currently have 8 nics on my VMWare boxes, they are connected to two pairs of 3750G cisco catalysts switches that are each connected together via a fiber link. (4 cisco switches in total). I have yet to figure out how to create an LACP aggregate across the 2 switch sets as the Port-channels don’t talk to each other from switch to switch unless they are connected via the cisco cross link cable… but then they share the same IOS so upgrades will take down the whole switch.

    What do you think?

    I’m trying to come up with a good solution for vSwitch setup and load balancing for my ESX Boxes, I’ve now got 11 ESX Servers with 8 nics each all connected to the 3750g switches.

    Here’s my current VMWare config
    http://universitytechnology.blogspot.com/2008/06/recommended-network-setup-for-vmware.html

  4. Duane Haas’s avatar

    Scott
    Great post, I am currently using an EMC NS502 via ISCSI, and was curious to know about the comment about multiple targets. Right now my software initiator is configured to go after one IP address. The one ip address is configured to go across three uplink ports(LACP) on the EMC. Are you suggesting that if I create another IP(target) on the NS502 that I might see improved performance. My environment isn’t that large @ the time being, but always love to deploy best practices when configuring and setting up my env. Thanks as always, your time and dedication to the vmware community is much appreciated.

  5. slowe’s avatar

    Duncan,

    Thanks for the feedback! Let me know when that iSCSI multipathing round-robin functionality is ready…. :-)

    Andrew,

    You shouldn’t mount the same NFS datastore on multiple IP addresses. If you can, use a multi-mode VIF on the NetApp (this will require configuration on the switches), then assign multiple IP address aliases to the VIF. Run each datastore on its own IP address. This should provide adequate traffic distribution across the links for both the ESX servers as well as the NetApp storage array.

    Matt,

    You can’t use cross-stack link aggregation without the cross-link cable; ordinary links won’t do it. There are drawbacks to using a stack, but you have to weigh those drawbacks against the ability to better distribute your traffic across the links. I reviewed your configuration and there are some changes I would probably make if it were me, but without knowing the rest of the environment it’s difficult to make good recommendations. Feel free to e-mail me (see the About page) and we can discuss it privately.

    Duane,

    The LACP configuration on the NS502 does help load balancing incoming iSCSI requests from multiple ESX servers/iSCSI initiators. If you want to have even the possibility of getting more than 1Gbps of throughput from an individual ESX server to the NS502, you will need to use EtherChannel for your ESX vSwitches and multiple iSCSI targets on the NS502. This wouldn’t necessarily help improve performance for a single LUN, since a single LUN is typically accessed via a single target, but it would help with multiple LUNs. I’ll defer to EMC experts (Chad, are you reading?) to tell me if I’m incorrect.

  6. Alastair’s avatar

    Scott,

    Thanks for a great blog, love your work.
    One thing to point out is that “Route based on Source MAC” is in the same camp as “Route based on source port ID” in that each virtual NIC uses one and only one physical NIC, rather than like “Route based on IP hash” where one vNIC will use multiple pNICs.
    To me the big risk around IP Hash is that the vSwitch must connect to a single switch config, so that become a much more significant point of failure.
    I definitely agree with the point about iSCSI storage and multiple targets being the primary place where IP Hash is better, personally I’d only use it there and I’d probably dedicate a vSwitch to that function.
    When you look at multiple VMK ports with differnet IP addresses remember they are all under one TCP stack so need to be on different subnets before the VMK route table will end up pointing to different vNICs then pNICs.
    To me IP Storage nirvana involves 10GBE (or infiniband) to let us have a much fatter pipe.

  7. Erwan’s avatar

    Hi,

    I use ESX VI3 for 6 month and really enjoy. I’m wondering if it make sense to install firewall as a VM. Each network interface of the firewall (3 at least) must be connect to different VLAN (tag must be configured on my switch and vswitch) and it’s not a problem for me. My real question is, what about security? I don’t see real risk with this implementation considering vSwith correctly deals VLAN Tag. The firewall i want as a VM will be the front firewall (connecting directly onto Internet).
    Please, give me your advice about this.
    Regards.

  8. Mike Astrosky’s avatar

    Came across your blog while searching for basic vmware/networking tutorials. I am new to vmware (just learned to spell it) and am having a lot of issues. Can you point me to some good turorials? While I actually read about 10% of your blog it will be a year or two before I understand half of what you are explaining.

    I have ESX loaded on a Dell 2950 server with redundant NICS and a separate quad port nic card. No underlying OS – just ESX server. I can access ESX just fine with the web interface and insalled on VM. I have Infrastructure server loaded on a different box. I have addedd the ESX host to the Infrastructure box. I cannot access the VM directly – only through the web interface of the ESX server or through the infrastructure client. How do I get the VM to get IP addresses?

    I appreciate your time for a response (either here of direct) even if it is to say that you cannot give out this kind of help at this time.

    Sorry to bother you.

    Michael

  9. slowe’s avatar

    Mike,

    You need to use the Infrastructure Client to access the console of the VM, at which point you can log into the guest OS and assign an IP address, etc. From there, it should be business as usual.

    Good luck!

  10. Meki Chan’s avatar

    hi, thank you for this great information

    Regards,
    Meki Chan
    Cisco Trick

  11. SlamRand’s avatar

    Scott,

    First, great articles! This is the second solution you’ve given me, led here by Google searches. (I used another article of yours to enable Jumbo Frames between ESXi and my Dell MD3000i iSCSI SAN.)

    I used this article plus your Dec ’06 article on NIC Teaming to configure NIC Teaming with LACP between ESXi and my Catalyst 3750 switch stack. I’ve enabled the teaming on the port groups for both my iSCSI VLAN and my Production LAN VLAN. Although both iSCSI communication and LAN communication are working fine, I’m not sure whether the teaming is actually configured/working properly – not because I’m having any connection troubles, but because I don’t know how to measure the network activity within ESX (or on the switch ports) to see which links are being utilized.

    The only thing I’ve found to go on is the same thing that’s making me question whether it’s working: on my Catalyst stack, both of the channel groups are reporting status as Down. (I’m using Cisco’s GUI configuration tool “Cisco Network Assistant”). I’ve also played around in the IOS command line looking for a way to display more verbose status information, but I haven’t found anything I know how to interpret as indicating one way or the other, whether the port groups are doing what we want them to do.

    Do you have any suggestions on ways to:

    1) Verify that the port groups are actually configured *and functioning* correctly on the switch? What commands can I use to explore this, and what output am I looking for?

    2) Verify that network traffic to different LUN targets on my iSCSI SAN is actually being carried across both links, thanks to the IP hash settings on ESX and etherchannel settings on the switch?

    FYI, my iSCSI SAN has a single controller with two host ports, each port configured with a unique IP, so I’m assuming that meets the requirements of multiple destination IPs needed for the NIC teaming to even matter. :-)

    Thanks for all the help you’re giving to the VMWare community — especially us newbies to ESX!

    Cheers,
    Bryan

  12. ken’s avatar

    good stuff, as usual.

  13. alex’s avatar

    Hi Scott,

    I’m in a situation where I may find myself having to implement iSCSI with software initiators.

    We have allocated 2 dual-port NICs for this, ie 4 NIC ports in total.

    The NAS has 2 controllers, one controller to network switch A, the second controller to network switch B. Then a connection from switch A and B to dual-port NIC #1, and ditto for dual-port NIC #2.

    Going by your blog, it would indicate I could only get throughput through 1 of the 4 NIC ports. Is there a way to configure so I can get throughput through more than one NIC port?

    Cheers
    Alex

  14. slowe’s avatar

    Alex,

    I’m guessing, although you didn’t state it, that your network switches don’t support cross-switch link aggregation (i.e., the ability to perform link aggregation across multiple physical switches). That being the case, it *is* possible to get more throughput, but it’s not as straightforward as it may seem.

    What you’ll need to do is create two VMkernel ports on different subnets and use multiple iSCSI targets on those different subnets. This will allow ESX to use more than one NIC, although it sounds like you’ll be hard-pressed to get ESX to use all four.

    Hope this helps!

  15. alex’s avatar

    Hi Scott

    Thanks for for your help.
    You are correct, our switches do not support cross-switch link aggregation. Are you able to advise me which Cisco switch(es) if any support this functionality… if no Cisco switches do, what switch does?

    Cheers,
    Alex

  16. alex’s avatar

    Also why do you say I will be hard-pressed to get ESX to use all four? Can I not create 4 vmKernel ports on 4 subnets rather than 2 vmkernel ports on 2 subnets?

    Cheers
    Alex

  17. slowe’s avatar

    Alex,

    In addition to creating 4 different VMkernel ports on 4 different subnets, you’d also need to configure your storage system to have 4 different iSCSI targets (one iSCSI target on each of the 4 different subnets). Then, *IF* your storage array is truly active/active and can present LUNs on multiple interfaces at the same time, you might be able to actively use all the interfaces. Otherwise, you’ll have to manually place LUNs on different iSCSI targets in order to get all of the links used. It’s certainly doable, but it will take effort and planning.

    With regards to the Cisco switches, I believe that the Cisco Catalyst 3750 switches offer a stacking option that also allows you to do cross-switch link aggregation. Some of the specific configurations of the high-end chassis switches, like the Catalyst 6500, also offer this functionality.

    Hope this helps!

  18. Robert Eanes’s avatar

    Scott,

    I’ve followed the advice to use link aggregation and nic teaming, and everything seems to be working as well as it can ( 1 link utilized for any one pair of ip addresses ). What I haven’t been able to find any information on, is what to set the other objects in ESXi to. The vSwitch needs to be IP hash, but what about the vm network or the vkernel object. Each of these settings have the ability to override the settings on the vswitch, and I was wondering what the effects are to changing these for the VM’s?

    Thanks,
    Rob

  19. Sean Clark’s avatar

    Wow! Great post. This was just the answer I was looking for. This is why putting VMotion on the same VMkernel nic as iSCSI is probably a bad idea. (Saw that one at customer site a couple weeks ago).

    But this is also the reason that the Equallogic’s best practice is to only use the ESX software iSCSI initiator for a VM’s system drive and use a guest-based iSCSI initiator for data. In that way, they can utilize multiple NICs to distribute the load.

  20. Daniel’s avatar

    Well, using multiple VMkernel NICs and multiple targets works just fine!

    For people like us, doing virtualization on a budget (we’re a small shop with rather strange IT requirements, I grant you that), the Dell and EMC iSCSI enclosures are rather expensive, so we ended up buying a Promise one (which works brilliantly, BTW). I was trying to do link aggregation on both sides, but unless you put a switch between the server and the NAS (which I wasn’t interested in, since we only have one server and won’t be needing vMotion) it will be redundant, but it won’t use the two uplinks (even though the Promise device was configured to aggregate the two ports).

    In the end I assigned each physical NIC to a different vSwitch (each with a VMkernel port group), so I had two paths to the iSCSI target. Then I used esxcfg-mpath to configure a custom load balancing policy, and I’m now getting both redundancy and a considerable increase in performance (also in part thanks to the hidden jumbo frames settings) when doing heavy I/O with multiple VMs!

    (I couldn’t use the VI client to set the load balancing policy because I’m using SW iSCSI so I only have one HBA, and the default round-robin policy is set to use “any” HBA and the “mru” target, rather than the other way around, which is what I would expect.)

    To be honest, without the info in this blog (even if only because it gives us the courage to tinker around with a lot of settings), I would have ended up with a rather mediocre installation. Now I have one that performs roughly 10 times faster than our old ESX 2.5 server (even if I’m doing a number of things that aren’t officially supported). Thanks, Scott!

  21. David’s avatar

    Scott,
    On 11 July 2008, PeterVG asked you a question around having two port channels connecting to a single load balanced IP hash vSwitch. I have a slightly different take on this, keep the 2 port channels connected to a single load balanced IP hash vSwitch, but when you configure the port groups you set the vmnics in port channel 1 to active and vmnics from port channel 2 to standby for port group 1 and the change it around for port group 2.
    If you had multiple port groups you would be able to play around with the vmnic order for each port group.
    My problem goes more around trying to solve the redundancy issues – for example – if you have one port channel per vSwitch and you lose a physical switch you lose all the port groups configured on that vSwitch.
    The VMware knowledge base article he refers to talks about 802.3ad aggregation and not about 802.1q trunking – don’t know whether this would change anything.

  22. slowe’s avatar

    David,

    Your suggestion regarding making all the NICs in the port channel active or standby should work; I’ve not tested it, but it seems to me it should be OK. This would allow you to have some flexibility in how the port groups communicated with the physical switches, but instead of changing NIC failover order on a per-NIC basis you would have to do it on a per-port channel basis. I’ll have to give that a swing to see how it behaves.

    As for redundancy, that’s the real issue, and there’s no easy fix. If you have physical switches that support cross-stack EtherChannel, then you’re OK; if you don’t, well….you’re not OK. I haven’t yet found a good workaround.

    802.1q VLAN trunking doesn’t change anything with regards to the behavior of port channels or NIC failover order. It’s a separate thing entirely.

    Thanks for reading!

  23. Loopie’s avatar

    Scott,
    I am a little confused about the VM connecting to a iSCSI target using one connection.
    I have our vSwitch setup to use two nic’s. Those feed into an etherchannel switch. That feeds into our EMC NS-120 SAN which has multiple NIC’s “trunked” using the same IP.
    I was under the impression the ESX would round robin on the way out seeing both NIC’s have the same IP and a route the the destination. The switch should grab all packets and route/round robin them to the two “trunked” NIC’s in the SAN with the same IP address.

    I think I have tried both IP hash and MAC hash and get the same results. No round robining. Only a single NIC caries all the traffic. The other NIC sits in standby all the time.
    Recomendations?

    thx,
    Loopie.

  24. Tony Aston’s avatar

    Hello, First time post for me…

    I have setup an iSCSI target on an Ubuntu server. ESX 3.5 Update3 host then points to it. We have 2x Gb NIC’s assigned for iSCSI traffic – configured as vSwitch2. As this switch patches to a dedicated Gb switch on a different subnet, i have also added Service console 2 to the vSwitch. The VMKernel also has VMotion enabled.

    When I copy a 2.5Gb file from a physical box to another pysical box, it copies in 3 minutes. When I copy from a physical box to a VM on the ESX host, it was taking 12 minutes. Throughput was 4-5%/s for about 30-40 seconds, then it would drop to 0.1%/s for 30-40 seconds, then up again.

    I noticed that the vSwitch load balancing was set to “Route based on virtual port ID”. Having read some posts and this one, I changed the vSwitch load balancing to “Route based on IP Hash”. This increased the thruput to 9-10%/s (which equates to 100Mb/s), but the ‘lull’ in traffic still presists. Without these ‘lulls’ in traffic I have no issues.

    Can anyone shed any light on why the Network traffic would drop and then return in this manner?

    Cheers in advance.
    Tony

  25. slowe’s avatar

    Loopie,

    That’s just the way that IP hash load balancing works. There’s nothing you can do about it–without multiple source-destination IP address pairs, only a single link will be utilized. The only way to get more than one link utilized is to have more than one source-destination IP address pair, i.e., run multiple iSCSI targets.

    Tony,

    “Route based on IP hash” should only be used when you also are using EtherChannel/static LACP/802.3ad link aggregation on the physical switch. As for the “lulls” in traffic, I would try to ascertain if these lulls are also associated with some other type of activity. At first guess, it sounds to me like the VMs are having problems with I/O throughput, so the copy progress pauses periodically to let the guest VM “catch up” and write data over the iSCSI connection.

    Good luck troubleshooting!

  26. Brett’s avatar

    Hello,
    Just to be sure I understand correctly, I think we are getting higher throughput in this scenario:

    3 ESX servers, teamed NICs on the storage network, physical switch and vSwitches set to 802.11ad static. Physical switch is a dedicated switch for handling NFS traffic. NFS server has 4 teamed NICs, also set to to 802.11ad static.

    Is is true that-
    We are still getting just 1GB worth of bandwidth to/from any ESX host, since the NFS server has a single IP, but the NFS server, communicating with 3 distinct ESX hosts, is seeing as many as 3 links utilized at once?

    Thanks!

  27. Kosch’s avatar

    Hello

    Just to say thanks for this article it is very informative and has helped me so much in understanding what is going on with the networking side.

    I’m trying to work out some network issues we have at the moment with Class C Blades, Cisco Ethernet Virtual Connects, 4 Cisco 3750′s stacked with cables and ESX.

    I’ve been reading our Catalyst manual and it talks about LACP (active,passive,on) but it doesnt mention anything about Static LACP as per VMWares guide when using route based on IP Hash. Could anyone enlighten me to this please?

    We had some consultants come in and setup our Blades, VC, and Cisco ethernet stack a while ago and I’m not convinced its performing correctly.

    Each blade has 4 NIC’s in, through VC we present all the vlans to ESX and do the tagging there.

    I’ve been poking through our switch stack config and have found the following.

    Ether Channel Detail

    Group: 1
    ———-
    Group state = L2
    Ports: 3 Maxports = 16
    Port-channels: 1 Max Port-channels = 16
    Protocol: LACP

    Port state = Up Mstr Assoc In-Bndl
    Channel group = 1 Mode = Active Gcchange = -
    Port-channel = Po1 GC = – Pseudo port-channel = Po1
    Port index = 0 Load = 0×00 Protocol = LACP

    Ether Channel Load Balancing set to “src-mac”

    ESX vSwitch set to “originating virtual switch port ID”, we have one vSwitch configured with all the vLans in it and service console

    So from what I understand this means even though we have lots of bandwidth (4GB per blade) it would have only ever been using 1GB and also potentially the etherchannel and vswitch configuration is wrong?

    I have also seen some MAC flapping events in the switchstack log but oddly the majority of MAC addresses reported are the MAC’s from the VC Ethernet stacking links.

    Kosch

  28. slowe’s avatar

    Kosch,

    If the switchports that connect to your blade servers are configured as EtherChannel bundles and your vSwitch is set to “Route based on originating virtual port ID,” then it’s definitely configured wrong. Go back and read my article on trunking and VLANs:

    http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

    That should help clarify things a bit.

  29. Kosch’s avatar

    Thanks I will. Still finding my feet with all of this :)

  30. Jfinn’s avatar

    Hi Scott, your Blog has been a great source of information for me; keep up the good work!

    Wold this work to give me switch redundancy?
    A FAS2020 set up for Single Mode VIF and each nic (there are only two!) is plugged into a separate switch. The switches would be connected to each other with a GBE cable. Then each switch would plug into an ESX host. Those pNics would both be part of a vSwitch for VMKernel for NFS. The NetApp would have a single IP as would each NFS VMKernel.

    Would this senario allow for a switch/nic/cable failure?

    I could use single mode VIF on the FAS since even with IPHash I’m only using 1 interface any way. right?

    Would I have to manually set active/passive pNic on ESX to mach the FAS config?

    I’m running procurve switches so cross-stack trunking isn’t an option right?

    thanks!

  31. John’s avatar

    Thanks for this and your previous article, as a beginner these posts have been invaluable in trying to understand this.

    We have set up NIC teaming and VLAN trunking as in your first article. In a bid to improve throughput over the Lan we implemented EtherChannel, static LACP/802.3ad on the physical switch (2 linked 3750’s) and “Route based on ip hash” on the vswitch.

    All seems to works fine but if I’m reading all this correctly (“VMkernel NIC will utilize multiple uplinks only if multiple destination IP addresses are involved”) the limitation is the 1 gig card in the esx host. The fact I have three of them teamed doesn’t matter if I’m connecting to one client (apart from the performance gain due to the distribution across the uplinks).

    In my case I was trying to increase throughput to a backup device we connect to over the Lan. We use VCB for all our backups but there is a need to backup some guests over the Lan. We don’t implement ip based storage so am I gaining much with this configuration or was mine just a 3 gig pipe dream?

    Thanks

  32. slowe’s avatar

    John,

    EtherChannel helps when there are multiple source-destination IP address pairs involved. If this is multiple guest OS instances backing up to a backup server, that’s fine; if it’s a single VMkernel NIC connecting to multiple iSCSI targets, that’s fine. If there’s multiple source-destination IP address pairs then you’re OK. If not, then it will do you absolutely no good whatsoever.

    Further, even IF there are multiple source-destination IP address pairs involved, the throughput of any single IP address pair cannot exceed the speed of a single link, i.e., 1Gbps. If you need more than 1Gbps for a single source-destination IP address pair, you’ll need to step up to 10Gbps.

    Hope this helps!

  33. John’s avatar

    Yes it does. I guess I was misinterpreting the way teaming works in both virtual and physical environments.

    Thanks

  34. Dan’s avatar

    Hi Scott, Long time reader first time poster here. We had many issues surrounding this very issue and a good friend of mine actually found this which was a true life saver with its almost dummy proof instructions. Hope it helps others too….

    http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1004048&sliceId=2&docTypeID=DT_KB_1_1&dialogID=3035461&stateId=0%200%203033691

  35. david c’s avatar

    I am hooking a esx 3.5 box to a cisco 3750 with 4 ports.
    When I configure the 3750 with “mode active” (lacp) I do not see any trunked vlans.
    When I configure the 3750 with “mode on” I see all the vlans active.

    It seems like the ESX box is not doing lacp, rather it is doing pagp.
    Is there a way to check how the ESX box is set ?

  36. slowe’s avatar

    David C,

    ESX does not support any form of dynamic link aggregation. Neither PAgP nor LACP are supported. You must use “channel group X mode on” in order for it to work with VMware ESX/ESXi. To my knowledge, this is not adjustable or settable from the VMware side.

    Hope this helps!

  37. david c’s avatar

    Thank you, the “channel group” worked. We had a non-technical problem.

    Is it possible to have redundant vswitches where each vswitch is made up of nics connected to multiple physical switches and if pSwitch_1 fails rollover to the second vSwitch & pSwitch_2. If I could team a VM to two vSwitches this would work but I do not think I can team an VM OS.

    This is for the case where we may have parallel 3560 switches connected to a single 3750 stack and can not connect all ESX hosts to the 3750.

    3750
    / \
    3560 3560
    |||| ||||
    vS1 vS2

  38. slowe’s avatar

    David C,

    You’ll have to tear apart your channel groups, but you could bind each vSwitch to a pair of NICs, and connect the NICs to separate physical switches. This would accomplish the same effect.

  39. Shai’s avatar

    Hello,

    this is a great post and interesting discussion. I’m actually having a very different experience with ESX 4 and NFS datastores.
    I setup 2 NICs a vswitch with a VMkernel configured to load ballance using IPHash. I than configure 2 NFS datastores to 2 different IPs (making sure the hash function returns to different values). I use Cisco 3750 and here is the interesting part:
    If I DO NOT configure etherchannel for the 2 ESX ports I get aggregate bandwidth across both NICs (I can see that quite clearly in the guest OS and esxtop with 1600Mbps total).

    If I DO turn on etherchannel on the switch I get HALF the bandwidth. Specifically esxtop shows both NICs sending packets at 800MBps but only 1 of them is recieving and the total is only around 800MBps.

    I am using src-dst-ip load balancing on the switch.

    In short I get double the bandwitdth out of 2 teamed NICs WITHOUT ethernchannel that I do with etherchannel.

    Anyone care to explain?
    Thanks,
    Shai

  40. Denys’s avatar

    I have two ESX server connected to one NFS storage, to links from each device to a dedicated physical switch, making it an isolated network. I do not use VLANs. The switch supports 802.3ad.

    On each of ESX servers, I have a separate vSwitch configured with VMkernel on one side and two vmnics on the other side. Teaming mode is ip hash-based.

    How do I achieve ‘multiple source-destination ip pairs’ in such setup? Do I need to add more VMKernels with different IPs? I do not see how can I assign different IPs to teh NFS storage since the NICs are ‘bonded’ and have only one IP.

  41. Robert’s avatar

    Sorry if this has already been asked but I am currently in the planning process of upgrading our core ESX infrastructure.

    Currently I have a LAB setup for testing and this is what i have so far.

    ESX4.0

    VSwich 0
    VLAN 10 MANAGEMENT-Network (VM Network)
    VLAN 10 MANAGEMENT-Console (Console)
    VLAN 100 iSCSI-VMotion (VMotion)
    VLAN 100 iSCSI-Console (Console)

    4 X pNIC 4Gb Etherchannel Trunk to 3560-1
    So far everything is working and tested out good! But we would like to setup redundancy between two 3560s via HSRP.

    Currently we have two core 3560s for layer 3 routing between vlans setup with HSRP for a total of 8 pNICS total 2 x 4 Gb Etherchannel Trunk to each ESX Server. I can setup 4 of the pNICs going to 3560-1 as active and the other 4 going to 3560-2 as standby. But, if i unplug ONE of the first pNICS the 5th one comes online and thus splits up my etherchannel and flapping occurs.

    Is there an easier way to do this with non stacking switches? IE: Can I configure ESX to ONLY bring TEAMB (pNIC 4,5,6,7) online only when TEAM A Completely fails (pNIC 0,1,2,3) ? Or Make TEAM B completely ACTIVE and TEAM A completely STANDBY if any one of the pNICS in TEAM A fails?

    Sorry if I’m not making any sense =/
    We want 4Gb of complete trunked redundancy and load balancing accomplished.

    /end rant

  42. Robert’s avatar

    I left out that that 3560-1 and 3560-2 are connected via a trunked 4Gb Etherchannel between each other.

  43. Michael’s avatar

    Thanks for the info.

    I am completely new to VMWare and I have a few questions. Here is my setup:

    I have one server with 4 network adapters running ESXi 4.0.

    There is one NAS device with a single network port being used for VM backups.

    I connect to the ESXi server using the vSphere client and do not have a vSphere server.

    NIC1 is connected to the internal network and is assigned to a virtual switch. This is the management interface and I’m running 4 VM’s that all use the assigned virtual switch.

    NIC2 is connected to my hardware firewall and is assigned to another virtual switch that is used by only one of my VM’s which is my proxy server.

    NIC’s 3 and 4 are not connected.

    I would like to know whether it would be possible to use NIC 3 or 4 together with NIC1 for load balancing and better network performance for my VM’s? If possible, how?

    Can I connect the last NIC to the firewall, configure a new VMkernel vSwitch and assign a public IP address to the interface so that I can manage the ESX server over the Internet when I can’t be at the office or can only one IP be used for management? (I still want to be able to manage the server in the local LAN using the existing private IP)

    Thanks!

  44. slowe’s avatar

    Michael,

    I know this sounds self-serving, but the best approach for you would be to pick up the book “Mastering VMware vSphere 4″ (which I authored)—it will give you all the information you need to configure and support vSphere. Given that you are, admittedly, new to vSphere, I think that this would be a very valuable resource for you.

    Good luck!

  45. Josh’s avatar

    Hi Scott,

    Great post, I’m glad that I came across it.

    Quick question; if the only way to get a VM to load balance across multiple physical NICs is to use the IP-Hash option, I am curious as to the behavior if you select IP-Hash as the policy on a vSwitch with multiple uplinks connected to a single pSwitch, but did not configure etherchannel/LACP on the pSwitch?

  46. slowe’s avatar

    Josh,

    Generally, you will lose connectivity when the vSwitch is set to IP hash but the pSwitch is not configured for LACP/EtherChannel.

  47. Josh’s avatar

    Got it, thanks for the quick reply.

  48. Josh’s avatar

    Scott,

    Another question; do you know the best practice for setting this up? Meaning, a different vSwitch for different ether channel groups? Or, one vSwitch, different port groups, with the port group LB policy overwriting that of the vSwitch and manually moving only those adapters in the eth chan group under active adapters? I have looked a bit on this and can’t find anything conclusive on which is better.

  49. slowe’s avatar

    Josh,

    You should only use EtherChannel (and the matching load balancing policy, “Route based on ip hash”) at the vSwitch level, not at a port group level. You can set it at a port group level, but it won’t work properly. This means you should use different vSwitches for different EtherChannel groups.

1 · 2 ·