VMware ESX VLAN Tagging with Cisco EtherChannel Trunks
In this article I will show you how to benefit from using VLAN Tagging with ESX Virtual Switches. This will allow you to increase bandwidth on demand and allow you to run multiple VLANs on your server without installing extra NICs on your ESX.
Probably some of you have heard or used EtherChannels in your switching environment, EtherChannels allows you to combine 2 or more switch ports and upto 8 ports max per channel, and treat it as 1 logical port sharing the full bandwidth of all the physical ports assigned to that port channel, this also provides fault-tolerance and high-speed links between switches.
The good news is that this also can be done between a physical switch and an ESX virtual switch as well. In the following example, I will be using a Cisco 2960 Switch and ESXi 5.0. You may need to check with your switch vendor if your switch supports EtherChannel if you are using a Cisco, Port-Sharing if you are using eXtreme Summit switch or Link-Aggregation on Dell PowerConnect switches, different vendors call it different names, regardless what the vendor calls it, the most important is that the switch must support IEEE 802.3ad Standard, because that is the same standard that ESX speaks.
You can then create a VLAN Trunk from that Port Channel to carry multiple VLANs to your ESX server, in other words, you can have 10 VLANs running on just 2 ESX NICs without having a separate physical NIC for each VLAN. This also can be useful if you want to add more bandwidth to the channel by just assigning extra ports on each end point of the channel.
Step1. Configuring the Cisco Switch:
Assuming that my network have 3 VLANs already and they are configured on my Cisco 2950 Switch as shown below, now we need to configure the EtherChannel, I will be using 2 ports Gig1/1 and Gig1/2 as the channel ports and then later on I will configure that EtherChannel as our VLAN Trunk – you may add more ports depending on how much bandwidth you need and how many NICs you have on you ESX server, but for the purpose of this demo, we will be using just 2 ports.
Note that on the Cisco 2950 switch you can have a maximum of 6 EtherChannels per switch, and on a Cisco Catalyst 3550 you can have upto 64 EtherChannels with 8 ports on each, so that means the more ESX servers you have the more advanced switch you will need as you have to create a new EtherChannel for each ESX server you have in you organization. but again, check with your vendor first.
SW01#show vlan brief VLAN Name Status Ports ---- -------------------------------- --------- ------------------------------- 1 default active Gig1/1, Gig1/2 10 OFFICE active Fa0/1, Fa0/2, Fa0/3, Fa0/4 Fa0/5, Fa0/6, Fa0/7, Fa0/8 20 PRODUCTION active Fa0/9, Fa0/10, Fa0/11, Fa0/12 Fa0/13, Fa0/14, Fa0/15, Fa0/16 30 FINANCE active Fa0/17, Fa0/18, Fa0/19, Fa0/20 Fa0/21, Fa0/22, Fa0/23, Fa0/24
a. Configuring your EtherChannel on the Switch:
When configuring your EtherChannel, you need to make sure that all the ports in the channel are configured as Full Duplex and Speed 1000 on both ends, whether the EtherChannel is running between 2 physical switches or if it’s a physical switch to ESX vSwitch, failing to do that will disable the channel. Also, each channel should be assigned a number from 1-6 (on Cisco 2950) or 1-64 on (Catalyst 3550) this is known as the ” channel group” and you must change its mode to “ON” (see example below) this is also known as a “static” mode on other vendors’ switches – this option will keep all the ports on the channel ON all the time on both ends, and this is the same protocol that ESX understands.
Note: DO NOT configure or enable LACP (Link Aggregation Control Protocol) as your channel protocol, ESX does NOT support LACP
SW01#config t Enter configuration commands, one per line. End with CNTL/Z. SW01(config)#int range gig1/1-gig1/2 SW01(config-if-range)#speed 1000 SW01(config-if-range)#duplex full SW01(config-if-range)#channel-group 1 mode on SW01(config-if-range)# no shutdown SW01(config-if-range)#^Z
Verify the EtherChannel Configuration by running “show etherchannel port-channel” command. Note that each channel will be assigned a name starts with “Po” and in our case its “Po1” and the second channel will be Po2 and so on, but most importantly you need to look at the “Ports in the port-channel” section and make sure that you have included the correct ports in you port group.
SW01#show etherchannel port-channel Channel-group listing: ---------------------- Group: 1 ---------- Port-channels in the group: --------------------------- Port-channel: Po1 ------------ Age of the Port-channel = 00d:00h:35m:41s Logical slot/port = 2/1 Number of ports = 2 GC = 0x00000000 HotStandBy port = null Port state = Port-channel Protocol = PAGP Port Security = Disabled Ports in the Port-channel: Index Load Port EC state No of bits ------+------+------+------------------+----------- 0 00 Gig1/1 On 0 0 00 Gig1/2 On 0 Time since last port bundled: 00d:00h:29m:58s Gig1/2
b. Configuring the Trunk on the Switch:
Use Po1 as the trunk that will carry all the VLANs traffic to the ESX vSwitch
SW01(config)#interface port-channel 1 SW01(config-if)#Description ESX_VLAN_TRUNK SW01(config-if)#switchport mode trunk SW01(config-if)#switchport trunk allowed vlan 10,20,30 SW01(config-if)#no shutdown SW01(config-if)#^Z
Verify your trunk configuration
SW01#show interfaces trunk Port Mode Encapsulation Status Native vlan Po1 on 802.1q trunking 1 Port Vlans allowed on trunk Po1 10,20,30 Port Vlans allowed and active in management domain Po1 10,20,30 Port Vlans in spanning tree forwarding state and not pruned Po1 10,20,30
Don’t forget to save your configurations
SW01#wr Building configuration... [OK]
Step2. Configuring the ESX Server:
Now we are done configuring our physical switch, and we have connected our CAT6e cables from ports gig1/1 and gig1/2 on the switch to vmnic2 and vmnic3 on my ESX server, we just need few more steps to get this working, so lets do the fun part.
a. Launch your vSphere Client, select your server, and then click on the “Configuration” tab, click on “Networking”
b. Create a new vSwitch by clicking “Add Networking …” from the top right corner of your screen and add any 2 available NICs, in my case I’ve selected vmnic2 and vmnic3, make sure to select the connection type as “Virtual Machine” when you create the new vSwitch.
c. on your new vSwitch1 click on “properties…” , select the “Ports” tab and then Click on the “Add” button
d. Now you need to add 3 new “Port Groups” that will represent your 3 VLANs, you can give them any name you want, but you have to assign the correct VLAN ID for each to match your Cisco switch VLAN configurations.
e. Your new vSwitch should look like this when you add the port groups and assign the VLAN IDs:
f. Now on the vSwitch you must configure the ports speed and duplexing from your vSwitch Properties to match ports gig1/1-gig1/2 on the Cisco Switch. Under “Network Adapters“, Select each vmnic and Click “Edit“
g. In the last step we just matched the speed and duplexing between the physical switch ports and the ESX ports, but we still need to match the protocol or the standard (IEEE 802.3ad) in order for both end points to understand and communicate with each other.
From the vSwitch1 Properties, select vSwitch from the Ports tab, and then click Edit. Under the NIC Teaming tab change the Load Balancing to “Route Based on IP Hash”
Route based on ip hash selects an uplink based on a hash of the source and destination IP addresses of each packet. For non-IP packets.
Step3. Assigning VMs to VLANs:
Now your Trunk should be working and both ends should understand and communicate with each other, so your next step is that you want to assign each VM in your ESX server to the VLAN that it belongs to in order to get each server to talk to the network.
a. Right-Click on each VM, then select “Edit Settings …“
b. Select the network that will be connected to your Virtual NIC on your VM like shown below.
c. Ping each VLAN interface IP from a Virtual Machine that belongs to the same VLAN to verify that your EtherChannel is working as a Trunk and it can carry the VLAN traffic to the network.