Learning NSX, Part 14: Using Logical Routing

Welcome to part 14 of the Learning NSX blog series, in which I discuss the ability for VMware NSX to do Layer 3 routing in logical networks. This post will also include a look at a very cool feature within VMware NSX known as distributed logical routing. This post will take a closer look at distributed logical routing within the context of an OpenStack environment that’s been integrated with VMware NSX. (Although NSX isn’t necessarily tied to OpenStack, I’ll assume you’re using OpenStack just to simplify the discussion.)

If you’re new to this series, you can find links to all the articles on my Learning NVP/NSX page. Ideally, I’d recommend you read all the articles, but if you’re just interested in some of the high-level concepts you probably don’t need to do that. For those interested in the deep technical details, I’d suggest catching up on the series before proceeding.

Overview of Logical Routing

One of the features of VMware NSX that can be useful, depending on customer requirements, is the ability to create complex network topologies. For example, creating a multi-tier network topology like the one shown below is easily accomplished via VMware NSX:

Sample network topology

Note that this topology has two tenant-specific routing entities—these are logical routers. A logical router is an abstraction created and maintained by VMware NSX on behalf of your cloud management platform (like OpenStack, which I’ll assume you’re using here). These logical entities perform the routing process just like a physical router would (forwarding traffic based on a routing table, changing the source and destination MAC address, maintaining an ARP cache of MAC addresses, decrementing the TTL, etc.). Of course, they are not exactly the same as physical routers; you can’t, for example, connect two logical routers directly to each other.

Logical routers also act as the logical boundary between one or more logical networks and an external network. Logical routers can be connected to multiple logical networks (each logical network with its own logical router interface), but can only be connected to a single external network. Thus, you can’t use a logical router as a transit path between two external networks (two VLANs, for example).

Now that you have a good understanding of logical routing, let’s take a closer look at the various components inside VMware NSX.

Components of Logical Routing

The components are pretty straightforward. In addition to the logical router abstraction that I’ve discussed already, you also have logical router ports (naturally, these are the ports on a logical router that connect it to a logical network or an external network), network address translation (NAT) rules (for handling address translation tasks), and a routing table (for…well, routing).

You can see all of these components in NSX Manager. Once you’re logged into NSX Manager, select Network Components > Logical Layer > Logical Routers, then click on a specific logical router from the list. This will display the screen shown below (click the image for a larger version):

Logical router detail in NSX Manager

A few things to note here:

  • You’ll note that the logical router has a port whose attachment is listed as “L3GW”. This denotes an attachment to a Layer 3 Gateway Service, an entity I described in part 9 of the series. This Layer 3 Gateway Service is itself comprised of two NSX gateway appliances; part 6 in the series discussed how to add a gateway appliance to your installation. The relationship between logical router, Layer 3 Gateway Service, and gateway appliance can be confusing for some; I plan to discuss that in more detail in the next post.
  • This particular logical router is not configured as a distributed logical router. This means that the actual routing function resides on a Layer 3 Gateway Service. The routing functionality is instantiated in a highly available configuration on two different gateway appliances within the Layer 3 Gateway Service.
  • NAT Synchronization is set to on; this refers to keeping NAT state synchronized between the active and standby routing functions instantiated on the gateway appliances.
  • As noted under Replication Mode, this router uses an NSX service node (refer to part 10 for more details on service nodes) for packet replication/BUM traffic.
  • You might notice that one of the logical router ports is assigned the IP address (and you’ll also note a corresponding “no NAT” rule and routing table entries for that same network). Astute readers recognize this as the network for Automatic Private IP Addressing (APIPA), also known as IPv4 Link-Local Addresses per RFC 3927. This exists to support an OpenStack-specific feature known as the metadata service, and is created automatically by OpenStack. (I’ll talk more about OpenStack later in this post.)

All of these components and settings are accessible via the NSX API, and since NSX Manager is completely an API client (it merely consumes NSX APIs and does not provide standalone functionality outside of some logging features), you could create, modify, and delete any of the logical routing components directly within NSX Manager. (Or, if you were so inclined, you could make the API calls yourself to do these tasks.) Typically, though, these tasks would be handled via integration between NSX and your cloud management platform, like OpenStack.

One key component of NSX’s logical routing functionality that you can’t see in NSX Manager is how the routing is actually implemented in the data plane. As with most features in NSX, the actual data plane implementation is handled via Open vSwitch (OVS) and a set of flow rules pushed down by the NSX controllers. These flow rules control the flow of traffic within and between logical networks (logical switches in NSX). You can see some of the flow rules in OVS using the ovs-dpctl dump-flows command, which will produce output something like what’s shown in this screenshot (note that the addresses are highlighted because I used grep to show only the flows matching a certain IP address):

List of flows in OVS

(Click the image above for a larger version.)

These flow rules include actions like re-writing source and destination MAC addresses and decrementing the TTL, both tasks carried out by “normal” routers when routing traffic between networks. These flow rules also provide some insight into the differences between a logical router and a distributed logical router. While both are logical entities, the way in which the data plane is implemented is different for each:

  • For a logical router, the flow rules will direct traffic to the appropriate gateway appliance in the Layer 3 Gateway Service. The logical router is actually instantiated on a gateway appliance, so all routed traffic must go to the logical router, get “routed” (routing table consulted, source and destination MAC re-written, TTL decremented, NAT rules applied, etc.), then get sent on to the final destination (which might be a VM on a hypervisor in NSX or might be a physical network outside of NSX).
  • For a distributed logical router, the flow rules will direct traffic either to the appropriate gateway appliance in the Layer 3 Gateway Service or to the destination hypervisor directly. Why the “either/or”? If the traffic is north/south traffic—that is, traffic being routed out of a logical network onto the physical network—then it must go to the gateway appliance (which, as I have mentioned before, is where traffic is unencapsulated and placed onto the physical network). However, if the traffic is east/west traffic—traffic that is moving from one server on a logical network to another server on a logical network—then the traffic is “routed” directly on the source hypervisor and then sent across an encapsulated connection to the hypervisor where the destination VM resides.

In both cases, there is only one logical router. For a non-distributed logical router, the data plane is instantiated on a gateway appliance only. For a distributed logical router, the data plane is instantiated both on the local hypervisors as well as on a gateway appliance. (This is assuming you’ve set an uplink on the logical router, meaning you have a north/south connection. If you haven’t set an uplink, then the routing functionality is instantiated on the hypervisors only.)

This should provide a good overview of how logical routing is implemented in VMware NSX, but there’s one more aspect I want to cover: logical routers in OpenStack with NSX.

Logical Routers in OpenStack

As you work with OpenStack Networking—Neutron, as it’s commonly called—you’ll find that the abstractions Neutron uses map really well to the abstractions that NSX uses. So, to create a logical router in NSX, you just create a logical router in OpenStack. Attaching an OpenStack logical router to a logical network tells NSX to create the logical switch port, create the logical router port, and connect the two ports together.

In OpenStack, there are a number of different ways to create a logical router:

  • OpenStack Dashboard (Horizon)
  • Command-line interface (CLI)
  • OpenStack Orchestration (Heat) template
  • API calls directly

When using the web-based Dashboard user interface, you can only create centralized logical routers, not distributed logical routers. The Dashboard UI also doesn’t provide any way of knowing if a logical router is distributed or not; for that, you’ll need the CLI (the command is provided shortly).

On a system with the neutron CLI client installed, you can create a logical router like this:

neutron router-create <router name>

This creates a centralized logical router. If you want to create a distributed logical router, it’s as simple as this:

neutron router-create <router name> --distributed True

The neutron router-show command will return output about the specified logical router; that output will tell you if it is a distributed logical router.

The neutron CLI client also offers commands to update a logical router’s routing table (to add or remove static routes, for example), or to connect a logical router to an external network (to set an uplink, in other words).

If you want to create a logical router as part of a stack created via OpenStack Orchestration (Heat), you could use this YAML snippet in a HOT-formatted template to create a distributed logical router (click here if you can’t see the code block below):

OpenStack Heat also offers resource types for setting the router’s external gateway and creating router interfaces (logical router ports). If you aren’t familiar with OpenStack Heat, you might find this introduction useful.

That wraps up this post on logical routing with VMware NSX. As always, I welcome your courteous feedback, so feel free to speak up in the comments below. In the next post, I’ll spend a bit of time discussing logical routers, gateway servies, and gateway appliances. See you next time!

Tags: , , , , , , ,

  1. ubaumann’s avatar

    Hi Scott
    Thanks for your good report. I have two questions.
    Is it possible to deploy the GW VM on every hypervisor and create a “distributed GW”?
    Use NSX with ESXi really OpenVSwitch?

  2. Xue baoping’s avatar

    scott,I always have quesiton about NSX multi-tenants support,especially for layer 3 seperation.For different tenants,Does NSX have to deploy one logical router for one tenant?Gateway router as well,Could one edge router serve for different tenants and still keep tenants seperation?If no,how the NSX support multi-tenant?Thanks.

  3. slowe’s avatar

    Xue, generally each tenant would have a tenant-specific logical router handling that tenant’s logical networks. There are other ways of handling it, but that’s the most common configuration.

  4. Kevin Reeuwijk’s avatar

    Hi Scott,

    out of curiosity, is it possible to configure a DLR to have a particular default gateway IP address (which could be a VM inside the logical network)? As the NVP L3 gateway doesn’t have VPN functions for outbound connectivity, this could be achieved by using a virtual appliance with one vNIC attached to the outside world to deliver those functions (e.g. pfSense or the NSX Edge VM from NSX-v). But if a (distributed) logical router is used in the logical network, it needs to be able to route outbound traffic to a user-specified IP address…

  5. slowe’s avatar

    Kevin, in this case what you would want to do is NOT to attach an external network to the distributed logical router. Then you should be able to assign a static route to the logical router to direct traffic through your custom edge device/VM.

  6. Geoff White’s avatar

    Hi Scott,
    Been searching the web for info on plugging DLRs to Edge Device to get my logical nets connected north bound. I need to achieve this manually first (without Openstack) as I am trying to automate some diagnostics.
    I have an Edge Device that is plugged in, talking to controllers (using STT) for transport. Ist question is when I create the L3 gateway service, I assume in my case the Device ID is breth2? (breth0 is mgmt, breth1 is Tunnel). Breth2 is connected to a port group that is attached to a VSS that is attached to a bonded physical trunk. The port group is assigned to VLAN 97.

    New setup NSX-MH is:
    Physical switch vip = (Vlan97)
    Edge IP = (Vlan97)
    DLR uplink to edge/physical switch = (Vlan97)
    Subnet1 = 10.140.50.x/24
    Subnet2 = 10.140.51.x/24

    So the DLR routes between the .50 and .51 nets just fine. It’s north bound that is not working.
    in setting up the properties for the DLR I have selected Single Default Route, DLR check box is checked, in this case I assume that the Default Gateway IP address is , using service node replication.

    I’ve created an L3 Gateway Service with the sole Edge Device, using breth2 as the Device ID and selecting Basic a single IP

    When the dust clears, I can ping the .253 address which is the uplink of the DLR from any VMs in the .50/.51 nets. But it looks like the Gateway service is not working. Any insight or troubleshooting steps would be appreciated.

  7. Geoff White’s avatar

    Also notice that after a time, the link status of the Logical Router Port attached to the Gateway Service goes from the initial Green to Unknown/down

  8. slowe’s avatar

    Geoff, you seem to be interchangeably mixing concepts from NSX when used in a pure vSphere environment versus NSX in multi-hypervisor environments. I focus primarily on the latter. In multi-hypervisor environments, there is only a single logical routing entity. There is no distinction between a distributed logical router (DLR) and an Edge device; those are NSX-v constructs. Instead, there is only one logical router entity, and it handles both east/west as well as north/south traffic. All you need to do to make north/south traffic work is add a gateway (uplink) to the logical router, and NSX handles the rest.

  9. Geoff White’s avatar

    Hi Scott,
    I’ve made some progress but not quite there yet. I might have mixed the concepts a little but I think I’m past that now. Actually I have more experience with NSX-MH when it was known as NVP :) But since I just passed the VCP-NV it is possible that I got some stuff confused. But as it stands now I seem to still have a problem with the Logical Router, the L3 Gateway Service and the actual Gateway Appliance VM.

    So the current set-up is as follows…

    NSX-MH is:
    Physical switch vip = (Vlan97)
    Gateway Appliance interface connected to vSphere port group (breth2) = (Vlan97)
    this was achieved via a

    set network interface breth2 ip config static

    on the CLI of the Edge Appliance

    There are two east-west logical network connected to the Logical Router
    Subnet1 = 10.140.50.x/24
    Subnet2 = 10.140.51.x/24

    Single Default Route is selected

    The Distributed Logical Router checkbox is checked under Distribution

    Default Gateway entered is

    For the uplink, I’m using Basic with an IP Address of
    L3 Gateway service has been created and the sole Gateway Appliance is attached to the Gateway service.

    From a VM ( on the network i can ping the router interfaces , , I cannot ping above this point.

    From the Edge device, I can tcpdump and see South bound traffic (OSPF Hello, RSTP, etc) North router can also ARP the 254 interface.
    From the Edge CLI I can ping, but not beyond.

    From the switch, I can ping the 254 interface but not beneath it.

    It looks like L2 is working at this level but not L3.
    What it looks like is that The Gateway is not forwarding traffic between the uplink and the rest of the logical networks, east-west forwarding is working.
    Or there is no L3 routing/forwading going on from/around the uplink interface.

    You said you would talk about the confusion between L3 Gateway Services, Logical Routers and Edge devices in “the next post” but that post is about doing it in Openstack and I’m trying to eventually write some API code to do this manually. And I think a few of us might need that explanation from a NSX Manager/API point of view.

    Any suggestions would help. I’ve scoured the web for any other info, even my former co-workers reference you as a resource on MH . So I’m posting here also in hopes that there are others who would benefit from an answer at this level.

    I suppose as a last resort I could delve into the Neutron plug-in but I’d like to avoid that if I can :)


Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>