ex0ns

about
A Computer Science Student's blog, about programming, algorithms, hack and a lot more.

LXC - Networking

17 Sep 2015

I've heard a lot about virtualization recently with, at the top, docker. I wanted to understand a bit more about those, and I started with LXC which stands for Linux Containers, the goal of the project was (and is) to provide tools for lightweight virtualization.

Contrary to machine virtualization, which create a whole isolated system (VMWare and VirtualBox for example), LXC uses a system of container, those containers share portions of the host kernel and even part of the host operating system, meaning that containers have less overhead than 'heavy' virtualization. On the other hand, those containers do not guarantee as much isolation (hence security) of the system as machine virtualization does, as always, both have pros and cons, but this is not the topic of this article.

When I started this journey in virtualization, I had no background on system administration nor networking, and even if setting up a LXC containers is really easy, I've had a lot of trouble to connect the containers to the internet, and this is the topic of this article.

Requirement

You will obviously need LXC, I let you use your favorite package manager to get the last version. Then we will create a Ubuntu container with the following command (on arch, there was an issue with debootstrap and ubuntu keyrings, I had to do a yaourt -S ubuntu-keyring and create a symlink for 'gpg1v' : sudo ln -s /usr/bin/gpgv /usr/bin/gpg1v in order to solve this):

sudo lxc-create --name natContainer -t ubuntu

This will simply create us a container named natContainer with an ubuntu on it, we do not need to start it or to do anything with it for now, but if you want more details on LXC basis I recommend the DigitalOcean guide.

By default, containers do not have any internet connection, we will need to create a bridge interface between them and the host so they can be reached from the 'out world'. In fact, we aren't going to create a real bridge (which can be viewed as a virtual switch), but we will be doing NAT to the container (ServerFault post about bridge and NAT).

The main interest of using this technique is that the bridge is not linked to any existing interface, and as Wireless interface can't (at least for what I know) be bridged, it will work the same way almost anywhere.

As I said, we need a bridge interface, we'll see two ways of creating a bridge interface:

  • The first one with netctl a profile based network manager
  • And the other one with systemd-networkd which is the networking tool created by Systemd to setup network configuration with container (which is exactly our case).

I'm using my server as an host, the main interface is eth0 and its IP is my public IP, which is 5.196.95.238 at the time I'm writing this article but we don't really need this at all (as we are not doing Bridged Connection). What we want is to create a subnet on 192.169.100.0/24 and connect our container(s) to it (containers will behave like computer in a local network).

Our previously created container will be at 192.168.100.10 (we are only interested in static configuration) and have a SSH server running on port 22 (with the default ubuntu/ubuntu login), the goal of this article is to establish a ssh connection from my laptop. Here is the following network schema (as you can see, NATint makes my host behaves like a router).

If you already have some networking knowledge, you might want to skip the next two sections, they are meant to be a very brief introduction of some network related element we'll need after.

NAT Bridge

A NAT bridge is a virtual interface which isn't connected to any existing interface (sometimes described as 'standalone interface') it basically creates a private subnet to which we can connect our VMs and containers. In fact, NATting is really common in our home private network, this is one of the main purpose of your router. By default, devices on a private network can't be reach (or reach) the ouside world, but they can reach other machines on the same private network.

When setting up a NAT (on the gateway) all outgoing packets are going to be 'parsed' by the NAT and inserted in a lookup table, this table stores the source ip and port and destination ip and port. With that, when the NAT receives the response, it can redirect the packet to the correct device on the private network. NAT itself it not the topic of this article, so as always, if you're interested, there are a lot of great resources out there.

One thing you need to keep in mind is that devices behind a NAT can't reach (or be reach) from the internet (by default), which is the main difference with the 'Host Bridge'.

Host Bridge

Contrary to the NAT Bridge, this type of bridge are actually 'connected' to an existing physical interface (eth0 in my case). Devices connected to this kind of bridge behave as if there were on the same network. Devices can reach other machines in the network, but also the internet. If a public IP is assigned to the devices in the network, they will even be accessible from the internet, without adding any routing ! In this case the bridge behave exactly as a switch does. In this article we will focus on NAT Bridge (there might be another one about host bridges and virtualization soon).

Creating a bridge using netctl

As I said, netctl is a profile based network manager, to create our new bridge interface, we will add a new file named lxcbridge in /etc/netctl/:

## sudo vim /etc/netctl/lxcbridge
Description="LXC Bridge"
Interface=lxcbr0
Connection=bridge
IP=static
Address='192.168.100.1/24'
SkipForwardingDelay=yes

This file will create lxcbr0 bridge interface with a static IP. To start the interface, we still need to load the profile:

sudo netctl start lxcbridge
sudo netctl enable lxcbridge # To start the interface at boot

You can check the interface's parameters with a simple ip addr:

74: lxcbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default 
    link/ether 3e:11:8b:90:0e:a0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global lxcbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c11:8bff:fe90:ea0/64 scope link 
       valid_lft forever preferred_lft forever

Creating a bridge using networkd

Creating a new interface with networkd requires to add a file in /etc/systemd/network/, namely lxcbridge.netdev:

[NetDev]
Name=lxcbr0
Kind=bridge

This file only contains the data about the virtual network device, we then need to configure it using a second file: lxcbridge.network in the same directory:

[Match]
Name=lxcbr0

[Network]
Address=192.168.100.1/24
Gateway=192.168.100.1
IPForward=kernel # Very important, otherwise no routing and the network will be isolated

We then need to start this inteface:

sudo ip link del dev lxcbr0 # We make sure to delete the interface if it already exists
sudo systemctl stop systemd-networkd #We could also restart it, but I've had issues with restarting services
sudo systemctl start systemd-networkd

You should now be able to see your new interface:

# ip addr
128: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether fa:d3:06:37:22:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.100.1/24 brd 192.168.100.255 scope global lxcbr0
       valid_lft forever preferred_lft forever

LXC Config

To enable the network on our LXC container, we'll have to change add some lines inside our container's configuration. The configuration file for the containers are located in /var/lib/lxc/natContainer/config:

# sudo vim /var/lib/lxc/natContainer/config
lxc.network.type = veth
lxc.network.name = veth0
lxc.network.flags = up
lxc.network.link = lxcbr0
lxc.network.veth.pair = veth0-natContainer
lxc.network.ipv4 = 192.168.100.10/24
lxc.network.ipv4.gateway = 192.168.100.1

The network type specifies that we want to have a Virtual network. The name is the adapter name inside the LXC container.

The link is the related bridge of the host (the one we created earlier) and the pair key will be the name of the LXC interface but from the Host point of view. We then set a static IP to our container (192.168.100.10) as well as the gateway (the host in this case).

Routing

With the previous configurations, if you start the container, it should be inside a private network, meaning that every container can ping the others (and the host).

## On the host
sudo lxc-start --name natContainer
sudo lxc-console --name natContainer
## Login on the container
ping 192.168.100.1

But unfortunately, we still can't reach the internet:

## ping google.fr
ping: unknown host google.fr

We need to add some rules to our host to enable packet routing/forwarding, we will need iptables for that:

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE # Redirect all incoming traffic on NAT interfaces to eth0
sudo sysctl -w net.ipv4.ip_forward=1 # Enable ipv4 forwarding

You'll also need to setup a dns server on the Gateway (the host in our case), I'll use dnsmasq for this.

sudo pacman -S dnsmasq
sudo dnsmasq

It's as easy ! You should now be able to reach any IP from inside your container, the NAT is working ! Even if our NAT is working, we can't reach the container from the internet, meaning that we still can't establish a SSH connection with them, we need to add a new rules to our iptables to redirect incoming traffic to our container:

sudo iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 31000 -j DNAT --to 192.168.100.10:22

From my laptop I should be able to connect to my server's IP(5.196.95.238) on port 3100 and have a SSH shell on the container !

ssh ubuntu@5.196.95.238 -p 31000
ubuntu@ex0ns.me's password: 
ubuntu@ex0ns:~$ whoami
ubuntu
ubuntu@ex0ns:~$ 

Here is the end of this quick overview of LXC and network.

PS: I finally have a PGP key, see my contact page.

Resources