<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>Shorewall and Multiple Internet Connections</title><link rel="stylesheet" href="html.css" type="text/css" /><meta name="generator" content="DocBook XSL Stylesheets V1.73.2" /></head><body><div class="article" lang="en" xml:lang="en"><div class="titlepage"><div><div><h2 class="title"><a id="id257523"></a>Shorewall and Multiple Internet Connections</h2></div><div><div class="authorgroup"><div class="author"><h3 class="author"><span class="firstname">Tom</span> <span class="surname">Eastep</span></h3></div></div></div><div><p class="copyright">Copyright © 2005, 2006, 2007, 2008 Thomas M. Eastep</p></div><div><div class="legalnotice"><a id="id257917"></a><p>Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled “<span class="quote"><a class="ulink" href="GnuCopyright.htm" target="_self">GNU Free Documentation License</a></span>”.</p></div></div><div><p class="pubdate">2008/12/15</p></div></div><hr /></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#Support">Multiple Internet Connection Support</a></span></dt><dd><dl><dt><span class="section"><a href="#Overview">Overview</a></span></dt><dt><span class="section"><a href="#providers">/etc/shorewall/providers File</a></span></dt><dt><span class="section"><a href="#Providers">What an entry in the Providers File Does</a></span></dt><dt><span class="section"><a href="#Provider_Doesnt">What an entry in the Providers File Does NOT Do</a></span></dt><dt><span class="section"><a href="#Martians">Martians</a></span></dt><dt><span class="section"><a href="#Example1">Example</a></span></dt><dt><span class="section"><a href="#morethan2">More than 2 Providers</a></span></dt><dt><span class="section"><a href="#Local">Applications running on the Firewall</a></span></dt><dt><span class="section"><a href="#IPSEC">IPSEC</a></span></dt><dt><span class="section"><a href="#route_rules">/etc/shorewall/route_rules</a></span></dt><dd><dl><dt><span class="section"><a href="#Routing_rules">Routing Rules</a></span></dt><dt><span class="section"><a href="#route_rules_columns">Columns in the route_rules file</a></span></dt><dt><span class="section"><a href="#Examples">Examples</a></span></dt></dl></dd><dt><span class="section"><a href="#USE_DEFAULT_RT">USE_DEFAULT_RT</a></span></dt><dt><span class="section"><a href="#Shared">Two Providers Sharing an Interface</a></span></dt></dl></dd></dl></div><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Warning</h3><p>This document describes the Multi-ISP facility in Shorewall 4.0 and later. If you are running an earlier release, please see the documentation for that release.</p></div><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Warning</h3><p>Reading just Shorewall documentation is probably not going to give you enough background to use this material. Shorewall may make iptables easy but the Shorewall team doesn't have the resources to be able to spoon-feed Linux policy routing to you (please remember that the user's manual for a tractor doesn't teach you to grow corn either). You will likely need to refer to the following additional information:</p><div class="itemizedlist"><ul type="disc"><li><p>The LARTC HOWTO: <a class="ulink" href="http://www.lartc.org" target="_self">http://www.lartc.org</a></p></li><li><p>Output of <span class="command"><strong>man ip</strong></span></p></li><li><p>Output of <span class="command"><strong>ip route help</strong></span> and <span class="command"><strong>ip rule help</strong></span></p></li></ul></div></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a id="Support"></a>Multiple Internet Connection Support</h2></div></div></div><p>Beginning with Shorewall 2.3.2, limited support is included for multiple Internet connections. Limitations of this support are as follows:</p><div class="itemizedlist"><ul type="disc"><li><p>It utilizes static routing configuration. As such, there is no provision for reacting to the failure of any of the uplinks.</p></li><li><p>The routing changes are made and the route cache is purged when Shorewall is started <span class="bold"><strong>and when Shorewall is restarted</strong></span> (unless you specify the "-n" option to <span class="command"><strong>shorewall restart</strong></span>). Ideally, restarting the packet filter should have no effect on routing.</p></li><li><p>Prior to Shorewall 3.4.0, the routes and route rules added by this support were not completely removed during <span class="command"><strong>shorewall stop</strong></span>, <span class="command"><strong>shorewall clear</strong></span> or <span class="command"><strong>shorewall restart</strong></span>.</p></li><li><p>For most routing applications, <a class="ulink" href="http://www.quagga.net/" target="_self">Quagga</a> is a better solution.</p></li></ul></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Overview"></a>Overview</h3></div></div></div><p>Let's assume that a firewall is connected via two separate Ethernet interfaces to two different ISPs as in the following diagram.</p><div align="center"><table border="0" summary="manufactured viewport for HTML img" cellspacing="0" cellpadding="0"><tr><td align="center" valign="middle"><img src="images/TwoISPs.png" align="middle" /></td></tr></table></div><div class="itemizedlist"><ul type="disc"><li><p>eth0 connects to ISP1. The IP address of eth0 is 206.124.146.176 and the ISP's gateway router has IP address 206.124.146.254.</p></li><li><p>eth1 connects to ISP 2. The IP address of eth1 is 130.252.99.27 and the ISP's gateway router has IP address 130.252.99.254.</p></li><li><p>eth2 connects to the local LAN. Its IP configuration is not relevant to this discussion.</p></li></ul></div><p>Each of these <em class="firstterm">providers</em> is described in an entry in the file <code class="filename">/etc/shorewall/providers</code>.</p><p>Entries in <code class="filename">/etc/shorewall/providers</code> can specify that outgoing connections are to be load-balanced between the two ISPs. Entries in <code class="filename">/etc/shorewall/tcrules</code> can be used to direct particular outgoing connections to one ISP or the other. Use of <code class="filename">/etc/shorewall/tcrules</code> is not required for <code class="filename">/etc/shorewall/providers</code> to work, but you must select a unique MARK value for each provider so Shorewall can set up the correct marking rules for you.</p><p>When you use the <span class="bold"><strong>track</strong></span> option in <code class="filename">/etc/shorewall/providers</code>, connections from the Internet are automatically routed back out of the correct interface and through the correct ISP gateway. This works whether the connection is handled by the firewall itself or if it is routed or port-forwarded to a system behind the firewall.</p><p>Shorewall will set up the routing and will update the <code class="filename">/etc/iproute2/rt_tables</code> to include the table names and numbers of the tables that it adds.</p><div class="caution" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Caution</h3><p>This feature uses <a class="ulink" href="traffic_shaping.htm" target="_self">packet marking</a> to control the routing. As a consequence, there are some restrictions concerning entries in <code class="filename">/etc/shorewall/tcrules</code>:</p><div class="itemizedlist"><ul type="disc"><li><p>Packet marking for traffic control purposes may not be done in the PREROUTING table for connections involving providers with 'track' specified (see below).</p></li><li><p>You may not use the SAVE or RESTORE options.</p></li><li><p>You may not use connection marking.</p></li></ul></div></div><p>The <code class="filename">/etc/shorewall/providers</code> file can also be used in other routing scenarios. See the <a class="ulink" href="Shorewall_Squid_Usage.html" target="_self">Squid documentation</a> for an example.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="providers"></a>/etc/shorewall/providers File</h3></div></div></div><p>Entries in this file have the following columns. As in all Shorewall configuration files, enter "-" in a column if you don't want to enter any value.</p><div class="variablelist"><dl><dt><span class="term">NAME</span></dt><dd><p>The provider name. Must begin with a letter and consist of letters and digits. The provider name becomes the name of the generated routing table for this provider.</p></dd><dt><span class="term">NUMBER</span></dt><dd><p>A number between 1 and 252. This becomes the routing table number for the generated table for this provider.</p></dd><dt><span class="term">MARK</span></dt><dd><p>A mark value used in your /etc/shorewall/tcrules file to direct packets to this provider. Shorewall will also mark connections that have seen input from this provider with this value and will restore the packet mark in the PREROUTING CHAIN. Mark values must be in the range 1-255.</p><p>Beginning with Shorewall version 3.2.0 Beta 6, you may use may set HIGH_ROUTE_MARKS=Yes in <code class="filename">/etc/shorewall/shorewall.conf</code>. This allows you to:</p><div class="itemizedlist"><ul type="disc"><li><p>Use connection marks for traffic shaping, provided that you assign those marks in the FORWARD chain.</p></li><li><p>Use mark values > 255 for provider marks in this column. These mark values must be a multiple of 256 in the range 256-65280 (hex equivalent 0x100 - 0xFF00 with the low-order 8 bits being zero).</p></li></ul></div></dd><dt><span class="term">DUPLICATE</span></dt><dd><p>Gives the name or number of a routing table to duplicate. May be 'main' or the name or number of a previously declared provider. For most applications, you want to specify 'main' here. This field should be be specified as '-' when USE_DEFAULT_RT=Yes in <code class="filename">shorewall.conf</code></p></dd><dt><span class="term">INTERFACE</span></dt><dd><p>The name of the interface to the provider. Where multiple providers share the same interface (which is not recommended), you must follow the name of the interface by a colon (":") and the IP address assigned by this provider (e.g., eth0:206.124.146.176). See <a class="link" href="#Shared" title="Two Providers Sharing an Interface">below</a> for additional considerations.</p></dd><dt><span class="term">GATEWAY</span></dt><dd><p>The IP address of the provider's Gateway router.</p><p>You can enter <span class="bold"><strong>detect</strong></span> here and Shorewall will attempt to automatically determine the gateway IP address.</p><p><span class="bold"><strong>Hint:</strong></span> <span class="bold"><strong>"detect"</strong></span> is appropriate for use in cases where the interface named in the INTERFACE column is dynamically configured via DHCP etc.</p><p>The GATEWAY may be omitted (enter '-') for point-to-point links.</p></dd><dt><span class="term">OPTIONS</span></dt><dd><p>A comma-separated list from the following:</p><div class="variablelist"><dl><dt><span class="term">track</span></dt><dd><p>If specified, connections FROM this interface are to be tracked so that responses may be routed back out this same interface.</p><p>You want to specify 'track' if Internet hosts will be connecting to local servers through this provider. Any time that you specify 'track', you will also want to specify 'balance' (see below).</p><p>Use of this feature requires that your kernel and iptables include CONNMARK target and connmark match support (<span class="bold"><strong>Warning</strong></span>: Standard <span class="trademark">Debian</span>™ and <span class="trademark">Ubuntu</span>™ kernels are lacking that support!).</p><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Warning</h3><p>A <span class="bold"><strong>bug</strong></span> in Shorewall versions 3.2.0-3.2.10, 3.4.0-3.4.6 and 4.0.0-4.0.2 prevents proper handling of PREROUTING marks when HIGH_ROUTE_MARKS=No and the <span class="bold"><strong>track</strong></span> option is specified. Patches are available to correct this problem:</p><p>Shorewall version 3.2.0-3.4.3: <a class="ulink" href="http://www1.shorewall.net/pub/shorewall/3.2/shorewall-3.2.10/errata/patches/Shorewall/patch-3.2.10-2.diff" target="_self">http://www1.shorewall.net/pub/shorewall/3.2/shorewall-3.2.10/errata/patches/Shorewall/patch-3.2.10-2.diff </a></p><p>Shorewall version 3.4.4-3.4.6: <a class="ulink" href="http://www1.shorewall.net/pub/shorewall/3.4/shorewall-3.4.6/errata/patches/Shorewall/patch-3.4.6-1.diff" target="_self">http://www1.shorewall.net/pub/shorewall/3.4/shorewall-3.4.6/errata/patches/Shorewall/patch-3.4.6-1.diff</a></p><p>Shorewall-shell version 4.0.0-4.0.2: <a class="ulink" href="http://www1.shorewall.net/pub/shorewall/4.0/shorewall-4.0.2/errata/patches/Shorewall-shell/patch-shell-4.0.2-2.diff" target="_self">http://www1.shorewall.net/pub/shorewall/4.0/shorewall-4.0.2/errata/patches/Shorewall-shell/patch-shell-4.0.2-2.diff</a></p></div><div class="important" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Important</h3><p>If you are using <code class="filename">/etc/shorewall/providers</code> because you have multiple Internet connections, we recommend that you specify 'track' even if you don't need it. It helps maintain long-term connections in which there are significant periods with no traffic.</p></div></dd><dt><span class="term">balance</span></dt><dd><p>The providers that have 'balance' specified will get outbound traffic load-balanced among them. Balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider.</p><p>By default, each provider is given the same weight (1) . You can change the weight of a given provider by following <span class="emphasis"><em>balance</em></span> with "=" and the desired weight (e.g., balance=2). The weights reflect the relative bandwidth of the providers connections and should be small numbers since the kernel actually creates additional default routes for each weight increment.</p><div class="important" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Important</h3><p>If you are using <code class="filename">/etc/shorewall/providers</code> because you have multiple Internet connections, we recommend that you specify 'balance' even if you don't need it. You can still use entries in <code class="filename">/etc/shorewall/tcrules</code> to force all traffic to one provider or another.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p>If you don't heed this advice then be prepared to read <a class="ulink" href="FAQ.htm#faq57" target="_self">FAQ 57</a> and <a class="ulink" href="FAQ.htm#faq58" target="_self">FAQ 58</a>.</p></div></div><div class="important" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Important</h3><p>If you specify 'balance' and still find that all traffic is going out through only one provider, you may need to install a kernel built with CONFIG_IP_ROUTE_MULTIPATH_CACHED=n. Several users have reported that this change has corrected similar problems.</p><p>The SuSE 10.0 kernel is subject to this problem, and <a class="ulink" href="https://bugzilla.novell.com/show_bug.cgi?id=190908" target="_self"> a kernel oops may result in this circumstance.</a> SUSE 10.1 and SLES 10 have CONFIG_IP_ROUTE_MULTIPATH_CACHED=n set by default. The source of the problem seems to be <a class="ulink" href="http://news.gmane.org/find-root.php?message_id=%3c00da01c5b35a%24b12b9860%241b00a8c0%40cruncher%3e" target="_self">an incompatibility between the LARTC patches and CONFIG_IP_ROUTE_MULTIPATH_CACHED.</a></p></div></dd><dt><span class="term">loose</span></dt><dd><p>Do not include routing rules that force traffic whose source IP is an address of the INTERFACE to be routed to this provider. Useful for defining providers that are to be used only when the appropriate packet mark is applied. Should not be specified together with <span class="bold"><strong>balance</strong></span>.</p></dd><dt><span class="term">optional (added in Shorewall 3.2.2)</span></dt><dd><p>Shorewall will determine of this interface is up and has a configured IPv4 address. If it is not, a warning is issued and this provider is not configured.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p>'optional' is designed to detect interface states that will cause <span class="command"><strong>shorewall start</strong></span> or <span class="command"><strong>shorewall restart</strong></span> to fail; just because an interface is in a state that Shorewall can [re]start without error doesn't mean that traffic can actually be sent through the interface.</p><p>Beginning with Shorewall-perl 4.0.3, you can supply an 'isusable' <a class="ulink" href="shorewall_extension_scripts.htm" target="_self">extension script</a> to extend Shorewall's interface state detection.</p></div></dd><dt><span class="term">src=<em class="replaceable"><code>source-address</code></em> (Added in Shorewall-perl 4.1.5)</span></dt><dd><p>Specifies the source address to use when routing to this provider and none is known (the local client has bound to the 0 address). May not be specified when an <em class="replaceable"><code>address</code></em> is given in the INTERFACE column. If this option is not used, Shorewall substitutes the primary IP address on the interface named in the INTERFACE column.</p></dd><dt><span class="term">mtu=<em class="replaceable"><code>number</code></em> (Added in Shorewall-perl 4.1.5)</span></dt><dd><p>Specifies the MTU when forwarding through this provider. If not given, the MTU of the interface named in the INTERFACE column is assumed.</p></dd></dl></div><p>For those of you who are terminally confused between<span class="bold"><strong> track</strong></span> and <span class="bold"><strong>balance</strong></span>:</p><div class="itemizedlist"><ul type="disc"><li><p><span class="bold"><strong>track</strong></span> governs incoming connections.</p></li><li><p><span class="bold"><strong>balance</strong></span> governs outgoing connections.</p></li></ul></div></dd><dt><span class="term">COPY</span></dt><dd><p>A comma-separated list if interface names. Wildcards specified using an asterisk ("*") are permitted (e.g., tun* ).</p><p>When you specify an existing table in the DUPLICATE column, Shorewall copies all routes through the interface specified in the INTERFACE column plus the interfaces listed in this column. Normally, you will list all interfaces on your firewall in this column except those Internet interfaces specified in the INTERFACE column of entries in this file.</p></dd></dl></div></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Providers"></a>What an entry in the Providers File Does</h3></div></div></div><p>Adding another entry in the providers file simply creates an alternate routing table for you. The table will usually contain two routes:</p><div class="orderedlist"><ol type="1"><li><p>A host route to the specified GATEWAY through the specified INTERFACE.</p></li><li><p>A default route through the GATEWAY.</p></li></ol></div><p>Note that the first route is omitted if "-" is specified as the GATEWAY; in that case, the default route does not specify a gateway (point-to-point link).</p><p>If the DUPLICATE column is non-empty, then routes from the table named in that column are copied into the new table. By default, all routes (except default routes) are copied. The set of routes copied can be restricted using the COPY column which lists the interfaces whose routes you want copied. You will generally want to include all local interfaces in this list. You should exclude the loopback interface (lo) and any interfaces that do not have an IPv4 configuration. You should also omit interfaces like <span class="bold"><strong>tun</strong></span> interfaces that are created dynamically. Traffic to networks handled by those interfaces should be routed through the main table using entries in <code class="filename">/etc/shorewall/route_rules</code> (see Example 2 <a class="link" href="#Examples" title="Examples">below</a>).</p><p>In addition:</p><div class="orderedlist"><ol type="1"><li><p>Unless <span class="bold"><strong>loose</strong></span> is specified, an ip rule is generated for each IP address on the INTERFACE that routes traffic from that address through the associated routing table.</p></li><li><p>If you specify <span class="bold"><strong>track</strong></span>, then connections which have had at least one packet arrive on the interface listed in the INTERFACE column have their connection mark set to the value in the MARK column. In the PREROUTING chain, packets with a connection mark have their packet mark set to the value of the associated connection mark; packets marked in this way bypass any prerouting rules that you create in <code class="filename">/etc/shorewall/tcrules</code>. This ensures that packets associated with connections from outside are always routed out of the correct interface.</p></li><li><p>If you specify <span class="bold"><strong>balance</strong></span>, then Shorewall will replace the 'default' route with weight 100 in the 'main' routing table with a load-balancing route among those gateways where <span class="bold"><strong>balance</strong></span> was specified. So if you configure default routes, be sure that their weight is less than 100 or the route added by Shorewall will not be used.</p></li></ol></div><p>That's <span class="bold"><strong>all</strong></span> that these entries do. You still have to follow the principle stated in the <a class="ulink" href="Shorewall_and_Routing.html" target="_self">Shorewall Routing documentation</a>:</p><div class="orderedlist"><ol type="1"><li><p>Routing determines where packets are to be sent.</p></li><li><p>Once routing determines where the packet is to go, the firewall (Shorewall) determines if the packet is allowed to go there.</p></li></ol></div><p>The bottom line is that if you want traffic to go out through a particular provider then you <span class="emphasis"><em>must </em></span>mark that traffic with the provider's MARK value in <code class="filename">/etc/shorewall/tcrules</code> and you must do that marking in the PREROUTING chain; or, you must provide the appropriate rules in <code class="filename">/etc/shorewall/route_rules</code>.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Provider_Doesnt"></a>What an entry in the Providers File Does NOT Do</h3></div></div></div><p>Given that Shorewall is simply a tool to configure Netfilter and does not run continuously in your system, entries in the providers file <span class="bold"><strong>do not provide any automatic failover in the event of failure of one of your Internet connections</strong></span>.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Martians"></a>Martians</h3></div></div></div><p>One problem that often arises with Multi-ISP configuration is 'Martians'. If your Internet interfaces are configured with the <span class="bold"><strong>routefilter</strong></span> option in <code class="filename">/etc/shorewall/interfaces</code> (remember that if you set that option, you should also select <span class="bold"><strong>logmartians</strong></span>), then things may not work correctly and you will see messages like this:</p><pre class="programlisting">Feb 9 17:23:45 gw.ilinx kernel: martian source 206.124.146.176 from 64.86.88.116, on dev eth1 Feb 9 17:23:45 gw.ilinx kernel: ll header: 00:a0:24:2a:1f:72:00:13:5f:07:97:05:08:00</pre><p>The above message is somewhat awkwardly phrased. The source IP in this incoming packet was 64.86.88.116 and the destination IP address is 206.124.146.176. Another gotcha is that the incoming packet has already had the destination IP address changed for DNAT or because the original outgoing connection was altered by an entry in <code class="filename">/etc/shorewall/masq</code> (SNAT or Masquerade). So the destination IP address (206.124.146.176) may not have been the destination IP address in the packet as it was initially received.</p><p>There a couple of common causes for these problems:</p><div class="orderedlist"><ol type="1"><li><p>You have connected both of your external interfaces to the same hub/switch. Connecting multiple firewall interfaces to a common hub or switch is always a bad idea that will result in hard-to-diagnose problems.</p></li><li><p>You are specifying both the <span class="bold"><strong>loose</strong></span> and <span class="bold"><strong>balance</strong></span> options on your provider(s). This causes individual connections to ping-pong back and forth between the interfaces which is guaranteed to cause problems.</p></li><li><p>You are redirecting traffic from the local system out of one interface or the other using packet marking in your <code class="filename">/etc/shorewall/tcrules</code> file. A better approach is to configure the application to use the appropriate local IP address (the IP address of the interface that you want the application to use). See <a class="link" href="#Local" title="Applications running on the Firewall">below</a>.</p></li></ol></div><p>If all else fails, remove the <span class="bold"><strong>routefilter</strong></span> option from your external interfaces. If you do this, you may wish to add rules to log and drop packets from the Internet that have source addresses in your local networks. For example, if the local LAN in the above diagram is 192.168.1.0/24, then you would add this rule:</p><pre class="programlisting">#ACTION SOURCE DEST DROP:info net:192.168.1.0/24 all</pre><p>Be sure the above rule is added before any other rules with <span class="emphasis"><em>net</em></span> in the SOURCE column.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Example1"></a>Example</h3></div></div></div><p>The configuration in the figure at the top of this section would be specified in <code class="filename">/etc/shorewall/providers</code> as follows.</p><pre class="programlisting">#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY ISP1 1 1 main eth0 206.124.146.254 track,balance eth2 ISP2 2 2 main eth1 130.252.99.254 track,balance eth2</pre><p>Other configuration files go something like this:</p><p><code class="filename">/etc/shorewall/interfaces</code>:</p><pre class="programlisting">#ZONE INTERFACE BROADCAST OPTIONS net eth0 detect … net eth1 detect …</pre><p><code class="filename">/etc/shorewall/policy</code>:</p><pre class="programlisting">#SOURCE DESTINATION POLICY LIMIT:BURST net net DROP</pre><p>Regardless of whether you have masqueraded hosts or not, the following entries are required in <code class="filename">/etc/shorewall/masq</code> if you plan to redirect connections from the firewall using entries in <code class="filename">/etc/shorewall/tcrules</code> or if you specify <span class="bold"><strong>balance</strong></span> on your providers.</p><pre class="programlisting">#INTERFACE SUBNET ADDRESS eth0 130.252.99.27 206.124.146.176 eth1 206.124.146.176 130.252.99.27</pre><p>Those entries ensure that traffic originating on the firewall and redirected via packet marks always has the source IP address corresponding to the interface that it is routed out of.</p><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p>If you have a Dynamic IP address on either of the interfaces, you can use shell variables to construct the above rules. For example, if <code class="filename">eth0</code> had a dynamic IP address, then:</p><p><code class="filename">/etc/shorewall/params</code>:</p><pre class="programlisting">ETH0_IP=$(find_first_interface_address eth0)</pre><p>For optional interfaces, use <span class="emphasis"><em>find_first_interface_address_if_any</em></span> in place of <span class="emphasis"><em>find_first_interface_address</em></span>.</p><p>/etc/shorewall/masq:</p><pre class="programlisting">#INTERFACE SUBNET ADDRESS eth0 130.252.99.27 $ETH0_IP eth1 $ETH0_IP 130.252.99.27</pre></div><p>If you have masqueraded hosts, be sure to update <code class="filename">/etc/shorewall/masq</code> to masquerade to both ISPs. For example, if you masquerade all hosts connected to <code class="filename">eth2</code> then:</p><pre class="programlisting">#INTERFACE SUBNET ADDRESS eth0 eth2 206.124.146.176 eth1 eth2 130.252.99.27</pre><div class="warning" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Warning</h3><p>Entries in <code class="filename">/etc/shorewall/masq</code> have no effect on which ISP a particular connection will be sent through. That is rather the purpose of entries in <code class="filename">/etc/shorewall/tcrules</code> or <code class="filename">/etc/shorewall/route_rules</code>.</p></div><p>Now suppose that you want to route all outgoing SMTP traffic from your local network through ISP 2. You would make this entry in <a class="ulink" href="traffic_shaping.htm" target="_self">/etc/shorewall/tcrules</a> (and if you are running a version of Shorewall earlier than 3.0.0, you would set TC_ENABLED=Yes in <a class="ulink" href="???" target="_self">/etc/shorewall/shorewall.conf</a>).</p><pre class="programlisting">#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST # PORT(S) 2:P <local network> 0.0.0.0/0 tcp 25</pre></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="morethan2"></a>More than 2 Providers</h3></div></div></div><p>When there are more than two providers, you need to extend the two-provider case in the expected way:</p><div class="orderedlist"><ol type="1"><li><p>For each external address, you need an entry in <code class="filename">/etc/shorewall/masq</code> to handle the case where a connection using that address as the SOURCE is sent out of the interfaces other than the one that the address is configured on.</p></li><li><p>For each external interface, you need to add an entry to <code class="filename">/etc/shorewall/masq</code> for each internal network that needs to be masqueraded (or use SNAT) through that interface.</p></li></ol></div><p>If we extend the above example to add eth3 with IP address 16.105.78.4 with gateway 16.105.78.254, then:</p><p><code class="filename">/etc/shorewall/providers</code>:</p><pre class="programlisting">#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY ISP1 1 1 main eth0 206.124.146.254 track,balance eth2 ISP2 2 2 main eth1 130.252.99.254 track,balance eth2 ISP3 3 3 main eth3 16.105.78.254 track,balance eth2</pre><p><code class="filename">/etc/shorewall/masq</code>:</p><pre class="programlisting">#INTERFACE SUBNET ADDRESS eth0 130.252.99.27 206.124.146.176 eth3 130.252.99.27 16.105.78.4 eth1 206.124.146.176 130.252.99.27 eth3 206.124.146.176 16.105.78.4 eth0 16.106.78.4 206.124.146.176 eth1 16.106.78.4 130.252.99.27 eth0 eth2 206.124.146.176 eth1 eth2 130.252.99.27 eth3 eth2 16.105.78.4</pre></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Local"></a>Applications running on the Firewall</h3></div></div></div><p>Experience has shown that in some cases, problems occur with applications running on the firewall itself. This is especially true when you have specified <span class="bold"><strong>routefilter</strong></span> on your external interfaces in /etc/shorewall/interfaces (see <a class="link" href="#Martians" title="Martians">above</a>). When this happens, it is suggested that you have the application use specific local IP addresses rather than 0.</p><p>Examples:</p><div class="itemizedlist"><ul type="disc"><li><p>Squid: In <code class="filename">squid.conf</code>, set <span class="bold"><strong>tcp_outgoing_address</strong></span> to the IP address of the interface that you want Squid to use.</p></li><li><p>In OpenVPN, set <span class="bold"><strong>local </strong></span>(<span class="bold"><strong>--local</strong></span> on the command line) to the IP address that you want the server to receive connections on.</p></li></ul></div><p>Note that some traffic originating on the firewall doesn't have a SOURCE IP address before routing. At least one Shorewall user reports that an entry in <code class="filename">/etc/shorewall/route_rules</code> with 'lo' in the SOURCE column seems to be the most reliable way to direct such traffic to a particular ISP.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="IPSEC"></a>IPSEC</h3></div></div></div><p>If you have an IPSEC gateway on your firewall, be sure to arrange for ESP packets to be routed out of the same interface that you have configured your keying daemon to use.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="route_rules"></a>/etc/shorewall/route_rules</h3></div></div></div><p>The <code class="filename">/etc/shorewall/route_rules</code> file was added in Shorewall version 3.2.0. The <code class="filename">route_rules</code> file allows assigning certain traffic to a particular provider just as entries in the <code class="filename">tcrules</code> file. The difference between the two files is that entries in <code class="filename">route_rules</code> are independent of Netfilter.</p><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="Routing_rules"></a>Routing Rules</h4></div></div></div><p>Routing rules are maintained by the Linux kernel and can be displayed using the <span class="command"><strong>ip rule ls</strong></span> command. When routing a packet, the rules are processed in turn until the packet is successfully routed.</p><pre class="programlisting">gateway:~ # <span class="command"><strong>ip rule ls</strong></span> 0: from all lookup local <=== Local (to the firewall) IP addresses 10001: from all fwmark 0x1 lookup Blarg <=== This and the next rule are generated by the 10002: from all fwmark 0x2 lookup Comcast 'MARK' values in /etc/shorewall/providers. 20000: from 206.124.146.176 lookup Blarg <=== This and the next rule are generated unless 20256: from 24.12.22.33 lookup Comcast 'loose' is specified; based in the output of 'ip addr ls' 32766: from all lookup main <=== This is the routing table shown by 'iproute -n' 32767: from all lookup default <=== This table is usually empty gateway:~ #</pre><p>In the above example, there are two providers: Blarg and Comcast with MARK 1 going to Blarg and mark 2 going to Comcast.</p></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="route_rules_columns"></a>Columns in the route_rules file</h4></div></div></div><p>Columns in the file are:</p><div class="variablelist"><dl><dt><span class="term">SOURCE (Optional)</span></dt><dd><p>An ip address (network or host) that matches the source IP address in a packet. May also be specified as an interface name optionally followed by ":" and an address. If the device 'lo' is specified, the packet must originate from the firewall itself.</p></dd><dt><span class="term">DEST (Optional)</span></dt><dd><p>An ip address (network or host) that matches the destination IP address in a packet.</p><p>If you choose to omit either SOURCE or DEST, place "-" in that column. Note that you may not omit both SOURCE and DEST.</p></dd><dt><span class="term">PROVIDER</span></dt><dd><p>The provider to route the traffic through. May be expressed either as the provider name or the provider number.</p></dd><dt><span class="term">PRIORITY</span></dt><dd><p>The rule's priority which determines the order in which the rules are processed.</p><p>1000-1999 Before Shorewall-generated 'MARK' rules</p><p>11000- 11999 After 'MARK' rules but before Shorewall-generated rules for ISP interfaces.</p><p>26000-26999 After ISP interface rules but before 'default' rule.</p><p>Rules with equal priority are applied in the order in which they appear in the file.</p></dd></dl></div></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h4 class="title"><a id="Examples"></a>Examples</h4></div></div></div><p><span class="bold"><strong>Example 1:</strong></span> You want all traffic entering the firewall on eth1 to be routed through Comcast.</p><pre class="programlisting">#SOURCE DEST PROVIDER PRIORITY eth1 - Comcast 1000</pre><p>With this entry, the output of <span class="command"><strong>ip rule ls</strong></span> would be as follows.</p><pre class="programlisting">gateway:~ # <span class="command"><strong>ip rule ls</strong></span> 0: from all lookup local 1000: from all iif eth1 lookup Comcast 10001: from all fwmark 0x1 lookup Blarg 10002: from all fwmark 0x2 lookup Comcast 20000: from 206.124.146.176 lookup Blarg 20256: from 24.12.22.33 lookup Comcast 32766: from all lookup main 32767: from all lookup default gateway:~ #</pre><p>Note that because we used a priority of 1000, the test for <code class="filename">eth1</code> is inserted before the fwmark tests.</p><p><a id="Openvpn"></a><span class="bold"><strong>Example 2:</strong></span> You use OpenVPN (routed setup w/tunX) in combination with multiple providers. In this case you have to set up a rule to ensure that the OpenVPN traffic is routed back through the tunX interface(s) rather than through any of the providers. 10.8.0.0/24 is the subnet chosen in your OpenVPN configuration (server 10.8.0.0 255.255.255.0).</p><pre class="programlisting">#SOURCE DEST PROVIDER PRIORITY - 10.8.0.0/24 main 1000</pre></div></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="USE_DEFAULT_RT"></a>USE_DEFAULT_RT</h3></div></div></div><p>Beginning with Shorewall 4.2.0 Beta3, Shorewall-perl has supported a USE_DEFAULT_RT option in <a class="ulink" href="manpages/shorewall.conf.html" target="_self">shorewall.conf</a> (5).</p><p>One of the drawbacks of the Mulit-ISP support as described in the preceding sections is that changes to the main table made by applications are not added to the individual provider tables. This makes route rules such as described in <a class="link" href="#Openvpn">one of the examples above</a> necessary.</p><p>USE_DEFAULT_RT=Yes works around that problem by passing packets through the main table first rather than last. This has a number of implications:</p><div class="orderedlist"><ol type="1"><li><p>Both the DUPLICATE and the COPY columns in the providers file must remain empty or contain "-". The individual provider routing tables generated when USE_DEFAULT_RT=Yes contain only a host route to the gateway and a default route via the gateway.</p></li><li><p>The <span class="bold"><strong>balance</strong></span> option is assumed for all interfaces that do not have the <span class="bold"><strong>loose</strong></span> option. When you want both <span class="bold"><strong>balance</strong></span> and <span class="bold"><strong>loose</strong></span>, both must be specified.</p></li><li><p>The default route generated by Shorewall is added to the <span class="emphasis"><em>default</em></span> routing table (253) rather than to the main routing table (254).</p></li><li><p>Packets are sent through the main routing table by a routing rule with priority 999. In ), the priority range 1-998 may be used for inserting rules that bypass the main table.</p></li><li><p>All provider gateways must be specified explicitly in the GATEWAY column. 'detect' may not be specified. Note that for ppp interfaces, the GATEWAY may remain unspecified ("-").</p></li><li><p>You should disable all default route management outside of Shorewall. If a default route is inadvertently added to the main table while Shorewall is started, then all policy routing will stop working except for those routing rules in the priority range 1-998.</p></li></ol></div><p>Although 'balance' is automatically assumed when USE_DEFAULT_RT=Yes, you can easily cause all traffic to use one provider except when you explicitly direct it to use the other provider via <a class="ulink" href="manpages/shorewall-route_rules.html" target="_self">shorewall-route_rules</a> (5) or <a class="ulink" href="manpages/shorewall-tcrules.html" target="_self">shorewall-tcrules</a> (5).</p><p>Example (send all traffic through the 'shorewall' provider unless otherwise directed).</p><p>/etc/shorewall/providers:</p><pre class="programlisting">#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS linksys 1 1 - wlan0 172.20.1.1 track,balance=1,optional shorewall 2 2 - eth0 192.168.1.254 track,balance=2,optional</pre><p>/etc/shorewall/route_rules:</p><pre class="programlisting">#SOURCE DEST PROVIDER PRIORITY - - shorewall 11999</pre></div><div class="section" lang="en" xml:lang="en"><div class="titlepage"><div><div><h3 class="title"><a id="Shared"></a>Two Providers Sharing an Interface</h3></div></div></div><p>Shared interface support is available only in Shorewall-perl 4.2.0 and later.</p><div class="orderedlist"><ol type="a"><li><p>Only Ethernet (or Ethernet-like) interfaces can be used. For inbound traffic, the MAC addresses of the gateway routers are used to determine which provider a packet was received through. Note that only routed traffic can be categorized using this technique.</p></li><li><p>You must specify the address on the interface that corresponds to a particular provider in the INTERFACE column by following the interface name with a colon (":") and the address.</p></li><li><p>Entries in <code class="filename">/etc/shorewall/masq</code> must be qualified by the provider name (or number).</p></li><li><p>This feature requires Realm Match support in your kernel and iptables.</p></li><li><p>You must add route_rules entries for networks that are accessed through a particular provider.</p></li><li><p>If you have additional IP addresses through either provider, you must add <code class="filename">route_rules</code> to direct traffic FROM each of those addresses through the appropriate provider.</p></li><li><p>You must manually add MARK rules for traffic known to come from each provider.</p></li><li><p>You must specify a gateway IP address in the GATEWAY column of<code class="filename"> /etc/shorewall/providers</code>; <span class="bold"><strong>detect</strong></span> is not permitted.</p></li></ol></div><p>Taken together, b. and h. effectively preclude using this technique with dynamic IP addresses.</p><p>Example:</p><p>This is our home network circa fall 2008. We have two internet providers:</p><div class="orderedlist"><ol type="1"><li><p>Comcast -- Cable modem with one dynamic IP address.</p></li><li><p>Avvanta -- ADSL with 5 static IP addresses.</p></li></ol></div><p>Because the old <span class="trademark">Compaq</span>™ <span class="trademark">Presario</span>™ that I use for a firewall only has three PCI slots and no onboard Ethernet, it doesn't have enough Ethernet controllers to support both providers. So I use a Linksys WRT300n pre-N router as a gateway to Comcast. Note that because the Comcast IP address is dynamic, I could not share a single firewall interface between the two providers directly.</p><p>On my personal laptop (ursa), I have 9 virtual machines running various Linux distributions. <span class="emphasis"><em>It is the Shorewall configuration on ursa that I will describe here</em></span>.</p><p>Below is a diagram of our network:</p><div align="center"><img src="images/Network2008a.png" align="middle" /></div><p>The local wired network in my office is connected to both gateways and uses the private (RFC 1918) network 172.20.1.0/24. The Comcast gateway has local IP address 172.20.1.1 while the Avvanta gateway has local IP address 172.20.1.254. Ursa's eth0 interface has a single IP address (172.20.1.130).</p><p>This configuration uses USE_DEFAULT_RT=Yes in <code class="filename">shorewall.conf </code>(see <a class="link" href="#USE_DEFAULT_RT" title="USE_DEFAULT_RT">above</a>).</p><p>Here is the <code class="filename">providers</code> file:</p><pre class="programlisting">#NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY comcast 1 1 - eth0:172.20.1.130 172.20.1.1 track,loose,balance,optional avvanta 2 2 - eth0:172.20.1.130 172.20.1.254 track,optional,loose wireless 3 3 - wlan0 172.20.1.1 track,optional</pre><p>Several things to note:</p><div class="orderedlist"><ol type="1"><li><p>172.20.1.130 is specified as the <code class="filename">eth0</code> IP address for both providers.</p></li><li><p>Both wired providers have the <span class="bold"><strong>loose</strong></span> option. This prevents Shorewall from automatically generating routing rules based on the source IP address.</p></li><li><p>Only <span class="bold"><strong>comcast</strong></span> has the <span class="bold"><strong>balance</strong></span> option. With USE_DEFAULT_RT=yes, that means that <span class="bold"><strong>comcast</strong></span> will be the default provider. While <span class="bold"><strong>balance</strong></span> is the default, with USE_DEFAULT_RT=Yes, it must be specified explicitly when <span class="bold"><strong>loose</strong></span> is also specified.</p></li><li><p>I always disable the <span class="bold"><strong>wireless</strong></span> interface when the laptop is connected to the wired network.</p></li><li><p>I use a different Shorewall configuration when I take the laptop on the road.</p></li></ol></div><p>Here is the route_rules file:</p><pre class="programlisting">#SOURCE DEST PROVIDER PRIORITY - 206.124.146.176/31 avvanta 1000 - 206.124.146.178/31 avvanta 1000 - 206.124.146.180/32 avvanta 1000</pre><p>Those rules direct traffic to the five static Avvanta IP addresses (only two are currently used) through the <span class="bold"><strong>avvanta</strong></span> provider.</p><p>Here is the tcrules file (MARK_IN_FORWARD_CHAIN=No in <code class="filename">shorewall.conf</code>):</p><pre class="programlisting">#MARK SOURCE DEST PROTO PORT(S) CLIENT USER TEST LENGTH TOS CONNBYTES HELPER # PORT(S) 2 $FW 0.0.0.0/0 tcp 21 2 $FW 0.0.0.0/0 tcp - - - - - - - ftp 2 $FW 0.0.0.0/0 tcp 119</pre><p>These rules:</p><div class="itemizedlist"><ul type="disc"><li><p>Use <span class="bold"><strong>avvanta</strong></span> for FTP.</p></li><li><p>Use <span class="bold"><strong>avvanta</strong></span> for NTTP</p></li></ul></div><p>The remaining files are for a rather standard two-interface config with a bridge as the local interface.</p><p><code class="filename">zones</code>:</p><pre class="programlisting">#ZONE IPSEC OPTIONS IN OUT # ONLY OPTIONS OPTIONS fw firewall net ipv4 kvm ipv4</pre><p><code class="filename">policy</code>:</p><pre class="programlisting">net net NONE fw net ACCEPT fw kvm ACCEPT kvm all ACCEPT net all DROP info all all REJECT info</pre><p>interfaces:</p><pre class="programlisting">#ZONE INTERFACE BROADCAST OPTIONS GATEWAY # net eth0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional,arp_ignore net wlan0 detect dhcp,tcpflags,routefilter,blacklist,logmartians,optional kvm br0 detect routeback #Virtual Machines</pre><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p><code class="filename">wlan0</code> is the wireless adapter in the notebook. Used when the laptop is in our home but not connected to the wired network.</p></div><p>masq:</p><pre class="programlisting">#INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth0 192.168.0.0/24 wlan0 192.168.0.0/24</pre><div class="note" style="margin-left: 0.5in; margin-right: 0.5in;"><h3 class="title">Note</h3><p>Because the firewall has only a single external IP address, I don't need to specify the providers in the masq rules.</p></div></div></div></div></body></html>