General Computing
firewall ssl iis static-ip load-balancer
Updated Mon, 15 Aug 2022 20:37:44 GMT

Http Request after passing through firewall and loadbalancer


I have a Firewall, then a LoadBalancer (IIS-ARR load balancer) and 2 IIS webservers. The 2 IIS web servers will be in LAN with private IPs. My website is hosted on both the webservers. My actual challenge is, I have a payment gateway in my website, and it will work as expected only if the request comes to the website from the website name (like www.abc.com) or public static IP configured to IIS-ARR load balancer.

I have 4 queries:

  1. When http request is handled by any of the 2 web servers, will the request come from website name/public static IP assigned to loadbalancer. Or it will be the WebServers Private IP.

  2. Is SSL offloading at IIS-ARR load balancer prone to any attack as the http request flows from load balancer to the real server in clear text within the LAN.

  3. Where to create SSL CSR request for my website. On IIS-ARR server or any of two WebServers. How many SSL certificates are required.

  4. How to retain https (SSL) throughout the request. From client browser to the firewall, then load balancer, then the real server. (Without SSL offloading)




Solution

1. The client will expect the answer to come from within the same ssl session as the one which it believes it has initiated with the firewall external address.

The client does not know if the firewall passes the tcp stream on to a different device such as an ssl terminating load balancer. Nor does it know if the load balancer passes on the terminated ssl session to an internal backend server (regardless of whether the load balancer passes the data to the backend server in reencrypted https or unencrypted http form). The client just knows that it somehow established an ssl session with an ip address, which happens to be the external ip of a firewall.

Through the layers of firewall and load balancer and ssl terminations, the request makes it all the way to a backend server. However, as the backend prepares the response, if the backend server looks at the sender ip address and sees the client external ip address there, it will respond directly to the ip address of the client. A response sent directly from backend to client would be received outside of the ssl session which the client has initiated and sent the request through. The client, naturally, does not expect such a response and will reject it.

So to ensure that the answer arrives by way of the client initiated ssl session, the load balancer must tweak the request before passing it on to the backend server.

It first decrypts the client ssl session, then modifies the original request so that the source ip address is overwritten with a source ip address belonging to the load balancer, before sending the request to the backend server.

The backend server now believes the request originated from the load balancer and sends its response to the load balancer instead of to the original client.

The load balancer again modifies the data, so that the response appears to come from the load balancer instead of from the backend server. After which the load balancer encrypts the data to be within the same ssl session it has established with the client and proceeds to send the response directly to the client.

The client happily accepts this and is oblivious to the real network paths taken to produce the response.

These ip modifications done by the load balancer are termed Source NAT (SNAT) and are common to all load balancers I have ever worked with.

For brevity I have not included the firewall translations between public and private address spaces. I suggest separating that question entirely so as not to confuse the rewrite done by the firewall with the rewrite done by the load balancer. This as the firewall rewrite can be done in a number of ways and merits its own question, once the choice of firewall brand has been decided or narrowed down. Until then think of it as magic which just happen in the firewall as each inbound or outbound packet passes through it.

A simple way of verifying a correct setup as described above would be to start off by using an internal client, and configuring unencrypted http sessions between client and load balancer, as well as between load balancer and backend server(s).

Using a packet sniffer such as Wireshark on the client, load balancer and backend, one can then see the effect of these rewrites in practice for any given request/response pair and for each of the network parts.

Once the setup is working and the process is understood, one can encrypt first the client-to load balancer path, then the load balancer-to-backend path. This as a suggestion for easing the learning curve and promoting correct final configuration.

One caveat lies in the backend server perception of the requests.

Regardless of the actual number of external clients, the backend will see and log only one client: the internal load balancer SNAT address.

This dilemma is produced by having the load balancer SNAT the requests, and is solved by having the load balancer inform the backend server of the actual client external ip address. As the request source ip itself is modified, the information of the real client ip address is instead passed to the backend by inserting an http header into the http request.

The header can have any valid name not already in use, a common choice is X-FORWARDED-FOR.

Configuring the load balancer to insert such a header is the first requirement for this fix to work, the second requirement is informing the backend server of the presence of this header. Configuration is specific to the brands of load balancer and backend server, which is easily googled. Here is for example how to configure a tomcat backend to log from x-forwarded-for. It has been a long time since I last configured an ARR and cannot recall by memory how x-forwarded-for was added, but recollect it took some experimenting and a bit of googling to get to work.

2) Yes, as the traffic can be sniffed by a protocol decoder such as Wireshark as suggested above, there is an attack vector.

This presumes an attacker to have access to the network traffic.

If the load balancer-to-backend traffic is sent in clear text, troubleshooting load balancer or backend server misconfiguration is easier, but carries the aforementioned risk.

How to make this design choice is a worthwhile discussion to have internally as well as with any external stakeholders.

3) The CSR is commonly made where SSL is terminated.

To encrypt the client-to load balancer traffic, create the csr request on the load balancer.

To encrypt the load balancer-to-backend traffic, create the csr on the backend server.

There are ways of exporting both a signed certificate and a corresponding private key as a bundle, which can be imported on a different server. This is useful for configuring a load balancer cluster where all members are desired to present the same certificate, or for configuring multiple identical backends to simplify load balancer client ssl configuration (that is, where the load balancer functions as an ssl client to the backend server as it reencrypts the http data).

4) Decide where client ssl termination should be done.

It is possible to terminate SSL in some firewalls, but more common is to just forward the tcp stream through the firewall to the load balancer which then terminates the client ssl session.

It is also possible to have the load balancer load balance pure tcp streams at which the backend server terminates ssl. This is uncommon and I will not explore the option here.

Once the initial ssl has been terminated decide if the data should be reencrypted, for example between the load balancer and the next jump. The next jump can be a backend server or another load balancer or...

Repeat until such a step as where data should be sent in clear or you reach the last server in the chain.

Terminating SSL is a requirement for the load balancer to be able to for example insert an http x-forwarded-for header, or to do other things which require access to clear text data. Therefore it is common to terminate SSL before the load balancer, or on the load balancer, or both.

It is also common both to send traffic to the backends encrypted and to send it unencrypted. It simply depends on the circumstance of the organisation and the purpose of the data sent.

SSL Offloading is just a term describing a process where one piece of tech does the SSL encrypting/decrypting in place of another piece of tech.

It can be a load balancer decrypting ssl and passing clear text to a backend - the backend has been ssl offloaded.

It can be a load balancer having specialized hardware circuits dedicated to ssl encryption/decryption - the load balancer CPU has been ssl offloaded. And so on...





Comments (5)

  • +0 – Thanks a lot for a detailed answer. This has cleared all my doubt's. :-) — Jul 13, 2020 at 23:03  
  • +0 – After reading about IIS-ARR, I came to know that it can be configured as Reverse-Proxy, probably to do all the tasks you have mentioned and the SNAT which you have mentioned (x-forwarded-for header). — Jul 13, 2020 at 23:09  
  • +0 – By chance if I get stuck any where while setting up the environment and I have any query, is there any way in stack exchange to directly reach or notify you for help. — Jul 13, 2020 at 23:16  
  • +0 – Thanks! I believe you can always ping me in a comment. But I do this kind of thing only when I have time and feel like it (as I work with this tech for a living and most often do other things when I'm not working). — Jul 14, 2020 at 10:52  
  • +1 – OK. Thanks. I have posted a query on stack overflow for SSL. "SSL for ARR load balancer to manage 2 websites". You can clarify in ur free time if possible. — Jul 14, 2020 at 11:08