Load balancer and ADC integration

DartRelay works behind a load balancer or application delivery controller (ADC) as an ordinary web application on an address of its own, optionally with the appliance signing people in first. This page lists what any appliance must do, explains the console settings that go with it, and gives configuration examples for the common products.

How DartRelay fits behind an appliance

There are two patterns, and every product below is one of them:

• Plain load balancing. The appliance terminates HTTPS, spreads visitors across your DartRelay servers and keeps each on one server. People see DartRelay's own sign-in page.
• The appliance signs people in. As well as load balancing, the appliance shows its own sign-in page (often with its own two-factor authentication), then passes the person into DartRelay so they do not type their password twice.

The same rules apply in front of a single DartRelay server as in front of a pool. For pools, see Running several servers (pools).

What is not supported

• Publishing under a path of another site. DartRelay needs an address of its own, such as https://relay.example.com/. Publishing it as https://www.example.com/relay/ is not supported: sign-in and several other redirects go to the root of the site and leave the path.
• Citrix NetScaler Gateway in front of DartRelay. None of NetScaler Gateway's modes can carry a DartRelay session. In clientless access the portal appears and an application can be chosen, but the session connection is never answered, so nothing opens. The ICA proxy and full VPN modes do not work either. Run NetScaler Gateway alongside DartRelay if you need it for other things, and use a NetScaler load-balancing virtual server in front of DartRelay, as described below.

What any load balancer must do

Whatever the product, check each of these. Most problems behind a load balancer come from one of the first three.

The System → Load balancer page

All load balancer settings that should be the same on every server are kept on one page, System → Load balancer, and stored in the database. A change made on one server reaches the others within about 30 seconds. Saving needs permission to manage both endpoints and security settings.

The page also shows how DartRelay sees your own connection: the address it arrived from and whether that is a trusted proxy, the client address it worked out, and the scheme. Open the page through the load balancer to confirm the forwarded headers are being read correctly.

Trusted proxies

Enter the address your load balancer uses when it connects to the DartRelay servers. On some appliances this is not the virtual address people browse to: on Citrix NetScaler, for example, it is the subnet IP (SNIP). The page refuses a save that would cause trouble:

• a range that is too broad (wider than /8 for IPv4 or /48 for IPv6);
• header sign-in switched on with no trusted proxy;
• a list that leaves out the proxy you are connected through, which would lock you out of the page you are using.

A setting that differs on one server

A value written in a server's own configuration file overrides the console on that server, and the field shows set on this server. Use this only to recover from a mistake; remove it afterwards so all servers match.

When the appliance signs people in

There are two ways to pass a person from the appliance into DartRelay.

Form sign-in

The appliance posts the username and password it collected to DartRelay's sign-in form at /portal/login, using the fields Input.Username and Input.Password. Because DartRelay receives the password, Resources that pass the user's password through to Windows keep working. If the appliance builds this post from a fixed template, tick Let the load balancer fill in the sign-in form.

Header sign-in

The appliance sends the authenticated username in a header (for example X-Forwarded-User), and DartRelay signs that person in. Switch on Sign-in by header and name the header; optionally name a second header for the display name. The appliance must remove any copy of these headers a visitor sends. Header sign-in carries no password, so Resources that pass the user's password through to Windows are refused for these sessions; use form sign-in if you need them.

Signing out of both

If the appliance signed someone in, signing out of DartRelay alone leaves the appliance's session alive, and the next visit lets them straight back in. To end both, give the address a sign-out address:

1. Open the address's binding. Go to Appearance → Tenancy & Branding and open the binding for the address people use. If the address has no binding yet, add one; see Tenancy & Branding.
2. Set Sign-out address. Enter the appliance's sign-out URL (see each product below) and save.

When someone signs out, DartRelay first ends its own session, then sends the browser to that address so the appliance can end its session too. Addresses with no sign-out address of their own use the Sign-out address on System → Load balancer.

Citrix NetScaler

Use a load-balancing virtual server. You can add AAA authentication to it so NetScaler signs people in first.

Plain load balancing

add serviceGroup sg_dartrelay HTTP
bind serviceGroup sg_dartrelay 10.0.0.11 8080
bind serviceGroup sg_dartrelay 10.0.0.12 8080

add lb monitor mon_dartrelay_ready HTTP -respCode 200 -httpRequest "GET /health/ready"
bind serviceGroup sg_dartrelay -monitorName mon_dartrelay_ready
set serviceGroup sg_dartrelay -downStateFlush DISABLED -svrTimeout 3600

add ns httpProfile http_dartrelay -webSocket ENABLED

add lb vserver vs_dartrelay_ssl SSL 10.0.0.10 443 -persistenceType COOKIEINSERT -lbMethod LEASTCONNECTION -cltTimeout 3600 -httpProfileName http_dartrelay
bind lb vserver vs_dartrelay_ssl sg_dartrelay
bind ssl vserver vs_dartrelay_ssl -certkeyName relay_cert

Add a virtual server on port 80 that redirects to HTTPS, and rewrite policies that replace the forwarded headers:

add rewrite action rwa_xff_del delete_http_header X-Forwarded-For
add rewrite action rwa_xff_set insert_http_header X-Forwarded-For CLIENT.IP.SRC
add rewrite action rwa_xfp_del delete_http_header X-Forwarded-Proto
add rewrite action rwa_xfp_set insert_http_header X-Forwarded-Proto "\"https\""
add rewrite policy rwp_xff_del true rwa_xff_del
add rewrite policy rwp_xff_set true rwa_xff_set
add rewrite policy rwp_xfp_del true rwa_xfp_del
add rewrite policy rwp_xfp_set true rwa_xfp_set
bind lb vserver vs_dartrelay_ssl -policyName rwp_xff_del -priority 100 -gotoPriorityExpression NEXT -type REQUEST
bind lb vserver vs_dartrelay_ssl -policyName rwp_xff_set -priority 110 -gotoPriorityExpression NEXT -type REQUEST
bind lb vserver vs_dartrelay_ssl -policyName rwp_xfp_del -priority 120 -gotoPriorityExpression NEXT -type REQUEST
bind lb vserver vs_dartrelay_ssl -policyName rwp_xfp_set -priority 130 -gotoPriorityExpression END -type REQUEST

In DartRelay, add NetScaler's subnet IP (SNIP) to Trusted proxies. -downStateFlush DISABLED is what keeps existing sessions running when a drained server is marked down.

NetScaler signs people in (AAA)

Bind an authentication virtual server to the load-balancing virtual server through an authentication profile.

add authentication vserver av_dartrelay SSL 10.0.0.10 4443
bind ssl vserver av_dartrelay -certkeyName relay_cert
bind authentication vserver av_dartrelay -policy pol_ldap_corp -priority 100 -gotoPriorityExpression NEXT

add authentication authnProfile ap_dartrelay -authnVsName av_dartrelay -authenticationHost relay.example.com:4443
set lb vserver vs_dartrelay_ssl -authentication ON -authnProfile ap_dartrelay

After sign-in, NetScaler may send people to /vpn/tmindex.html, which DartRelay does not have. A responder policy sends them to the portal instead (a session action's home page setting alone does not fix this):

add responder action rsa_relay_home redirect "\"https://\" + HTTP.REQ.HOSTNAME.HTTP_URL_SAFE + \"/portal\"" -responseStatusCode 302
add responder policy rsp_relay_home "HTTP.REQ.URL.PATH.SET_TEXT_MODE(IGNORECASE).EQ(\"/vpn/tmindex.html\")" rsa_relay_home
bind lb vserver vs_dartrelay_ssl -policyName rsp_relay_home -priority 100 -gotoPriorityExpression END -type REQUEST

Then pass the person into DartRelay with either method from When the appliance signs people in:

• Header sign-in: a rewrite policy that removes any visitor-supplied X-Forwarded-User and inserts it with the AAA username, for example insert_http_header X-Forwarded-User AAA.USER.NAME. Switch on Sign-in by header in DartRelay.
• Form sign-in: a traffic management form SSO action that posts to /portal/login with Input.Username and Input.Password, bound through a traffic policy.

Signing out of NetScaler too

Set the address's Sign-out address on Tenancy & Branding to the authentication host's logout URL:

https://relay.example.com:4443/cgi/tmlogout

Only the authentication host and port end the AAA session. The same path on the load-balancing virtual server (port 443) shows a "logged out" message without ending anything, so people would be signed straight back in.

Kemp LoadMaster

A Kemp virtual service balances DartRelay. With the Edge Security Pack (ESP) switched on, Kemp can also sign people in, and it serves its own sign-in page on the same address and port as the portal, so people never see a second address.

Virtual service

• Virtual address 10.0.0.10, port 443, with your certificate; real servers 10.0.0.11:8080 and 10.0.0.12:8080.
• Persistence: Active Cookie.
• Health check: HTTP, URL /health/ready.
• Make sure WebSocket connections are passed and the idle timeout is at least an hour.

ESP options (Kemp signs people in)

DartRelay settings for Kemp

1. Trust the LoadMaster. On System → Load balancer, add the address the LoadMaster uses to connect to your DartRelay servers to Trusted proxies, and set Proxies in the chain to 2.
2. Allow Kemp's form sign-in. Tick Let the load balancer fill in the sign-in form. Kemp builds its sign-in post from a template, so this is required for its single sign-on.
3. Set the sign-out address. On Appearance → Tenancy & Branding, open the binding for relay.example.com and set Sign-out address to https://relay.example.com/portal/signed-out.

nginx

Open-source nginx makes a capable plain load balancer for DartRelay. It does not sign people in.

map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream dartrelay {
    hash $cookie_relay_device consistent;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=30s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=30s;
}

server {
    listen 80;
    server_name relay.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name relay.example.com;
    ssl_certificate     /etc/nginx/ssl/relay.example.com.crt;
    ssl_certificate_key /etc/nginx/ssl/relay.example.com.key;

    client_max_body_size 0;

    location / {
        proxy_pass http://dartrelay;
        proxy_http_version 1.1;
        proxy_set_header Upgrade           $http_upgrade;
        proxy_set_header Connection        $connection_upgrade;
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        proxy_buffering         off;
        proxy_request_buffering off;
    }
}

• X-Forwarded-For $remote_addr replaces whatever the visitor sent. Do not use $proxy_add_x_forwarded_for, which passes a forged value along.
• Add nginx's address to Trusted proxies.

Two limits of open-source nginx

• Persistence. Cookie persistence is not part of open-source nginx. The example keeps each browser on one server by hashing DartRelay's own device cookie, which spreads visitors well once their browser has it. The alternative, ip_hash, keeps visitors by address, which sends everyone behind one office's public address to the same server.
• No active health checks. nginx drops a server only after real requests to it fail, and never asks /health/ready. Drain on the Load balancer page has no effect behind nginx. For planned maintenance, mark the server down in the upstream block and reload nginx.

IIS Application Request Routing (ARR)

If you already run IIS, ARR can balance DartRelay at no extra cost. It does not sign people in.

Prerequisites

• WebSocket Protocol (Server Manager → Web Server → Application Development). It is off by default on Windows Server; without it the portal appears and nothing launches.
• URL Rewrite. ARR can appear installed while URL Rewrite is missing, and then forwards nothing.

Proxy settings

%windir%\system32\inetsrv\appcmd set config -section:system.webServer/proxy /enabled:"True" /preserveHostHeader:"True" /reverseRewriteHostInResponseHeaders:"False" /responseBufferLimit:"0" /timeout:"00:05:00" /commit:apphost

• preserveHostHeader passes the address people used, which DartRelay uses to choose the tenant, theme and sign-out address.
• responseBufferLimit 0 lets downloads and print jobs stream rather than being held.
• These settings apply to the whole IIS server; review them if other sites on the same server also use ARR.

Server farm

1. Create a farm named dartrelay. Add 10.0.0.11 and 10.0.0.12 with HTTP port 8080.
2. Answer No when the wizard offers to create a URL rewrite rule. Its rule applies to every site on the server.
3. Set the health test. URL http://dartrelay/health/ready, interval 10 seconds, expected status 200.
4. Turn on client affinity. ARR then keeps each browser on one server with its own cookie.
5. Allow the server variable. At server level, in URL Rewrite → View Server Variables, add HTTP_X_FORWARDED_PROTO.

Rule in the DartRelay site's web.config

Create an IIS site bound to relay.example.com on HTTPS with your certificate, and give it this rule:

<rule name="DartRelay" stopProcessing="true">
  <match url="(.*)" />
  <serverVariables>
    <set name="HTTP_X_FORWARDED_PROTO" value="https" />
  </serverVariables>
  <action type="Rewrite" url="http://dartrelay/{R:1}" />
</rule>

In DartRelay, add the IIS server's address to Trusted proxies. Proxies in the chain can be 2: ARR adds the real visitor address to X-Forwarded-For, and anything a visitor forged before it is not believed because it did not come from a trusted proxy. No sign-out address is needed, because nothing signs people in ahead of DartRelay.

Other load balancers

For any other product, including F5 BIG-IP and cloud load balancers, work through What any load balancer must do. In practice:

1. Create a server pool of your DartRelay servers on their HTTP port, with a health monitor on /health/ready expecting 200, and a member-down action that leaves existing connections in place.
2. Create an HTTPS virtual server for your address, with WebSocket support, cookie persistence, an idle timeout of at least an hour, and no response buffering or content rewriting.
3. Replace the forwarded headers. Remove visitor-supplied X-Forwarded-For and X-Forwarded-Proto and set your own. Take care not to have both a header rule and a built-in "insert X-Forwarded-For" option adding the header twice.
4. Trust the load balancer. Add the addresses it connects from to Trusted proxies.
5. If it signs people in, configure form sign-in to /portal/login or header sign-in, land people on /portal, and set the address's Sign-out address to the appliance's logout URL. If the appliance intercepts a logout URL itself, have it watch /portal/signed-out, never /portal/logout.

Checking your configuration

1. Open System → Load balancer through the load balancer's address. It should say your connection came from a trusted proxy, show your own client address, and show the scheme as https.
2. Launch an application. If the portal appears but nothing opens, WebSocket upgrades are not being passed.
3. Check persistence. In the browser's developer tools, open the page request (not the request list) and read the X-DartRelay-Node response header. It should stay the same as you move around the portal and launch applications.
4. Check the audit log. Sign-in entries should show visitors' real addresses, not the load balancer's.
5. If the appliance signs people in, sign out and confirm the next visit asks for credentials again.

If something goes wrong

The portal works but nothing launches

WebSocket upgrades are not reaching DartRelay. Enable WebSocket on the appliance (on IIS, install the WebSocket Protocol feature).

"Running on another server"

Persistence is missing or not cookie-based. Turn on cookie persistence. With nginx, use the device-cookie hash shown above.

Everyone appears with the load balancer's address, or everyone is locked out together

The load balancer is not in Trusted proxies, or it connects from a different address than you entered (on NetScaler, the SNIP).

An endless redirect after NetScaler sign-in

The authentication profile's host is the same host and port as the load-balancing virtual server. Use a different port, such as 4443.

A "page not found" straight after NetScaler sign-in

NetScaler sent the browser to /vpn/tmindex.html. Add the responder policy shown above.

After signing out, people are let straight back in

The appliance's own session is still alive. Set the address's Sign-out address: on NetScaler, the authentication host's /cgi/tmlogout; on Kemp, /portal/signed-out with the matching Logoff String.

Appliance single sign-on works sometimes

One server has a different setting in its own configuration file. Look for set on this server on the Load balancer page of each server.

Resources that pass the password through are refused

People signed in by header sign-in, which carries no password. Use form sign-in instead.

Related pages