See the real client behind your web proxy with X-Forwarded-For

Field	Value
App Note ID	AN-008
Audience	SOC analysts, Trisul admins, NOC and network engineers
Applies to	Trisul in packet capture mode, monitoring HTTP traffic on the upstream side of a web proxy

Summary

When Trisul monitors traffic upstream of a web proxy, every web request appears to come from the proxy. Trisul can read the X-Forwarded-For (XFF) HTTP header that the proxy adds, and replace the proxy IP address with the real client IP address for the rest of that connection. You turn on the XFF header in your proxy, confirm four settings in trisulProbeConfig.xml on the probe, and restart the probe context.

The problem

Trisul works best at a trust boundary, for example on either side of the corporate firewall. A few monitored external ports then show the network and security data that crosses the boundary.

A web proxy such as Squid, Bluecoat, F5 or Cisco breaks this. The proxy opens its own connection to each web server, so traffic upstream of the proxy carries the proxy IP address, not the address of the endpoint that made the request. In the HowTo example, all web traffic appears to come from the proxy at 10.0.0.200, and you cannot see the activity of the user at 10.0.0.166. Host reports, flows and alerts all point at the proxy.

You could place Trisul behind the proxy instead, but that often means mirroring several ports. The documented solution is the X-Forwarded-For HTTP header.

How it works

Most proxies, including Squid, can add an X-Forwarded-For header to each HTTP request. The header carries the IP address of the client that sent the request to the proxy. Its general form is X-Forwarded-For: client1, proxy1, proxy2: the left-most address is the original client, and each proxy on the path appends the address it received the request from.

The XFF deproxy feature is part of TCP reassembly on the probe:

1. TCP flow tracking follows each TCP connection. TCP reassembly rebuilds the byte stream, so an HTTP header that spans several packets is read correctly.
2. When the reassembled stream is identified as HTTP, Trisul parses the request headers. If a header named X-Forwarded-For holds a valid IPv4 address and EnableXFFDeproxy is on, Trisul records that address for the connection.
3. From then on, Trisul replaces the proxy address with the client address in every packet of that connection, in both directions. The proxy is the side that opened the connection.
4. All monitored objects (traffic, flows, packets, alerts and resources) then carry the endpoint address, as if the proxy were not there. Raw packets that Trisul stores are re-stamped with the endpoint address.

Two consequences follow from this order. The few bytes that pass before Trisul reads the XFF header stay metered against the proxy. If a connection carries no usable XFF header, the whole flow stays metered against the proxy.

The probe sits at the firewall, upstream of the proxy. Only connections with a usable XFF header move from the proxy address to the client address.

Reference: Trisul Probe Configuration File – Reassembly

Prerequisites

• Trisul in packet capture mode. The reassembly settings apply to packet capture mode only, and in NetFlow mode Trisul disables all TCP reassembly options.
• A Trisul probe that monitors the link upstream of the proxy, where the HTTP requests from the proxy to the web servers pass.
• A web proxy that can add the X-Forwarded-For header to HTTP requests, and admin access to its configuration.
• Root access on the probe node. Root privileges are required to edit trisulProbeConfig.xml.
• The name of the context to change, for example default.

Procedure

Part A – Make the proxy send the XFF header

1. In your proxy configuration, turn on the X-Forwarded-For header for HTTP requests, if it is not already on by default. Use the documentation of your proxy product for the exact setting.

Part B – Check the probe settings

All four settings are in the Reassembly section of the probe configuration file. The XFF option sits inside TCPReassembly › Applications.

1. Log in to the probe node as root.
2. Open the probe configuration file of the context. For the default context the path is:

/usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml

For a context named data1, the last folder is context_data1 instead of context0.

3. Under Reassembly › TCPReassembly › Applications, check that EnableXFFDeproxy is True. This is the default.
4. Under Reassembly › TCPFlowTrack, check that Enabled is True. This is the default. XFF deproxy, like every reassembly-based application, needs TCP flow tracking.
5. Under Reassembly › TCPReassembly, check that Enabled is True. This is the default. While you are there, check two related settings:
   • Direction is INOUT (the default), so that both directions of each connection are reassembled.
   • Ports is empty (the default), or lists the port your proxy uses to reach web servers. When Ports holds a list, Trisul reassembles only traffic on those ports.
6. Under Reassembly › IPDefrag, set Enabled to True. This setting is False by default because IP reassembly can load the CPU and memory on busy links. The value MetricsOnly only collects fragmentation metrics and does not reassemble.
7. Save the file.

After steps 3 to 6 the relevant part of the file looks like this. Other settings in these sections stay as they are.

<Reassembly>
    <IPDefrag>
        <Enabled>True</Enabled>
    </IPDefrag>
    <TCPFlowTrack>
        <Enabled>True</Enabled>
        <!-- HiWater, LoWater unchanged -->
    </TCPFlowTrack>
    <TCPReassembly>
        <Enabled>True</Enabled>
        <!-- MaxBytes, KickoffBytes unchanged -->
        <Ports></Ports>
        <Direction>INOUT</Direction>
        <Applications>
            <EnableXFFDeproxy>True</EnableXFFDeproxy>
            <!-- other applications unchanged -->
        </Applications>
    </TCPReassembly>
</Reassembly>

Reference: Trisul Probe Configuration File

Part C – Restart the probe context

1. Restart the probe context so that the changes take effect. For the default context on probe0:

trisulctl_probe restart context default@probe0

Replace default with your context name.

Reference: trisulctl_probe Command Reference

Verify the result

The setup works when the probe log shows XFF deproxy enabled and your HTTP traffic is metered against endpoint addresses instead of the proxy.

1. Settings loaded. Right after the restart, view the probe log of the context:

trisulctl_probe log default@probe0 log=ns tail

At startup the probe logs one line per reassembly setting. Look for lines that contain this text:

TCP Flow Tracking is : TRUE
TCP Reass is..............: TRUE
TCP Reass IN..............: TRUE , OUT TRUE
TCP Reass XFF.............: ENABLED

2. Reports. Generate HTTP traffic from an endpoint behind the proxy. In the web interface, the endpoint address now appears for that traffic in hosts, flows, alerts and resources. The proxy address still shows the few bytes of each connection that pass before the XFF header, and any flows that carry no usable XFF header.

Troubleshooting

The messages below appear in the probe log of the context (see Verify the result, step 1).

Symptom or message	Cause	Fix
TCP Reass XFF.............: DISABLED	EnableXFFDeproxy is not True	Set it to True and restart the context.
TCP Flow Tracking is : FALSE	TCPFlowTrack › Enabled is False, or its water marks are invalid (next row)	Set Enabled to True and restart the context.
Error in config file TCPFlowTrack : Lowater cant be greater than Hiwater	TCPFlowTrack › LoWater is equal to or greater than HiWater, so flow tracking is turned off	Set LoWater below HiWater (defaults 60000 and 80000) and restart the context.
TCP Reass is..............: FALSE	TCPReassembly › Enabled is False	Set it to True and restart the context.
Mode is NETFLOW_TAP , all TCP Reassembly options are disabled	The context runs in NetFlow mode	XFF deproxy needs packet capture mode.
[IPDEFRAG]Disabling due to Invalid value for config param <Enabled>. Expect True,False,MetricsOnly val= followed by the value	IPDefrag › Enabled holds another value, so IP defragmentation is turned off	Use True, False or MetricsOnly and restart the context.
IPReassl Config Error : Lowater cant be greater than Hiwater	IPDefrag › LoWater is equal to or greater than HiWater, so IP defragmentation is turned off	Set LoWater below HiWater and restart the context.
XFF is ENABLED in the log, but HTTP traffic is still metered against the proxy	The request has no X-Forwarded-For header; the header value is not a single IPv4 address; the traffic is HTTPS; or Ports lists ports that do not include the port the proxy uses	Check that the proxy adds the header (Part A). Check the Ports setting (Part B, step 5). See Limitations for the other cases.

Limitations

• Packet capture mode only. XFF deproxy depends on TCP reassembly, which is not available in NetFlow mode.
• Plain HTTP only. Trisul reads the X-Forwarded-For header from reassembled HTTP requests. Traffic that is HTTPS upstream of the proxy stays metered against the proxy.
• One IPv4 address per header. Trisul uses the header only when its value is a single IPv4 address. A list such as client1, proxy1 from a chain of proxies, or an IPv6 address, is not used, and the flow stays with the proxy.
• IPv4 connections only. Address replacement is not implemented for IPv6 traffic.
• First bytes stay with the proxy. The bytes of each connection that pass before the XFF header is read are metered against the proxy.
• CPU and memory. IPDefrag set to True can add CPU and memory load on busy links.

Related

• Trisul Probe Configuration File – Reassembly
• trisulctl_probe Command Reference
• Glossary: Context, Flow, Mode selection
• AN-005 – Send IDS alerts to Trisul
• AN-007 – Offline processing of packet captures