Adding random delay for specific HTTP Requests with HAProxy + Lua

Lately, i wanted to delay specific http requests and i wanted to have a random delay for every of these requests (for example in a range from 1000ms to 2000ms).

One possible use case was to absorb and slow down traffic bursts that came from bots/crawlers/spiders or abusers (bad behavior clients in general). I’ve classified the traffic from these clients by multiple factors (http requests/errors per time interval, etc.). For all requests from this traffic class I want to distribute the requests over a timeframe and absorb bursts. A fixed, static delay value isn’t really helpful here because if an abuser/spider/crawler is doing many requests at the same time/same second, all requests are delayed for xxxx ms. But if the delay is over, all requests are bursting anyway at the same point in time. I do not want to block the clients or return a 503 (for many reasons, one of them is that false positives in the detection logic should not block a client etc.).

Another reason for using a random delay was that many applications behave much better if the requests are distributed over a timeframe and not as a burst. Nice side effect is that you do not need extra server capacity only for handling bursts.

As you can imagine, there are many other use cases, especially in the context of abuser handling.

Note: I recommend that you whitelist search engines (when response time is a ranking factor, for example) or the tune the limits for the “delay traffic class” accordingly.

As an ugly hack, you can use the following in HAProxy 1.5 to delay requests:

 tcp-request inspect-delay 2000ms
 tcp-request content accept if WAIT_END

But as you can see, the value is fixed and the same for every request. Besides that, “inspect-delay” was originally not build for this case and it’s not recommended for this use case.

Let’s try HAProxy 1.6.

Update 2015-10-20: Adapted configuration for 1.6.0 Stable Release

HAProxy 1.6 comes with Lua scripting support and i gave it a try. I was able to fulfill my requirements with the following (HAProxy 1.6.0):

haproxy.cfg:

global
 lua-load /etc/haproxy/delay.lua

defaults
 mode http
 timeout connect 10s
 timeout client 10s
 timeout server 10s

frontend fe
 bind 127.0.0.1:8001 name fe
 http-request lua.delay_request if  { ... your condition ... }
 default_backend be

backend be
 server s 127.0.0.1:80

delay.lua:

function delay_request(txn)
    core.msleep(1000 + math.random(1000))
end

core.register_action("delay_request", { "http-req" }, delay_request);

“core.msleep” is a non-blocking function, so there aren’t any problems with concurrent requests.

Another example of HAProxy’s great flexibility! I think the HAProxy Lua integration has enormous potential and i’am looking forward to see other examples and use cases.

 

4 thoughts on “Adding random delay for specific HTTP Requests with HAProxy + Lua

  1. How did you get this working? I am getting below error. if I use http-request lua

    [ALERT] 290/174657 (27738) : parsing [haproxy1.cfg:14]: ‘http-request’ expects ‘allow’, ‘deny’, ‘auth’, ‘redirect’, ‘tarpit’, ‘add-header’, ‘set-header’, ‘replace-header’, ‘replace-value’, ‘set-nice’, ‘set-tos’, ‘set-mark’, ‘set-log-level’, ‘add-acl’, ‘del-acl’, ‘del-map’, ‘set-map’, ‘set-src’, ‘set-var(*)’, ‘use-service’, ‘silent-drop’, ‘capture’, ‘set-method’, ‘set-path’, ‘set-query’, ‘set-uri’, ‘sc-inc-gpc0(*)’, ‘sc-set-gpt0(*)’, but got ‘lua’.

Leave a Reply