This is a funtoo rev-bump of apache-2.2.11.ebuild to address the slowloris denial of service:
Patch was applied, and still-vulnerable MPM's were masked for the time being.
Without this patch, apache-2 servers can be brought down with the slowloris DOS in a matter of seconds.
I did confirm that the anti-slowlaris.diff allowed apache to continue working - tested myself on a Drupal installation I had lying around. Before the patch, apache stopped responding. After the patch, apache was a tiny bit laggy but kept working.
Please use the Security component for such bugs and let us handle the CC, thanks.
At the moment upstream does not consider this to be a bug , the unofficial PoC patch you are using is also known to the Debian maintainers but they have not yet reviewed or applied it , so I am not sure if we should apply it.
For an immediate workaround, I suggest using any of the the techniques described in .
Hi Alex, I have created this bug report for the benefit of Gentoo users, in case you might be interested in providing a version of Apache that is not exploitable by default.
I cc'd hollow and patrick because they're friends of mine who I know have an interest in improving Gentoo; patrick has explicitly requested that I cc him on all bugs I submit, and hollow compiled the 2.2.11 patch set. In the interest of full disclosure of this issue to people who may be interested in resolving it, I ask that you do not remove them from the cc list.
The apache team may not consider this to be a bug, and that is their choice. However, I don't see how IT staff worth their salt can consider Apache to be a good choice for a Web browser if it is so trivially exploitable by default, as it currently is. The IT staffs of the world also have the right to form an opinion, don't they?
This is a critical issue for people using Apache in a production environment, where the DOS can have significant impact. Consider health care services, financial services, or other important services that use Apache in some way.
I hope you will take the lead on this issue rather than waiting to see what Debian does. If you consider Gentoo users' Apache installs to be valuable, then addressing this issue in a timely manner should be a priority.
s/web browser/web server/
The man offers you a patch to fix, and you offer work arounds? The very fact that the work arounds exist for the issue being patched admits that it actaully is a bug. Where is the logic in any of your response Alex?
Waiting to see what someone else will do before making a decision?
Thank you Daniel.
This is just a simple variation of known tcp/ip socket exhaustion attacks, it's just slightly different in keeping the connections open, one could also start a download and limit the transfer to a few bytes/s or never close connections by using the methods described in the 25c3 talk "TCP Denial of Service Vulnerabilities".
> However, I don't see how IT staff worth their salt can consider Apache to be a
> good choice for a Web browser if it is so trivially exploitable by default, as
> it currently is.
I don't see how IT staff worth their salt can consider not using rate-limiting in their firewall systems to prevent a well-known attack. This issue just pops up in slightly new variations now and then (1998, 2005, early 2007, December 2008, now). It's just a fact that keeping 1000 connections open to a server that can only handle 1000 connections will prevent further connections to it; slowloris just does this a bit differently.
> This is a critical issue for people using Apache in a production environment,
> where the DOS can have significant impact. Consider health care services,
> financial services, or other important services that use Apache in some way.
I'm sure these institutions would add additional security systems on the network level which will easily prevent this attack. Besides that, critical health care and financial services run on isolated networks, not on the public internet.
> I hope you will take the lead on this issue rather than waiting to see what
> Debian does. If you consider Gentoo users' Apache installs to be valuable, then
> addressing this issue in a timely manner should be a priority.
I'm hosting Apache on ~100 production servers and I'm not worried. Why? I've known these kinds of attacks for a long time and they can be prevented easily.
IMHO, there really is no reason to panic right now.
In my opinion, disabling all MPMs except prefork (which we could quickly patch) is not an option. The patch should be applied if it does not have any side effects, but I think there is no rush; applying it does not fix the underlying problems of tcp/ip socket exhaustion; one will still need rate-limiting to fix that.
Other opinions (especially from the apache team) are welcome and should be discussed.
Craig, connection rate limiting is not a fool-proof defense against slowloris, and Gentoo does not perform connection rate limiting by default. This is all about getting a configuration of Apache that is not exploitable *by default*.
Yes, there are possible ways to set up your infrastructure so that you will not be affected by this bug. However, this is actually quite a bit trickier than you think.
Take for example an organization that *does* have the infrastructure in place, such as a hardware load balancer in front of Apache. Even if you have a load balancer in front of Apache, slowloris can traverse your load balancer if you don't have tcp splicing/delayed binding enabled.
Next, let's talk about SSL. For SSL connections, load balancers *can't* protect against this unless they are configured to decrypt your traffic prior to forwarding the SSL to Apache, so that they can inspect the HTTP header. So this DOS is actually a lot more effective than you seem to think it is.
This generally applies to all popular load balancers, including F5 Big IP, Cisco, Citrix etc.
Many people will not or have not deployed an additional layer of defense against this attack. Some may have additional layers of defense but may have a false sense of confidence that their existing defenses are "good enough". You seem to be in that camp.
The fact that an easy-to-run perl script exists, that can allow anyone who has 5 minutes and malicious intent to take a Web site offline with ease, makes this an issue. Similar attacks may have existed for a while, but this is a particularly nasty form of it that happens to impact the world's most popular Web server.
So, Craig, I do think your post contains some misleading statements. For anyone reading this thread, I'd recommend that you download slowloris.pl and test your infrastructure yourself. Collect some empirical evidence about *your* infrastructure. If you're not affected, great. Just don't assume that you're OK. Because of the complexity of the issue, it makes sense to ensure that Apache is not affected *by default*.
Credit for the technical content in this post goes to Ryan Vick, who tested the infrastructure of a large organization for vulnerability against slowloris.
(In reply to comment #5)
> This is just a simple variation of known tcp/ip socket exhaustion attacks, it's
And buffer attacks are all variations on, as are any resource exhaustion attack, or... Every current attack vector is a variation on a past attack vector. (Windows is continually being exploited. The Windows RPC service has been exploited repeatedly (RPC DCOM, RPC LSASS, RPC OOB). To say that this is a simple variation is fine, but to justify this as a reason to do nothing is not.
> > However, I don't see how IT staff worth their salt can consider Apache to be a
> > good choice for a Web browser if it is so trivially exploitable by default, as
> > it currently is.
> I don't see how IT staff worth their salt can consider not using rate-limiting
> in their firewall systems to prevent a well-known attack. This issue just pops
> up in slightly new variations now and then (1998, 2005, early 2007, December
> 2008, now). It's just a fact that keeping 1000 connections open to a server
> that can only handle 1000 connections will prevent further connections to it;
> slowloris just does this a bit differently.
I've run DNS Blacklists for ages, and your trust in any IT staff at anything other than a large enterprise being able to deploy a firewall that does anything productive let alone rate limiting is _severely_ misplaced.
> > This is a critical issue for people using Apache in a production environment,
> I'm hosting Apache on ~100 production servers and I'm not worried. Why? I've
> known these kinds of attacks for a long time and they can be prevented easily.
Would you like a cookie? I absolutely 100% guarantee it's not malicious.
> IMHO, there really is no reason to panic right now.
Yes and that's exactly what we say just before there is reason to panic. This needs to be fixed, or apache needs to be masked (which I think would be pretty embarassing).
I've done a LOT of additional reading, and note that this is a huge problem, it's not restricted to Gentoo, and the patch at best mitigates this problem barely. Load balancers won't work for obvious reasons, firewalls shouldn't be trusted for this sort of work as often other attacks can cripple their ability to deal with this sort of attack utilizing only slightly more bandwidth than slowloris uses.
Further to this a firewall should never be used, or expected to mitigate the flaws in underlying software. The firewall must be correctly configured, but the underlying software MUST be fixed.
This is a serious issue and a GLSA should be published noting that there is not a fix, noting that there is a workaround with significant caveats.
Of course, connection limiting per IP or subnet might block legitimate users, too; but I'd prefer partly blocking access from an IP that does 1000 connects than to allow them to exhaust all available tcp sockets.
In order to mitigate the other well-known tcp/ip issues, rate-limiting is still needed (you should really have a look at the 25C3 talk I mentioned earlier).
If running productive systems, one should really have considered those DOS attacks in general and not just the slowloris specific method of keeping sockets open. If you cannot mitigate this attack, because you're using loadbalancers that are not able to do it, you should still have considered rate limiting on the apache servers themselves. Everyone will agree that this is *not* a beautiful solution, of course.
I'm not saying that we should not patch (if we can patch all MPMs, and if it does not have side effects; disabling worker is not an option IMHO), I'm just saying: this is not the kind of drama you make out of it.
> Some may have additional layers of defense but may have a
> false sense of confidence that their existing defenses are "good enough". You
> seem to be in that camp.
I'm in the "I've-thought-about-these-kind-of-attacks-years-ago"-camp.
Andrew, it's nice to see another funtoo staff member join in, I was still sleeping when you queried me on IRC, but we can also discuss here, so that other people can take part in the discussion.
>And buffer attacks are all variations on, as are any resource exhaustion
My point is: slowloris is just another form of socket exhaustion. Even if you patch this specific method of keeping connections open, the underlying problem is not fixed as there are other methods which can only be mitigated by connection limiting. The patch tastes a bit snakeoilish to me.
Fixing a buffer overflow totally different as it really fixes the problem, there is no other, trivially different way to still exploit it.
>Load balancers won't work for obvious reasons,
Why? I'm interested in your technical arguments. For example, Varnish is said to easily block a slowloris attack. If your shiny, proprietary loadbalancer can't, well...
>firewalls shouldn't be trusted for this sort of work as often other attacks
>can cripple their ability to deal with this sort of attack utilizing only
>slightly more bandwidth than slowloris uses.
Firewalls are MADE for filtering DOS attacks. What other attacks are you talking about?
>Further to this a firewall should never be used, or expected to mitigate the
>flaws in underlying software. The firewall must be correctly configured, but
>the underlying software MUST be fixed.
So you will allow a single IP to eat up all of your connections? That issue will still exist if slowloris is patched, you might want to think about that.
BTW: You can trivially DOS any webserver (any I've tested) with a for loop and curl --limit-rate, even lighttpd...
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 100 -j DROP
As a usual browser opens 2 connections and keeps them open for about 30-90s (HTTP KeepAlive), this would allow 50 Users/IP. This needs to be adjusted for your environment, of course (you also might want to REJECT).
A friend of mine whose website is under attack by slowloris (he has compared headers) has confirmed this to mitigate the attack.
masking all except the prefork mpm ist no an option, neither is masking apache. itk should be trivial to patch, since it's a prefork fork, all other mpms are out of my scope.
i agree with craig that there is no reason to panic, temporary workarounds are known, and we can wait for patches.
Craig, I've already explained why load balancers are not always effective, you should read what I've already written.
I can look at iptables connection limiting. This entered the kernel in 2.6.23 - earlier kernels don't have this option available, and most people are not using connlimit on their Gentoo installs (whereas many, many people have Apache installed.)
For funtoo, we have this issue patched. As more MPMs are patched, I'll add the fixes for those as well. Please keep in touch with me about this as you eventually address it. The patch we are using has been tested to be effective, and looks nice.
I will be collating all this information into a document, which I hope to have ready around tomorrow, which will provide info this issue. I will include info on the patch, iptables connlimit, load balancer configuration, etc.
I found a good quote from Paul Querna, Apache committer, about this - finally, an honest answer:
"Mitagation is the wrong approach.
We all know our architecture is wrong.
We have started on fixing it, but we need to finish the async input
rewrite on trunk, but all of the people who have hacked on it, myself
included have hit ENOTIME for the last several years.
Hopefully the publicity this has generated will get renewed interest
in solving this problem the right way, once and for all :)"
I like the patch because it is not a mitigation, it fixes the problem. As you become more aware of more complete patches, let me know, they will get added to funtoo.
> I can look at iptables connection limiting. This entered the kernel in 2.6.23
> earlier kernels don't have this option available, and most people are not using
> connlimit on their Gentoo installs (whereas many, many people have Apache
Maybe they have. On their homeserver. Or for development. Not on live, important sites. And If you have the freedom to use gentoo there, your boss will probably allow a kernel update for security reasons, too. :)
> For funtoo, we have this issue patched. As more MPMs are patched, I'll add the
> fixes for those as well. Please keep in touch with me about this as you
> eventually address it. The patch we are using has been tested to be effective,
> and looks nice.
My opinion from the sysadmin point of view: Has it been tested in a real-life scenario? Prefork performs badly under high load. Using the patch and waiting for the other MPMs to be fixed would require additional servers. Additionational servers require money, rackspace and setup time, which costs more money than a new loadbalancer or firewall (or even linux server that is used as firewall).
> I will be collating all this information into a document, which I hope to have
> ready around tomorrow, which will provide info this issue. I will include info
> on the patch, iptables connlimit, load balancer configuration, etc.
You might want to read Willys (haproxy dev) statement: http://haproxy.1wt.eu/.
Current phrack is worth a read, too (nkiller2). The slowloris page links to other pages which contain more apache-dos tools.
The connlimit will unfortunately not work for my servers, as I am using OpenVZ, and the highest-quality OpenVZ kernel is based on a RHEL5 kernel, which is at 2.6.18 and does not include connlimit. Last night, I tried using patch-o-matic to get connlimit into the kernel, but it looks like patch-o-matic includes an older version, and I don't have the experience to back-port it (tried a basic back-port, and connlimit is not plug 'n' play with 2.6.18).
It looks like Red Hat did a one-off connlimit kernel for this issue: https://bugzilla.redhat.com/show_bug.cgi?id=483588 . Hopefully they will integrate it into their mainline kernel soon. I may look at using existing netfilter modules to provide similar functionality.
It looks like the convesration on the apache-dev mailing list is more pro-active than the hoser who closed the slowloris bug report, so hopefully they will address slowloris and other trivial DOS sooner rather than later.
Put Apache on Port 8080 and varnish or haproxy (they have an anti-slowloris and anti-nkiller config on their page) on 80, redirect the traffic to the (firewalled!) port 8080.
It's easier to just switch to cherokee...
I have tested the anti-slowloris patch that I submitted, fairly extensively. It is not a perfect fix for the slowloris issue. It prevents slowloris from doing any damage when slowloris is used with basic settings. When you crank slowloris up quite a bit though, then Apache is still unresponsive.
The best fix I have tested for this issue is to use the Cherokee web browser, and to configure ulimits for the cherokee user to allow up to 4096 file descriptors, then use cherokee-admin to set cherokee to use all 4096 file descriptors (it may not auto-detect how many it should use as the help in cherokee-admin states). 4096 file descriptors (up from the max per user default of 1024) was enough to allow cherokee to handle a pretty serious slowloris attack without running out of resources.
With the much higher threshold, it should be possible to safely do connection rate limiting, at a high threshold (to protect against extremely concentrated slowloris attacks) and not be at a serious risk of blocking legitimate traffic.
http://thread.gmane.org/gmane.comp.apache.devel/37773 (where the funtoo proposed patch came from)
And what about very popular sites with enterprise users (of course from behind NAT)?
I think firewall solution can be just workaround, but not fixes the problem itself.
(In reply to comment #10)
> iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 100 -j
> As a usual browser opens 2 connections and keeps them open for about 30-90s
> (HTTP KeepAlive), this would allow 50 Users/IP. This needs to be adjusted for
> your environment, of course (you also might want to REJECT).
> A friend of mine whose website is under attack by slowloris (he has compared
> headers) has confirmed this to mitigate the attack.
I forgot to post this here. Ryan Vick and I have published a fairly comprehensive Slowloris DOS Mitigation Guide, which is available here:
This should be useful for enterprise as well as non-enterprise users.
One area we did not cover are software load balancers, but we cover hardware load balancers in detail - the same concepts apply to software load balancers.
I have addressed this issue in Funtoo by adding a pkg_postinst() message pointing to our Slowloris guide. Going to mark the bug as resolved.
I don't see how that solves the bug in Gentoo.
Rbu, it does not address the bug in gentoo, but gentoo is not going to be applying the anti-slowloris.diff. Are you wanting to use this bug for tracking the slowloris issue in general, until its resolution in the official gentoo ebuilds? Based on the response, it does not seem that this issue is actively being addressed or treated as a bug by Gentoo.
(In reply to comment #23)
> Rbu, it does not address the bug in gentoo, but gentoo is not going to be
> applying the anti-slowloris.diff. Are you wanting to use this bug for tracking
> the slowloris issue in general, until its resolution in the official gentoo
> ebuilds? Based on the response, it does not seem that this issue is actively
> being addressed or treated as a bug by Gentoo.
We are not going to apply the anti-slowloris.diff which was not accepted upstream and is marked as "proof-of-concept" by its author. However, I am following development on the mod_noloris which made its way into SVN:
The fact that no commit has been made does not mean that we are not actively following the bug. Craig (who is practically a member of the Security Team) has replied several times on this bug, and I have replied to you in private.
It seems to me you think we do not care about this issue, and I don't know why that would be the case.
Regarding your impressions of what I am thinking, I appreciate you trying to read my mind over the Internet, but you're not very good at it. I am simply trying to clean up a bug that I opened, to which I have received an adequate response from the Gentoo developers - and the response so far has been to direct me towards mitigations, which I have added to my guide, and which is fine. I appreciate the feedback I have received. To this point I had not seen anything in this bug to indicate to me that this issue was still open, so I closed it as I think everyone including me agrees that anti-slowloris is not a good fix.
There is no <www-servers/apache-2.2.14-r1 in portage any more.
What about adding noloris as a module with the apache ebuild?
Or adding it to the tree?
This would at least allow those of us that want to use it to do so.
WONTFIX doesn't look like the correct solution for a valid security issue...
We should add ebuilds for mod_noloris or something alike and suggest it in apache postinst.
Reopening, as the issue is indeed not fixed.
The Apache HTTP Server 1.x and 2.x allows remote attackers to cause a denial
of service (daemon outage) via partial HTTP requests, as demonstrated by
Slowloris, related to the lack of the mod_reqtimeout module in versions
Interpretation of the CVE: mod_reqtimeout was added in 2.2.15 to mitigate slowloris, and if Gentoo enables it in the default config, updates the initscript to throw a warning if it isn't enabled, and puts a message at the end of merge about it, we can probably call this bug closed?
Added to existing GLSA draft
This issue was resolved and addressed in
GLSA 201309-12 at http://security.gentoo.org/glsa/glsa-201309-12.xml
by GLSA coordinator Sean Amoss (ackle).