Two months ago, one of the websites we manage was hit by a wave of Distributed Denial of Service attacks, each with a changing attack vector as the systems mitigated the attacks. These started with basic UDP floods, moved through some NTP and DNS reflections and the only remaining attack which was effective was a layer 7 attack on the site directly. This final, layer 7 attack was a WordPress Pingback attack. While it initially took the site down, we were able to mitigate quickly and see the full force of what the attack was.
While we performed some initial analysis when the attack first occurred, we’ve now taken the time to run though the data in detail to see if anything odd stands out. As this will be an analytical report, there will be lots of numbers. The first:
2.2 million requests in 15 minutes, which peaked at 4,500 requests per second.
This is a lot of traffic for a site which previously averaged around 20,000 requests a day and while it’s low in terms of global DDOS attacks, it was still a significant issue for the destination content management system (not WordPress).
How does the attack work?
Unfortunately, out of the box WordPress has one component which can be exploited for malicious use. By default, the pingback (which allows you to record who linked to your blog) doesn’t have any form of validation to ensure the “pingback” was from the requesting server. This means that a request can be made against a fully patched, clean server and the return request will go to your intended target.
The request from the attacker only needs to be a simple script or even a cURL based call, which is trivial to do.
Despite locking parts of it down, this option is still enabled by default and still an issue with version 4.6.1 (the latest as of this article). We highly recommend installing this plugin to disable the pingback: https://wordpress.org/support/plugin/disable-xml-rpc-pingback
This will still leave the remaining XML-RPC code functional, but prevent your site from being used in a pingback dos attack. Chances are, if you don’t know what a pingback is then you probably don’t need it.
For more information on how widespread the attacks are, Sucuri have two great blog articles about it:
Attack Mitigation
To stop the effects of the attack, all we did was drop any user agent which was set to WordPress. The site itself wasn’t WordPress based, so there’s no reason to allow the pingback or any requests which originate from WordPress. This can be done with the following snippet in nginx:
if ($http_user_agent ~* "WordPress") {
return 403;
}
As we hadn’t disabled logging for this code block, all requests were still logged and it gives us the opportunity to analyse the data further.
Ideally of course we’d tweak a rule within our Web Application Firewall (WAF) to detect and rate limit requests, however being time sensitive this was the fastest approach to bring the site back online.
The Data
To run through the data, we used our Elastic stack so that we could use a high speed index to manipulate. If you haven’t tried the Elasticsearch / Kibana combo for log analysis, I highly recommend looking at it sooner rather than later. It’s absolutely invaluable when it comes to analysing patterns and being able to perform further analysis.
Here’s a few key findings:
Unique Sites: 9,984
Unique IP’s: 10,268
The distributed nature of nearly 10,000 sites making the calls means it’s impossible to block via IP. This is why attackers favour a distributed attacks, as the blocks from both the originating ASN and the destination aren’t feasible to do manually.
Especially since the pingback is a layer 7 attack, this means any DDOS scrubbing service needs to be intelligent to block proactively.
WordPress Versions
WordPress Versions in the Attack: 1,795
The number of WordPress versions seemed abnormally high, which upon investigation showed that many seemed to be obfuscating their version.
Here’s an example from the raw log:
GET / HTTP/1.0" 403 455 "-" "WordPress/2359; https://<realsiteremoved>; verifying pingback from xxxx.xxxx.xxxx.xxxx
I’m not aware of any plugins which obfuscate the version numbers like this, however over 1,400 sites had version numbers which didn’t match a standard WordPress pattern. I manually checked around a dozen sites to see if they were simply faking being WordPress based and all were legitimately running WordPress. If anyone knows the reason for this, please let me know in the comments below!
If we run a very rudimentary filter on the results by simply scanning for a period in the vesion (eg so that 4.6 etc match), this reduced the number to a more sensible figure:
Filtered WordPress Versions in the Attack: 137
Even this is fairly high, considering how easy WordPress makes it to update your software I was hoping to see less of a spread.
If we look closer at the remaining versions, there’s some scary stats. The oldest version is 2.8.1, which was released back in 2009! Thankfully this was only one hit and one manually verified out of disbelief (it really was that old).
Disturbingly though, there were still a significant number of sites older than 3.0:
If we look at the top versions used (grouped by unique sites), we see a spread like this:
As a percentage overall, sites running 4.5.3 were 26% of overall sites and only 11% of sites were running the latest 4.6 release (the latest when the data was gathered).
The official WordPress stats (https://wordpress.org/about/stats/) show more sites which have updated (thankfully), but it’d be nice to see a much lower spread in the future.
Country Analysis
Being an Australian based hosting company, we were interested in seeing where the pingbacks originated from and performed a geo-ip lookup of the requesting server’s IP address.
Countries in the Attack: 109
We then analysed how many requests came from each country and graphed the top 15 by request:
Just to see if certain sites were being used more than others, we also plotted the country data against unique sites:
The spread remains roughly the same. Given that a significant amount of cloud services and hosting companies are based in the USA, this figure doesn’t seem out of place.
Origin IP Analysis
One enhancement WordPress added to the pingbacks in 3.7, which at least tracked the originating IP of the request. While this doesn’t solve the problem, it at least allows you to trace where the calls are coming from. Unless the attacker is very, very naive however, this IP will simply trace back to another infected machine or site. Generally these requesting systems are part of a botnet to mask and distribute the requests. The only information it does give is how distributed the requests are:
Originating IP’s: 149
Again, this simply highlights why the verification doesn’t really assist in any meaningful way. We at least know that there were 149 bots / infected sites used in generating the calls, which had a spread like this:
I’ve masked the real IP’s, and most of the requests came from two systems. There were also a number of sites which were using private IP’s and still a significant number of requests which weren’t from the two main IP’s. To give a bit more perspective, if we plot against the originating IP’s country, we see the following:
This plot shows that the majority of the botnet / originating IP’s came from Russia. I’m not sure if other attack analyses have found any correlation between where the botnets are located and where the attack originated from, but there’s no way to tell from the data we have. It could be that the random IP selection of the bot has some predisposition to certain IP ranges, which I have seen before in other attacks. Either way, it does highlight how much of a global problem DDOS attacks are and the difficulties surrounding the ability to close associated botnets down.
Conclusion
The pingback tool within WordPress still remains an exploitable system for any WordPress site which hasn’t explicitly stopped it. From a web host’s perspective, this is quite frustrating. With enhanced firewall rules in our system, we’re now logging around 2.5 million malicious pingback requests per month (inbound). In world hosting terms we’re only a small player too, no doubt the larger providers are seeing pingbacks in the hundreds of millions or more per month.
There’s two immediate changes WordPress could make to resolve the issue going forward. The first is to disable pingback by default. Those who require it would still be able to easily enable it. Chances are if you don’t know about it, then you probably don’t require it anyway. This would still leave those sites vulnerable to being used in attacks, so it’s only reducing the number of potential sites which can be used.
The second method is to consider the use of a third party system (such as Jetpack) to handle pingback requests as the default. It’s far easier for a centralised platform to be able to detect abnormal behaviour and implement basic rate limiting / blacklisting of bad requests. This is no different to their default installation of the Akismet plugin for spam handling, so it would be a sensible option for WordPress to adopt going forward.
With WordPress 4.7 just around the corner featuring a big upgrade to the REST API, we’re hoping that this is the catalyst to consider disabling XML-RPC by default. Those who still require it could still re-enable it, but it would at least reduce the amount of potential sites hackers can use for malicious use. At the end of the day we still absolutely love and recommend WordPress, especially since features such as the auto-updates are invaluable to patching security issues. If a few of the legacy issues like the pingback vulnerability are resolved then it’ll simply be even better.
