I'm seeing some very strange behavior from my Apache web server, and I'm afraid it may have been compromised. Every time I start it, my router is saturated with the maximum number of connections it can handle, and my access_log starts filling with lines like:
208.115.242.252 - - [29/Mar/2014:22:04:54 -0400] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=300x250%C2%A7ion=3730175&... HTTP/1.0" 200 5463 74.63.219.228 - - [29/Mar/2014:22:04:54 -0400] "GET http://ads.yahoo.com/st?ad_type=iframe&ad_size=728x90%C2%A7ion=5101980&a... HTTP/1.0" 200 5432 198.100.123.53 - - [29/Mar/2014:22:04:54 -0400] "GET http://www.superficialgirl.com/miss/160x600.php HTTP/1.0" 200 374 198.100.121.56 - - [29/Mar/2014:22:04:53 -0400] "GET http://content.yieldmanager.edgesuite.net/atoms/14/0d/e5/b4/140de5b4c2f26ddb... HTTP/1.0" 200 20393 199.83.93.35 - - [29/Mar/2014:22:04:54 -0400] "GET http://ro2.biz/pixel.png HTTP/1.0" 200 151 172.246.127.211 - - [29/Mar/2014:22:04:54 -0400] "GET http://ads.yahoo.com/imp?Z=300x250&s=5507180&_salt=2285596723&B=...
I don't serve any adds from this server. It's just a few webapps that I use personally. Has anyone seen anything like this before? Any advice?
Thanks, =-Jameson
I'm seeing some very strange behavior from my Apache web server, and I'm afraid it may have been compromised. Every time I start it, my router is saturated with the maximum number of connections it can handle, and my access_log starts filling with lines like:
Start whatever HTTP server in place of Apache, and see if you still get these requests by analyzing their access.logs. Then you will know if you really get these requests or they are fake.
On Sat, Mar 29, 2014 at 10:41 PM, Nowaker enwukaer@gmail.com wrote:
I'm seeing some very strange behavior from my Apache web server, and I'm afraid it may have been compromised. Every time I start it, my router is saturated with the maximum number of connections it can handle, and my access_log starts filling with lines like:
Start whatever HTTP server in place of Apache, and see if you still get these requests by analyzing their access.logs. Then you will know if you really get these requests or they are fake.
Thanks for the idea. I had just been approaching it from the idea of trying to figure out what was going on with Apache. I installed, and started Nginx, and sure enough, it started getting blown up with those requests. Now, I guess I have to figure out why on earth those requests would be coming to my humble home web server.
If you are worried about the possibility of a system compromise here are a few things you could try
1. Check for the presence of any unusual files on your server. A lot of cracks aimed at webservers have the goal of hosting files from the cracked server (usually porn and warez). It might be a good idea to try using a live cd for this because some rootkits can hide the presence of the files from system tools such as the ls command. 2. Use a tool like wireshark to monitor the incoming and outgoing traffic to the server and look for anything unusual (see https://wiki.archlinux.org/index.php/Wireshark) 3.Check the contents of your /etc/passwd file and look for any unusual user accounts (I also recommend a live CD for this). 4. Use the ps command to check the running processes, and look for any ususual processes. A lot of cracks modify the ps . A cracked ps often has a much smaller filesize than a regular ps. 5. Look at the output of the history command to view the past commands used on the server. If it does not return any output or returns commands that you did not enter then this could indicate a problem. 6. Run some rootkit detection programs like chkrootkit or rkhunter (these return a LOT of false positives). 7. Has anything else been acting up with the server? A lot of cracks break other things. ---Theo
Date: Sat, 29 Mar 2014 22:45:35 -0400 From: imntreal@gmail.com To: arch-general@archlinux.org Subject: Re: [arch-general] My Apache Sever Compromised?
On Sat, Mar 29, 2014 at 10:41 PM, Nowaker enwukaer@gmail.com wrote:
I'm seeing some very strange behavior from my Apache web server, and I'm afraid it may have been compromised. Every time I start it, my router is saturated with the maximum number of connections it can handle, and my access_log starts filling with lines like:
Start whatever HTTP server in place of Apache, and see if you still get these requests by analyzing their access.logs. Then you will know if you really get these requests or they are fake.
Thanks for the idea. I had just been approaching it from the idea of trying to figure out what was going on with Apache. I installed, and started Nginx, and sure enough, it started getting blown up with those requests. Now, I guess I have to figure out why on earth those requests would be coming to my humble home web server.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
You can also use pacman -Qkk
For example, boot from iso, mount your root device as /mnt pacman -Qkk -r /mnt
Not every warning means a compromission.
To find files not owned by a package, use: find /mnt/ -exec pacman -Qqo {} -r /mnt ;
Maybe you need better filtering here, like using /mnt/usr, /mnt/etc, ... instead /mnt, because lots of files should not be owned by a package.
I haven't tried this, it is only a comment.
Am 31.03.2014 07:05, schrieb Theo Jones:
On Mon, Mar 31, 2014 at 6:36 AM, Simon Brand simon.brand@postadigitale.de wrote:
You can also use
Thanks for all the suggestions, guys. I'll probably do some further checking just to make sure nothing else is going on, but once I created a rewrite rule to drop those connections instead of sending them 404s, they went away. I suspect it's some type of ad click fraud.
On Sun, Mar 30, 2014 at 4:37 AM, Jameson imntreal@gmail.com wrote:
I'm seeing some very strange behavior from my Apache web server, and I'm afraid it may have been compromised. Every time I start it, my router is saturated with the maximum number of connections it can handle, and my access_log starts filling with lines like:
Do you know what looks interesting from you log? Take for example this line
199.83.93.35 - - [29/Mar/2014:22:04:54 -0400] "GET http://ro2.biz/pixel.png HTTP/1.0" 200 151
See? The request asks for all the URL, http:// and host name included, just as if you were a proxy. The normal GET request for a web server asks only for the file part ("/pixel.png" in this case). But the most interesting part is that your apache is replying with "200", that is OK! And the 151 is the real size of this file, I just checked with "curl http://ro2.biz/pixel.png | wc -c". I would expect your server to reply with an error to such a request, not to reply with the real file.
So my guess is that you have your apache configured as an unrestricted proxy, somebody discovered it, and is abusing it.
Hope this helps.
199.83.93.35 - - [29/Mar/2014:22:04:54 -0400] "GET http://ro2.biz/pixel.png HTTP/1.0" 200 151
But the most interesting part is that your apache is replying with "200", that is OK!
Nice catch! It's certainly a proxy.
See? The request asks for all the URL, http:// and host name included, just as if you were a proxy. The normal GET request for a web server asks only for the file part ("/pixel.png" in this case).
It's because of HTTP/1.0 protocol. Should the client use HTTP/1.1, it would look more usual.
On Tue, Apr 1, 2014 at 9:30 AM, Nowaker enwukaer@gmail.com wrote:
199.83.93.35 - - [29/Mar/2014:22:04:54 -0400] "GET http://ro2.biz/pixel.png HTTP/1.0" 200 151
But the most interesting part is that your apache is replying with "200", that is OK!
Nice catch! It's certainly a proxy.
Thanks for everyone's help with this. I did in fact have ProxyRequests set to On thinking it was needed for reverse proxies as well, and have turned it off. Now, when I open up port 80, it looks like they're still trying, but I'm replying with 404. Is that what it should be doing? I probably also need to make sure I have some throttling setup in case this is too much for my Internet connection.
Am 09.04.2014 19:32, schrieb Jameson:
On Tue, Apr 1, 2014 at 9:30 AM, Nowaker enwukaer@gmail.com wrote:
199.83.93.35 - - [29/Mar/2014:22:04:54 -0400] "GET http://ro2.biz/pixel.png HTTP/1.0" 200 151
But the most interesting part is that your apache is replying with "200", that is OK!
Nice catch! It's certainly a proxy.
Thanks for everyone's help with this. I did in fact have ProxyRequests set to On thinking it was needed for reverse proxies as well, and have turned it off. Now, when I open up port 80, it looks like they're still trying, but I'm replying with 404. Is that what it should be doing? I probably also need to make sure I have some throttling setup in case this is too much for my Internet connection.
If you know the IP addresses (or address-ranges) you use to connect to your server, I suggest you block everything else for the time being with an iptables rule.
On Wed, Apr 9, 2014 at 7:38 PM, ProgAndy admin@progandy.de wrote:
Am 09.04.2014 19:32, schrieb Jameson:
On Tue, Apr 1, 2014 at 9:30 AM, Nowaker enwukaer@gmail.com wrote:
199.83.93.35 - - [29/Mar/2014:22:04:54 -0400]
"GET http://ro2.biz/pixel.png HTTP/1.0" 200 151
But the most interesting part is that your apache is replying with
"200", that is OK!
Nice catch! It's certainly a proxy.
Thanks for everyone's help with this. I did in fact have ProxyRequests set to On thinking it was needed for reverse proxies as well, and have turned it off. Now, when I open up port 80, it looks like they're still trying, but I'm replying with 404. Is that what it should be doing? I probably also need to make sure I have some throttling setup in case this is too much for my Internet connection.
If you know the IP addresses (or address-ranges) you use to connect to your server, I suggest you block everything else for the time being with an iptables rule.
fail2ban can do that automatically for you, with some work configuring it.
In general I think it's better not to send a 404 when someone is obviously trying to abuse your servers, that tells the bad guys that there is a web server listening there and may leak some information about your setup. It's better to block them at the firewall level, which costs you less server resources. I'd sugest that the firewall is configured to deny (that is, just drop their packets) instead of reject (which sends a rejection packet which, again, gives the bad guys more information than strictly necessary).
On 2014-04-09 19:32, Jameson wrote:
On Tue, Apr 1, 2014 at 9:30 AM, Nowaker enwukaer@gmail.com wrote:
199.83.93.35 - - [29/Mar/2014:22:04:54 -0400] "GET http://ro2.biz/pixel.png HTTP/1.0" 200 151
But the most interesting part is that your apache is replying with "200", that is OK!
Nice catch! It's certainly a proxy.
Thanks for everyone's help with this. I did in fact have ProxyRequests set to On thinking it was needed for reverse proxies as well, and have turned it off. Now, when I open up port 80, it looks like they're still trying, but I'm replying with 404. Is that what it should be doing? I probably also need to make sure I have some throttling setup in case this is too much for my Internet connection.
One approach I've seen mentioned and which seemed fun, but -- I hasten to add -- have never personally tried is to start returning shock site images for all such requests (obviously not for all 404s, just attempts at abusing you as a proxy).
Regards,
arch-general@lists.archlinux.org