My site was hacked

Normally, I do a site:h3manth.com search on Google, and the first thing that comes up is:

site-was-hacked

But last Firday evening things were shockingly different, I noticed none of my indexes where reflecting on Google search, rather it was showing some spam titled links!

spamed

Wondering what could be wrong, I logged into the Google search console and request the bot to live test my URL.

The bot replied saying the site is not reachable 404! Even though the site was loading perfectly well with a 200OK!

So, h3manth.com has a strange hosting scenario and here is a quick time line on the investigation I did:

Tech support circus:

A ray of hope:

As I was waiting and replying to each of the main thread async, I was asking the lazyweb if someone else has seen something like this before and Å ime Vidas highlighted that my domain was responding with a 404 when the User-Agent was of Googlebot.

Well, my robots.txt and .htaccess looked fine, at this point of time I started doubting if it was something to do with cloudflares firewall settings, read few posts and disabled few of the advanced settings:

cloud-flare-disbale

Well, still the googlebot was being blocked.

The final move:

From the cloudflare conversation with Damian Parker we tried surpassing the cloudflare proxy via the curl call by directly connecting to the godaddy's host to rule out cloudflare issues:

Surpassing cloudflare with Googlebot UA:

$ curl -svko /dev/null https://h3manth.com/ -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' --connect-to ::godaddy.ip
* Connecting to hostname: godaddy.ip
* Trying godaddy.ip:443...
* Connected to godaddy.ip (godaddy.ip) port 443 (#0)
> GET / HTTP/2
> Host: h3manth.com
> user-agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
> accept: */*

< HTTP/2 404
< date: Tue, 08 Jun 2021 09:17:54 GMT
< server: Apache
< content-length: 315
< content-type: text/html; charset=iso-8859-1

Surpassing cloudflare with a browser UA:

$ curl -svko /dev/null https://h3manth.com/ \
- A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36' \
--connect-to ::godaddy.ip

* Connecting to hostname: godaddy.ip
* Trying godaddy.ip:443...
* Connected to godaddy.ip (godaddy.ip) port 443 (#0)
> GET / HTTP/2
> Host: h3manth.com
> user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36
> accept: */*

< HTTP/2 200
< date: Tue, 08 Jun 2021 09:19:16 GMT
< server: Apache
< last-modified: Sun, 06 Jun 2021 19:52:10 GMT
< etag: "2d2227c-22db-5c41e42b70a59"
< accept-ranges: bytes
< content-length: 8923
< cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< expires: Sun, 19 Nov 1978 05:00:00 GMT
< vary: Accept-Encoding,User-Agent
< content-type: text/html; charset=utf-8

Finally, it was clear that the issue is in the hosting server and not in Google Workplace or Cloudflare, so I was back to discussion with godaddy, who suggested to disable the DNS proxy and test, it was still the same, they later suggested that if I would buy a security package they would do a detailed investigation:

godaddy-email.png

During the discussion, we came across a php file that was high it size and strange, the goddady team suspected a mallware and was suggesting to do a virusscan, but I had remembered seeing such a file and had deleted it assuming that it was created by mistake, the src made no sense though, it was almost like uglified and minified JS (At least the JS would have still made some sense, this wasn't binary but just had some random refs) which they were suggesting not to delete as it might replicate, none the less I had already deleted it and didn't keep a copy of it for future investigation. I asked them for more time and think through this before I bought the security plan.

Later on I was find . -type f -exec du -h {} + | sort -r -h finding and sorting files by size to see if there was any else that is fishy, there wasn't anything suspicious, I decided to look into the .htaccess yet again and voilà this time my eyes finally caught something suspicious:

rewrite-engine

Well, I guess I had removed hungers-jemie.php already before, but this was screwing it up! Cleaned this up and testing the bot was able to fetch my URLs!

Root Cause:

I was thinking hard on how did an external user get access to my .htaccess then I realized that I was also hosting an old drop blog of mine with 300+ posts, logged in as admin to make sure user rules were fine and noticed:

drupal-visitors-can

That was still ok, but making a quick security scan made me realize that this was the backdoor that was used for the breach.

drupal-issues

^ few many of those red flags were not initiated by me for sure! (maybe few module got outdated and was never checked)

So, I quickly got into my DB and checked, surprise, surprise! There thousands of fake users been created! I a hurry I did a delete * from users and then when and check my site, dang, none of posts where loading, I had forgot the fact that deleting users will also delete their reference posts and there is no was to recover them, I was so disappointed that after all this I have lost all of my posts, 300+ posts!! Realized that I had done few experiments in migrating away from drupal just before I moved to octopress (2013) wasn't able to find it in my cloud stores, then I recalled there was a demon that was running backing up the DB, luckily I was able to find the latest DB backup, restored it, but that had the fake users, so cleaned it up and this time didn't delete myself:

db-users

Finally, my weekend was over! (Remember this started on Friday evening) well, it was an awesome feeling at the end, the site was back on track, as I type this the site has been indexed by search engines like ever before, but there are still few cached indices that I have requested to search engine to flush.

P.S: If the intruder is reading this post, I would like to say 👋

Feel free to share this article. You may as well ping me on Twitter.

Published