Dealing With Google's Malware Robocop

We’re now self-hosting a number of capabilities, such as our Mastodon server and our PixelFed server (plus of course this blog!) An unexpected problem this has raised is the way Google repeatedly red-lists our domains for hosting phishing sites and malware. The problem with that? We do not and never have done any such thing. This problem is not unique to us; almost everyone who self-hosts via YunoHost encounters it and there is a massive thread on their support forum.

Why it’s happening

Frankly I don’t know. I mean, it’s obviously part of Google’s necessary and welcome defences against the bad guys. I’m glad they do that. But the assumptions on which they appear to be working, their triggers for action and the asymmetry of the process all make me very concerned. Because of some undocumented trigger, Google unilaterally causes everyone’s web browser to treat my domain as if it is hosting malware and it prevents staff and clients accessing our production services. Even after clearing a block their bot will sometimes re-apply it the same day - see the screenshot above.

It is false and baseless and the way they misrepresent us to clients approaches defamation (see the warning below) but they do it anyway because we are self-hosting so for them the risk/reward ratio is skewed towards blocking rather than investigating.

Whatever the reasons, the really big practical problem is the asymmetry of their process for us and every other self-hoster. They are fast and devastating on the “shoot first” front, but slow, opaque and uncooperative when it comes to “asking questions later”. Once Google has red-listed one of my domains, a sequence of adverse consequence follow:

• They automatically add the domain to a list in everyone’s browser that makes it block the domain with the red screen above.
• Since GMail (and I suspect other webmail providers) use the same blocklist, any e-mail with a link to that domain gets classified as phishing.
• That in turn can get anyone using the sites we host automatically reported to their mail administrator for abuse (and with almost no context - just a statement they are sending malware links around).
• Sometimes, they add the domain to a more dangerous blocklist that makes DNS providers delist the domain so it won’t resolve to an IP address - there’s a helpful tool for detecting this because even my ISP (Hover) does not tell me they are blocking my domain.
• All my Yunohost-managed sites use a shared LDAP, so the LDAP domain then also gets red-listed, impacting every production system.

All this happens within a few minutes of the relevant bot deciding my Yunohost LDAP is a phishing site. It can then take several hours for me to be advised they have done this, via the search.google.com portal where I have registered all our domains. If I had not done this, I would get no notifications – in Google’s world, convicted phishers get no chances to prove they are innocent and there’s no deterministic appeals process.

Once they have red-listed my domain, what can I do to get it unblocked? It turns out Google don’t even admit to the possibility of a false positive. All their processes are heavy on gaslighting you into believing you are the problem. So you have to go along with their game and tell them what you did to remove the malware and beg for unblocking via the portal. There’s no ticket and no tracking. Even when they finally unblock my domains, they never explain anything - they just tell me to take more care not to let the malware be implanted next time.

Removing The Block

So how do we remove these blocks? I have a well-trodden process that often gets the block delisted within a few hours (although it can take 3-5 days, and the confirmation via the search portal can take even longer).

• Add the domain to your list at https://search.google.com/, ideally via a DNS authentication. Doing it this way also authenticates all subdomains.
• Check the report is false using Virus Total - you never know, this might be the one time Google has it right!
• Ask for a review on search.google.com, explaining it is a false positive and details of the software you are using.
• Ask all other users to report a false positive via the red screen (press the “Details” button).

Taking Action

I can’t help thinking Google is out of order here. Making my domains repeatedly unusable, effectively without notification or appeal, and falsely telling my clients and staff that I and my company are a bad actor on the Internet, seems an extreme consequence of their defence actions. It’s as if their worldview excludes the possibility that people might be self-hosting servers on the Internet. Or, worse, as if the extremity of their action has the useful side-effect of driving business their way. Personally I think it’s time some regulators took a look at things. How about you?

From an original Mastodon thread