Craft vs Bitdefender: Certificate Wars

tl;dr: If you're having issues where Craft CMS is throwing 500 bad request errors for POST requests when Bitdefender is installed, check your browser proxy settings, nuke the Bitdefender Certificate in both your browser and Windows, and try reimporting manually (this has been a journey); skip to the bottom for details.

---

Realistically, with modern operating systems, there has become little need for dedicated antivirus software on personal computers. Still, I grew up hammering the message into relatives and friends that AV programs were important, so having a backup to Windows Defender gives me peace of mind. For the past few years, that safety net has been provided by Bitdefender. With the rise and fall of both AVG and Avast, it's the last free option I've not heard any negative comments about, and it's served me well. It happily sits in the background, keeping itself up-to-date, working alongside Windows Firewall without complaint, and silently blocking the occasional malicious advert or redirect^[1].

However, recently that peace of mind began to be chipped away. Starting about a month ago, I started noticing a weird pattern of behaviour when using the backend to this 'ere website: my installation of Craft CMS. Each time I logged in, things would behave as normal, letting me write up a post, make some quick edits, fiddle around with settings, etc. But after 10-15 minutes or so, the site would begin throwing errors all over the place. It first affected the autosave function for drafts, but then it began to bleed into other areas of the dashboard: loading plugin settings, fetching latest news, displaying category lists, and on, and on.

Once the errors had started there was nothing I could do. Closing and reopening my browser seemed to help but, of course, any changes since the last successful save would be lost (unless first copied somewhere else temporarily). Then even that began to make no difference; the errors would just keep going, now also blocking login attempts 😤

Eventually I caused one too many errors in quick succession (likely in frustration hammering F5 😅) and wound up triggering a server routine that fully blocked my account! Right, off to Krystal's (my web host) customer support to raise a ticket. The initial investigation confirmed the security routine had been a bit overzealous, so my account was green-listed for whatever trigger it had set off, and initially that seemed to fix things... initially!

Unfortunately, within an hour or two the original errors were back. Sure, they didn't seem to be bothering the server anymore, but they were still bothering me! Back I went to the support ticket. Now, to Krystal's credit, their support team really tried to help out. I had my ticket reopened a total of four times, escalated twice, with various tactics attempted over a multiple week period. Ultimately, though, they were unable to find a root cause, only confirming that it must be something on my end.

---

During that time, my own investigation had raised a couple of commonalities with when the errors would occur. First, they only affected one machine (weird, right); my phone and laptop, my work laptop, my partner's computer, all were fine. Second, they were being triggered by HTTP requests, mainly of the POST variety (though hefty GET requests could sometimes have the same effect). After a little further prodding, I was able to successfully capture HTTP packets for identical requests, one that went through fine and one that triggered an error! The difference? Two things:

1. The error message was missing a content-type header;
2. And the request that errored was digitally signed by Bitdefender, whereas the one that wasn't was covered by Let'sEncrypt.

Yeah, that's why this post began talking about antivirus software. Bitdefender was somehow involved in this infuriating knee-capping of my content management system! How? Why? It wasn't clear.

Still, it was an easy theory to test: wait for the errors to start and turn off Bitdefender, then refresh the page. Do the errors continue? No. Wait, no! That worked? Basically, I found that disabling my antivirus "fixed" the issue. Adding my website to the "exclusion" list in Bitdefender, however, did nothing. This new information was dutifully passed along to Krystal, but they didn't have any suggestions and felt it was likely a red herring. I also asked around online, including in the Craft Discord servers, to no avail^[2].

After another week – now spent dutifully disabling my antivirus software whenever I needed to use Craft, a "solution" that continued to work flawlessly – the team at Krystal managed to confirm that this was absolutely not something on their end. To their credit, they helped a huge amount and did confirm what I'd suspected about content-type being missing, but ultimately that ended up being the red herring.

See, Craft recently added some config settings around content-type, so when I found those release notes I got all excited thinking I'd uncovered the root cause, but no matter how I changed the corresponding values, the errors always came back. Another fortnight wasted, I was close to giving up and uninstalling Bitdefender, when I hopelessly began throwing keywords into their FAQ search bar in vain one last time instead.

Searching for "headers" turned out to be the golden ticket, unearthing an article about browser proxy settings. Huh, Bitdefender has some kind of VPN or proxying service in place? Wouldn't that route all HTTP requests through it? Maybe that was causing the issues? 🤔

Sure enough, opening up Firefox's Settings menu and navigating to Network Settings showed that I was using "system proxy settings" as my preference. I changed this to "no proxy" and held my thumbs... 🤞

---

Now, the first time I published this article, that was where the story concluded. For hours I used the CMS without the slightest hint of an error message, and I wasn't holding back. I deliberately went and modified a bunch of old entries (updating some fields I've been meaning to for a while) to really test the server connection – no errors! Next, I went and browsed around the plugin store, something that would always trigger a few issues – nothing! Finally, I wrote the first version of this article. It came to over 1,000 words, took more than 40 minutes, triggered dozens of draft saves – all without issue. I thought I'd genuinely fixed the problem. For over a month, I'd never had a spell this long without running into a bad request. It felt conclusive.

Yet it wasn't.

...

🤬🌪🎇⚡👆🔥

Mere minutes after setting this article live, I noticed a typo in the blurb and went to fix it. When I hit resave, Craft errored. When I refreshed the page, Craft errored. When I navigated to a different menu – you guessed it – Craft errored. I opened up Network settings with a heavy sense of dread and glumly confirmed that the error had, indeed, returned.

Still, at least I had a few additional ideas to go on; a couple of new keywords to try out. I began searching through Bitdefender forums, subreddits, and their ilk, and lo and behold I actually uncovered some more potential solutions. These were largely complaints about incorrect HTTPS handling, from people who were finding Bitdefender a little overzealous with its blocklist, but there were several commonalities I began to notice. Whilst issues were present across browsers, Firefox popped up more frequently than most. Plus, wherever chatter online became loud enough to attract a customer support employee, the suggestions always focused on corrupt security certificates.

That was a new avenue to explore, so off I went. Each attempt resulted in the same mildly infuriating sequence of events:

1. Make some changes to my browser or OS settings and clear the error;
2. Use Craft for an hour or so without any hiccoughs;
3. Reopen the browser, or restart my PC, and find that the error had returned;
4. Changing the settings again with no effect, or at best a short-lived one.

It was almost like Bitdefender was learning. Like it was watching my problem-solving attempts and then running off to work out ways to circumvent them. Frustrating was an understatement.

The My Solution

Then, at the very end of my tether, I found someone who claimed that a specific sequence of solutions I'd tried in the past offered some form of salvation. Fine, let's give it a go (this is for Windows/Firefox, specifically):

1. Open the Windows Certificate Manager (type Certificate Manager into search or open certmgr.msc from command line);
2. Browse to Trusted Root Certificates, then open the Certificates subfolder;
3. Delete any instances of Bitdefender certificates listed here (they all start with "Bitdefender ..." so pretty easy to see);
4. Once deleted, right-click the Certificates subfolder and choose All Tasks → Import;
5. Navigate to C:\Program Files\Bitdefender\Bitdefender Security\mitm_cache\fake-ca.crt and select/open the file, then press Next (FYI, this assumes you installed the software in the default location; if not, navigate to wherever you did install it. You can also ignore the second "fake" file in that folder);
6. Make sure "Automatically select the certificate store" is checked (this appears to be crucial), then press Next and finally Finish;
7. Reboot your browser (and possibly PC).

That set of instructions seemed to work for me (🤞🤞🤞). Step 6 appears to be the most important. I'd previously tried deleting the certificate and letting Bitdefender repopulate; deleting it and importing with default options; and deleting it without letting it reimport or refresh. None of those attempts had any effect.

Another trick I'd tried had been to open up my browser certificate store and delete/reimport the Bitdefender certificate there as well (and every possible variation/combination of the two). Again, this worked for a while, but ultimately still broke. If you feel it's worth giving it a go, here are the steps for Firefox:

1. Open your settings menu (hamburger icon at top-right → Settings, or type about:preferences into the address bar);
2. Go to Privacy & Security and scroll down to the Certificates section;
3. Press "View Certificates" to open the modal, and make sure the Authorities tab is selected;
4. Scroll through and remove any Bitdefender certificates by selecting them and pressing "Delete or Distrust" (again, normally nested under a Bitdefender section, though worth double-checking throughout I found);
5. Hit OK and reload the browser, see if it worked;
6. If it didn't, go back to that panel and press "Import", navigate to the same file path as above and import the fake-ca.crt file, then again reboot the browser and your PC.
   

That worked for a while for me, but ultimately the error returned. I also came across anecdotal evidence that you should delete the certificate within Windows Certificate Manager (using steps above) and then immediately remove it from Firefox as well. Some people claim you should then reimport to Firefox, but not Windows; others say import to Windows, but not Firefox. I tried all of these variations and none of them worked for me, but clearly YMMV with these things 😅

---

It's been a couple of days since I implemented the latest solution and – so far – it's held. There was one moment when I saved a post and received a server error, but it immediately cleared on refresh (something the old errors never did) and my server logs didn't flag the same behaviour, so I'm hopeful that was just a genuine dropped packet (as incredibly unlucky as that would have been).

Still, I'll remain a little sceptical that this truly was the silver bullet I've been looking for. Fool me once and all that! Right now, though, it looks like I've finally found the true issue causing Craft to fall over: Bitdefender and a corrupt security certificate (and possible something to do with the proxy service, or Firefox, or HTTPS). Huzzah! 🎉🎉