How I (Almost) Took Down the Whole Office

How I (Almost) Took Down the Whole Office

There’s a certain kind of IT mistake you don’t fully appreciate until the next morning when you walk into the office and everyone is staring at you.

Enabling HTTPS decryption on a Sophos Firewall sounds like a smart security move and honestly, it is. But if you just check the box and walk out the door without testing anything, you might come back the next morning to an office full of panicked colleagues and a very uncomfortable conversation with your boss. That’s exactly what happened to me.

The Click That Started It All

I was in the Sophos Firewall settings one afternoon, poking around. Not because something was broken, just because I wanted to understand things better. Maybe tighten up security a little. Maybe make myself feel like a proper sysadmin for five minutes.

And there it was: “Decrypt HTTPS during web proxy filtering.”
It sounded smart. It sounded like the kind of thing a real sysadmin would enable. The idea is that your firewall intercepts encrypted HTTPS traffic, decrypts it, scans it for malware or policy violations, and then re-encrypts it before sending it on its way. More visibility. More security. Better control over what’s actually flowing through your network.

I checked it. I saved it. I went home.

The New Guy

Before I left, a new colleague came over and told me he couldn’t connect. Now, look… he was brand new. Day one or two. And in my head, I was already halfway out the door. New people sometimes just don’t know how to connect to things. Maybe he missed a step. Maybe he forgot a password.
So I told him we’d look at it tomorrow. It’s end of shift, it can wait.

I left.

The Next Morning

I walked into the office and everyone was looking at me. Not in a good way. My boss, my colleagues — everyone was in panic mode because nobody could connect to the German servers. The Omnissa Horizon client, the software we use to remote into our AutoCAD workstations in Germany, was completely dead. Couldn’t establish a connection. Nothing. The whole workflow had stopped overnight.

I won’t lie, I panicked for about 20–30 seconds.

Then I remembered. Oh. Oh no. The firewall.

What Actually Happened (The Technical Part)

Here’s why that one checkbox caused so much damage.

When you enable “Decrypt HTTPS during web proxy filtering” on Sophos, the firewall essentially performs what’s called a man-in-the-middle on your encrypted traffic. Every HTTPS connection leaving your network goes through it and the firewall intercepts it, decrypts it using its own internal Certificate Authority, scans the content, and then re-encrypts it before sending it to its destination.

For regular browser traffic this can work fine, but there’s a critical catch: you need to properly set it up first. That means distributing the Sophos CA certificate to all machines on your network so they trust it, and crucially creating exceptions for applications that do their own certificate validation outside of the browser.

The Omnissa Horizon client is exactly one of those applications. Horizon doesn’t use your browser. It establishes its own encrypted tunnel directly to the remote connection servers in Germany, and it validates the certificates on those servers very strictly. When Sophos suddenly started intercepting those connections and presenting its own re-signed certificate instead of the real one from Germany, Horizon looked at it, didn’t recognize the Sophos CA as a trusted authority, and refused to connect. No graceful error. No fallback. Just silence.

This is actually a known problem with SSL/TLS inspection and it breaks any application that pins or strictly validates certificates without going through the system’s browser trust store. Remote desktop clients, some email clients, update services, VPNs – all of them can fall apart the moment a firewall starts intercepting their encrypted traffic without the proper exclusions in place.

The new guy was the first person to try connecting after I made the change. He was the canary. I just didn’t listen to him.

The Fix

I went to the Sophos Firewall settings, unchecked the box, and saved. Connections came back immediately. The whole thing… the panic, the morning chaos, the confused colleagues – caused by one checkbox I enabled the afternoon before and never tested.

The Part Where I Owned It

My boss asked what happened, and I explained exactly what I did. No excuses, no “maybe it was something else.” It was me, I did it, here’s why, here’s what I should have done differently. He was fine about it, he’s worked in IT long enough to know that these things happen. You touch something, something else breaks. It’s part of the job.
I also sent a message in the team group chat apologizing to everyone, and they were fine too. The new guy I dismissed the night before? I didn’t apologize to him specifically. But I should have, he noticed the problem first, and I waved him off because he was new. Lesson learned on that one too.

What I’d Do Differently

Test after every change. Even small ones. Especially firewall changes. Before going home. If I had opened the Horizon client for 10 seconds after enabling that setting, I would have caught it immediately and nobody would have known.

Don’t dismiss reports because someone is new. Brand new colleague, day two, says he can’t connect, right after I tinkered with the firewall. I had all the information I needed and still told him it could wait until tomorrow.

HTTPS decryption needs proper setup before you turn it on. It does have real security benefits by being able to scan encrypted traffic for malware is genuinely useful. But it requires distributing the firewall’s CA certificate to all machines, and setting up exceptions for any application that handles its own certificate validation. Remote desktop clients like Horizon, VPN software, anything that isn’t a regular browser – these need to be excluded or they will break.

Conclusion

The internet itself was never actually down. Everyone could still browse. It was one broken connection to one set of servers, but it happened to be the connection the whole office needed to do their work. One checkbox, one afternoon, one morning of chaos. You learn more from that kind of mistake than from a week of reading documentation.

That’s usually how it goes.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *