PHP, TLS, etc… We appear to be facing a tsunami of vulnerabilities being discovered in record time. Further, the window to act on these vulnerabilities has basically closed to zero; hackers’ tools now automatically fold the very latest vulnerabilities into their botnets within minutes/hours of discovery. I have always been extremely reluctant to expose any of my services - even with the latest and greatest security tools. My choice to stay totally within the confines of tailscale seems vindicated given this week’s news. How are you all taking this?
I have my ops agent constantly checking SBOMs against current CVEs and if something triggers, I get a report with severity and repair options researched. I can then decide how I want it to proceed. I’m much less concerned about internal only services. SBOMs are refreshed after updates for future checks.
I’m re-thinking a few things, mostly moving exposed services onto a VM or two that are on their own VLAN, and don’t contain any important data (it would be my website, gameservers like Minecraft, etc…). I might move to podman as well so nothing has a possibility of running as root, docker can run rootless but that setup seems more confusing to me.
I already use a VPN via Wireguard on my router for access to my private services when out of the house, so my changes are just separating the public from the private as much as possible on the server side of things…
Essentially if someone manages to drop a crypto miner or something on my website that’s fine, I’ll get an alert pretty quickly through Beszel that the server load is high and I can go roll back a backup or whatever. I just want to make sure there’s no access to anything else for them.
I moved all my files to one share. At the root of that share is a robot_readme.md file that lists all the contents of my share with a note that says read everything you want, you will anyway.
They left a note that said “nah, bruh, you’re good”.
I guess I’m okay.
This was the final push that convinced me to rebuild my server from scratch! I started doing so 3 weeks ago :D
I was running my containers on a single good old Debian, that I replaced with Proxmox. I set up several VM that don’t share anything between them :
- One for all the services I don’t really care about, like game servers and utilities with no personal data.
- A second one for services in need of a GPU passthrough (Immich and Jellyfin) and partial access to some of my data (photos and movies in this case)
- And a third one that gets access to all my personal data, only hosting a container for WebDAV access and a web file browser. I might hide these two services behind a VPN in the future if I feel like doing so, for now it is still to practical to have them freely available…
- A last VM is running network related app like my reverse proxy.
I should have done this years ago, but I’m still learning and learning costs time :p
Next I want to hide all of Internet facing services behind an identity provider/application proxy (I don’t really know if it is the right way to name it ?) like Authentic to totally block access to any content from unauthorized visitors.
I guess I’ll be mostly fine once this is done ^^
Your separation between VMs is exactly what I want to do. But given how I’m still rocking a single truenas machine running all my containers, I’ve got a quite a bit of setup to do… Like you said, rebuilding everything is a slog.
My threat model is not changed: I want to serve something and I want to not be busy with "oh suddenly there’s
atwo PHP vulnerabilityies. My enemy is the world, but that is not really unchanged from before in a world where there exists capitalism and rent seekers want to collect from us even for breathing. I’m not intended to be unpaid tech support. So I just serve the things with reasonable scopes (under wireguard, or in a borrowed VPS, etc) and let it go. If an automated system manages to find a hole, computers are so fast that it’s not going to make a difference if I’m sitting in front of the chair vs enjoying some modicum of social life visiting a friend three cities away.I only have a VPN endpoint exposed. If I need to access any of my web services I simply VPN in and away I go.
This
Tailscale is the super easy version of this. Otherwise make sure your containers don’t have root etc. block inter container networking as much as you can. Lots of relatively simple things work unless you’re actually an interesting target.
I completely folded all services to the “inside”.
Answering the titular question, why would the advent of AI-found vulnerabilities substantially change one’s threat model? Before LLMs, the prospect of an undiscovered vulnerability was non-zero probability then, so what exactly has changed now?
In terms of attacker capabilities, it would be a mistake to solely observe that attackers now have LLMs and conclude that this means the threat landscape has changed. The use of LLMs does not magically create a root escalation where one does not or cannot exist. Part of your threat model should include defense-in-depth mitigations, so that when (not if) an attacker finds a 0-day exploit, they are severely limited in gained functionality on your machines. In the simplest cases, jails or containers serve to keep separate the web-facing processes. VM isolation would be a further step, because an attacked cannot gain machine control without both a software exploit as well as a VM escape route. People hosting services can and should expect that their front line defense will fall, so they better have more defenses in place.
Phrased another way, the people who have done their threats analysis properly and have accepted the residual risk of hosting their own services that face the web, should not really be additionally perturbed by AI advances. This isn’t the same caliber of landscape change as would, say, the invention of a cryptographically relevant quantum computer. If the worst thing that can happen to your home IRC server is that it gets flooded by a script kiddie, AI does not mean your server can now burst into flames.
Much like with human hygiene, some basic rules are unusually effective against most threats. The equivalent to washing one’s hands is to stay on top of system and software updates. Institutionally funding the organizations (eg PHP Foundation, curl, USA CDC) so they can fix these problems would also help.
We’re talking self hosting, not an enterprise setup. Industry best practices and layers of defense mitigation are not realistic expectations for something like a media server that is only intended to be accessed by the owner and maybe a couple other people.
I disagree: self hosting and enterprise are not mutually exclusive. In fact, “enterprise on-prem” is the marketing description for self-hosting. I do not accept any definition of self-hosting that gate-keeps to any degree.
It is incongruous to talk about “only home self-hosting” or “only enterprise on-prem” when it comes to security mitigations, threat modeling, and risk management: it is merely a question of scale.
Sure, the user base for a home setup might be single digits, but the attackers are identically situated: their capabilities do not grow or shrink based on whether they are targeting a bank or a Jellyfin server.
Before LLMs, the prospect of an undiscovered vulnerability was non-zero probability then, so what exactly has changed now?
The problem is not the prospect of an undiscovered vulnerability (there is always a vulnerability somewhere) but rather the prospect of a reduced response window.
There’s a wide gap of difference between “non-zero probability” (0.0076%) and “non-zero probability” (82.01734%).
If you’re dealing in probabilities, that’s not threat modeling; it might be in the larger umbrella of risk management though. Defining a threat model means to qualify what sort of attacker you expect to face and what they are assumed to be capable of. An assumption such as “this adversary can send arbitrary packets to the public IP address” is what would be documented in a threat model.
The raison d’etre of threat modeling is to methodically tackle the actual or assumed risks that an attacker can pose against you. It is not an exercise in swinging at shadows, though I wouldn’t discourage overstating the capabilities of an attacker, if needing to protect assets like patient info or PII. Doing threat modeling based on “gut feeling” is a recipe for not recognizing risk where it’s present and seeing risk when there isn’t any.
The fact remains: you cannot squeeze blood from a stone. Even the most sophisticated AI cannot steal PII from a computer that doesn’t store PII. Proper threat modeling means you don’t put anything on the line that you haven’t squared up against the risks, however improbable. For any non-zero concern, a good threat model will discuss its mitigation, whether it will be absorbed, or if it is an open question. For an out-of-scope concern, a good threat model will explicitly state that assumption.
Defining a threat model means to qualify what sort of attacker you expect to face and what they are assumed to be capable of
Which is exactly what OP is talking about. 5 years ago if you were only trying to defend against script kiddies, using containers was good enough. But now you have to consider that script kiddies might find container escapes.
I still do, but I don’t host anything on 80 or 443. I use a random large port number, and I’ve seen zero illegitimate requests in 20 months.
IF there’s an active exploit AND it affects a container AND you have a container escape route AND you pick my domain you’d need to statistically try thousands of times to even hit the single open NAT port.
I also practice domain/IP obfuscation in addition to high port:
- TLS certs are wildcard certs, so bots can’t get my subdomains from transparency logs.
- My DDNS IP is on a subdomain (random string) not on the base domain.
- My public services are also on random subdomains, CNAME’d individually to the IP subdomain.
- There is no wildcard CNAME that the bots can query to get the IP.
This eliminates scanning almost completely. They can’t find an IP to go with my base domain, first of all. And even if a bot is scanning the IPv4 address space and all the ports and it finds the reverse proxy it can’t get past it without knowing a FQDN. I still get the occasional hit from such bots but it’s very rare.
Also, all my public services are secured with mTLS (preferably) or at the very least a long key in a custom HTTP header (both enforced at the proxy).
Thank you for sharing this, vey insightful
How do u use a port other than 443 for https. AFAICT all web browsers are pinned to it (for ssl checks I think) and I could never get an actual https connection through an alt port. Maybe I didn’t understand it properly.
Edit: I think actually it’s the auto cert provisioning workflow. The CA I’m signing against hardcodes port 443 :/
Edit, edit: this was the issue https://caddy.community/t/non-standard-ports-for-https/6916/4
The API based challenges are better since you don’t even need to make the site externally accessible to get certificates.
Let’s encrypt and similar services support those and for popular registrars (aws, porkbun, etc,) here are existing plugins for caddy, swag (nginx proxy with built in cert support) and a few others.
Yeah, if you use something like Nginx Proxy Manager you can’t use the automatic flow, you need to use the token based challenge.
Boo, no idea why acme forces these specific ports. Swapping them seems like a pretty basic obfuscation technique that could help a lot to secure any services.
Edit: actually seems like this might be possible with the dns challenge type. I’m gonna test this out later https://github.com/caddy-dns/porkbun
It is, that’s what I had meant by token based challenge lol. You need a DNS API key for your manager to do it directly, it flags a DNS record with a token to validate ownership. I was just calling it out from memory, sorry!
Much appreciated stranger. Hopefully now I can stop exposing the two most attacked ports in existence from my LAN XD.
Hasn’t changed much. I have accepted the risk and, for now, the desire to have publicly accessible stuff outweights my lack of desire to deal with the realities of a VPN. I mean, I want to share stuff with others, and I want to be hosting it myself because I have grown to distrust someone else’s service… so what else can I do but risk it?
Besides: I expect tailscale or the underlying VPN services to already be a target. I would be surprised if they don’t find anything. What then?
If they find anything we fix it, update our services, and move on.
The chances of being hit by a zero-day as a random self-hoster are very small, but anyway if you keep backups and can restore a clean state (and patch your system) you should be good.
The goal is not to have perfect security, it’s to mitigate the risks and to handle fallout correctly.
I’ve always treated my home server with public facing services as presumptively compromised. I don’t put any data on there I wouldn’t want to see become public, and any data I actually care about losing is backed up to the cloud with an append-only key. The NixOS config lives in a git repo not on the server. If something bad happens, I can wipe the machine clean, redeploy the image, restore my backups, and we’re back to the status quo.
So, my threat model hasn’t changed much!
not at home but i work as a software dev for a very big company you heard of and all we do noe is fix ai found security bugs. hell the stuff i work on isnt that public and we are still fixing. i can only imagine what the team of the public website are handling.
I moved to closing everything down and wireguarding in to home 5 years ago and I’m glad I did. I only expose game server these days which have a different vuln profile.
If/when I open up for Jellyfin/etc, I’ll probably run them on a different box to my NAS in a DMZ with strong firewall rules. And use a VPS for anything else that doesn’t require NAS storage to reduce blast radius.
Can we go back to late 90s internet yet?
exposure is unchanged. automated patching and upgrades are working. rollbacks, etc are working. minimal attack surface exposed and properly blocking traffic by geography. rotation of keys/passwords as well as short lived rotated TLS certs.
not one of those things alone is a solution, but together can make a solid dent in securing your infrastructure.
I had a public facing jellyfin. I took it down because of that [ai].
Immich and nextcloud are still up.
I’m not saying there is a rational reasoning behind it.
.jpg){kind=link}