Hi, folks! I’ve learned a lot these past 8-9 months in self-hosting, but I’m not sure what’s going on with my reverse proxy setup. At this point, I’m only running Nginx Proxy Manager, Jellyfin, and Komga, separated from my home network via VLANs and firewall rules. My home network is 192.168.1.X and hosted services are on 192.168.10.X. Yet when I look at access logs for Jellyfin or Komga, both are reporting access from 192.168.65.1 when accessed from outside my home network. I don’t have a 65 subnet, so this is strange. The reverse proxy is run on a mini PC at .10.201, and Jellyfin and Komga are hosted on a NAS at .10.202; all of them run via Docker containers.

The instructions for Jellyfin for reverse proxy just say to list the address for the reverse proxy, which I did using IPv4 and the domain, and various configurations of only one or the other, as well as the 192.168.65.1 address that manifested out of nowhere. I haven’t observed any kind of change in IP address reporting after making changes here. I haven’t begun to try to configure Komga for the reverse proxy, because I figure until I get it working for Jellyfin and understand how to do it, it probably doesn’t matter; I only listed it here as evidence that two different services were reporting the same phantom IP address. I did ask for help on this issue on a Discord server a little while ago, and some of those folks suggested looking at the Docker networks. I left network settings on basically default for all three of these applications, and they all seem to generate something like a 172.17.0.X address, not the 192.168.65.1 that I’m seeing in logs.

I know there are supposed to be extra headers on network packets that these services can be aware of if they know it’s going through a reverse proxy first, and I figure this is important for proper monitoring of folks accessing my hosted services, so i want to get this figured out before I start running a few more apps. Any help anyone can offer would be appreciated. A lot of times, the documentation for this stuff is written up with an assumption of more understanding than the person reading it might have. Or maybe I just glanced right by something obvious and stupid.

  • lemmyvore@feddit.nlEnglish
    1·
    23 hours ago

    Just to understand better:

    • When you say “when accessed from outside my home network”, what kind of access is that? Port forward? VPN?
    • Is 192.168.1.1 your router?
    • What’s your LAN netmask?
    • NPM listens on 192.168.10.201 ports and Jellyfin/Komga on 192.168.10.202 ports?
    • A connection from the outside comes through the router, gets forwarded to 192.168.10.201:443, the proxy sends it to 192.168.10.202:8096, and it shows up in the Jellyfin logs as coming from 192.168.65.1?

    If all of the above is correct I would look at the network interfaces that are up on the mini-PC and the NAS, as well as inside the containers of the proxy and of Jellyfin, and see if there isn’t a 192.168.65.1 up on there somewhere for whatever reason.

    You can also try making a manual request (wget, curl, nc) from the proxy container to Jellyfin and see what it shows up as.

    May also want to see what’s the DNS situation in your LAN and on those machines. You mentioned “names” being used, how are those defined and resolved?

    Last but not least are the containers you mentioned using docker bridge networks or ipvlan?

    • ampersandrew@lemmy.worldOPEnglish
      2·
      22 hours ago

      When you say “when accessed from outside my home network”, what kind of access is that? Port forward? VPN?

      Accessing via 443 forwarding to my reverse proxy from the internet. My dad will hit my Jellyfin server from his house, and I can hit that server from mobile networks when I’m not on wi-fi.

      Is 192.168.1.1 your router?

      Yes. It’s also .10.1 on the 10 VLAN.

      What’s your LAN netmask?

      255.255.255.0 on both VLANs

      NPM listens on 192.168.10.201 ports and Jellyfin/Komga on 192.168.10.202 ports?

      Correct.

      A connection from the outside comes through the router, gets forwarded to 192.168.10.201:443, the proxy sends it to 192.168.10.202:8096, and it shows up in the Jellyfin logs as coming from 192.168.65.1?

      Correct. Jellyfin is actually listening on a different, auto generated, arbitrary port that I just stuck with, but other than that, you got it.

      I did check every IP address I could find anywhere on my network and couldn’t find the 192.168.65.X anywhere, but @anamethatisnt@sopuli.xyz here in this thread seems to have identified this as some NAT setting built in to Docker. Running on default settings for network, because I had no idea why I would want to modify any of those settings, they are running as bridge networks.

      • lemmyvore@feddit.nlEnglish
        1·
        9 hours ago

        seems to have identified this as some NAT setting built in to Docker

        TBH I also thought about it being the default range used by Docker for its bridge networks, but I was under the impression it uses 172. ranges by default. The official example on their site says it uses 172.30.0.0/16 and 172.31.0.0/16. Also I have an unmodified docker running on one of my machines and it allocated 172.18.0.0/16 for a bridge network.

        You can try to docker network create a bridge network and see what range it gets, you’ll see if it gets a 172. or a 168..

        BTW I hope you’re looking at the docker on the reverse proxy machine, not on the jellyfin machine.

          • lemmyvore@feddit.nlEnglish
            1·
            3 hours ago

            Hope you saw my other comment. Docker now also makes 192.168. networks in addition to 172. I’ve explained how to edit your daemon.json to stop it from doing that.

            • ampersandrew@lemmy.worldOPEnglish
              1·
              3 hours ago

              I can take a look after work, but won’t that just change the proxy’s IP listed in my logs instead of properly forwarding along the source IP address?

              • lemmyvore@feddit.nlEnglish
                1·
                2 hours ago

                Yep, you’ll have to fix that by telling your proxy to put the client IP in a header that the target app can recognize and use:

                https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Forwarded-For

                But, also, it’s a good idea to have Docker network ranges that don’t overlap your LAN. That would make me uncomfortable too.

                You can also look into using IPv4 link-local addresses for Docker bridges (169.254.). Here’s an explanation:

                https://github.com/11notes/RTFM/blob/master/linux/container/docker/daemon.md#networking

                That frees you from accidental overlaps with pretty much everything else, but it comes with its own gotcha: link-local isn’t routable. This can come back to bite you if you have an app that relies on announcing a public UDP ip:port to a peer and announces the link-local IP. When a peer tries to connect back, it won’t work. So you have to either tweak the app’s config to make sure you announce a non-169.254. IP, or put the app container on a bridge/macvlan network so it auto-detects and announces a routable IP. There’s all kinds of apps that can be affected by this kind of peer-to-peer behavior (file transfer apps, games, VoIP apps etc.)

                But TBH I’ve been using link-local ranges for a while now and only got bitten once (Deluge).

                • ampersandrew@lemmy.worldOPEnglish
                  1·
                  2 hours ago

                  Yep, you’ll have to fix that by telling your proxy to put the client IP in a header that the target app can recognize and use.

                  I thought that’s what I was trying to do by adjusting both NPM proxy host settings as well as informing Jellyfin where to expect proxy headers from.

                  But, also, it’s a good idea to have Docker network ranges that don’t overlap your LAN. That would make me uncomfortable too.

                  Does it overlap my LAN? Is 192.168.65.X not distinguishable from my defined subnets with a 24-bit mask, now that I know where they’re coming from?

                  And to be honest, you’ve talked past my understanding with a lot of that paragraph, so I’ll have to comb through it later tonight when I have more time. Keep in mind that all of my Docker network settings are default, so it’s a setting I’ve never dabbled with. From what I can tell from the settings it defined on its own, these containers created a 172.17.0.X address, and somehow the proxy responds with (“announces”?) a 192.168.65.1 address. Are you saying that I need to define for it an unused address that’s on my existing 10 VLAN? Somehow all of the tutorials I’ve found in the past months to set up services behind a reverse proxy never mentioned any of these gotchas with Docker and networks.

                  • lemmyvore@feddit.nlEnglish
                    2·
                    1 hour ago

                    Yeah that’s because most people are under the impression that Docker only allocates 172. addresses for its bridge network. I was under that impression too.

                    Turns out it also allocates 192.168 addresses.

                    Does it overlap my LAN? Is 192.168.65.X not distinguishable from my defined subnets with a 24-bit mask, now that I know where they’re coming from?

                    The problem is that the default says 192.168.0.0/16, so it can also generate overlapping addresses. The fact it generated a 192.168.65.x address instead of an 192.168.1.x address was pure luck.

                    Somehow all of the tutorials I’ve found in the past months to set up services behind a reverse proxy never mentioned any of these gotchas with Docker and networks.

                    The common-sense assumption is that you will adjust your Docker daemon settings so that you prevent overlap with your LAN. (Also, I expect lots of tutorials aren’t that high quality to begin with.)

                    If you edit daemon.json like I explained and explicitly only put 172. ranges in there you should be safe from overlaps in the future.