Troubleshooting Podman Container Access to the Host Network

Abstract: This article documents the troubleshooting process for a classic networking issue encountered when migrating services from Docker to Podman (in rootless mode): a container being unable to access a host service via the host’s IP address. By analyzing Podman’s network model and working through a complex SSH proxy scenario, the issue was ultimately identified and resolved.

1. Problem Statement

#

During the migration, a service that relied on the “container-accesses-host” pattern failed. In the Docker environment, this service could communicate directly using the host’s LAN or public IP address, but in a Podman rootless container, the connection timed out.

2. Principle Analysis: Docker vs. Podman Rootless Network Models

#

Initial analysis revealed that the problem stemmed from fundamental differences in their network models.

• Docker (Rootful): The Docker daemon runs with root privileges, creating kernel-level virtual bridges (like docker0) and actively modifying the host’s iptables/nftables rules to support “Hairpin NAT”. This allows traffic from a container destined for the host’s real IP to be correctly routed back to the loopback interface.
• Podman (Rootless): For security reasons, Podman in rootless mode uses a user-space network stack (like Netavark) and lacks the permission to modify system-level iptables/nftables. Consequently, when a container tries to access the host’s real IP, the resulting Hairpin NAT traffic is dropped by the kernel’s default policy, causing the connection to fail.

3. Solution: The Correct Internal Communication Mechanism

#

Since the external loopback path is blocked, the internal communication mechanisms provided by Podman must be used.

3.1. The Abstraction Layer: host.containers.internal

#

Podman provides a special DNS name, host.containers.internal, which is automatically resolved inside the container to the host’s address within the current container network. This is the preferred way to access the host, as it decouples the dependency on a specific IP.

Testing connectivity from within the container:

1$ podman exec -it my_container /bin/sh
2ping host.containers.internal
3PING host.containers.internal (169.254.1.2): 56 data bytes
464 bytes from 169.254.1.2: seq=0 ttl=42 time=0.285 ms
5...

The test result confirms that Layer 3 connectivity from the container to the host is working.

3.2. Firewall Policy

#

Traffic must be allowed through the host’s firewall. The core of the rule is to permit inbound traffic from the Podman subnet.

• Get the Podman network subnet:

1$ podman network inspect podman | grep subnet
2      "subnet": "10.89.0.0/24",

• Add a rule to nftables (using target port TARGET_PORT/tcp as an example):

1# Add a rule to the input chain of the default inet filter table
2sudo nft add rule inet filter input ip saddr 10.89.0.0/24 tcp dport TARGET_PORT accept

• Persist nftables rules: Rules added via the nft command will be lost on reboot. To make them permanent, save the current ruleset to the configuration file and enable the service.

1# Write the current ruleset to the configuration file
2sudo nft list ruleset > /etc/nftables.conf
3
4# Ensure the nftables service starts on boot to load the rules
5sudo systemctl enable nftables.service

4. Real-World Case Study: Accessing the Host’s SSH via a sing-box Proxy

#

After verifying the theory, I applied it to a complex real-world scenario for testing.

• Scenario: Remote SSH client (WindTerm) -> Local sing-box client -> Server’s sing-box container -> Host’s SSH service (listening on TARGET_PORT).
• Client Configuration: WindTerm’s proxy was set to the local sing-box (listening on LOCAL_PROXY_PORT), with “Remote DNS resolution” enabled. The SSH hostname was set to host.containers.internal.

Despite this setup, the connection still failed. This indicated the problem was not with the underlying network but with the logic of the proxy chain.

4.1. Server-Side Validation

#

To rule out server-side issues, I initiated an SSH connection directly from within the sing-box container.

1# Install and run ssh inside the sing-box container
2$ podman exec -it sing-box /bin/sh
3/ # apk add openssh-client
4/ # ssh [email protected] -p TARGET_PORT
5The authenticity of host ... can't be established.
6...
7Are you sure you want to continue connecting (yes/no)? yes
8# Successfully received the password prompt
9[email protected]'s password:

Conclusion: The server-side configuration was completely correct. The successful ssh connection proved that the network path from the container to the host, the firewall, and the SSH service itself were all working properly.

4.2. Client-Side Log Analysis and Final Diagnosis

#

Since the server side was fine, the problem had to be on the client’s proxy chain. I checked the logs of the local sing-box client and found the following critical entries:

1error [timestamp] connection: open outbound connection: NXDOMAIN
2info  [timestamp] outbound/direct[]: outbound connection to host.containers.internal:TARGET_PORT

The logs clearly pointed out the problem:

• The client received a request destined for host.containers.internal.
• However, its routing rules incorrectly matched this request to the direct (direct connection) outbound.
• The client then attempted to resolve host.containers.internal locally, resulting in an NXDOMAIN (domain does not exist) error.

4.3. Solution

#

Correct the routing configuration in the client-side sing-box, adding a high-priority rule to forcibly route traffic for host.containers.internal to the server proxy outbound (tagged here as proxy-out).

 1{
 2  "routing": {
 3    "rules": [
 4      {
 5        "domain": ["host.containers.internal"],
 6        "outbound": "proxy-out"
 7      },
 8      // ... other rules
 9    ]
10  }
11}```
12
13After applying this rule and restarting the client, the SSH connection succeeded.
14
15## 5. Summary
16
17*   The isolation of Podman's rootless networking is fundamental to its security but leads to different network behavior compared to Docker, especially in Hairpin NAT scenarios.
18*   `host.containers.internal` is the standard abstraction layer for a Podman container to access its host and should be used preferentially.
19*   You must configure corresponding firewall inbound rules for the Podman subnet.
20*   In complex proxy or network chains, end-to-end log analysis and testing with the native protocol at key nodes are the most efficient and reliable troubleshooting methods.