Solution somewhat like a bastion host with port knocking and forwarding

[cro]

Greetings,

I'm building an SSH proxy server that dynamically looks up its target and I have it almost completely working.

Client --> host 1           --> host 2       --> socat port forward   --> container or kvm instance 
           asyncssh server  asyncssh target      listening on random      running openssh-sshd
                                                 port, forwarding to      on port 22
                                                 container non-routable
                                                 ip, port 22

I can ssh to host 1 with a specially-formed username, the server on host 1 dynamically looks up what host the container is running on, connects to the right host (host 2 in the diagram). The Asyncssh script picks a random port and spawns a socat pointing from the random port to port 22 on the non-routable IP where the container is, then directs the incoming connection from host 1 to use that port. Thus the client connects transparently to the ssh daemon in the container.

Where I'm getting tripped up is the ssh port-forwarding functionality. Without any additional logic I can do ssh special-username@host1 -L 9000:localhost:9000 but nc localhost 9000 on the client seems to connect me to host2's ssh daemon (or maybe the container's?). So clearly I need to override something somewhere but I'm not sure what. Any suggestions?

[ronf]

In your AsyncSSH server on host 1, have you enabled port forwarding? To do so, you'd need to have your own subclass of SSHServer which implements the connection_requested() method. By default, this returns False, which denies all such requests.

If you just need to do standard port forwarding to the host/port requested by the SSH client, you can have this method always return True. However, you can also look at the original host & port and the destination host & port to decide whether to allow each forwarding request, if you want to limit who can connect or where they can connect to.

You can also return a custom session class or callable if you want to actually get involved in viewing and/or modifying the data being forwarded, but you probably don't need that here.

[cro]

Thanks so much for responding so quickly!

In your AsyncSSH server on host 1, have you enabled port forwarding? To do so, you'd need to have your own subclass of SSHServer which implements the connection_requested() method. By default, this returns False, which denies all such requests.

Yes, we've done this.

You can also return a custom session class or callable if you want to actually get involved in viewing and/or modifying the data being forwarded, but you probably don't need that here.

Yes, I think we need to, so we've written this. Here's the connection_requested() method from our subclassed SSHServer and _open_backend_stream()

    async def connection_requested(
        self, dest_host: str, dest_port: int, orig_host: str, orig_port: int
    ) -> bool:
        """Allow local port forwarding"""
        await self.hub.log.info(
            f"Local forward requested: {dest_host}:{dest_port} "
            f"(origin {orig_host}:{orig_port})"
        )

        result = await self._open_backend_stream(dest_host, dest_port, orig_host=orig_host, orig_port=orig_port, request_type="direct-tcpip")
        return result

    async def _open_backend_stream(
        self, dest_host, dest_port, *, orig_host=None, orig_port=None, request_type: str
    ):
        """Open a direct-tcpip channel on the backend to the requested destination."""
        if dest_host is None or dest_port is None:
            raise asyncssh.ChannelOpenError(
                asyncssh.OPEN_CONNECT_FAILED,
                f"Missing destination for {request_type} forwarding",
            )

        if isinstance(dest_host, (bytes, bytearray)):
            dest_host = dest_host.decode("utf-8", "ignore")
        else:
            dest_host = str(dest_host)

        try:
            dest_port_int = int(dest_port)
        except (TypeError, ValueError):
            raise asyncssh.ChannelOpenError(
                asyncssh.OPEN_CONNECT_FAILED,
                f"Invalid destination port {dest_port!r} for {request_type} forwarding",
            )

        # backend will contain the SSHClient instance for the current open connection
        backend = await self.get_backend_connection()
        await self.hub.log.debug(
            f"Opening backend {request_type} stream to {dest_host}:{dest_port_int}"
        )

        try:
            tcpcon = await backend.create_connection(asyncssh.SSHTCPSession, dest_host, dest_port_int, orig_host, orig_port)
        except Exception:
            await self.hub.log.error(
                f"Failed to open backend {request_type} stream to "
                f"{dest_host}:{dest_port_int}",
                exc_info=True,
            )
            raise

        return tcpcon

If I simply return True from connection_requested() I get this traceback:

  File "/usr/local/lib/python3.13/dist-packages/asyncssh/channel.py", line 231, in _cleanup
    self._session.connection_lost(exc)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'bool' object has no attribute 'connection_lost'

If I don't return True and go through the rest of the code there in _open_backend_stream() I should be returning a tuple of an SSHTCPChannel and SSHTCPSession. In that case I get a similar traceback saying

Traceback (most recent call last):
  File "/usr/local/lib/python3.13/dist-packages/asyncssh/channel.py", line 231, in _cleanup
    self._session.connection_lost(exc)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'connection_lost'
[asyncssh         ][INFO    ] [conn=0, chan=2] Channel closed: 'tuple' object has no attribute 'connection_made'

So I'm obviously not returning the right thing, and I'm confused.

To add a little extra detail, to test I'm ssh'ing with -L 9999:localhost:9999 and in the ultimate target shell I'm running nc -l 9999. Then from the client side I'm running nc localhost 9999. If I don't have the listener side of nc running I get a connection refused which is expected and seems to indicate that the port forwarding is being setup correctly.

Thanks again!

[ronf]

Does the example at https://asyncssh.readthedocs.io/en/latest/#id16 work for you? It shows an example of implementing connection_requested() and returning True. I connected to it using ssh -L 8080:localhost:80 -p 8022 -N localhost and could then connect to port 8080 with a browser and the traffic should get forwarded to a local web server running on port 80.

You can also try the example at https://asyncssh.readthedocs.io/en/latest/#forwarded-tcp-connections to see if returning a custom TCPSession works. It should accept a TCPChannel and TCPSession tuple as well, but there's very little reason to create your own custom TCPChannels. It's mainly useful if you want to read & write Unicode strings instead of bytes or set different window or max packet sizes.

If you do decide you need to be involved in reading/writing the data, you can also return a callable that takes reader & writer arguments. That's typically simpler than building your own custom TCPSession with things like data_received() and eof_received() callbacks. There's an example of that just below the link I sent in the last paragraph.

[ronf]

FYI here's the code which handles the value you return from connection_requested():

        if not result:
            raise ChannelOpenError(OPEN_CONNECT_FAILED, 'Connection refused')

        if result is True:
            result = cast(SSHTCPSession[bytes],
                          self.forward_connection(dest_host, dest_port))
        elif isinstance(result, SSHClientConnection):
            result = cast(Awaitable[SSHTCPSession[bytes]],
                          self.forward_tunneled_connection(
                              result, dest_host, dest_port))

        if isinstance(result, tuple):
            chan, result = result
        else:
            chan = self.create_tcp_channel()

        session: SSHTCPSession[bytes]

        if callable(result):
            session = SSHTCPStreamSession[bytes](result)
        else:
            session = cast(SSHTCPSession[bytes], result)

As you can see, it converts result into a TCPChannel and TCPSession before it returns, or it raises an error. It has handling for any value which maps to a boolean value of False to error out, as well as the specific value True which leverages a built-in forward_connection() method in that case (which you can also call and return yourself if you want to connect to a different dest_host/dest_port than what the client requests). It will also automatically create a TCPChannel if you don't return a tuple, and it has a special case for opening a new direct-tcpip session on some other SSHConnection if you return one of those. It also handles you returning a callable for the stream version of this, automatically creating an SSHTCPStreamSession for you. If the object you return is none of those things, it assumes it's an SSHTCPSession or something which mimics one.

[ronf]

Is backend here an SSHConnection? I'm thinking one issue might be that create_connection() isn't quite the right way to forward data. If you want to accept data on one SSH connection but forward it to another as an outbound direct-tcpip connection, you should just return your upstream SSHConnection object as the result of connection_requested(). As you can see from the above code, there's a special forward_tunneled_connection() method for doing this. It uses create_connection(), but also sets up SSHForwarder objects which are needed for this:

    async def forward_tunneled_connection(
            self, conn: SSHClientConnection,
            dest_host: str, dest_port: int) -> SSHForwarder:
        """Forward a tunneled TCP connection between SSH connections"""

        _, peer = await conn.create_connection(
            cast(SSHTCPSessionFactory[bytes], SSHForwarder),
            dest_host, dest_port)

        self.logger.info('  Forwarding TCP connection to %s via SSH tunnel',
                         (dest_host, dest_port))

        return SSHForwarder(cast(SSHForwarder, peer))