proxy

README

proxy

SSH tunnel + dual-protocol proxy management for HPC compute nodes.

Two Modes

	Shared	Per-job
Started on	Login node	Compute node (inside job)
Bind	0.0.0.0:PORT	127.0.0.1:PORT
PID file	config.GetUserDataDir()/proxy.pid (NFS)	utils.GetTmpDir()/proxy.pid (local)
Lifetime	Until stopped	Until job ends

Daemon Architecture

job/login node
  RunDaemon()
    ├── tryEstablishTunnel()   ← SSH tunnel to login node
    │     ├── 1. DialGoSSH()        Go SSH library (no subprocess)
    │     ├── 2. ssh -D unix-sock   system ssh, Unix socket SOCKS5
    │     └── 3. ssh -D 127.0.0.1  system ssh, TCP SOCKS5 (OpenSSH <6.7)
    └── RunProxy(ctx, listenAddr, dial)
          ├── SOCKS5 (RFC 1928)      detected by first byte 0x05
          └── HTTP CONNECT           everything else

RunProxy serves both protocols on the same port. The DialFunc from whichever tunnel method succeeded is passed directly — no intermediate SOCKS5 layer for the go-ssh path.

SSH Tunnel Fallback Chain (daemon.go, ssh_dial.go)

tryEstablishTunnel tries three methods in order, returning on the first success:

Priority	Method	Condition
1	DialGoSSH — Go SSH library	Any non-interactive auth available
2	ssh -D /path/to.sock	system ssh binary in PATH
3	ssh -D 127.0.0.1:PORT	fallback for OpenSSH < 6.7 (no Unix socket -D)

Go SSH Auth Methods (ssh_dial.go)

buildAuthMethods assembles all available non-interactive auth methods — never prompts:

Method	Condition
ssh.Hostbased	ssh-keysign setuid binary found + host key in /etc/ssh/
ssh.PublicKeysCallback via agent	$SSH_AUTH_SOCK is set and connectable
ssh.PublicKeys via key files	~/.ssh/id_{ed25519,ecdsa,ecdsa_sk,ed25519_sk,rsa} readable

RSA keys are wrapped in rsaSHA2Signer to advertise and sign with rsa-sha2-256 consistently (fixes a golang.org/x/crypto/ssh mismatch where Type() returns ssh-rsa but Sign() uses SHA-256).

The Go SSH client sends a keepalive@openssh.com request every 60 s to prevent firewalls from silently dropping idle connections.

Key Functions

proxy.GetJobProxy() (string, bool)                   // cached; returns "", false on login nodes
proxy.FindActiveProxy() (string, bool)               // per-job → shared
proxy.ResolveViaHost(flagVia string) (string, error) // --via flag, or error if missing
proxy.StartOnNode(host, via string, port int) error  // SSH delegation; waits for NFS PID file
proxy.ProxyAlive(host string, port int) bool         // TCP dial, 500ms timeout
proxy.RunDaemon(sshDest string, port int, localOnly bool, reportFd int) error
proxy.DialGoSSH(sshDest string) (DialFunc, func(), <-chan struct{}, error)

Usage

proxy.GetJobProxy() is the call site for injection — resolves once per process via sync.Once, no-ops on login nodes. Used by internal/exec/run.go (container env vars) and internal/build/fetch.go (HTTP transport).

Daemon Readiness Protocol

proxy start uses an os.Pipe to synchronise with the daemon:

1. Parent creates (pr, pw), passes pw as ExtraFiles[0] (fd 3 in daemon) and --report-fd 3
2. Daemon closes pw (EOF) once tunnel is up and PID file is written → success
3. Daemon writes an error message before closing → startup failed, surfaced to user
4. Parent blocks on io.ReadAll(pr) — no polling needed

Documentation

Index

• func FindActiveProxy() (string, bool)
• func FreePort() (int, error)
• func GetJobProxy() (string, bool)
• func LocalPidFilePath() string
• func LocalSockPath() string
• func PidFilePath() string
• func ProxyAlive(host string, port int) bool
• func ProxyEnvList(httpURL string) []string
• func RemovePidFile()
• func ResolveViaHost(flagVia string) (string, error)
• func RunDaemon(sshDest string, port int, localOnly bool, reportFd int) error
• func RunProxy(ctx context.Context, listenAddr string, dial DialFunc) error
• func SockPath() string
• func StartOnNode(host, via string, port int) error
• func WillSurviveLogout() (survives, killUserProcesses, linger bool)
• func WritePidFileAt(path string, ps ProxyState) error
• type DialFunc
  • func DialGoSSH(sshDest string) (DialFunc, func(), <-chan struct{}, error)
  • func DialSystemSSH(node string) (DialFunc, func(), <-chan struct{}, error)
• type ProxyState
  • func ReadLocalPidFile() (*ProxyState, error)
  • func ReadPidFile() (*ProxyState, error)
  • func ReadPidFileAt(path string) (*ProxyState, error)

Functions

func FindActiveProxy

func FindActiveProxy() (string, bool)

FindActiveProxy returns the proxy URL to use for the current job. Checks the local per-job PID file first, then the shared NFS PID file.

func FreePort

func FreePort() (int, error)

FreePort asks the OS for an available TCP port on all interfaces and returns it.

func GetJobProxy

func GetJobProxy() (string, bool)

GetJobProxy returns the active proxy URL for injection into container envs and HTTP clients inside a scheduler job. Returns "", false on login nodes — they have direct internet access and need no proxy. Result is cached after the first call.

func LocalPidFilePath

func LocalPidFilePath() string

LocalPidFilePath returns the node-local proxy PID file path (per-job mode). Uses utils.GetTmpDir() which respects SLURM_TMPDIR, PBS_TMPDIR, etc.

func LocalSockPath

func LocalSockPath() string

LocalSockPath returns the Unix socket path for the per-job SSH SOCKS5 tunnel. Lives in the node-local tmpdir alongside LocalPidFilePath.

func PidFilePath

func PidFilePath() string

PidFilePath returns the NFS-shared proxy PID file path (shared mode).

func ProxyAlive

func ProxyAlive(host string, port int) bool

ProxyAlive checks if the SOCKS5 proxy is reachable at host:port.

func ProxyEnvList

func ProxyEnvList(httpURL string) []string

ProxyEnvList returns env var assignments for an http:// proxy URL. Both http:// and socks5:// point to the same dual-protocol port.

func RemovePidFile

func RemovePidFile()

RemovePidFile removes the shared NFS proxy PID file.

func ResolveViaHost

func ResolveViaHost(flagVia string) (string, error)

ResolveViaHost resolves the SSH server to tunnel through for a per-job proxy. Uses the --via flag value, or returns an error if not provided.

func RunDaemon

func RunDaemon(sshDest string, port int, localOnly bool, reportFd int) error

RunDaemon starts a dual-protocol (SOCKS5 + HTTP CONNECT) proxy on the visible port. It creates an SSH tunnel and runs a Go dual-protocol proxy in front of it.

localOnly=false: shared mode — visible proxy binds 0.0.0.0:port, NFS PID file. localOnly=true: per-job mode — visible proxy binds 127.0.0.1:port, node-local PID file.

reportFd, if > 0, is a file descriptor for a pipe back to the parent process: the daemon closes it (EOF) once the tunnel and PID file are ready, or writes an error message before closing if startup fails. The parent blocks on this pipe instead of polling the PID file.

The daemon blocks until SIGTERM/SIGINT, then cleans up the PID file and socket.

func RunProxy

func RunProxy(ctx context.Context, listenAddr string, dial DialFunc) error

RunProxy listens on listenAddr and serves both SOCKS5 and HTTP CONNECT, forwarding connections through dial. Stops when ctx is cancelled.

func SockPath

func SockPath() string

SockPath returns the Unix socket path for the shared-mode SSH SOCKS5 tunnel. Lives in the state dir (same node as the daemon; not accessed by compute nodes).

func StartOnNode

func StartOnNode(host, via string, port int) error

StartOnNode SSHes to host and runs "condatainer proxy start" there (delegation). Passes --via and --port if set. Waits up to 10s for the NFS PID file to appear.

func WillSurviveLogout

func WillSurviveLogout() (survives, killUserProcesses, linger bool)

WillSurviveLogout returns true if the proxy daemon is expected to survive user logout. Safe when KillUserProcesses=no OR Linger=yes. Returns (survives, killUserProcesses, linger).

func WritePidFileAt

func WritePidFileAt(path string, ps ProxyState) error

WritePidFileAt writes proxy state to an arbitrary PID file path as JSON.

Types

type DialFunc

type DialFunc func(ctx context.Context, network, addr string) (net.Conn, error)

DialFunc dials a network connection to addr. Compatible with net.Dialer.DialContext and http.Transport.DialContext.

func DialGoSSH

func DialGoSSH(sshDest string) (DialFunc, func(), <-chan struct{}, error)

DialGoSSH dials sshDest using the Go SSH library with all available non-interactive auth methods. Returns a DialFunc (wrapping client.Dial), a stop closer, and a done channel closed when the connection drops.

func DialSystemSSH

func DialSystemSSH(node string) (DialFunc, func(), <-chan struct{}, error)

DialSystemSSH dials a remote service by spawning `ssh -W node:port node` for each connection, piping the subprocess stdin/stdout as a net.Conn. This serves as a fallback when DialGoSSH fails (e.g. GSSAPI/Kerberos auth, strict known_hosts). The returned done channel is closed only on explicit stop(), not on connection drop, because there is no persistent underlying connection to monitor.

type ProxyState

type ProxyState struct {
	Host string `json:"host"`
	Via  string `json:"via"`
	Port int    `json:"port"`
	PID  int    `json:"pid"`
}

ProxyState holds the state written to a proxy PID file.

func ReadLocalPidFile

func ReadLocalPidFile() (*ProxyState, error)

ReadLocalPidFile reads the node-local per-job proxy PID file.

func ReadPidFile

func ReadPidFile() (*ProxyState, error)

ReadPidFile reads the shared NFS proxy PID file.

func ReadPidFileAt

func ReadPidFileAt(path string) (*ProxyState, error)

ReadPidFileAt reads and parses a JSON proxy PID file.