SSH Directly into Slurm Job

Connect your editor or terminal to the requested Slurm resources like a breeze.

Command

Host <bastion>
  HostName     <bastion-host>
  User         <user>
  IdentityFile <identity-file>

Host <bastion>-sleeper
  ProxyCommand ssh -q <bastion> "bash -l -c 'j=\$(squeue --me --states=R --noheader --format=%%i --name="sleeper" | head -n1); n=\$(scontrol show hostnames \$(squeue -j \$j --noheader --format=%%N) | head -n1); exec srun -u --jobid=\$j --overlap --nodes=1 --ntasks=1 --nodelist=\$n nc \$n 22'"
  User         <user>
  IdentityFile <identity-file>

  StrictHostKeyChecking no
  UserKnownHostsFile    /dev/null
  LogLevel              ERROR

Add this configuration to your ~/.ssh/config, and then SSH to your latest job named sleeper with ssh <bastion>-sleeper, or connect your Visual Studio Code Remote SSH or Zed Remote using the host <bastion>-sleeper.

A benefit of this approach: you don't need to use a fixed node, or copy-paste the node allocated for your job. All nodes running the sleeper job are collapsed into one host. This also keeps a clean history for your shell and editor workspaces.

Explanation

ProxyCommand

This command creates a remote socket to the sleeper job. It first connects to the <bastion> host. Then, it executes a command to find the compute node running your job:

• Query Slurm job ID: squeue --me --states=R --noheader --format=%i --name=sleeper | head -n1
  • squeue queries Slurm for your (--me) currently running (--states=R) jobs named sleeper (--name=sleeper), omits the table header (--noheader), and prints only the job ID (--format=%i).
  • head -n1 selects the first matching job ID.
• Query node hostname: scontrol show hostnames $(squeue -j $j --noheader --format=%N) | head -n1
  • Inner squeue looks up the nodelist (--format=%N) for job $j. If a job spans multiple nodes, squeue returns a compressed host-list expression (e.g., gpu[01-02]), which nc cannot resolve as a hostname.
  • Outer scontrol show hostnames expands the host-list expression into one hostname per line, and head -n1 picks the first node.
• Netcat: srun -u --jobid=$j --overlap --nodes=1 --ntasks=1 --nodelist=$n nc $n 22
  • -u (--unbuffered) disables srun's default line-buffering of task stdout. SSH binary handshake does not use newlines like interactive texts, with line-buffering the connection will hang indefinitely.
  • --jobid=$j --overlap --nodes=1 --ntasks=1 --nodelist=$n launches an overlapped job sharing resources (CPUs, GPUs, memory) already allocated to the sleeper job. Restricted to 1 task on 1 node.
  • nc $n 22 runs a netcat within the srun step. The srun will relay stdin/stdout between our local ssh client and nc.

nc $(squeue --me --states=R --noheader --format=%N --name="debug" | head -n 1) 22

Host Key Checking

  StrictHostKeyChecking no
  UserKnownHostsFile    /dev/null
  LogLevel              ERROR

Since the compute node assigned to your job changes every time you start a new session, the host key will constantly change.

• StrictHostKeyChecking no: Disables the safety check that blocks connection when a host key changes.
• UserKnownHostsFile /dev/null: Prevents saving the host key to your known_hosts file, avoiding "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" errors in the future.
• LogLevel ERROR: Suppresses the warning messages about the host key being added to the list of known hosts (which is /dev/null anyway).

Prerequisites

Ensure you have an interactive job running with the matching name (debug in this case) before connecting. And this job should occupy 1 node only (otherwise Slurm might output node[01-02] which is not a valid hostname).

salloc --name=sleeper --time=1:00:00 --node 1 ...

Also this assumes your cluster uses node names as hostnames. Otherwise you might need to use scontrol show hostnames to resolve it first.