smy-skills/linux-ssh-operator-skill/references/ssh-playbook.md

# SSH playbook (Linux server ops)

## Defaults and conventions

- Prefer SSH keys (ed25519) and `~/.ssh/config` aliases for repeatable runs.
- Avoid putting passwords in files, prompts, or chat logs. If password auth is required, use an interactive terminal/TTY.
- Start with read-only inspection, then apply changes, then verify.

Recommended env vars for wrappers:

- `REMOTE_USER`: default SSH user
- `REMOTE_PORT`: default SSH port (usually 22)
- `REMOTE_KEY`: path to identity file (private key)
- `REMOTE_CONNECT_TIMEOUT`: connect timeout seconds

## SSH key setup (recommended)

Generate a new key:

```bash
ssh-keygen -t ed25519 -C "codex" -f ~/.ssh/id_ed25519
```

Copy the public key to the server:

```bash
ssh-copy-id -i ~/.ssh/id_ed25519.pub -p 22 USER@SERVER_IP
```

Add a host alias:

```sshconfig
Host my-server
  HostName SERVER_IP
  Port 22
  User USER
  IdentityFile ~/.ssh/id_ed25519
  IdentitiesOnly yes
```

## Common tasks

### Connectivity and OS info

```bash
ssh my-server "whoami && hostname && uname -a"
ssh my-server "cat /etc/os-release"
```

### Disk and memory

```bash
ssh my-server "df -h"
ssh my-server "free -h"
ssh my-server "du -sh /var/log/* | sort -h | tail"
```

### Processes and ports

```bash
ssh my-server "ps aux --sort=-%mem | head"
ssh my-server "ss -lntp"
ssh my-server "lsof -i :8080"
```

### Logs (systemd)

```bash
ssh my-server "journalctl -u SERVICE -n 200 --no-pager"
ssh my-server "journalctl -u SERVICE -f --no-pager"
ssh my-server "journalctl -u SERVICE --since '1 hour ago' --no-pager"
```

### Services (systemd)

Status:

```bash
ssh my-server "systemctl status SERVICE --no-pager"
```

Restart (often needs sudo and TTY):

```bash
ssh -tt my-server "sudo systemctl restart SERVICE"
```

Non-interactive sudo (fails if a password prompt would be required):

```bash
ssh my-server "sudo -n systemctl restart SERVICE"
```

### Docker

```bash
ssh my-server "docker ps"
ssh my-server "docker logs --tail=200 -f CONTAINER"
ssh my-server "docker exec -it CONTAINER bash"
ssh my-server "docker stats --no-stream"
```

### File editing shortcuts

```bash
# View a config file
ssh my-server "cat /etc/nginx/nginx.conf"

# Quick in-place line replacement
ssh my-server "sed -i 's/old/new/g' /path/to/file"

# Append a line
ssh my-server "echo 'new line' >> /path/to/file"
```

## SSH Tunneling / Port Forwarding

### Local forward

Access a remote service on a local port (server-side port does not need to be publicly open):

```bash
# remote MySQL → localhost:13306
ssh -L 13306:127.0.0.1:3306 my-server -N

# remote Redis → localhost:16379
ssh -L 16379:127.0.0.1:6379 my-server -N
```

Add `-f` to background the tunnel:

```bash
ssh -fNL 13306:127.0.0.1:3306 my-server
```

### Remote forward

Expose a local service on the server (useful for demos or webhooks):

```bash
# local :8080 → server :18080
ssh -R 18080:127.0.0.1:8080 my-server -N
```

### Jump host / bastion

```bash
# Single jump
ssh -J bastion user@target

# Multi-hop
ssh -J jump1,jump2 user@target
```

`~/.ssh/config` version:

```sshconfig
Host target-internal
  HostName 10.0.0.50
  User ubuntu
  ProxyJump bastion
```

## SSH Multiplexing

Reuse the same TCP connection for multiple SSH sessions — avoids repeated handshakes:

```sshconfig
Host *
  ControlMaster auto
  ControlPath ~/.ssh/ctrl-%h-%p-%r
  ControlPersist 60s
```

Check active master sockets:

```bash
ls ~/.ssh/ctrl-*
```

Close a master socket manually:

```bash
ssh -O exit my-server
```

## Safer host key handling

- Prefer verifying the host key fingerprint out-of-band on first connect.
- If you must automate first-connect for ephemeral hosts, use `StrictHostKeyChecking=accept-new` (OpenSSH 7.6+).
- If you see a "host key changed" warning, treat it as a potential security incident until you confirm the change is expected.

Remove a stale known_hosts entry:

```bash
ssh-keygen -R SERVER_IP
```

## Troubleshooting quick hits

| Symptom | Likely cause | Fix |
|---|---|---|
| `Permission denied (publickey)` | Wrong user/key, pub key not on server, sshd config | Check `~/.ssh/authorized_keys` on server; verify key matches |
| `Connection timed out` | Firewall/security group, wrong port, server down | Check port, security group, ping |
| `No route to host` | Network path missing (VPN, ACL, subnet) | Check VPN, routing table |
| `Host key changed` warning | Server rebuilt, or MITM | Verify with out-of-band channel; `ssh-keygen -R HOST` if expected |
| `Too many authentication failures` | SSH agent offering too many keys | Use `-o IdentitiesOnly=yes -i ~/.ssh/id_ed25519` |
| `sudo: a terminal is required` | No TTY allocated | Use `ssh -tt` or pass `-t` to wrapper |

### Debug connection verbosely

```bash
ssh -vvv my-server
```

### Check sshd config on server

```bash
ssh my-server "sudo sshd -T | grep -E 'pubkeyauthentication|passwordauthentication|permitrootlogin'"
```

### Check authorized_keys permissions (common cause of publickey failure)

```bash
ssh my-server "ls -la ~/.ssh/ && cat ~/.ssh/authorized_keys"
# .ssh should be 700, authorized_keys should be 600
```