Troubleshooting SSH Connection Failures: From Network to Service and Emergency Access

The biggest problem when SSH is unreachable is that it is the only entry point; if you can’t connect, you can’t troubleshoot. Therefore, you must first distinguish between “network connectivity issues” and “service problems,” and know whether there are other entry points available.

First, Determine If There Are Other Entry Points

This step should be clarified during normal operations, rather than discovered only when connection fails:

• VNC / Remote Console via the cloud control panel (provided by almost all cloud providers)
• Rescue Mode / System Repair via the cloud control panel
• Whether there are other machines with internal network access to reach this server

VNC does not depend on the SSH service, so even if sshd crashes, configuration is broken, or network rules block it, VNC usually remains accessible. This is your last resort.

Layered Troubleshooting

Layer 1: Is the Network Connected?

# Test from your local machine (do not test only on the server)
ping <Server_IP>
nc -zv <Server_IP> 22
telnet <Server_IP> 22

# Check for Timeout vs. Refused
# Timeout → Network/Firewall blocking
# Refused → No service listening on the port

“Timeout” and “Refused” are two different signals that directly distinguish between blocking and a non-running service.

Layer 2: Security Groups and Firewalls

This layer is outside the server and must be checked in the cloud console:

• Are inbound rules allowing port 22?
• Does the source IP range include your current outbound IP? (This is the most common pitfall when changing network environments)

# If you have another entry point, check the system firewall on the machine
firewall-cmd --list-all
iptables -L -n | grep 22

Layer 3: sshd Service

# Check after logging in via VNC
systemctl status sshd
ss -lntp | grep :22

# Check logs (most important)
journalctl -u sshd --no-pager | tail -50
tail -50 /var/log/secure        # CentOS/RHEL family
tail -50 /var/log/auth.log      # Debian/Ubuntu family

Layer 4: Configuration Errors

# If you can't connect after changing config, it's likely here
sshd -t                          # Syntax validation (finds issues without restarting)
grep -E 'Port|PermitRootLogin|PasswordAuthentication|AllowUsers' /etc/ssh/sshd_config

Common configuration issues:

• Changed Port but still trying to connect on 22
• Disabled password login but didn’t configure keys properly
• AllowUsers accidentally excluded yourself

# Confirm the port
grep '^Port' /etc/ssh/sshd_config
ss -lntp | grep sshd

Layer 5: Authentication Issues

# Client-side troubleshooting (view detailed process)
ssh -vvv user@<Server_IP>

# Common issue: Key permissions too broad
chmod 600 ~/.ssh/id_rsa
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

SSH will refuse to use private keys with overly permissive permissions, manifesting as “password prompt” or “permission denied”.

Layer 6: Resource Exhaustion

The characteristic of this situation is: it worked before, suddenly stopped working, but network and services appear normal.

# Check via VNC
free -m          # Memory exhausted? (OOM kills sshd)
df -h            # Disk full? (Prevents session creation)
df -i            # Inodes exhausted?
top              # Extremely high load?

A full disk causes SSH login failures, a point often overlooked—even if network and services seem fine.

Troubleshooting Commands Summary

# Local (Client Side)
nc -zv <Server_IP> 22          # Timeout or Refused?
ssh -vvv user@<Server_IP>       # View detailed handshake process

# Server Side (Login via VNC)
systemctl status sshd
ss -lntp | grep :22
sshd -t
journalctl -u sshd --no-pager | tail -30
df -h; df -i; free -m
tail -30 /var/log/secure

Preventive Measures

• When modifying SSH config, keep one established connection open; verify in a new window before disconnecting the old one.
• Do not disable password login unless keys are already configured and tested.
• Ensure whitelisting rules do not exclude your common outbound IPs.
• Regularly confirm VNC/Rescue Mode availability (this is the most critical fallback).

Decision Tree

SSH Connection Failed
├─ nc -zv Result
│  ├─ Timeout   → Network/Firewall Block → Check Security Group & System Firewall
│  └─ Refused   → No Service Listening   → sshd Not Running
├─ Service Normal But Still Failing
│  ├─ Config Changed      → Run sshd -t, Check Port/AllowUsers
│  ├─ Auth Failure Prompt → Key Permissions, Username, AllowUsers
│  └─ Ping OK But Handshake Hangs → Check Resources (Memory/Disk/Inodes)
├─ No Issues Found Above
│  └─ Use VNC / Rescue Mode Login (Last Resort)
└─ Disk Full / Inodes Full
   └─ Clean Up and Retry (via VNC)

Scenario Walkthrough: Changed SSH Port, Saved, Exited, Now Unreachable

Symptom: To reduce noise, you changed the remote login port.
After saving config and exiting, connecting with the new port fails, and the old port also doesn’t work.

1. First, confirm if there are other entry points → Cloud platform console login methods do not rely on the remote login service.
   This is the only remaining channel to enter the machine. This step should have been confirmed before changing config.
2. Enter via console, confirm host status → Host is running normally;
   the issue lies solely with the login service or the path.
3. Check if the service is actually listening on the new port → Listening is normal,
   indicating the config change was correct.
4. Distinguish between two layers of control → System firewall allowed it, but the cloud platform’s security group did not;
   external connections are still blocked.
5. Restore access after allowing traffic in the cloud platform → Connection succeeds immediately.
6. Standardize this sequence as a habit → Add new port first, verify connectivity, then remove old port;
   and always confirm console entry availability before making changes.

# First check service status and listening ports in the console
systemctl status sshd --no-pager | head -5
ss -lntp | grep -E ':22|:<New_Port>'

# Self-connection from the machine is the most direct boundary test
ssh -p <New_Port> user@127.0.0.1 'echo Local Connection OK'

# System Firewall
firewall-cmd --list-ports 2>/dev/null
iptables -L -n | grep <New_Port>

# Cloud Platform Security Groups can only be confirmed in the console, not inside the server

# Leave one connected session before modifying; don't break all connections at once.
# After modification, verify via a "new session" before closing the old one.

Root Cause: During the change, only one layer of the chain (system firewall) was opened;
the other layer (cloud platform security group) was not synchronized; and the original connection was dropped without reserving a verification window.

Two key takeaways:

• Before changing the remote port, confirm that a “second channel independent of it” is available.
  The cloud platform console login is this channel; you must verify it works before making changes.
• Do not disconnect the current connection immediately after changes.
  Keep the original session open, verify via a new session, and only then exit. This habit prevents locking yourself out.

Layered Symptoms and Verification Commands Reference

Troubleshooting order must be bottom-up: First confirm the host is alive, then confirm port reachability, and finally check SSH configuration.
Doing it in reverse leads to misleading results—spending time tweaking SSH config with no effect, while the actual cause is an unopened security group.

# Test port reachability from outside (execute on another machine)
nc -zv 1.2.3.4 22 -w 3

# Server-side: Confirm listening and connection count
ss -lntp | grep -E ':22|:[0-9]{4,5}'
ss -tn state established '( dport = :22 or sport = :22 )' | wc -l

# Check SSH server logs to determine where the handshake hangs
journalctl -u sshd --since '10 min ago' | tail -30
# Or (CentOS 6 / Legacy Systems)
tail -50 /var/log/secure

The step of “first determining if there are other entry points” comes before troubleshooting for practical reasons:
The cloud platform console’s VNC/Rescue Mode is the only entry point independent of SSH.
Confirming its availability before touching SSH config ensures a fallback if changes go wrong; skipping this step means a mistake leads directly to total loss of access.

8 Common Pitfalls

1. Changing port without opening firewall/security group: Changing only one side has no effect.
2. Disabling password login without verifying keys: Locks yourself out.
3. AllowUsers excludes yourself: Login becomes impossible after config takes effect.
4. Private key permissions too broad: SSH refuses to use the key.
5. Only pinging, ignoring ports: Ping success does not mean port 22 is open.
6. Ignoring full disk: A full disk causes login failures.
7. Not knowing where VNC is: Cannot use the last resort.
8. Failing to update security group after changing networks: Source IP changed, causing blocks.

Real Customer Feedback (From Cloud Marketplace Reviews and Inquiries for Similar Products)

The following excerpts are taken from real buyer reviews and inquiries for similar O&M services on Alibaba Cloud / Tencent Cloud / Huawei Cloud marketplaces (providers include our shop and peers, provided for reference only, not fictional tickets from this site):

b****4 Rating: 5.0 Jun 23, 2025 16:57 b****4 Rating: 5.0 Jun 23, 2025 16:56 Jian***Wei Rating: 5.0 Jan 20, 2021 15:03 Excellent technical skills, very fast! It turned out to be a trojan causing Baidu global hijacking!!!!!! f****u Rating: 5.0 Jan 02, 2021 20:43 Thanks for assisting in maintaining Yuncai’s normal operation. 1****1 Rating: 5.0 Nov 20, 2020 10:03 Consistently good service 1****1 Rating: 5.0 Nov 17, 2020 09:13 Thanks for the consistently good service. 1****1 Rating: 5.0 Sep 25, 2020 15:42 Many thanks. 1****1 Rating: 5.0 Aug 24, 2020 15:34 Many thanks to your company’s technical staff 1****1 Rating: 5.0 Jul 24, 2020 09:49 Many thanks! 1

10***77 2022-09-19 5.0 Affordable price | Skilled technology | Patient and professional Received filing rectification notice, found this service just in time, handled quickly 10***49 2022-04-12 5.0 On-time delivery | Affordable price | Skilled technology Needed rectification, fast configuration, not bad