I'm receiving "Connection refused" or "Connection timed out" errors when trying to connect to my EC2 instance with SSH. How do I resolve this?

 

Error message: "ssh: connect to host ec2-X-X-X-X.compute-1.amazonaws.com port 22: Connection timed out". This error message comes from the SSH client. The error indicates that the server didn't respond to the client and the client program gave up (timed out). The following are common causes for this error:

  • The security group or network ACL doesn't allow access.
  • There's a firewall on the instance's operating system.
  • There's a firewall between the client and the server.
  • The host doesn't exist.

Error message: "ssh: connect to host ec2-X-X-X-X.compute-1.amazonaws.com port 22: Connection refused". This message comes remotely from a host. The following are common causes for this error:

  • The host reached the instance but there was no service listening on the SSH port.
  • A firewall blocked and was set to reject the package instead of dropping it.

Resolution

For the “Connection timed out” error, verify the following

  • The instance's IP address or hostname is correct.
  • The instance is passing its health checks.
  • The security group of the instance allows incoming traffic on TCP port 22.
  • The network ACLs of instance subnet allows incoming traffic on TCP port 22 and allow ephemeral port for the outgoing traffic.
  • The route table of the instance’s subnet is configured properly to provide connectivity between EC2 instance and the SSH client.
  • There isn't a firewall blocking the connection between SSH client and the EC2 instance.
  • SSH isn't blocked by TCP Wrappers in the instance.

Note: The last two verification steps require OS-level access of the instance.

For the “Connection refused” error, verify the following

  • There's no firewall on the instance rejecting the SSH connection.
  • The SSH daemon (sshd) is running and listening on port 22.

Note: Both verification steps require OS-level access of the instance.

If the instance passes both health checks, use one of the following four listed methods with your configuration

  • Method 1: Use the EC2 Serial Console for Linux.
  • Method 2: Use AWS Systems Manager Session Manager.
  • Method 3: Run the AWSSupport-TroubleshootSSH automation runbook.
  • Method 4: Use a user data script.

Method 1: Use the EC2 Serial Console for Linux

If configured, you can use the EC2 Serial Console for Linux to troubleshoot OS-level issues on supported Nitro-based instance types. The serial console allows troubleshooting of boot issues, network configuration, and SSH configuration issues. The serial console is accessible using the Amazon EC2 console or the AWS Command Line Interface (AWS CLI).

Before using the serial console, grant access to it at the account level. Then, create AWS Identity and Access Management (IAM) policies granting access to your IAM users.

Note: Each instance using the serial console must include at least one password-based Linux user with sudo access.

For more information on configuring the EC2 Serial Console for Linux, see Configure access to the EC2 Serial Console. If there isn’t a Linux account with a login password configured, you must run ssm-user to reset the password for an account with sudo access. For more information on running ssm-user commands, see the section Managing ssm-user sudo account permissions on Linux and macOS.

After configuration, connect to the EC2 instance through the EC2 serial console using a password-configured Linux user.

Next, run the following commands. These commands verify that the SSH connections aren't being blocked by the OS firewall or TCP wrapper. The commands also verify that the sshd service is running and listening on port 22.

1.    If you have iptables rules configured, then run following command to add a rule in iptables accepting all SSH connections on default port 22:

$ sudo iptables -I INPUT 1 -p tcp --dport 22 -j ACCEPT

Because it’s a best practice to use security groups instead of an OS-based firewall, the firewall can be deactivated altogether. To deactivate the OS-based firewall, use one of the following set of commands, depending on your operating system:

Important: The following commands flush all main iptables rules. They also add a rule allowing incoming SSH connections. Additionally, they change the default policy of the main chain to ACCEPT so that flushing the iptables rule doesn’t affect instance network connectivity. Because these commands flush all main iptables, they also flush any existing rules.

Distributions that use UFW (Ubuntu, Debian)

$ sudo iptables -F
$ sudo iptables -P INPUT ACCEPT
$ sudo ufw disable

Distributions that use firewalld (Red Hat, CentOS)

$ sudo iptables -F 
$ sudo iptables -P INPUT ACCEPT
$ sudo systemctl disable firewalld

Note: After you regain access to your instance, review your firewall configuration (UFW, firewalld, iptables).

2.    Verify that SSH is running and verify that the SSH TCP port (22) is in listening state:

$ sudo systemctl restart sshd
$ sudo ss -tpln | grep -iE '22|ssh'
LISTEN 0 128 *:22 *:* users:(("sshd",pid=1901,fd=3))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1901,fd=4))

Note: If your system doesn't have the ss command, you can use the legacy netstat command with the same syntax shown in the preceding example.

3.    Make sure that the TCP wrapper isn’t blocking an SSH connection:

$ if [[ $( cat /etc/hosts.[ad]* | grep -vE '^#' | awk 'NF' | wc -l) -ne 0 ]];\
then sudo sed -i '1i sshd2 sshd : ALL: allow' /etc/hosts.allow; fi

4.    Next, connect to the instance using SSH.

5.    Disconnect the EC2 Serial Console session if it's no longer required.

Method 2: Use AWS Systems Manager Session Manager

Note: To use this method, the instance must be an SSM managed instance and its SSM agent ping status must be Online. For more information on Session Manager and a complete list of prerequisites, see Setting up Session Manager.

To confirm that SSH connections aren't being blocked by the firewall or TCP wrapper and that the sshd service is running and listening on port 22:

1.    Open the AWS Systems Manager.

2.    Start a session for the instance using Session Manager.

3.    Follow steps 1 - 4 from Method 1: Use EC2 Serial console for Linux.

4.    Close the Session Manager session if it's no longer required.

Method 3: Run the AWSSupport-TroubleshootSSH runbook

The AWSSupport-TroubleshootSSH automation runbook installs the Amazon EC2Rescue tool for Linux on the instance. This tool checks for and attempts to fix issues that prevent a remote connection Linux host though SSH.

To run the AWSSupport-TroubleshootSSH runbook:

1.    Open the AWSSupport-TroubleshootSSH page.

2.    Select Run this Automation (console).

To learn more, see How can I use the AWSSupport-TroubleshootSSH Automation workflow to troubleshoot SSH connection issues?

Method 4: Use a user data script

If none of the methods described are suitable in your environment, then use an EC2 user-data script. The EC2 user-data script turns off the OS-level firewall and the TCP wrapper, and then restarts the sshd service.

Important:

  • This procedure requires a stop and start of the EC2 instance. If the instance has any data stored on instance store volumes, that data is deleted after stopping the instance.
  • If the instance is part of an Amazon EC2 Auto Scaling group, then terminating the instance might also stop instances within the Auto Scaling group.
  • If the instance is launched by services that use AWS Auto Scaling, then terminating the instance might also stop instances within the Auto Scaling group.
  • Instance termination depends on the instance scale-in protection settings for the Auto Scaling group. If the instance is part of an Auto Scaling group, temporarily remove the instance from the Auto Scaling group before starting the resolution steps.
  • Stopping and starting the instance changes the public IP address of the instance. It's a best practice to use an Elastic IP address instead of a public IP address when routing external traffic to the instance.

Follow these steps to configure user-data for the instance:

1.    Open the Amazon EC2 console.

2.    Choose Instances from the navigation pane, and then select the instance that you plan to connect to.

3.    Stop the instance.

4.    Choose Actions, Instance Settings, Edit User Data.

5.    Copy the following user data script into the Edit User Data dialog box, and then choose Save.

Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
 
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
 
#cloud-config
cloud_final_modules:
- [scripts-user, always]
 
--//
Content-Type:
    text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
 
#!/bin/bash
iptables -P INPUT ACCEPT
iptables -F
systemctl restart sshd.service || service sshd restart
if [[ $( cat /etc/hosts.[ad]* | grep -vE '^#' | awk 'NF' | wc -l) -ne 0 ]];\
then sudo sed -i '1i sshd2 sshd : ALL: allow' /etc/hosts.allow; fi
--//

6.    Connect to the instance using SSH.

7.    The preceding user data script is set to run on every reboot of the instance. After regaining access to the instance, remove the user data script.

Note: The preceding command flushes all main iptables rules. After regaining access to the instance, review the firewall configuration for accuracy (for example, UFW, firewalld, iptables).

To remove user data:

1.    Complete steps 1–4 in the Method 4: Use a user data script section.

2.    Delete the user data script in the Edit User Data dialog box.

Comments

Post a Comment

Popular Posts