SSH stops working after upgrade to Ubuntu 18.04 on AWS Lightsail

:warning: IMPORTANT, please fill the questions

We assume you are using Bitnami to deploy your application.

  • Which version of the application are you using?:
    LAMP PHP 7(7.1.20-1)

  • Please choose how you got the application: Installer (Windows, Linux, macOS), cloud image (AWS, GCE, Azure, …) or VM (VMDK, VBOX):
    AWS Lightsail

  • Have you installed any plugin or modified any configuration file?:
    Stopped working after Ubuntu upgraded to 18.04

  • Describe here your question/suggestion/issue (expected and actual results):
    Had a working ssh for my AWS lighsail instance.

I upgraded by Ubuntu from 16.04 to 18.04 using $ sudo do-release-upgrade.

Upgrade went smoothly and asked for reboot in the end to complete the upgrade After that I have lost the ssh connect and since then reconnect does not work.Tried both putty and web ssh.Not working. I can see even sftp FileZilla has stopped working.

The instance is running (in AWS lighsail console) and I can also access the lightsail website that means upgrade has completed successfully and not crashed.

I have checked the lightsail networking tab and port 22 for ssh is enabled.I did a remove and add but did not help.

  • Steps to reproduce the issue (if relevant):

  • Copy the apache log (if relevant):

PASTE HERE

Hi @Ashd

I could not reproduce this from a LAMP with PHP7 from the AWS MarketPlace. I will check it using an AWS LightSail instance and will back to you.


Was my answer helpful? Click on :heart:

Hi @Ashd

I could not reproduce the issue. However, it seems the upgrade action starts a new ssh daemon at port 1022:

Continue running under SSH?

This session appears to be running under ssh. It is not recommended
to perform a upgrade over ssh currently because in case of failure it
is harder to recover.

If you continue, an additional ssh daemon will be started at port
'1022'.
Do you want to continue?

Continue [yN] y

Starting additional sshd

To make recovery in case of failure easier, an additional sshd will
be started on port '1022'. If anything goes wrong with the running
ssh you can still connect to the additional one.
If you run a firewall, you may need to temporarily open this port. As
this is potentially dangerous it's not done automatically. You can
open the port with e.g.:
'iptables -I INPUT -p tcp --dport 1022 -j ACCEPT'

To continue please press [ENTER]

This daemon might be still running so we can connect to the instance through it. Can you give it a try?

Looking for a reason, it seems the upgrade task also ask for modifying the ssh configuration:

Configuration file '/etc/ssh/ssh_config'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** ssh_config (Y/I/N/O/D/Z) [default=N] ?

If we select showing the differences, we get:

--- /etc/ssh/ssh_config 2018-08-15 10:44:39.331636732 +0000
+++ /etc/ssh/ssh_config.dpkg-new        2019-01-31 13:58:34.000000000 +0000
@@ -20,8 +20,6 @@
 #   ForwardAgent no
 #   ForwardX11 no
 #   ForwardX11Trusted yes
-#   RhostsRSAAuthentication no
-#   RSAAuthentication yes
 #   PasswordAuthentication yes
 #   HostbasedAuthentication no
 #   GSSAPIAuthentication no
@@ -33,16 +31,14 @@
 #   AddressFamily any
 #   ConnectTimeout 0
 #   StrictHostKeyChecking ask
-#   IdentityFile ~/.ssh/identity
 #   IdentityFile ~/.ssh/id_rsa
 #   IdentityFile ~/.ssh/id_dsa
 #   IdentityFile ~/.ssh/id_ecdsa
 #   IdentityFile ~/.ssh/id_ed25519
 #   Port 22
 #   Protocol 2
-#   Cipher 3des
-#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
-#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
+#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
+#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
 #   EscapeChar ~
 #   Tunnel no
 #   TunnelDevice any:any
@@ -53,7 +49,3 @@
     SendEnv LANG LC_*
     HashKnownHosts yes
     GSSAPIAuthentication yes
-    GSSAPIDelegateCredentials no
-
-Host *
-    UseRoaming no

I chose to install the package maintainer’s version as I think the changes will not break our current connection. After rebooting, I could ssh the instance and the ssh daemon is working as expected:

$ sudo netstat -plnt | grep 22
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1116/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1116/sshd

Was my answer helpful? Click on :heart:

1 Like

Hi jdrios, Many thanks for your checks and responses.

I did try on port 1022 but the issue is the same.

Regarding the modify the ssh configuration warning.I selected the default i.e “keep your currently-installed version”
Could this be the reason.Could you try doing that if that reproduces the issue for you.

In show the differences can you see many major highlights which can be causing this,Unfortunately I am not an expert in reading them appreciate if you have an interpretation.

In Lightsail there is an option to create a instance with the replica of the current instance and during the creation there is a option to run a start up script.Is there something based on the above interpretation I can out in the start script .
I tried creating an instance from replica as is (without running any startup script.)I see the same issue is on the new instance as well.

Looking forward for your inputs.Thanks…

Regards

Hi @Ashd

I reported this issue to the team and they will help you to dig into this from now on. About the differences you mentioned:

--- /etc/ssh/ssh_config 2018-08-15 10:44:39.331636732 +0000
+++ /etc/ssh/ssh_config.dpkg-new        2019-01-31 13:58:34.000000000 +0000
@@ -20,8 +20,6 @@
 #   ForwardAgent no
 #   ForwardX11 no
 #   ForwardX11Trusted yes
-#   RhostsRSAAuthentication no
-#   RSAAuthentication yes
 #   PasswordAuthentication yes
 #   HostbasedAuthentication no
 #   GSSAPIAuthentication no
@@ -33,16 +31,14 @@
 #   AddressFamily any
 #   ConnectTimeout 0
 #   StrictHostKeyChecking ask
-#   IdentityFile ~/.ssh/identity
 #   IdentityFile ~/.ssh/id_rsa
 #   IdentityFile ~/.ssh/id_dsa
 #   IdentityFile ~/.ssh/id_ecdsa
 #   IdentityFile ~/.ssh/id_ed25519
 #   Port 22
 #   Protocol 2
-#   Cipher 3des
-#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc
-#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160
+#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
+#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
 #   EscapeChar ~
 #   Tunnel no
 #   TunnelDevice any:any
@@ -53,7 +49,3 @@
     SendEnv LANG LC_*
     HashKnownHosts yes
     GSSAPIAuthentication yes
-    GSSAPIDelegateCredentials no
-
-Host *
-    UseRoaming no

Most of the lines have a # at the beginning. This means that the lines are no “enabled” in the configuration. Therefore, the only important change is this:

@@ -53,7 +49,3 @@
     SendEnv LANG LC_*
     HashKnownHosts yes
     GSSAPIAuthentication yes
-    GSSAPIDelegateCredentials no
-
-Host *
-    UseRoaming no

But it should not have an important impact on the configuration.

As Lightsail supports running a script on startup, we could try showing the current sshd configuration and starting an ssh daemon on a different port to access your instance (this port should be opened in the firewall first).


Was my answer helpful? Click on :heart:

Hi @Ashd, this is to be expected as Ubuntu 18.04, as it includes OpenSSH 7.6 which includes breaking changes, see: https://www.openssh.com/txt/release-7.6

Regarding what @jdrios mentioned, that is the /etc/ssh/ssh_config file for the SSH client. However the SSH daemon/server uses a different file, /etc/ssh/sshd_config which may also suffer important changes.

If you have a backup and can restore the instance, or if you still happen to be connected before restarting it, use this guide for fixing it: https://jupiterkenji888.wordpress.com/2018/05/16/ssh-bad-ssh2-cipher-spec/

If you do not have a backup, and you have important data inside the instance, unfortunately you will need to change /etc/ssh/sshd_config like explained here in hard way with a different instance, see: https://docs.bitnami.com/google/how-to/troubleshoot-ssh-issues/#hard-way-launching-a-new-instance

NOTE: You can find the full difference of sshd_config below. As you will see there are many required options not configured so using this file is not an option either (e.g. HostKey and others).

Line by line differences between versions

--- /etc/ssh/sshd_config root.root 0644 2019-02-20 14:10:37
+++ /tmp/fileLHavcu root.root 0644 2019-02-20 14:35:08
@@ -1,80 +1,76 @@
-# Package generated configuration file
-# See the sshd_config(5) manpage for details
+# $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $

-# What ports, IPs and protocols we listen for
-Port 22
-# Use these options to restrict which interfaces/protocols sshd will bind to
-#ListenAddress ::
+# This is the sshd server system-wide configuration file. See
+# sshd_config(5) for more information.
+
+# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
+
+# The strategy used for options in the default sshd_config shipped with
+# OpenSSH is to specify options with their default value where
+# possible, but leave them commented. Uncommented options override the
+# default value.
+
+#Port 22
+#AddressFamily any
 #ListenAddress 0.0.0.0
-Protocol 2
-# HostKeys for protocol version 2
-HostKey /etc/ssh/ssh_host_rsa_key
-HostKey /etc/ssh/ssh_host_dsa_key
-HostKey /etc/ssh/ssh_host_ecdsa_key
-HostKey /etc/ssh/ssh_host_ed25519_key
-#Privilege Separation is turned on for security
-UsePrivilegeSeparation yes
-
-# Lifetime and size of ephemeral version 1 server key
-KeyRegenerationInterval 3600
-ServerKeyBits 1024
+#ListenAddress ::
+
+#HostKey /etc/ssh/ssh_host_rsa_key
+#HostKey /etc/ssh/ssh_host_ecdsa_key
+#HostKey /etc/ssh/ssh_host_ed25519_key
+
+# Ciphers and keying
+#RekeyLimit default none

 # Logging
-SyslogFacility AUTH
-LogLevel INFO
+#SyslogFacility AUTH
+#LogLevel INFO

 # Authentication:
-LoginGraceTime 120
-PermitRootLogin prohibit-password
-StrictModes yes
-
-RSAAuthentication yes
-PubkeyAuthentication yes
-#AuthorizedKeysFile %h/.ssh/authorized_keys

+#LoginGraceTime 2m
+#PermitRootLogin prohibit-password
+#StrictModes yes
+#MaxAuthTries 6
+#MaxSessions 10
+
+#PubkeyAuthentication yes
+
+# Expect .ssh/authorized_keys2 to be disregarded by default in future.
+#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
+
+#AuthorizedPrincipalsFile none
+
+#AuthorizedKeysCommand none
+#AuthorizedKeysCommandUser nobody
+
+# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
+#HostbasedAuthentication no
+# Change to yes if you don't trust ~/.ssh/known_hosts for
+# HostbasedAuthentication
+#IgnoreUserKnownHosts no
 # Don't read the user's ~/.rhosts and ~/.shosts files
-IgnoreRhosts yes
-# For this to work you will also need host keys in /etc/ssh_known_hosts
-RhostsRSAAuthentication no
-# similar for protocol version 2
-HostbasedAuthentication no
-# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
-#IgnoreUserKnownHosts yes
+#IgnoreRhosts yes

-# To enable empty passwords, change to yes (NOT RECOMMENDED)
-PermitEmptyPasswords no
+# To disable tunneled clear text passwords, change to no here!
+PasswordAuthentication no
+#PermitEmptyPasswords no

 # Change to yes to enable challenge-response passwords (beware issues with
 # some PAM modules and threads)
 ChallengeResponseAuthentication no

-# Change to no to disable tunnelled clear text passwords
-PasswordAuthentication no
-
 # Kerberos options
 #KerberosAuthentication no
-#KerberosGetAFSToken no
 #KerberosOrLocalPasswd yes
 #KerberosTicketCleanup yes
+#KerberosGetAFSToken no

 # GSSAPI options
 #GSSAPIAuthentication no
 #GSSAPICleanupCredentials yes
-
-X11Forwarding yes
-X11DisplayOffset 10
-PrintMotd no
-PrintLastLog yes
-TCPKeepAlive yes
-#UseLogin no
-
-#MaxStartups 10:30:60
-#Banner /etc/issue.net
-
-# Allow client to pass locale environment variables
-AcceptEnv LANG LC_*
-
-Subsystem sftp /usr/lib/openssh/sftp-server
+#GSSAPIStrictAcceptorCheck yes
+#GSSAPIKeyExchange no

 # Set this to 'yes' to enable PAM authentication, account processing,
 # and session processing. If this is enabled, PAM authentication will
@@ -87,8 +83,40 @@
 # and ChallengeResponseAuthentication to 'no'.
 UsePAM yes

-ClientAliveInterval 180
+#AllowAgentForwarding yes
+#AllowTcpForwarding yes
+#GatewayPorts no
+X11Forwarding yes
+#X11DisplayOffset 10
+#X11UseLocalhost yes
+#PermitTTY yes
+PrintMotd no
+#PrintLastLog yes
+#TCPKeepAlive yes
+#UseLogin no
+#PermitUserEnvironment no
+#Compression delayed
+#ClientAliveInterval 0
+#ClientAliveCountMax 3
+#UseDNS no
+#PidFile /var/run/sshd.pid
+#MaxStartups 10:30:100
+#PermitTunnel no
+#ChrootDirectory none
+#VersionAddendum none

-Ciphers aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se
+# no default banner path
+#Banner none
+
+# Allow client to pass locale environment variables
+AcceptEnv LANG LC_*
+
+# override default of no subsystems
+Subsystem sftp /usr/lib/openssh/sftp-server

-TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub
+# Example of overriding settings on a per-user basis
+#Match User anoncvs
+# X11Forwarding no
+# AllowTcpForwarding no
+# PermitTTY no
+# ForceCommand cvs server

Hi Marcos

Thanks for your reply.

Mine is a AWS Lightsail Instance.
So I don’t have the options to try the fixes that has been suggested by you.

Is this issue reproducible by you on a AWS Lightsail by choosing default to
the warning that asks about ssh configuration.@jdrios

Looking forward for your continued support
Thanks

Regards

Hi @Ashd, you are right. It seems that Lightsail does not offer the possibility of de-attaching a disk.

However, not all hope is lost! It seems that Lightsail offers the possibility of executing custom instance init scripts. We’ll take advantage of that like described below:

  • Create a snapshot of the instance with broken SSH

  • Once the snapshot is created, navigate to Snapshots > click on the arrow next to “1 instance snapshots” > Create new instance.

  • In the snapshot launch wizard, go to Optional > Add launch script > Enter the script below:

      sudo sed -i 's/^Ciphers .*/Ciphers +aes256-cbc,aes192-cbc,aes128-cbc/' /etc/ssh/sshd_config
      sudo service sshd stop
      sudo service sshd start
    
  • Create the instance, and wait some minutes for it to initialize.

You will be seeing “ssh: connect to host … port 22: Connection refused” for some minutes, but if everything goes right, in the end you will be able to connect to your instance.

Hope it helps!

Note: If you still find trouble you could always export the snapshot to Amazon EC2. It will be copied as an EBS disk meaning you will be able to follow the guide we mentioned earlier.

4 Likes

Hi Marcos

Created the instance using the snapshot with your suggested launch script
command.
It worked like a charm :slight_smile: Many Many Thanks. :slight_smile:

Appreciate if you could shed some light on the fix and reasoning behind it
so I can understand how and why things broke with an upgrade.(so I know it
for any next upgrade;)

Regards,

2 Likes

That’s awesome! Thanks for letting us know. :smiley:

The reason is that the OpenSSH version bundled with Ubuntu 18.04 includes breaking changes, meaning the previous “Ciphers” configuration was not properly set. See this post for more information.

Therefore in order to fix this, we needed to find some way to 1) replace /etc/ssh/sshd_config with a proper Ciphers configuration and 2) restart sshd.

Fortunately Lightsail includes a way to run customization scripts, which is a perfect place for performing such fix to an instance with broken SSH.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.