Checking a server's public host key on the server
If you receive a message from a
Secure
Shell (SSH) or
Secure File
Transfer Protocol (SFTP) application regarding the host key of the server
to which you are attempting to connect being unknown or changed, such as
the message from WinSCP below, you can check the server's public host key on the
server itself, if it is a
Linux server, using the
ssh-keygen utility.
$ ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub
2048 96:f3:8b:03:13:06:13:4d:3c:7c:4b:fa:94:33:90:83 (RSA)
$
The -l option shows the fingerprint of a specified public key file.
Private RSA1 keys are also supported. For
RSA
and DSA keys, ssh-keygen tries to find the matching public key
file and prints its fingerprint. If the -l option is combined
with -v, an ASCII art representation of the key is supplied with
the fingerprint. The -f filename option allows you to
specify the file name of the key file.
The ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key.pub command isn't
showing the key itself, but instead shows the
"fingerprint" for
the key, which is a sequence of 32
hexadecimal
digits. You can see the much larger key value itself by issuing the
command cat /etc/ssh/ssh_host_rsa_key.pub.
[/network/ssh]
permanent link
Running a command on a remote system using SSH
The
Secure Shell (SSH) protocol allows you to interactively log into remote
systems. Once logged into a remote system, you have a shell prompt where you
can enter commands on the remote system. But you can use an SSH client to
execute a command on a remote system without logging into that system and
obtaining a shell prompt on the remote system. E.g., if you wanted to
get a
command line interface (CLI) on the remote system, you
might enter a command similar to the following one:
But, if you just were logging in to enter one command, say you wanted
to find the hardware platform of the remote system using the
uname
command uname --hardware-platform, you could simply append that
command to the end of the above ssh command you would have used to log into
the remote system. E.g.:
$ ssh jdoe@example.com uname --hardware-platform
jdoe@example.com's password:
x86_64
$ uname --hardware-platform
i386
In the example above, issuing the same command on the local system, i.e.,
the one on which the SSH command is being issued shows that the result returned
when the uname command was issued at the end of the ssh command line returned
a result from the remote system.
You may even be able to use a text-based editor, such as the
vi editor, though you may see
error messages like the ones below:
$ ssh jdoe@example.com vi temp.txt
jdoe@example.com's password:
Vim: Warning: Output is not to a terminal
Vim: Warning: Input is not from a terminal
When you enter an ssh command in the form ssh
user@host the remote system allocates a
pseudo-tty (PTY), a
software abstraction used to handle keyboard input and screen
output. However, if you request SSH to run a command on the remote
server by appending that command after ssh user@host, then
no interactive terminal session is required and a PTY is not allocated, so
you see the error messages when you use a screen-based program intended for
use with a terminal, such as the vi editor.
For such cases you should inclde the -t option to the SSH
command.
| -t |
Force pseudo-tty allocation. This can be used to execute arbitrary
screen-based programs on a remote machine, which can be very useful,
e.g. when implementing menu services. Multiple -t options force tty
allocation, even if ssh has no local tty. |
E.g.:
$ ssh jdoe@example.com -t vi temp.txt
[/network/ssh]
permanent link
Break out of SSH session
Sometimes after I've established an SSH connection to an SSH server, I
encounter a situation where the remote system isn't responding to keyboard
input and I want to terminate the SSH session and return to a command prompt.
E.g., often when I've connected to a Microsoft Windows system running SSH
software from my Ubuntu Linux laptop, I find that I'm in a situation where
after I've entered a command at the Windows system's command prompt the
remote system no longer seems to be accepting keyboard input from the
Linux system. Sometimes it seems to occur when I've mistyped a Windows
command and the Windows system may be waiting for further input, but doesn't
seem to accept what I type. In such cases, rather than close the Terminal
tab on the Linux system to terminate the connection, which then requires me
to open a new tab and establish a new SSH session, I'd prefer to break out
of the current SSH session and return to the shell prompt on the Linux
system where I can re-establish the SSH connection. In such cases,
Ctrl-C,
Ctrl-D, and
Ctrl-Z don't help me.
But there is an
escape sequence
that will allow me to terminate the current SSH session. Hitting the
three keys listed below will allow me to terminate the session.
↲ Enter, ~, .
[ More Info ]
[/network/ssh]
permanent link
Keeping an SSH connection alive
If you need to keep a
SSH connection alive, e.g., when you won't be
entering any commands for awhile after logging into the remote system via
SSH, you can use the
-o option with
ServerAliveInterval
. You can specify the interval in seconds which will be used by the SSH
client to send
keepalive packets with
-o ServerAliveInterval x
where
x is the frequency for sending the keepalive packets. E.g.,
if I wanted the SSH client to send keepalive packets every minute (60 seconds)
to the remote SSH server, I could use a command like the one below when
establishing the SSH session:
$ ssh -o ServerAliveInterval=60 jdoe@example.com
By using this option, you should be able to reduce the likelihood that
your SSH connection will get dropped after a certain amount of time due to
no activity for the session.
You can also use the
ServerAliveCountMax
parameter with ServerAliveInterval to drop the connection, if the SSH
client hasn't received a response from the server to the prior "heartbeat"
signal when the time comes to send another keepalive packet. E.g., ssh
-o ServerAliveInterval=60 -o ServerAliveCountMax=1 jdoe@example.com
would result in the connection being dropped if the client was awaiting
a response to even one outstanding keepalive packet.
There is also a TCPKeepAlive option in
OpenSSH.
That option is used to recognize when a connection is no longer active due
to some problem such as the SSH client application crashing or a prolonged
network outage. If the SSH server never recognizes that the client is no
longer communicating with it, it will continue to allocate resources,
such as memory, for the connection. The option is turned on by default
in the OpenSSH configuration file /etc/ssh/sshd_config. You
will see the following line in that file:
#TCPKeepAlive yes
You don't need to uncomment the line by removing the pound sign, since
"yes" is the default value. The option causes
Transmission Control Protocol (TCP) to periodically
transmit keepalive messages. If it doesn't receive responses within the
expected time, it returns an error to the sshd process, which will then
shut down the connection. The purpose of this option is to prevent half-dead
connections building up over time and consuming more and more system
resources as the number grows. The keepalive interval is typically in the
order of hours rather than minutes to minimize the network load for the
server. If the keepalive period was made shorter, that would affect all TCP
connections on the system, not just the SSH ones, potentially increasing the
network load unnecessarily and also causing connections to be dropped even
for transient issues, such as a short and temporary network issue.
The TCPKeepalive option is for dealing with longer term issues for a
connection rather than the loss of connectivity due to firewall, proxying, or
Network Address Translation (NAT) timeouts.
You can specify the option on the command line at the SSH client end as
follows:
$ ssh -o TCPKeepAlive=yes joe@example.com
References:
-
SSH, The Secure Shell: The Definitive
Guide
By: Daniel Barrett, Richard Silverman, Robert Byrnes
[/network/ssh]
permanent link
OpenSSH Roaming Vulnerability
Ars Technica published an article today
titled
Bug that can leak crypto keys just fixed in widely used OpenSSH which
explained how a compromise of a
SSH server running
OpenSSH software could
lead to the server being used to capture data from the memory of systems
that have connected to the server via SSH including private keys for users
connecting to the server.
The vulnerability resides only in the version end users use to connect to
servers and not in versions used by servers. A maliciously configured server
could exploit it to obtain the contents of the connecting computer's memory,
including the private encryption key used for SSH connections. The bug is the
result of code that enables an experimental roaming feature in OpenSSH
versions 5.4 to 7.1
"The matching server code has never been shipped, but the client code was
enabled by default and could be tricked by a malicious server into leaking
client memory to the server, including private client user keys," OpenSSH
officials wrote in an advisory published Thursday. "The authentication of the
server host key prevents exploitation by a man-in-the-middle, so this
information leak is restricted to connections to malicious or compromised
servers."
The roaming feature was intended to allow users to resume broken SSH
connections, even though the feature was disabled in OpenSSH server software
years ago. E.g., when I connected to a server I have running OpenSSH server
software, I saw the folowing:
$ ssh -v jdoe@127.0.0.1 2>&1 >/dev/null | grep -i 'roaming'
debug1: Roaming not allowed by server
The Red Hat article on the vulnerability
OpenSSH: Information-leak
vulnerability (CVE-2016-0777) notes:
Since version 5.4, the OpenSSH client supports an undocumented feature called
roaming. If a connection to an SSH server breaks unexpectedly, and if the SSH
server supports roaming as well, the client is able to reconnect to the server
and resume the interrupted SSH session. The roaming feature is enabled by
default in OpenSSH clients, even though no OpenSSH server version implements
the roaming feature.
For affected products, the article also notes:
Red Hat Enterprise Linux 7 since version 7.1 has provided OpenSSH 6.6 for
which the default configuration is not affected by this flaw. OpenSSH 6.6 is
only vulnerable to this issue when used with certain non-default ProxyCommand
settings. Security update
RHSA-2016-0043
corrects this issue.
So CentOS 7 systems using a
default OpenSSH configuration should be unaffected, since CentOS is derived from
Red Hat
Enterprise Linux.
On a Linux system, you can check the version of SSH installed with
ssh -V.
$ ssh -V
OpenSSH_6.6.1p1, OpenSSL 1.0.1e-fips 11 Feb 2013
On a CentOS Linux system using the
RPM Package Manager
you can also use rpm -qi openssh | grep Version.
$ rpm -qi openssh | grep Version
Version : 6.6.1p1
On a CentOS system, you can update the software from the command line, aka a
shell prompt, using the command yum update openssh.
If you are using a vulnerable OpenSSH client, you can also specify the
-oUseRoaming=no parameter on the command line to ensure that
a malicious server can't take advantage of the vulnerability. E.g.
ssh -oUseRoaming=no jdoe@example.com. Or the feature can
be disabled for all users on a system by putting UseRoaming no in
/etc/ssh/ssh_config or by an individual user for his account
by adding the line to ~/.ssh/config.
echo 'UseRoaming no' >> /etc/ssh/ssh_config
References:
-
Bug that can leak crypto keys just fixed in widely used OpenSSH
Date: January 14, 2016
Ars Technica
-
Fixing The New OpenSSH Roaming Bug
By: ScriptRock
ScriptRock Blog
-
OpenSSH: Information-leak vulnerability (CVE-2016-0777)
Updated: January 14, 2016
Red Hat Customer Portal
-
Security OpenSSH Security Bug CVE-2016-0777 & CVE-2016-0778
Date: January 4, 2016
[/network/ssh]
permanent link
Transferring files with scp through a bastion host on an OS X system
I need to transfer files through a
bastion host
periodically. To edit files on a webserver, I need to first establish a
Secure Shell (SSH)
connection to the bastion host, logging in using an
RSA SecurID token.
Once I've provided my login credentials, the bastion host prompts me for
the system to which I want to connect to from it, which in this case
is the web server. So my ssh login to the webserver is
tunneled through the bastion host.
I normally go through the process once a month from my MacBook Pro laptop
running the OS X operating system when I need to place a monthly newsletter on
the web server. I use an SSH command similar to the following to log into
the bastion host where gold.example.com is the
fully qualified domain name (FQDN) of the web server and
bastion1.example.com is the bastion host.
ssh -L 22001:gold.example.com:22 jasmith1@bastion1.example.com
The -L option specifies I want to tunnel a local port on
my laptop, in this case I chose 22001, to port 22 on the web server,
gold.example.com. A tunnel is set up from my laptop to the web server
through the bastion host by using that option once my login is completed
to the bastion host.
Then, to transfer a file via
secure copy from my laptop to the web server, I can use a command like
the following one to transfer a file named July.txt from the laptop
to the web server:
$ scp -P 22001 July.txt jasmith1@127.0.0.1:/data/htdocs/clubs/groot/newsletter/2015/.
jasmith1@127.0.0.1's password:
The -P option to the scp command specifies I want to use
TCP
port 22001, since that is the port for the end of the tunnel on my laptop.
The 127.0.0.1 address I'm specifying is the
localhost, aka
"loopback", address on my laptop. I.e., I'm connecting to port 22001 on
the laptop itself. The tunnel I set up earlier results in any connection
to that port being forwared through the tunnel to the web server, so
I'm specifying my userid for the web server and the password prompt I
receive is for the web server. The file July.txt will thus be placed
in the directory /data/htdocs/clubs/groot/newsletter/2015
on the web server with the same name, July.txt.
If I wanted to pull a file from the webserver via the tunnel, I could
use a command such as the following:
scp -P 22001 jasmith1@127.0.0.1:/data/htdocs/clubs/groot/July.html .
That command would retrieve the file July.html from the web server and
place it on the laptop with the same name.
[/network/ssh]
permanent link
Monitoring Failed SSH Logins to a CentOS System
If you have ssh enabled on a system that is accessible to the Internet, it is
probable that malicious individuals will try to gain access to the system
by brute force login attempts. I.e., since a Linux, Unix, or OS X system is
likely to have a root account, an attacker may use "root" as the userid and
then attempt to login with commonly used passwords. There are sites on the
Internet that provide lists of passwords commonly used and an attacker can
easily use a
dictionary attack where he tries every word in a dictionary as a possible
password. Attackers can use dictionaries for multiple languages, lists of
sports teams, name dictionaries, e.g., dictionaries of names parents might
check to aid in selecting a name for a baby, etc. So a root or another
administrator account should have a
strong password.
If it doesn't, the system will likely be cracked by an attacker eventually.
Attackers also routinely use name dictionaries to break into systems with
any accounts that have weak passwords. E.g., an attacker may use a name
dictionary to pick names to use as the userid. Let's say the first name in
the name dictionary is Aaron. The attacker might then use a word dictionary
to try every word in the English language, or some other language, as a
possible password for an
account with the userid of aaron. If an aaron account doesn't exist on the
system or has a strong password, once the attacker has gone through every
word in the word dictionary or whatever other password list he is using, he
will then go onto the next name in his name dictionary, e.g., perhaps Abe.
The attacker will proceed in this manner until he finds an account with a
weak password he can compromise or exhausts all possible combinations of
names for accounts and words to use for possible passwords. Of course it would
take a human an inordinate amount of time to type all such possible userid
and password combinations, but an attacker will let a program make such guesses
for him. He merely needs to start the program and let it run. His program
may be able to check many thousands of userid and password combinations
in minutes.
If the system isn't monitored for such brute-force password attempts, an
attacker can run unchecked for days. Even if he can't get in, he will be
using bandwidth to/from the system under attack as well as CPU cycles, etc.,
so may slow down access to the system for legitimate users. I've seen periods
where a system has been under attack from 5 such attackers in different
countries at once.
On CentOS Linux,
you can check the /var/log/secure log to find instances of
such attacks.
# grep 'Failed password' /var/log/secure | tail -5
Oct 28 09:47:43 frostdragon sshd[32246]: Failed password for root from 123.125.219.130 port 11859 ssh2
Oct 28 09:47:47 frostdragon sshd[32249]: Failed password for root from 123.125.219.130 port 13894 ssh2
Oct 28 09:47:52 frostdragon sshd[32253]: Failed password for root from 123.125.219.130 port 15886 ssh2
Oct 28 09:47:56 frostdragon sshd[32256]: Failed password for root from 123.125.219.130 port 17740 ssh2
Oct 28 09:48:01 frostdragon sshd[32259]: Failed password for root from 123.125.219.130 port 19477 ssh2
You can see the number of failed ssh login attempts from various login
addresses with the command grep 'Failed password' /var/log/secure | grep
sshd | awk '{print $11}' | sort | uniq -c - the IP address from which
the failed login attempt was made is the 11th item on the line.
If you pipe the output of the awk command into sort, you can sort the
output by IP address; uniq -c will then provide you the count
of failed SSH login attempts from particular IP addresses.
# grep 'Failed password' /var/log/secure | grep sshd | awk '{print $11}' | sort | uniq -c
1 101.227.71.40
409 117.27.158.71
2 117.27.158.91
84 122.225.109.104
315 122.225.109.108
232 122.225.109.118
321 122.225.109.197
247 122.225.109.212
115 122.225.109.217
458 122.225.97.103
309 122.225.97.108
96 122.225.97.110
377 122.225.97.117
478 122.225.97.120
121 122.225.97.83
63 122.225.97.84
81 122.225.97.88
36 122.225.97.98
382 123.125.219.130I can see from the above output from that command that there were
382 failed ssh login attempts from the 123.125.219.130 address
at the time I ran the command.
From a search on that IP address at the
American Registry for Internet Numbers (ARIN), I found the address was
part of a block of addresses managed by the
Asia Pacific Network Information Centre (APNIC)
. A whois search on the APNIC site showed the IP address is part of
a large block of addresses, 123.112.0.0 - 123.127.255.255, allocated
to an organization in Beijing, China. I often see attacks from IP addresses
allocated to entities in China.
You can manually block further attempts to compromise a system in this
manner using a route
reject command or through the firewall software
on the system. The default firewall software for CentOS 7 is
FirewallD.
You can configure it through a
Graphical User
Interface (GUI), which can be opened using the command
firewall-config or through a command line interface at a shell
prompt by using the command firewall-cmd. I blocked the IP
address from any access to the system using the command below, though by
the time I blocked it, the login attempts had ceased:
# firewall-cmd --add-rich-rule="rule family='ipv4' source address='123.125.219.130' reject"
success
The block can be viewed through the graphical interface for FirewallD
by running firewall-config. E.g., in this case under "Rich
Rules" for the public zone, I can see the blocked IP when starting the
application after issuing the firewall-cmd command.
The command above will put in place a firewall rule that will apply
to the default firewall zone, but will only remain
until the firewall service is restarted, e.g., with a system reboot.
To put in place a permanent block, I could have used the commands below.
Instituting a permanent change requires a restart of the firewall service,
though.
firewall-cmd --permanent --add-rich-rule="rule family='ipv4' source address='123.125.219.130' reject"
systemctl restart firewalld.service
To have a block apply to a specific firewall zone, e.g., the public zone,
I could use the commands below.
firewall-cmd --permanent --zone='public' --add-rich-rule="rule family='ipv4' source address='123.125.219.130' reject"
systemctl restart firewalld.service
The output of the grep command run against /var/log/secure
displayed above was sorted by IP address; if you, instead, would like
to sort the output by count of failed login attempts you can pipe the
output of the commands above into sort again adding the
-n argument to sort by the number that appears first on
each line.
# grep 'Failed password' /var/log/secure | grep sshd | awk '{print $11}' | sort | uniq -c | sort -n
1 176.222.201.154
1 85.132.71.83
1 91.220.131.33
1 a
1 pi
1 ubnt
2 client
4 ubuntu
4 usuario
27 git
48 122.225.97.117
64 221.228.205.196
71 61.174.51.223
78 admin
129 122.225.97.79
191 122.225.109.198
237 122.225.97.116
268 117.27.158.88
306 113.200.188.55
336 117.27.158.89I can see from the above output that the greatest number of failed
SSH login attempts made on the day I ran the command, which was November 9,
2014, were made from 117.27.158.89. Checking the APNIC site again,
I see that IP address is also assigned to an entity in China.
If you want to reverse the sorting order, so that the largest number
appears first, simply add the -r argument to the last sort command.
# grep 'Failed password' /var/log/secure | grep sshd | awk '{print $11}' | sort | uniq -c | sort -nr
336 117.27.158.89
306 113.200.188.55
268 117.27.158.88
237 122.225.97.116
191 122.225.109.198
129 122.225.97.79
78 admin
71 61.174.51.223
64 221.228.205.196
48 122.225.97.117
27 git
4 usuario
4 ubuntu
2 client
1 ubnt
1 pi
1 a
1 91.220.131.33
1 85.132.71.83
1 176.222.201.154In the above output, some of the failed entries are associated with
userids the attacker attempted to use to login. E.g., for the case of
the usuario one, I can see that the illegitimate login attempts
where that name was used for the userid orginated from the 221.228.205.196
IP address. There is no account on the system with that userid. The IP address
is also assigned to an entity in China.
# grep usuario /var/log/secure
Nov 9 10:53:01 localhost sshd[23516]: Invalid user usuario from 221.228.205.196
Nov 9 10:53:01 localhost sshd[23516]: input_userauth_request: invalid user usuario [preauth]
Nov 9 10:53:03 localhost sshd[23516]: Failed password for invalid user usuario from 221.228.205.196 port 52710 ssh2
Nov 9 10:53:04 localhost sshd[23568]: Invalid user usuario from 221.228.205.196
Nov 9 10:53:04 localhost sshd[23568]: input_userauth_request: invalid user usuario [preauth]
Nov 9 10:53:06 localhost sshd[23568]: Failed password for invalid user usuario from 221.228.205.196 port 53534 ssh2
Nov 9 10:53:07 localhost sshd[23654]: Invalid user usuario from 221.228.205.196
Nov 9 10:53:07 localhost sshd[23654]: input_userauth_request: invalid user usuario [preauth]
Nov 9 10:53:10 localhost sshd[23654]: Failed password for invalid user usuario from 221.228.205.196 port 55193 ssh2
Nov 9 10:53:12 localhost sshd[23657]: Invalid user usuario from 221.228.205.196
Nov 9 10:53:12 localhost sshd[23657]: input_userauth_request: invalid user usuario [preauth]
Nov 9 10:53:14 localhost sshd[23657]: Failed password for invalid user usuario from 221.228.205.196 port 56072 ssh2
To count just by IP address so that the login failurers for particular
usernames don't appear in the output, I can put another grep
command, one that will filter the output of the prior grep command so any lines of output from it are eliminated if they contain "invalid user", before the
awk command.
# grep 'Failed password' /var/log/secure | grep sshd | grep -v "invalid user" | awk '{print $11}' | sort | uniq -c | sort -n
1 176.222.201.154
1 85.132.71.83
1 91.220.131.33
48 122.225.97.117
64 221.228.205.196
71 61.174.51.223
129 122.225.97.79
191 122.225.109.198
237 122.225.97.116
268 117.27.158.88
306 113.200.188.55
336 117.27.158.89If you wish to see what userids are being used most frequently for the
failed login attempts, the string of commands entered above need to be
modified to search for the userids used in failed login attempts. The above
commands don't show the most commonly used userid, which is
root, since almost all Unix/Linux systems will have a root
account.
For failed login attempts the lines that appear in the output are slightly
different depending upon whether the userid used exists on the system. E.g.,
if the account doesn't exist on the system, as in the case for client
and git below, the lines appear as follows:
Nov 9 10:34:14 localhost sshd[21745]: Failed password for invalid user client f
rom 91.220.131.33 port 60223 ssh2
Nov 9 10:52:00 localhost sshd[23204]: Failed password for invalid user git from
221.228.205.196 port 60513 ssh2
If the account does exist, e.g., the root account, then
the lines have the following format:
Nov 9 04:58:50 localhost sshd[21319]: Failed password for root from 122.225.97.
79 port 7951 ssh2
The sed command can be used to strip out the "invalid user"
from lines to make the format of those lines containing "invalid user" the
same as for those for valid userids on the system. You can then use
the awk command to display the contents of the 9th entry on
the line, which is the userid used.
# grep "sshd.*: Failed password for" /var/log/secure | sed 's/invalid user //' | awk '{print $9}' | sort | uniq -c | sort -n
1 a
1 operator
1 pi
1 ubnt
2 client
4 ubuntu
4 usuario
27 git
78 admin
1844 rootThe output from a check of the /var/log/secure file shows
that the most common user name used in attempts to log into the system
by attackers is root.
References:
-
Firewalld - Block an IP Address
By: up2long
Date: February 26, 2014
Fedoraforum.org
[/network/ssh]
permanent link
