If you need to update the public key you use for logging in to a server using ssh, you can do the following:

Just in case it’s a good idea to have a first shell open in parallel with a connection to the remote server:

ssh root@YOUR_SERVER_NAME

Replace YOUR_SERVER_NAME by the hostname or IP address of the remote server. If you login with another user than root, replace root by the appropriate user name.

Then open a new shell and go to the subdirectory where your keys are stored:

cd ~/.ssh

First you should first backup the existing keys in case you run into problems:

cp id_dsa id_dsa.OLD
cp id_dsa.pub id_dsa.pub.OLD

Then you can generate new keys. For this, you need to generate them using new files since you still need the old key in order to copy the new key to the server:

ssh-keygen -t dsa -f ~/.ssh/MY_NEW_KEY

Press Enter for each question asked.

This will create two files MY_NEW_KEY and MY_NEW_KEY.pub.

Now you need to copy the new public key to your server:

cat ~/.ssh/MY_NEW_KEY.pub | ssh root@YOUR_SERVER_NAME "cat >> ~/.ssh/authorized_keys" 

Now you have both public keys in the authorized_keys file on the server and you need to delete the old key from the file:

ssh root@YOUR_SERVER_NAME
vi ~/.ssh/authorized_keys

Then go to the line with the old key, press “dd” to delete the line and “ZZ” to save the file.

Then go back to your local shell:

exit

Now you can replace your old public key by the new one and test whether it still works:

mv MY_NEW_KEY id_dsa
mv MY_NEW_KEY.pub id_dsa.pub
ssh root@YOUR_SERVER_NAME

If nothing works anymore and you want to roll it all back, you can do the following:

In the second shell:

cat ~/.ssh/id_dsa.pub.OLD

Copy the output of “cat” and in the first shell:

vi ~/.ssh/authorized_keys

Then delete the new key (with “dd”) and paste the old key back in there (press “i” before pasting to get in insert mode and escape after pasting to exit insert mode, and then “ZZ” to save).

We saw everyday (and often multiple times a day) traces of brute force attacks on our server: many failed attempts to log into our server with SSH using the same or different usernames. Very often the username used were parts of domains hosted on our server. Only once did it succeed (on an old server we were about to get rid of and only had a few private domains) but even when they fail it’s just disturbing to have auth.log filled with thousands of message a day related to such attacks.

Of course in the beginning, we just started adding an iptables rule to drop packets from the IP address which kept coming up in auth.log:

# iptables -A INPUT -s xxx.xxx.xxx.xxx -j DROP

It basically means  that all packets with the specified IP addressed will be dropped.

It is OK if it just happens once in a while but if have to do it for multiple IP addresses a day, it really calls for an automated solution. So we then introduced a script running as a cron job:

#!/bin/bash

IPLIST="/tmp/hack.ips.log"
ALREADYDENIED="/tmp/denied.ips.log"

MAIL="/usr/bin/mail"
EMAILS="henri.benoit@gmail.com xxx.xxx@googlemail.com"
TEMPLOG="/root/deny.ips.$$.log"

/sbin/iptables -n -L | /bin/grep DROP | awk '{ print $4; }' > $ALREADYDENIED

rm /tmp/hack.ip.*.log
grep -I "Invalid user " /var/log/auth.log* | awk '{ print $10; }' | sed '/^$/d' | awk '{ print $0 > "/tmp/hack.ip."$1".log" }'
wc -l /tmp/hack.ip.*.log | /bin/grep -v " total" | /bin/grep -v " insgesamt" | sed 's,/tmp/,,g' | sed 's/hack.ip.//g' | sed 's/.log//g' | awk '{ print $2" : "$1; }' | sort -k3 -n -r | /bin/grep " : [0-9][0-9]" | awk '{ print $1; }' > $IPLIST

while read line
do

if /bin/grep -Fxq "$line" $ALREADYDENIED
then
echo "Access already denied to IP address: $line"
else
echo "Denying access to IP address: $line" | tee -a $TEMPLOG
/sbin/iptables -A INPUT -s $line -j DROP | tee -a $TEMPLOG
fi

done < $IPLIST
echo "-------------------------------------------------------------"

if [ -s $TEMPLOG ] ; then
for EMAIL in $EMAILS
do
$MAIL -s "Denied access to some IP addresses" $EMAIL < $TEMPLOG
done
fi
rm -f $TEMPLOG

It basically goes through all auth.log* log files and looks for failed ssh login attempts and if an IP address is found at least 10 times (i.e. a two digits number of time), it will add an iptables rule. It also checks before that which DROP rules already exist so that we don’t add duplicate rules. In the end it sends me an email with the list of affected IP addresses.

This worked great. The only problem is that it adds quite a lot of rules to our firewall configuration. It has 2 main disadvantages:

• At some point in time, the firewall will start needed much more time to start because of the thousands of rules.
• When I list all rules, I get a huge list and need much more time to actually find what I need (and if I forget the -n argument and it start resolving IP addresses, it can last forever…).

So we decided to implement a few iptables rules to get rid of these guys temporarily and hope they’ll give up and it will save us a a permanent iptables rule. In order to do it , you need to be able to temporarily store who opened a connection to port 22 so that we can figure out whether it’s opening to many connections or not. This can be done using the iptables module “recent”. So for the following rules, you’ll need this module installed:

iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 22 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 5 -j DROP

The first rule writes with the –set option the IP address of a new connection on port 22 and a time stamp in the proc file /proc/net/ipt_recent/DEFAULT.
The second rule adds with –update  another timestamp to the file and checks whether there were 5 timestamps within the last 60 seconds. If it is the case, the connection is dropped. These rules only affect SSH access, so other services might still be vulnerable.

Note that on the iptables level, we do not know whether the login with ssh failed or not so you might also sometimes temporarily lock out a user who just happened to need 5 connections and opened them all within 1 minute. But then he just needs to wait for a minute and things are fine again.

Using the combination of both the script above and the iptables rules above proved to work quite fine.

I’ve got a script running on a regular basis and executing rkhunter. After I finally got to installing all security updates on our new server, I started getting emails saying:

First, it’s always recommended to not use only one tool checking for root kits but multiples. Here is how to configure cron jobs to check the system everyday and email you the findings (only if something was found).

First rkhunter:

00 23 * * * (/usr/bin/rkhunter –versioncheck; /usr/bin/rkhunter –update; /usr/bin/rkhunter –cronjob –report-warnings-only)
This will first check for latest version of rkhunter (–versioncheck), check for updates to database files (–update) and then run rkhunter.

You can enter email address which should receive a message if something was found in /etc/rkhunter.conf:

MAIL-ON-WARNING=henri.benoit@gmail.com xxx.xxx@googlemail.com

00 23 * * * /usr/sbin/chkrootkit -q | grep -v “bindshell.*465)” > /tmp/chkrootkit.log && [ -s /tmp/chkrootkit.log ] && mail -s ‘chkrootkit findings’ henri.benoit@gmail.com xxx.xxx@googlemail.com < /tmp/chkrootkit.log

Just installed chkrootkit on our (relatively) new Debian Squeeze server and run it. Wasn’t expecting anything sind rkhunter reported everything was fine. But it did indeed report something:

# chkrootkit
…
Checking `bindshell’… INFECTED (PORTS: 465)
…


So checking what’s running on port 465:

# netstat -pan | grep “:465 ”
tcp 0 0 0.0.0.0:465 0.0.0.0:* LISTEN 8325/master
tcp6 0 0 :::465 :::* LISTEN 8325/master


Since I wasn’t too sure what this master was:

# ps -F -p 8325
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
root 8325 1 0 9292 2404 6 Apr14 ? 00:00:17 /usr/lib/postfix/master


There it is, it’s nothing to worry about, just postfix…

So if a mailserver is running tls on port 465, chkrootkit wrongfully identifies bindshell.

If you stop postfix and rerun chkrootkit, nothing is reported:

# service postfix stop
Stopping Postfix Mail Transport Agent: postfix.
# chkrootkit | grep INFECTED
# service postfix start
Starting Postfix Mail Transport Agent: postfix.