We upgraded our server to Debian Wheezy and Plesk 11.5 about a week ago and started getting many emails from cron. Luckily I had entered the email address of a colleague as administrator and he got the spam 😉

One of them was this one:

———- Forwarded message ———-
From:
Date: 2013/9/16
Subject: Daily AIDE report for xx-xxxx.myserver.de
To: root@xx-xxxx.myserver.de

This is an automated report generated by the Advanced Intrusion Detection
Environment on xx-xxxx.myserver.de started at 2013-09-16 07:45:17.

******************************************************************************
* The cron job was terminated by a fatal error. *
******************************************************************************

******************************************************************************
* the cron job was interrupted before AIDE could return an exit code. *
******************************************************************************
******************************************************************************
* script errors *
******************************************************************************
Fatal error: The AIDE database ‘/var/lib/aide/aide.db’ does not exist!
This may mean you haven’t created it, or it may mean that someone has removed it.
End of script errors

AIDE produced no errors.

funny, AIDE did not leave a log.

End of AIDE daily cron job at 2013-09-16 07:45, run time 0 seconds

I didn’t know we had AIDE installed but since it was there, it’d make sense to initialize it properly and see whether it works fine. For those of you who do not know AIDE: it is an intrusion detection software which works by checking file and directory integrity. In order to work, AIDE needs to first have a database it can use to then detect changes.

So the first step is to initialize the database. I found a nice article at HowToForge. It looked pretty easy to I started with the first step which was to download a sample configuration file:

wget securehostingdirectory.com/aide.conf

And I got a 404 error:

–2013-09-16 20:32:46– http://securehostingdirectory.com/aide.conf
Resolving securehostingdirectory.com (securehostingdirectory.com)… 69.65.27.131
Connecting to securehostingdirectory.com (securehostingdirectory.com)|69.65.27.131|:80… connected.
HTTP request sent, awaiting response… 404 Not Found
2013-09-16 20:32:47 ERROR 404: Not Found.

Well… it starts fine… So I checked in Google whether I could find any alternative location but couldn’t find any. Then I thought that there must be some kind of configuration already available since it was looking for the database at a specific path. So I checked the cron job sending us this nice email:

# grep "aide.conf" /etc/cron.daily/aide
CONFFILE="/var/lib/aide/aide.conf.autogenerated"
# grep aide configuration data from aide config
update-aide.conf

Then opened /var/lib/aide/aide.conf.autogenerated to check the content and saw the following:

#########
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# this file is generated dynamically from /etc/aide/aide.conf and the files
# in /etc/aide/aide.conf.d
# Any changes you make here will be lost.
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# WARNING WARNING WARNING
# WARNING WARNING WARNING
#########

aide.conf as well as /etc/aide/aide.conf.d were there and it looks like they were used to generate this file. So I just needed to create the databse. So I just skipped the wget part of the tutorial and went to the next step (Step 2: Initialize the AIDE database):

# nice -19 aide --init --config=/etc/aide/aide.conf

AIDE, version 0.15.1

### AIDE database at /var/lib/aide/aide.db.new initialized.

It was pretty fast (you wish AIDE was actually that fast to initialize…). Then I checked whether AIDE was working properly:

 nice -19 aide -C --config=/etc/aide/aide.conf
Couldn't open file /var/lib/aide/aide.db for reading

Ok, this one is obvious, it has created a aide.db.new file and is actually looking for aide.db so I just had to rename the file (well, I thought so because the next step of the tutorial was “cp aide.db.out aide.db.in”):

# mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
# nice -19 aide -C --config=/etc/aide/aide.conf
Database does not have attr field.
Comparation may be incorrect
Generating attr-field from dbspec
It might be a good Idea to regenerate databases. Sorry.
db_char2line():Error while reading database

OK, so the initialization was actually too fast and it didn’t generate a proper database… So after googling again, I found out you can initialize AIDE with the following:

aideinit

It then runs forever using a CPU core at about 80% to 100%. It might display a few warnings like:

/run/lock/mailman/master-qrunner.xx-xxxx.myserver.de.8246 mtime in future
/run/lock/mailman/master-qrunner mtime in future

But I wouldn’t worry about them.

Looking at the running processes, I can see that it actually caused a aide — init to be called but with the auto generated configuration file. Actually I should have thought that it is the one I should use, otherwise all rules are missing…

The initialization of AIDE took over an hour. I then made the second mistake: I immediately run a check, got the same error, assumed that the created database was again not working and restarted an initialization. Only later did I think that the problem was probably that it created a aide.db.new database and I needed to copy it to aide.db otherwise the check would still run with the old database.

After the second initialization and copying the file, I didn’t get the same error again when running the check but a new one:

/usr/bin/aide --config /var/lib/aide/aide.conf.autogenerated -C
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
File database must have one db_spec specification
...

The message is not 100% clear (does it mean it expected one and found none or that it expected one and found two ?). I thought it might have something to do with the second initialization. Maybe it did not reset the file but just appended to it… So I deleted both db files and tried again:

rm /var/lib/aide/aide.db.new /var/lib/aide/aide.db
aideinit
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db
/usr/bin/aide --config /var/lib/aide/aide.conf.autogenerated --checked

It then looked better and I actually got a list of differences. I now need to check the list and also check what I need to add to the exclusion list and other configuration option to try to keep everything secure but avoid unnecessary spamming.