Debian / Ubuntu: viewing the apt change log

I am running rkhunter as a cron job and thus receiving notification any time I’ve installed something. Ideally I should be checking the output immediately and updating its database if it is just reporting a change due to a software update I’ve performed. But very often I don’t have time to do it right away and end up forgetting for days. At some point in time, I do check the logs and see that some pieces of software have been changed. By that time I usually do not remember what packages I’ve updated since last rkhunter database update. So I needed a way to check the history of installation of packages.

The bad news is that there is no apt-history command which would easily display the history of installed, updated and removed packages. The good news is that there are some log files which contain the whole history and can be used as a basis to extract this information.

There are actually two location where you can find such log files:

  • /var/log/apt/history.log* –> This contains the apt history logs
  • /var/log/dpkg.log* –> This contains the dpkg logs

Note that older log files are compressed with gzip. For the apt history logs, only the latest file is uncompressed. For the dpkg logs, the two latest files are uncompressed. Luckily, you can use the -f option of zcat to output the contents of those log files, no matter whether they are compressed or not. And combining it with ls -tr you can basically decompress them and concatenate them on the fly in the proper order:

zcat -f $(ls -tr /var/log/apt/history.log*)
zcat -f $(ls -tr /var/log/dpkg.log*)

Before we start, I wanted to mention that I do know that aptitude also writes a log file (/var/log/aptitude) but when I looked at it, it only contained information about what happened in the past 8 days. So it wasn’t really useful for my purpose. There is probably a way to configure aptitude to store more history in there but since I found another good solution, let’s stick to this… Also if you use Synaptic, there is also a “View History” function in the GUI. But it only shows changes made with Synaptic, not from the command line or automatically e.g. by Plesk auto-installer.

First I had a look at the contents of the apt history log files. It looks like this:

Start-Date: 2014-02-17  21:53:00
Install: libserf1:amd64 (1.3.4-1+WANdisco, automatic), php5-imagick:amd64 (3.1.0~rc1-1+b2, automatic), php5-pgsql:amd64 (5.4.4-14+deb7u7, automatic), php5-mcrypt:amd64 (5.4.4-14+deb7u7, automatic), php5-intl:amd64 (5.4.4-14+deb7u7, automatic)
Upgrade: owncloud:amd64 (5.0.13-1, 6.0.1-1), subversion:amd64 (1.7.13-1+WANdisco, 1.7.14-1+WANdisco), libsvn1:amd64 (1.7.13-1+WANdisco, 1.7.14-1+WANdisco), libapache2-svn:amd64 (1.7.13-1+WANdisco, 1.7.14-1+WANdisco)
End-Date: 2014-02-17  21:53:20

Processing it with Unix tools is possible but it is a pain because you have to consider groups of lines, between a Start-Date and an End-Date, and the number of lines inbetween is not always the same (potentially one line for installs, one line for upgrades and one line for removals). This can of course be procesed with some grep, awk, sed and other utilities. But this has two problems: first I’m lazy, second if the command is too complex I will never remember it even if I use it daily.

So I decided to have a look at the dpkg log files and see whether they were friendlier:

2014-02-17 21:53:00 startup archives unpack
2014-02-17 21:53:00 install libserf1:amd64  1.3.4-1+WANdisco
2014-02-17 21:53:00 status half-installed libserf1:amd64 1.3.4-1+WANdisco
2014-02-17 21:53:00 status unpacked libserf1:amd64 1.3.4-1+WANdisco
2014-02-17 21:53:00 status unpacked libserf1:amd64 1.3.4-1+WANdisco
2014-02-17 21:53:01 install php5-intl:amd64  5.4.4-14+deb7u7
2014-02-17 21:53:01 status triggers-pending libapache2-mod-php5:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:01 status not-installed php5-intl:amd64 
2014-02-17 21:53:01 status half-installed php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:01 status unpacked php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:01 status unpacked php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:02 install php5-mcrypt:amd64  5.4.4-14+deb7u7
2014-02-17 21:53:02 status not-installed php5-mcrypt:amd64 
2014-02-17 21:53:02 status half-installed php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:02 status unpacked php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:02 status unpacked php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:02 install php5-pgsql:amd64  5.4.4-14+deb7u7
2014-02-17 21:53:02 status not-installed php5-pgsql:amd64 
2014-02-17 21:53:02 status half-installed php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:03 status unpacked php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:03 status unpacked php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:03 upgrade subversion:amd64 1.7.13-1+WANdisco 1.7.14-1+WANdisco
2014-02-17 21:53:03 status half-configured subversion:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:03 status unpacked subversion:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:03 status half-installed subversion:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:03 status triggers-pending man-db:amd64 2.6.2-1
2014-02-17 21:53:04 status half-installed subversion:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:04 status half-installed subversion:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:04 status unpacked subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:04 status unpacked subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:04 upgrade libsvn1:amd64 1.7.13-1+WANdisco 1.7.14-1+WANdisco
2014-02-17 21:53:04 status half-configured libsvn1:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:04 status unpacked libsvn1:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:04 status half-installed libsvn1:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:05 status half-installed libsvn1:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:05 status unpacked libsvn1:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:05 status unpacked libsvn1:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:05 upgrade libapache2-svn:amd64 1.7.13-1+WANdisco 1.7.14-1+WANdisco
2014-02-17 21:53:05 status half-configured libapache2-svn:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:05 status unpacked libapache2-svn:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:05 status half-installed libapache2-svn:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:05 status half-installed libapache2-svn:amd64 1.7.13-1+WANdisco
2014-02-17 21:53:05 status unpacked libapache2-svn:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:05 status unpacked libapache2-svn:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:06 upgrade owncloud:all 5.0.13-1 6.0.1-1
2014-02-17 21:53:06 status half-configured owncloud:all 5.0.13-1
2014-02-17 21:53:06 status unpacked owncloud:all 5.0.13-1
2014-02-17 21:53:06 status half-installed owncloud:all 5.0.13-1
2014-02-17 21:53:11 status half-installed owncloud:all 5.0.13-1
2014-02-17 21:53:11 status unpacked owncloud:all 6.0.1-1
2014-02-17 21:53:11 status unpacked owncloud:all 6.0.1-1
2014-02-17 21:53:12 install php5-imagick:amd64  3.1.0~rc1-1+b2
2014-02-17 21:53:12 status half-installed php5-imagick:amd64 3.1.0~rc1-1+b2
2014-02-17 21:53:12 status unpacked php5-imagick:amd64 3.1.0~rc1-1+b2
2014-02-17 21:53:12 status unpacked php5-imagick:amd64 3.1.0~rc1-1+b2
2014-02-17 21:53:12 trigproc libapache2-mod-php5:amd64 5.4.4-14+deb7u7 5.4.4-14+deb7u7
2014-02-17 21:53:12 status half-configured libapache2-mod-php5:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:12 status installed libapache2-mod-php5:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:13 trigproc man-db:amd64 2.6.2-1 2.6.2-1
2014-02-17 21:53:13 status half-configured man-db:amd64 2.6.2-1
2014-02-17 21:53:13 status installed man-db:amd64 2.6.2-1
2014-02-17 21:53:14 startup packages configure
2014-02-17 21:53:14 configure libserf1:amd64 1.3.4-1+WANdisco 
2014-02-17 21:53:14 status unpacked libserf1:amd64 1.3.4-1+WANdisco
2014-02-17 21:53:14 status half-configured libserf1:amd64 1.3.4-1+WANdisco
2014-02-17 21:53:14 status installed libserf1:amd64 1.3.4-1+WANdisco
2014-02-17 21:53:14 configure php5-intl:amd64 5.4.4-14+deb7u7 
2014-02-17 21:53:14 status triggers-pending libapache2-mod-php5:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:14 status unpacked php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:15 status unpacked php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:15 status half-configured php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:15 status triggers-awaited php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:15 configure php5-mcrypt:amd64 5.4.4-14+deb7u7 
2014-02-17 21:53:15 status unpacked php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:15 status unpacked php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:15 status half-configured php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:16 status triggers-awaited php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:16 configure php5-pgsql:amd64 5.4.4-14+deb7u7 
2014-02-17 21:53:16 status unpacked php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:16 status unpacked php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:16 status half-configured php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:17 status triggers-awaited php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:17 configure php5-imagick:amd64 3.1.0~rc1-1+b2 
2014-02-17 21:53:17 status unpacked php5-imagick:amd64 3.1.0~rc1-1+b2
2014-02-17 21:53:17 status half-configured php5-imagick:amd64 3.1.0~rc1-1+b2
2014-02-17 21:53:18 status installed php5-imagick:amd64 3.1.0~rc1-1+b2
2014-02-17 21:53:18 trigproc libapache2-mod-php5:amd64 5.4.4-14+deb7u7 
2014-02-17 21:53:18 status half-configured libapache2-mod-php5:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:18 status installed php5-pgsql:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:18 status installed php5-mcrypt:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:18 status installed php5-intl:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:18 status installed libapache2-mod-php5:amd64 5.4.4-14+deb7u7
2014-02-17 21:53:18 configure owncloud:all 6.0.1-1 
2014-02-17 21:53:18 status unpacked owncloud:all 6.0.1-1
2014-02-17 21:53:19 status unpacked owncloud:all 6.0.1-1
2014-02-17 21:53:19 status unpacked owncloud:all 6.0.1-1
2014-02-17 21:53:19 status half-configured owncloud:all 6.0.1-1
2014-02-17 21:53:19 status installed owncloud:all 6.0.1-1
2014-02-17 21:53:19 configure subversion:amd64 1.7.14-1+WANdisco 
2014-02-17 21:53:19 status unpacked subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status unpacked subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status unpacked subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status unpacked subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status half-configured subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status installed subversion:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 configure libsvn1:amd64 1.7.14-1+WANdisco 
2014-02-17 21:53:19 status unpacked libsvn1:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status half-configured libsvn1:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:19 status installed libsvn1:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:20 configure libapache2-svn:amd64 1.7.14-1+WANdisco 
2014-02-17 21:53:20 status unpacked libapache2-svn:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:20 status unpacked libapache2-svn:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:20 status unpacked libapache2-svn:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:20 status half-configured libapache2-svn:amd64 1.7.14-1+WANdisco
2014-02-17 21:53:20 status installed libapache2-svn:amd64 1.7.14-1+WANdisco

It’s much more verbose but each line contains both the date and time and what was performed. All you need to do is use a grep to extract the information you need. So depending what you’re after you’ll use one of these 3 commands:

zcat -f $(ls -tr /var/log/dpkg.log*) | grep " install "
zcat -f $(ls -tr /var/log/dpkg.log*) | grep " upgrade "
zcat -f $(ls -tr /var/log/dpkg.log*) | grep " remove "

You will then see something like this:

# zcat -f $(ls -tr /var/log/dpkg.log*) | grep " remove "
2014-12-01 18:43:15 remove libneon27-gnutls:amd64 0.29.6-3 <keine>
2014-12-01 18:43:16 remove libnss3-1d:amd64 2:3.14.5-1+deb7u3 <keine>
2014-12-01 18:43:16 remove libtommath0:amd64 0.42.0-1 <keine>

Note that keine means “none” in German, so removing a package is going from a certain version to none.

Mac OS X: PHP Notice: Use of undefined constant MCRYPT_RIJNDAEL_128

My setup:

  • OS: Mac OS X Yosemite (10.10.1)
  • Using Homebrew
  • PHP: 5.5.14

I’ve installed an empty Laravel installation and got the following error message when navigating to http://localhost/kanban/public/:

Notice: Use of undefined constant MCRYPT_RIJNDAEL_128 – assumed ‘MCRYPT_RIJNDAEL_128′ in /Library/WebServer/Documents/xxx/config/app.php on line 83

Googling for this error message return many tutorials on how to install mcrypt on Mac OS X (whether building it from source or using Homebrew). The problem was that both the mcrypt and the php55-mcrypt packages were properly installed:

$ brew install mcrypt
Warning: mcrypt-2.6.8 already installed
$ brew install php55-mcrypt
Warning: php55-mcrypt-5.5.20 already installed

Mcrypt was also properly loaded by PHP:

$ php -m | grep mcrypt
mcrypt

$ php -i | grep mcrypt
Additional .ini files parsed => /usr/local/etc/php/5.5/conf.d/ext-mcrypt.ini,
Registered Stream Filters => zlib.*, bzip2.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk, mcrypt.*, mdecrypt.*
mcrypt
mcrypt support => enabled
mcrypt_filter support => enabled
mcrypt.algorithms_dir => no value => no value
mcrypt.modes_dir => no value => no value

Looking at the results of phpinfo() in a Web browser, I could notice two things:

  1. No mention of mcrypt being loaded
  2. Scan this dir for additional .ini files: /Library/Server/Web/Config/php

Actually the directory /Library/Server/Web/Config/php didn’t exist (neither did the parent directories). Since the command line php seemed to scan the directory /usr/local/etc/php/5.5/conf.d for additional .ini files, I did the following:

First, I created the directory PHP wanted to scan:

sudo mkdir -p /Library/Server/Web/Config/php

Then, I created a symbolic link from the crypt ini file loaded by my PHP CLI to this directory:

sudo ln -s /usr/local/etc/php/5.5/conf.d/ext-mcrypt.ini /Library/Server/Web/Config/php/ext-mcrypt.ini

Finally, I restarted Apache:

sudo apachectl restart

After that checking the output of phpinfo() in my browser, I could see the following:

mcrypt

mcrypt support enabled
mcrypt_filter support enabled
Version 2.5.8
Api No 20021217
Supported ciphers cast-128 gost rijndael-128 twofish arcfour cast-256 loki97 rijndael-192 saferplus wake blowfish-compat des rijndael-256 serpent xtea blowfish enigma rc2 tripledes
Supported modes cbc cfb ctr ecb ncfb nofb ofb stream

 

Directive Local Value Master Value
mcrypt.algorithms_dir no value no value
mcrypt.modes_dir no value no value

And the Laravel application was working !

Powershell: Check whether a file is locked

When opening a StreamReader for a file in powershell, you will get an exception if the file is locked:

The process cannot access the file ‘xxx’ because it is being used by another process.

You can use this to check whether a file is locked. You just need to setup a trap which will say that the file is locked if an exception occurs and then open the file:

trap {
	Write-Output "$filename is locked"
	continue
}
$stream = New-Object system.IO.StreamReader $filename
if ($stream) {$stream.Close()}

Of course, you might also get an exception if the file doesn’t exist, so non-existing files will be reported as locked. So before checking the file, we need to make sure the file exists:

$file = gi (Resolve-Path $filename) -Force
if ($file -is [IO.FileInfo]) {
	trap {
		Write-Output "$filename is locked"
		continue
	}
	$stream = New-Object system.IO.StreamReader $file
	if ($stream) {$stream.Close()}
}

So if the file doesn’t exist, you will get an exception before we define the trap and the trap will only be activated if you cannot open the file. Here’s the output when the file doesn’t exist:

Resolve-Path : Cannot find path ‘D:\Temp\AuditAnalysis\test.xlsx2′ because it does not exist.
At D:\Temp\AuditAnalysis\test_lock.ps1:13 char:25
+ $file = gi (Resolve-Path <<<< $filename) -Force
+ CategoryInfo : ObjectNotFound: (D:\Temp\AuditAnalysis\test.xlsx2:String) [Resolve-Path], ItemNotFoundException
+ FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.ResolvePathCommand

Get-Item : Cannot bind argument to parameter ‘Path’ because it is null.
At D:\Temp\AuditAnalysis\test_lock.ps1:13 char:11
+ $file = gi <<<< (Resolve-Path $filename) -Force
+ CategoryInfo : InvalidData: (:) [Get-Item], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.GetItemCommand

Here’s the full code with parameter handling:

############################################################
#                      Test file lock
#                      by Henri Benoit
############################################################

Param(
    [parameter(Mandatory=$true)]
    $filename
)

Write-Output "Checking lock on file: $filename"

$file = gi (Resolve-Path $filename) -Force
if ($file -is [IO.FileInfo]) {
	trap {
		Write-Output "$filename is locked"
		continue
	}
	$stream = New-Object system.IO.StreamReader $file
	if ($stream) {$stream.Close()}
}
exit

If the file is locked it will display the following:

Checking lock on file: test.xlsx
test.xlsx is locked

If it is not locked, only the first line will be displayed:

Checking lock on file: test.xlsx

If you want to check whether a file is locked within your powershell code, you can define this in a function and instead of writing something to the output, you can use Set-Variable to set a variable to true and return this variable.

If you actually do not care whether the file is locked or not but just want to read the contents of the file no matter what, you could use this to check whether the file is locked and if yes copy the file before reading it.

Of course you might want to also define an additional function checking whether the file exists by trapping the error in Resolve-Path.

PHP: composer install or update causes a PHP Fatal error

When running composer install or update I was getting a PHP fatal error because PHP was using more than 512MB:

$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev)
PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php on line 123
PHP Stack trace:
PHP 1. {main}() /usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar:0
PHP 2. require() /usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar:14
PHP 3. Composer\Console\Application->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/bin/composer:43
PHP 4. Symfony\Component\Console\Application->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:83
PHP 5. Composer\Console\Application->doRun() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:121
PHP 6. Symfony\Component\Console\Application->doRun() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:117
PHP 7. Symfony\Component\Console\Application->doRunCommand() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:191
PHP 8. Symfony\Component\Console\Command\Command->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:881
PHP 9. Composer\Command\InstallCommand->execute() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:241
PHP 10. Composer\Installer->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Command/InstallCommand.php:110
PHP 11. Composer\Installer->doInstall() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:210
PHP 12. Composer\DependencyResolver\Solver->solve() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:449
PHP 13. Composer\DependencyResolver\RuleSetGenerator->getRulesFor() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/Solver.php:166
PHP 14. Composer\DependencyResolver\RuleSetGenerator->addRulesForUpdatePackages() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php:275
PHP 15. Composer\DependencyResolver\RuleSetGenerator->addRulesForPackage() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php:235
PHP 16. Composer\DependencyResolver\RuleSetGenerator->createConflictRule() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php:204

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php on line 123

Call Stack:
0.0100 385808 1. {main}() /usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar:0
0.0103 387560 2. require(‘phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/bin/composer’) /usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar:14
0.0308 2886688 3. Composer\Console\Application->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/bin/composer:43
0.0334 3185464 4. Symfony\Component\Console\Application->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:83
0.0345 3310400 5. Composer\Console\Application->doRun() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:121
0.0354 3389704 6. Symfony\Component\Console\Application->doRun() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Console/Application.php:117
0.0355 3390560 7. Symfony\Component\Console\Application->doRunCommand() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:191
0.0356 3390896 8. Symfony\Component\Console\Command\Command->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Application.php:881
0.0360 3394928 9. Composer\Command\InstallCommand->execute() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/vendor/symfony/console/Symfony/Component/Console/Command/Command.php:241
0.1072 6425464 10. Composer\Installer->run() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Command/InstallCommand.php:110
0.1081 6525856 11. Composer\Installer->doInstall() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:210
3.8979 41690568 12. Composer\DependencyResolver\Solver->solve() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/Installer.php:449
3.8985 41756784 13. Composer\DependencyResolver\RuleSetGenerator->getRulesFor() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/Solver.php:166
3.9001 41901248 14. Composer\DependencyResolver\RuleSetGenerator->addRulesForUpdatePackages() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php:275
3.9001 41903408 15. Composer\DependencyResolver\RuleSetGenerator->addRulesForPackage() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php:235
104.6757 536552128 16. Composer\DependencyResolver\RuleSetGenerator->createConflictRule() phar:///usr/local/Cellar/composer/1.0.0-alpha8/libexec/composer.phar/src/Composer/DependencyResolver/RuleSetGenerator.php:204

This wasn’t caused by an old version of PHP as I was using PHP 5.5.19:

$ php –version
PHP 5.5.19 (cli) (built: Nov 23 2014 20:46:27)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
with Xdebug v2.2.6, Copyright (c) 2002-2014, by Derick Rethans

So increasing the maximum memory limit in php.ini would probably solve it but more than 512MB just for a composer install seemed a little bit excessive. So since my PHP install had a current version, I checked composer:

$ composer –version
Composer version 1.0.0-alpha8 2014-01-06 18:39:59

OK, it wasn’t that old, but I checked anyway whether there was a newer version and found one:

$ composer selfupdate
Updating to version 029f7093009f621bd20aef98c7bfc61631f18cf1.
Downloading: 100%
Use composer self-update –rollback to return to version 1.0.0-alpha8

After updating to alpha9, I was able to run composer install !! Sometimes fixing your problem is as easy as making sure you use the latest version of all involved software !

It looks like the memory usage of composer was improved in the latest update. But since composer is still using a lot of memory (depending on the packages you have configured), when using it on your development machine, you should set a high memory limit in php.ini and when deploying to your production server it is best to use a composer.lock especially if you’re on shared hosting.

So you should allocate a lot of memory to PHP on your development machine, only run composer update there and only run composer install with a composer.lock file on your deployment machine.

Also, I have tried it but it looks like it helped some people to disable xdebug.

JavaScript: variables in asynchronous callback functions

What’s the problem?

First let’s assume you have declared a variable called myVar:

var myVar;

If you immediately log the contents of the variable, it will return undefined.

Now if you define any of the following:

setTimeout(function() { myVar = "some value"; }, 0);

or:

$.ajax({
	url: '...',
	success: function(response) {
		myVar = response;
	}
});

or:

var script = document.createElement("script");
script.type = "text/javascript";
script.src = "myscript.js";
script.onload = function(){
	myVar = "some value";
};
document.body.appendChild(script);

and immediately afterwards display the contents of myVar, you will also get undefined as result.

Why is the variable not set?

In all examples above, the function defined are what’s called asynchronous callbacks. This means that they are immediately created but not immediately executed. setTimeout pushes it into the event queue, the AJAX calls will execute it once the call returns and onload will be executed when the DOM element is loaded.

In all three cases, the function does not execute right away but rather whenever it is triggered asynchronously. So when you log or display the contents of the variable, the function has not yet run and the content has not yet been set.

So even though JavaScript actually works with a single-thread model, you’ve landed in the asynchronous problem zone.

Synchronous vs. Asynchronous

Basically the difference between a synchronous function call and an asynchronous one can be shown with these 2 pieces of code:

var myVar;

function doSomething() {
	myVar = "some value";
}

//synchronous call
doSomething();

log(myVar);

And:

var myVar;

function doSomething() {
	myVar = "some value";
}

log(myVar);

//asynchronous call
doSomething();

In the asynchronous case, our function is also defined before logging the variable but is call some time later.

How to make it work?

You will need to rewrite your code in order to use callbacks which will be called when the processing (in this case setting the value of myVar) is done.

First example: Instead of using the following:

var myVar;

setTimeout(function() { 
    myVar = "some value"; 
}, 0);

alert(myVar);

You should rather do the following:

var myVar;

function callback() {
    alert(myVar);
}

setTimeout(function() { 
    myVar = "some value"; 
    callback();
}, 0);

Instead of this:

var myVar;

var script = document.createElement("script");
script.type = "text/javascript";
script.src = "d3.min.js";
script.onload = function(){
	myVar = "some value";
};
document.body.appendChild(script);

alert(myVar);

Use this:

function callback(response) {
    alert("loaded");
}

var script = document.createElement("script");
script.type = "text/javascript";
script.src = "d3.min.js";
script.onload = callback;
document.body.appendChild(script);

And instead of doing the following:

var myVar;

$.ajax({
	url: '...',
	success: function(response) {
		myVar = response;
	}
});

alert(myVar);

Do the following:

function callback(response) {
    alert(response);
}

$.ajax({
	url: '...',
	success: callback
});

Of course, here another solution is to make the AJAX call synchronous by adding the following to the options:

async: false

Also if you use $.get or $.post, you’ll need to rewrite it to use $.ajax instead.

But this is not a good idea. First if the call lasts longer than expected, you will block the UI making your browser window unresponsive. And at some point in the time the browser will ask your user whether you want to stop the unresponsive script. So, even though programing everything in an asynchronous way with callbacks is not always trivial, doing everything in a synchronous way will cause more sporadic and difficult to handle issues.

So to summarize, you should either use a callback or directly call a function after processing and not only rely on setting a variable in the asynchronous part.

Debian/Ubuntu: The following packages have been kept back

Note that for all commands below, you might need to add a leading sudo depending on your system.

When updating your packages using apt-get, you might get the following message:

# apt-get upgrade
Building Dependency Tree… Done
The following packages have been kept back:
xxxxxx xxxxxx xxxxxx xxxxxx
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.

This means that there are packages which can be updated but they were not. So what are the reasons why some packages wouldn’t be updated ?

First let’s look at what apt-get upgrade does:

apt-get upgrade installs the newest versions of all packages which are already installed on the system from the configured sources. What apt-get upgrade doesn’t do is install additional packages or remove existing ones. The fact that it doesn’t install new packages is the reason why your packages are being kept back. These packages have an updated list of dependencies and apt-get would need to install new packages in order to install the new version of the software.

So what can you do about it ? There are basically 3 ways to handle it.

dist-upgrade

The first one is to use dist-upgrade. apt-get dist-upgrade does basically the same as upgrade. But additionally, it removes and installs packages in order to accommodate the updated list of dependencies of newer versions of the install packages.

When using diet-upgrade, you will be presented with a list of changes which will be applied. There will be a list of packages being additionally installed and a list of packages which will be removed.

You have to especially pay attention to the list of packages to be removed. In the list of packages to be removed, you will not only find packages being removed because they are not needed anymore but also packages which will be removed because they are not compatible with the latest version of the packages you are updating.

On my system, this is very often the case for Plesk which tends to force me to keep outdated packages for a little longer. Forcing an update of these packages would result in Plesk being uninstalled. Recovering from this is not very easy… Also, if you install beta versions of some software, you might also end up getting long list of packages to be removed because they would not be compatible anymore.

Also, you might in some cases have packages required by others but not cleanly listed as a dependency (once had this problem with subversion).

install

If you want to have the required packages installed but do not want to have packages removed, you can use apt install:

apt-get install <list of packages>

This will first resolve the kept-back dependencies and will offer to install additionally required packages.

aptitude

Start aptitude, select the list of upgradable packages, press “g” twice to install.

Then answer the questions and follow the instructions.

Still problems?

If all of this doesn’t work, you may want to give some of the following a try. These are various things I’ve needed in some cases to get rid of conflicts using apt-get.

Reinstalling the packages kept back:

apt-get install --reinstall <list of packages>

Checking for a given package, the installed and the versions available from the configured sources:

apt-cache policy <package name>

Use aptitude instead of apt-get:

aptitude update && aptitude upgrade

Use aptitude safe-upgrade to upgrade currently installed packages as well as install new packages to resolve new dependencies, but without removing installed packages:

aptitude safe-upgrade

 

JavaScript: Workaround for Object.defineProperty in IE8

Even though Internet Explorer 8 is now very old and outdated, there are unfortunately still many corporate users who are still stuck with this old browser. Many things are broken in IE8. One of them is the implementation of Object.defineProperty.

I noticed it when using the polyfill for String.startswith you can find on MDN looks. It looks like this:

if (!String.prototype.startsWith) {
  Object.defineProperty(String.prototype, 'startsWith', {
    enumerable: false,
    configurable: false,
    writable: false,
    value: function(searchString, position) {
      position = position || 0;
      return this.lastIndexOf(searchString, position) === position;
    }
  });
}

While Object.defineProperty is fully supported in IE9, it’s only partially supported in IE8. Internet Explorer 8 only supports defineProperty for DOM objects. If you use it for anything else, IE will complain that there is nothing like Object.defineProperty.

This is especially bad since calling defineProperty will cause and error which will prevent the rest of your JavaScript code to run.

So what can we do about it ?

The first thing I tried was:

        if (defineProperty){
            Object.defineProperty(String.prototype, 'startsWith', {
                enumerable: false,
                configurable: false,
                writable: false,
                value: function (searchString, position) {
                    position = position || 0;
                    return this.indexOf(searchString, position) === position;
                }
            });
        }

Unfortunately I then got an error in the if. So the easiest way to do this turned out to be to try to execute the defineProperty and ignore exceptions:

if (!String.prototype.startsWith) {
    try{ 
        if (defineProperty){
            Object.defineProperty(String.prototype, 'startsWith', {
                enumerable: false,
                configurable: false,
                writable: false,
                value: function (searchString, position) {
                    position = position || 0;
                    return this.indexOf(searchString, position) === position;
                }
            });
        }
    } catch (e) { };
}

After that, you can check again whether startsWith exists and add this function:

if (!String.prototype.startsWith) {
    String.prototype.startsWith = function(searchString, position) {
        position = position || 0;
        return this.indexOf(searchString, position) === position;
    };
}

 

‘console’ is undefined

As shown in this post logging to the console can be very useful when writing JavaScript code. Unfortunately, the JavaScript console is not evenly supported by all browsers. You first have browsers which don’t know it at all (like IE8) and will throw the error which I use as the title of this post. And then even two browsers which do have a console object do not all implement the same functions.

Here’s a short overview of what’s supported by which browser (note that if you do not have the latest version of a specific browser, this particular version may not support all listed functions):

Function Firefox Chrome Internet Explorer Safari Opera
assert X  X  X
clear X  X
count X X  X  X  X
debug X X  X  X  X
dir X X  X  X
dirxml X  X  X  X
error X X  X  X
_exception X
group X X  X  X  X
groupCollapsed X X  X  X  X
groupEnd X X  X  X  X
info X X  X  X
log X X  X  X
markTimeline  X
profile X X  X
profileEnd X X  X
table X
time X X  X  X  X
timeEnd X X  X  X  X
timeline X
timelineEnd X
timeStamp X
trace X  X  X  X
warn X X  X  X

So even though there are quite a few functions supported by all browsers, you should still check whether the console exists and whether the particular function you want to use exists. But adding before each call to a console function a check for the console and for the function quickly becomes a pain.

A way to work around it is to:

  1. Create a dummy console object if console is not defined
  2. Create dummy functions for all functions not supported by the console object in this particular browser

The dummy console object is created by this code:

if (!window.console) {
        window.console = {};
}

So simple, if the global console variable doesn’t exist, create one as an empty object. Now we’ve got either a real console object or a dummy object, we got rid of the error message saying that console is undefined. But we’ll still get errors when our code calls undefined functions. So let’s go through all functions and for each of them create a dummy function if missing:

(function() {
	var funcList = ["assert", "clear", "count", 
		"debug", "dir", "dirxml", "error", 
		"_exception", "group", "groupCollapsed", 
		"groupEnd", "info", "log", "markTimeline", 
		"profile", "profileEnd", "table", "time", 
		"timeEnd", "timeline", "timelineEnd", 
		"timeStamp", "trace", "warn"];

	var funcName;

	for (var i=0; i < funcList .length; i++) {
		funcName = funcList [i];
		if (!window.console[funcName]) {
			window.console[funcName] = function() {};
		}
	}
})()

Here a few explanations how this works. First wrapping it all in (function() { … })() is used to scope the variables we define in there, so that we do not pollute the global scope.

Then we define an array containing all known console functions and iterate through them. For each function, we check whether the console object (the real one or the dummy one) have such a function defined. If not, we assign an empty function. This is done by using the fact that functions of an object are just special properties of the object and that object properties are indexed.

So, using this, you will get rid of all errors related to the console being undefined or console functions being undefined, but of course since we add empty implementations, using these empty implementations will still have no effect.

Of course, instead of using empty implementations you could log the calls to console functions somewhere (e.g. an array) so that you can access it. If I ever need it, I might actually write some code for it and update this post.

Chrome: Resource interpreted as Font but transferred with MIME type font/x-woff

When loading one of my website in Chrome, I noticed the following error message in the JavaScript console:

Resource interpreted as Font but transferred with MIME type font/x-woff: “https://xxx.xxx/xxx.woff”.

Actually it looks like it took too long to define and official MIME type for WOFF fonts and a few different MIME types have been used until the official one was defined:

  • font/x-woff
  • application/x-font-woff
  • application/font-woff – This is actually the official MIME type for WOFF fonts
  • font/woff

By default IIS7 will not know what to do with the WOFF font file, so you will get a 404 error when fetching it.

In my case I was using some third party files which defined the following in order to solve the 404 error:

  <system.webServer>
    <staticContent>
      <remove fileExtension=".woff" />
      <mimeMap fileExtension=".woff" mimeType="font/x-woff" />
    </staticContent>
  </system.webServer>

But this MIME type is now not in use anymore. So I tried the following:

  <system.webServer>
    <staticContent>
      <remove fileExtension=".woff" />
      <mimeMap fileExtension=".woff" mimeType="application/font-woff" />
    </staticContent>
  </system.webServer>

And the error in the console was gone. Actually using application/x-font-woff instead of application/font-woff also works. This is probably because for a very long time, Chrome didn’t support the new MIME type and really expected application/x-font-woff.

If you’re using Apache as a Web Server, you will need to add the AddType directive to your .htaccess file:

AddType application/font-woff .woff

Note that you may get any of the following error messages which basically all mean that the mime type provided is not the expected one and you need to update the configuration of your web server:

resource interpreted as font but transferred with mime type font/x-woff
resource interpreted as font but transferred with mime type application/octet-stream
resource interpreted as font but transferred with mime type application/x-woff
resource interpreted as font but transferred with mime type application/x-font-otf
resource interpreted as font but transferred with mime type font/woff
resource interpreted as font but transferred with mime type font/truetype

If you have a similar problem with other types of files, please refer to the official list of Media Types (formerly called MIME types).

The exception listener on JMS connections is never called

Since we were having a few issues with memory leaks while handling JMS connections, we wanted to setup am Exception Listener (using the setExceptionListener method of the connection) to handle a connection failure and reconnect. Especially, if you consume messages asynchronously, this seems like a good way to learn that your connection has failed and reconnect.

Unfortunately, we never were notified although we could clearly see that the connection failed.

The problem is that we were in a JBoss container and as the J2EE specification says:

This method must not be used in a Java EE web or EJB application. Doing so may cause a JMSException to be thrown though this is not guaranteed.

This is actually true not only for setExceptionListener but for all the following methods of the JMS Connection class:

  • createConnectionConsumer
  • createSharedConnectionConsumer
  • createDurableConnectionConsumer
  • createSharedDurableConnectionConsumer
  • setClientID
  • setExceptionListener
  • stop

The first four of them are not allowed because of restrictions on the use of threads in the container. The other ones because they may interfere with the
connection management functions of the container.

In the past this was not always enforced by J2EE container vendors but in order to pass the compatibility test suites, they have started really enforcing this policy. So with many containers, you will get an IllegalStateException or a JMSException when invoking setExceptionListener (which states that you’re not allowed to call it).