Fixing the Shellshock GNU bash shell vulnerability

What’s shellshock?

Shellshock (also known as Bashdoor) is a vulnerability in GNU bash shell. It allows an attacker access to run remote commands on your system. It seems to affect the Bash versions 1.13 to 4.3. In the past few days, botnets have used compromised computers for distributed denial-of-service (DDOS) attacks and vulnerability scanning. Bash is installed as the default command line interface on many Unix-based systems. It is used to interpret commands and is itself a command which can be executed. As such it has both a list of environment variable and a list of internal functions. When launching a new bash interpreter, a new list of environment variables and functions can be exported to the new process. This vulnerability causes Bash to execute commands stored in environment variables in a special way. This happens when a newly started instance of Bash scans its environment variables for values in a special format, which should be converted to internal functions. In the affected versions, Bash creates fragments of code on-the-fly and executes them without checking whether the fragments are only function definitions. Using this vulnerability, the attacker can execute any arbitrary command.

How to check whether you are affected?

In order to check whether your system is affected you can execute the following from a shell:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If the ouput contains “vulnerable”, you are affected. Otherwise, it means the patch for Bash has already been installed (but keep in mind that there different ways to exploit this vulnerability and even if this check is negative, it’d be safe to update to newer versions of Bash as they are released).

Your server is not only vulnerable when spawning a Bash instance from the command line but also when Apache is used in combination with mod_cgi and spawns either a Bash shell or anything else (e.g. a Perl script) which executes commands and passes environment variables.

A few examples are:

  • A Perl script using the built-in functions exec or system to invoke a new shell process
  • A PHP script using the built-in functions exec or system to invoke a new shell process
  • A Python script using os.system or os.popenx (depending on how it is called)

Note that an attack on CGI scripts will only work if Bash is used as default shell.

Just to show you how easy it is to use this vulnerability against a server running Apache, mod_cgi and some CGI scripts spawning a shell:

curl "https://mysiteipaddress.com/index.cgi" --insecure -H "User-Agent: () { :; }; echo 'Hello Server' > /tmp/shellshock.log"

This will return some HTML code and will create a file /tmp/shellshock.log on the affected server.

How to fix it?

In most systems, it’s as easy as installing the latest Bash update. On Debian, Ubuntu and other systems using apt-get:

apt-get update && apt-get upgrade

On Red Hat, Fedory, CentOS and other systems using yum:

yum -y update bash

On Arch Linux:

pacman -Syu

On SLES:

zypper ref -s
zypper up bash

SUSE Linux Enterprise Server

Note that updatse for SLES are only available in LTSS versions and for SLES 11 SP3. If you are running an older non-LTSS version of SLES, you’ll have to install Bash from source as described below. Here the list of SLES versions for which an update is available:

  • SUSE Linux Enterprise Server 10 SP3 LTSS
  • SUSE Linux Enterprise Server 11 SP2 LTSS
  • SUSE Linux Enterprise Server 11 SP1 LTSS
  • SUSE Linux Enterprise Server 10 SP4 LTSS
  • SUSE Linux Enterprise Server 11 SP3

Red Hat Enterprise Linux

RHEL updates are available for the following versions:

  • Red Hat Enterprise Linux 4 Extended Lifecycle Support
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 5.6 Long Life
  • Red Hat Enterprise Linux 5.9 Extended Update Support
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 6.2 Advanced Update Support
  • Red Hat Enterprise Linux 6.4 Extended Update Support
  • Red Hat Enterprise Linux 7

 Mac OS X

Apple has released updates for Mac OS X Lion, Mountain Lion and Mavericks. Make sure that you install these updates even if Apple has been arguing that most Mac OS X users are not exposed unless they have configured advanced UNIX services. Since most Max OS X users do not have a web server running, the risk of having this vulnerability exploited is not as high as for Unix-based web servers accessible from the internet but you do not want to risk anything especially when the fix is as easy as updating your software (which should be done on a regular basis anyway).

How to install from source?

Latest Bash version (4.3)

If no update is available for your system (e.g. because you’re on an older version), you’ll have to download Bash 4.3, apply all patches, compile it and install it. If you have a direct access from your machine to the internet, do the following:

cd /tmp
mkdir bash
cd bash
wget https://ftp.gnu.org/gnu/bash/bash-4.3.tar.gz
for i in $(seq -f "%03g" 1 27); do wget https://ftp.gnu.org/gnu/bash/bash-4.3-patches/bash43-$i; done
tar zxvf bash-4.3.tar.gz 
cd bash-4.3
for i in $(seq -f "%03g" 1 27);do patch -p0 < ../bash43-$i; done
./configure && make && make install

If there are more than 27 patches for Bash 4.3 when you read this replace both occurences of 27 by the appropriate number.

If you do not have direct access to the internet from the machine, you’ll have to download Bash 4.3 and all updates and then copy them through (S)FTP to /tmp/bash:

tar zxvf bash-4.3.tar.gz 
cd bash-4.3
for i in $(seq -f "%03g" 1 27);do patch -p0 < ../bash43-$i; done
./configure && make && make install

Note: If the files are renamed to bash43-0xx.txt when downloading, you’ll have to remove the .txt extension. And change 27 to the current number of patches for Bash 4.3.

Also note that bash will be installed in /usr/local/bin/bash. you will then need to make sure that the old bash is removed and replaced by a symlink to the new one:

cd /bin; rm bash; ln -s /usr/local/bin/bash

(thanks Gerhard for the hint)

Older Bash version (e.g. 3.2)

If you’re using an older version of Bash (e.g. because you’re running SLES 11 SP1), you might want to only update Bash to the latest update within the same version.

First create the working directory /tmp/bash using: mkdir /tmp/bash

Download https://ftp.gnu.org/gnu/bash/bash-3.2.48.tar.gz instead of the 4.3 package. This package already includes the updates until update 48. Additionally download updates 49 to 54 from https://ftp.gnu.org/gnu/bash/bash-3.2-patches/.

Note: If the files are renamed to bash32-0xx.txt when downloading, you’ll have to remove the .txt extension.

Upload all downloaded files to /tmp/bash. Go to /tmp/bash using: cd /tmp/bash and execute the following:

tar zxvf bash-3.2.48.tar.gz 
cd bash-3.2.48/
for i in $(seq -f "%03g" 49 54);do patch -p0 < ../bash32-$i; done
./configure && make && make install

Change 54 to the current number of patches for Bash 3.2.

Also note that this uses the patch command. If it is not installed, you’ll have to install it or you’ll have to perform all steps except the last one on a machine where patch is installed and then copy the resulting folder to the machine where it should be installed before executing the last command. Before executing this last command you might need to add execution permissions to the configure file:

chmod +x configure

If you get the following error message during the install step:

/home/install -m 0755 bash /usr/local/bin/bash
make: execvp: /home/install: Permission denied
make: *** [install] Error 127

You will have to execute the following:

install -m 0755 bash /usr/local/bin/bash

Note that bash will be installed in /usr/local/bin/bash. you will then need to make sure that the old bash is removed and replaced by a symlink to the new one:

cd /bin; rm bash; ln -s /usr/local/bin/bash

(thanks Gerhard for the hint)

 LD_LIBRARY_PATH

I’ve noticed that with the new version of Bash there may be some issues when setting LD_LIBRARY_PATH  before starting some software. Actually you should not be using LD_LIBRARY_PATH on a production system but rather rely on ldconfig. In order to do this, you should add all directories containing shared libraries to /etc/ld.so.conf and execute the following:

ldconfig -v

This way you make sure that even with an updated version of Bash your software will find the shared libraries without issues.

 

Windows: Port 80 is already in use

I wanted to start the Apache web server using XAMPP and got an error message that port 80 was already in use. My first thought was that IIS might be running on port 80. But on this machine I hadn’t installed IIS. So I executed the following to check which process might be using it:

c:\> netstat -nabo
...
  TCP    [::]:80                [::]:0                 LISTENING       4
 Can not obtain ownership information
...

So it’s process ID 4. I also used TCPView from Sysinternals and also saw that the process ID 4 was using this port. The process name was shown as “System”. In Task Manager, the image name for process ID 4 was System and the description was “NT Kernel & System”. It didn’t help me much either.

So I went through the list of running services and saw a service called “Web Deployment Agent Service” with the description “Remote agent service for the Microsoft Web Deploy 3.5.”. After stopping this service I was able to start Apache on port 80 successfully.

You can check the services by pressing Windows-R to get to the Run prompt, typing services.msc and pressing enter.

If this doesn’t help, I’d recommend searching for a service called “World Wide Web Publishing Service” and stop it. And of course, if IIS is running, you should stop it.

Mod_pagespeed: clear the cache

The Apache module mod_pagespeed can be used to automatically make web pages faster without effort. It promises a 50% speed boost. I’ve been using for some time and can confirm that there is a definitive speed increase.

But there is a big problem when I update the styling of my blog. When I change the theme css file, I do not see the changes. In the beginning I thought it was caused by the caching plugin I use. But after deactivating the plugin, the problem persisted. Using Ctrl-F5 did not help either. I also opened the CSS in the browser and pressed Ctrl-F5. Nothing worked.

Finally, I understood that mod_pagespeed was responsible for it. So I just needed to clear its cache. Unfortunately it seems to be only possible from the command line. So first open a shell. Then you need to find out where the cache is. It is configured in pagespeed.conf. So either open the file and search for ModPagespeedFileCachePath or execute the following:

# grep "^ *ModPagespeedFileCachePath" /etc/apache2/mods-enabled/pagespeed.conf
    ModPagespeedFileCachePath            "/var/cache/mod_pagespeed/"

Now you need to go to this directory and touch a file (don’t worry if the file doesn’t exist yet, it will be created by touch if it doesn’t already exist), or just touch the file with it’s complete path:

touch /var/cache/mod_pagespeed/cache.flush

You then have to wait for a few second and reload your web page and you’ll see that the new style is used.

Instead of doing all these manual steps, you can do the following:

touch `grep "^ *ModPagespeedFileCachePath" /etc/apache2/mods-enabled/pagespeed.conf | awk ' { print $2; } ' | sed 's/"//g'`/cache.flush

You need to adapt the path to the apache configuration depending on where it is stored e.g. use the following if it is stored in /etc/httpd/conf.d instead of /etc/apache2/mods-enabled:

touch `grep "^ *ModPagespeedFileCachePath" /etc/httpd/conf.d/pagespeed.conf | awk ' { print $2; } ' | sed 's/"//g'`/cache.flush

(thanks Kieran for the hint).

Linux: Apache logs

Location

The Apache Web Server log files contains all errors found while serving requests as well as records of all incoming requests. The location of the log files is configurable. By default they are stored in one of these directories depending on the Linux distribution:

  • /var/log/httpd
  • /var/log/apache2
  • sometimes directly in /var/log
  • /usr/local/apache/log

The exact location of the log files is configured in the apache2.conf or httpd.conf log file. Here’s where you should look for it:

  • /etc/apache2/httpd.conf
  • /usr/local/etc/apache2/httpd.conf
  • /etc/httpd/conf/httpd.conf

To find the location of the error log file, open the apache configuration file and search for ErrorLog. Alternatively you can use grep to get the information without having to open the file in an editor:

# grep ^ErrorLog /etc/apache2/httpd.conf
ErrorLog /var/log/apache2/error_log

Replace /etc/apache2/httpd.conf by the actual location of the apache configuration file.

The access log file containing the records of all incoming requests, is usually stored at the same location. To get its name, you usually just have to replace error by access in the path to the error log file.

You can also get the actual location of the access log file using grep:

# grep ^CustomLog /etc/apache2/httpd.conf
CustomLog /var/log/apache2/access_log combined

Format

The “combined” after the log file path means that the “combined” log format should be used. The other log format you can use is the “common” log format. Both of them are pretty similar. The common log format will store the following information on each line:

  • IP address of the client
  • identity of the client
  • user ID
  • time that the server finished processing the request
  • method used (e.g. GET) and requested resource
  • status code sent back to the client
  • size of the object returned to the client

Additionally, the combined log format will also store the following information:

  • site the client has been referred from
  • User-Agent

Graceful restart

A graceful restart will instruct the web server to reopen the log files for new requests. When you rename the log files, apache will continue writing to the renamed log files. By performing a graceful restart of apache, you will cause apache to reopen the log files at the configured path and write to the new files for new requests. Requests being processed at the time of the graceful restart will be logged to the renamed file. So after a graceful restart, you will need wait for all pending requests to be finally processed before you can work on the renamed logfiles (e.g. compress them).

Here a short example:

mv /var/log/apache2/access_log /var/log/apache2/access_log.save
mv /var/log/apache2/error_log /var/log/apache2/error_log.save
/usr/sbin/apache2ctl -k graceful

The pending requests will be logged to access_log.save and error_log.save and new requests to access_log and error_log.

Note that instead of /usr/sbin/apache2ctl you might have to use one of the following:

  • /usr/sbin/apachectl
  • /usr/sbin/httpd2
  • /usr/local/apache2/bin/apachectl

Also if it complains that the arguments are not fine try executing it without the -k i.e. only with graceful as command line argument.

Log rotation

Using a graceful restart, you can manually rotate logs (or rotate them with a cron job) but there’s an even nicer solution. You can used piped logs. Piped logs just mean that the web server will write log files through a pipe to another process instead of directly to the files. Like this you can basically redirect the log files to rotatelogs which will create new log files on a regular basis.

Instead of:

CustomLog /var/log/apache2/access_log combined

configure the following:

CustomLog "|/usr/sbin/rotatelogs2 /var/log/apache2/access_log 86400" combined

86400 is the number of seconds after which the log file should be rotated (in this case 24 hours).

Instead of /usr/sbin/rotatelogs2 you might have to use one of the following paths:

  • /usr/sbin/rotatelogs
  • /usr/local/apache/bin/rotatelogs
  • /usr/local/sbin/rotatelog

Other log files

There are also other log files created by the apache log server or its modules. They are usually stored in the same location as the error and access logs. Some of them are:

  • ScriptLog: log for mod_rewrite, input to and output from CGI scripts
  • RewriteLog: log for mod_rewrite, rewrite transforms requests
  • JkLogFile : log for mod_jk, communication between Tomcat and Apache

Apache2 graceful restart: seg fault or similar nasty error detected in the parent process

This morning, we noticed that all our web pages on one server were not accessible. So first I checked the apache error logs. /var/log/apache2/error.log was 0 bytes long. So I went to error.log.1:

# tail /var/log/apache2/error.log.1
…
[Sun Jul 14 07:51:20 2013] [notice] Graceful restart requested, doing restart
[Sun Jul 14 07:51:21 2013] [notice] seg fault or similar nasty error detected in the parent process

So it looks like apache hung during a graceful restart.

For those who do not know what a graceful restart is:

The graceful signal causes the parent process to advise the children to exit after their current request (or to exit immediately if they’re not serving anything). The parent re-reads its configuration files and re-opens its log files. When a child dies off the parent replaces it with a child of the new generation of the configuration. This immediately begins serving new requests.

You can send such a graceful signal to apache2 like this:

# apachectl graceful

If it worked fine, you should see the following in /var/log/apache2/error.log:

[Sun Jul 14 12:50:59 2013] [notice] Graceful restart requested, doing restart
[Sun Jul 14 12:51:03 2013] [notice] Digest: generating secret for digest authentication ...
[Sun Jul 14 12:51:03 2013] [notice] Digest: done
[Sun Jul 14 12:51:03 2013] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Sun Jul 14 12:51:03 2013] [notice] mod_python: using mutex_directory /tmp 
[Sun Jul 14 12:51:03 2013] [notice] Apache/2.2.16 (Debian) DAV/2 mod_fcgid/2.3.6 Phusion_Passenger/3.0.11 mod_python/3.3.1 Python/2.6.6 mod_ssl/2.2.16 OpenSSL/0.9.8o mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations

Depending on which mods are installed, your output might be different.

Now to the problem we had this morning.

When apache2 is running, you have the parent process (usually running under user root in order to be able to listen to the standard HTTP/S ports) and a few children processes:

# ps -Af | grep apache2 | grep -v grep
www-data 10777 22889  0 12:31 ?        00:00:02 /usr/sbin/apache2 -k start
www-data 14601 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14618 22889  1 12:51 ?        00:00:02 /usr/sbin/apache2 -k start
www-data 14619 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14620 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14621 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14622 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14623 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14627 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14629 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 14636 22889  0 12:51 ?        00:00:00 /usr/sbin/apache2 -k start
root     22889     1  0 11:12 ?        00:00:01 /usr/sbin/apache2 -k start

This morning, it looked like this:

# ps -Af | grep apache2 | grep -v grep
www-data 14601     1  3 12:51 ?        00:00:10 /usr/sbin/apache2 -k start

So the parent process as well as almost all children processes are gone. Only one child process seems to hang.

The solution is simple, kill the last apache2 process and restart apache2:

# killall apache2
# service apache2 start

If you’ve already checked the processes as shown above you can also just use kill with the PID instead of killall:

# kill 14601
# service apache2 start

I haven’t yet been able to spend much time looking for the reason why this happened. A graceful restart of apache is usually done because the configuration changed or because the log files were rotated.

Many *.log.1 log files have the same timestamp as the entry in /var/log/apache2/error.log.1 and I could see that rsyslogd also got HUPed 1 second later:

syslogd: [origin software="rsyslogd" swVersion="4.6.4" x-pid="1244" x-info="http://www.rsyslog.com"] rsyslogd was HUPed, type 'lightweight'.

So I assume it’s related to rotating log files. CPanel seems to use graceful restarts a lot but our server runs Plesk and it seems not to use graceful restarts by default (still need to check the configuration in the database).