Linux: change the keyboard layout from the console

A short description of how to change the keyboard layout from the console on different Linux distributions:

Debian

See my separate post regarding this.

SLES or OpenSuSE

Use loadkeys:

# sudo loadkeys de
Loading /usr/share/kbd/keymaps/i386/qwertz/de.map.gz
assuming iso-8859-15 euro

Ubuntu

Use loadkeys as for SLES or OpenSuSE.

Note that you may first need to install console-tools by using:

sudo apt-get install console-tools

Fedora

You can use:

loadkeys de

If you are in a terminal window in X-Windows, it will not work and you’ll need to do the following:

setxkbmap -layout de

Puppy Linux

As for Fedora, loadkeys should work. If not use setxkbmap.

Linux: Grep IP addresses

I very often have to either find log entries related to a specific IP address or just find IP addresses in a log file or process a file containing IP addresses and other things. Here are a few commands I use for this.

First if you want to search for an IP address in a log file, you can just use grep:

grep 18.194.233.1 /var/log/auth.log

Unfortunately it might return more than expected: 118.194.233.1 and 18.194.233.14 also match. So we need to match 188.194.233.1 only using a whole word matching:

grep -w 18.194.233.1 /var/log/auth.log

Now if you want to search for lines containing IP addresses, you’ll need to use some regular expressions. An IP address is basically a dot separated sequence of 4 numbers each having 1 to 3 digits. So we can represent it this way:

grep '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' ...

If the IP addresses are stored alone on a line:

grep '^[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}$' ...

Of course if you have something like:

999.888.777.666

in your file, it will be matched although it is not a valid IP address (each part of the IP address cannot exceed 255). If you know you’ll never get such strings, then you can use the expression above otherwise:

Replace:

[0-9]\{1,3\}

By:

\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)

So using:

grep '\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)\.\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)\.\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)\.\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)' ...

If you do not want to allow IP address containing e.g. 001 instead of 1, you can replace the expression above:

\(25[0-5]\|2[0-4][0-9]\|[01][0-9][0-9]\|[0-9][0-9]\)

By:

\(25[0-5]\|2[0-4][0-9]\|[1][0-9][0-9]\|[1-9][0-9]|[1-9]\)

Linux: which vs. whereis

“which” is a very useful command used to find the location of an executable which is in your path e.g.:

# which yacc
/usr/bin/yacc

I actually didn’t know that there is also a command “whereis” which is kind of similar:

 # whereis yacc
yacc: /usr/bin/yacc /usr/share/man/man1p/yacc.1p.gz

It doesn’t only return the location of the executable but also of source files and man pages. Another difference is also that it doesn’t search the configured path but looks through an internal list of common Unix paths.

So this basically means that there are executables which will be found by “which” but not by “whereis” because they are in a non-standard directory which is in the path. On the other hand, “whereis” will find other executables not found by which if some of the standard directories are not in the path.

So if you do not find an executable with one of the two commands, you should give the other one a try.

“which” stays my preferred command for this (and I’ll only use “whereis” if which fails to find something) because it’s just returning one full path without anything else which makes it so easy to combine it with other commands e.g.:

# cat `which yacc`
#! /bin/sh
exec /usr/bin/bison -y "$@"

Of course you can also tell “whereis” to only show binaries and then use awk to get the output you need:

# whereis -b yacc | awk '{ print $2; }'
/usr/bin/yacc

So combining it with cat:

# cat `whereis -b yacc | awk '{ print $2; }'`
#! /bin/sh
exec /usr/bin/bison -y "$@"

But it’s of course much longer and more complex…

SLES: Reconfigure a fixed IP address

We recently had a problem with a SLES virtual machine which was copied and where the network connectivity was broken because we ended up being in a different network. So we needed to reconfigure the IP-Address.

It’s easy to solve temporally:

ifconfig eth0 192.168.190.200 netmask 255.255.255.0
route add default gw 192.168.190.1 eth0

Replace 192.168.190.200 by the IP address you want to set and 192.168.190.1 by the gateway address.

The IP address configuration will probably get lost on reboot.

A permanent fix with yast:

start yast from the command line.
Then select Network Devices > Network Card
Alt-h (Change…)
Delete the card with the old Mac-Address (probably called Ethernet Network…)
Edit the other one
Alt-t (Static address setup)
Alt-i (IP Address)
Type in the IP address you want the server to have (e.g. 192.168.190.200).
Alt-o (Routing)
Type in the IP address of the gateway (e.g. 192.168.190.1)
Alt-o (OK)
Alt-n (Next)
Alt-f (Finish)
Alt-q (Quit)

Execute the following to check that it’s fine:

ifconfig eth0

Then to check that the default gateway is properly setup, connect with putty.

Reboot the computer and check that everything is still OK.

Linux: just installed Linux on my last Windows machine at home

I switched to a MacBook Pro 3,5 years ago. Until then, I’ve mostly had Windows machines at home even though I’ve always been more of a Unix/Linux guy. We still had a netbook which was delivered with Windows 7 Starter. I quickly upgraded to Windows Home Basic. Not sure why this Starter Edition even exists. It’s just an operating system with so many restrictions that any OS available 10 years ago would actually be better. Well, the netbook has an Atom N455 processor and 2 GB of RAM. As always with Windows, the longer the machine was in use, the slower it was getting and with this low spec machine, it was even worse. So yesterday, we got fed up with waiting all the time while planning our next vacation and decided to install Linux.

I first had a look at which distribution would make sense. I of course needed a low footprint distribution so that it can still run smoothly on this netbook which is already a few years old. Also I didn’t want to spend weeks trying to get all drivers working. Also I needed to be able to install it without an optical drive which I do not have in the netbook (and I didn’t want to spend hours looking for the external DVD drive I have somewhere in a box in the cellar).

I finally decided to install Easy Peasy.
easypeasy
Easy Peasy is based on Ubuntu (so you can use the Ubuntu package administration) and was specially build for netbooks. It has a low footprint, the ISO is less 900 MB and it brings most drivers you need on a netbook. The UI is also optimize for smaller displays. It uses the Netbook-Launcher from the Ubuntu Netbook Remix and application windows are maximized and displayed without a title bar.

Easy Peasy also brings quite a few open source applications like OpenOffice, Songbird and Firefox. The nice update management makes sure that you always have the latest version of the different software.

You can write the ISO on a memory stick using UNetbootin. Like this you can also first test it and see whether it works for you before overwriting your previous OS. I had a small problem. My Wireless adapter needed a Broadcom driver which wasn’t part of the default package and needed to be downloaded. Of course without driver, no Wireless connection and thus no Internet to download the driver from. So I first had to find a network cable and connect the netbook directly to the router. Installing the driver was no issue at all and I only needed the network cable for a few minutes.

Easy Peasy is a further development of Ubuntu Eee but is now not only limited to Asus Eee PCs (although I actually do have one) but is meant to be used on any netbook.

I still need to test it a little bit more but I’m already very happy with the UI, the performance and the ease of installation and update.

Linux: truncate all lines in a file

I had the following problem. I was analyzing a dump of a Java process. There was a String object taking a few hundred megabytes of heap. I saved it to a file and wanted to look at it. This string contained a few hundred lines many of them were very long. It’s not so easy to scroll through such a file. Actually, only the beginning of each line was interesting since the rest of the line contained only a huge list of ascii codes.

Here is how you get only the first 30 characters of each lines:

cut -c1-30 INPUT_FILE_PATH > OUTPUT_FILE_PATH

It’s actually pretty easy ! Just love how much you can actually do with a one liner as soon as you’ve found out whether you need to use grep, awk, sed, cut, tr or a combination of them !

Debian: use SSHFS to mount a remote directory

In a previous post, I’ve shown how to use SSHFS to mount a remote directory as a volume in Mac OS X. Of course the same can be done to connect two Linux boxes.

Here’s a short description how to do it on Debian.

First you’ll need to install sshfs using apt-get:

apt-get install sshfs

Note that this will also install fuse as a dependency.

Then you need to have a public key authentication in place. I’m assuming that ssh is already installed on the server. You now should do the following, on the Linux box where the file system should be mounted:

apt-get install openssh-client

Now you can generate a key:

mkdir ~/.ssh
chmod 700 ~/.ssh
cd ~/.ssh
ssh-keygen -t rsa -C "henri.benoit@gmail.com"

Of course replace my email address by yours. You can then print the key:

cat id_rsa.pub

This should be added to the file ~/.ssh/authorized_keys on the server.

You also need to make sure that PubkeyAuthentication is set to yes in /etc/ssh/sshd_config so that public key authentication is supported at all.

Now you can connect using ssh without password and can mount over ssh:

mkdir ~/amazingweb
sshfs -p 22 USER@xxx.xxx.xxx.xxx:/var/www/vhosts ~/amazingweb -oauto_cache,reconnect

Replace USER by the user for which you’ve setup the public key authentication.
Replace xxx.xxx.xxx.xxx by the IP address of the remote server.
Replace ~/amazingweb by the path where you want to mount the remote file system.

It’s basically the same command as under Mac OS X except that you have to remove the 3 following options which are not available under Debian:

  • defer_permissions
  • noappledouble
  • volname

Debian: using netselect-apt to find the fastest archive mirror

If you need to quickly find the fastest Archive mirror to download Debian packages, you can either use apt-spy as shown here or use a tool called netselect-apt.

netselect-apt can be installed with apt-get:

# apt-get install netselect-apt
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  netselect
The following NEW packages will be installed:
  netselect netselect-apt
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 37.9 kB of archives.
After this operation, 171 kB of additional disk space will be used.
Do you want to continue [Y/n]? 
Get:1 http://ftp.de.debian.org/debian/ squeeze/main netselect amd64 0.3.ds1-14+squeeze1 [25.8 kB]
Get:2 http://ftp.de.debian.org/debian/ squeeze/main netselect-apt all 0.3.ds1-14+squeeze1 [12.0 kB]
Fetched 37.9 kB in 0s (201 kB/s)         
Preconfiguring packages ...
Selecting previously deselected package netselect.
(Reading database ... 106538 files and directories currently installed.)
Unpacking netselect (from .../netselect_0.3.ds1-14+squeeze1_amd64.deb) ...
Selecting previously deselected package netselect-apt.
Unpacking netselect-apt (from .../netselect-apt_0.3.ds1-14+squeeze1_all.deb) ...
Processing triggers for man-db ...
Setting up netselect (0.3.ds1-14+squeeze1) ...
Setting up netselect-apt (0.3.ds1-14+squeeze1) ...

You can then start it with the distribution (e.g. stable, testing, unstable, lenny, squeeze…) as parameter:

# netselect-apt stable
Using distribution stable.
Retrieving the list of mirrors from www.debian.org...

--2012-12-25 21:20:46--  http://www.debian.org/mirror/mirrors_full
Resolving www.debian.org... 86.59.118.148, 2001:858:2:2:214:22ff:fe0d:7717
Connecting to www.debian.org|86.59.118.148|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 337765 (330K) [text/html]
Saving to: “/tmp/netselect-apt.UJfxkZ”

100%[==================================================================================================================================================================>;] 337,765     1.68M/s   in 0.2s    

2012-12-25 21:20:46 (1.68 MB/s) - “/tmp/netselect-apt.UJfxkZ” saved [337765/337765]

Choosing a main Debian mirror using netselect.
netselect: 389 (23 active) nameserver request(s)...       
Duplicate address 212.36.0.70 (http://ftp.bg.debian.org/debian/, http://debian.spnet.net/debian/); keeping only under first name.
...
Duplicate address 128.61.240.89 (http://128.61.240.89/debian/, http://debian.gtisc.gatech.edu/debian/); keeping only under first name.
Running netselect to choose 1 out of 379 addresses.      
..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
The fastest server seems to be:
        http://debian.cruisix.net/debian/

Writing sources.list.
Done.

It is much faster than apt-spy and produces a sources.list file in the current directory:

# cat ./sources.list 
# Debian packages for stable
deb http://debian.cruisix.net/debian/ stable main contrib
# Uncomment the deb-src line if you want 'apt-get source'
# to work with most packages.
# deb-src http://debian.cruisix.net/debian/ stable main contrib

# Security updates for stable
deb http://security.debian.org/ stable/updates main contrib

netselect-apt is faster than apt-spy but supports less option and it is e.g. not so easy to select a region or country.

netselect-apt is also faster because it doesn’t actually benchmark the mirrors but just pings them. So the speed measurements made by netselect-apt are not as accurate as the one done by apt-spy.

Another difference is that netselect-apt automatically gets the latest list of archive mirror. With apt-spy you’d need to first run it with the update option.

Debian: using apt-spy to find the fastest archive mirror

apt-spy is a tool which tests the bandwidth of Debian archive mirrors and generates a file in /etc/apt/sources.list.d with the source list for the fastest mirror.

The most useful options for apt-spy are:

  • -d to define which distribution to use. You can use stable, testing or unstable. You can alternatively also use a distribution name e.g. squeeze. You need to set a value to this parameter.
  • -a to define the area to check. If not set, apt-spy will check all known mirrors world-wide. Use one of these values: Africa, Asia, Europe, North-America, Oceania or South-America.

To check all mirrors worldwide and generate a sources.list file for stable packages:

# apt-spy -d stable

Of course, if apt-spy is not installed yet, you’ll need to install it first:

# apt-spy -d stable
-bash: apt-spy: command not found
# apt-get install apt-spy
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
apt-spy
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 33.4 kB of archives.
After this operation, 209 kB of additional disk space will be used.
Get:1 http://ftp.de.debian.org/debian/ squeeze/main apt-spy amd64 3.1-19 [33.4 kB]
Fetched 33.4 kB in 0s (190 kB/s)
Selecting previously deselected package apt-spy.
(Reading database ... 106526 files and directories currently installed.)
Unpacking apt-spy (from .../apt-spy_3.1-19_amd64.deb) ...
Processing triggers for man-db ...
Setting up apt-spy (3.1-19) ...

Without using the -a parameter, checking the bandwidth of all mirrors will take some time.

Eventually it will find some pretty fast servers:

SERVER: ftp.at.debian.org
Benchmarking FTP...
Downloaded 7100595 bytes in 2.10 seconds
Download speed: 3302.32 kB/sec

Some very slow servers:

SERVER: debian.indika.net.id
Benchmarking HTTP...
Downloaded 67776 bytes in 20.24 seconds
Download speed: 3.27 kB/sec

Some extremely fast servers:

SERVER: ftp.sk.debian.org
Benchmarking FTP...
Downloaded 7100595 bytes in 1.27 seconds
Download speed: 5442.81 kB/sec

And some servers with issues:

...

SERVER: mirror.publicns.net
Benchmarking FTP...
Error: Couldn't resolve host 'mirror.publicns.net'

...

SERVER: ftp.linuxforum.net
Benchmarking FTP...
Error: connect() timed out!

...

SERVER: ftp.it.debian.org
Benchmarking FTP...
Error: couldn't connect to host

...

SERVER: debian.uni.edu.ni
Benchmarking HTTP...
Error: name lookup timed out

...

SERVER: ftp.3logic.net
Benchmarking FTP...
Error: Server denied you to change to the given directory

...

When it is done it will write a sources.list file:

...

Writing new sources.list file: /etc/apt/sources.list.d/apt-spy.list 
# cat /etc/apt/sources.list.d/apt-spy.list 
# sources.list generated by apt-spy v3.1
#
# Generated using:
#
# apt-spy \
# 	-d stable
#
deb ftp://ftp.sk.debian.org/debian/ stable main #contrib non-free
deb-src ftp://ftp.sk.debian.org/debian/ stable main #contrib non-free
deb http://security.debian.org/ stable/updates main

 

Next time you run apt-get update you’ll see that the new mirror is used.

GitHub: Cloning your own project

Today I released the first version of MySQLBuddy. It is basically a fork of SybaseBuddy where I’ve removed a few things (for Sybase) and made a few changes for MySQL. So I wanted to fork SybaseBuddy in GitHub to a new project. Unfortunately it looks like GitHub doesn’t support this. It seems to assume that the owner of a project will not fork his own project.

Here is what I did to fork the project:

I first created a new GitHub repository: MySQLBuddy.
Then I created a clone of SybaseBuddy on my machine:

$ git clone https://github.com/benohead/SybaseBuddy.git MySQLBuddy
Cloning into 'MySQLBuddy'...
remote: Counting objects: 105, done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 105 (delta 33), reused 74 (delta 7)
Receiving objects: 100% (105/105), 6.48 MiB | 1.38 MiB/s, done.
Resolving deltas: 100% (33/33), done.

Now I had a clone but needed to teach it that it’s not linked to the SybaseBuddy GitHub repository anymore but to the MySQLBuddy repository. For this you need to update the .git/config in your clone:

Under Linux:

$ sed -i 's/SybaseBuddy/MySQLBuddy/g' .git/config

Under Mac OS X:

$ sed -i .tmp 's/SybaseBuddy/MySQLBuddy/g' .git/config

Or simply edit the file and update the URL to the repository.

Now you need to push this clone to the new repository:

$ git push -u origin master

You might get the following error message:

$ git push -u origin master
To https://github.com/benohead/MySQLBuddy.git
 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'https://github.com/benohead/MySQLBuddy.git'
To prevent you from losing history, non-fast-forward updates were rejected
Merge the remote changes (e.g. 'git pull') before pushing again.  See the
'Note about fast-forwards' section of 'git push --help' for details.

What you need to do is first pull from the repository before pushing:

$ git pull

You might also get there the following error message:

$ git pull
warning: no common commits
remote: Counting objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From https://github.com/benohead/MySQLBuddy
 + c1c2949...820a952 master     -> origin/master  (forced update)
Auto-merging README.md
CONFLICT (add/add): Merge conflict in README.md
Automatic merge failed; fix conflicts and then commit the result.

So you’ll need to handle the merge manually. Just update the file (here README.md) and pull again.

For some reason, I did the following before modifying the file:

$ git pull
U	README.md
Pull is not possible because you have unmerged files.
Please, fix them up in the work tree, and then use 'git add/rm <file>'
as appropriate to mark resolution, or use 'git commit -a'.
$ git commit -a
[master 00dd552] Merge branch 'master' of https://github.com/benohead/MySQLBuddy

But you don’t need to do it like this and can directly modify the file and pull.

After that you’ll be able to push:

$ git push -u origin master
To https://github.com/benohead/MySQLBuddy.git
   820a952..00dd552  master -> master
Branch master set up to track remote branch master from origin.

And you’re done !