Downloading files and directories via SFTP using SSH.Net

SSH.NET is a .NET library implementing the SSH2 client protocol. It is inspired by a port of the Java library JSch called Sharp.SSH. It allows you to execute SSH commands and also provides both SCP and SFTP functionality.

In this article, I’ll show you how to download a complete directory tree using SSH.NET.

First you’ll need to add a few usings:

using System;
using System.IO;
using Renci.SshNet;
using Renci.SshNet.Common;
using Renci.SshNet.Sftp;

SSH.NET can be added to your project using NuGet.
In order to work with SFTP, you’ll need to get an instance of SftpClient. You can either directly give details like host, port, username and password or you can provide a ConnectionInfo object. In this example we’ll use a KeyboardInteractiveConnectionInfo. This is required if the server expects an interactive keyboard authentication providing the password. The alternatives are PasswordConnectionInfo and PrivateKeyConnectionInfo.

First we’ll create the ConnectionInfo object:

var connectionInfo = new KeyboardInteractiveConnectionInfo(Host, Port, Username);

Host, Port and Username are constants I’ve defined before.

Then we need to define a delegate which will get the prompts returned by the server and will send the password when requested:

connectionInfo.AuthenticationPrompt += delegate(object sender, AuthenticationPromptEventArgs e)
{
	foreach (var prompt in e.Prompts)
	{
		if (prompt.Request.Equals("Password: ", StringComparison.InvariantCultureIgnoreCase))
		{
			prompt.Response = Password;
		}
	}
};

It waits for the prompt “Password: ” and sends the password I’ve defined in a constant called Password.

Then using this SFTP client, we’ll connect to the server and download the contents of the directory recursively:

using (var client = new SftpClient(connectionInfo))
{
	client.Connect();
	DownloadDirectory(client, Source, Destination);
}

Source is the directory you want to download on the remote server and destination is the local directory e.g.:

private const string Source = "/tmp";
private const string Destination = @"c:\temp";

Now we’ll define the DownloadDirectory method. It will get the directory listing and iterate through the entries. Files will be downloaded and for each directory in there, we’ll recursively call the DownloadDirectory method:

private static void DownloadDirectory(SftpClient client, string source, string destination)
{
	var files = client.ListDirectory(source);
	foreach (var file in files)
	{
		if (!file.IsDirectory && !file.IsSymbolicLink)
		{
			DownloadFile(client, file, destination);
		}
		else if (file.IsSymbolicLink)
		{
			Console.WriteLine("Ignoring symbolic link {0}", file.FullName);
		}
		else if (file.Name != "." && file.Name != "..")
		{
			var dir = Directory.CreateDirectory(Path.Combine(destination, file.Name));
			DownloadDirectory(client, file.FullName, dir.FullName);
		}
	}
}

I am ignoring symbolic links because trying to download them just fails and the SftpFile class provides no way to find what this link points to. “.” and “..” are also ignored.

Now let’s see how to download a single file:

private static void DownloadFile(SftpClient client, SftpFile file, string directory)
{
	Console.WriteLine("Downloading {0}", file.FullName);
	using (Stream fileStream = File.OpenWrite(Path.Combine(directory, file.Name)))
	{
		client.DownloadFile(file.FullName, fileStream);
	}
}

It’s pretty easy: you create a file stream to the destination file and use the DownloadFile method of the SFTP client to download the file.

That’s it ! Here the full code for your convenience:

using System;
using System.IO;
using Renci.SshNet;
using Renci.SshNet.Common;
using Renci.SshNet.Sftp;

namespace ConsoleApplication1
{
    public static class SftpTest
    {
        private const string Host = "192.168.xxx.xxx";
        private const int Port = 22;
        private const string Username = "root";
        private const string Password = "xxxxxxxx";
        private const string Source = "/tmp";
        private const string Destination = @"c:\temp";

        public static void Main()
        {
            var connectionInfo = new KeyboardInteractiveConnectionInfo(Host, Port, Username);

            connectionInfo.AuthenticationPrompt += delegate(object sender, AuthenticationPromptEventArgs e)
            {
                foreach (var prompt in e.Prompts)
                {
                    if (prompt.Request.Equals("Password: ", StringComparison.InvariantCultureIgnoreCase))
                    {
                        prompt.Response = Password;
                    }
                }
            };

            using (var client = new SftpClient(connectionInfo))
            {
                client.Connect();
                DownloadDirectory(client, Source, Destination);
            }
        }

        private static void DownloadDirectory(SftpClient client, string source, string destination)
        {
            var files = client.ListDirectory(source);
            foreach (var file in files)
            {
                if (!file.IsDirectory && !file.IsSymbolicLink)
                {
                    DownloadFile(client, file, destination);
                }
                else if (file.IsSymbolicLink)
                {
                    Console.WriteLine("Ignoring symbolic link {0}", file.FullName);
                }
                else if (file.Name != "." && file.Name != "..")
                {
                    var dir = Directory.CreateDirectory(Path.Combine(destination, file.Name));
                    DownloadDirectory(client, file.FullName, dir.FullName);
                }
            }
        }

        private static void DownloadFile(SftpClient client, SftpFile file, string directory)
        {
            Console.WriteLine("Downloading {0}", file.FullName);
            using (Stream fileStream = File.OpenWrite(Path.Combine(directory, file.Name)))
            {
                client.DownloadFile(file.FullName, fileStream);
            }
        }
    }
}

So except for the issue with the symbolic link, it works pretty good and is also quite fast.

 

Debian: use SSHFS to mount a remote directory

In a previous post, I’ve shown how to use SSHFS to mount a remote directory as a volume in Mac OS X. Of course the same can be done to connect two Linux boxes.

Here’s a short description how to do it on Debian.

First you’ll need to install sshfs using apt-get:

apt-get install sshfs

Note that this will also install fuse as a dependency.

Then you need to have a public key authentication in place. I’m assuming that ssh is already installed on the server. You now should do the following, on the Linux box where the file system should be mounted:

apt-get install openssh-client

Now you can generate a key:

mkdir ~/.ssh
chmod 700 ~/.ssh
cd ~/.ssh
ssh-keygen -t rsa -C "henri.benoit@gmail.com"

Of course replace my email address by yours. You can then print the key:

cat id_rsa.pub

This should be added to the file ~/.ssh/authorized_keys on the server.

You also need to make sure that PubkeyAuthentication is set to yes in /etc/ssh/sshd_config so that public key authentication is supported at all.

Now you can connect using ssh without password and can mount over ssh:

mkdir ~/amazingweb
sshfs -p 22 USER@xxx.xxx.xxx.xxx:/var/www/vhosts ~/amazingweb -oauto_cache,reconnect

Replace USER by the user for which you’ve setup the public key authentication.
Replace xxx.xxx.xxx.xxx by the IP address of the remote server.
Replace ~/amazingweb by the path where you want to mount the remote file system.

It’s basically the same command as under Mac OS X except that you have to remove the 3 following options which are not available under Debian:

  • defer_permissions
  • noappledouble
  • volname

Mac OS X: use SSHFS to mount a remote directory as a volume

While working for amazingweb, I often need to edit existing files. This can be done with the web-based management UI but it’s so much nicer to be able to use the same editors and tools you’re used to on your local machine.

In order to be able to do this, you need either to have your editors/tools support SFTP and get and put files on the fly or be able to mount the remote file system as a volume on your local machine. Even if you find some solutions for a few of your editors (e.g. the Sublime Text editor with the SFTP plugin) you’ll still not be able to do everything as if it was all local because at least one tool doesn’t support SFTP.

So the more generic solution is to mount the remote file system. Of course since your remote server is remote and needs to be secured, you can only access it using SSH. Fortunately, SSH provides many extensions, some of which are used by pretty much everybody (e.g. SFTP) and others which are less known.

One of these extensions is SSHFS (Secure SHell FileSystem). It implements a file system and can be used on the Linux operating system and other platforms where FUSE is ready. FUSE (Filesystem in Userspace) is a kernel module for Unix systems, which enables file system drivers to shift from kernel mode to user mode. It thus allows non-privileged users to mount their own file systems.

In the past MacFUSE used to be the most prominent FUSE implementation for Max OS X. MacFUSE is not maintained anymore and has been replaced by “FUSE for OS X” (OSXFUSE). An alternative is Fuse4X which is a fork off MacFUSE but unlike MacFUSE it is fully compatible with FUSE.

I’ve used Fuse4X and it worked fine so I haven’t tried OSXFUSE.

You have three ways install Fuse4X:

  1. Download it from here and install it manually.
  2. Install it using Macport:
    sudo port install fuse4x
  3. Install it using Homebrew:
    brew install fuse4x

I used Homebrew and got the following errors:

Warning: Could not link fuse4x. Unlinking...
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
You can try again using `brew link fuse4x'
==> Summary
/usr/local/Cellar/fuse4x/0.9.2: 18 files, 720K, built in 34 seconds
Error: You must `brew link fuse4x' before sshfs can be installed

Unfortunately it doesn’t tell you what’s the actual problem… So I just did what I was told to:

$ brew link fuse4x
Linking /usr/local/Cellar/fuse4x/0.9.2... Warning: Could not link fuse4x. Unlinking...

Error: Could not symlink file: /usr/local/Cellar/fuse4x/0.9.2/lib/pkgconfig/fuse.pc
Target /usr/local/lib/pkgconfig/fuse.pc already exists. You may need to delete it.
To force the link and delete this file, do:
  brew link --overwrite formula_name

To list all files that would be deleted:
  brew link --overwrite --dry-run formula_name

Ok, so let’s do it with --overwrite:

$ brew link --overwrite fuse4x
Linking /usr/local/Cellar/fuse4x/0.9.2... 7 symlinks created

It’s installed !

Ok, now that you have installed Fuse4X using one of the three methods above, you’ll need to install sshfs.

With Homebrew, it is pretty easy:

$ brew install sshfs
==> Installing sshfs dependency: xz
==> Downloading http://tukaani.org/xz/xz-5.0.4.tar.bz2
######################################################################## 100.0%
==> ./configure --prefix=/usr/local/Cellar/xz/5.0.4
==> make install
/usr/local/Cellar/xz/5.0.4: 58 files, 1.5M, built in 41 seconds
==> Installing sshfs dependency: libffi
==> Downloading http://mirrors.kernel.org/sources.redhat.com/libffi/libffi-3.0.11.tar.gz
######################################################################## 100.0%
==> ./configure --prefix=/usr/local/Cellar/libffi/3.0.11
==> make install
==> Caveats
This formula is keg-only: so it was not symlinked into /usr/local.

Mac OS X already provides this software and installing another version in
parallel can cause all kinds of trouble.

Some formulae require a newer version of libffi.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you'll need to add to your
build variables:

    LDFLAGS:  -L/usr/local/opt/libffi/lib

==> Summary
/usr/local/Cellar/libffi/3.0.11: 13 files, 312K, built in 19 seconds
==> Installing sshfs dependency: glib
==> Downloading http://ftp.gnome.org/pub/gnome/sources/glib/2.34/glib-2.34.2.tar.xz
######################################################################## 100.0%
==> Downloading patches
######################################################################## 100.0%
######################################################################## 100.0%
########################################################                  77.9%
==> Patching
patching file glib/gunicollate.c
patching file aclocal.m4
patching file config.h.in
patching file configure
patching file configure.ac
patching file gio/gdbusprivate.c
patching file gio/xdgmime/xdgmime.c
patching file gio/gsocket.c
patching file gio/tests/socket.c
==> ./configure --disable-maintainer-mode --disable-dtrace --prefix=/usr/local/Cellar/glib/2.34.2 --localstatedir=/usr/local/var
==> make
==> make install
/usr/local/Cellar/glib/2.34.2: 407 files, 15M, built in 4.5 minutes
==> Installing sshfs
==> Downloading https://github.com/fuse4x/sshfs/tarball/sshfs_2_4_0
######################################################################## 100.0%
==> autoreconf --force --install
==> ./configure --prefix=/usr/local/Cellar/sshfs/2.4.0
==> make install
==> Caveats
Make sure to follow the directions given by `brew info fuse4x-kext`
before trying to use a FUSE-based filesystem.
==> Summary
/usr/local/Cellar/sshfs/2.4.0: 8 files, 132K, built in 12 seconds

Also check the following instruction before continuing:

$ brew info fuse4x-kext
fuse4x-kext: stable 0.9.2 (bottled)
http://fuse4x.github.com
/usr/local/Cellar/fuse4x-kext/0.9.2 (6 files, 284K) *
https://github.com/mxcl/homebrew/commits/master/Library/Formula/fuse4x-kext.rb
==> Caveats
In order for FUSE-based filesystems to work, the fuse4x kernel extension
must be installed by the root user:

  sudo /bin/cp -rfX /usr/local/Cellar/fuse4x-kext/0.9.2/Library/Extensions/fuse4x.kext /Library/Extensions
  sudo chmod +s /Library/Extensions/fuse4x.kext/Support/load_fuse4x

If upgrading from a previous version of Fuse4x, the old kernel extension
will need to be unloaded before performing the steps listed above. First,
check that no FUSE-based filesystems are running:

mount -t fuse4x

Unmount all FUSE filesystems and then unload the kernel extension:

  sudo kextunload -b org.fuse4x.kext.fuse4x

Now, you have both Fuse4X and SSHFS installed, so you can mount a remote directory as a volume using:

$ mkdir ~/amazingweb
$ sshfs -p 22 root@antagus2:/var/www/vhosts ~/amazingweb -oauto_cache,reconnect,defer_permissions,noappledouble,volname=amazingweb

Note that this works without needing any password because I use public-key cryptography to authenticate.

You only need to create the directory using mkdir once.

The first parameter (-p 22) means that port 22 should be used (the standard SSH port). The second parameter is the username, hostname and path to the remote filesystem to be mounted. The third parameter is the local path. The forth parameter is the list of options used:

  • auto_cache: enable caching based on modification times
  • reconnect: reconnect to server
  • defer_permissions: certain shares may mount properly but cause permissions denied errors when accessed (an issue caused by the way permissions are translated and interpreted by the Mac OS X Finder). This option works around this problem
  • noappledouble: to prevent Mac OS X to write .DS_Store files on the remote file system
  • volname: the volume name to be used

You can now access it just like a normal local folder. Of course you have to remount it after restarting the computer.

You can also automate the mounting of the remote folder using an application. To create one:

  • Open the AppleScript Editor
  • Create a script like the one which follows
  • Save it as an application
  • You can then start the application to mount your folder instead of going to the terminal

Here the script I use:

set homePath to POSIX path of (path to home folder)
set localPath to homePath & "amazingweb"
set remoteLogin to "root"
set remoteHost to "antagus2"
set remotePath to "/var/www/vhosts"
set volumeName to "amazingweb"

set message to "Unmounting SSHFS volume " & localPath
log message
UnmountSSHFSVolumne(localPath)
set message to "Mounting SSHFS volume " & localPath
log message
MountSSHFSVolumne(localPath, remoteLogin & "@" & remoteHost & ":" & remotePath, volumeName)

on UnmountSSHFSVolumne(aLocalPath)
	tell application "Finder"
		try
			if exists aLocalPath as POSIX file then
				set mountedVolumes to every paragraph of (do shell script "mount | awk -F' on ' '{print $2}' | awk -F' \\\\(' '{print $1}'")
				if aLocalPath is in mountedVolumes then
					do shell script "/sbin/umount " & aLocalPath
					set message to "Unmounted SSHFS volume " & aLocalPath
					log message
					display dialog message buttons {"OK"}
				end if
			end if
		on error errStr number errorNumber
			display dialog "Error: " & errStr buttons {"OK"}
		end try
	end tell
end UnmountSSHFSVolumne

on MountSSHFSVolumne(aLocalPath, aRemotePath, aVolumeName)
	tell application "Finder"
		try
			if exists aLocalPath as POSIX file then
			else
				do shell script "mkdir " & aLocalPath
			end if
			do shell script "/usr/local/bin/sshfs -p 22 " & aRemotePath & " " & aLocalPath & " -oauto_cache,reconnect,defer_permissions,noappledouble,volname=" & aVolumeName
			set message to "Mounted SSHFS volume " & aLocalPath
			log message
			display dialog message buttons {"OK"}
		on error errStr number errorNumber
			display dialog "Error: " & errStr buttons {"OK"}
		end try
	end tell
end MountSSHFSVolumne

Update: An alternative is to use ExpanDrive it makes it easy to setup a drive and reconnect automaticallyand allows you to use the drive anywhere. It’s technically a whole different beast since it’s actually using SFTP and showing the remote filesystem as a drive, but you won’t really feel the difference. The main reason why I do not use it is that it doesn’t work with authentication key instead of password: my Server doesn’t not allow authentication with password (only with registered keys). Theyhave a 7-days free trial period, so you can just give it a try.

Update: Fuse4x is not maintained anymore and has merged with osxfuse. So now, you should use osxfuse instead of Fuse4x. Osxfuse can also be installed with Homebrew. The package is not yet available in the repository so the following will fail:

$ brew install osxfuse
Error: No available formula for osxfuse

You have to do the following to install it:

$ brew install https://raw.github.com/bfleischer/homebrew/osxfuse/Library/Formula/osxfuse.rb

After that execute the following:

$ sudo /bin/cp -RfX /usr/local/Cellar/osxfuse/2.6.1/Library/Filesystems/osxfusefs.fs /Library/Filesystems
$ sudo chmod +s /Library/Filesystems/osxfusefs.fs/Support/load_osxfusefs

If this fails saying that the file does not exist, it means that building osxfuse failed and the resulting file is missing. To check what happened, execute the following:

$ cd /Library/Caches/Homebrew/osxfuse--git
$ ./build.sh -t homebrew -f /usr/local/Cellar/osxfuse/2.6.1

I got the following messgage:

/usr/bin/xcodebuild: line 2: -version: command not found
OSXFUSEBuildTool() : skip unsupported Xcode version in ‘/Applications/Xcode.app/Contents/Developer’.
OSXFUSEBuildTool() failed: no supported version of Xcode found.

It’s due to a trick I used some time ago to build gem native extensions without XCode… All you have to do is restore the original xcrun file, remove the already partially installed osxfuse and reinstall it:

sudo mv /usr/bin/xcrun /usr/bin/xcrun.sav2
sudo mv /usr/bin/xcrun.sav /usr/bin/xcrun
brew remove osxfuse
brew install --reinstall https://raw.github.com/bfleischer/homebrew/osxfuse/Library/Formula/osxfuse.rb