File Transport

• Introduction
• Transport Configuration
  • File Transport Listener (VFS Listener)
  • File Transport Sender (VFS Sender)
    • File Locking
    • FTP Passive Mode
    • Retrying on Error
    • Using Temporary Files
    • Appending to Files
    • Out-only Message Exchange Pattern
• Using SFTP
• Known Issues

Introduction

The file transport, also known as the VFS (Virtual File System) transport, can be used to read, mediate and write file content using Synapse. This transport allows Synapse to interface with the local file system and remote file systems via file transfer protocols such as FTP.

The file transport is based on Apache Commons VFS, and supports all the file transfer protocols supported by Commons VFS. This includes interactions with the local file system, HTTP, HTTPS, FTP and SFTP (i.e. file transfer over SSH).

There is a fundamental difference between the file transport and transports such as HTTP, and it is important to understand this difference to be able to use the file transport correctly. The HTTP transport binds to a single protocol endpoint, i.e. a TCP port on which it accepts incoming HTTP requests. These requests are then dispatched to the appropriate service based on the request URI. On the other hand, the file transport only receives the payload of a message (i.e. the file), but no additional information that could be used to dispatch the message to a service. This means that file system locations must be explicitly mapped to services. This is done using a set of service parameters. For Synapse this means that the VFS transport listener can only be used in conjunction with proxy services. The relevant service parameters are specified in the proxy service configuration as follows:

In the above example the file system location file:///var/spool/synapse/in is explicitly bound to MyVFSService. Any file dropped in that location will be pre-dispatched to MyVFSService, bypassing any other configured dispatch mechanisms that would normally apply to messages received via HTTP.

[Back to top]

Transport Configuration

The file transport consists of a transport listener component and a transport sender component. Proxy services can read files using the file transport listener, and they can write file content using the file transport sender. Following sections describe how to configure these two components of the transport.

Using SFTP

To avoid man-in-the-middle attacks, SSH clients will only connect to hosts with a known host key. When connecting for the first time to an SSH server, a typical command line SSH client would request confirmation from the user to add the server and its fingerprint to the list of known hosts.

The VFS transports supports SFTP through the JSch library and this library also requires a list of known hosts. Since Synapse is not an interactive process, it can't request confirmation from the user and is therefore unable to automatically add a host to the list. This implies that the list of known hosts must be set up manually before the transport can connect.

JSch loads the list of known hosts from a file called known_hosts in the .ssh sub-directory of the user's home directory, i.e. $HOME/.ssh in Unix and %HOMEPATH%\.ssh in Windows. The location and format of this file are compatible with the OpenSSH client.

Since the file not only contains a list of host names but also the fingerprints of their host keys, the easiest way to add a new host to that file is to simply use the OpenSSH client to open an SSH session on the target host. The client will then ask to add the credentials to the known_hosts file. Note that if the SSH server is configured to only allow SFTP sessions, but no interactive sessions, the connection will actually fail. Since this doesn't rollback the change to the known_hosts file, this error can be ignored.

Known issues

The VFS listener will start reading a file as soon as it appears in the configured location. To avoid processing half written files, the creation of these files should be made atomic. On most platforms this can be achieved by writing the data to a temporary file and then moving the file to the target location. Note however that a move operation is only atomic if the source and destination are on the same physical file system. The location for the temporary file should be chosen with that constraint in mind.

It should also be noted that the VFS transport sender doesn't create files atomically. Use the transport.vfs.UseTempFile endpoint parameter to get around this issue.