The Amazon Elastic File System is one of the most affordable storage options available. It's seamless integration with EC2 and ECS using the NFS protocol makes it a perfect match for analytics, machine learning, stateful web applications, backup storage and more. With the SFTP Gateway for Amazon EFS there is now an easy way to provide secure and scalable access to your Elastic File System for end users or applications that cannot directly connect to the EFS endpoints.
The SFTP Gateway is a proxy server that provides a secure and convenient way to browse, upload and download files from Amazon EFS via the popular SFTP file transfer protocol. The gateway is able to handle thousands of concurrent users and connections on a single server due to its unique implementation.
Unlike other implementations, the SFTP Gateway for Amazon EFS has been engineered from the ground up for speed and efficiency. The SFTP server is a custom implementation that directly communicates with the EFS endpoints over a TLS encrypted TCP connection. SFTP commands are directly translated into NFS remote procedure calls. This unique design makes it possible to support thousands of concurrent user sessions. Each session can be a different user with their own distinct access permissions.
Public key and password authentication are supported. User accounts are managed centrally in the AWS Systems Manager Parameter Store, including UID, GID and root directory mappings. Refer to the resources listed below for further documentation, detailed instructions and FAQs.
Figure 1: Network topology of the deployment
HIGHLY SCALABLE Streaming, non-blocking data transfer ensures high throughput and low latency. CONFIGURABLE Supports thousands of users, each can have their own UID, GID and root path.
We recommend to deploy the product using the provided CloudFormation template. This deployment option provides high availability by setting up an auto-scaling group and a network load balancer with corresponding health checks to replace failed instances. The auto-scaling group can also be configured to scale capacity automatically to match demand. Even scaling to zero is possible to achieve significant cost savings in times where the service is not needed.
Users are centrally managed in the AWS Systems Manager Parameter Store. The /netcubed/sftpgw/default/user/ namespace is used to lookup access credentials and user-specific settings. The following parameters are supported:
| Parameter Name | Data Type | Comments |
|---|---|---|
password |
SecureString | Password |
publicKeys |
String | Newline-separted list of OpenSSH public keys |
uid |
String | Numeric POSIX user id |
gid |
String | Numeric POSIX group id |
root |
String | Root path that the user will be locked in to |
Either the password or the publicKey parameter or both are required. The other parameters will default to the parameters provided to the CloudFormation template.
We recommend the use of the AWS CLI to create new users. Alternatively, you can also use the AWS Web Console for the parameter store which can be found at https://console.aws.amazon.com/systems-manager/parameters.
To create a new user testuser with the password testpassword, execute the following command:
aws ssm put-parameter --name /netcubed/sftpgw/default/user/testuser/password --type SecureString --value testpassword
To also provide an SSH public key, execute the following. You can also provide multiple public keys. Each public key must be separated by a new-line character.
aws ssm put-parameter --name /netcubed/sftpgw/default/user/testuser/publicKeys --type String --value 'ssh-rsa AAAAB3NzaC1yc2EAAA...'
For high availability, we recommend launching at least two servers in separate availability zones. The Elastic Load Balancer (ELB) continuously checks the health of all instances. Instances that fail their health check will automatically be replaced with new instances and the failing instances are terminated.
Updates to the software are deployed using the rolling update policy. Instances will be replaced one-by-one without any interruption to the service.
This service listens on port 22 of the Network Load Balancer for connections which forwards the connection to port 2222 on the EC2 instance. No additional ports need to be opened. The SSH host certificate is unique to the instance. The SFTP Gateway only supports the subset of SSH commands that are required for SFTP (over SSH) and SCP connections. A full SSH shell cannot be opened. The service runs under an unprivileged user sftpgw with no write access to the local file system.
The OpenSSH server is accessible on port 22 on the EC2 instance for administrative access. The username is ec2-user.
The application will write an Audit Log of users who have logged on and transferred files to AWS CloudWatch Logs. The audit log contains information about the session start date, end date, instance id, IP and protocol of the session. The CloudWatch Logs group is called /netcubed/sftpgw.
The server can be patched manually or by using the AWS Systems Manager (preferred). Follow the guide at https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-patch-working.html to enable automated patching of the server through SSM. The SSM service is preinstalled and enabled. The required IAM policy is already attached to the instance.
To manually patch the system, log in via SSH. Execute sudo yum update -y to upgrade packages and the operating system. Reboot the instance to ensure that all patches are applied.
The following features are currently not supported:
Please keep your eyes on our release notes as we are actively working on adding these features.
We have tested SFTP Gateway successfully with the following clients. Since we are fully compliant with the SFTP (over SSH) standards, we expect other clients to work as well. Please contact us if you are having trouble to connect with a client that is not listed here.
Most likely, the SFTP server is unable to create the root path because the parent directory has not sufficient permissions to allow the operation. Make sure that the parent directory is world-writable and that the user ID of the user is able to create the folder.
Please refer to events in the log group /netcubed/sftpgw for additional details.
You will receive an "Access Denied" error when you do not have the necessary permissions to create the folder. The likely cause are the ownership and permissions on the root path. Make sure that the root path is either owned by the user ID or extend the permissions to be world writable. These changes need to be executed from a server that has the EFS volume mounted. To make the root folder world-writable, execute chmod 0777 /mnt/efs where /mnt/efs is the mount point of the EFS volume.
Please refer to events in the log group /netcubed/sftpgw for additional details.
Yes. Please get in touch at support@netcubed.de. We are currently working with AWS to get the product released on the AWS Marketplace for Containers.
No. All data transferred between the SFTP client and the EFS service is never stored on the SFTP Gateway. All data transferred is buffered in memory and directly written to the EFS endpoint. Swappiness is disabled which guarantees that memory content is never written to disk. Additionally, all processes are launched in a disk read-only container.
Yes. Connections from the SFTP Gateway to the EFS endpoint are protected by TLS v1.2. Connections from the client to the SFTP Gateway is protected by the SSH protocol.
Access Points are a fairly recent addition to the EFS service. They are currently not supported as endpoints for the SFTP Gateway. However, some of the features that Access Points provide can also be achieved with the SFTP Gateway by defining individual User IDs, Group IDs and root paths per user.
For paid support, email sales@netcubed.de for further information. Free support is provided via support@netcubed.de.
For free support, we do not provide a guaranteed response time. However, we do our best to respond to questions within 24 hours Monday through Friday.