The SFTP Gateway is a proxy server that provides a secure and convenient way to upload and download files from S3 buckets over the SFTP and SCP protocols. Manage access through IAM users and authenticate with the SFTP Gateway using IAM user credentials. No separate user management is necessary.
This implementation is unique because it does not buffer files on the local EBS volume. Instead, files are streamed directly to and from S3. This eliminates disk IO as a potential bottleneck for throughput and allows terabyte-sized files to be transferred efficiently.
Refer to the resources listed below for further documentation, detailed instructions and FAQs.
ENCRYPTION AT REST All files stored on S3 are encrypted at rest using Server Side Encryption (SSE).
FULLY INTEGRATED INTO IAM Authenticate with IAM user credentials. Access logs are written to CloudWatch Logs.
HIGHLY SCALABLE Streaming, non-blocking data transfer ensures high throughput and low latency.
This solution is best deployed through CloudFormation templates. CloudFormation is an Infrastructure as Code (IaC) service provided by AWS which makes the set up of complex cloud infrastructures fast and reliable.
We are offering two templates. We highly recommend to get started with the Basic Template. This template will setup a turn-key deployment in a new VPC with minimal configuration options while providing high availability with two availability zones. The Advanced Template is the best choice if you need to launch the SFTP Gateway in an existing VPC with more control over instance placement.
Please follow the instructions in the Marketplace product page to subscribe to the product. Once subscribed you can launch the stack by following the Launch Stack links.
This template creates a VPC with two subnets in two availability zones, including Internet Gateway and corresponding routing tables. A Network Load Balancer will distribute traffic between the instances in an Auto Scaling group. The whole deployment is accessible over the internet through the load balancer's public DNS endpoint which you can find in the Outputs section of the CloudFormation stack.
Deploy this CloudFormation template if you need to deploy the SFTP Gateway in an existing VPC. The template provides parameters for the VPC and subnets where the instances and load balancer should be launched into. Additionally, the load balancer can be configured to be internal, which means that connections are only accepted from within the VPC. Please ensure that instances in the subnet have internet access.
The CloudFormation stack takes roughly 5 minutes to create. The Outputs section of the CloudFormation stack will provide you with the hostname of the SFTP load balancer (LoadBalancerHostName). The SFTP Gateway is running on port 22 and can be accessed through any SFTP or SCP client (see Recommended Clients).
Authentication is managed through AWS IAM user accounts. The SFTP Gateway will accept an Access Key ID as user name and the Secret Access Key as password.
The server will then assume the identity of the access key's owner. The IAM user requires the necessary policies that allows them to list buckets, read bucket locations, list objects and access objects. See below for a least-privileged IAM policy or apply the AmazonS3FullAccess AWS managed policy.
To connect to the linux shell for administrative purposes use SSH on port 22 of the EC2 instance and the username ec2-user.
For high availability, we recommend launching at least two servers in separate availability zones. The Elastic Load Balancer (ELB) continuously checks the health of all instances. Instances that fail their health check will automatically be replaced with new instances and the failing instances are terminated.
Updates to the software are deployed using the rolling update policy. Instances will be replaced one-by-one without any interruption to the service.
This service listens on port 22 of the Network Load Balancer for connections which forwards the connection to port 2222 on the EC2 instance. No additional ports need to be opened. The SSH host certificate is unique to the instance. The SFTP Gateway only supports the subset of SSH commands that are required for SFTP (over SSH) and SCP connections. A full SSH shell cannot be opened. The service runs under an unprivileged user s3gw with no write access to the local file system.
The OpenSSH server is accessible on port 22 on the EC2 instance for administrative access. The username is ec2-user.
All files are uploaded with AES256 server-side encryption enabled.
The application will write an Audit Log of users who have logged on and transferred files to AWS CloudWatch Logs. The audit log contains information about the session start date, end date, instance id, IP and protocol of the session. The CloudWatch Logs group is called /netcubed/s3gw.
The server can be patched manually or by using the AWS Systems Manager (preferred). Follow the guide at http://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-patch-consolewalk.html to enable automated patching of the server through SSM. The SSM service is preinstalled and enabled. The required IAM policy is already attached to the instance.
To manually patch the system, log in via SSH. Execute sudo yum update -y to upgrade packages and the operating system. Reboot the instance to ensure that all patches are applied.
An active connection can consume up to 10 megabytes of memory when up- or downloading a file from S3, regardless of the size of the object that is being transferred. This is achieved by using multi-part up- and downloads. The limiting factor in terms of transfer speed will therefore be the network bandwidth of the instance.
The SFTP Gateway supports multi-core environments and therefore fully leverages instance types that provide more than one core. The server is implemented using a non-blocking event loop. Therefore, the server can handle many concurrent connections.
The EC2 instances are located behind a Network Load Balancer which provides high throughput and low latency. The Auto Scaling Group is configured to replace instances if health checks to the instance fail.
The recommended instance type for small production environments is m4.large. T2 instances should only be used for testing. They tend to run out of CPU credits if they are heavily utilized and will grind to a halt. The server will become unresponsive and you will need to reboot the instance or upgrade to larger instance type.
We have tested SFTP Gateway successfully with the following clients. Since we are fully compliant with the SCP and SFTP (over SSH) standards, we expect other clients to work as well. Please contact us if you are having trouble to connect with a client that is not listed here.
Yes, you can but with caveats. Buckets in regions other than the one where the SFTP Gateway was launched are accessible. However, please be reminded that this will incur cross-region file-transfer charges. Due to the higher latency and reduced bandwidth between the SFTP Gateway and the S3 endpoint expect worse performance. It is highly recommended to only access S3 buckets in the same region as the SFTP Gateway.
Yes, you can. You can simply cd into a bucket that you have read access to, even though it doesn't show in the root directory listing. For example, try to cd into the cloudformation-examples bucket. In graphical SFTP clients you should be able to set a path that the client will change directory into.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:List*"
],
"Resource": "*"
}]
}
No, local user accounts cannot be used for authentication. The SFTP Gateway uses the credentials it is passed at sign in to authenticate with AWS.
To limit a user to one or more buckets, the IAM policy attached to the user to the user has to explicitly allow access to certain buckets only. The following example provides the user access to example-bucket.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:List*"
],
"Resource": ["arn:aws:s3:::example-bucket", "arn:aws:s3:::example-bucket/*"]
}, {
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "*"
}]
}
These network diagrams give an idea on how the SFTP Gateway can be deployed.
A common scenario where you want to grant access to your S3 buckets to an external entity. The SFTP Gateway can be launched in a public subnet and is therefore also accessible to clients that connect from the internet.
This topology especially useful in hybrid networks where on-premises applications need to work with objects on S3 but are not allowed to connect to S3's public endpoints. In that case, the on-premises application will be able to connect to the SFTP Gateway in the securely connected VPC (via VPN or Direct Connect) which will relay the request to S3.
The server will not attempt to delete or rename buckets if instructed to do so. Instead, an error is returned to the client.
File attributes, such as file modes, cannot be changed since S3 does not support them. Similarly, symbolic links are not supported.
S3 is not a real file system and therefore some operations cannot be implemented efficiently. Renaming a directory on a regular file system is a single system call. Renaming a directory on S3 requires all containing files to be moved to the new directory which is a slow, and potentially costly operation (since you will be charged for each COPY command).
Currently, only authentication through AWS IAM credentials is available. Please contact support@netcubed.de if you are interested in public key authentication.
For paid support, email sales@netcubed.de for further information. Free support is provided via support@netcubed.de.
For free support, we do not provide a guaranteed response time. However, we do our best to respond to questions within 24 hours Monday through Friday.
v1.3.2
v1.3.1
v1.3.0
v1.2.0
ListAllMyBuckets permissionsv1.1.0
v1.0.0