So you’re thinking you might DIY your own SFTP solution to Amazon S3?
When faced with needing a secure file transfer solution in the cloud, many organizations consider building their own SFTP implementation. After all, how hard could it be?
You can spin up a server with OpenSSH and be done, right? Unfortunately, this approach leads down a rabbit hole of complexity, cost, and compromise, making a DIY SFTP solution a decision you’ll likely regret.
Let’s talk about some things to consider before you DIY your own SFTP solution.
The Deceptive Simplicity of OpenSSH
The most obvious starting point for a custom SFTP solution is OpenSSH, which seems straightforward enough. However, OpenSSH has a fundamental limitation: it can only write files to the local Linux file system. This constraint immediately creates a cascade of problems that compound as you try to build a production-ready solution.
The Storage Dilemma
With files writing only to the local disk, you’ll face the inevitable problem of unbounded growth. Your disk size will expand without limit, forcing you to over-provision storage to prevent your system from running out of space. However, local EBS volumes aren’t considered durable storage, putting your data at risk of corruption or loss.
The natural next step might be to switch to EFS (Elastic File System) to address durability and size constraints. However, this introduces a new set of challenges. EFS operates as a network disk, making disk I/O a significant bottleneck. The situation worsens when you discover that EFS uses burstable I/O, meaning your performance degrades to an unusable crawl once you exhaust your burst credits.
To maintain consistent performance, you’d need provisioned throughput for EFS. Unfortunately, even modest provisioned throughput speeds come with prohibitively expensive price tags, making this approach economically unfeasible for most use cases.
The High Availability Challenge
In any production environment, high availability becomes a critical requirement. This is where the DIY approach hits another significant roadblock. OpenSSH stores SFTP users in local files like /etc/passwd, but there’s no built-in way to synchronize users between servers in a high-availability setup.
You might consider replacing the user store with LDAP using PAM, which creates complications. The LDAP database can’t run on EFS due to poor I/O performance, but placing it on a local disk means each VM needs its own database. To preserve your database, you’d then need to configure multi-master replication, handle sync conflicts, and ensure at least one VM stays running at all times.
Ideally, you’d want your database managed in RDS, but connecting all these pieces adds layers of complexity that continue to multiply.
The S3 Integration Problem
Storing files in S3 rather than EFS makes much more sense for cloud-based solutions. S3 offers superior durability, significantly lower costs, and seamless integration with other AWS services for post-processing workflows. However, getting files from your local Linux disk to S3 introduces another challenge.
You’d need to implement file event services like incron to handle the transfer, but sync issues inevitably arise where files get stuck on the local filesystem without making it to S3. To achieve actual read/write operations directly to S3 using the SFTP protocol, you’d need to stream data in memory, which OpenSSH cannot do. This means re-implementing the entire SFTP protocol using a custom application.
Management and User Experience
Beyond the technical hurdles lies the practical challenge of user management. Command-line user administration doesn’t scale in production environments. You need a proper user interface to manage SFTP users and settings efficiently, which means building yet another component from scratch.
DIY Your Own SFTP Solution: The Reality Check
While it’s technically feasible to create a basic SFTP solution using OpenSSH and EFS, you’ll end up with a system that lacks high availability, requires CLI-based user management, doesn’t store files in S3, suffers from slow transfer speeds, and has expensive throughput costs. Solving these problems correctly would require a senior developer to invest significant time and effort.
The Economic Argument
Here’s where the economics become clear: the cost of an in-house developer far exceeds the per-hour cost of marketplace SFTP solutions like SFTP Gateway. Even if you already have developers on staff, there’s a substantial opportunity cost. Those same developers could be working on projects that deliver direct business value instead of recreating existing solutions.
Companies specializing in SFTP products have already solved not just the basic requirements, but also the countless edge cases and related problems that emerge from supporting customers worldwide. They’ve spread their development costs across multiple customers and cloud platforms, making their solutions more economical than any DIY approach.
The Bottom Line If You Want to DIY Your Own SFTP Solution
Building your own SFTP solution might seem like a cost-saving measure, but the hidden complexity, ongoing maintenance burden, and opportunity costs make it a poor business decision. Amazon Marketplace file transfer solutions offer better reliability, more features, professional support, and ultimately lower total cost of ownership.
Instead of reinventing the wheel, focus your development resources on building features that differentiate your business and drive revenue. Leave the SFTP infrastructure to companies that have made it their specialty. Check out the video below to learn about why SFTP Gateway might be the best option if you decide not to DIY your own SFTP solution to S3.
