Direct Multipart Uploads to S3 in Rails

When building Bitcast.io we took a hard look at the available open source uploading options.  We wanted to create an uploading experience that took full advantage of our users’ upstream bandwidth, delivering videos as quickly as possible to their destination, but that duplicated effort as little as possible on our end. We also wanted to use Amazon’s S3, a service that continues to rank among the most economical hosting solution on the market.

Solutions such as Carrierwave and Paperclip bear a polish that took years of combined effort to achieve, but were conceived with small file uploads in mind. Extensions like Carrierwave Direct cut down on many of the pain points inherent in larger file uploads, but were built on frameworks that don’t solve the problem and lack basic features (such as multiple file uploads). Javascript-based plugins such as Sebastian Tschan’s jQuery File Upload offer up similar solutions, but lack a well thought-out integration with Rails and don’t utilize the latest technological advancements.

To fill the perceived gap we created S3 Multipart, a gem that makes direct multi-part uploading to S3 as painless as possible to implement in Rails. With the recent introduction of the Blob API in the browser, we were able to offload a great deal of responsibility to the client. Files are split into many chunks and uploaded directly to Amazon both in parallel and succession. Built-in initiation and completion events allow the appropriate callbacks to be executed on our servers and we never have to touch a single byte as it makes its way to its home on S3. All data flows directly to Amazon’s servers, and with multi-part uploading, our users can now max out their upload speeds - a feat that, for most, is not possible when uploading one piece at a time.

S3 Multipart is still a work in progress, but it is quickly nearing a production-ready state. Please check out the repository on Github and let us know what you think. Pull requests are always welcome.

If you like it make sure to sign up at http://www.bitcast.io/ so we can let you know when we launch.