AWS CloudFront Large Download Delivery for Canto Cumulus

Challenges

Many DAM users frequently need to deliver collection basket downloads ranging from 500 MB to several Gigabytes. These downloads can either consist of a single file, but potentially up to hundreds of files, all of which get delivered contained in a single zip file. The challenges with such a large gamut of potential download scenarios are quite a few:

1. Scaling the hardware of their server becomes a challenge, as monthly server cost is usually a limitation.
2. It often leads to extremely long wait times until the download will start, as the web server needs several minutes to create very large zip files that can be downloaded via the browser.
3. While the server generates the zip file, web server performance drastically decreases for all other users.
4. Lastly, downloading these files via the same web server takes away from the available network bandwidth, also negatively affecting the user experience.

Nextware’s task was to address all four challenges for one of our Canto Cumulus clients, without limiting their need of downloading such large basket collections.

Solution & Benefits

Nextware opted to introduce a dedicated, yet smaller-scaled processing server running Canto RoboFlow. A customization of the Cumulus Web Client’s collection basket now checks the total size of the collection basket. If certain thresholds are surpassed (in this client’s case 500 MB), a RoboFlow download request workflow instead of the regular HTTP direct download will kick in. This will off-load the creation of the download zip file creation to the processing server, thereby not affecting CPU performance of the Cumulus web server.

We opted for a delivery of the finished download zip via the AWS Cloud, specifically AWS S3 / AWS CloudFront, thereby not decreasing the network bandwidth of the Cumulus web server. Once a download zip is finished by the processing server, it is handed over to AWS and the enduser will receive an AWS CloudFront download URL by email. This also allows for employing very sophisticated content delivery strategies, with download zips being delivered from the server most local to the enduser.

To keep AWS S3 storage costs in check, an AWS LifeCycle rule will delete all download zip files after 7 days, so that never more than the most recent 7 days of requested download zip files are stored in the AWS Cloud.

Architecture

All servers in the architecture are hosted in the AWS cloud, using a virtual private cloud (VPC). Assets are stored on a vault server (alternatively the S3 AssetStore could be used), the Cumulus Web Client (CWC) runs on a web server and RoboFlow is running on the processing server, thus enabling this workflow to provide the CPU and network traffic offloading benefits outlined above. This also means that with an in-house server scenario, different network bandwidth offloading strategies would have to be implemented.

Interested?

This custom download workflow can easily be adapted to any Cumulus DAM architecture, starting with your personalized download thresholds and continuing with the download zip delivery method. The RoboFlow workflow is flexible enough to deliver the finished download zip file to any other CDN, FTP server, file server or custom HTTP download delivery platform.

Please contact us for more information. We’d be happy to advise you on the best strategy for your Cumulus DAM solution.