Film and livestreaming projects generate large amounts of data. Depending on the project, the data must be stored for varying lengths of time. As a result, a large pool of data accumulates over time. The challenge is to set up a reliable, cost-effective, and secure system. In this article, we explain how we handle this and share our experiences from the past few years.
Intended Use
We use large data pools to back up current project data. Once a project is completed, the data is moved to long-term storage. We also operate our own services to share project data with partners—for example, Nextcloud.
Requirements
As media producers, we have certain basic requirements for a storage solution.
Easy to maintain
Even the best solution is useless if it isn't easy to maintain and operate. Many processes should be automated.
Fast
We have a lot of data, so it must be possible to write to or read from it within a reasonable amount of time. The network connection must be designed accordingly.
Flexibly Expandable
The pool is getting bigger and bigger because certain customer data needs to be retained for longer periods. So the pool must be able to grow accordingly.
Cost-effective
A data pool like this incurs costs. First, there are hardware costs. Components can suddenly fail, since all data storage devices—such as hard drives and SSDs—have a limited lifespan. Or the pool may need to be expanded. In addition, running equipment, such as a server, requires electricity.
Security
Our History in Data Management
Our data strategy in the early years was as follows:
Project data from a film shoot ended up on the editing computer.
For backup purposes, the data was automatically copied to a second hard drive in the editing computer.
The backup hard drive was regularly swapped out for a duplicate hard drive. The hard drive that was not in the editing computer was stored in a bank safe deposit box.
It worked right away with the archive. We always bought two identical hard drives. One was kept in the office and one in a safe deposit box.
As the business grew, so did the data volume. At some point, the archiving solution became unsustainable. The hard drives had to be replaced manually too often. It took longer to locate project data. And at times, replacing the backup hard drives was neglected.
So the time had come to adopt a new solution.
What do we store, and for how long?
Depending on the project and the client, we retain data for varying lengths of time. Here are a few examples:
Promotional Video Project
We store this type of data for years at the customer's request. After all, it's quite possible that, after a certain amount of time, the customer may want to make changes to the promotional video. Or they may want to use footage from it for a new project.
Live streamings
Streaming projects generate a large amount of data in a short period of time. This is because we often record not only the final stream but also all individual sources, such as cameras or presentations. This allows us to create custom video clips for social media or correct any errors. For streaming projects, we delete all source data after one year.
Master Files
We keep the final edited version of each project for reference.
Why not move to the cloud?
We explored whether a cloud solution would be suitable for our data archive and backup. We decided against it for the following reasons:
Since we need a lot of storage space, the costs would be extremely high. And as demand increases, so would the costs.
Access to the data is limited by the capacity of the Internet connection. Uploading or downloading a large project would take a very long time.
The data should remain in Switzerland. And we want to avoid becoming dependent on individual services.
Our System Architecture
For security reasons, we don't explain everything down to the last detail. But here's a general overview of how our system is set up:
Cutting Computer
All the projects we're working on are stored on very fast SSDs in the editing suite. This allows for data access speeds of up to 8 GB/s. The data is backed up regularly to Server 1.
Server 1
The main server is connected to the core network at 2x10 Gbps. The editing computer is also connected at this speed. This is where backups and data from completed projects are stored. Services such as Nextcloud and others also run here. TrueNAS is used as the operating system. The hardware consists of an old editing computer that has been repurposed for server use.
The server can accommodate up to 24 3.5-inch HDDs and 8 2.5-inch SSDs. It currently has a total capacity of approximately 150 TB.
Server 2
This server is located at a second site. It is connected to the core network via the Internet using an encrypted connection. All new data is regularly mirrored from Server 1 to this server. The hardware also consists of an old editing computer. Since this server does not have many tasks, the hardware can be less powerful and smaller than that of Server 1.
This server has space for only 8 3.5-inch HDDs. It currently has a total capacity of 120 TB.
Costs
Investment
These are estimates based on current market data, assuming the system were to be rebuilt from scratch.
| Object | Details | Costs [CHF] |
| Server 1 Hardware | Server chassis with CPU, RAM, GPU, and enough bays for HDDs/SSDs | 4,000 |
| Server 1 Hard Drives | 150 TB usable, +20% safety margin, 190 TB total | 5,700 at CHF 30 per TB |
| Server 1 SSDs | 20 TB usable, +20% safety margin, 25 TB total | 3,750 at CHF 150 per TB |
| Server 2 Hardware | Server chassis with CPU, RAM, GPU, and enough bays for HDDs/SSDs | 3,000 |
| Server 2 Hard Drives | 120 TB usable, +20% safety margin, 150 TB total | 4,500 at CHF 30 per TB |
| Network Components | Switches, Network Cards, Cables | 2,000 |
| Total | 22,950 |
Operation and Maintenance
These costs are based on current electricity consumption and current hardware prices. The components will need to be replaced at some point. Money must be set aside for this on a regular basis.
| Position | Details | Costs [CHF] |
| Server 1 Power Consumption | Per year, higher consumption, since this server is significantly more powerful | 1,000 |
| Server 2 Power Consumption | Per year | 150 |
| Replace Server 1 Hardware | Assuming a service life of 8 years | 500 |
| Server 1: Replacing HDDs | Assuming a service life of 5 years | 1,140 |
| Server 1: SSD Replacement | Assuming a service life of 5 years | 750 |
| Replace Server 2 Hardware | Assuming a service life of 8 years | 375 |
| Server 2: Replacing HDDs | Assuming a service life of 5 years | 900 |
| Total per year | 4,815 |