self-encryption src/lib.rs:L187 says
/// The maximum size of file which can be self-encrypted, defined as 1GB
pub const MAX_FILE_SIZE: usize = 1024 * 1024 * 1024;
This would seem to be a temporary limit.
What is the planned largest file size, if any?
The data map may become very large if there's no largest file size.
eg a 1 MB data map (assuming it only includes a list of chunk names each 256 bits ie 32 bytes) would be able to contain a list of 32768 chunk names, which would map to 32 GB of data. Still less than a bluray disc.
A 100 GB file would have a data map greater than 3 MB. This seems potentially problematic to me (considering all chunks are 1 MB max).
Some possible ideas for self-encrypting large files:
- remove the 1 GB limit and accept data maps greater than 1 MB
- data maps can be a list of chunk names or other data map names, ie large data maps may themselves be chunked
- apps split large files into parts less than 1 GB before upload, then recombine them in the app (but this is ugly and will lead to many different incompatible implementations)
Related topic: RFC: Increase limits on file sizes in self_encryption
Just curious to understand the longer-term plans with large files and self encryption.