That said, I think the compression system could really use some improvements in a few key areas, and figured I'd throw them in a V12 wishlist.
- Allow compressing each top-level folder into individual archives.
This would allow you to "encrypt file names" of all contents of a folder, instead of having to choose one massive archive, or no encrypted metadata.
- Allow encrypting file names of individual files.
This could be (somewhat) easily attained by creating a local DB where you store a map of a randomly generated string (the new file name of the archive), and the original name. This DB could then be stored on the destination as a hidden file (in addition to in %localappdata% with the rest of the profile data), and read on restore to get the original names back. Alternatively, you could just not store the original name anywhere, randomly generate a new one for the archive, and on restore, the original name (contained in the zip with the randomized name) would be restored without issue.
- Better multithreading support.
Backing up many millions of files into archives is obviously going to be slow, but even more so when you are forced to use a single thread to do it. Syncback has a feature to compress files in parallel, but it's somewhat flawed...
If you present Syncback with a folder of say, 4 million files, ranging from a few bytes to tens of gigabytes, it seems to get rather confused when it starts zipping them up. It creates a bunch of SBSE____XXXXXXXX files in the destination root directory, but it does so for many, many more than the parallel threads selected. Unfortunately, due to the extremely small identifier that SBP uses in the file name (seems to be 8 characters), this allows for hash collisions during the backup. The result of this is the temp file getting overwritten later in the backup, and SBP putting the wrong file inside of the archive with the correct file's name, which is extremely confusing and rather concerning. This does however raise a warning event, stating that the compressed file is different than expected, but it still shouldn't be possible for this to happen at all.
I had this happen over 100 times in my backup of 4 million files.
I'd suggest either using the full SHA256 hash as the identifier or reducing the number of files that are able to be placed in the directory with the temp name to the parallel thread limit, rather than allowing tens or hundreds of thousands.
- Allow Multipart archives with LZMA2/XZ.
7-Zip allows this, so I know it's possible. Syncback should implement this with all the standard features available to multipart archives.
- When the destination is the cloud, allow sending multipart archives in chunks.
If compressing a directory totaling many TBs, the temp directory used becomes a problem when compressing into a single archive. If you split the resulting compressed file into chunks, SBP should upload them as it goes, rather than at the end. Obviously, there might be a few parts that it needs to keep around to facilitate this, perhaps the first one, and the ones immediately before and after each part that gets uploaded, but it should significantly reduce the requirement for the temp directory by doing this. I have a ~15TB profile, split into 4GB chunks. When running the initial backup, dealing with the temp folder is a huge pain.
Thanks.