Before posting, and to avoid disappointment, please read the following:

  • This forum is not for 2BrightSparks to provide technical support. It's primarily for users to help other users. Do not expect 2BrightSparks to answer any question posted to this forum.
  • If you find a bug in any of our software, please submit a support ticket. It does not matter if you are using our freeware, a beta version or you haven't yet purchased the software. We want to know about any and all bugs so we can fix them as soon as possible. We usually need more information and details from you to reproduce bugs and that is better done via a support ticket and not this forum.

Newbie Question: Name/Date vs. Hash vs. Binary

For technical support visit https://support.2brightsparks.com/
Post Reply
ChrisW828
Newbie
Newbie
Posts: 5
Joined: Wed Mar 29, 2017 2:50 am

Newbie Question: Name/Date vs. Hash vs. Binary

Post by ChrisW828 »

I have been using SyncBack "forever" and am trying the Pro version now. I fear I may have been deleting files I want to keep for years.

Does SyncBack only compare file names and dates, or does it compare more in depth? I have been Googling and reading the forums for hours and it seems like either binary comparison or hash comparison is what I am asking about, but I'm not sure which one, and every thread I am finding via search discusses pieces, but doesn't tell me which is default, how to find and turn one "deeper" comparison, etc.

If possible, I'd like to synchronize in such a way that files are compared beyond name/date for differences and files ARE removed if they 1. don't share name/date, but 2. are actually the same exact file regardless, and ARE NOT removed if they 1. share name/date and 2. are actually different files regardless.

I hope I am making sense. Thank you in advance.
ChrisW828
Newbie
Newbie
Posts: 5
Joined: Wed Mar 29, 2017 2:50 am

Re: Newbie Question: Name/Date vs. Hash vs. Binary

Post by ChrisW828 »

I continued searching and think I may have found my answer. Posting in case anyone else finds this thread via similar search. If this is incorrect, or there is more to it, I'd appreciate a head's up. :)

https://www.2brightsparks.com/syncback/ ... ttings.htm
Use slower but more reliable method of file change detection: By default, SyncBackPro will not compute the hash value of a file. The reason is that it can dramatically increase the time taken for a profile to run. However, if you want to be absolutely certain that SyncBackPro detects if a file has changed, so that it is copied, then you can enable this option. The only reason to enable this option is if you do not trust the last file modification date & time of the files, and the file size may not change. For example, by default, TrueCrypt drive container files never change size or last modification date & time (note that you can configure TrueCrypt to change the contains last modification date & time via the Settings->Preferences main menu).
thorsten26
Newbie
Newbie
Posts: 1
Joined: Wed May 09, 2018 6:41 am

Re: Newbie Question: Name/Date vs. Hash vs. Binary

Post by thorsten26 »

I was also worried about my Data especially Videos and Pictures which are all stored on one Fixdisk - so i have a mirror to an other fixdisk.

However, there is a big Myth? Or a Fact? Thats called BITROT or Bit rotation.

If u search in the Internet u find a lot of Threads about this issue and it seems to be a never ending Story.... :shock:

For those, who didn't know what that is: It seems, that if a File is sitting on the Harddisk - over a longer time - that one Bit can change from 0 to 1 or to the other way arround. U may have seen this already, typically in jpg Pictures if u see suddenly only a half picture or colered lines - in streaming media like a video u may not see it at all.

So i looked for a way to

1. Mirror my Data from one Disk to a second
2. Verify, that I have no Bitrot ( Bit rotation)

Syncback is a good Tool for doing it - but it is not perfect and should be enhanced a little bit to follow the logic to recognize Bitrot faster.

Now what exactly is happening?

Lets say on the left side u have ur original files and u want to sync them to the right side with verify, secure copy and all the nice things.
As a result u should have an exact duplicate on the right side.

Now in ur original Files, u have in one or more pictures a Bitrot (because of bad Harddisk surface or what ever) - How u will recognize this with Syncback?

I guess, if there is a Bit rotation, that Fileattributes keep the same as they where: Date Time last modified, Creation Date Time and so on is the same as before also the file size is the same - only one Bit changed.

That means, if u do a Sync without using Hashing and comparing, the defective file will not be recognized (on left side ( source))

On the other Hand if u are using Hashing the File, which is defective!!!!, will be recognized and synced to the right side, overwriting the original, good File!!!!

A good solution would be, if u could use the integrety database which is an option in Syncback. But u can't do that today because there is no logic behind it for recognizing BitRot.

Sure, u can build hashes for Files on left and right side and u can add hashes for new files - but with integrety check u are getting a lot of errors because files are changing (by modifying them) and the already recorded hashes are not updated.
So there should be in the database also the date/time last modified and hash from a file. If u then do a Integrity check for only files which didn't changed (via Date/Time modified) u will recognize the Bitrot.

So my solution is, that i'm using two profiles - a Profile for syncing and one Profile for verify. To be secure im using also Versioning in a hidden Top-Folder

Because a Verify is very time consuming - evertime hashes from source and destination must be calculated u are now able to do it from time to time only.

The syncing Profile is configured to use the fastest syncing possible with Syncback - no binary compare, no secure copy nothing - simply using file size, date time modified to do the sync.

So, if ur Hardware is not having a malfunction, after using the Sync-Profile u should have on the right side an exact copy.

With the Verify Profile i'm using on both sides Hashing for doing a compare and also here i'm using Versioning in a hidden folder in the Top.

Regarding BitRot:

In the first Sync, using file size and date time modified, a bit rot will not be recognized and the defective File will not overwrite thg intact file in the mirror.

If u run the verify profile u should get as a result that it is nothing todo! If u get any results here then u have a bitrot in a file or somthing went wrong by syncing.

Ok, fine... but it is very time consuming because all hashes must be calculated again and again.

I would be happy if there would be an option to use a hash database for left and right wich is using the following logic:
For new files add hash and date time last modified
For old files, if date time last modified is different then update hash
For old files, if date time last modified is the same, but the hash is different = Error!

For doing all my test i have used the tool SetFileDate 2.0 - where u can manipulate the datetime modified, Accessed and Created Flag for a file.
konnak
Newbie
Newbie
Posts: 1
Joined: Thu Oct 11, 2018 3:49 pm

Re: Newbie Question: Name/Date vs. Hash vs. Binary

Post by konnak »

thorsten26 wrote:
Wed May 09, 2018 7:40 am
I would be happy if there would be an option to use a hash database for left and right wich is using the following logic:
For new files add hash and date time last modified
For old files, if date time last modified is different then update hash
For old files, if date time last modified is the same, but the hash is different = Error!
This! I am planning out a backup strategy, I have a small file server set-up to hold backups, plus Crashplan, and the only remaining concern to eliminate is the possibility of bit rot. SyncBack is such a great tool in that it provides great versatility, yet it seems that regarding integrity check they made it perfect except for this small detail!

To be honest, I still can't figure how your two-profile solution makes amends for legitimately modified files, i.e. files that were supposed to be modified since the last Source integrity check - hash creation. The regular profile (no integrity check) would either overwrite the previously backed up file (or version it). When the integrity check profile is run, it will detect the hash mismatch and either prompt or also attempt to version the already backed up file - which however now is the same as the one one Source. If, instead, bitrot has affected the Source file, then the regular profile will skip it and the integrity check one will, well, basically do the same as in the case of a properly modified file. So, do you do a manual integrity check before running the integrity check profile? And you go over all modified files to see if there is a case that seems to be bitrot? Unless you do this scheme on files that are not expected to be modified, e.g. photo original files.

In any case, the quoted option would be a perfect addition, which could also be marketed accordingly ("Backup solution with bitrot detection"), which also seems as a logical completion to what SyncBack originally intended to do with integrity check - it now seems unfinished. As I see it, the only logical way to use integrity check is only on Destination, and use other means of preventing/detecting it on Source so that it does not contaminate your backup copies (e.g. ReFS and... pray). Or just use reliable copy, integrity check, etc. and a single profile, hope to identify a bit rot visually (e.g. a photo displaying those lines, etc.), and use versioning to make sure you also can get the original back.
Post Reply