Differences

This shows you the differences between two versions of the page.

--- sneaker_net_sync [2013/05/22 10:04] – [Advanced Options] stephen
+++ sneaker_net_sync [2017/01/01 20:05] (current) – external edit 127.0.0.1
@@ Line 22: / Line 22: @@
 ===== How does it work? =====
-**Master Computer**: This is the computer that contains the //source// files. Sneaker Net Sync only **reads** these files and does not make any changes to them.
+**Master Computer**: This is the computer that contains the //source// files. These files are only ever **read** and are not modified.
 **Secondary Computer**: This is the computer that contains the files that will be **changed** to match the files on the **master** computer.
@@ Line 36: / Line 36: @@
 The ''Secondary Source'' in this case is our backup directory. The directory must exist, but it can be empty.
-The ''Index File'' is the file that contains an index of all of the files in the ''Secondary Source''. This is used by Sneaker Net Sync to identify changes between the files on the master and secondary computers. This file must be transferred to the master computer. In this case, ''D:'' is the USB drive that we are going to sneaker net to the master computer.
+The ''Index File'' is the file that contains an index of all of the files in the ''Secondary Source''. This is used to identify changes between the files on the master and secondary computers. This file must be transferred to the master computer. In this case, ''D:'' is the USB drive that we are going to sneaker net to the master computer.
 The ''Generate hash values'' will add a hash value for each file into the index file. This will **significantly** increase the amount of processing time, both at this stage and later when the delta file is created.
@@ Line 50: / Line 50: @@
 {{ :snscreatedelta.png?nolink | Create the delta file.}}
-The ''Master Source'' in this case is the directory where the files we want backed up are. These file are only **read** by Sneaker Net Sync and are not modified.
+The ''Master Source'' in this case is the directory where the files we want backed up are. These files are only ever **read** and are not modified.
 The ''Index File'' is the file created in [[#create_the_index_file_secondary_computer|step 1]].
@@ Line 62: / Line 62: @@
 {{ :snscreatedeltaadvancedoptions.png?nolink | Advanced options for creating the delta file.}}
-If the ''Use Temp Dir'' option is checked, Sneaker Net Sync will first create the delta file in the Window temporary directory and then move it to the final location. This is much faster if the final delta file location is on a USB drive or an external hard drive, but will be slightly slower if the delta file is on the local hard drive.
+If the ''Use Temp Dir'' option is checked, the delta file will be created in the Windows temporary directory and then moved to the final location. This is much faster if the final delta file location is on a USB drive or an external hard drive, but will be slightly slower if the delta file is on the local hard drive.
 The ''Limit Size'' can be used to set the maximum size of the delta. Most USB drives are FAT32, which means that the maximum file size they can support is 4GB. In this case, the ''Limit Size'' should be set to 3.9 GB. If there are more than 3.9 GB of files to sync, the extra ones are ignored. This means that the files on the secondary computer will not be fully in sync first time around, but the sync process can be repeated until they 'catch up'.
@@ Line 84: / Line 84: @@
 The ''Delta File'' is the file created in [[#create_the_delta_file_master_computer|step 3]].
-If the ''Re-index'' is checked, Sneaker Net Sync will recreate the index file (using the settings on the 'Create Index' tab) after the delta has been applied. This allows [[#create_the_index_file_secondary_computer|step 1]] to be skipped when the sync process is repeated.
+If the ''Re-index'' is checked, the index file is recreated (using the settings on the 'Create Index' tab) after the delta has been applied. This allows [[#create_the_index_file_secondary_computer|step 1]] to be skipped when the sync process is repeated.
 Click ''Start'' to apply the delta file.
+===== Edge Cases =====
+There is a possible situation that would prevent Sneaker Net Sync from detecting that a file has changed. When a computer program modifies the contents of a file the operating system (Windows) automatically updates the modified date of the file. However, a program can also directly set the modified date to anything. A program could theoretically read the modified date of a file, change the contents of the it and then set the modified date back to its original value. This would be very strange behaviour indeed. If the length of the file did not change, Sneaker Net Sync would not detect the change and would not ever re-sync the file.
+This edge case is very unlikely to happen, but if it was a concern, hash values could be used. This would prevent the edge case, but using hash values dramatically slows down the indexing and delta creation process.
+Well... almost. Hash values do not //fully// eliminate the edge case. While it is incredibly unlikely that two files with different content would have the same hash value, it is technically //possible//. Sneaker Net Sync uses a 128-bit hash, which means that there are approximately 3.402 × 10<sup>38</sup> possible unique hash values. Assuming truly random distribution, the chances of two files having the same hash value is about the same as throwing three pins at the Earth from space such that they land at some random point on the Earth's surface, and all three land within 1 mm of each other (the Earth's surface area is approximately 5.1 × 10<sup>20</sup>mm<sup>2</sup>).
+If this possibility is unacceptable, the only option would be to fully recopy **all** the files every time a re-sync is done. Even this would not absolutely and completely guarantee that the files are synced - there could be read or write errors while copying the files to and from the transport media.
 ===== Daylight Saving Issues =====
+Obviously, if both the master and secondary computers are set to a time zone that does not have daylight saving, they will not be affected by this issue. If both computers are Windows 7+ and the file systems are NTFS, they should also not be affected. Otherwise, they //might// be affected.
+==== The Problem ====
+After changing to daylight saving time or back to standard time, Sneaker Net Sync may incorrectly determine that //all// the files have been modified and attempt to resync them all.
+==== The Cause ====
+Sneaker Net Sync relies on the name, length, modified date and hash (if generated) of a file to determine if it has been modified, added or deleted. It turns out that there are two issues that can make the modified date unreliable:
+  * On FAT32 file systems, the modified date is recorded using the **local** system date and no time zone offset is stored. When a file is moved to another computer it is no longer possible to exactly determine what the real modified date is. On NTFS file systems, the modified date is stored as UTC. It will always be correct, even if the host system changes its time zone.
+  * When Windows XP converts the modified date of a file between UTC time and local time it adds or subtracts the current time zone offset. This method does not deal with daylight saving time correctly. The correct method is to add or subtract the time zone offset that was in use //when the file was modified//. Windows 7 and above uses the correct method.
+Unfortunately, the bottom line is that file modified dates can not always be reliably compared between different systems.
+==== The Workaround ====
+There is no practical way of 'fixing' the daylight saving issue. The best that can be done is a workaround. On the ''Create Delta'' tab, select the ''Advance Option'' button:
+{{ :snscreatedeltaadvancedoptions.png?nolink |}}
+''None'': Nothing special is done. **All** files will be resynced when going into daylight saving time and back to standard time. Typically, this will take a long time and will create a huge delta file and/or may require multiple resyncs to 'catch up'. Once the files are resynced, the file modified dates will match between the master and secondary computers.
+Only use this option if Sneaker Net Sync is not affected by the daylight saving issue, or there a concern about the [[#the_daylight_saving_edge_case|edge case]].
+''Ignore time differences'': If the name, file size and hash (if used) match and the modified date is different by the amount specified by the ''Time Difference'' setting (+/- 3 seconds) the difference will be ignored and the file will be considered to be 'in sync'. Nothing special will happen when going into daylight saving time or back to standard time. Files added or modified during daylight saving time will have modified dates that do not match the master computer's files. These files will also have modified dates that are incorrect relative to the files that were //not// modified during daylight saving time. The incorrect modified dates will never resync.
+Use this option if whether or not the modified dates remain synced is completely unimportant.
+''Correct time differences'': If the name, file size and hash (if used) match and the modified date is different by the amount specified by the ''Time Difference'' setting (+/- 3 seconds) the modified will be resynced, but not the contents of the file. When going into daylight saving time or back to standard time, **all** the files will have their modified resynced. Typically, this does not take too long and does not result in a very large delta file. Once the modified dates are resynced, they will remain synced.
+Use this option if it is important that the modified date remain synced as much as possible.
+''Correct time ... ignore time...'': When it is daylight saving time, modified dates are ignored as described above. When it is standard time, modified dates corrected as described above. When going into daylight saving time, nothing special will happen. When going back to standard time, incorrect modified dates will be resynced. This will only affect files that were added or updated during the daylight saving period so will typically be much faster than the ''Correct time differences'' option. Modified dates will be synced during standard time. During daylight saving time, the modified dates of new or updated files will be correct.
+Use this option if it is important that the modified dates //eventually// sync up.
+''Time Difference'': The amount of time that should be ignored or corrected.
+==== The Daylight Saving Edge Case ====
+If the Time Zone Options is set to anything other than ''None'', there is a known edge case that could potentially cause Sneaker Net Sync to fail recognise that a file has changed - although the chances of this occurring are very remote. It would have to happen like this:
+  - A file is modified on the master computer.
+  - Sneaker Net Sync creates a delta file.
+  - The same file is modified, without changing its length, one hour (+/- 3 seconds) after it was originally modified.
+Because the modification date of the file will have increased by one hour and the length remained the same, Sneaker Net Sync will either ignore the change or update the modified date without updating the contents.
+If this edge case is a concern, the Time Zone Options can be set to ''None'' or hash values can be used.