Differences

This shows you the differences between two versions of the page.

--- sneaker_net_sync [2013/05/21 23:48] – created stephen
+++ sneaker_net_sync [2017/01/01 20:05] (current) – external edit 127.0.0.1
@@ Line 3: / Line 3: @@
 ======  Sneaker Net Sync ======
+Sneaker Net:
+<blockquote>
+To transfer files or data by physically moving media (typically USB drives or external hard drives) from one computer to another. Sneaker refers to a type of footwear sometimes associated with the stereotypically causal IT professional.
+</blockquote>
+Sneaker Net Sync is an application that synchronises files between two computers without using a computer network connection. No need to figure out which files or directories have been changed, added or deleted - Sneaker Net Sync will do that out for you and make the required changes. Sneaker Net Sync is great for creating off-site backups.
+Note: Sneaker Net Sync only does one-way synchronisation, meaning that it makes changes to the files on one computer to make them exactly match the files on the other computer. Technically, this would be better described as a mirror, but somehow 'sneaker net mirror' doesn't quite roll off the tongue as easily.
+==== Why would anyone use a sneaker net? ====
+  * Either one or both of the computers may not be connected to a computer network.
+  * The changed files may be very large. Sneaker nets using a USB drive or an external hard drive can often have a much higher bandwidth than a computer network.
+  * The contents of the changed files may be sensitive. Files on a physical USB drive are often much easier to secure and keep private than sending them over a computer network.
+  * It may be inappropriate to send certain types of files over a particular computer network. For example, sending personal files over a corporate computer network may violate the usage policy.
+===== How does it work? =====
+**Master Computer**: This is the computer that contains the //source// files. These files are only ever **read** and are not modified.
+**Secondary Computer**: This is the computer that contains the files that will be **changed** to match the files on the **master** computer.
+==== 1. Create the index file (secondary computer) ====
+{{ :snscreateindex.png?nolink | Create the index file.}}
+In this example, we are going to create a backup of our pictures. We start by going to our **secondary** computer.
+The drop down at the top of the window and the ''Save'' and ''Delete'' buttons allow all of the settings to be saved for later use or deleted when they are no longer required.
+The ''Secondary Source'' in this case is our backup directory. The directory must exist, but it can be empty.
+The ''Index File'' is the file that contains an index of all of the files in the ''Secondary Source''. This is used to identify changes between the files on the master and secondary computers. This file must be transferred to the master computer. In this case, ''D:'' is the USB drive that we are going to sneaker net to the master computer.
+The ''Generate hash values'' will add a hash value for each file into the index file. This will **significantly** increase the amount of processing time, both at this stage and later when the delta file is created.
+Click ''Start'' to create the index file.
+==== 2. Sneaker net the index file to the master computer ====
+{{ :walkingright.gif?nolink | Walking to the master computer}}
+==== 3. Create the delta file (master computer) ====
+{{ :snscreatedelta.png?nolink | Create the delta file.}}
+The ''Master Source'' in this case is the directory where the files we want backed up are. These files are only ever **read** and are not modified.
+The ''Index File'' is the file created in [[#create_the_index_file_secondary_computer|step 1]].
+The ''Delta File'' is the file that will be created. It will contain all of the modified and new files and details about how to change the files on the secondary computer to exactly match the master computer. This file must be transferred to the secondary computer. In this case, ''D:'' is the USB drive that we are going to sneaker net to the secondary computer.
+Click ''Start'' to create the delta file.
+=== Advanced Options ===
+{{ :snscreatedeltaadvancedoptions.png?nolink | Advanced options for creating the delta file.}}
+If the ''Use Temp Dir'' option is checked, the delta file will be created in the Windows temporary directory and then moved to the final location. This is much faster if the final delta file location is on a USB drive or an external hard drive, but will be slightly slower if the delta file is on the local hard drive.
+The ''Limit Size'' can be used to set the maximum size of the delta. Most USB drives are FAT32, which means that the maximum file size they can support is 4GB. In this case, the ''Limit Size'' should be set to 3.9 GB. If there are more than 3.9 GB of files to sync, the extra ones are ignored. This means that the files on the secondary computer will not be fully in sync first time around, but the sync process can be repeated until they 'catch up'.
+Note that the delta file is compressed and the size limit is applied **before** the compression is done. This is because there is no easy way (other than actually doing the compression) to know how much a file will be compressed and therefore how many files can be added without the final file going over the size limit. By limiting the size before compression, we can at least ensure that the final file will always be less than the limit.
+TODO: Rename this option to ''Size Limit''.
+The ''Time Zone Options'' and ''Time Difference'' settings are used to rectify daylight saving issues that can sometimes occur. See [[#daylight_saving_issues|Daylight Saving Issues]] for more details.
+==== 4. Sneaker net the delta file to the secondary computer ====
+{{ :walkingleft.gif?nolink | Walking to the secondary computer}}
+==== 5. Apply the delta file (secondary computer) ====
+{{ :snsapplydelta.png?nolink | Apply the delta file.}}
+The ''Secondary Source'' in this case is our backup directory. It should be the same as what was used in [[#create_the_index_file_secondary_computer|step 1]].
+The ''Delta File'' is the file created in [[#create_the_delta_file_master_computer|step 3]].
+If the ''Re-index'' is checked, the index file is recreated (using the settings on the 'Create Index' tab) after the delta has been applied. This allows [[#create_the_index_file_secondary_computer|step 1]] to be skipped when the sync process is repeated.
+Click ''Start'' to apply the delta file.
+===== Edge Cases =====
+There is a possible situation that would prevent Sneaker Net Sync from detecting that a file has changed. When a computer program modifies the contents of a file the operating system (Windows) automatically updates the modified date of the file. However, a program can also directly set the modified date to anything. A program could theoretically read the modified date of a file, change the contents of the it and then set the modified date back to its original value. This would be very strange behaviour indeed. If the length of the file did not change, Sneaker Net Sync would not detect the change and would not ever re-sync the file.
+This edge case is very unlikely to happen, but if it was a concern, hash values could be used. This would prevent the edge case, but using hash values dramatically slows down the indexing and delta creation process.
+Well... almost. Hash values do not //fully// eliminate the edge case. While it is incredibly unlikely that two files with different content would have the same hash value, it is technically //possible//. Sneaker Net Sync uses a 128-bit hash, which means that there are approximately 3.402 × 10<sup>38</sup> possible unique hash values. Assuming truly random distribution, the chances of two files having the same hash value is about the same as throwing three pins at the Earth from space such that they land at some random point on the Earth's surface, and all three land within 1 mm of each other (the Earth's surface area is approximately 5.1 × 10<sup>20</sup>mm<sup>2</sup>).
+If this possibility is unacceptable, the only option would be to fully recopy **all** the files every time a re-sync is done. Even this would not absolutely and completely guarantee that the files are synced - there could be read or write errors while copying the files to and from the transport media.
+===== Daylight Saving Issues =====
+Obviously, if both the master and secondary computers are set to a time zone that does not have daylight saving, they will not be affected by this issue. If both computers are Windows 7+ and the file systems are NTFS, they should also not be affected. Otherwise, they //might// be affected.
+==== The Problem ====
+After changing to daylight saving time or back to standard time, Sneaker Net Sync may incorrectly determine that //all// the files have been modified and attempt to resync them all.
+==== The Cause ====
+Sneaker Net Sync relies on the name, length, modified date and hash (if generated) of a file to determine if it has been modified, added or deleted. It turns out that there are two issues that can make the modified date unreliable:
+  * On FAT32 file systems, the modified date is recorded using the **local** system date and no time zone offset is stored. When a file is moved to another computer it is no longer possible to exactly determine what the real modified date is. On NTFS file systems, the modified date is stored as UTC. It will always be correct, even if the host system changes its time zone.
+  * When Windows XP converts the modified date of a file between UTC time and local time it adds or subtracts the current time zone offset. This method does not deal with daylight saving time correctly. The correct method is to add or subtract the time zone offset that was in use //when the file was modified//. Windows 7 and above uses the correct method.
+Unfortunately, the bottom line is that file modified dates can not always be reliably compared between different systems.
+==== The Workaround ====
+There is no practical way of 'fixing' the daylight saving issue. The best that can be done is a workaround. On the ''Create Delta'' tab, select the ''Advance Option'' button:
+{{ :snscreatedeltaadvancedoptions.png?nolink |}}
+''None'': Nothing special is done. **All** files will be resynced when going into daylight saving time and back to standard time. Typically, this will take a long time and will create a huge delta file and/or may require multiple resyncs to 'catch up'. Once the files are resynced, the file modified dates will match between the master and secondary computers.
+Only use this option if Sneaker Net Sync is not affected by the daylight saving issue, or there a concern about the [[#the_daylight_saving_edge_case|edge case]].
+''Ignore time differences'': If the name, file size and hash (if used) match and the modified date is different by the amount specified by the ''Time Difference'' setting (+/- 3 seconds) the difference will be ignored and the file will be considered to be 'in sync'. Nothing special will happen when going into daylight saving time or back to standard time. Files added or modified during daylight saving time will have modified dates that do not match the master computer's files. These files will also have modified dates that are incorrect relative to the files that were //not// modified during daylight saving time. The incorrect modified dates will never resync.
+Use this option if whether or not the modified dates remain synced is completely unimportant.
+''Correct time differences'': If the name, file size and hash (if used) match and the modified date is different by the amount specified by the ''Time Difference'' setting (+/- 3 seconds) the modified will be resynced, but not the contents of the file. When going into daylight saving time or back to standard time, **all** the files will have their modified resynced. Typically, this does not take too long and does not result in a very large delta file. Once the modified dates are resynced, they will remain synced.
+Use this option if it is important that the modified date remain synced as much as possible.
+''Correct time ... ignore time...'': When it is daylight saving time, modified dates are ignored as described above. When it is standard time, modified dates corrected as described above. When going into daylight saving time, nothing special will happen. When going back to standard time, incorrect modified dates will be resynced. This will only affect files that were added or updated during the daylight saving period so will typically be much faster than the ''Correct time differences'' option. Modified dates will be synced during standard time. During daylight saving time, the modified dates of new or updated files will be correct.
+Use this option if it is important that the modified dates //eventually// sync up.
+''Time Difference'': The amount of time that should be ignored or corrected.
+==== The Daylight Saving Edge Case ====
+If the Time Zone Options is set to anything other than ''None'', there is a known edge case that could potentially cause Sneaker Net Sync to fail recognise that a file has changed - although the chances of this occurring are very remote. It would have to happen like this:
+  - A file is modified on the master computer.
+  - Sneaker Net Sync creates a delta file.
+  - The same file is modified, without changing its length, one hour (+/- 3 seconds) after it was originally modified.
+Because the modification date of the file will have increased by one hour and the length remained the same, Sneaker Net Sync will either ignore the change or update the modified date without updating the contents.
+If this edge case is a concern, the Time Zone Options can be set to ''None'' or hash values can be used.