|TS2351: Recovering from Data Errors 16 & 18 (Corrupt Byte Stream Object)|
This technote is written for Helix 6.2 and later. If you are using Helix 6.0 or 6.1, see the archived technote here.
On rare occasions, a picture or document pasted or imported into Helix is stored improperly. When this happens, Helix Utility reports “Data errors that Helix Utility is unable to repair were detected in the collection…” when you Verify or Repair the collection.
This technote explains how to determine if errors reported by Helix Utility are of this type, and how to potentially resolve them without having to contact us for assistance.
|16 & 18 Errors Explained||
Data errors can take many shapes. Typically, errors occur when the data that Helix is working with in memory is not successfully transferred to the collection file stored on disk. This can happen as a result of network noise, hardware issues, bugs in macOS, bugs in Helix, or even external influences.
This technote is strictly about errors that Helix Utility reports as either type 16 or 18, as seen when the “Detailed Logging” option is turned on. These data errors are collectively referred to as Byte Stream Errors, occurring during storage of a picture, document, or long stream of text (greater than 16,383 characters) in a data field, label, or command rectangle.
Curiously, the data often appears to display correctly, but even minute errors are detected and reported by Helix Utility’s data integrity tests. Essentially, these errors indicate that the checksum Helix calculated does not match the actual data stored. The error may be in the actual data or in the checksum, but either way, the problem is contained within the data object stored in Helix.
Helix Utility responds to all errors with the generic, distressing suggestion that “If you have a backup of this collection, it is strongly suggested that you use it. Continued use of this collection may decrease the chances for successful recovery,” but there are exceptions. (This is a limitation we hope to address in a future release.)
In the case of 16- & 18-type errors, this message is too strong. It is not necessary to revert to a backup to resolve these errors.
16- & 18-type errors are self-contained and do not affect other areas of the collection. If all of the errors reported by Helix Utility are 16- & 18-type errors, you can continue to use the collection without fear; you can correct the errors any time that is convenient. (Of course, we don’t recommend that you leave the errors there indefinitely; as that could cause you to ignore other, more dangerous errors, should they arise.)
Consider this analogy: You visit the doctor’s office because of a pain in your wrist. He diagnoses it as carpal tunnel and suggests that you have it corrected. This is not an ‘emergency’ situation, and you can continue with your normal routine until you can schedule a time to have the minor surgery done.
16- & 18-type errors are much the same: they are annoying, but they are not life-threatening, and fixing them is something that can be scheduled at a convenient time. Keep on using your collection without fear, keeping in mind that there is a minor issue that needs to be corrected.
During operation, the contents of a Helix collection are in flux, as internal objects are created and destroyed in the process of supporting the operations. When a collection is closed normally these transient objects are discarded. However, some situations (crash, power fluctuation, etc.) can result in transient objects being left in the collection. These are also susceptible to data errors.
Helix RADE, Engine, and Server automatically dispose of transient objects that were left by accident the next time the collection is opened. So the first step after an abnormal shut down (or anytime data error are reported) is to reopen the collection with Helix RADE, Engine or Server. Then close the collection normally and recheck with Helix Utility to determine whether the data errors have been resolved.
|Identifying Byte Stream Data Errors||
To see errors in the Helix Utility Activity Log, you must turn on the “Detailed Logging” checkbox at the bottom of the window.
Data errors are flagged in orange text and typically appear as:
18 4C04 0022 0002 AE0E 0000 2222 7777 C818 0001
Helix 7.0.3 includes additional information for each error:
18 4C04 0022 0002 AE0E 0000 2222 7777 C818 0001 0CAB 0000 0002
If you see lines that start with 16 or 18, this technote might enable you to correct the issue that is being reported as an error.
Note: If the log details do not contain any lines that begin with 16 or 18, the information in this technote will not be of any use in repairing the collection.
|Interpreting Byte Stream Data Errors||
The log details shows each data error as a string of hexadecimal codes that are typically used by Helix Technical Support to identify the section of the collection where the error is found. Unfortunately, this does not help the average user identify the error. The log is generated with the expectation that it will be read by people trained in doing collection repair.
However, in the case of these error types, Helix 6.2’s AppleScript capabilities can be used in conjunction with this information to correct the error.
Step one is to identify whether the error is in a rectangle or a record. Look at the seventh segment in the hex log line. If the value there is 2222 (as in the example above) the error is in a label rectangle. 3333 indicates a command rectangle, and FFFF (or any other value) indicates that it is in a record.
|Fixing Errors in Objects (Label and Command Rectangles)||
If the error is in a rectangle, download the Error Identifier script from our AppleScripts page. (It’s in the ‘Miscellaneous’ section.) Open the collection with Helix RADE — switch into Design Mode if necessary — and run the script; a dialog asks you to enter an ‘Object ID’ — enter the four characters from segment ten of the error line followed by the previous four characters. Using the sample line above, the value you would enter into the dialog is 0001C818.
If you enter that information correctly, clicking OK results in a dialog containing information to help you identify the object. i.e: the template it is on. You should now be able to go to that template, cut the picture from the rectangle, commit the template, and then re-add the picture. If all goes well, the error will not recur and the collection will now pass all utility checks.
If the script informs you that it can’t read the object, or it points at some totally unrelated object, either you didn’t enter the hex value correctly or the collection has been modified since the error was logged. If all of the characters in the object ID are number, the script may interpret it as a decimal number. To avoid this, type an ‘x’ before the value, as in x0001C818.
|Fixing Errors in Records (Pictures, Documents, Large Text Blocks)||
When the error log line indicates that the problem is in record data (seventh section is FFFF or ‘random’ characters) the error can only be in picture, document, text or styled text data stored in a record. In these cases, the fix is basically the same: locate the offending data (the picture, document, or text), delete it, replace the record, then re-add the data. The collection should now pass the utility check.
The difficulty here is in identifying which record contains the data error. Download the Find 18 Error in Record application (last update: August 8, 2017), then open your collection in Design Mode and run the app. It will assist you in identifying the records containing the errors, which can usually be fixed simply by cutting and repasting the bad data, then replacing the record.
Helix 7.0.3 adds additional information to the error log line. The current version of the Find 18 Error in Record application makes use this new information when available to further check the affected record(s).
|A Note on Picture Fields||
The Picture field datatype is one of the original options in Helix. It stores picture data in PICT format, which has been replaced by various cross-platform formats in macOS. Although we continue to use the PICT format in Helix 6.2, it is clear that macOS itself will one day render it obsolete. In fact, under macOS 10.6, the Preview application can not display PICT files unless the application is run in 32-bit mode. (Weirdly, QuickLook can display the PICT file contents.)
Consequently, our recommendation is to use the Document field datatype instead. Documents containing images of various formats can be displayed in Helix by creating an abacus containing a picture tile (with the document field in its hole) and placing that abacus on any template where the picture should be displayed.
Our file server has some sample scripts that show techniques used to extract PICTs from Helix so they can be saved as files. Helix 7.0 and later convert PICT data into PDF (Apple’s current official format for image interchange between applications).