New in version 7.0, Helix Server can be optionally configured for encrypted communication with Helix Clients. Helix uses the same strong communication technologies that protect data on the Internet and used by VPN tunnels and https for web server private communication with a web browser.
From Wikipedia:
Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), both frequently referred to as “SSL”, are cryptographic protocols that provide communications security over a computer network … [TLS] aims primarily to provide privacy and data integrity between two communicating computer applications…
A digital certificate certifies the ownership of a public key… This allows others … to rely upon signatures or on assertions made by the private key that corresponds to the certified public key.
A macOS computer with a properly configured certificate can be used by Helix Server to encrypt data exchanges with Helix Client, guarding against ‘eavesdropping’, ‘spoofing’ and other security issues.
The process of securing a Helix Server can be outlined in the following steps:
- Acquire a certificate
- Install the certificate in macOS
- Connect Helix Server to the certificate
It is beyond the scope of this technote to explain the many ways of acquiring and installing a certificate in macOS or macOS Server. Many excellent tutorials for these tasks exist on the internet. QSA ToolWorks technical support is also available to assist with this process at our standard support rates. You may also find our notes on Purchasing and Installing a Digital Certificate useful.
Important: the certificate, private key, and any intermediate certificates should be installed in the System keychain. Unexpected results may occur if certificate components are installed in the Login keychain.
Once your certificate has been successfully installed in macOS, all that remains is to connect Helix Server to it as follows:
- Quit Helix Server if it is running
- Launch Helix Preference Editor 7.0*, choose Server ⟶ Server Settings ⟶ HxSecureCertificateName, and enter the exact certificate name (the fully qualified domain name used when requesting the certificate)
- Apply the change and Quit Helix Preference Editor 7.0
- Launch Helix Server
*The HxSecureCertificateName preference can also be set via the command line or any property list editor.
If the certificate is properly configured, the Helix Server splash screen displays a locked padlock next to the IP Address line, which displays a secure server name instead of the server’s IP Address. The image on the right displays a secured Server for qsatoolworks.com.
The locked padlock in the Helix Server splash screen indicates that all Client/Server communications are automatically handled via TLS/SSL. No configuration is necessary for Clients, and there are no special steps a Client must take when connecting to a secured Server. A Client can not connect non-securely to a secured Server.
There are a number of visible indicators to inform the user when a Server is secured…
Server Splash Screen: As shown above, the Helix Server’s splash screen displays a padlock next to the IP Address line along with the secure server name. A non-secure Server displays the server’s IP Address(es) instead.
-
Server Information Window: A new “SSL status” line in the ‘Server’ panel reports the status as one of the following:
- Unsecured
- Secure to secure server name
- Secure to secure server name (expires in x days)
- Secure to secure server name (expired)
The first image shown on the right is an example of a secure SSL Status as seen in the Server information window.
-
Helix Client icon: When Helix Client connects to a secured Server, the Dock icon replaces the traditional ‘two-persons’ badge with a padlock. This icon is displayed in the Application Switcher as well.
Note that Helix Server’s dock icon does not display a padlock when secured; the padlock in the splash screen is always visible, so no other indication is necessary.
When the certificate is within 30 days of expiring, the padlock is replaced by a warning icon (⚠️) as a visual reminder that action must be taken to resume full security. In this situation, communications are still encrypted and host verification is still assured. Connecting Clients are not informed when this situation occurs.
The warning icon also appears for an expired certificate, and in the case where the Certificate Authority (CA) certificate has expired. In these situations, communications are still encrypted, but host verification may be compromised, due to the invalidity of the expired certificate. Connecting Clients are informed via dialog when this situation occurs.
If a valid certificate matching the name specified in the HxSecureCertificateName preference is not found, a dialog providing the option of quitting (to fix the problem) or proceeding with non-encrypted communications is displayed when Server is launched. If the option to proceed is chosen, an ‘open padlock’ icon is displayed to indicate that communications are not encrypted, and the host can not be verified. Connecting Clients are informed via dialog when this situation occurs.
In all cases, the Server Information Window’s SSL Status line provides additional information.
Prior to macOS 10.8, Apple Secure Transport lacks support for recoverable SSL errors (expired certificate or name mismatch). For this reason (and others) we recommend that Helix Server always be run on the most recent version of macOS possible.
Helix Server running on macOS 10.6 or 10.7 can not accept secure connections from Clients running on macOS 10.8 or later; an ‘Authentication Failed’ error message is always returned.
Helix Clients running on macOS 10.6 or 10.7 can not connect to a Server that has an expired or otherwise invalid certificate, and an ‘Authentication Failed’ error message is returned. Clients running on macOS 10.8 or later are informed via a dialog and given an option to proceed, albeit with a potentially compromised connection.
macOS vs macOS Server should present no issues for Helix, the primary difference being that macOS Server provides an improved interface for installing certificates.
Although a properly configured server will work reliably, the nature of internet security and of digital certificates is such that a proper configuration won’t remain that way forever.
Certificates expire. Apple releases new versions of macOS and interim security updates as needed. The installation and configuration process changes. TLS/SSL sometimes feels like a high-wire balancing act, where the slightest deviation results in an invalid (or worse: semi-valid) certificate.
What follows is a series of error dialogs we encountered during testing of Helix’s SSL security feature, and the proximate cause and solution for each. Consider this an ‘anecdotal field guide’ to configuration issues we have seen, and how to resolve them.
In all cases, it should be possible to resolve these issues. If you encounter any of them as a recurring issue, or need help diagnosing the problem, please contact technical support.
These two dialogs appeared on the Server in response to certificate installation or configuration errors.
The first dialog indicates that the certificate for the Certificate Authority (GeoTrust) was not properly installed. Probably the most frustrating aspect of certificate management is in getting the right intermediate certificates. Certificate vendors offer an array of choices, and matching the correct CA certificate to the product you purchased is sometimes less than obvious. Our solution to this was: when in doubt, install them all.
The second dialog indicates that one or more certificates in the ‘trust chain’ are not installed in the correct location, or that the fine-grained access settings found in Keychain Access were preventing Helix Server from automatically accessing the certificate.
Our solution was to delete the certificate (and its trust chain) and reinstall from scratch.
These two dialogs are a result of the same error: macOS rejected the attempt to create a secure connection. When a new connection is rejected, the error is displayed in the stauts line of the ‘New Connection’ dialog. When an attempt to reconnect to a Server via a connection document is rejected, the dialog displays the error returned by macOS.
In our case, the failure was caused by attempting a secure connection across the macOS 10.7/10.8 boundary that was noted in the System Version Issues section above.
The solution is to update the Server or Client to macOS 10.8 or later, or to disable secure connections.
This dialog appears when the certificate used by Helix Server can’t be verified by Helix Client. The Client may be running on a version of macOS than the Server, or there may be a configuration issue on the Server, such as using a self-signed certificate.
Contact your network administrator if you see this dialog. The solution is to determine whether the situation is such that you can trust that the connection is valid and to proceed, or to cancel and wait until the situation is resolved.
This dialog appears when an IP Address was previously used to make connection to the Server, but now that security is enabled a certificate name is required.
When a Helix Client makes its first connection to a Server, the user enters a Hostname or IP Address in the New Connection dialog. This information is automatically saved in a ‘connection document’ that makes reconnection easier and faster. (Saved connection documents are listed in the ‘Connect To’ menu seen when Helix Client is not connected to a Server.)
Since Helix 6.2 did not support secure connections, most users connected via an IP Address. Upon first connection to a secured Server in Helix 7.0, this dialog will most likely appear warning that the stored address does not match the certificate name.
The solution is to determine whether the situation is such that you can trust that the connection is being made to the correct machine and to proceed, or to cancel and initiate a new connection using the correct security certificate name.
When in doubt, contact your network administrator.
This dialog appears when the hostname stored in the connection document points to the correct machine, but does not match the certificate name. This can happen when a macOS server is used for multiple services and the initial connection was made to another hostname pointing to the same machine, prior to the certificate being specified. It could also appear if the administrator switches which certificate is used by Helix Server.
The solution is to determine whether the situation is such that you can trust that the connection is being made to the correct machine and to proceed, or to cancel and initiate a new connection using the correct security certificate name.
When in doubt, contact your network administrator.
We haven’t seen this one in a while, so we think it has been resolved. (If you encounter it, please let us know immediately!)
The error indicates that the encryption mechanism in macOS failed for some unknown reason. Apple claims this is ‘not fatal’ but rather than insert code to trap this error and resubmit the request, we choose to halt and immediately terminate the connection. Since this only occurred during the initial connection phase, and a subsequent reconnection attempt was usually successful, the irritation factor is minimal.
The solution is to re-initiate the connection.