|RAMJet: Helix 5.x and Helix 6.0.x Classic Applications|
October 21, 2002 As noted last time, were going to shift gears a bit with this posting and talk about a subject near and dear to our hearts...how to manage memory in Helix 5.
Since this experiment began back in June, weve made every effort to solve each technical problem presented to us in a timely and effective manner. A few times in the past few months, weve encountered situations where the lack of comprehensive or definitive information about this memory management has resulted in, to be generous, a less-than-satisfying experience for some Helix users. It appears that, in spite of the availability of rather profuse documentation on the subject, these users either have not seen, or just cant figure out what to do when they find their way to the Memory Preferences dialog depicted above.
Too often, when faced with a new feature or a change to an old feature, theres a tendency on the part of users to attempt to apply old knowledge to new problems. The rules have changed in Helix 5 and you need to look at memory management issues through new eyes. This is our first--and hopefully only--proactive attempt to explain what these differences mean on a practical level. Before moving into the discussion, we need to point out that there are three major obstacles to creating the aforementioned, much-needed comprehensive or definitive information, illustrated by the three obviously ridiculous questions that follow:
What does it all have to do with managing memory in Helix 5?
Well, there are, sadly, no hard and fast rules that can be applied to the great diversity of Helix situations. However, and in keeping with our theme of attempting to be very general, there are three areas that you as a user must be responsible for managing. These are:
Lets begin with the most frequently asked question:
Do I Have To Use These Features? What If I Just Ignore That Screen?
OK. Suppose you just download 5.0.x and install it, crank the RAM up to where you last had it and go back to work. Is there a downside to this?
All things being equal, everything should work, memory-wise, about the way it worked in 4.5.5 or whatever you were using before you upgraded. Actually, prior to this version of Helix, when you increased the RAM allocation to a Helix application in the Finder, Helix simply divided that additional RAM between its own internally-managed Data and Structure caches. For those of you who have delved deeper into this subject in the past, we know its much more complicated than that, but remember, the purpose of this discussion is to arrive at some very general guidelines.
The downside is simple: youre not taking advantage of one of the key features of the Helix 5 product line. You paid for your upgrade and youre ignoring a potentially powerful tool that could considerably lighten your workload.
Where Is This Screen? How Does It Work?
As noted above, were going to attempt to create some very general rules. Were going to call them the "General Rules of RAMJet()" and well start, as the Marx Brothers would say, with the first number:
1st General Rule of RAMJet: Turn off Virtual Memory
That seemingly small detail has surfaced in a number of calls, and unless you have VM off, none of the rest of this discussion will make any sense. If you must use some application that requires VM to be on, use it either on another machine or when Helix is not being used.
2nd General Rule of RAMJet: Turn AutoSave On and Save and Clear Caches Off.
3rd General Rule of RAMJet: Know Where It Is
The most dramatic performance improvements generally seen--and this is based strictly on anecdotal evidence--is in the use of the Helix Utility. Users with enough RAM to get maximum benefit from this feature have reported that Collections that used to take an hour to run through the Helix Utility now run in under 10 minutes, some in under 5. Thats a pretty significant improvement. Whether we like it or not, one of the big sins frequently committed by Helix users is not running the Helix Utility often enough and one of the principal reasons given in informal surveys of why they dont is that it takes so long to run.
Later, youre going to learn about certain kinds of Helix errors that address Data and Structural memory caching. If you should see one of these errors while using Helix Client, or Helix Update Collection, which do not have Memory Preference settings, increase the amount of memory you allocate to these applications in the Finder.
4th General Rule of RAMJet: To use RAMJet, the Custom Caches must be On
You really dont have much of a choice about this. You cant use the Data and Structure caches separately or exclusively. Turning on one turns on both. And without these, RAMJet cannot be turned on.
There are rare instances where it might make sense for you to adjust the size of the Data Cache. Some operations, for example, require that the Data cache be of a certain size (the exact numbers depend on multiple factors: number of records, number of Fields in records, total size of record data, number of Relations involved in the action, etc.) so setting it to its minimum value can sometimes result in 5xxx errors (more about which below). In these cases it is necessary to increase the data cache. You should view this as a concession, not an improvement. Having a smaller Data cache is better than having to have a larger Data cache. Sometimes the larger cache is unavoidable, but no less regrettable.
Generally speaking, however, the Data cache will always fill to overflowing and there's nothing you can do about it. No matter what you do, it will always fill to a point where there is around 72K free (this number may vary).
The purpose of the Data cache is to speed Helix by keeping recently read data in RAM, avoiding disk access. If youre going to use RAMJet, the Data cache is redundant and pointless. It should be kept out of the way as much as possible. The best approach is to minimize the delay and effect by keeping the data cache as small as possible.
At the risk of oversimplifying this, most often, increasing the size of the Data cache is like putting a larger bandage on a bleeding wound. Itll keep blood from dripping on the floor sooner, but eventually, the patient will still bleed to death if the bleeding doesnt stop.
5th General Rule of RAMJet: Start Small and Work Your Way Up
Since the Data cache setting is for the most part irrelevant, our advice is to start by leaving the custom caches at their default settings. Start low and observe the error codes. If you get 7xxx errors, increase the structure cache (in 2 MB increments). If you get 5xxx errors, increase the data cache (in 1 MB increments). If either problem persists beyond a doubling of the default settings, stop what youre doing and call technical support.
Now if you absolutely do know it all and you want to get any deeper into this, you should start by making periodic inspections of the Server Info screen during the day. Both caches will fill as the day goes on. The Structure cache needs to be set large enough that it never becomes completely full. We recommend keeping it large enough so there is always 1,000-2,000K of free space. And keep in mind that a good network administrator will keep an eye on changes to the size of the Collection over time and make appropriate adjustments so that the amount of overall RAM devoted to their Helix RADE or Server provides an adequate ceiling as the Collection grows.
6th General Rule of RAMJet: Unless you have enough real RAM to get your entire Collection into RAM, you will probably not derive maximum benefit from the use of this feature.
However, a significant number of Helix users have been using Helix for years and years and have Collections that have grown extremely large. Some of these users--and you know who you are--cant seem to bring themselves to part with any of their information so their Collections never achieve any kind of size stability and just keep on growing. These Collections would probably benefit most from the use of custom caches and RAMJet, provided, of course, that the machines are expandable to the amount of RAM needed and that someone actually buys and installs it. Remember what we said earlier about the price of memory!
Having said this, let us not discourage you from using RAMJet if you dont have enough RAM to totally enclose your Collection. Plenty of users have reported that the Helix Utility runs much faster than before on the same size Collection even though they dont have nearly enough RAM to get the whole thing in there. Just make sure that when you try it, do it on a Collection you have properly tested and backed up.
7th General Rule of RAMJet: Performance and Stability are NOT directly proportional
Theres almost always a trade-off between performance and stability. The faster you drive, the less control you have over your car. As you increase memory allocations you increase stability (to a point) and you decrease performance (again, to a point).
Obviously, too little memory causes problems of its own. but once you've gotten past the 'memory starved' situation of, for example, a 200 MB Collection running on Helix Server with 20 MB allocated, you then face the foggy world of not knowing where the perfect settings are.
Each type of memory (OS, Helix application, RAMJet, Structure cache, Data cache, plus other internal caches) has its own pair of curves, where too little or too much is bad. In between too little and too much is a pretty wide range of acceptable or "workable" values and "just right" is at some unknown spot in between. Because each memory type has its own curve, getting the settings right for one parameter might move another into a danger zone. Finding the compromise that allows everything to function acceptably is the hard part.
One of the purposes of RAMJet feature, besides the obvious RAM disk emulation, is that it gives you control over the Structure and Data caches. The larger and more complex the Collection, the longer the battle will be to find the set of values for your Collection. Working to get those settings optimized is WORK, but in all but the rarest of cases, persistence will pay off.
Bottom line question: How much RAM do I allocate to my Helix application in the Finder?
Depending upon the Helix application youre using (e.g., Helix RADE, Helix Server, etc.), Get Info in the Finder and find out what the minimum memory requirement is for that application. To this number, add the total of your custom caches plus the size of your Collection and then add another 10% or so to give it some breathing room, both for during the day as it uses memory, and over time as it grows in size.