I was recently surprised to learn that a Hotfix Rollup shipped from SDL Tridion is something quite different to what you'd expect from the title. For at least the last 10 years, and probably longer, the distinction between a hotfix and a service pack was very simple:

Service Pack

A collection of product improvements shipped between full version releases. The improvements would include bug fixes, and possibly new features, but never "breaking" changes. The intention was that customers should install the latest service pack for their current version. The service pack would have been thoroughly tested by R&D and would be the basis for on-going support until the next release.

Hotfix

If an issue was found in software in the field, a hotfix could be created to address this issue. There wouldn't be an installer - just some files and some instructions. Often a hotfix would be seen as suitable for any customer to install, but other hotfixes were riskier, and if you didn't have the problem, installing the hotfix would be a bad idea. Hotfixes were tested by customer support. The next service pack or full release would supersede any hotfix. In a reasonably thorough risk-management strategy, the standard play was to avoid taking hotfixes until you needed them. The official advice from Tridion as of 2011 was this:

IMPORTANT NOTE: Hotfixes are released at the discretion of SDL Tridion based on technical complexity, customer business requirements and schedules. Hotfixes are made and tested only for the described problem on a particular environment/configuration and therefore should only be installed if approved by SDL Tridion Customer Support. Hotfixes should be replaced as soon as possible by the subsequent service pack where the problem is fixed.

And then along came Hotfix Rollups...

Hotfix rollups

You might be forgiven for thinking that a hotfix rollup was, well a sort of erm... roll-up of hotfixes. A collection of hotfixes. A gathering together of a handy bunch of hotfixes to make life easier for the less risk-averse who like to install everything. (Like me, when I'm installing my own dev image. Love the handiness of it.) That's what the name means in any normal interpretation of the English language. The point here is that this is not what SDL Tridion mean when they say Hotfix Rollup. From discussions with various SDL people, it seems that they see a hotfix rollup as having the following characteristics:

• It is not expected to cause any problems on your system and can safely be installed.
• To this end, it has been tested by the relevant specialists in R&D
• In the same way that you are expected to install a service pack, you are expected to install a hotfix rollup. Should further hotfixes become necessary, they will have the hotfix rollup as a dependency, not specific hotfixes. (This means that if you need that hotfix, you'll end up installing the hotfix rollup too, probably at a moment that you'd prefer to have chosen yourself.)

This is my best understanding at the current moment, but I am not aware of any formal communication from SDL that makes this clear, or otherwise updates the advice from 2011. Obviously, feel free to get formal confirmation via the usual channels

And as for you, SDL: your customers' risks are not your risks. You owe it to your customers to communicate correctly and in a timely way about this kind of thing. If anyone thought this would engender trust and confidence, that person was not thinking clearly. I wouldn't be saying this, but people out in the field often spend significant effort trying to balance risks like this, and it's in all our interests to make sure it goes well.

When designing and implementing web-content-managed web sites with Tridion, the usual arrangement is to have at least four distinct environments, designated for specific purposes. Development, Test, Acceptance, Production. Often we refer to this as the DTAP street. Each environment has its own peculiarities. The production environment serves web pages to the visiting public, so will have at least some servers in the "demilitarized zone". There will probably be multiple web servers behind a load balancer, and particular attention will be paid to defences against the ne'er-do-wells of the Internet. The acceptance environment will be used for the final testing of software releases before they are allowed on to the production system. If the production system is load-balanced, so will the acceptance system be, and lots of attention will be paid to ensuring that the A-environment is a truly representative copy of P. The hardware will be close to identical, and all software will be patched to exactly the same levels as on the production system (unless, of course, a patch is being rolled out, in which case this will take place first in A). The other two environments belong to the development team. The Test environment is used for testing during a development project, to ensure that the necessary quality levels are achieved before moving on to acceptance testing. New versions of the software may be frequently deployed - perhaps daily or several times a day. In general, the environment will be maintained to be a good representation of the Production environments, but not to quite the same levels of obsession as for the Acceptance environment. The Development environment will be quite similar to the Test environment, but is likely to have extra software installed for use in development. Programming software, automated build and test software - that kind of thing. Typically the programmers will have more access privileges in the Development environment than in the other environments. Depending on the organisation, they may well be system administrators in D, and have significant privileges in T. This makes sense, because often they will need to try new approaches, and set up new configurations, or perhaps they might need to attach a debugger to the running software to analyse its processing.

All of this adds up to a significant investment. There will be an entire team of people busy for quite a while to get this set up and to maintain it. Hardware (although often virtualised these days - still complex enough), software - operating systems, databases, security, etc, etc, Then you have to add in all the work of simply managing the whole thing. It's not cheap, and then on top of that, the licenses usually aren't free. So there's a temptation to cut corners. This can mean missing out an environment entirely, or even two - although even the most miserly will usually draw the line at doing development work on the same system that serves the public. It can also mean taking shortcuts in configuration expenses. Maybe you can't afford to have your system administrator spend his time making special configurations for the development environment. The thing is, making sure everything is done right can be unpalatably expensive. So, of course, the first thing to do is ensure that such an expensive set-up is a good fit for your needs. For the vast majority of web sites, you definitely don't need a high-end enterprise web content management system like Tridion. If you do, however, then it's probably a pretty good sign that the expense of running a proper DTAP street is also worth it.

But what if you want all that goodness without having to pay for it? Well in that case, the responsible technicians need to make it clear what the trade-offs are. You can save money, but it's a gamble. The problem is that getting these things working doesn't just cost money. In an emergency, you can usually get more of that. The trouble is, that it also costs time, and the definition of an emergency is that you don't have any of that spare. So - imagine a situation where you would like to be able to debug a problematic piece of software, but your security requirements are pretty heavy, and cast in procedural concrete. So you attempt to set up the necessary tools in your development system, but it doesn't work. To get it working, you estimate you'll need to spend a couple of days of research (say - a day each for a developer and a sysadmin). Maybe it's twice that, maybe it's half, and maybe you need to write a report on all the possible approaches, and have it approved by a committee of architects. Whatever - it's more expensive than you'd like... so you choose not to do it. This is the point at which clear communication is essential.

Living, as I do, in Amsterdam, I can't help feeling just a bit smug as I listen to the news on the radio. In England, the Somerset Levels have been flooded for a long time, and it's still raining. The amount of rain landing on the South of England, and on Wales just now is more than normal - that is to say, it only comes down like that a few times in a century. The people of the Somerset Levels are complaining vociferously that maybe they'd have stood for being flooded for a week, but the water won't go away so they've had it for weeks and weeks. Now some people near London are getting their feet wet too, so suddenly it's important. :-) On my way home, I felt some of that weather. The rain was lashing down, with a pretty solid wind behind it. Was I worried? Not in the slightest, even though I live at least as far below sea level as the people of the levels. You see, here, if the weather gets like that, the only real effect you'll see, is perhaps some smoke coming out of the chimney at your local friendly pumping station. The whole landscape is littered with places for water to go. Every little canal or pool has big sloping sides that will accommodate several times the normal amount of water.

So - when it rains in Somerset (nothing personal, folks), they get their feet very wet indeed, and have some very uncomplimentary things to say about the government's Environment Agency, whose job it is to build and maintain the DTAP street. Various government agencies turn up to provide sandbags. When it rains here, the pumps kick in, and we're good. Sometimes I find it hard to articulate to budget holders exactly why I'd like them to spend money on the odd pumping station that's never really going to get used, is it? I mean come on, what are the chances? Did I say pumping station?

Seriously - if you're going to cut corners on your infrastructure, make sure all your stakeholders know the difference between Somerset and Amsterdam.