This page describes techniques for debugging Windows Installer based setups.
Unit tests are your most important tool for avoiding bugs before you even perform a test install. The Windows Installer SDK supplies a suite of over a hundred different specific unit tests for merge modules and MSI databases. The tests evaluate the internal consistency of your database tables and are called internal consistency evaluators or ICEs. The ICEs are described in the Windows Installer SDK documentation under "Package Validation".
You can also write your own ICEs. I recommend that anytime you identify an error in your merge module or MSI database that can be detected by a unit test that you write an ICE to perform this test. This will help you from repeating the same error again and in many cases can be used to detect classes of errors that are likely to recur during ongoing development.
For instance, it is easy to make a change to the CustomAction table that results in a misspelled entry point into a DLL for a DLL-based custom action. Using the MSI API and a C++ ICE you can query the database for the entry point into a DLL custom action. For each entry point you can attempt to perform LoadLibrary on the DLL and then GetProcAddress to obtain the address of the entry point. You can therefore detect spelling errors in DLL entry point names. With this early detection you can avoid costly testing and retesting of your install due to "little" errors that are easy to make.
The SDK documentation describes how to construct an ICE unit test and provides a very simple C++ example. rtICE01 is an example ICE written in VBScript that validates the UIText table.
Logging is your most important tool for debugging your installs. With a verbose log file you can find out the reason for unexpected failures during any phase of your install and additional information about how Windows Installer processes your MSI to make changes to the system.
You can obtain a verbose log for a particular invocation of msiexec with a command-line argument. You can also enable logging for the whole machine through group policy. I recommend you enable logging through group policy because then you will obtain a verbose log for any invocation of msiexec, even those initiated by the system in response to an automatic repair operation.
When you enable logging with group policy, the log file is called MSIxxxx.LOG, where "xxxx" is replaced with a random string. The log file is placed in the %TEMP% folder. An easy way to browse to this folder is to type '%TEMP%' (sans quotes) into the address bar of an open Windows Explorer folder window.
To enable logging through group policy, do Start / Run... / gpedit.msc and then drill down into Local Computer Policy / Computer Configuration / Administrative Templates / Windows Components / Windows Installer / Logging to enable logging with the setting 'voicewarmup' (sans quotes).
You can write to the log file with a call to the API function MsiProcessMessage from C++, or with the Session.Message method from the automation model. If you have a custom action invoked from DoAction in a ControlEvent, then calls to MsiProcessMessage are not written to the log. In this case you may need to use your own log file, or the debug output stream.
The debug output stream is available from C++ by calling the OutputDebugString function. Unfortunately, there is no easy way to call OutputDebugString from script, but you can create a C++ custom action and invoke it through script by Session.DoAction, or you can create a separate COM object for writing to the debug output stream from script.
A useful technique is to write a wrapper for MsiProcessMessage to write strings to both the MSI log and the debug output stream. You can run dbmon.exe from the Platform SDK in parallel with your install. Then as your custom actions write to the MSI log through the wrapper, the message strings also appear in the dbmon window for instant feedback of what is taking place inside your install. You have a permanent record in the MSI log.
If you have a complex custom action with many locations where it could fail, be sure that you always write a unique message to the log that indicates the exact point of the failure. This will help you avoid having to retest your installation to determine the specific point of failure.
If Windows Installer initiates a repair action because a damaged component was detected when a shortcut was launched or COM object activated through a Windows Installer descriptor, then messages will be written into the system event log giving details on the damaged component and feature that initiated the repair operation.
The Debug machine policy allows all logging output to be sent to the debug output stream while the log is being generated. This is very useful for tracking the current state of your install while it is running. DbMon.exe can be used to monitor the output as it is generated.
To test your install, you will need a way to reset your test system to a known good state after each test. You can use virtual machine software like Virtual PC or VMWare or you can use a system imaging technology like Ghost. Resetting your test environment to a known good state after each install test is critical to getting reliable results from your install.
Windows Installer treats an installation like a database transaction: the changes are made and then committed to the system if no failures occur. If a failure occurs, then the changes made to the system so far are rolled back and the system is returned to its original state.
You will want to test your install for unexpected failures to ensure that no permanent modifications are made to the machine after the rollback has completed. You can create a test framework for this by inserting a type 19 (force failure) custom action into your InstallExecuteSequence table just before InstallFinalize. InstallFinalize is the standard action that ends an installation transaction. If you force a failure just before InstallFinalize, then you will be testing rollback at the last possible moment in your install. You can insert the type 19 custom action and conditionalize it on a property that normally evaluates to false, allowing the install to proeed normally. To test rollback, you set this property from the command-line so that the failure path is taken.