This is a common error with custom action development, especially if you are working with your own custom installer class, not using wix or some other windows installer derivative.

Since many of the Google entries for this were unresolved, thought I would give it a quick tackle. 😉

First Solution

Throw an exception.

This solution seems to be the most popular among Google results.. You can throw a new Exception or InstallException, but this creates an unsightly error message which you would not want your users to see.

 public override void Install(System.Collections.IDictionary stateSaver)
        {
            Process proc = new Process();
            string path = Path.GetDirectoryName(System.Reflection.Assembly.GetAssembly(typeof(InstallHelper)).Location);
            proc.StartInfo.FileName = path + "\\somesubprocess.exe";
            
//note ProductCode argument is an installer variable
            proc.StartInfo.Arguments = "\"" + this.Context.Parameters["action"] + "\" \"" + this.Context.Parameters["ProductCode"] + "\" \"" + this.Context.Parameters["src"];

            proc.Start();

//optional showwindow code to force window into foreground
 IntPtr msiwindowhandle;
            msiwindowhandle = FindWindow("#32770", InstallTitle);
            ShowWindow(msiwindowhandle, 0);

if (proc.ExitCode == 1)
            {
throw new InstallException("Installation cancelled. Any changes made will now rollback.");
                Process.GetCurrentProcess().Kill();
                if (stateSaver == null)
                {
                    stateSaver = new Dictionary<string, string>();
                    //Add some default values to the dictionary here...if you dare..
                }

                base.Uninstall(stateSaver);
                base.Rollback(stateSaver); //will give you article title error
            }
            else
            {
                base.Install(stateSaver);
            }
}

The code above was a bit much to simply explain the concept of throwing a new “InstallException”, but does illustrate there are definitely things you can do from your custom action bootstrap DLL which have room for error (like spawning a sub process!).

The Alternative

…is to simply click/call the cancel button for the base installer from your custom action code.

Since the msi installer process is separate from your bootstrap custom action DLL, you will have to use some old school tricks to work around the constraints.

Simply import User32.dll and call the “SendMessage” function, passing in the handle of cancel button. See code example below:

using System.Runtime.InteropServices;

[DllImport("User32.dll", EntryPoint = "SendMessage")]
        public static extern int SendMessage(IntPtr hWnd, int Msg, IntPtr wParam, IntPtr lParam);

        [DllImport("user32.dll", SetLastError = true)]
        private static extern IntPtr FindWindowEx(IntPtr parentHandle, IntPtr childAfter, string className, string windowTitle);

        [DllImport("user32.dll", SetLastError = true)]
        public static extern IntPtr SetActiveWindow(IntPtr hWnd);

        [DllImport("user32.dll", SetLastError = true)]
        static extern IntPtr FindWindow(string lpClassName, string lpWindowName);

        [DllImport("user32.dll")]
        static extern bool ShowWindow(IntPtr hWnd, ShowWindowCommands nCmdShow);

        enum ShowWindowCommands : int
        {

            Hide = 0,
            Normal = 1,
            ShowMinimized = 2,
            Maximize = 3,   
            ShowMaximized = 3,
            ShowNoActivate = 4,
            Show = 5,
            Minimize = 6,
            ShowMinNoActive = 7,
            ShowNA = 8,
            Restore = 9,
            ShowDefault = 10,
            ForceMinimize = 11
        }

public static string InstallTitle = "AppbarTitle"; //used to get the window handle

  public override void Install(System.Collections.IDictionary stateSaver)
        {
ShowWindow(msiwindowhandle, ShowWindowCommands.Show);
                IntPtr cancelbuttonhandle;
                const int BM_CLICK = 0x00F5;
                msiwindowhandle = FindWindow("#32770", InstallTitle);
                cancelbuttonhandle = FindWindowEx(msiwindowhandle, IntPtr.Zero, "Button", "Cancel");
                SetActiveWindow(msiwindowhandle); //necessary for button click to fire
                SendMessage(cancelbuttonhandle, BM_CLICK, IntPtr.Zero, IntPtr.Zero);
}

The MSDN Social article listed in my references below comes up in Google rankings high for the error the OP received when calling the Rollback method manually. To track any social momentum my solution may possibly gain, refer to this link.

References
MSDN Social, http://social.msdn.microsoft.com/Forums/en/winformssetup/thread/ea7d09d1-2812-4f04-b6ed-e7293f2a2d75
wix, http://wix.sourceforge.net/
MSDN, “Installer.Rollback Method”, http://msdn.microsoft.com/en-us/library/system.configuration.install.installer.rollback.aspx
MSDN, “Installer.BeforeRollback Method”, http://207.46.16.248/en-us/library/system.configuration.install.installer.beforerollback%28VS.80%29.aspx