How does the MSI Installer InstallValidate determi

I'm trying to work out how to prevent the Restart Manager from detecting that a restart is required when I uninstall my application.

My application has a Windows Service which uses a native DLL (Tanuki Wrapper) and creates another process (Java) which writes to some log files. I'm using WiX Toolset but I'm more interested in the MSI Installer itself. I've been trying some changes manually using Orca. For reference the relevant WiX configuration is (there are some other components for JARs etc that I have omitted):

<DirectoryRef Id="logs3327407xx">
    <Component Guid="344ec345-bdd7-4c1d-801f-55ddf9e07735" Id="logs_wrapper_log88580873x">
        <File DiskId="1" Id="fl_logs_wrapper_log88580873x" Name="wrapper.log" Source="logs\wrapper.log"/>
    </Component>
</DirectoryRef>
<DirectoryRef Id="bin97543xxxx">
    <Component Guid="67c93dd8-36ad-427f-9d79-64a07c719eea" Id="bin_wrapper_windows_x86_64_exe189026768">
        <File DiskId="1" Id="fl_bin_wrapper_windows_x86_64_exe189026768" KeyPath="yes" Name="wrapper-windows-x86-64.exe" Source="bin\wrapper-windows-x86-64.exe"/>
        <ServiceInstall Account="LocalSystem" Arguments="-s &quot;..\conf\wrapper.conf&quot; wrapper.console.flush=true" Description="My Example Service." DisplayName="My Service" ErrorControl="ignore" Id="srvc_i_bin_wrapper_windows_x86_64_exe189026768" Interactive="no" Name="MyService" Start="auto" Type="ownProcess" Vital="yes"/>
        <ServiceControl Id="srvc_c_bin_wrapper_windows_x86_64_exe189026768" Name="MyService" Remove="uninstall" Start="install" Stop="both" Wait="yes"/>
    </Component>
</DirectoryRef>
<DirectoryRef Id="bin_wrapper_native490235675">
    <Component Guid="d7e4295a-1ce5-4dd2-aa92-230caac34247" Id="bin_wrapper_native_wrapper_windows_x86_64_dll156404367">
        <File DiskId="1" Id="fl_bin_wrapper_native_wrapper_windows_x86_64_dll156404367" Name="wrapper-windows-x86-64.dll" Source="bin\wrapper\native\wrapper-windows-x86-64.dll"/>
    </Component>
</DirectoryRef>

I understand that there is logic in the InstallValidate action that determines whether files are in use. It will either use the Restart Manager or the FilesInUse based on the MSIRESTARTMANAGERCONTROL property.

If I use the Restart Manager then it opens the dialog saying a restart is required. The logs say:

MSI (s) (1C:7C) [12:27:14:679]: Doing action: InstallValidate
Action ended 12:27:14: MigrateFeatureStates. Return value 0.
MSI (s) (1C:7C) [12:27:14:679]: PROPERTY CHANGE: Deleting MsiRestartManagerSessionKey property. Its current value is 'f2947dee632d694f8b4f1795ff254092'.
...
MSI (s) (1C:7C) [12:27:14:679]: Component: bin_wrapper_windows_x86_64_exe189026768; Installed: Local;   Request: Absent;   Action: Absent;   Client State: Local
MSI (s) (1C:7C) [12:27:14:679]: Component: bin_wrapper_native_wrapper_windows_x86_64_dll156404367; Installed: Local;   Request: Absent;   Action: Absent;   Client State: Local
MSI (s) (1C:7C) [12:27:14:679]: Component: logs_wrapper_log88580873x; Installed: Local;   Request: Absent;   Action: Absent;   Client State: Local
...
MSI (s) (1C:7C) [12:27:14:741]: RESTART MANAGER: Detected that application with id 11368, friendly name 'java.exe', of type RmCritical and status 1 holds file[s] in use.
MSI (s) (1C:7C) [12:27:14:741]: RESTART MANAGER: Did detect that a critical application holds file[s] in use, so a reboot will be necessary.
MSI (s) (1C:7C) [12:27:14:741]: Note: 1: 1610

It doesn't actually say the files but if I disable the Restart Manager and use the FilesInUse instead then no dialog appears. This time the logs say:

Info 1603. The file C:\...\wrapper-windows-x86-64.exe is being held in use by the following process: Name: wrapper-windows-x86-64, Id: 11004, Window Title: '(not determined yet)'. Close that application and retry.
MSI (s) (1C:8C) [12:33:23:458]: 2 application(s) had been reported to have files in use.
Info 1603. The file C:\...\wrapper-windows-x86-64.dll is being held in use by the following process: Name: java, Id: 8284, Window Title: '(not determined yet)'. Close that application and retry.
MSI (s) (1C:8C) [12:33:23:458]: Note: 1: 2727 2:  
MSI (c) (AC:28) [12:33:23:458]: File In Use: -wrapper-windows-x86-64- Window could not be found. Process ID: 11004
MSI (c) (AC:28) [12:33:23:458]: File In Use: -java- Window could not be found. Process ID: 8284
MSI (c) (AC:28) [12:33:23:458]: No window with title could be found for FilesInUse

Can someone please explain how the InstallValidate determines which files are in use?

Additionally how can I prevent the Restart Manager from saying that files are in use which will be released once the service is stopped?

As a side question, why is the Restart Manager not showing the MsiRMFilesInUse dialog? I have checked all the requirements on MsiRMFilesInUse Dialog and as far as I can tell they are all true.

I remember reading somewhere that using a ServiceControl rather than a custom action helps the Restart Manager know that those Files within that Component are used by the service. I have tried adding the files to that Component but it doesn't seem to make any difference.

The behavior of InstallValidate and in-use files related to ServiceControl isn't complicated. If there are files in use by a service and that service is in the ServiceControl table and marked to be stopped on uninstall then it will ignore those files as far as in-use behavior. Putting files in the same component makes no difference, and obviously Windows can't know that the service code is going to shut down processes that are causing files-in-use situations. To my knowledge there is no way of telling InstallValidate that some files really won't be in-use when they are due to be uninstalled.

You won't get an old-style FilesInUse dialog when Restart Manager isn't used because it requires an active window that the user can be prompted to close to shut down the app. This is the reason tray apps without an active window don't cause a FilesInUse dialog. I can't be sure, but it appears to me that the first log extract referring to Restart Manager actually IS RMFilesInUse, and I think that's the dialog you're seeing.

If the only files-in-use issues you're seeing are related to the java process that the service will shut down then a possible solution is:

Suppress all files-in-use dialogs during uninstall.
Use ServiceControl to shut down your service.
Ensure that your ServiceControl was Wait=1 and that it does not respond to the "stop service" request until the java process has been terminated, assuming you can explicitly shut it down.

Windows isn't so dumb that it will force a reboot just because files were in use at InstallValidate time but that isn't true at actual removal time. So the reboot won't be necessary if you can make sure that the files are no longer in use at the actual time they need removing/replacing/whatever. It's not unusual to see the dialog and then find that a reboot is not in fact necessary. So if you suppress the dialog and get everything shut down you won't see a reboot request.