Let's say I'm developing an application that installs COM component and installer registers them. This works great.
Now the software is required to run from a memory stick. How can I register my libraries runtime and make sure, that the registry is cleaned up after running the application?
Are you always running on XP or later? Don't bother. Use "registration free COM".
Essentially what you do is create a manifest file for the COM component so the Windows loader know what its registration is without having to put that in the registry.
A walkthrough of what to do is in this article on MSDN:
Registration-Free Activation of COM Components: A Walkthrough
"Step 6" and "Step 7" in that article show how to create the manifests. If your main program is native, not managed, you may have to make some API calls as well, also explained there.
We can use shell commands to do it. Call Regsvr32 with the dll name using Shell.
I once saw an article about side-by-side COM which might be what you want. Note, however, that it is not available on all versions of Windows.