I cannot find the MSI format specification. It says that its an open standard, but I don't see any documentation - just bits and pieces of information here and there.
I want to modify some .msi installer, but I want to understand first what it is doing.
Don't forget MakeMSI which is a free app to create MSI files.
Sounds like you got a misunderstanding of Microsoft's promise.
The full list of the open specifications can be found at
http://www.microsoft.com/openspecifications/en/us/programs/osp/default.aspx
Since MSI is not listed within, you can safely assume that it is a private standard and won't be shared to general public.
Are you sure that .MSI database format is required to modify some installer? Have you heard about the Orca tool? If not, this MS KB article is for you:
http://support.microsoft.com/kb/255905
MSI files are COM structured storage, which is described here: Structured Storage (Windows).
The binary format is not open, I wouldn't even go as far as calling it a standard.
However I assume it's not the file format you're concerned with (which is undocumented), but rather the tables, actions, sequences, etc. that are visible when interfacing with the MSI via the Windows Installer SDK, API, Orca, etc.
Everything is covered in the SDK documentation, however for some reason search engines don't rank the documentation very highly. Also you can no longer download the Windows Installer SDK separately, it is now bundled as part of the Windows SDK.
The MSI file format is not open and is in fact undocumented, however the API required to access these files is part of the Windows Installer SDK. That link describes both the API functions required to access and modify MSI databases (files), and the contents of the tables in an MSI database.
That said if you want to modify or view the contents of an MSI then the easiest way to do it is probably just to use an editor that someone else has made (which will in turn use these functions). The ones that I know of are:
If you really want to know more about the details of the internals of the MSI file format then take a read through of these two blog articles by Rob Mensching (the author of WiX)