Py_InitModule4() returns NULL on embedded Python-i

I'm writing a C-Plugin using Cython. In final, this should embed the Python-Interpreter into iTunes on Windows. In order for this to work, a bootstrapper is needed. It implements the iTunes plugin entry-point, initializes or finalizes, or whatever is needed, to then call code generated from Cython.

I'm using MinGW gcc 4.6.2 on Windows 7 64Bit and CPython 2.7.

Preamble

The iTunes plugin entry-point on Windows is called iTunesPluginMain. As far as I understood it, the shared library that implements the plugin is not kept all the time iTunes is running and therefore you can not store any global-variables once the entry-point was called. That's why iTunes want's the developer to store a void* pointer to a handle that is passed every call to iTunesPluginMain.

iTunesPluginMain is called for several notifications, such as initialization and cleanup.

The bootstrapper looks like this:

/** coding: utf-8
    file:   bootstrap.c
    Copyright (c) 2012 by Niklas Rosenstein

    This file implements the bootstrapper for loading the PyTunes plugin. **/

#include <Python.h>
#include <stdlib.h>
#include <windows.h>
#include <iTunesVisualAPI/iTunesAPI.h>

#include "pytunes.c"

#define EXPORT(type) __declspec(dllexport) type

extern void     initpytunes(void);
extern OSStatus PyTunes_Main(OSType, PluginMessageInfo*, void*);

EXPORT(OSStatus) iTunesPluginMain(OSType message, PluginMessageInfo* msgInfo, void* refCon) {

    OSStatus status             = unimpErr;
    char     handlePyMain       = 1;

    switch(message) {
        case kPluginInitMessage: {
            // Sent to notify the plugin that this is the first time it is loaded
            // and should register itself to iTunes

            char** argv = malloc(sizeof(char*) * 1);
            argv[0]     = malloc(sizeof(char)  * 256);

            // WinAPI call, retrieves the path to iTunes.exe
            GetModuleFileName(0, argv[0], 256);

            // Initialize the Python-interpreter
            Py_SetProgramName(argv[0]);
            PyEval_InitThreads();
            Py_Initialize();
            PySys_SetArgvEx(1, argv, 0);
            handlePyMain = 1;

            free(argv[0]);
            free(argv);
            break;
        }

        case kPluginCleanupMessage: {
            // Sent to cleanup the memory when the plugin gets unload

            status       = PyTunes_Main(message, msgInfo, refCon);
            handlePyMain = 0;
            Py_Finalize();
            break;
        }

        default: break;
    }

    if (handlePyMain != 0) {
        initpytunes();
        status = PyTunes_Main(message, msgInfo, refCon);
    }

    return status;
}

pytunes.c is generated by Cython. Now what the bootstrapper does, or should do, is the following:

Identify what iTunes wants to tell the plugin
- If iTunes notifies it about the initialization, it retrieves the path to iTunes.exe through a Windows API call and initializes the Python interpreter.
- If iTunes notifies it to cleanup (e.g. iTunes closes), it finalizes the Python interpreter. Note that the "Cython-call" is done before this happens and handlePyMain is set to zero so it isn't executed again when the interpreter is already finalized.
If handlePyMain is not set to zero, which indicates the Cython-call should not be executed, initpytunes and PyTunes_Main, which are generated from Cython, are called. The call to initpytunes is necessary as Cython does initializations to global variables there. PyTunes_Main is finally the Cython implementation of what the plugin does.

The Cython Implementation

The call to PyTunes_Main, which is implemented in pytunes.pyx, runs smooth. The following implementation opens a file at my desktop and writes a message into it.

cimport iTunesVisualAPI     as itapi

cdef public itapi.OSStatus PyTunes_Main(itapi.OSType message,
                                        itapi.PluginMessageInfo* msgInfo,
                                        void* refCon):
    fl = open("C:/Users/niklas/Desktop/feedback.txt", "w")
    print >> fl, "Greetings from PyTunes!"
    fl.close()
    return itapi.unimpErr

When i start iTunes, the file is created and the text is written into it.

iTunesVisalAPI.pxd contains the cdef extern from "iTunesVisualAPI/iTunesVisualAPI.h" declarations to make the API available for Cython, but that is of lesser importance here.

Problem description

The problem occurs, just for example, when importing the sys module in Cython and using it. Simple example:

cimport iTunesVisualAPI     as itapi

import sys

cdef public itapi.OSStatus PyTunes_Main(itapi.OSType message,
                                        itapi.PluginMessageInfo* msgInfo,
                                        void* refCon):
    fl = open("C:/Users/niklas/Desktop/feedback.txt", "w")
    print >> fl, sys
    fl.close()
    return itapi.unimpErr

This causes iTunes to crash. This is the full gdb-session, which will tell us what the problem actually is.

C:\Program Files (x86)\iTunes>gdb -q iTunes.exe
Reading symbols from c:\program files (x86)\itunes\iTunes.exe...(no debugging symbols found)...done.
(gdb) b pytunes.c:553
No symbol table is loaded.  Use the "file" command.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (pytunes.c:553) pending.
(gdb) r
Starting program: c:\program files (x86)\itunes\iTunes.exe
[New Thread 3244.0x3a8]
[New Thread 3244.0xd90]
[New Thread 3244.0x11c0]
[New Thread 3244.0x125c]
[New Thread 3244.0x1354]
[New Thread 3244.0x690]
[New Thread 3244.0x3d8]
[New Thread 3244.0xdb8]
[New Thread 3244.0xe74]
[New Thread 3244.0xf2c]
[New Thread 3244.0x13c0]
[New Thread 3244.0x1038]
[New Thread 3244.0x12b4]
[New Thread 3244.0x101c]
[New Thread 3244.0x10b0]
[New Thread 3244.0x140]
[New Thread 3244.0x10e4]
[New Thread 3244.0x848]
[New Thread 3244.0x1b0]
[New Thread 3244.0xc84]
[New Thread 3244.0xd5c]
[New Thread 3244.0x12dc]
[New Thread 3244.0x12fc]
[New Thread 3244.0xf84]
warning: ASL checking for logging parameters in environment variable "iTunes.exe.log"

warning: ASL checking for logging parameters in environment variable "asl.log"

BFD: C:\Windows\SysWOW64\WMVCORE.DLL: Warning: Ignoring section flag IMAGE_SCN_MEM_NOT_PAGED in section .reloc

Breakpoint 1, PyTunes_Main (__pyx_v_message=1768843636, __pyx_v_msgInfo=0xd7e798, __pyx_v_refCon=0x0)
    at C:/Users/niklas/Desktop/pytunes/pytunes/build-cython/pytunes.c:553
553       __pyx_t_1 = __Pyx_GetName(__pyx_m, __pyx_n_s__sys); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno
 = 75; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
(gdb) print __pyx_m
$1 = (PyObject *) 0x0
(gdb) print __pyx_n_s__sys
$2 = (PyObject *) 0x92f42c0
(gdb) print __pyx_t_1
$3 = (PyObject *) 0x0
(gdb) step
__Pyx_GetName (dict=0x0, name=0x92f42c0) at C:/Users/niklas/Desktop/pytunes/pytunes/build-cython/pytunes.c:788
788         result = PyObject_GetAttr(dict, name);
(gdb) step

Program received signal SIGSEGV, Segmentation fault.
0x1e089f57 in python27!PyObject_GetAttr () from C:\Windows\SysWOW64\python27.dll
(gdb)

Sidenote: Line 553 is the line where the Python statement print >> fl, sys was handled by Cython. You can find the full generated source-code of pytunes.c on paste.pocoo.org.

The debug-session tells us that __pyx_m_t is used in context with using the sys module in the Cython code (why ever?). Anyway, it is a NULL-pointer. It should be initialized on line 699. Py_InitModule4 obviously returns NULL and therefore an ImportError should be raised within initpytunes. (You can find the corresponding implementation of goto __pyx_L1_error at line 751).

To check this, I've modified the code a bit, and the result is "positive" in that context.

    if (handlePyMain != 0) {
        initpytunes();
        if (PyErr_Occurred()) {
            PyObject* exception, *value, *traceback;
            PyErr_Fetch(&exception, &value, &traceback);
            PyObject* errString = PyObject_Str(exception);

            // WinAPI call
            MessageBox(NULL, PyString_AsString(errString), "PyErr_Occurred()?", 0);
            status = paramErr;
        }
        else {
            // WinAPI call
            MessageBox(NULL, "No error, calling PyTunes_Main.", "PyPyErr_Occurred()?", 0);
            status = PyTunes_Main(message, msgInfo, refCon);
        }
    }

The Question

Do you know or have an idea what I am doing wrong? Maybe I initialize the Python-interpreter wrong? The most bizarre part is, I had a working prototype of this. But I can't get it to work anymore! (see below)

References and Notes

You may want to see the full source. You can find the working prototype here (Virustotal) and the actual project here (Virustotal). (Both, links to mediafire.com)

As I am not allowed to distribute the iTunesVisualSDK with it, here is a link to download it from apple.com.

Please do not comment "Why not work with the prototype?" or alike. It is a prototype and I write prototypes dirty and unclean, and usually I achieve better results when rewriting the whole thing.

Thanks for anyone looking at this, reading it carefully and investing time into helping me to solve my problem. :-)
-Niklas

An ImportError indicates that Python could not import a module. Check the value of the exception to see which module wasn't found.

Module init functions return void so you should always call PyErr_Occurred() after it to check if it failed. If an error occurred, you have to handle it, preferably by showing it to the user in some way. If stdout is available, PyErr_Print() will print out a complete traceback.

I'm not sure how iTunes plugins work but if it indeed unloads DLLs between calls then Python will be unloaded as well and its state will be lost. You would need to call Py_Initialize() and Py_Finalize() in every call to iTunesPluginMain() which means that all your Python objects would be lost as well. Most likely not what you want.

One idea to prevent that could be to re-open your plugin DLL in kPluginInitMessage and close it in kPluginCleanupMessage. Windows keeps track of how many times a DLL has been opened by a process. A DLL is unloaded only after the count reaches 0. So if you call LoadLibrary() in your DLL, the count will increase to 2 and the DLL will be unloaded only after both iTunes and your code call FreeLibrary().

Note that this is just an (untested) idea.