creating standalone exe using pyinstaller with may

Question

I have a program that helps visualize some data in 3D by plotting a surface and a cloud of points to see how they relate to the surface. For the visualization I am using mayavi since it was easy to set up and does a better job than matplotlib in 3D. The program works great when I run it in my python environment and makes beautiful visualizations. However, I need to distribute this to a few people who don't have Python and would prefer not to have to install python and all the add-ins on each computer, so I have been using pyinstaller to create standalone .exe files that they can run after I develop a program.

For reference, this is all done on Windows 10, Python 3.6, pyqt 4.11.4, pyface 6.0.0, traits 4.6.0, pyinstaller 3.3.1, mayavi 4.5.0+vtk81. Just about every module I use was installed using pip.

The problem is that I can't seem to get a working exe if I use/import the mayavi module. I have been reading so much github documentation on different hook files and hidden-imports and each time I fix one error another pops up. It all started with scipy but I believe I have those worked out. So I have a few questions that could help me solve the problem:

1) Has anyone successfully created a standalone exe using pyinstaller with a mayavi import (specifically from mayavi import mlab)? What is your secret?!?

This seems similar but I haven't been able to figure it out yet... SO_link

I have used the following links (scipy,h5py,pandas,traits/qt4,ETS_TOOLKIT) to add hidden imports or fix other issues but I am stuck now after setting my ETS_TOOLKIT=qt4. Before setting it, I would get a pyface/traits error RuntimeError: No traitsui.toolkits plugin found for toolkit null, but now it says the same thing for qt4 instead of null. I have qt4 installed so I don't understand that... It is in the import_toolkit function in pyface/base_toolkit.py.

2) Is there a better route to go in terms of 3D visualization / exe creation? I need a 3D program that can accurately render if the points are in front of or behind the surface and be able to rotate/zoom/pan interactively, plus it needs to be intuitive. Mayavi had very simple commands similar to matplotlib but others seem very complicated working around how the UI interacts with everything.

3) How do I interpret all of these error codes I get? They are usually pretty cryptic saying a certain line threw an exception nested in 10 other function calls and it seems very difficult to back out where exactly things went wrong, especially when nothing pops up on Google that seems to be related.

EDIT

While I am still very confused, I have been able to change where the error occurs. Following the traceback, I commented out a line setting the marker color in traitsui/editors/code_editor.py (line 49), at which point the exception then started the next time the Color method was called... but I still get the same RuntimeError. So that doesn't tell me much other than I am still looking for what hidden import I need to include for this thing to work.

Also note that I get the exact same error with both PyInstaller and cx_Freeze, in case that helps...

Edit 2

I have now tried it using anaconda for python 2.7, SAME EXACT ISSUE..... I'm starting to believe the universe doesn't want this to happen. Is there somewhere else I should bring this issue up?? I have posted on the traitsui GitHub but that wasn't very helpful. This seems to be bigger than pyinstaller/cx_freeze since it happens in both....

Answer 1

I dealt with the same problem and finally switched to cx_freeze, which now works fine on linux and windows. The problems you are dealing with arise from statements like in the SE answer, you found, i.e. dynamic import statements, where what is imported is only determined at runtime:

    be = 'pyface.ui.%s.' % tk
    __import__(be + 'init')

I couldn't fix that in pyinstaller, while in cx_freeze it works, when you explicitely add the required packages in the build file. Here is the package list I used:

"packages": ["pyface.ui.qt4", "tvtk.vtk_module", "tvtk.pyface.ui.wx", "matplotlib.backends.backend_qt4",'pkg_resources._vendor','pkg_resources.extern','pygments.lexers',
                              'tvtk.pyface.ui.qt4','pyface.qt','pyface.qt.QtGui','pyface.qt.QtCore','numpy','matplotlib','mayavi']

Here is a full build script that works with python3.6, cx_freeze 5.0.2, mayavi 4.5.0+vtk71, traits 4.6.0, pyface 5.1.0 and traitsui 5.1.0.

import os
from cx_Freeze import setup, Executable
import cx_Freeze.hooks
def hack(finder, module):
    return
cx_Freeze.hooks.load_matplotlib = hack
import scipy
import matplotlib

scipy_path = os.path.dirname(scipy.__file__) #use this if you are also using scipy in your application

build_exe_options = {"packages": ["pyface.ui.qt4", "tvtk.vtk_module", "tvtk.pyface.ui.wx", "matplotlib.backends.backend_qt4",'pygments.lexers',
                                  'tvtk.pyface.ui.qt4','pyface.qt','pyface.qt.QtGui','pyface.qt.QtCore','numpy','matplotlib','mayavi'],
                     "include_files": [(str(scipy_path), "scipy"), #for scipy
                    (matplotlib.get_data_path(), "mpl-data"),],
                     "includes":['PyQt4.QtCore','PyQt4.QtGui','mayavi','PyQt4'],
                     'excludes':'Tkinter',
                    "namespace_packages": ['mayavi']
                    }


executables = [
    Executable('main.py', targetName="main.exe",base = 'Win32GUI',)
]

setup(name='main',
      version='1.0',
      description='',
      options = {"build_exe": build_exe_options},
      executables=executables,
      )

I import pyface in the following way:

os.environ['ETS_TOOLKIT'] = 'qt4'

import imp
try:
    imp.find_module('PySide') # test if PySide if available
except ImportError:
    os.environ['QT_API'] = 'pyqt' # signal to pyface that PyQt4 should be used

from pyface.qt import QtGui, QtCore

before importing mayavi