Python imports wrong version of sqlite when runnin

I'm trying to set up a Windows slave for our Jenkins server, but some of the unit tests are failing due to a sqlite error.

We're using conda and have built a custom version of sqlite that includes rtree. Then our application is built using this custom sqlite. When we install our app on the windows server, the correct version of sqlite is installed:

$ conda list

sqlite                    3.16.2                    vc9_0  [vc9]  file://[package_server]

Running the app from the command line it works as expected. When we run tests though,

$ nosetests test

...

======================================================================
ERROR: test_vectorise_1 (test.system_test_wallet.test_rasters.test_vectorise.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "farm\rasters\vectorise.py", line 189, in run_vectorise
    proc(progressfn)
  File "farm\process.py", line 1278, in __call__
    for feature in lyr:
  File "C:\bin\Miniconda2\envs\farm_system_test_windows\lib\site-packages\osgeo\
ogr.py", line 2876, in next
    feature = self.GetNextFeature()
  File "C:\bin\Miniconda2\envs\farm_system_test_windows\lib\site-packages\osgeo\
ogr.py", line 1460, in GetNextFeature
    return _ogr.Layer_GetNextFeature(self, *args)
RuntimeError: sqlite3_exec(CREATE VIRTUAL TABLE "rtree_GB_TEST_RP100_VE_5m_27700
_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: no such module: rtree

I added some logging to show python executable path and sqlite3 path and it is the same every time, so it's not a virtualenv issue:

C:\bin\Miniconda2\envs\farm_system_test_windows\python.exe
C:\bin\Miniconda2\envs\farm_system_test_windows\lib\sqlite3\__init__.pyc

Printing the sqlite version though, shows some differences:

print('sql ver', sqlite3.sqlite_version_info)

Running the app from the command line, you get (3, 16, 2), which is the correct version. Running nosetests, you get (3, 8, 11) - an older version and presumably the reason for rtree not being found.

Weirdly, if you call a test file with

python path/to/test.py

and let unittest do its stuff, version (3, 16, 2) is initially imported, but then when the test runs, you get (3, 8, 11) and the test errors.

What is going on here? Why do I get a different version of sqlite depending on how I call the application?

I'm using python 2.7.14, conda 4.4.11 on Windows Server 2012 R2.

UPDATE: I'm still getting this error on Jenkins and now on my dev machine too. It's not just a nose issue, as when I run the test using unittest, I get the error. The PYTHONPATH and sys.path are the same in both cases, and the call stack is identical, except with unittest, there are an extra two frames at the top.

File "C:\ProgramData\Anaconda2\envs\farm\lib\runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "C:\ProgramData\Anaconda2\envs\farm\lib\runpy.py", line 72, in _run_code
exec code in run_globals

Interestingly, the rest of the call stack shows an absolute path for main.py, the entry point, and relative paths for everything else when using unittest. When calling main.py directly however, main.py has a relative path and all the other files have absolute paths.

Does anyone have any idea why we're seeing this difference? It's really frustrating having to skip a load of tests when we're running on windows.

By looking for a solution in sys.path you are looking at the wrong place. You need to see what os.environ['PATH'] is at the critical points here since it's the Windows Loader that has picked the 'wrong' DLL, where by wrong I mean the first one it found according to its well documented DLL lookup procedure.