In Python 3.x, why is there not an itertools share

Question

Is the itertools C module included somehow in the main Python binary in 3.x?

Assuming that the C module is built and included, which it appears to be:

>>> import inspect
>>> import itertools
>>>
>>> inspect.getsourcefile(itertools)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 571, in getsourcefile
    filename = getfile(object)
  File "/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/inspect.py", line 518, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module 'itertools' (built-in)> is a built-in module

I can't find an itertools.so for Python 3.x on my system, but there's one for 2.7.

I noted that some other C modules exist as shared objects (locate '.so' | grep -E '^/usr/local/' | grep '.so' e.g. mmap.so) on disk, so what's the deal with itertools? How can I use it if there's not a shared library?

Answer 1

In Python 3, the itertools extension is compiled into the main Python binary:

>>> import sys
>>> 'itertools' in sys.builtin_module_names
True

See the sys.builtin_module_names documentation:

A tuple of strings giving the names of all modules that are compiled into this Python interpreter.

The module was added to the Python binary because it is used very widely in the Python standard library.

The list of modules to include is taken from the Modules/Setup.dist file in the Python distribution; itertools was added together with _collections, as it is a transient dependency of that module. See issue #9545.

Answer 2

There are hints in the makefile that's near the Python wrapper of inspect.py:

/usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/config-3.4m/Makefile

We can see the build rules for the itertools.c source:

1668 Modules/itertoolsmodule.o: $(srcdir)/Modules/itertoolsmodule.c; $(CC) $(PY_CORE_CFLAGS)  -c $(srcdir)/Modules/itertoolsmodule.c -o      Modules/itertoolsmodule.o

And then trace it a little to see that it's being bundled in:

 24 MODOBJS= ..  Modules/itertoolsmodule.o  ... Modules/xxsubtype.o

 462 # objects that get linked into the Python library
 463 LIBRARY_OBJS_OMIT_FROZEN=   \
...
 470         $(MODOBJS)
 471
 472 LIBRARY_OBJS=   \
 473         $(LIBRARY_OBJS_OMIT_FROZEN) \
 474         Python/frozen.o
 ...
 553 # Build the interpreter
 554 $(BUILDPYTHON): Modules/python.o $(LIBRARY) $(LDLIBRARY) $(PY3LIBRARY)
 555     $(LINKCC) $(PY_LDFLAGS) $(LINKFORSHARED) -o $@ Modules/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) $(LDLAST)
 556
 557 platform: $(BUILDPYTHON) pybuilddir.txt
 558     $(RUNSHARED) $(PYTHON_FOR_BUILD) -c 'import sys ; from sysconfig import get_platform ; print(get_platform()+"-"+sys.version[0:     3])' >platform
 589 # Build static library
 ...
 598     $(AR) $(ARFLAGS) $@ $(MODOBJS)
 599     $(RANLIB) $@

 944 $(LIBRARY_OBJS) $(MODOBJS) Modules/python.o: $(PYTHON_HEADERS)

Or if made via distutils, the path will be something like: /usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/python3.4/_sysconfigdata.py

And assuming that this gets built into a dynamic library:

Ξ ~ → strings /usr/local/Cellar/python3/3.4.3_2/Frameworks/Python.framework/Versions/3.4/lib/libpython3.4.dylib | grep itertools
itertools
itertools._tee_dataobject
itertools._tee
itertools._grouper
itertools.groupby
itertools.repeat
itertools.product
...

Which means that at build time, the itertools.c module gets included in the libpython dynamic library, meaning that it's now part of the standard library.