Blender Python `bpy` `__init__.py`, apparently imp

I'm using Mac OS X with PyCharm and Python 3.5.1 and Blender 2.77.

I'm using Blender with Python scripts. I understand that in order to run Python scripts that use Blender (i.e. that imports bpy), I need to run it from command line using blender -b -P /path/to/my_python_script.py (although I don't really know why). That's all fine and it works, but I wish I could just run it from inside Python, because I use these scripts with other non-Blender Python code and I like to use PyCharm to debug and to do pretty much everything. I understand that I can't just run the Blender Python script from PyCharm, but I thought I'd try anyway. I took care to ensure that PyCharm can see the bpy module by editing the "Project Structure" settings in "Preferences" to include the parent directory in which the bpy module lives, which on my machine is /Applications/blender.app/Contents/Resources/2.77/scripts/modules. However, when I try to run the script, it gives ImportError: No module named '_bpy'. I followed the source of the error and the culprit was a line in the __init__.py file in the bpy module (whose location on my machine is /Applications/blender.app/Contents/Resources/2.77/scripts/modules/bpy/__init__.py); the line is:

from _bpy import types, props, app, data, context

So I tried to search for the module _bpy on my machine, and couldn't find it anywhere. So it seems to be importing things from a module that doesn't exist. However, I know that my script works because I've successfully run it in Blender.

So my question is, what witchcraft is going on with the mysterious _bpy module, that neither I nor PyCharm can find, but that the Blender app doesn't have a problem with? I looking to gain a general understanding of what might be going on here, so educated guesses (as well as precise answers obviously) are welcome.

Did you happen to notice the line before from _bpy import... it says # internal blender C module which should be the giveaway.

The witchcraft that makes this work is that the blender binary includes _bpy as a binary python module, blender makes this module accessible within the python interpreter included with blender, it does this during the initialization of the python interpreter. The normal blender binary cannot be imported into a python interpreter outside of blender.

To figure out how to do it you can start with the python docs on python's c-api. You may also want to look through blender's source code within source/blender/python where you will find the c files used to build blender's main c based modules such as bpy, bgl, bmesh, mathutils.

Also see this answer which has a link to info on building blender as a python module so it can be imported (without the gui) into an external python interpreter. If you search for pycharm at blender.stackexchange.com you will find several answers about using bpy in pycharm and eclipse, including ways to have blender running as an external interpreter for debugging.