NameError: name 'N_TOKENS' is not defined

Question

I am new on Python and just got around to install PyCharm for Windows. Downloaded some sample code from Skype for testing their SkypeKit API. But... As soon as I hit the debug button, I get this: (I have Python 2.7 and Django 1.4 installed)

Traceback (most recent call last):
  File "C:\Program Files (x86)\JetBrains\PyCharm 2.0.2\helpers\pydev\pydevd.py", line 2, in <module>
    from django_debug import DjangoLineBreakpoint
  File "C:\Program Files (x86)\JetBrains\PyCharm 2.0.2\helpers\pydev\django_debug.py", line 1, in <module>
    import inspect
  File "C:\Program Files (x86)\Python27\lib\inspect.py", line 39, in <module>
    import tokenize
  File "C:\Program Files (x86)\Python27\lib\tokenize.py", line 38, in <module>
    COMMENT = N_TOKENS
NameError: name 'N_TOKENS' is not defined

Process finished with exit code 1

What does this mean and what can I do to fix it?

Answer 1

In my case, I had created as a parsing exercise a file named token.py that caused the initial namespace conflict. However, simply renaming the file in the Eclipse project doesn't always fix the problem. Sometimes a compiled bytecode file with a *.pyc ending is created when running python. You may also have to find and delete that using filesystem commands.

Answer 2

Maybe you have named a module of your program "token". Rename that and change your import statements and you may fix your issue. If it is the developer's fault, point them to my answer please. I had the same error while trying to fix pylaga on sourceforge, and solved it that way. In that case, the python import path was also being manipulated, which may have forced python to import the program's token.py instead of its own and failed to tokenize things (in fact, N_TOKENS is defined in Python's own token.py, such as /usr/lib64/python2.7/token.py).

Answer 3

I saw this when I had activated a virtualenv, but had not changed to the venv directory. Attempting to import tokenize (from the zip of the system modules) was failing.

Changing my working directory to the root of the virtual environment solved the issue (presumably a relative path problem).

Answer 4

it worked for me.

1. rename file /usr/lib/python2.7/token.py to /usr/lib/python2.7/token2.py
2. open /usr/lib/python2.7/tokenize.py
3. change from token import * to from token2 import *

done.

Answer 5

The tokenize.py module is probably loading the wrong token.py module. See error importing numpy. Solution 1) rename the new token.py (token2.py) and update references to it in tokenize.py etc. Solution 2) if the new token.py is in a python package you can disambiguate the import statement:

import CorrectPythonPackage.token as token2

#or 

from CorrectPythonPackage.token import *

Where CorrectPythonPackage is the folder name containing the token.py file.