Python3 Error: TypeError: Can't convert 'b

Question

I am working on exercise 41 in learnpythonthehardway and keep getting the error:

  Traceback (most recent call last):
  File ".\url.py", line 72, in <module>
    question, answer = convert(snippet, phrase)
  File ".\url.py", line 50, in convert
    result = result.replace("###", word, 1)
TypeError: Can't convert 'bytes' object to str implicitly

I am using python3 while the books uses python2, so I have made some changes. Here is the script:

#!/usr/bin/python
# Filename: urllib.py

import random
from random import shuffle
from urllib.request import urlopen
import sys

WORD_URL = "http://learncodethehardway.org/words.txt"
WORDS = []

PHRASES = {
            "class ###(###):":
                "Make a class named ### that is-a ###.",
            "class ###(object):\n\tdef __init__(self, ***)" :
                "class ### has-a __init__ that takes self and *** parameters.",
            "class ###(object):\n\tdef ***(self, @@@)":
                "class ### has-a funciton named *** that takes self and @@@ parameters.",
            "*** = ###()":
                "Set *** to an instance of class ###.",
            "***.*** = '***'":
                "From *** get the *** attribute and set it to '***'."
}

# do they want to drill phrases first
PHRASE_FIRST = False
if len(sys.argv) == 2 and sys.argv[1] == "english":
    PHRASE_FIRST = True

# load up the words from the website
for word in urlopen(WORD_URL).readlines():
    WORDS.append(word.strip())

def convert(snippet, phrase):
    class_names = [w.capitalize() for w in
                    random.sample(WORDS, snippet.count("###"))]
    other_names = random.sample(WORDS, snippet.count("***"))
    results = []
    param_names = []

    for i in range(0, snippet.count("@@@")):
        param_count = random.randint(1,3)
        param_names.append(', '.join(random.sample(WORDS, param_count)))

    for sentence in snippet, phrase:
        result = sentence[:]

        # fake class names
        for word in class_names:
            result = result.replace("###", word, 1)

        # fake other names
        for word in other_names:
            result = result.replace("***", word, 1)

        # fake parameter lists
        for word in param_names:
            result = result.replace("@@@", word, 1)

        results.append(result)

    return results

# keep going until they hit CTRL-D
try:
    while True:
        snippets = list(PHRASES.keys())
        random.shuffle(snippets)

        for snippet in snippets:
            phrase = PHRASES[snippet]
            question, answer = convert(snippet, phrase)
            if PHRASE_FIRST:
                question, answer = answer, question

            print(question)

            input("> ")
            print("ANSWER: {}\n\n".format(answer))
except EOFError:
    print("\nBye")

What exactly am I doing wrong here? Thanks!

Answer 1

urlopen() returns a bytes object, to perform string operations over it you should convert it to str first.

for word in urlopen(WORD_URL).readlines():
    WORDS.append(word.strip().decode('utf-8')) # utf-8 works in your case

To get the correct charset : How to download any(!) webpage with correct charset in python?

Answer 2

Explicitly convert byte type 'word' into string

result = result.replace("###", sre(word), 1)

it should work

Answer 3

In Python 3, the urlopen function returns an HTTPResponse object, which acts like a binary file. So, when you do this:

for word in urlopen(WORD_URL).readlines():
    WORDS.append(word.strip())

… you end up with a bunch of bytes objects instead of str objects. So when you do this:

result = result.replace("###", word, 1)

… you end up trying to replace the string "###" within the string result with a bytes object, instead of a str. Hence the error:

TypeError: Can't convert 'bytes' object to str implicitly

The answer is to explicitly decode the words as soon as you get them. To do that, you have to figure out the right encoding from the HTTP headers. How do you do that?

In this case, I read the headers, I can tell that it's ASCII, and it's obviously a static page, so:

for word in urlopen(WORD_URL).readlines():
    WORDS.append(word.strip().decode('ascii'))

But in real life, you usually need to write code that reads the headers and dynamically figures it out. Or, better, install a higher-level library like requests, which does that for you automatically.