scrapy, how to separate text within a HTML tag ele

Question

Code containing my data:

        <div id="content"><!-- InstanceBeginEditable name="EditRegion3" -->
      <div id="content_div">
    <div class="title" id="content_title_div"><img src="img/banner_outlets.jpg" width="920" height="157" alt="Outlets" /></div>
    <div id="menu_list">
<table border="0" cellpadding="5" cellspacing="5" width="100%">
    <tbody>
        <tr>
            <td valign="top">
                <p>
                    <span class="foodTitle">Century Square</span><br />
                    2 Tampines Central 5<br />
                    #01-44-47 Century Square<br />
                    Singapore 529509</p>
                <p>
                    <br />
                    <strong>Opening Hours:</strong><br />
                    7am to 12am (Sun-Thu &amp;&nbsp;PH)<br />
                    24 Hours (Fri &amp; Sat&nbsp;&amp;</p>
                <p>
                    Eve of PH)<br />
                    Telephone: [phone]</p>
            </td>
            <td valign="top">
                <img alt="Century Square" src="/assets/images/outlets/century_sq.jpg" style="width: 260px; height: 140px" /></td>
            <td valign="top">
                <span class="foodTitle">Liat Towers</span><br />
                541 Liat towers #01-01<br />
                Orchard Road<br />
                Singapore 238888<br />
                <br />
                <strong>Opening Hours: </strong><br />
                24 hours (Daily)<br />
                <br />
                Telephone: [phone]</td>
            <td valign="top">
                <img alt="Liat Towers" src="/assets/images/outlets/century_liat.jpg" style="width: 260px; height: 140px" /></td>
        </tr>

**i want to get

place name: Century Square, Liat Towers

address : 2 Tampines Central 5, 541 Liat towers #01-01

postal code: Singapore 529509, Singapore 238888

Opening hours: 7-12am, 24 hours daily**

For example:

the first <"p> in '<"td valign="top">' have 3 data which i want (name,adress,postal). How do i split them?

here is my spider code:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
import re
from todo.items import wendyItem

class wendySpider(BaseSpider):
    name = "wendyspider"
    allowed_domains = ["wendys.com.sg"]
    start_urls = ["http://www.wendys.com.sg/outlets.php"]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        values = hxs.select('//td')
        items = []
        for value in values:
            item = wendyItem()
            item['name'] = value.select('//span[@class="foodTitle"]/text()').extract()
            item['address'] = value.select().extract()
            item['postal'] = value.select().extract()
            item['hours'] = value.select().extract()
            item['contact'] = value.select().extract()
            items.append(item)
        return items

Answer 1

I would select all <td valign="top"> which contain a <span class="foodTitle">

//div[@id="menu_list"]//td[@valign="top"][.//span[@class="foodTitle"]]

and then for each one of these td cell, get all text nodes

.//text()

You get something like that:

['\n                ',
 '\n                    ',
 'Century Square',
 '\n                    2 Tampines Central 5',
 '\n                    #01-44-47 Century Square',
 '\n                    Singapore 529509',
 '\n                ',
 '\n                    ',
 'Opening Hours:',
 u'\n                    7am to 12am (Sun-Thu &\xa0PH)',
 u'\n                    24 Hours (Fri & Sat\xa0&',
 '\n                ',
 '\n                    Eve of PH)',
 '\n                    Telephone: [phone]',
 '\n            ']

and

['\n                ',
 'Liat Towers',
 '\n                541 Liat towers #01-01',
 '\n                Orchard Road',
 '\n                Singapore 238888',
 'Opening Hours: ',
 '\n                24 hours (Daily)',
 '\n                Telephone: [phone]']

Some of these text node have a string representation that is all whitespace, so strip them and look for "Opening hours" and "Telephone" keywords to process lines in a loop:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
import re
from todo.items import wendyItem

class wendySpider(BaseSpider):
    name = "wendyspider"
    allowed_domains = ["wendys.com.sg"]
    start_urls = ["http://www.wendys.com.sg/outlets.php"]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        cells = hxs.select('//div[@id="menu_list"]//td[@valign="top"][.//span[@class="foodTitle"]]')
        items = []
        for cell in cells:
            item = wendyItem()

            # get all text nodes
            # some lines are blank so .strip() them
            lines = cell.select('.//text()').extract()
            lines = [l.strip() for l in lines if l.strip()]

            # first non-blank line is the place name
            item['name'] = lines.pop(0)

            # for the other lines, check for "Opening hours" and "Telephone"
            # to store lines in correct list container

            address_lines = []
            hours_lines = []
            telephone_lines = []

            opening_hours = False
            telephone = False

            for line in lines:
                if 'Opening Hours' in line:
                    opening_hours = True
                elif 'Telephone' in line:
                    telephone = True
                if telephone:
                    telephone_lines.append(line)
                elif opening_hours:
                    hours_lines.append(line)
                else:
                    address_lines.append(line)

            # last address line is the postal code + town name
            item['address'] = "\n".join(address_lines[:-1])
            item['postal'] = address_lines[-1]

            # ommit "Opening hours" (first element in list)
            item['hours'] = "\n".join(hours_lines[1:])

            item['contact'] = "\n".join(telephone_lines)

            items.append(item)

        return items