Scrapy weird output

I have a scrapy spider that parses this link

My spider looks as follows:

from scrapy.spider import BaseSpider
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.http import request
from scrapy.selector import HtmlXPathSelector
from medsynergies.items import MedsynergiesItem

class methodistspider(BaseSpider):

    name="samplemedsynergies"
    allowed_domains=['msi-openhire.silkroad.com/epostings/']
    start_urls=['https://msi-openhire.silkroad.com/epostings/index.cfm?fuseaction=app.jobinfo&jobid=1284&source=ONLINE&JobOwner=992700&company_id=16616&version=1&byBusinessUnit=NULL&bycountry=0&bystate=0&byRegion=&bylocation=NULL&keywords=&byCat=NULL&proximityCountry=&postalCode=&radiusDistance=&isKilometers=&tosearch=yes']

    #rules=(
    #Rule(SgmlLinkExtractor(allow=("epostings/index.cfm?fuseaction=app%2Ejobsearch&company_id",))),
    #Rule(SgmlLinkExtractor(allow=("epostings/index.cfm?fuseaction=app.jobinfo&jobid",)),callback="parse_job",follow=True),
    #)

    def parse(self, response):
        hxs=HtmlXPathSelector(response)
    titles=hxs.select('//*[@id="jobDesciptionDiv"]')
    items = []

    for titles in titles:
        item=MedsynergiesItem()
        item['job_id']=response.url
        item['title']=titles.select('//*[@id="jobTitleDiv"]/text()').extract()
        item['tracking_code']=titles.select('//*[@id="trackCodeDiv"]/text()').extract()
        item['job_description']=titles.select('.//p/text()').extract()
        item['responsibilities']=titles.select('.//ul/li/text()').extract()
        item['required_skills']=titles.select('//*[@id="jobRequiredSkillsDiv"]/ul/text()').extract()
        item['job_location']=titles.select('//*[@id="jobPositionLocationDiv"]/text()').extract()
        item['position_type']=titles.select('//*[@id="translatedJobPostingTypeDiv"]/text()').extract()
        items.append(item)
    print items
    return items

The output i get looks like the following:

> [{'job_description': [u'The Operations Solution Architect creates the
> technical vision for Revenue Cycle Management delivery capabilities,
> ensuring that interdependent applications and infrastructures are
> aligned. The SA effectively translates business needs into supportable
> solutions that deliver an excellent customer experience.',
>                      u'Responsibilities:'],  'job_id': 'https://msi-openhire.silkroad.com/epostings/index.cfm?fuseaction=app.jobinfo&jobid=1284&source=ONLINE&JobOwner=992700&company_id=16616&version=1&byBusinessUnit=NULL&bycountry=0&bystate=0&byRegion=&bylocation=NULL&keywords=&byCat=NULL&proximityCountry=&postalCode=&radiusDistance=&isKilometers=&tosearch=yes',
> 'job_location': [u'\r\n\t\t\t\t\t\tIrving, Texas, United
> States\r\n\t\t\t\t\t'],  'position_type':
> [u'\r\n\t\t\t\t\t\tFull-Time/Regular\r\n\t\t\t\t\t'], 
> 'required_skills': [u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n'],  'responsibilities': [u'Utilizes technical expertise to create strategic technical vision and
> architecting solutions for Revenue Cycle Manage delivery
> capabilities.',
>                       u'Responsible for gathering requirements, architecting the overall design, and executing the design and build
> phases to ensure RCM solutions and related infrastructures are
> effectively aligned.',
>                       u'Defines key milestones and deliverables related to new developments in collaboration with senior management
> and stakeholders.',
>                       u'Collaborates with Solutions Design, ITS and Operations Implementation team to define, design, price and execute
> new service requirements, new customer accounts, and expanded scope of
> services.',
>                       u'Develops portfolio strategic plan to ensure alignment with Industry trends and market needs to retaining
> MedSynergies industry leadership status.',
>                       u'Provides analysis, opportunity assessments and recommendations to optimize and profitably grow portfolio in alignment
> with established business strategy and goals.\xa0',
>                       u'Performs risk evaluations to ensure that business strategies and evaluations are implemented with clarity and
> consistency.',
>                       u'Serves as senior subject matter expert on content, processes, and procedures for applicable portfolio
> offerings.',
>                       u'Tracks project milestones and deliverables. Develops and delivers progress reports presentations to stake holders
> and senior management',
>                       u'Assists with the transfer of knowledge of technical skills. Provides coaching to less experienced employees.',
>                       u'Participates in special projects and/or completes other duties as assigned.'],  'title':
> [u'\r\n\t\t\t\t\tSolutions Architect\r\n\t\t\t\t'],  'tracking_code':
> [u'\r\n\t\t\t\t\t\tTracking Code\r\n\t\t\t\t\t']}]

So my question is: I would like to know if there is a better way of defining my xpaths so that i do not get the newline(\n) and tab(\t) characters in my output. Also the required_skills Field was not able to scrape any text from the field. I would like to know where i have the mistake.

Thank you in advance!

回答1:

If you know you can expect 1 output string value from an XPath expression, you can wrap your XPath in normalize-space(). Also, within the for title in titles loop, you should use relative XPath expressions (starting with .//, not absolute XPath expressions starting with //)

For example:

item['tracking_code']=titles.select('normalize-space(.//*[@id="trackCodeDiv"]/text())').extract()

For required_skills, I suggest you try normalize-space(.//*[@id="jobRequiredSkillsDiv"]/ul):

item['required_skills']=titles.select('normalize-space(.//*[@id="jobRequiredSkillsDiv"]/ul)').extract()

回答2:

You can clean it with python:

def clean(item):
    data = {}
    for k, v in item.iteritems():
        data[k] = ' '.join([val.strip() for val in v]).strip()
    return data