With rvest, how to extract html contents from the

2019-05-27 01:14发布

站内问答 / 前沿技术
1941 2 6

I am trying to download some traffic data from pems.dot.ca.gov, following this topic. 

rm(list=ls())
library(rvest)
library(xml2)
library(httr)
url <- "http://pems.dot.ca.gov/?report_form=1&dnode=tmgs&content=tmg_volumes&tab=tmg_vol_ts&export=&tmg_station_id=74250&s_time_id=1369094400&s_time_id_f=05%2F21%2F2013&e_time_id=1371772740&e_time_id_f=06%2F20%2F2013&tod=all&tod_from=0&tod_to=0&dow_5=on&dow_6=on&tmg_sub_id=all&q=obs_flow&gn=hour&html.x=34&html.y=8"
pgsession <- html_session(url)
pgform <-html_form(pgsession)[[1]]
filled_form <- set_values(pgform,
                          'username' = 'omitted',
                          'password' = 'omitted')
resp = submit_form(pgsession, filled_form)
resp_2 = resp$response
cont = resp_2$content

I checked the class() of these items and found that the resp is a 'session', resp_2 is a 'response', and cont is 'raw'. My question is: how can I extract the html content correctly so that I can proceed with XPath to pick out the actual data I want from this page? My intuition is that I should parse the resp_2 which is a response, but I just can not make it work. Your help are highly appreciated!

2条回答
再贱就再见
2楼-- · 2019-05-27 01:50

This should do it:

pg <- content(resp$response)

html_nodes(pg, "table.inlayTable") %>% 
  html_table() -> tab

head(tab[[1]])
##                 X1      X2           X3           X4
## 1                          Data Quality Data Quality
## 2             Hour 8 Lanes   % Observed  % Estimated
## 3 05/24/2013 00:00   1,311           50            0
## 4 05/24/2013 01:00     729           50            0
## 5 05/24/2013 02:00     399           50            0
## 6 05/24/2013 03:00     487           50            0

(you'll obviously need to modify the column names)

查看更多
0人赞 添加讨论(0) 举报
神经病院院长
3楼-- · 2019-05-27 01:54

You need httr::content, which parses a response into content, which in this case is HTML that can easily be parsed with rvest:

resp_2 %>% content()
## {xml_document}
## <html style="height: 100%">
## [1] <head>\n  <!-- public -->\n  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/     ## ...
## [2] <body class="yui-skin-sam public">\n  <div id="maincontainer" style="height: 100%">\n\n      \n\     ## ...
查看更多
0人赞 添加讨论(0) 举报
登录 后发表回答
相关问题
查看全部
相关文章
查看全部
收藏的人(6)