Import data from URL

Question

The St. Louis Federal Reserve Bank has a great set of data available on a variety of their web pages, such as:

http://research.stlouisfed.org/fred2/series/OILPRICE/downloaddata?cid=32217 http://www.federalreserve.gov/releases/h10/summary/default.htm http://research.stlouisfed.org/fred2/series/DGS20

The data sets get updated, some as often as daily. I tend to have an interest in the daily data (see the above settings on the URLS)

I'd like to import these kinds of price or rate data streams (accessible as CSV or Excel files at the above URLs) directly into Mathematica.

I've looked at the documentation on Importing[] but I find scant documentation (actually none) on how to go about something like this.

It looks like I need to navigate to the pages, send some data to select specific files and formats, trigger the download, then access the downloaded data from my own machine. Even better if I could access the data directly from the sites.

I had hoped Wolfram Alpha might make this sort thing easy, but I haven't had any success.

FinancialData[] would seem natural for this sort of thing, but I don't see anyway to do it. Financial data has lots of features, but I don't see a way yo get this sort of thing.

Does anyone have any experience with this or can someone point me in the right direction?

Answer 1

You can Import directly from a URL. For example, the data from federalreserve.gov can be obtained and visualized as follows.

url = "http://www.federalreserve.gov/datadownload/Output.aspx?";
url = url<>"rel=H10&series=a660e724c705cea4b7bd1d1b85789862&lastObs=&";
url = url<>"from=&to=&filetype=csv&label=include&layout=seriescolumn";
data = Import[url, "CSV"];
DateListPlot[data[[7 ;;]], Joined -> True]

I broke up url for convenience, since it's so long. I had to examine the contents of data before I knew exactly how to plot it - a step that is typically necessary. I'm sure that the data from stlouisfed.org can be obtained in a similar way, but it requires the use of an API with key to access it.

Answer 2

discussion on how to do this with your API key here:

http://library.wolfram.com/infocenter/MathSource/7583/

the function is based on the API documentation. I haven't looked at the code for a couple of years and from memory I put it together rather quickly but I have used it regularly for over 2 years without problems. Here is an example for monthly non seasonally adjusted retail sales from early 1992 to now:

wolfram alpha also uses FRED data so you could use that as an alternative to direct import but it is more tricky to get the query right. I prefer to use FRED directly. Also from memory the data is only available on alpha the day after the release, which is not what you would typically want.

Answer 3

As Mark said, you can get the data directly from a URL. Your oil data can be imported from a different URL than you had:

http://research.stlouisfed.org/fred2/data/OILPRICE.txt

With that URL, you can do this:

oil = Import["http://research.stlouisfed.org/fred2/data/OILPRICE.txt",
"Table", "HeaderLines" -> 12, "DateStringFormat" -> {"Year", "Month", "Day"}];
DateListPlot[oil, Joined -> True, PlotRange -> All]

Note that "HeaderLines"->12 option strips off the header text in the first 12 lines (you have to count the header lines to know how many to remove). I've also specified the date format.

To find that URL, do as you did before, but click on a data series and then choose View Data from the menu on the left when you see the chart.

Answer 4

The documentation has a short example on extracting data out of a webpage:

http://reference.wolfram.com/mathematica/howto/CleanUpDataImportedFromAWebsite.html

Of course, what actually needs to be done will vary significantly from page to page.