Download Kaggle Dataset by using Python

2020-07-11 06:43发布

站内问答 / Python
1667 6 4

I have trying to download the kaggle dataset by using python. However i was facing issues by using the request method and the downloaded output .csv files is a corrupted html files.

import requests

# The direct link to the Kaggle data set
data_url = 'https://www.kaggle.com/crawford/gene-expression/downloads/actual.csv'

# The local path where the data set is saved.
local_filename = "actsual.csv"

# Kaggle Username and Password
kaggle_info = {'UserName': "myUsername", 'Password': "myPassword"}

# Attempts to download the CSV file. Gets rejected because we are not logged in.
r = requests.get(data_url)

# Login to Kaggle and retrieve the data.
r = requests.post(r.url, data = kaggle_info)

# Writes the data to a local file one chunk at a time.
f = open(local_filename, 'wb')
for chunk in r.iter_content(chunk_size = 512 * 1024): # Reads 512KB at a time into memory

    if chunk: # filter out keep-alive new chunks
        f.write(chunk)
f.close()

Output file

<!DOCTYPE html>
<html>
<head>
    <title>Gene expression dataset (Golub et al.) | Kaggle</title>
    <meta charset="utf-8" />
    <meta name="robots" content="index, follow"/>
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">    <meta name="theme-color" content="#008ABC" />
    <link rel="dns-prefetch" href="https://www.google-analytics.com" /><link rel="dns-prefetch" href="https://stats.g.doubleclick.net" /><link rel="dns-prefetch" href="https://js.intercomcdn.com" /><link rel="preload" href="https://az416426.vo.msecnd.net/scripts/a/ai.0.js" as=script /><link rel="dns-prefetch" href="https://kaggle2.blob.core.windows.net" />
    <link href="/content/v/d420a040e581/kaggle/favicon.ico" rel="shortcut icon" type="image/x-icon" />
    <link rel="manifest" href="/static/json/manifest.json">
    <link href="//fonts.googleapis.com/css?family=Open+Sans:400,300,300italic,400italic,600,600italic,700,700italic" rel='stylesheet' type='text/css'>
                    <link rel="stylesheet" type="text/css" href="/static/assets/vendor.css?v=72f4ef2ebe4f"/>
        <link rel="stylesheet" type="text/css" href="/static/assets/app.css?v=d997fa977b65"/>
        <script>

            (function () {
                var originalError = window.onerror;

                window.onerror = function (message, url, lineNumber, columnNumber, error) {
                    var handled = originalError && originalError(message, url, lineNumber, columnNumber, error);
                    var blockedByCors = message && message.toLowerCase().indexOf("script error") >= 0;
                    return handled || blockedByCors;
                };
            })();
        </script>
    <script>
        var appInsights=window.appInsights||function(config){
        function i(config){t[config]=function(){var i=arguments;t.queue.push(function(){t[config].apply(t,i)})}}var t={config:config},u=document,e=window,o="script",s="AuthenticatedUserContext",h="start",c="stop",l="Track",a=l+"Event",v=l+"Page",y=u.createElement(o),r,f;y.src=config.url||"https://az416426.vo.msecnd.net/scripts/a/ai.0.js";u.getElementsByTagName(o)[0].parentNode.appendChild(y);try{t.cookie=u.cookie}catch(p){}for(t.queue=[],t.version="1.0",r=["Event","Exception","Metric","PageView","Trace","Dependency"];r.length;)i("track"+r.pop());return i("set"+s),i("clear"+s),i(h+a),i(c+a),i(h+v),i(c+v),i("flush"),config.disableExceptionTracking||(r="onerror",i("_"+r),f=e[r],e[r]=function(config,i,u,e,o){var s=f&&f(config,i,u,e,o);return s!==!0&&t["_"+r](config,i,u,e,o),s}),t
        }({
            instrumentationKey:"5b3d6014-f021-4304-8366-3cf961d5b90f",
            disableAjaxTracking: true
        });
        window.appInsights=appInsights;
        appInsights.trackPageView();
    </script>

6条回答
欢心
2楼-- · 2020-07-11 06:50

Basically, if you want to use the Kaggle python API (the solution provided by @minh-triet is for the command line not for python) you have to do the following:

import kaggle

kaggle.api.authenticate()

kaggle.api.dataset_download_files('The_name_of_the_dataset', path='the_path_you_want_to_download_the_files_to', unzip=True)

I hope this helps.

查看更多
0人赞 添加讨论(0) 举报
叼着烟拽天下
3楼-- · 2020-07-11 06:52

Just to make things easy for the next person, I combined the fantastic answer from CaitLAN Jenner with a little bit of code that takes the raw csv info and puts it into a Pandas DataFrame, assuming that row 0 has the column names. I used it to download the Pima Diabetes dataset from Kaggle, and it worked swimmingly.

I'm sure there are more elegant ways to do this, but it worked well enough for a class I was teaching, is easily interpretable, and lets you get to analysis with minimal fuss.

import pandas as pd
import requests
import csv

payload = {
    '__RequestVerificationToken': '',
    'username': 'username',
    'password': 'password',
    'rememberme': 'false'
}

loginURL = 'https://www.kaggle.com/account/login'
dataURL = "https://www.kaggle.com/uciml/pima-indians-diabetes-database/downloads/diabetes.csv"

with requests.Session() as c:
    response = c.get(loginURL).text
    AFToken = response[response.index('antiForgeryToken')+19:response.index('isAnonymous: ')-12]
    #print("AntiForgeryToken={}".format(AFToken))
    payload['__RequestVerificationToken']=AFToken
    c.post(loginURL + "?isModal=true&returnUrl=/", data=payload)
    download = c.get(dataURL)
    decoded_content = download.content.decode('utf-8')
    cr = csv.reader(decoded_content.splitlines(), delimiter=',')
    my_list = list(cr)
    #for row in my_list:
    #    print(row)


df = pd.DataFrame(my_list)
header = df.iloc[0]
df = df[1:]
diab = df.set_axis(header, axis='columns', inplace=False)

# to make sure it worked, uncomment this next line:
# diab

`

查看更多
0人赞 添加讨论(0) 举报
够拽才男人
4楼-- · 2020-07-11 06:52

Ref https://github.com/Kaggle/kaggle-api

Step _1, Try Insatling Kaggle

pip install kaggle # Windows
pip install --user kaggle # **Mac/Linux**.

Step 2,

Update your Credentials, so that kaggle can authenticate on .kaggle/kaggel_json based on your token generated from Kaggle. ref: https://medium.com/@ankushchoubey/how-to-download-dataset-from-kaggle-7f700d7f9198

Step 3 Now Instaed ofkaggle competitions download ..

run ~/.local/bin/kaggle competitions download .. to avoid Command Kaggle Not Found

查看更多
0人赞 添加讨论(0) 举报
Summer. ? 凉城
5楼-- · 2020-07-11 06:53

I would recommend checking out Kaggle API instead of using your own code. As per latest version, an example command to download dataset is kaggle datasets download -d zillow/zecon

查看更多
0人赞 添加讨论(0) 举报
你好瞎i
6楼-- · 2020-07-11 06:56

kaggle api key and usersame is available on kaggle profile page and dataset download link is available on dataset details page on kaggle

#Set the enviroment variables
import os
os.environ['KAGGLE_USERNAME'] = "xxxx"
os.environ['KAGGLE_KEY'] = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition
查看更多
0人赞 添加讨论(0) 举报
Fickle 薄情
7楼-- · 2020-07-11 06:58

Before anything:

pip install kaggle

For the dataset:

import os
os.environ['KAGGLE_USERNAME'] = "uname" # username from the json file
os.environ['KAGGLE_KEY'] = "kaggle_key" # key from the json file
!kaggle datasets download -d zynicide/wine-reviews

For the competitions:

import os
os.environ['KAGGLE_USERNAME'] = "uname" # username from the json file
os.environ['KAGGLE_KEY'] = "kaggle_key" # key from the json file
!kaggle competitions download -c dogs-vs-cats-redux-kernels-edition

Some time ago I provided another similar answer.

查看更多
0人赞 添加讨论(0) 举报
登录 后发表回答
相关问题
查看全部
相关文章
查看全部
收藏的人(4)