I'm using the bigquery.go library.
While investigating some of the performance I have found that my export (.csv to GCS) job (and export job only) started from the client takes about 60seconds on average while the same job started from the WebGUI takes about 20seconds. What could be the reason for this?
The code is the following:
time1 := time.Now()
job_extract, err := extractor.Run(ctx)
if err != nil {
return err
}
status, err = job_extract.Wait(ctx)
if err != nil {
return err
}
if status.Err() != nil {
log.Fatalf("Job failed with error %v", status.Err())
return status.Err()
}
time2 := time.Since(time1)
The WEB UI usually has a polling mechanism to check when a job has finished, so you might see longer times. Usually the files exported to GCS appear sooner than the job actually finishes in the WebUI.
To make sure and see exact timing, please use the cli tool to obtain the most recent jobs:
bq ls -j -a --max_results=15
running this would display a table with job ids and the respective timing.
jobId Job Type State Start Time Duration
--------------------------------- ---------- --------- ----------------- ----------
bquijob_1864e679_15a84d8878a query SUCCESS 28 Feb 07:11:06 0:00:04
bquijob_770b512_15a84d8122c query FAILURE 28 Feb 07:10:35 0:00:00
bquijob_de0df03_15a84d6a4fa query FAILURE 28 Feb 07:09:02 0:00:00
bquijob_52c4f7d7_15a84d660e6 query FAILURE 28 Feb 07:08:44 0:00:00
bquijob_76a2c1be_15a84d5e769 query FAILURE 28 Feb 07:08:13 0:00:00
bquijob_7f51dde5_15a84d55afb query SUCCESS 28 Feb 07:07:41 0:00:08
bquijob_34f25864_15a84d50503 query SUCCESS 28 Feb 07:07:18 0:00:08
job_Ca0cuRTAjY7MEHAs7vTJMxtVYTs query SUCCESS 28 Feb 07:00:47 0:00:09
job_hHfmcdwyBsPsYF5dDvvOdR1Rmd0 load SUCCESS 28 Feb 07:00:26 0:00:20
job_mkiLf_mFHLKSplGJOtg-XDKzvv4 load SUCCESS 28 Feb 02:52:50 0:00:02
job_3RsPvttxWwv3SzVoOI9Cv_2yWtA query SUCCESS 27 Feb 21:18:40 0:00:08
job_JLsqJO0NEIlKNac6jkDWbwneGMg extract SUCCESS 27 Feb 11:35:04 0:00:17
job_KOS7vKX4aX0FNbK6dibE7cxzcQA query SUCCESS 27 Feb 11:33:44 0:00:37
bquijob_44046bec_15a802f703a query SUCCESS 27 Feb 09:27:48 0:00:07
job_2qQ6YSWeXaP2y2doONQJsIoga3c query SUCCESS 27 Feb 08:53:20 0:00:06
You would be able this way to check the extract job DURATION. In case you validate indeed it's a problem, please post such a table back in your question as sooner/later a Google engineer will check it. But without proper details, we can just assume your measurement is wrong.
