Can I use Dataflow for Python SDK from a Jupyter n

Question

I want to play with Dataflow for Python SDK from a Jupyter notebook. I am not sure what are the dependencies needed and if I can spread the code over multiple notebook cells or not. What are the steps involved?

Answer 1

Yes! There are no special steps involved. For example, using a Conda environment (recommended for using IPython/Jupyter notebooks) the commands to start a Jupyter notebook are:

1. conda create -n TESTENV jupyter
2. source activate TESTENV
3. pip install https://github.com/GoogleCloudPlatform/DataflowPythonSDK/archive/v0.2.3.tar.gz
4. jupyter notebook

The commands above install version v0.2.3 of Python Dataflow. Please change it to the version desired. In the first notebook cell execute the following import statement:

import google.cloud.dataflow as df

Now you are all set. You can spread the workflow code over multiple cells. Check out the following notebook describing a very simple workflow: https://github.com/silviulica/WorkflowExamples/blob/master/notebooks/HelloWorld.ipynb