Can I use third party libraries with Cloud Dataflo

Question

Does Cloud Dataflow allows you to use it with third party library jar files? How about non-Java libraries?

Kaz

Answer 1

Yes you can use third party library files just fine. By default when you run your Dataflow main program to submit your job, Dataflow will analyze your classpath and upload any jars it sees and add them to the class path of the workers.

If you need more controlthen you can use the command line option --filesToStage to specify additional files to stage on the workers.

Another common technique is building a single bundled jar which contains all your dependencies. One way to build a bundled jar is to use a maven plugin like shade.