I have two python scripts running once a day in my local environment. One is to fetch data and another is to format it.
Now I want to deploy those scripts to Google's cloud environment and run those once/twice a day.
Can I do that using Google Cloud Function or do I need App Engine?
Why NO cron job: Because I don't want my system/VM to run whole day (when not in use).
Can I use Cloud Composer to achieve that?
You can use Google Cloud Scheduler which is a fully managed enterprise-grade cron job scheduler. It allows you to schedule virtually any job, including batch, big data jobs, cloud infrastructure operations.
The first 3 jobs per month are free. Next are $0.10 per job/month.
Yes, you will need App Engine and the Cron service to schedule those scripts. You have some more or less straight-forward options to run those scripts on GCP:
- Deploy each one as different functions, and do something as mentioned in here: Basically deploy a simple function in App Engine that sends an HTTP request to your(s) function(s).
- Deploy each one as different handlers in App Engine, and schedule each calls.
- Deploy each one as different services and then schedule the calls, bearing in mind that, if you want to schedule a call to a different service, you must specify to which one, in your
cron.yaml
file, using the target
keyword and the name of the service.