I’m working on a project that requires a lot algorithm development and data visualization. The workflow involves lots of small tweaks, with checks to see if the output matches expectations.
To fit that workflow, I’m trying to set up my work environment to be very responsive:
- Watch for file changes in given directory (like watchdog or tornado’s debug mode)
- On file change, re-run a given script (possibly the script that was modified)
- Do it in such a way that external modules (e.g. pandas, matplotlib) don’t need to be reloaded.
- If possible, do it in such a way that some large data inputs don’t need to be reloaded either.
Currently, I’m using watchdog’s watchemdo utility to re-run scripts every time they’re edited. The problem is that all the scripts have large-ish dependencies (matplotlib, etc.) that take a couple of seconds to load. It’s not much, but it really slows down the workflow.
Can anyone recommend an architecture/utility/etc. that satisfies 1, 2, 3, and (ideally) 4 as well?
