I have a relatively simple linear regression lambda in AWS. Each instance the function is called the logs display the following:
/opt/python/sklearn/externals/joblib/_multiprocessing_helpers.py:38: UserWarning: [Errno 38] Function not implemented. joblib will operate in serial mode
warnings.warn('%s. joblib will operate in serial mode' % (e,))
I suspect this is due to sklearn running on a lambda (i.e. 'serverless') and trying to determine it's multi-processing capabilities as per this question and this GH issue.
I am also understanding from the GH that this is not a 'fixable' issue, it will always happen when deploying with these dependencies on this hardware. I am getting back my expected results (even though I am currently maxing out the default, minimum lambda memory of 128mb).
I aim to control the warnings and would know if there is a way to either:
- stop sklearn looking for multiprocessing, so preventing the warning from issuing
- capture this specific warning and prevent it from being passed from my function into the cloudwatch logs
- if both are possible, which would be preferable from a aws architecture/pythonic opinion?
