I have an apache web server which I have setup a website using python based flask using mod_wsgi. I'm having a problem with my router for python's flask hanging while processing huge number of json data. Here inside a route there is function which collects data and process.
@app.route('/bulkprocess/', methods=['GET', 'POST'])
def bulk_request():
res_vrf = mainapp.bulkfun(request.data)
return jsonify(res_vrf)
The python script mainapp.bulkfun() reads in a lot of data, process and then return back to the client in son format. If i send 2000 Json data to process it will take less than 6min and I will get the data. if i try to send more than 2000 data to process the application will hang. But the data will be send to the server and the whole process will take place and won't return back to client.
The mainapp.bulkfun() script contains a multiprocess module to increase the performance of the script. I have used this script from the below link.
Multiprocessing
In Mypool() function we are using 40 process per request.
While using Top command i can see the multiple apache process going on after the son data is passed. but no data is returned to client if we use large son data. At this point my memory usage is less than 30% too.
Below is my apache error log.
[Tue Oct 09 06:25:03.625139 2018] [mpm_worker:notice] [pid 3908:tid 140377528133504] AH00292: Apache/2.4.18 (Ubuntu) SVN/1.9.3 mod_wsgi/4.6.4 Python/3.6 configured -- resuming norm$
[Tue Oct 09 06:25:03.625179 2018] [core:notice] [pid 3908:tid 140377528133504] AH00094: Command line: '/usr/sbin/apache2'
My Flask-app error Log
[Thu Oct 11 06:46:46.126464 2018] [wsgi:error] [pid 3175:tid 140377306687232] (70007)The timeout specified has expired: [client 137.97.88.45:43530] mod_wsgi (pid=3175): Failed to proxy response to client.
