I'm running Django REST framework with fairly low request per minute rate during the day. I've noticed one problem that I could't explain or reproduce.
Every day, during the night or early in the morning, when my RPM is close to zero I have 1 - 10 requests that are super slow. Average response time for me us between 100 and 200 ms, but this ones are from 50s to 1 minute.
On this screen you can see how it happens:
As my primary database I'm using MySQL and I've implemented logging for each request with something similar to DRF-tracking.
In my logs I can see that it's actually two different problems:
- Sometimes INSERT into my logging table is super slow
- Sometimes after
finalize_response
is finished django/uwsgi takes 40-50 seconds to actually serve the response.
The problem with MySQL only occurred after table size became more than 2 millions rows. But this table doesn't have any indexes (except for primary one) and on average INSERT takes 0.5 ms, but sometimes it goes crazy.
Did anyone has this problems or know which way I should dig to find a solution?
Here you can see difference between this two problems:
Slow Dhango
Slow MySQL