How to fix memory leak in my application?

Question

In my GAE app I add rows to Google Spreadsheet.

taskqueue.add(url='/tabletask?u=%s' % (user_id),
retry_options=taskqueue.TaskRetryOptions(task_retry_limit=0),
                            method='GET')

class TableTaskHandler(webapp2.RequestHandler):
    def get(self):
        user_id = self.request.get('u')
        if user_id:
            try:
                tables.add_row(
                    user_id
                )
            except Exception, error_message:
                pass



def get_google_api_service(scope='https://www.googleapis.com/auth/spreadsheets', api='sheets', version='v4'):
    ''' Login to Google API with service account and get the service
    '''
    service = None
    try:
        credentials = AppAssertionCredentials(scope=scope)
        http = credentials.authorize(httplib2.Http(memcache))
        service = build(api, version, http=http)
    except Exception, error_message:
        logging.exception('Failed to get Google API service, exception happened - %s' % error_message)
    return service

def add_row(user_id, user_name, project_id, question, answer, ss_id=SPREADSHEET_ID):
    service = get_google_api_service()
    if service:
        values = [
            [
                user_id, user_name, project_id, question, answer # 'test1', 'test2'
            ],
            # Additional rows ...
        ]
        body = {
            'values': values
        }
        # https://developers.google.com/sheets/api/guides/values#appending_values
        response = service.spreadsheets().values().append(
            spreadsheetId=ss_id,
            range='A1:E1000',
            valueInputOption='RAW',
            body=body).execute()

I add many tasks with different row values.

In result I get critical errors 'Exceeded soft private limit of 128 Mb with 158 Mb' after servicing 5 requests in total.

What could be wrong here?

Answer 1

It seems to be that you are running instance class B1 or F1, which has a memory limit of 128 MB.

A possible solution would be to use a higher instance class. But please keep in mind that choosing a different instance class will have impact on your pricing and quotas.

Answer 2

At first glance there’s nothing special in your code that might lead to a memory leak. I don’t think anybody can locate it unless he’s very deeply familiar with the 3rd party libraries used and their existing bugs. So I’d approach the problem as follows:

1. First lets find out where exactly memory is leaking and whether it’s leaking at all.

   Refer to tracemalloc, memory_profiler, heapy or whatever else you’re familiar with. Most profilers available are listed here Which Python memory profiler is recommended?

   Expected outcome: you clearly know where exactly the memory is leaking, up to a code line / python expression

2. If the problem is in a 3rd party code, try to dig deeper into its code and figure out what’s up there

3. Depending on p.2 outcome

   a. Post another SO question like ‘why this python code excerpt leads to a memory leak’ - ideally it should be a standalone code snippet that leads to a weird behavior free of any third party libraries and reproducible locally. Environment specification - at least python version, is appreciated

   b. If the problem is in a 3rd party library and you’ve located the problem, open a bug report on github/anywhere the target project is hosted

   c. If the problem is clearly in a 3rd party library and you’re unable to find the cause, open a ticket describing the case with the profiler's report attached