Performance of django-CMS with remote mysql DB

2019-08-12 04:18发布

问题:

I have tried to run a django-CMS (+filer +easy_thumbnails) with a remote mysql DB on GoogleAppengine, with cloudstorage. After fixing file-system related performance problems and fork-fixing a django-google-cloud-storage module (http://github.com/locandy/django-google-cloud-storage), the performance is still beyond terrible (4s per request pre cache). Most things are configured as default from the tutorial.

Timing is for generic page rendering requests (no cache, not logged in, excluding instance start up time). The fastest is an empty page (no images, no text) with 1.7 seconds and 40 database RPCs. The slowest a full page with many images and some text at 4 seconds and 100 rdbms.Exec calls. I used the appengine python profiling module.

On average that would be 45ms per query.

Is there any configuration we have missed?

Did anyone succeed in deploying a CMS in the cloud with remote DB that is usable?

TEMPLATE_CONTEXT_PROCESSORS = (
    'django.contrib.auth.context_processors.auth',
    'django.contrib.messages.context_processors.messages',
    'django.core.context_processors.i18n',
    'django.core.context_processors.request',
    'django.core.context_processors.media',
    'django.core.context_processors.static',
    'cms.context_processors.media',
    'sekizai.context_processors.sekizai',
)

MIDDLEWARE_CLASSES = (
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.locale.LocaleMiddleware',
    'django.middleware.doc.XViewMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
    'cms.middleware.page.CurrentPageMiddleware',
    'cms.middleware.user.CurrentUserMiddleware',
    'cms.middleware.toolbar.ToolbarMiddleware',
    'cms.middleware.language.LanguageCookieMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
)

DATABASES = {
    'default': {
        'ENGINE': 'google.appengine.ext.django.backends.rdbms',

Profiling:

  (1) 2014-01-15 12:15:03.358 "GET .../benefits/" 200 real=4636ms api=0ms overhead=9ms (89 RPCs, cost=0, billed_ops=[])
  (2) 2014-01-15 12:14:56.862 "GET .../preise/" 200 real=5200ms api=0ms overhead=9ms (94 RPCs, cost=0, billed_ops=[])
  (3) 2014-01-15 12:14:47.673 "GET .../einstieg/" 200 real=4684ms api=0ms overhead=8ms (87 RPCs, cost=0, billed_ops=[])
  (4) 2014-01-15 12:14:01.054 "GET .../moeglichkeiten/" 200 real=5341ms api=0ms overhead=10ms (98 RPCs, cost=0, billed_ops=[])
  (5) 2014-01-15 12:13:31.516 "GET .../werkzeuge/" 200 real=5176ms api=0ms overhead=9ms (96 RPCs, cost=0, billed_ops=[])
  (6) 2014-01-15 12:13:00.507 "GET .../einstieg/" 200 real=5460ms api=0ms overhead=9ms (94 RPCs, cost=0, billed_ops=[])
  (7) 2014-01-15 12:12:59.891 "GET .../" 302 real=369ms api=0ms overhead=0ms (7 RPCs, cost=0, billed_ops=[])

回答1:

After profiling django-CMS with appstats in the recommended setup with filer and easy-thumbnails and after excluding cloudstorage, it is not possible to render new uncached empty pages in less than 500ms with a remote DB.

Every placeholder or feature added to a page adds several sequentially executed queries. A complex page has up to 100 queries and a moderate page about 50. This adds up to 4-5 seconds rendering time for a page.

We have run a remote DB connection to a SDK instance on a local machine Europe-USA, which increased the page rendering time to 32 seconds. Access times seems to scale linear with network delay and number of queries.

The system can still be used with django's database cache, but the administrative backend is too slow to be used conveniently. Conclusion: django-CMS is not compatible with a remote DB setting in any cloud.

As a result of our tests we have fixed a uncached django storage module compatible with cloudstorage http://github.com/locandy/django-google-cloud-storage