Deploying Django to AWS; static files for dummies

Question

I am utterly lost on one of the last steps of this project.

So far, I've been able to develop a django app that works the way I want it to on localhost; I've been able to deploy the website to AWS EC2, but I must be missing something fundamental about serving the static files. (I haven't even tried media files yet.) I've read the Django Deployment page and How-To manage static files, but I have never deployed a website from scratch before. The tutorials I've found seem to be contradicting (or outdated?).

Here are the questions I think I have at this time:

1. Do I need to host static (and/or media) files in a bucket, or is this merely a good idea?
2. When I set up STATIC_ROOT and STATIC_URL, should I have a STATICFILE_DIRS setup? (I mean, I think I really need a tutorial on how they even go together, their settings, and how 'static' works in the templates.)
3. I've tried to get whitenoise going; I get a message that STATIC_URL isn't set up correctly; I can't find the documentation to tell me what it should be. Is this a viable root to take?

EDIT

Even with @DirkGroten 's amazingly detailed answer, I'm still not getting how to serve static files. I can run the server and then web browser in to see pages with no static files. But, I now have a new problem: Pages that have static files on them return a 500 27 error (whereas they used to return an error for only the file). So, here is my folder structure, and below that is the relevant part of my settings file (which is actually split into base, dev, and prod).

[mainsite]/
|---[mainsite]/
|   |---[settings]/
|       |---base.py
|       |---dev.py
|       |---prod.py
|---[app1]/
|   |---[migrations]
|   |---[static]/
|   |   |---[app1]/
|   |   |   |---app1_file1.jpg (etc)
|   |   |---app1_style.css
|   |---[templates]/
|   |   |---[app1]/
|   |       |---about.html (etc)
|---[app2]/ (etc)
|---[static]  (this gets populated after running collectstatic)

development settings:

Debug = False
ALLOWED_HOSTS = [###.###.###.###]  

MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'whitenoise.middleware.WhiteNoiseMiddleware',   
... (the rest of the middleware)
]

STATIC_URL = '/static/'
STATIC_ROOT = os.path.abspath(os.path.join(BASE_DIR, 'static'))
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'app1/'),
                    os.path.join(BASE_DIR, 'app2/'),
                    os.path.join(BASE_DIR, 'app3/'),
                    ]
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'

I have made sure that I am importing whitnoise in production requirements file.

What am I missing?

Also, I have never used the AWS support system. (I'm on the free tier.) Is this the kind of thing I can get their help on? Do I start a case?

TIA.

Answer 1

First your questions:

1. No, it's one of the options, but it's definitely a good idea to use a centralised, shared storage, for when you'll need to serve your site from multiple EC2 instances. S3 is one option, I use EFS which I find easier (see below).
2. Yes, you need to tell collectstatic where to find the static files so you should have STATICFILES_DIRS.
3. Yes, I like whitenoise, it works well in combination with a CDN. Makes it easier not to worry about expiry/cache headers.

I've written a blog post about this.

This is a setup that works for one EC2 instance:

• Static files stored locally on the EC2 instance (on the EBS volume) -> STATIC_ROOT setting (where are they stored)
• Whitenoise to serve the static files and setting cache headers correctly
• CompressedManifestStaticFilesStorage as the storage (STATICFILES_STORAGE setting)
• Preferably CloudFront or other CDN to serve the static files -> STATIC_URL setting.

Eventually you'll want to have your static files stored centrally so that multiple EC2 instances can access them (otherwise you'd have to duplicate them on each machine and ensure the manifest files are in sync), so my own setup is:

• Static files stored on an EFS volume that is mounted and shared by all EC2 instances (much easier to setup and use than S3, as it's just a mounted disk as far as python is concerned). I set the mount point of the EFS volume to be the same location as STATIC_ROOT, so nothing to change in settings.
• Then the same as above: Whitenoise, CloudFront and the manifest storage.

Try first to get it to work with the first setup, doing the following:

• STATIC_URL should be the URL that will show up in the HTML. Just /static/ should work with WhiteNoiseMiddleware but if you use a CDN, you need to precede that with the full host name of your CDN instance.
• STATICFILES_STORAGE should be one of the storages offered by Whitenoise, I recommend CompressedManifestStaticFilesStorage.
• STATICFILES_DIRS tells collectstatic where it can find the static files to collect. Note that if you only have static files in a /static directory inside each app, collectstatic will find them automatically, and you should not specify those directories here. If you have a /static directory in your main project directory, you should add (os.path.join(BASE_DIR, "my_site", "static"),) to STATICFILES_DIRS where BASE_DIR (or PROJECT_ROOT in some projects) is the root directory of your project.
• STATIC_ROOT should be the physical location of the static files on disk. This is where collectstatic will copy them and index them, start with on level up from your BASE_DIR, i.e. os.path.abspath(os.path.join(BASE_DIR, '../static'))

Now every time you update your code and static files, you should run django-admin collectstatic. This will copy all your static files to your STATIC_ROOT directory where whitenoise will fetch them to serve them. You'll see MD5 hashes added to the filename each time the file changes, this is so that browsers fetch the new version and don't use the cached version (Whitenoise tells browsers to cache static files for 2 years by default).

If you get this to work, start adding a CDN (set it up so it fetches the files from your server, whitenoise now will only have to serve each file once to the CDN, browsers will get it from the CDN); you'll need to change STATIC_URL to add the CDN's host name (e.g. https://die9493v4034.cloudfront.net/static/).

Then move the location of the files from the local instance to a shared storage (e.g. EFS).