Elastic BeanStalk EC2 instance's log uses up w

Question

I have an Elastic BeanStalk environment where I run my application on 1 EC2 instance. I've added load balancer, when I configured the environment initially, but since then I set it only use 1 instance.

Application run within container apparently produces quite a lot of logs - after several days they use up whole disk space and then application crash. Health check drops to severe.

I see that terminating instance manually helps - environment removes old instance and creates a new one that works (until it fills up the whole disk again).

What are my options? A script that regularly cleans up logs? Some log rotation? Trigger that reboots instance when disk is nearly full?

I do not write anything to file myself - my application only log to std out and std err, so writing to file is done by EC2/EBS wrapper. (I deploy the application as a ZIP containing a JAR, a bash script and Procfile if that is relevant).

Answer 1

Logrotation is the way forward. You can create a configuration file in `/etc/logrotate.d/' where you state your options in order to avoid having large log files.

You can read more about the configurations here https://linuxconfig.org/setting-up-logrotate-on-redhat-linux

A sample configuration file would look something like this:

/var/log/your-large-log.log {
    missingok
    notifempty
    compress
    size 20k
    daily
    create 0600 root root
}

You can also test the new configuration file from the cli by running the follow:

logrotate -d [your_config_file]

This will test if the log rotation will be successful or not but only in debugging mode, therefore the log file will not be actually rotated.

Answer 2

By default EB will rotate some of the logs produced by the Docker containers, but not all of them. After contacting support on this issue I received the following helpful config file, to be placed in the source path .ebextensions/liblogrotate.config:

files:
  "/etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.containers.conf":
    mode: "00644"
    owner: "root"
    group: "root"
    content: |
      /var/lib/docker/containers/*/*.log {
        size 10M
        rotate 5
        missingok
        compress
        notifempty
        copytruncate
        dateext
        dateformat %s
        olddir /var/lib/docker/containers/rotated
      }
    "/etc/cron.hourly/cron.logrotate.elasticbeanstalk.containers.conf":
    mode: "00755"
    owner: "root"
    group: "root"
    content: |
      #!/bin/sh
      test -x /usr/sbin/logrotate || exit 0
      /usr/sbin/logrotate /etc/logrotate.elasticbeanstalk.hourly/logrotate.elasticbeanstalk.containers.conf

container_commands:
  create_rotated_dir:
    command: mkdir -p /var/lib/docker/containers/rotated
    test: test ! -d /var/lib/docker/containers/rotated
  99_cleanup:
    command: rm /etc/cron.hourly/*.bak /etc/logrotate.elasticbeanstalk.hourly/*.bak
    ignoreErrors: true

What this does is install an additional log rotation configuration and cron task for the /var/lib/docker/containers/*/*.log files which are the ones not automatically rotated on EB.

Eventually, however, the rotated logs themselves will fill up the disk if the host lives long enough. For this, you can add shred in the list of logrotation options (along side compress notifempty etc).

(However, I'm not sure if the container logs that are already configured for rotation are set to be shredded, probably not - so those may accumulate too and require modification of the default EB log rotation config. Not sure how to do that yet. But the above solution in most cases would be sufficient since hosts typically do not live that long. The volume of logging and lifetime of your containers may force you to go even further.)