How to customize the configuration file of the off

Question

I'm using the official Postgres Docker image trying to customize its configuration. For this purpose, I use the command sed to change max_connections for example:

sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf

I tried two methods to apply this configuration. The first is by adding the commands to a script and copying it within the init folder "/docker-entrypoint-initdb.d". The second method is by running them directly within my Dockerfile with "RUN" command (this method worked fine with a non-official Postgresql image with a different path to the configuration file "/etc/postgres/..."). In both cases the changes fail because the configuration file is missing (I think it's not created yet).

How should I change the configuration?

Edit 1:

Here is the Dockerfile used to create the image:

# Database (http://www.cs3c.ma/)

FROM postgres:9.4
MAINTAINER Sabbane <[e-mail]>

ENV TERM=xterm

RUN apt-get update
RUN apt-get install -y nano

ADD scripts /scripts
# ADD scripts/setup-my-schema.sh /docker-entrypoint-initdb.d/

# Allow connections from anywhere.
RUN sed -i -e"s/^#listen_addresses =.*$/listen_addresses = '*'/" /var/lib/postgresql/data/postgresql.conf
RUN echo "host    all    all    0.0.0.0/0    md5" >> /var/lib/postgresql/data/pg_hba.conf

# Configure logs
RUN sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_directory = 'pg_log'.*$/log_directory = '\/var\/log\/postgresql'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_filename = 'postgresql-\%Y-\%m-\%d_\%H\%M\%S.log'.*$/log_filename = 'postgresql_\%a.log'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_file_mode = 0600.*$/log_file_mode = 0644/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_truncate_on_rotation = off.*$/log_truncate_on_rotation = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_rotation_age = 1d.*$/log_rotation_age = 1d/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_min_duration_statement = -1.*$/log_min_duration_statement = 0/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_checkpoints = off.*$/log_checkpoints = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_connections = off.*$/log_connections = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_disconnections = off.*$/log_disconnections = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^log_line_prefix = '\%t \[\%p-\%l\] \%q\%u@\%d '.*$/log_line_prefix = '\%t \[\%p\]: \[\%l-1\] user=\%u,db=\%d'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_lock_waits = off.*$/log_lock_waits = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_temp_files = -1.*$/log_temp_files = 0/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#statement_timeout = 0.*$/statement_timeout = 1800000        # in milliseconds, 0 is disabled (current 30min)/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^lc_messages = 'en_US.UTF-8'.*$/lc_messages = 'C'/" /var/lib/postgresql/data/postgresql.conf

# Performance Tuning
RUN sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^shared_buffers =.*$/shared_buffers = 16GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#effective_cache_size = 128MB.*$/effective_cache_size = 48GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#work_mem = 1MB.*$/work_mem = 16MB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#maintenance_work_mem = 16MB.*$/maintenance_work_mem = 2GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#checkpoint_segments = .*$/checkpoint_segments = 32/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#checkpoint_completion_target = 0.5.*$/checkpoint_completion_target = 0.7/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#wal_buffers =.*$/wal_buffers = 16MB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#default_statistics_target = 100.*$/default_statistics_target = 100/" /var/lib/postgresql/data/postgresql.conf


VOLUME ["/var/lib/postgresql/data", "/var/log/postgresql"]

CMD ["postgres"]

With this Dockerfile the build process shows the error: sed: can't read /var/lib/postgresql/data/postgresql.conf: No such file or directory

Answer 1

You can put your custom postgresql.conf in a temporary file inside the container, and overwrite the default configuration at runtime.

To do that :

• Copy your custom postgresql.conf inside your container
• Copy the updateConfig.sh file in /docker-entrypoint-initdb.d/

---

Dockerfile

FROM postgres:9.6

COPY postgresql.conf      /tmp/postgresql.conf
COPY updateConfig.sh      /docker-entrypoint-initdb.d/_updateConfig.sh

---

updateConfig.sh

#!/usr/bin/env bash

cat /tmp/postgresql.conf > /var/lib/postgresql/data/postgresql.conf

---

At runtime, the container will execute the script inside /docker-entrypoint-initdb.d/ and overwrite the default configuration with yout custom one.

Answer 2

With Docker Compose

When working with Docker Compose, you can use command: postgres -c option=value in your docker-compose.yml to configure Postgres.

For example, this makes Postgres log to a file:

command: postgres -c logging_collector=on -c log_destination=stderr -c log_directory=/logs

Adapting Vojtech Vitek's answer, you can use

command: postgres -c config_file=/etc/postgresql.conf

to change the config file Postgres will use. You'd mount your custom config file with a volume:

volumes:
   - ./customPostgresql.conf:/etc/postgresql.conf

---

Here's the docker-compose.yml of my application, showing how to configure Postgres:

# Start the app using docker-compose pull && docker-compose up to make sure you have the latest image
version: '2.1'
services:
  myApp:
    image: registry.gitlab.com/bullbytes/myApp:latest
    networks:
      - myApp-network
  db:
     image: postgres:9.6.1
     # Make Postgres log to a file.
     # More on logging with Postgres: https://www.postgresql.org/docs/current/static/runtime-config-logging.html
     command: postgres -c logging_collector=on -c log_destination=stderr -c log_directory=/logs
     environment:
       # Provide the password via an environment variable. If the variable is unset or empty, use a default password
       - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-4WXUms893U6j4GE&Hvk3S*hqcqebFgo!vZi}
     # If on a non-Linux OS, make sure you share the drive used here. Go to Docker's settings -> Shared Drives
     volumes:
       # Persist the data between container invocations
       - postgresVolume:/var/lib/postgresql/data
       - ./logs:/logs
     networks:
       myApp-network:
         # Our application can communicate with the database using this hostname
         aliases:
           - postgresForMyApp
networks:
  myApp-network:
    driver: bridge
# Creates a named volume to persist our data. When on a non-Linux OS, the volume's data will be in the Docker VM
# (e.g., MobyLinuxVM) in /var/lib/docker/volumes/
volumes:
  postgresVolume:

---

Permission to write to the log directory

Note that when on Linux, the log directory on the host must have the right permissions. Otherwise you'll get the slightly misleading error

FATAL: could not open log file "/logs/postgresql-2017-02-04_115222.log": Permission denied

I say misleading, since the error message suggests that the directory in the container has the wrong permission, when in reality the directory on the host doesn't permit writing.

To fix this, I set the correct permissions on the host using

chgroup ./logs docker && chmod 770 ./logs

Answer 3

A fairly low-tech solution to this problem seems to be to declare the service (I'm using swarm on AWS and a yaml file) with your database files mounted to a persisted volume (here AWS EFS as denoted by the cloudstor:aws driver specification).

  version: '3.3'
  services:
    database:
      image: postgres:latest
      volumes:
        - postgresql:/var/lib/postgresql
        - postgresql_data:/var/lib/postgresql/data
    volumes:
       postgresql:
         driver: "cloudstor:aws" 
       postgresql_data:
         driver: "cloudstor:aws"

1. The db comes up as initialized with the image default settings.
2. You edit the conf settings inside the container, e.g if you want to increase the maximum number of concurrent connections that requires a restart
3. stop the running container (or scale the service down to zero and then back to one)
4. the swarm spawns a new container, which this time around picks up your persisted configuration settings and merrily applies them.

A pleasant side-effect of persisting your configuration is that it also persists your databases (or was it the other way around) ;-)

Answer 4

Inject custom postgresql.conf into postgres Docker container

The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.

To customize PostgreSQL configuration in Docker consistently, I suggest using config_file postgres option together with Docker volumes like this:

Production database (PGDATA dir as Persistent Volume)

docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf

Testing database (PGDATA dir will be discarded after docker rm)

docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf

---

Debugging

1. Remove the -d (detach option) from docker run command to see the server logs directly.

2. Connect to the postgres server with psql client and query the configuration:

   docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
   
   psql (9.6.0)
   Type "help" for help.
   
   postgres=# SHOW all;
   

Answer 5

I looked through all answers and there is another option left. You can change your CMD value in docker file (it is not the best one, but still possible way to achieve your goal).

Basically we need to

• Copy config file in docker container
• Override postgres start options

---

Docker file example:

FROM postgres:9.6
USER postgres

# Copy postgres config file into container
COPY postgresql.conf /etc/postgresql

# Override default postgres config file
CMD ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]

---

Though I think using command: postgres -c config_file=/etc/postgresql/postgresql.confin your docker-compose.yml file proposed by Matthias Braun is the best option.

Answer 6

The postgres:9.4 image you've inherited from declares a volume at /var/lib/postgresql/data. This essentially means you can't copy any files to that path in your image; the changes will be discarded.

You have a few choices:

• You could just add your own configuration files as a volume at run-time with docker run -v postgresql.conf:/var/lib/postgresql/data/postgresql.conf .... However, I'm not sure exactly how that will interact with the existing volume.

• You could copy the file over when the container is started. To do that, copy your file into the build at a location which isn't underneath the volume then call a script from the entrypoint or cmd which will copy the file to correct location and start postgres.

• Clone the project behind the Postgres official image and edit the Dockerfile to add your own config file in before the VOLUME is declared (anything added before the VOLUME instruction is automatically copied in at run-time).