How to stream the logs in docker python API?

Question

I am building an image from a Dockerfile using the docker python API.

import os
import sys
import os.path
import docker


client = docker.from_env()
try:
    here = os.path.dirname(__file__)
    no_cache = False
    dockerfile = os.path.join(here, 'app', 'nextdir')
    image = client.images.build(path=dockerfile, tag='app:v.2.4', nocache=no_cache, stream=True)

The operation finishes successfully, however I was not able to stream the logs. The API says:

Return a blocking generator you can iterate over to retrieve build output as it happens

when stream=True.

How can I get these logs in python?

Answer 1

You may use the low-level API client. The build() function will return a generator that you can iterate to grab chunks of the build log.

The generator will yield a string containing a JSON object, you may call json.loads() on it or you can use the decode=True parameter in the build() function that will do that for you.

Once you grab the 'stream' key from the yielded dictionary you may just print() it, but if you need to send it to a logger it may be better to do it line by line, as the chunk received will contain more than one line.

One option of such code is as follows:

from docker import APIClient

client = APIClient(base_url='unix://var/run/docker.sock')

# Build docker image
log.info('Building docker image ...')
streamer = client.build(
    decode=True,
    path=args.path,
    tag=args.tag,
)

for chunk in streamer:
    if 'stream' in chunk:
        for line in chunk['stream'].splitlines():
            log.debug(line)

Answer 2

The docs state...

If you want to get the raw output of the build, use the build() method in the low-level API.

Have you tried that?

Answer 3

Streaming the docker build logs can be done using the low-level APIs given in docker-py as follows,

        here = os.path.dirname(__file__)
        dockerfile = os.path.join(here, 'app', 'nextdir')
        docker_client = docker.APIClient(base_url='unix://var/run/docker.sock')
        generator = docker_client.build(path=dockerfile, tag='app:v.2.4', rm=True)
        while True:
            try:
                output = generator.__next__
                output = output.strip('\r\n')
                json_output = json.loads(output)
                if 'stream' in json_output:
                    click.echo(json_output['stream'].strip('\n'))
            except StopIteration:
                click.echo("Docker image build complete.")
                break
            except ValueError:
                click.echo("Error parsing output from docker image build: %s" % output)

Answer 4

According to the docs, the image build now returns a tuple with both the image and the build logs

The first item is the Image object for the image that was build. The second item is a generator of the build logs as JSON-decoded objects.

And modifying @havock solution accordingly:

import docker

client = docker.from_env()
image, build_logs = client.images.build(**your_build_kwargs)

for chunk in build_logs:
    if 'stream' in chunk:
        for line in chunk['stream'].splitlines():
            log.debug(line)

Answer 5

Docker low level build has the ability to directly decode the output to a dictionary. For that use decode=True.

I used mashumaro to convert the dictionary to a dataclass. This is useful if you like type hints. To use dataclasses you need at least python3.7.

Install mashumaro:

pip install mashumaro

The dataclass:

@dataclass
class DockerStreamOutput(DataClassDictMixin):
    stream: str = None
    aux: str = None

Build your image:

from mashumaro import DataClassDictMixin, MissingField
generator = client.api.build(
            path=container_path,
            tag=tag_name,
            dockerfile=dockerfile_path,
            decode=True  # encode to dict
        )

for line in generator:
    try:
        stream_output: DockerStreamOutput = DockerStreamOutput.from_dict(line)
        stream: str = stream_output.stream
        if stream:
            logging.info(stream.strip())
    except MissingField:
        logging.error(line)

Hope this helps.