Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Compose Develop does not sync changes to services without build section #12065

Open
sfc-gh-kbregula opened this issue Aug 17, 2024 · 2 comments
Labels
area/watch Related with watch feature kind/bug

Comments

@sfc-gh-kbregula
Copy link

sfc-gh-kbregula commented Aug 17, 2024

Description

Hello,

My docker-compose file consists of one image, but it is used by multiple services. So I defined three services:

  • The first service contains the build definitions and image name. This service builds image.
  • The second and third service uses this image and depends on first service.

I found this trick on Stackoverflow:
https://stackoverflow.com/questions/40899236/how-to-prevent-docker-compose-building-the-same-image-multiple-times

Next, I wanted to add integration with docker compose watch, so I added develop sections like below:

  • The first service rebuilds the image if dependencies are changed.
  • Second and third service sync application code

Unfortunately, if my services with the application do not have a build section, the files are not synchronized without any error message, if I add the build section, everything starts working, but this causes docker compose build to build the same image multiple times after executing docker compose build.

From what I understand from the source code this is not the expected behavior. The sync action should work without a build section. The rebuild action requires a build section.

if trigger.Action == types.WatchActionRebuild && service.Build == nil {
return nil, fmt.Errorf("service %s doesn't have a build section, can't apply 'rebuild' on watch", service.Name)
}

Steps To Reproduce

I have a simple application as below:

#./src/main.py
import textwrap

from flask import Flask

import os;
from pathlib import Path

INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'Hello World!')

app = Flask(__name__)

@app.route("/")
def hello():
    response = f"INSTANCE_NAME: {INSTANCE_NAME}\n"
    response += f"REQUIREMENTS: \n{Path('requirements.txt').read_text()}\n"
    response += f"MAIN_APP: \n{Path(__file__).read_text()}\n"
    return textwrap.dedent(response)


if __name__ == "__main__":
    app.run()
# requirements.txt
flask

# Dockerfile
# syntax = docker/dockerfile:1.9
# check=error=true

FROM python:3.10-slim-bullseye

WORKDIR /app

RUN apt-get update; apt-get install -y dumb-init curl

COPY requirements.txt /app/

RUN pip install -r requirements.txt

ENV PYTHONPATH=/app/src
ENV FLASK_APP=/app/src/main.py

COPY ./src /app/src

RUN curl "http://host.docker.internal:9500/$(hostname)" || true

ENTRYPOINT ["/usr/bin/dumb-init", "--"]
# docker-compose.yaml
services:
  my-app-image:
    build:
      context: .
      dockerfile: Dockerfile
    image: ${COMPOSE_PROJECT_NAME}-my-app
    command: ['echo', 'The docker image (${COMPOSE_PROJECT_NAME}-docparser) is ready to be used.']
    develop:
      watch:
        - path: ./requirements.txt
          action: rebuild

  my-app-instance-1:
    depends_on:
      my-app-image:
        condition: service_completed_successfully
        required: false
    environment:
      INSTANCE_NAME: instance1
    build:
      context: .
      dockerfile: Dockerfile
    image: ${COMPOSE_PROJECT_NAME}-my-app
    ports:
      - "3001:3001"
    command: [ 'flask', 'run', '--host=0.0.0.0', '--port=3001' ]
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src

  my-app-instance-2:
    depends_on:
      my-app-image:
        condition: service_completed_successfully
        required: false

    image: ${COMPOSE_PROJECT_NAME}-my-app
    environment:
      INSTANCE_NAME: instance2
    ports:
      - "3002:3002"
    command: [ 'flask', 'run', '--host=0.0.0.0', '--port=3002' ]
    develop:
      watch:
        - action: sync
          path: ./src
          target: /app/src

Now I run my environment:

docker compose build --no-cache
docker compose up --wait

And then I run Compose Watch

docker --debug --log-level=debug compose watch

I have the following output:

DEBU[0000] Enabled Docker Desktop integration (experimental) @ unix:///Users/my-user/Library/Containers/com.docker.docker/Data/docker-cli.sock
DEBU[0000] Watch configuration for service "my-app-instance-1":
  - Action sync for path "/Users/my-user/compose_develop_seperate_image/src"
DEBU[0000] Watch configuration for service "my-app-image":
  - Action rebuild for path "/Users/kbregula//compose_develop_seperate_image/requirements.txt"
Watch enabled
DEBU[0005] otel error                                    error="<nil>"

As you can see, one service is not tracked.

In a separate terminal, I check if synchronization is working.

$ curl localhost:3001
INSTANCE_NAME: instance1
REQUIREMENTS:
flask

MAIN_APP:
import textwrap

from flask import Flask

import os;
from pathlib import Path

INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'Hello World!')

app = Flask(__name__)

@app.route("/")
def hello():
    response = f"INSTANCE_NAME: {INSTANCE_NAME}\n"
    response += f"REQUIREMENTS: \n{Path('requirements.txt').read_text()}\n"
    response += f"MAIN_APP: \n{Path(__file__).read_text()}\n"
    return textwrap.dedent(response)


if __name__ == "__main__":
    app.run()
$ curl localhost:3002
INSTANCE_NAME: instance2
REQUIREMENTS:
flask

MAIN_APP:
import textwrap

from flask import Flask

import os;
from pathlib import Path

INSTANCE_NAME = os.environ.get('INSTANCE_NAME', 'Hello World!')

app = Flask(__name__)

@app.route("/")
def hello():
    response = f"INSTANCE_NAME: {INSTANCE_NAME}\n"
    response += f"REQUIREMENTS: \n{Path('requirements.txt').read_text()}\n"
    response += f"MAIN_APP: \n{Path(__file__).read_text()}\n"
    return textwrap.dedent(response)


if __name__ == "__main__":
    app.run()

Then I make a change in the source code:

echo "# TEST $(date)" >> src/main.py

Next I checked again:

$ curl -s localhost:3001 | grep TEST
# TEST Sat Aug 17 17:42:27 CEST 2024
$ curl -s localhost:3002 | grep TEST

Here's the full log.

$ docker --debug --log-level=debug compose watch --no-up
DEBU[0000] Enabled Docker Desktop integration (experimental) @ unix:///Users/kbregula/Library/Containers/com.docker.docker/Data/docker-cli.sock
DEBU[0000] Watch configuration for service "my-app-image":
  - Action rebuild for path "/Users/my-user/compose_develop_seperate_image/requirements.txt"
DEBU[0000] Watch configuration for service "my-app-instance-1":
  - Action sync for path "/Users/my-user/compose_develop_seperate_image/src"
Watch enabled
DEBU[0005] otel error                                    error="<nil>"
DEBU[0090] change for /Users/my-user/compose_develop_seperate_image/src/main.py - comparing with /Users/my-user/compose_develop_seperate_image/src
DEBU[0090] batch start: service[my-app-instance-1] count[1]
Syncing "my-app-instance-1" after changes were detected
DEBU[0090] batch complete: service[my-app-instance-1] duration[29.016667ms] count[1]
DEBU[0095] otel error                                    error="<nil>"

I will add that if a change is made in dependencies (requirements.txt), the image is correctly updated.

$ echo "### TEST2 $(date)" >> requirements.txt
curl -s localhost:3001 | grep TEST2
### TEST2 Sat Aug 17 17:45:45 CEST 2024
$ curl -s localhost:3002 | grep TEST2
### TEST2 Sat Aug 17 17:45:45 CEST 2024

Compose Version

Docker Compose version v2.29.1-desktop.1

Docker Environment

Client:
 Version:    27.1.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /Users/my-user/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /Users/my-user/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/my-user/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/my-user/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/my-user/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/my-user/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/my-user/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/my-user/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/my-user/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /Users/my-user/.docker/cli-plugins/docker-scout

Server:
 Containers: 10
  Running: 2
  Paused: 0
  Stopped: 8
 Images: 18
 Server Version: 27.1.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.0-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 10
 Total Memory: 16.07GiB
 Name: docker-desktop
 ID: 
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/my-user/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Anything else?

No response

@idsulik
Copy link
Collaborator

idsulik commented Aug 18, 2024

continue
here is the line where it skips the service, because next line requires the build context.
watch.LoadDockerIgnore(service.Build.Context)
dockerIgnores, err := watch.LoadDockerIgnore(service.Build.Context)

@ndeloof
Copy link
Contributor

ndeloof commented Aug 19, 2024

each of your service should be fully defined, even those share a common image. To avoid copy-pasting the build definition you can use extends or rely on yaml anchors to get the same build section applied to other services

@jhrotko jhrotko added area/watch Related with watch feature and removed status/0-triage labels Oct 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/watch Related with watch feature kind/bug
Projects
None yet
Development

No branches or pull requests

4 participants