Container does not get updated

[realies]

Describe the bug

When the image of a docker compose service is updated, it gets pulled, but the docker container does not get recreated or stopped. I'm not sure if it's related to compose.

Steps to reproduce

version: "3"
services:
  watchtower:
    container_name: watchtower
    image: containrrr/watchtower
    restart: unless-stopped
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 20 5 * * *
      - WATCHTOWER_REMOVE_VOLUMES=true
      - WATCHTOWER_DISABLE_CONTAINERS=wake-on-lan-service
      - WATCHTOWER_NOTIFICATIONS=shoutrrr
      - WATCHTOWER_NOTIFICATION_URL=x
      - WATCHTOWER_DEBUG=true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Expected behavior

The container stops and gets recreated with the latest image

Screenshots

No response

Environment

• rpi4
• arm64
• Docker version 27.0.3, build 7d4bcd8

Your logs

watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Parsing image ref" host=index.docker.io image=zoeyvid/npmplus normalized=docker.io/zoeyvid/npmplus tag=latest
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/zoeyvid/npmplus/manifests/latest"
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Found a remote digest to compare with" remote="sha256:661f67996f2293717f8f82dfc7b546494d63bea5689c30aa90aa1bd71b4abeb4"
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg=Comparing local="sha256:adb7e1f2d47724ceb27e494a1f2db3d21401b08311ccba2393b2b9e79017fdfa" remote="sha256:661f67996f2293717f8f82dfc7b546494d63bea5689c30aa90aa1bd71b4abeb4"
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Digests did not match, doing a pull."
watchtower  | time="2024-07-14T05:26:03Z" level=debug msg="Pulling image" container=/npmplus image="zoeyvid/npmplus:latest"
watchtower  | time="2024-07-14T05:26:04Z" level=info msg="Found new zoeyvid/npmplus:latest image (43ff4b4c4210)"
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="Trying to load authentication credentials." container=/dpaste image="darrenofficial/dpaste:latest"
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="No credentials for index.docker.io found" config_file=/config.json
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="Got image name: darrenofficial/dpaste:latest"
watchtower  | time="2024-07-14T05:26:04Z" level=debug msg="Checking if pull is needed" container=/dpaste image="darrenofficial/dpaste:latest"
...
watchtower  | time="2024-07-14T05:26:10Z" level=info msg="Session done" Failed=0 Scanned=7 Updated=1 notify=no

Additional context

No response

[github-actions]

Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏

[ciberboy]

I have a similar issue.

This is the configuration I am using:
docker create
--name watchtower
-e PUID=0
-e PGID=0
-e WATCHTOWER_CLEANUP=TRUE
-e WATCHTOWER_DEBUG=TRUE
-e WATCHTOWER_POLL_INTERVAL=300
-v /var/run/docker.sock:/var/run/docker.sock
-v $HOME/.docker/config.json:/config.json
--restart unless-stopped
containrrr/watchtower

These are the logs:
time="2024-08-01T12:32:04Z" level=debug msg="Trying to load authentication credentials." container=/photoprism image="photoprism/photoprism:latest"
time="2024-08-01T13:04:59Z" level=debug msg="Retrieving running containers"
time="2024-08-01T12:32:04Z" level=debug msg="Loaded auth credentials for user XXXXXXXXXXXXX, on registry index.docker.io, from file /config.json"
time="2024-08-01T12:32:04Z" level=debug msg="Got image name: photoprism/photoprism:latest"
time="2024-08-01T12:32:04Z" level=debug msg="Credentials loaded"
time="2024-08-01T12:32:04Z" level=debug msg="Checking if pull is needed" container=/photoprism image="photoprism/photoprism:latest"
time="2024-08-01T12:32:04Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2024-08-01T12:32:05Z" level=debug msg="Got response to challenge request" header="Bearer realm="https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2024-08-01T12:32:05Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2024-08-01T12:32:05Z" level=debug msg="Setting scope for auth token" image=docker.io/photoprism/photoprism scope="repository:photoprism/photoprism:pull"
time="2024-08-01T12:32:05Z" level=debug msg="Credentials found."
time="2024-08-01T12:32:05Z" level=debug msg="Parsing image ref" host=index.docker.io image=photoprism/photoprism normalized=docker.io/photoprism/photoprism tag=latest
time="2024-08-01T12:32:05Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/photoprism/photoprism/manifests/latest"
time="2024-08-01T12:32:06Z" level=debug msg="Found a remote digest to compare with" remote="sha256:b8dc65e14808021dff7fdb60723fe7e7cd224db1fd356c4e3f1892fa4912e4e4"
time="2024-08-01T12:32:06Z" level=debug msg=Comparing local="sha256:3b6a64d86abb566b5314dc7b168476e421ca7322b9102c1bd9c79834c6bc6756" remote="sha256:b8dc65e14808021dff7fdb60723fe7e7cd224db1fd356c4e3f1892fa4912e4e4"
time="2024-08-01T12:32:06Z" level=debug msg="Digests did not match, doing a pull."
time="2024-08-01T12:32:06Z" level=debug msg="Pulling image" container=/photoprism image="photoprism/photoprism:latest"
time="2024-08-01T12:32:07Z" level=debug msg="No new images found for /photoprism"
time="2024-08-01T12:32:07Z" level=debug msg="Trying to load authentication credentials." container=/mariadb image="mariadb:10.11"
time="2024-08-01T12:32:07Z" level=debug msg="Loaded auth credentials for user XXXXXXXXXXXXX, on registry index.docker.io, from file /config.json"
time="2024-08-01T12:32:07Z" level=debug msg="Got image name: mariadb:10.11"
time="2024-08-01T12:32:07Z" level=debug msg="Credentials loaded"
time="2024-08-01T12:32:07Z" level=debug msg="Checking if pull is needed" container=/mariadb image="mariadb:10.11"
time="2024-08-01T12:32:07Z" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/"
time="2024-08-01T12:32:08Z" level=debug msg="Got response to challenge request" header="Bearer realm="https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized"
time="2024-08-01T12:32:08Z" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io
time="2024-08-01T12:32:08Z" level=debug msg="Setting scope for auth token" image=docker.io/library/mariadb scope="repository:library/mariadb:pull"
time="2024-08-01T12:32:08Z" level=debug msg="Credentials found."
time="2024-08-01T12:32:08Z" level=debug msg="Parsing image ref" host=index.docker.io image=library/mariadb normalized=docker.io/library/mariadb tag=10.11
time="2024-08-01T12:32:08Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/mariadb/manifests/10.11"
time="2024-08-01T12:32:08Z" level=debug msg="Found a remote digest to compare with" remote="sha256:75f6e61397758489d1dccf95db33b6b49ebfc7ec1253d40060fdf8ceb7f938a3"
time="2024-08-01T12:32:08Z" level=debug msg=Comparing local="sha256:385c0ba6270307511a511ad915350736602c33fafb3f63f663c853920424c365" remote="sha256:75f6e61397758489d1dccf95db33b6b49ebfc7ec1253d40060fdf8ceb7f938a3"
time="2024-08-01T12:32:08Z" level=debug msg="Digests did not match, doing a pull."
time="2024-08-01T12:32:08Z" level=debug msg="Pulling image" container=/mariadb image="mariadb:10.11"
time="2024-08-01T12:32:11Z" level=debug msg="No new images found for /mariadb"
time="2024-08-01T12:32:11Z" level=debug msg="This is the watchtower container /watchtower"
time="2024-08-01T12:32:11Z" level=info msg="Session done" Failed=0 Scanned=11 Updated=0 notify=no
time="2024-08-01T12:32:11Z" level=debug msg="Scheduled next run: 2024-08-01 12:36:48 +0000 UTC"
time="2024-08-01T12:33:20Z" level=info msg="Waiting for running update to be finished..."
time="2024-08-01T13:04:58Z" level=debug msg="Sleeping for a second to ensure the docker api client has been properly initialized."
time="2024-08-01T13:04:59Z" level=debug msg="Making sure everything is sane before starting"

There's a specific message regarding authentication, but I made sure I am using a current and valid docker token:

    "auths": {
            "https://index.docker.io/v1/": {
                    "auth": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX>
            },
            "https://index.docker.io/v2/": {}
    }

}
(I have replaced the token generated by the login action with X)

I am using for all my docker images the platform tag for x64. I don't know if that is that is the root cause of the issue, but I doubt it.

Additionally, watchtower was working fine up until a month ago. I haven't changed nothing in the meantime.

Is there a chance someone can help?

Thank you in advance for your help.

[ciberboy]

Anyone? Please help!

[PopC0rnwalk]

Since las 2-3 month I think, container images are not getting updated.

Checking the logs I see that watchtower is detecting same image hash, somehow it is not getting the new image hash from index.docker.io.

I have been checking the images in the docker hub and they have been updated, watchtowerr is not getting that info and new hash.

Here are some logs:

time="2024-08-11T05:30:01+02:00" level=debug msg="Trying to load authentication credentials." container=/website-wordpress-1 image="wordpress:latest" time="2024-08-11T05:30:01+02:00" level=debug msg="No credentials for index.docker.io found" config_file=/config.json time="2024-08-11T05:30:01+02:00" level=debug msg="Got image name: wordpress:latest" time="2024-08-11T05:30:01+02:00" level=debug msg="Checking if pull is needed" container=/website-wordpress-1 image="wordpress:latest" time="2024-08-11T05:30:01+02:00" level=debug msg="Built challenge URL" URL="https://index.docker.io/v2/" time="2024-08-11T05:30:01+02:00" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\"" status="401 Unauthorized" time="2024-08-11T05:30:01+02:00" level=debug msg="Checking challenge header content" realm="https://auth.docker.io/token" service=registry.docker.io time="2024-08-11T05:30:01+02:00" level=debug msg="Setting scope for auth token" image=docker.io/library/wordpress scope="repository:library/wordpress:pull" time="2024-08-11T05:30:01+02:00" level=debug msg="No credentials found." time="2024-08-11T05:30:01+02:00" level=debug msg="Parsing image ref" host=index.docker.io image=library/wordpress normalized=docker.io/library/wordpress tag=latest time="2024-08-11T05:30:01+02:00" level=debug msg="Doing a HEAD request to fetch a digest" url="https://index.docker.io/v2/library/wordpress/manifests/latest" time="2024-08-11T05:30:02+02:00" level=debug msg="Found a remote digest to compare with" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb627d46634" time="2024-08-11T05:30:02+02:00" level=debug msg=Comparing local="sha256:4bc1649284319b09f6f8da1a181616c196f4e225ad9a7d7901b179e3fb421070" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb62 7d46634" time="2024-08-11T05:30:02+02:00" level=debug msg=Comparing local="sha256:6c49d06a62950e6866c0fc9ee0fed6b0605da895db3ec6fa1036660a054a5215" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb62 7d46634" time="2024-08-11T05:30:02+02:00" level=debug msg=Comparing local="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb627d46634" remote="sha256:92951775334a184513ebc2a7bee22ad9848507be924c5df9f0b3ddb62 7d46634" time="2024-08-11T05:30:02+02:00" level=debug msg="Found a match" time="2024-08-11T05:30:02+02:00" level=debug msg="No pull needed. Skipping image." time="2024-08-11T05:30:02+02:00" level=debug msg="No new images found for /website-wordpress-1"

[cattomgithub]

Unfortunately, I meeted the same issue.
Here is my log :

time="2024-08-12T10:04:03Z" level=debug msg="This is the watchtower container /watchtower"
time="2024-08-12T10:04:03Z" level=info msg="Session done" Failed=0 Scanned=7 Updated=0 notify=no
time="2024-08-12T10:04:03Z" level=debug msg="Scheduled next run: 2024-08-12 10:04:57 +0000 UTC"
time="2024-08-12T10:04:57Z" level=debug msg="Checking containers for updated images"
time="2024-08-12T10:04:57Z" level=debug msg="Retrieving running containers"
time="2024-08-12T10:04:57Z" level=debug msg="Trying to load authentication credentials." container=/watchtower image="<my_private_registries>/containrrr/watchtower:1.7.1"
time="2024-08-12T10:04:57Z" level=debug msg="Loaded auth credentials for user cattom, on registry <my_private_registries>, from file /config.json"
time="2024-08-12T10:04:57Z" level=debug msg="Got image name: <my_private_registries>/containrrr/watchtower:1.7.1"
time="2024-08-12T10:04:57Z" level=debug msg="Credentials loaded"
time="2024-08-12T10:04:57Z" level=debug msg="Checking if pull is needed" container=/watchtower image="<my_private_registries>/containrrr/watchtower:1.7.1"
time="2024-08-12T10:04:57Z" level=debug msg="Built challenge URL" URL="https://<my_private_registries>/v2/"
time="2024-08-12T10:04:57Z" level=debug msg="Got response to challenge request" header="Basic realm=\"restricted\"" status="401 Unauthorized"
time="2024-08-12T10:04:57Z" level=debug msg="Parsing image ref" host=<my_private_registries> image=containrrr/watchtower normalized=<my_private_registries>/containrrr/watchtower tag=1.7.1
time="2024-08-12T10:04:57Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://<my_private_registries>/v2/containrrr/watchtower/manifests/1.7.1"
time="2024-08-12T10:04:58Z" level=debug msg="Found a remote digest to compare with" remote="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038"
time="2024-08-12T10:04:58Z" level=debug msg=Comparing local="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038" remote="sha256:6dd50763bbd632a83cb154d5451700530d1e44200b268a4e9488fefdfcf2b038"

[ldexterldesign]

You had one job 🙄

[ciberboy]

Apologies. I haven't been able to reproduce the error since I enabled debug logging. Will do as soon as it happens again

…

On Sat, 31 Aug 2024, 16:42 Lewis Dexter Litanzios, ***@***.***> wrote: You had one job 🙄 — Reply to this email directly, view it on GitHub <#1999 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJSISCCAN774OLQYVLYNGTZUHP4RAVCNFSM6AAAAABK24GRN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRSHEZTQMJSGU> . You are receiving this because you commented.Message ID: ***@***.***>

[yutotakano]

I encountered this issue today with Watchtower version 1.7.1. Similar relevant logs, where the digests do not match between local and remote, but Watchtower is unable to find new images after a subsequent pull.

I've had to redact the image and organization names.

time="2024-09-28T17:35:46Z" level=debug msg="Trying to load authentication credentials." container=/<image> image="ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:35:46Z" level=debug msg="Loaded auth credentials for user <redacted-sa>, on registry ghcr.io, from file /config.json"
time="2024-09-28T17:35:46Z" level=debug msg="Got image name: ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:35:46Z" level=debug msg="Credentials loaded"
time="2024-09-28T17:35:46Z" level=debug msg="Checking if pull is needed" container=/<image> image="ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:35:46Z" level=debug msg="Built challenge URL" URL="https://ghcr.io/v2/"
time="2024-09-28T17:35:46Z" level=debug msg="Got response to challenge request" header="Bearer realm=\"https://ghcr.io/token\",service=\"ghcr.io\",scope=\"repository:user/image:pull\"" status="401 Unauthorized"
time="2024-09-28T17:35:46Z" level=debug msg="Checking challenge header content" realm="https://ghcr.io/token" service=ghcr.io
time="2024-09-28T17:35:46Z" level=debug msg="Setting scope for auth token" image=ghcr.io/<redacted>/<image> scope="repository:<redacted>/<image>:pull"
time="2024-09-28T17:35:46Z" level=debug msg="Credentials found."
time="2024-09-28T17:35:46Z" level=debug msg="Parsing image ref" host=ghcr.io image=<redacted>/<image> normalized=ghcr.io/<redacted>/<image> tag=latest
time="2024-09-28T17:35:46Z" level=debug msg="Doing a HEAD request to fetch a digest" url="https://ghcr.io/v2/<redacted>/<image>/manifests/latest"
time="2024-09-28T17:35:47Z" level=debug msg="Found a remote digest to compare with" remote="sha256:aa91f7c79a35fce42037a2bfc8f83cba061ba53c70196372a26265f525a3bd9d"
time="2024-09-28T17:35:47Z" level=debug msg=Comparing local="sha256:04039f7a7e3a2983efdaefba3abf4813931db4534a088cd1511e007c2e28203a" remote="sha256:aa91f7c79a35fce42037a2bfc8f83cba061ba53c70196372a26265f525a3bd9d"
time="2024-09-28T17:35:47Z" level=debug msg="Digests did not match, doing a pull."
time="2024-09-28T17:35:47Z" level=debug msg="Pulling image" container=/<image> image="ghcr.io/<redacted>/<image>:latest"
time="2024-09-28T17:36:03Z" level=debug msg="No new images found for /<image>"