containerd can't pull local core images #425

nordanz · 2023-02-08T11:39:06Z

nordanz
Feb 8, 2023

Hello everybody,

I was wondering if somebody else as well had the same issue that I have. after deploying the core images this is the status of my pods:

meep-auth-svc-57ff44554b-g2qbz                                0/1     ImagePullBackOff   0             5m3s
meep-auth-svc-5b46695b7c-j8cxx                                0/1     ImagePullBackOff   0             24h
meep-couchdb-couchdb-0                                        1/1     Running            1 (24h ago)   24h
meep-docker-registry-86b6b6b849-9w7pz                         1/1     Running            1 (24h ago)   24h
meep-grafana-66fff596d-5ct54                                  1/1     Running            1 (24h ago)   24h
meep-influxdb-0                                               1/1     Running            1 (24h ago)   24h
meep-ingress-controller-ck65s                                 1/1     Running            1 (24h ago)   24h
meep-ingress-defaultbackend-5d8778d54f-5zhkc                  1/1     Running            1 (24h ago)   24h
meep-kube-state-metrics-6c75f9d88c-xzmcf                      1/1     Running            1 (24h ago)   24h
meep-mon-engine-57d4b44f64-z8876                              0/1     ErrImagePull       0             24h
meep-open-map-tiles-5695d8f67-w8fkm                           1/1     Running            1 (24h ago)   24h
meep-platform-ctrl-8b9fd68c4-mhfxg                            0/1     ImagePullBackOff   0             24h
meep-postgis-0                                                2/2     Running            2 (24h ago)   24h
meep-prometheus-operator-7bffcbf4d7-8l6mk                     1/1     Running            1 (24h ago)   24h
meep-prometheus-prometheus-node-exporter-rff9m                1/1     Running            1 (24h ago)   24h
meep-redis-master-0                                           2/2     Running            2 (24h ago)   24h
meep-redis-slave-0                                            2/2     Running            4 (23h ago)   24h
meep-virt-engine-6c8b9f7f68-hchg2                             0/1     ImagePullBackOff   0             24h
meep-webhook-f97d4599b-dx6rf                                  0/1     ImagePullBackOff   0             24h
nvidia-container-toolkit-daemonset-9qds4                      1/1     Running            1 (24h ago)   25h
nvidia-cuda-validator-xfl4k                                   0/1     Completed          0             23h
nvidia-dcgm-exporter-9flp4                                    1/1     Running            1 (24h ago)   25h
nvidia-device-plugin-daemonset-5685s                          1/1     Running            1 (24h ago)   25h
nvidia-device-plugin-validator-r85pv                          0/1     Completed          0             23h
nvidia-operator-validator-v55g5                               1/1     Running            1 (24h ago)   25h
prometheus-meep-prometheus-prometheus-0                       2/2     Running            9 (24h ago)   24h

As you can see it cannot pull the images, this is the output of manually pulling the image using docker

$ docker pull meep-docker-registry:30001/meep-ams
Using default tag: latest
latest: Pulling from meep-ams
Digest: sha256:e2c8a5c6c11ce46cff9382fc4672bc8af6e0d412bd4c8eb953d25ce2d58a9173
Status: Image is up to date for meep-docker-registry:30001/meep-ams:latest
meep-docker-registry:30001/meep-ams:latest

this is the output of the host resolve commands

$ nslookup meep-docker-registry
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   meep-docker-registry
Address: 10.14.56.42

$ resolvectl query --legend=true meep-docker-registry
meep-docker-registry: 10.14.56.42

-- Information acquired via protocol DNS in 718us.
-- Data is authenticated: yes

And this is the result of pulling image using ctr:

$ sudo ctr image pull meep-docker-registry:30001/meep-auth-svc:latest
INFO[0000] trying next host - response was http.StatusNotFound  host="meep-docker-registry:30001"
meep-docker-registry:30001/meep-auth-svc:latest: resolving      |--------------------------------------|
elapsed: 0.1 s                                   total:   0.0 B (0.0 B/s)
ERRO[0000] active check failed                           error="context canceled"
ctr: failed to resolve reference "meep-docker-registry:30001/meep-auth-svc:latest": meep-docker-registry:30001/meep-auth-svc:latest: not found

And at the end this is the Pod error:

$ kubectl describe pods meep-auth-svc-57ff44554b-g2qbz
Name:             meep-auth-svc-57ff44554b-g2qbz
Namespace:        default
Priority:         0
Service Account:  meep-auth-svc
Node:             fe2z0380/10.14.56.42
Start Time:       Wed, 08 Feb 2023 12:18:39 +0100
Labels:           app=meep-auth-svc
                  meepOrigin=core
                  pod-template-hash=57ff44554b
                  release=meep-auth-svc
Annotations:      kubectl.kubernetes.io/restartedAt: 2023-02-08T12:18:38+01:00
Status:           Pending
IP:               10.32.0.27
IPs:
  IP:           10.32.0.27
Controlled By:  ReplicaSet/meep-auth-svc-57ff44554b
Init Containers:
  init-system-kube-dns:
    Container ID:  containerd://e79e79c40193f9cf2b9afb8e232b05c5c87e36bc944128a50e3f97a6543bd575
    Image:         busybox:1.28
    Image ID:      docker.io/library/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      until nslookup kube-dns.kube-system ; do echo waiting for kube-dns; sleep 0.25; done;
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 08 Feb 2023 12:18:42 +0100
      Finished:     Wed, 08 Feb 2023 12:18:42 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2r84 (ro)
Containers:
  meep-auth-svc:
    Container ID:
    Image:          meep-docker-registry:30001/meep-auth-svc:latest
    Image ID:
    Ports:          80/TCP, 9000/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:
      MEEP_HOST_URL:                   http://10.14.56.42
      MEEP_MAX_SESSIONS:               10
      MEEP_OAUTH_GITHUB_AUTH_URL:
      MEEP_OAUTH_GITHUB_ENABLED:       false
      MEEP_OAUTH_GITHUB_REDIRECT_URI:
      MEEP_OAUTH_GITHUB_TOKEN_URL:
      MEEP_OAUTH_GITLAB_API_URL:
      MEEP_OAUTH_GITLAB_AUTH_URL:
      MEEP_OAUTH_GITLAB_ENABLED:       false
      MEEP_OAUTH_GITLAB_REDIRECT_URI:
      MEEP_OAUTH_GITLAB_TOKEN_URL:
      MEEP_SVC_PATH:                   /auth/v1
      MEEP_OAUTH_GITHUB_CLIENT_ID:     <set to the key 'client-id' in secret 'github-secret'>      Optional: true
      MEEP_OAUTH_GITHUB_SECRET:        <set to the key 'secret' in secret 'github-secret'>         Optional: true
      MEEP_OAUTH_GITLAB_CLIENT_ID:     <set to the key 'client-id' in secret 'gitlab-secret'>      Optional: true
      MEEP_OAUTH_GITLAB_SECRET:        <set to the key 'secret' in secret 'gitlab-secret'>         Optional: true
      MEEP_SESSION_KEY:                <set to the key 'encryption-key' in secret 'meep-session'>  Optional: true
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r2r84 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-r2r84:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  13m                  default-scheduler  Successfully assigned default/meep-auth-svc-57ff44554b-g2qbz to fe2z0380
  Normal   Pulled     13m                  kubelet            Container image "busybox:1.28" already present on machine
  Normal   Created    13m                  kubelet            Created container init-system-kube-dns
  Normal   Started    13m                  kubelet            Started container init-system-kube-dns
  Warning  Failed     11m (x5 over 13m)    kubelet            Error: ImagePullBackOff
  Normal   Pulling    11m (x4 over 13m)    kubelet            Pulling image "meep-docker-registry:30001/meep-auth-svc:latest"
  Warning  Failed     11m (x4 over 13m)    kubelet            Failed to pull image "meep-docker-registry:30001/meep-auth-svc:latest": rpc error: code = NotFound desc = failed to pull and unpack image "meep-docker-registry:30001/meep-auth-svc:latest": failed to resolve reference "meep-docker-registry:30001/meep-auth-svc:latest": meep-docker-registry:30001/meep-auth-svc:latest: not found
  Warning  Failed     11m (x4 over 13m)    kubelet            Error: ErrImagePull
  Normal   BackOff    3m9s (x41 over 13m)  kubelet            Back-off pulling image "meep-docker-registry:30001/meep-auth-svc:latest"

Did I miss any step in setting up the docker-registery or some containerd configuration? any idea how can I fix this?

containerd: containerd://1.6.14
docker: 1.20.22
K8s version: 1.26.0

dilallkx · 2023-04-11T13:53:47Z

dilallkx
Apr 11, 2023
Maintainer

Hi @nordanz,

Thank you for reaching out, and apologies for the delayed response!

Do you still observe the above issue?
If so, could you please confirm the versions you are currently using?

Best regards!

2 replies

nordanz Sep 21, 2023
Author

Hello,

I tried with another machine and it was working there. couldn't find the root cause and it was the latest version.

dilallkx Sep 21, 2023
Maintainer

Hi @nordanz,
Thank you for the update!

Urwah92 · 2023-09-11T12:02:20Z

Urwah92
Sep 11, 2023

Hi @dilallkx ,
I am receiving a similar error. Please can you inform me about its solution. I have adhered to the versions specified in the tutorials.
Thanks.

3 replies

Urwah92 Sep 11, 2023

I am receiving the following error while following the Advantedge installation tutorial.

dilallkx Sep 19, 2023
Maintainer

Hi @Urwah92,

Are you still observing this issue?
Could you please confirm that you are using the latest AdvantEDGE release?

Regards!

nordanz Sep 21, 2023
Author

Check this as well, not sure but if it helps:
https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration
https://github.com/containerd/containerd/blob/main/docs/cri/registry.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

containerd can't pull local core images #425

{{title}}

Replies: 2 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

containerd can't pull local core images #425

nordanz Feb 8, 2023

Replies: 2 comments · 5 replies

dilallkx Apr 11, 2023 Maintainer

nordanz Sep 21, 2023 Author

dilallkx Sep 21, 2023 Maintainer

Urwah92 Sep 11, 2023

Urwah92 Sep 11, 2023

dilallkx Sep 19, 2023 Maintainer

nordanz Sep 21, 2023 Author

nordanz
Feb 8, 2023

Replies: 2 comments 5 replies

dilallkx
Apr 11, 2023
Maintainer

nordanz Sep 21, 2023
Author

dilallkx Sep 21, 2023
Maintainer

Urwah92
Sep 11, 2023

dilallkx Sep 19, 2023
Maintainer

nordanz Sep 21, 2023
Author