Earthly doesn't respect a host defined in a local /etc/hosts file when doing SAVE IMAGE

[stevenj]

What went wrong?

I have this line in my earthfile:

SAVE IMAGE --push --insecure registry.cluster.test/cat-voices-docs:latest

And in my /etc/hosts file i have defined:

192.168.58.10   registry.cluster.test       # The local container registry

If i run the target with --push it fails with:

 output | --> exporting outputs
              output | [----------] 100% exporting outputs
Error: failed to do request: Head "https://registry.cluster.test/v2/cat-voices-docs/blobs/sha256:57e0e85367e114e9012cfa0cc7f3a815f90441bd011418289516dae791fb287a": dial tcp: lookup registry.cluster.test on 192.168.69.1:53: no such host

So, it's not using the machines local /etc/hosts file at all, and resorting to upstream DNS query.

What should have happened?

A hostname lookup from earthly for SAVE IMAGE should resolve from /etc/hosts before resorting to an upstream DNS query.
Ideally this would apply to all hostname resolution, not just SAVE IMAGE.

What earthly version?

earthly version v0.8.12 7dbb0ea linux/amd64; Manjaro Linux

Buildkit Logs

Latest relevant log messages:

time="2024-05-29T07:02:05Z" level=error msg="/moby.buildkit.v1.frontend.LLBBridge/Export returned error: rpc error: code = Unknown desc = failed to do request: Head \"https://registry.cluster.test/v2/cat-voices-docs/blobs/sha256:57e0e85367e114e9012cfa0cc7f3a815f90441bd011418289516dae791fb287a\": dial tcp: lookup registry.cluster.test on 192.168.69.1:53: no such host" revision=531b303aa8ec03c29c2ceaa140eb0a6d32e6f6f3 version=v0.8.12
time="2024-05-29T07:02:25Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to SAVE IMAGE: failed to do request: Head \"https://registry.cluster.test/v2/cat-voices-docs/blobs/sha256:57e0e85367e114e9012cfa0cc7f3a815f90441bd011418289516dae791fb287a\": dial tcp: lookup registry.cluster.test on 192.168.69.1:53: no such host" revision=531b303aa8ec03c29c2ceaa140eb0a6d32e6f6f3 version=v0.8.12

Other Helpful Information

Curl can resolve the address as expected:

❯ curl -qv registry.cluster.test/v2/
* Host registry.cluster.test:80 was resolved.
* IPv6: (none)
* IPv4: 192.168.58.10
*   Trying 192.168.58.10:80...
* Connected to registry.cluster.test (192.168.58.10) port 80
> GET /v2/ HTTP/1.1
> Host: registry.cluster.test
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< Content-Length: 2
< Content-Type: application/json; charset=utf-8
< Date: Wed, 29 May 2024 05:39:40 GMT
< Docker-Distribution-Api-Version: registry/2.0
< Vary: Origin
< X-Content-Type-Options: nosniff
<
* Connection #0 to host registry.cluster.test left intact
{}⏎                                

As can docker push

❯ docker push registry.cluster.test/nginy.later
Using default tag: latest
The push refers to repository [registry.cluster.test/nginy.later]
14773070094d: Pushed
7d2fd59c368c: Pushed
56f8fe6aedcd: Pushed
9f4d73e635f1: Pushed
747b290aeba8: Pushed
fc1cf9ca5139: Pushed
5d4427064ecc: Pushed
latest: digest: sha256:0e1ac7f12d904a5ce077d1b5c763b5750c7985e524f6083e5eaa7e7313833440 size: 1778

[alexcb]

This issue stems from buildkitd running out of a seperate container that does use the host's network resolver (or /etc/hosts file).

e.g. docker exec earthly-buildkitd /bin/sh -c 'cat /etc/hosts' will be different from the host's version.

Futhermore, go (by default) does not use the /etc/hosts file (a questionable decision IMO).

[alexcb]

does it work with the IP address instead? unfortunately this isn't a trivial fix.