proxy.golang.org, cmd/go: 403 Forbidden on go get

[EliRibble]

Go version

go version go1.23.4 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/eliribble/.cache/go-build'
GOENV='/home/eliribble/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/eliribble/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/eliribble/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/jfv85qbj4vb1dafcg6kncg4vrbq2bbxv-go-1.23.4/share/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/jfv85qbj4vb1dafcg6kncg4vrbq2bbxv-go-1.23.4/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.4'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/eliribble/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/eliribble/src/sovr.cloud/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell-107044-0/go-build3527668268=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm working on NixOS, building my own flake of my own software. Ran nix develop which does a bunch of Nix stuff you don't care about. Within that command is a go get. The command worked fine on my laptop, fails on a cloud VPS with a 403 Forbidden.

What did you see happen?

$ nix develop
error: builder for '/nix/store/kiyqr9llfcxxjnri7n3ifh4ar71mbxwj-sovr-server-1.0.0-go-modules.drv' failed with exit code 1;
       last 25 log lines:
       > go: downloading google.golang.org/protobuf v1.34.1
       > go: downloading gopkg.in/yaml.v3 v3.0.1
       > go: downloading github.com/bytedance/sonic v1.11.6
       > go: downloading github.com/goccy/go-json v0.10.2
       > go: downloading github.com/json-iterator/go v1.1.12
       > go: downloading golang.org/x/sys v0.23.0
       > go: downloading github.com/x448/float16 v0.8.4
       > go: downloading github.com/gabriel-vasile/mimetype v1.4.3
       > go: downloading github.com/go-playground/universal-translator v0.18.1
       > go: downloading github.com/leodido/go-urn v1.4.0
       > go: downloading golang.org/x/text v0.17.0
       > go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
       > go: downloading github.com/modern-go/reflect2 v1.0.2
       > go: downloading github.com/go-playground/locales v0.14.1
       > go: downloading github.com/cloudwego/base64x v0.1.4
       > go: downloading golang.org/x/arch v0.8.0
       > go: downloading github.com/bytedance/sonic/loader v0.1.1
       > go: downloading github.com/klauspost/cpuid/v2 v2.2.7
       > go: downloading github.com/twitchyliquid64/golang-asm v0.15.1
       > go: downloading github.com/cloudwego/iasm v0.2.0
       > go: sovr.io imports
       >         github.com/gin-gonic/gin imports
       >       github.com/gin-gonic/gin/binding imports
       >       github.com/go-playground/validator/v10 imports
       >         github.com/gabriel-vasile/mimetype: github.com/gabriel-vasile/mimetype@v1.4.3: reading https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.3.zip: 403 Forbidden
       For full logs, run 'nix log /nix/store/kiyqr9llfcxxjnri7n3ifh4ar71mbxwj-sovr-server-1.0.0-go-modules.drv'.
error: 1 dependencies of derivation '/nix/store/ba086j7fnga3zh2c7sp27pzv9x27c8p7-sovr-server-1.0.0-env.drv' failed to build

I can reproduce the specific failure directly with go get:

GODEBUG=http2debug=1 GOPROXY=direct go get github.com/gabriel-vasile/mimetype@v1.4.3 2>&1 | tee go-mod-download-mimetype.txt
http2: Transport failed to get client conn for golang.org:443: http2: no cached connection was available
http2: Transport creating client conn 0xc000004000 to [2607:f8b0:4003:c0c::8d]:443
http2: Transport encoding header ":authority" = "golang.org"
http2: Transport encoding header ":method" = "GET"
http2: Transport encoding header ":path" = "/x/net?go-get=1"
http2: Transport encoding header ":scheme" = "https"
http2: Transport encoding header "accept-encoding" = "gzip"
http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
http2: Transport received SETTINGS len=18, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=65536
http2: Transport received WINDOW_UPDATE len=4 (conn) incr=983041
http2: Transport received SETTINGS flags=ACK len=0
http2: Transport received HEADERS flags=END_HEADERS stream=1 len=96
http2: Transport received DATA flags=END_STREAM stream=1 len=1584 data="<!DOCTYPE html>\n<html lang=en>\n  <meta charset=utf-8>\n  <meta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\">\n  <title>Error 403 (Forbidden)!!1</title>\n  <style>\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-ser" (1328 bytes omitted)
http2: Transport received PING len=8 ping="\x00\x00\x00\x00\x00\x00\x01c"

My IP addresses are:

    inet 194.135.104.85/24 brd 194.135.104.255 scope global eth0
    inet6 2605:e440::2:3e/64 scope global 

What did you expect to see?

I expected the mimetype module to download and be available to build software.

[gabyhelp]

Related Issues

• proxy.golang.org 403 forbidden much like #48107 #62448
• proxy.golang.org: 400 Bad Request #32765 (closed)
• proxy.golang.org 403 forbidden #48107 (closed)
• golang.org/x/text #64036 (closed)
• x/website: build broken since CL 322790 #46501 (closed)
• website: 403 Forbidden from golang.org #15642 (closed)
• net/http: ProxyFromEnvironment does not support socks5 #18508 (closed)
• cmd/go: get panics with "can't find reason for requirement on" #57037 (closed)
• go mod download 404 Not Found for private module #69381 (closed)
• go: writing stat cache:, permission denied #26814 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[EliRibble]

Working through some error triangulation steps from other related issues:

Specify a proxy directly, get 403 Forbidden

$ GOPROXY=https://proxy.golang.org go get github.com/gabriel-vasile/mimetype@v1.4.3
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: github.com/gabriel-vasile/mimetype@v1.4.3: reading https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.3.zip: 403 Forbidden

Specify direct (without a proxy) and get a different 403 Forbidden

$ GOPROXY=direct go get github.com/gabriel-vasile/mimetype@v1.4.3
go: golang.org/x/arch@v0.8.0: unrecognized import path "golang.org/x/arch": reading https://golang.org/x/arch?go-get=1: 403 Forbidden

Specify a third party proxy, don't get a 403 Forbidden

$ GOPROXY=https://goproxy.cn go get github.com/gabriel-vasile/mimetype@v1.4.3
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading golang.org/x/net v0.25.0
$

Another third party

$ GOPROXY=https://goproxy.io go get github.com/gabriel-vasile/mimetype@v1.4.3
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading golang.org/x/net v0.25.0
$

[seankhliao]

can you use curl to download the zip file?

[EliRibble]

Maybe? I'm not sure how exactly to translate the go get command to a comparable curl command. Do you know how?

[EliRibble]

Assuming this is valid:

curl -Lvo /tmp/mimetype https://proxy.golang.org/github.com/gabriel-vasile/mimetype/@v/v1.4.3.zip

I end up with /tmp/mimetype:

$ file /tmp/mimetype 
/tmp/mimetype: Zip archive data, at least v2.0 to extract, compression method=deflate

It appears to unzip to a valid archive.

[hyangah]

Looks like you got 403 not only while interacting with https://proxy.golang.org, but also when reading https://golang.org/x/arch?go-get=1 (in #71094 (comment)) and https://golang.org/x/net?go-get=1 (in the original report) in GOPROXY=direct mode. Given that the only shared part between proxy.golang.org and golang.org is the Google network (they are on different servers, and on different corners in Google), it may be an issue between Google frontend and the client. OTOH the last curl (on proxy.golang.org) succeeded, so that's strange. Does the problem persist?

[EliRibble]

Yeah, I can repro at will with:

$ go clean -modcache
$ go get github.com/gabriel-vasile/mimetype@v1.4.3

[hyangah]

Is it possible to try the similar setup from a machine with a different IP address?

[EliRibble]

Sure, tested from my laptop, works fine.

$ curl -4 icanhazip.com
98.171.80.66

$ go clean -modcache

$ go get github.com/gabriel-vasile/mimetype@v1.4.3
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
go: downloading golang.org/x/net v0.25.0

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/eliribble/.cache/go-build'
GOENV='/home/eliribble/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/eliribble/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/eliribble/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/nix/store/zhq8dnjbhwspzdglyq28j74axvqyk86q-go-1.23.3/share/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/nix/store/zhq8dnjbhwspzdglyq28j74axvqyk86q-go-1.23.3/share/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.3'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/eliribble/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/eliribble/src/sovr.cloud/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/nix-shell-19324-0/go-build1867468048=/tmp/go-build -gno-record-gcc-switches'