Running into what appears to be a fairly common issue with git-lfs failing to authenticate during push (verify locks)

[haikusw]

Reporting as a bug but not 100% sure what the root cause is: user issue, a git-lfs assumption/spec interpretation that GitHub server isn't agreeing with, a bug in git-lfs, false assumption about user systems, … ¯_(ツ)_/¯

I'm inclined to think user issue, but I can't figure out what it is. If nothing else, there's some documentation that needs writing (I've read everything I could find in the GitHub.com documents and elsewhere online and tried a lot of different things).

Describe the bug
git lfs push <remote> <branch> fails with Authentication error 403

This issue has been reported by numerous people over the years (e.g., #2114, #2219, #1852, #2361, etc, etc), and I've read all of those I could find. Few of them have actual fixes, and the few that do don't seem to fix the issue I'm having. Sometimes the response was "GitHub server issue - go talk to GitHub support", which was surprising. Surely a GitHub dev can get a ticket filed that gets paid a lot more attention to than we do; and maybe someone familiar with the git-lfs internals can shed enough light on the specific issue to help the server devs find and fix it, especially since it's been tripping people up for at least 5 years). If nothing else, there's a documentation improvement issue (I'd be happy to write something up if I could figure out what the root cause was).

To Reproduce

• I am attempting to use a GitHub personal access token.
• I have a GitHub hosted read-write access fork ("haikusw_origin") of a GitHub repo to which I don't have write access ("origin").
• This repo is using lfs.
• I have a branch ("async") checked out and tracking a branch on the read-only repo ("origin").
• I wish to push this branch ("async") to the fork I have read-write access to ("haikusw_origin").
• I execute the command git lfs push haikusw_origin async
• …and receive the following error:

error: Authentication error: Authentication required: You must have push access to verify locks

Expected behavior
The branch "async" should be created in the repo "haikusw_origin"

System environment
The version of your operating system, plus any relevant information about platform or configuration (e.g., container or CI usage, Cygwin, WSL, or non-Basic authentication). If relevant, include the output of git config -l as a code block.

• macOS X 12.3.1 on an M1 MBP

• zsh (default for macOS X 12.x)

• git version 2.35.2

• git-lfs version 3.1.2 (GitHub; Darwin arm 64; go 1.17.6)

• ssh-agent is running (ssh-agent dumps SSH_AUTH_SOCK and SSH_AGENT_PID values, set | grep SSH shows them set).

• echo "host=github.com\nprotocol=https" | git credential fill
  returns the expected output:

protocol=https
host=github.com
username=haikusw
password={correct personal token}

However, when I do GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async immediately before the POST that returns 403 I see the following two log entries which I then tried on the command line directly:

% /usr/bin/security 'list-keychains'

"/Users/tyler/Library/Keychains/login.keychain-db"
 …
"/Library/Keychains/System.keychain"

% /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'gist.github.com' '/Library/Keychains/System.keychain'
%
(no output)

There is no output from the second one, which doesn't seem all that surprising - I don't expect user certificates will be in the System keychain (?).

I wonder if this is a clue? I see in the git-lfs source code that there is an intentional ignoring of the login keychain "to match browsers" or something like that. Maybe this is unrelated, but it did catch my eye.

Git Config:

credential.helper=osxkeychain
mergetool.keepbackup=true
user.name=Tyler
user.email={email address}
core.excludesfile=/Users/tyler/.gitignore_global
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
difftool.Kaleidoscope.cmd=ksdiff --partial-changeset --relative-path "$MERGED" -- "$LOCAL" "$REMOTE"
mergetool.Kaleidoscope.cmd=ksdiff --merge --output "$MERGED" --base "$BASE" -- "$LOCAL" --snapshot "$REMOTE" --snapshot
mergetool.Kaleidoscope.trustexitcode=true
commit.template=/Users/tyler/.stCommitMsg
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.haikusw_origin.url=https://haikusw@github.com/haikusw/isowords.git
remote.haikusw_origin.gtserviceaccountidentifier=58ebcb006ed046c6e32cbbaad31fb77d0139ba9080ff2071fb3886dd99355400
remote.haikusw_origin.fetch=+refs/heads/*:refs/remotes/haikusw_origin/*
branch.haikusw_main.remote=haikusw_origin
branch.haikusw_main.merge=refs/heads/main
remote.origin.url=git@github.com:pointfreeco/isowords.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.main.remote=origin
branch.main.merge=refs/heads/main
branch.async.remote=origin
branch.async.merge=refs/heads/async
user.name=haikuty
user.email={email address}
lfs.repositoryformatversion=0

Output of git lfs env

Ok - I'm wondering why this output says "(auth=none)" at the end of the "haikusw_origin" endpoint?
That seems incorrect and possibly an indication of the root problem? (he says hopefully…).

% git lfs env
git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
git version 2.35.2

Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  SSH=git@github.com:pointfreeco/isowords.git
Endpoint (haikusw_origin)=https://haikusw@github.com/haikusw/isowords.git/info/lfs (auth=none)
LocalWorkingDir=/Users/Shared/OpenSource/isowords
LocalGitDir=/Users/Shared/OpenSource/isowords/.git
LocalGitStorageDir=/Users/Shared/OpenSource/isowords/.git
LocalMediaDir=/Users/Shared/OpenSource/isowords/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/Shared/OpenSource/isowords/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/Shared/OpenSource/isowords/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/opt/homebrew/Cellar/git/2.35.2/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Additional context
Any other relevant context about the problem here.

A different repo forked from the same organization doing the same operation (pushing a branch they have but my fork doesn't to my fork) succeeds as expected. It's only when using git-lfs that the error happens.

Checking git lfs env (though it doesn't have the lfs git hooks installed and I don't think it's using lfs), I see the same "(auth=none)" after the endpoint name. Maybe a red herring.

% git lfs env   
git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
git version 2.35.2

Endpoint=https://github.com/pointfreeco/swift-parsing.git/info/lfs (auth=none)
Endpoint (haikusw-origin)=https://haikusw@github.com/haikusw/swift-parsing.git/info/lfs (auth=none)
LocalWorkingDir=/Users/Shared/OpenSource/PointFree-swift-parsing
LocalGitDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git
LocalGitStorageDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git
LocalMediaDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git/lfs/objects
LocalReferenceDirs=
TempDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git/lfs/tmp
ConcurrentTransfers=8
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=/Users/Shared/OpenSource/PointFree-swift-parsing/.git/lfs
AccessDownload=none
AccessUpload=none
DownloadTransfers=basic,lfs-standalone-file,ssh
UploadTransfers=basic,lfs-standalone-file,ssh
GIT_EXEC_PATH=/opt/homebrew/Cellar/git/2.35.2/libexec/git-core
git config filter.lfs.process = "git-lfs filter-process"
git config filter.lfs.smudge = "git-lfs smudge -- %f"
git config filter.lfs.clean = "git-lfs clean -- %f"

Happy to try more things if there's something useful to try. Thanks.

[chrisd8088]

Hey, I'm sorry you're having trouble. If I understand correctly, I wonder if this is your problem:

I have a GitHub hosted read-write access fork ("haikusw_origin") of a GitHub repo to which I don't have write access ("origin").
I wish to push this branch ("async") to the fork I have read-write access to ("haikusw_origin").

According to their documentation, GitHub says (in the "Tips" box):

In forks, bandwidth and storage usage count against the root of the repository network....
Forking and pulling a repository counts against the parent repository's bandwidth limit.

Forges like GitLab and GitHub typically store all the forks of repository together in a way that minimizes duplicate files between the forks. However, at least for GitHub, this means Git LFS usage is tracked against the parent repository. So you may not be able to push your Git LFS objects even though your fork is read/write.

I would definitely recommend getting in touch with GitHub Support as they can look into your personal situation and (with your permission) repositories, whereas this project is just for the open-source Git LFS client.

As for the (auth=none) annotation in the git lfs env output, I agree that's rather mysterious. What it means is that you haven't explicitly configured a section in your Git config that looks like, for example:

[lfs "https://haikusw@github.com/haikusw/isowords.git/info/lfs"]
        access = basic

But typically you should not need to do that; Git LFS will infer the authorization scheme from the response it gets from the server.

[haikusw]

Thank you for the reply. Interesting to learn about the bandwidth and storage use counting against the root of the repository network - quite unexpected and there's no way I would have figured out that was what that error message meant. Hopefully that can get improved somehow if this is root cause. I'll give support a hello and report back what they say.

Also - thanks for the clarification about "(auth=none)". Definitely sends the wrong message :)
"(auth=unspecified)" or "(auth=inferred)", while longer, might be clearer.

[chrisd8088]

I've continued to try to investigate this from the client side, and I notice that a difference you report between the working and non-working environments is that the latter has an SSH endpoint while the former does not.

Are you able to switch to an HTTPS endpoint in your Git configuration and see if that works?

And, would you be able to provide a complete set of steps to reproduce the problem, starting from a git clone of your fork?

And as I noted, please do be mindful of the fact that fetching and adding Git LFS objects will be counted against the parent repository's usage by GitHub.

[haikusw]

Interesting! Thank you.
I should be able to do both those things later this evening and will report back.

Update on my support query: Support verified some basic facts and that it seems like it should work and then opened a ticket with engineering to investigate.

[haikusw]

During edits to my original post I seem to have lost the test for ssh connectivity that I did based on GitHub documentation:

% ssh -T git@github.com
Hi haikusw! You've successfully authenticated, but GitHub does not provide shell access.

[haikusw]

Hi @chrisd8088 - so I made a new folder and cloned the repo fresh being sure to use the https urls.

It appears that I git-lfs still get an authentication error but it's a 401 instead of a 403, and then git-lfs retried using basic repo access method and that succeeded!

I hope this gives you some clue to the what and why of this issue that people seem to run into off and on.

Here's the sequence and results (note that I did a different test immediately after this one; it's below this):

% git clone https://github.com/pointfreeco/isowords.git
...
% cd isowords

% git remote add haikusw_origin https://github.com/haikusw/isowords.git

% git remote -v
haikusw_origin	https://github.com/haikusw/isowords.git (fetch)
haikusw_origin	https://github.com/haikusw/isowords.git (push)
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

% git checkout async
branch 'async' set up to track 'origin/async'.
Switched to a new branch 'async'

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
20:39:20.625526 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
20:39:20.626129 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
20:39:20.681171 trace git-lfs: exec: git 'version'
20:39:20.686781 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
20:39:20.690735 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
20:39:20.694321 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
20:39:20.698481 trace git-lfs: exec: git 'config' '--includes' '-l'
20:39:20.701379 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:39:20.703902 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
20:39:20.707879 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
20:39:20.713282 trace git-lfs: Upload refs [async] to remote haikusw_origin
20:39:20.713425 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:39:20.717220 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:39:20.720169 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'
20:39:21.188744 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:39:21.219142 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
20:39:21.237585 trace git-lfs: HTTP: POST https://github.com/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}20:39:21.748874 trace git-lfs: HTTP: 401


< HTTP/2.0 401 Unauthorized
< Content-Length: 153
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 03:39:21 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: ECAC:3720:5DCDE6:C0B8F1:627C8169
< 
20:39:21.749126 trace git-lfs: HTTP: {"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"ECAC:3720:5DCDE6:C0B8F1:627C8169"}

{"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"ECAC:3720:5DCDE6:C0B8F1:627C8169"}

20:39:21.751221 trace git-lfs: setting repository access to basic
20:39:21.751411 trace git-lfs: exec: git 'config' '--includes' '--replace-all' 'lfs.https://github.com/haikusw/isowords.git/info/lfs.access' 'basic'
20:39:21.764002 trace git-lfs: api: http response indicates "basic" authentication. Resubmitting...
20:39:21.764397 trace git-lfs: creds: git credential fill ("https", "github.com", "")
20:39:21.764466 trace git-lfs: exec: git 'credential' 'fill'
20:39:21.808731 trace git-lfs: Filled credentials for https://github.com/haikusw/isowords.git
20:39:21.809149 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:39:21.822844 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
20:39:21.836043 trace git-lfs: HTTP: POST https://github.com/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}{"ref":{"name":"refs/heads/async"}}20:39:22.165743 trace git-lfs: HTTP: 200


< HTTP/2.0 200 OK
< Content-Length: 41
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 03:39:22 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: ECAD:611C:3C5467:92CD5E:627C816A
< 
20:39:22.165938 trace git-lfs: creds: git credential approve ("https", "github.com", "")
20:39:22.166057 trace git-lfs: exec: git 'credential' 'approve'
20:39:22.300295 trace git-lfs: HTTP: {"ours":[],"theirs":[],"next_cursor":""}

{"ours":[],"theirs":[],"next_cursor":""}
20:39:22.300799 trace git-lfs: tq: running as batched queue, batch size of 100
20:39:22.300915 trace git-lfs: run_command: git rev-list --objects --ignore-missing --not --remotes=haikusw_origin --stdin --
20:39:22.302014 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
20:39:22.303645 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'

push 73b2f912875308b6669232c2bad2456ae93f5aae954bc44cc2e45a0a50d34fdc => App/AppClip/Assets.xcassets/AppIcon.appiconset/default-1024-1024@1x-1.png
... many lines like the previous ...

20:39:22.341846 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
20:39:22.341869 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
20:39:22.341894 trace git-lfs: filepathfilter: accepting "tmp"

---

I don't understand why git remote -v shows this difference that you called out in my original clone and test folder

Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  SSH=git@github.com:pointfreeco/isowords.git

It's the HTTPS url but then has that SSH line below it.

Only think I can think of is that I added the remote with the Git Tower macOS git app (which I do generally use; when it started giving me errors I switched to command line to see if it was a Tower bug and ).

---

Out of curiosity I tried removing the haikusw_origin and then adding it back via the git@github.com… url.
Different behavior with different looking failures, but ultimately recovering and completing.

Here's that sequence of commands and outputs (which begin with things in the state at the end of the above sequence/test):

% git remote remove haikusw_origin
% git remote -v
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

% git remote add haikusw_origin git@github.com:haikusw/isowords.git
% git remote -v
haikusw_origin	git@github.com:haikusw/isowords.git (fetch)
haikusw_origin	git@github.com:haikusw/isowords.git (push)
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
20:49:25.953540 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
20:49:25.953948 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
20:49:25.999396 trace git-lfs: exec: git 'version'
20:49:26.005434 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
20:49:26.011366 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
20:49:26.016236 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
20:49:26.019658 trace git-lfs: exec: git 'config' '--includes' '-l'
20:49:26.022775 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
20:49:26.025358 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
20:49:26.029124 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'

20:49:26.033910 trace git-lfs: attempting pure SSH protocol connection

20:49:26.035187 trace git-lfs: run_command: ssh git@github.com git-lfs-transfer haikusw/isowords.git upload
20:49:26.036802 trace git-lfs: exec: ssh 'git@github.com' 'git-lfs-transfer haikusw/isowords.git upload'

20:49:27.039835 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF

20:49:27.041305 trace git-lfs: Upload refs [async] to remote haikusw_origin
20:49:27.041744 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:49:27.052994 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
20:49:27.060126 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'

20:49:28.276084 trace git-lfs: attempting pure SSH protocol connection

20:49:28.277091 trace git-lfs: run_command: ssh git@github.com git-lfs-transfer haikusw/isowords.git upload
20:49:28.277299 trace git-lfs: exec: ssh 'git@github.com' 'git-lfs-transfer haikusw/isowords.git upload'

20:49:29.244726 trace git-lfs: pure SSH protocol connection failed: Unable to negotiate version with remote side (unable to read capabilities): EOF

20:49:29.245339 trace git-lfs: run_command: ssh git@github.com git-lfs-authenticate haikusw/isowords.git upload
20:49:29.245621 trace git-lfs: exec: ssh 'git@github.com' 'git-lfs-authenticate haikusw/isowords.git upload'
20:49:30.318900 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:49:30.357009 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'lfs.github.com' '/Library/Keychains/System.keychain'
20:49:30.374200 trace git-lfs: HTTP: POST https://lfs.github.com/haikusw/isowords/locks/verify
> POST /haikusw/isowords/locks/verify HTTP/1.1
> Host: lfs.github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: RemoteAuth gitauth-v1-ga2G9UNrcUDLsS2bddhFKAHLLUjpHCo6sBfURxbQmhRl9lZElZLOYnzYKoKmbWVtYmVys3VzZXI6MjIyMjcxOmhhaWt1c3elcHJvdG-jc3No
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}

20:49:30.851353 trace git-lfs: HTTP: 200

< HTTP/2.0 200 OK
< Content-Length: 41
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 03:49:30 GMT
< X-Github-Request-Id: ECF7:419E:2A5BD8:7E3DC4:627C83CA
< 
20:49:30.851456 trace git-lfs: HTTP: {"ours":[],"theirs":[],"next_cursor":""}

{"ours":[],"theirs":[],"next_cursor":""}
20:49:30.853497 trace git-lfs: tq: running as batched queue, batch size of 100
20:49:30.853932 trace git-lfs: run_command: git rev-list --objects --ignore-missing --not --remotes=haikusw_origin --stdin --
20:49:30.856315 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
20:49:30.857915 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
push 73b2f912875308b6669232c2bad2456ae93f5aae954bc44cc2e45a0a50d34fdc => App/AppClip/Assets.xcassets/AppIcon.appiconset/default-1024-1024@1x-1.png

... many lines like the above for each asset ...

20:49:30.921261 trace git-lfs: filepathfilter: creating pattern ".git" of type gitignore
20:49:30.921284 trace git-lfs: filepathfilter: creating pattern "**/.git" of type gitignore
20:49:30.921312 trace git-lfs: filepathfilter: accepting "tmp"

---

It's odd that SSH connection isn't working. I tried connecting to GitHub via SSH again via ssh -T git@github.com and turned on verbose and it noted:

debug1: Authentication succeeded (publickey).
Authenticated to github.com ([140.82.114.4]:22).

and is using my ed25519 type 3 ssh key.

(there's a lot of debug output from ssh -Tv and I'm not sure how much of that is kosher to post).

---

Well, hopefully some of that leads to an "ah ha!" moment and either clarity on a bug or something I'm doing wrong or ?

[haikusw]

The other thing I thought of later:

Why does having the SSH= as part of the fork root impact the connection to the forked repo?

Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
  SSH=git@github.com:pointfreeco/isowords.git
Endpoint (haikusw_origin)=https://haikusw@github.com/haikusw/isowords.git/info/lfs (auth=none)

The SSH=git… is for the pointfreeco repo which I don't have write access too, and shouldn't need write access to.

The haikusw_origin Endpoint is just a pure HTTPS entry with no SSH aspect to it. So why does an aspect on the fork root have anything to do with the connection to verify locks on the dest repo?

The second question is: why is it failing when ssh -T git@github.com succeeds?

---

Realized I hadn't posted the full output of the failing connection, so let me post the here.

*** Odd thing is, I don't see it doing any authentication at all for this HTTP POST request.. and that just seems like a bug.

Perhaps I'm missing something.

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
08:32:10.571461 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
08:32:10.571963 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
08:32:10.646061 trace git-lfs: exec: git 'version'
08:32:10.651554 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
08:32:10.657385 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
08:32:10.664772 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
08:32:10.667904 trace git-lfs: exec: git 'config' '--includes' '-l'
08:32:10.670946 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
08:32:10.673662 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
08:32:10.677387 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
08:32:10.682163 trace git-lfs: Upload refs [async] to remote haikusw_origin
08:32:10.682243 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:32:10.685946 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:32:10.688650 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'
08:32:11.166265 trace git-lfs: exec: /usr/bin/security 'list-keychains'
08:32:11.190638 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
08:32:11.207274 trace git-lfs: HTTP: POST https://haikusw@github.com/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}08:32:11.652653 trace git-lfs: HTTP: 403


< HTTP/2.0 403 Forbidden
< Content-Length: 154
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 15:32:11 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: C59B:59A6:B2F112:18C7D70:627D287B
< 
08:32:11.652738 trace git-lfs: HTTP: {"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C59B:59A6:B2F112:18C7D70:627D287B"}

{"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C59B:59A6:B2F112:18C7D70:627D287B"}
error: Authentication error: Authentication required: You must have push access to verify locks

---

Just to be 100% sure this is the issue in my original repo that shows this issue I removed the origin repo and added it back as an HTTPS origin:

% git remote remove origin
% git remote add origin https://github.com/pointfreeco/isowords.git
% git remote -v
haikusw_origin	https://haikusw@github.com/haikusw/isowords.git (fetch)
haikusw_origin	https://haikusw@github.com/haikusw/isowords.git (push)
origin	https://github.com/pointfreeco/isowords.git (fetch)
origin	https://github.com/pointfreeco/isowords.git (push)

I then re-did the push command again and it fails in the same way even though the remote doesn't have the SSH bit on it for the root repo.

So, perhaps not the actual issue...?

Happy to try anything else you think might be useful / illuminating.

Here's the failure after changing origin to an HTTPS one in that working copy (still isn't visibly doing any authentication on that HTTPS POST call that I see in the logging?):

% GIT_TRACE=1 GIT_CURL_VERBOSE=1 git lfs push --dry-run haikusw_origin async
08:43:07.022683 git.c:745               trace: exec: git-lfs push --dry-run haikusw_origin async
08:43:07.023298 run-command.c:654       trace: run_command: git-lfs push --dry-run haikusw_origin async
08:43:07.063859 trace git-lfs: exec: git 'version'
08:43:07.068549 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
08:43:07.072833 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
08:43:07.077491 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
08:43:07.080421 trace git-lfs: exec: git 'config' '--includes' '-l'
08:43:07.083359 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
08:43:07.085969 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' ':.lfsconfig'
08:43:07.089603 trace git-lfs: exec: git 'config' '--includes' '-l' '--blob' 'HEAD:.lfsconfig'
08:43:07.094018 trace git-lfs: Upload refs [async] to remote haikusw_origin
08:43:07.094082 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:43:07.097252 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
08:43:07.099927 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'haikusw_origin'
08:43:07.556945 trace git-lfs: exec: /usr/bin/security 'list-keychains'
08:43:07.588748 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'github.com' '/Library/Keychains/System.keychain'
08:43:07.608552 trace git-lfs: HTTP: POST https://haikusw@github.com/haikusw/isowords.git/info/lfs/locks/verify
> POST /haikusw/isowords.git/info/lfs/locks/verify HTTP/1.1
> Host: github.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 35
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.1.2 (GitHub; darwin arm64; go 1.17.6)
> 
{"ref":{"name":"refs/heads/async"}}08:43:08.074734 trace git-lfs: HTTP: 403


< HTTP/2.0 403 Forbidden
< Content-Length: 153
< Content-Type: application/vnd.git-lfs+json
< Date: Thu, 12 May 2022 15:43:08 GMT
< X-Frame-Options: DENY
< X-Github-Request-Id: C5F0:67FD:181BB4:252D10:627D2B0B
< 
08:43:08.074941 trace git-lfs: HTTP: {"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C5F0:67FD:181BB4:252D10:627D2B0B"}

{"documentation_url":"https://github.com/contact","message":"You must have push access to verify locks","request_id":"C5F0:67FD:181BB4:252D10:627D2B0B"}
error: Authentication error: Authentication required: You must have push access to verify locks

[chrisd8088]

Thanks for the all the detail! Just to summarize where I believe things stand, and make a few notes:

1. It sounds like you have a workaround by using an HTTPS endpoint, which is great. Please correct me if I'm wrong about that.
2. You asked about the where the SSH part comes from here in the git lfs env output:

   Endpoint=https://github.com/pointfreeco/isowords.git/info/lfs (auth=none)
     SSH=git@github.com:pointfreeco/isowords.git
   

   That means that Git LFS was configured to use either a ssh:// URL or a "bare" git@github.com:... connection string, not an https:// one. (The HTTPS line is a bit misleading because when an SSH URL is provided the HTTPS one shown by git lfs env is constructed by Git LFS as a "backup" URL from the configured SSH one; it's not actually from your configuration.) You might look for that in a .lfsconfig or .git/config file in your original local repo (or in your global ~/.gitconfig, but that seems unlikely as it would affect your other recent fresh test clones as well). It would look something like (maybe with a ssh:// prefix):

   [lfs]
   	url = git@github.com:pointfreeco/isowords.git
   

3. You asked why the SSH connection isn't working re the example where the HTTPS connection works (in this comment above). I suspect this is because you noticed the pure SSH protocol connection failed error message in the trace output. If so, that's a red herring, and one I've been caught by too. Git LFS recently introduced a "pure SSH" transfer protocol on the client side, but no one has implemented it in production on the server side yet. So the client checks for support and then falls back to using HTTPS for the actual transfer; it only uses SSH for the authentication and authorization. You can see that in that example the SSH authorization did succeed; there are no error messages after the exec: ssh 'git@github.com' 'git-lfs-authenticate haikusw/isowords.git upload' trace line.

What I'm interested in is how to reproduce the failing case. I suspect this might have something to do with an [lfs] configuration in that repo's .lfsconfig or .git/config files, which would tell Git LFS to use an SSH (or "bare") URL for its endpoint. Note that this is different from the remotes you have configured with normal Git via git remote.

I'm still unsure if this is a client-side bug or a GitHub-side one.

[chrisd8088]

One thing that might help a bit is if you could post the output of git config -l in your original repo where things are failing. Just make sure to redact any personal info like authorization credentials or email addresses.

[haikusw]

Thanks for the all the detail! Just to summarize where I believe things stand, and make a few notes:

1. It sounds like you have a workaround by using an HTTPS endpoint, which is great. Please correct me if I'm wrong about that.

I have a second clone using and HTTPS endpoint and it works. The original cloned repo still doesn't work. Simply changing the remote to an HTTPS url in that clone didn't fix it (the last section of my most recent post details this).

So I'm still curious why that one continues to fail (and interested in helping track it down, if we can).

2. You asked about the where the SSH part comes from here in the git lfs env output:
   …

Very interesting!

I looked and I'm not seeing anything lfs related that overrides the endpoint in ~/.gitconfig, or the problematic clone's .git/config and no .lfsconfig files locally or globally that I'm seeing.

Here's the output of git config -l from the directory that doesn't work (after having done the remote change to an HTTPS endpoint as noted above), as you requested:

% git config -l
credential.helper=osxkeychain
mergetool.keepbackup=true
user.name=Tyler
user.email={REDACTED}
core.excludesfile=/Users/tyler/.gitignore_global
difftool.sourcetree.cmd=opendiff "$LOCAL" "$REMOTE"
difftool.sourcetree.path=
mergetool.sourcetree.cmd=/Applications/Sourcetree.app/Contents/Resources/opendiff-w.sh "$LOCAL" "$REMOTE" -ancestor "$BASE" -merge "$MERGED"
mergetool.sourcetree.trustexitcode=true
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
difftool.Kaleidoscope.cmd=ksdiff --partial-changeset --relative-path "$MERGED" -- "$LOCAL" "$REMOTE"
mergetool.Kaleidoscope.cmd=ksdiff --merge --output "$MERGED" --base "$BASE" -- "$LOCAL" --snapshot "$REMOTE" --snapshot
mergetool.Kaleidoscope.trustexitcode=true
commit.template=/Users/tyler/.stCommitMsg
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.haikusw_origin.url=https://haikusw@github.com/haikusw/isowords.git
remote.haikusw_origin.gtserviceaccountidentifier= {REDACTED}
remote.haikusw_origin.fetch=+refs/heads/*:refs/remotes/haikusw_origin/*
branch.haikusw_main.remote=haikusw_origin
branch.haikusw_main.merge=refs/heads/main
user.name=haikuty
user.email= {REDACTED}
lfs.repositoryformatversion=0
remote.origin.url=https://github.com/pointfreeco/isowords.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*

3. …
   Thanks for the explanation. Makes sense now :)

What I'm interested in is how to reproduce the failing case. I suspect this might have something to do with an [lfs] configuration in that repo's .lfsconfig or .git/config files, which would tell Git LFS to use an SSH (or "bare") URL for its endpoint. Note that this is different from the remotes you have configured with normal Git via git remote.

Me also! I have included (above) the git config -l you requested and I looked for any .lfsconfig files (none) or related entries in git config files (not seeing any).
Let me know if there's something else I can do to help.

I'm still unsure if this is a client-side bug or a GitHub-side one.

:) Or a user error (always a possibility!).
I appreciate your interest in tracking this down!

From GitHub support:

The engineering team are keeping an eye on the public issue you created so they can see everything going on there. If there's anything they need you to do in addition, or any output they need that shouldn't be posted publicly, I'll be back in touch!

So if we can figure out root cause and it's on GitHub server side, they should hear about it.

(I'm going to be really embarrassed if it's something dumb I've done :), but this has come up enough times for others here that even if that's the issue we can at least get it documented - "don't do this dumb thing" - to help others :)). Thanks again for your assistance in tracking this down.

[haikusw]

Ok. New Test. Used git tower to checkout the root repo and then added my fork as an additional remote and then checked out the async branch again and then tried the push of that branch to my remote and it failed the same way.

So there's something about the clone that Tower produces that is triggering this.

---

Just to confirm, I made a clone of the root repo on the command line, checked out the async branch, added my remote repo, and then tried the git lfs push of the async branch to my repo and the dry run succeeded.

I then compared the output of git config -l for the command line clone that works and the tower created one that doesn't.

The key line seems likely to be this one:

lfs.https://github.com/haikusw/isowords.git/info/lfs.access=basic

which is in the command line one but not in the Tower created one.

[haikusw]

Hmm. Now somehow I made a command line clone that doesn't have that line and it is failing. So that line in the git config -l seems to be the difference.

[haikusw]

Ah, more complicated than that.

Either the destination remote needs to not have the user name in the url:

https://haikusw@github.com/haikusw/isowords.git // fails unless… (see below)

vs

https://github.com/haikusw/isowords.git // always works

What makes this complicated to test back and forth in a single repo is that once you have successfully done a git lfs command in a repo it seems to add:

[lfs "https://github.com/haikusw/isowords.git/info/lfs"]
	access = basic

to .git/config for that repo.

Once git-lfs has done that, then the username in the remote url no longer causes authentication to fail.

---

So, the authentication issue is reproducible when there is a username in the destination remote url AND lfs.https access is not overridden to "basic" via the config file.

Thus, to reproduce this you can do the following:

1. on github.com:
   Fork the https://github.com/pointfreeco/isowords to your GitHub account so you have a fork with write access (let's say it's at https://github.com/yourname/isowords )

2. in terminal:

git clone https://github.com/pointfreeco/isowords.git
git checkout -b test42
git remote add my_origin https://yourname@github.com/yourname/isowords.git
git lfs push --dry-run my_origin test42

and that last command should fail with the authentication error.
(Though likely this will depend on something else relating to ssh keys, personal access tokens, etc, but we'll have to start by seeing if you can reproduce it with your current setup and if not we can figure out what is different between our setups).

...

3. To then make the git lfs push work, either:
   3a) add the following to your .git/config file manually:

   [lfs "https://github.com/yourname/isowords.git/info/lfs"]
   	access = basic`
   

   3b) or, change your remote to no longer include your username:
   git remote set-url my_origin https://github.com/yourusername/isowords.git

Note that if you do (3b) and then do the git lfs push command (or any other git lfs command?) it will add the entry from (3a) to your .git/config file. Once that's done you can switch the git remote url back to including your username and it will then work.

---

Anyway, that's how to reproduce the issue there. (let me know if you have any trouble reproducing it).

---

Now the question is: Why does having non-basic authentication with a remote with a user name in the url cause authentication to fail.

As a reminder of things that may be involved:

• I have a .ssh key added to my GitHub account.
• I have a Personal Access Token set for Tower and
  echo "host=github.com\nprotocol=https" | git credential fill seems to have it set

Not sure what else.

[chrisd8088]

Hey, thanks for all the investigation! I'll try to use your steps to reproduce it myself in the next few days, and with luck that will shed some light on the source of the problem. Thanks again!