Skip to content

Conversation

thxCode
Copy link

@thxCode thxCode commented Aug 5, 2024

With the development of AI, more and more Git repositories use LFS to store model weights. The ADD command already supports pulling from Git URLs, https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#adding-files-from-a-url. This PR intends to support the ADD command to pull content managed by Git LFS.

Signed-off-by: thxCode <[email protected]>
Copy link
Member

@crazy-max crazy-max left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Left some comments, I don't have much knowledge with LFS but happy to see some support here.

Would it be possible to add tests?

if err != nil {
return nil, err
}
_, err = checkoutGit.Run(ctx, "lfs", "install", "--local")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should check if lfs is supported (git >= 1.8.3.1): https://github.com/git-lfs/git-lfs/wiki/Installation#requirements

if err != nil {
return nil, err
}
_, err = checkoutGit.Run(ctx, "lfs", "install", "--local")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we check if there are files being tracked by LFS first?

git lfs ls-files

Or maybe look for .gitattributes with filter=lfs like:

*.png filter=lfs diff=lfs merge=lfs -text

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe git lfs status?

@crazy-max
Copy link
Member

https://github.com/moby/buildkit/actions/runs/10241968157/job/28396722664?pr=5212#step:8:5671

=== FAIL: frontend/dockerfile TestIntegration/TestGitProvenanceAttestation/worker=containerd/frontend=builtin (0.86s)
    dockerfile_provenance_test.go:305: 
        	Error Trace:	/src/frontend/dockerfile/dockerfile_provenance_test.go:305
        	            				/src/util/testutil/integration/run.go:97
        	            				/src/util/testutil/integration/run.go:211
        	Error:      	Received unexpected error:
        	            	failed to read dockerfile: failed to install lfs for repo at /tmp/bktest_containerd3672960127/root/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs: git stderr:
        	            	git: 'lfs' is not a git command. See 'git --help'.

@thxCode
Copy link
Author

thxCode commented Aug 6, 2024

https://github.com/moby/buildkit/actions/runs/10241968157/job/28396722664?pr=5212#step:8:5671

I will try to address your comments, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants