git lfs migrate --no-rewrite couldn't see files in commit?

[bitinn]

Say I have following branches:

• master
• update (git checkout -b update from master)

And if I run following command on it:

$ git lfs migrate import --no-rewrite LightingData.asset

It gives me this warning:

fatal: could not rewrite "LightingData.asset": unable to find entry LightingData.asset in tree

But if I run following command:

$ git lfs migrate import --include-ref=update --include="LightingData.asset"

Then it passes just fine:

migrate: Sorting commits: ..., done                                             
migrate: Rewriting commits: 100% (4/4), done                                    
  convert	7b208bf9ddfc5c814e50b5c649fa99f41cef1b2f -> 14da0c07bd6a215b8fd71335579b492b7d7327ee
  master 	7b208bf9ddfc5c814e50b5c649fa99f41cef1b2f -> 14da0c07bd6a215b8fd71335579b492b7d7327ee

$ git lfs ls-files | grep asset

9af25f1c37 - leocities-starter-kit/Assets/Leocities/Scenes/TestSceneA/LightingData.asset
190a5dd172 - leocities-starter-kit/Assets/Leocities/Scenes/TestSceneB/LightingData.asset
108a9f53fd - leocities-starter-kit/Assets/Leocities/Scenes/TestSceneC/LightingData.asset

My .gitattributes contains these 2 lines:

LightingData.asset filter=lfs diff=lfs merge=lfs -text
...
*.asset merge=unityyamlmerge eol=lf

So my question, is this a bug? Is --no-rewrite supposed to do what I think it would do: that is, to migrate the files into LFS without rewriting history?

[bk2204]

Hey, sorry to hear you're having trouble with this.

--no-rewrite is designed to migrate your files in a single commit on top of the branches (or other refs) without rewriting history, yes. When you use it, you can specify branches (but not files) on the command line as arguments, or you can specify files and refs as arguments to options.

It isn't clear to me what you're expecting to see that's different, so it's hard for me to provide a more helpful response. It would be helpful to know what branch you're on when seeing this as well. A few thoughts that did come up for me, though, are the following:

• I think the syntax you want for --include-ref is a full ref name: refs/heads/update instead of just update.
• Note that if you're in a directory other than the root of the repository, git lfs ls-files doesn't always print the right thing (git-lfs ls-files showing different OID states in output depending upon position within working tree #3406).

[bitinn]

It isn't clear to me what you're expecting to see that's different

I am on branch update in this case, you are saying the following is expected?

$ git lfs migrate import --no-rewrite LightingData.asset

fatal: could not rewrite "LightingData.asset": unable to find entry LightingData.asset in tree

Please see the manual for --no-rewrite because that's what I am asking for: why --no-rewrite doesn't work when --include-ref does.

• https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-migrate.1.ronn#import-no-rewrite
• https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-migrate.1.ronn#migrate-without-rewriting-local-history

For a workaround I have switch to remove them with git rm instead, which is largely the same in terms of what I want.

$ git rm --cached LightingData.asset
$ git add .
(then re-commit to git lfs)

[bk2204]

I am on branch update in this case, you are saying the following is expected?

Thanks, this clarification is helpful.

$ git lfs migrate import --no-rewrite LightingData.asset

fatal: could not rewrite "LightingData.asset": unable to find entry LightingData.asset in tree

Yes, this is expected. The --no-rewrite option takes a list of paths, not a list of pathspecs or patterns. In other words, we look them up literally, and they have to already be tracked by .gitattributes.

What you might want to do is something like the following to import all the asset files:

$ git lfs migrate import --no-rewrite $(find . -name '*.asset' | cut -c3-)

[bitinn]

Thx for your reply, but if you are right, the manual is very misleading:

https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-migrate.1.ronn#migrate-without-rewriting-local-history

[bk2204]

We do specify that they're files a bit farther up, but I agree this can be improved:

• [file ...]
  The list of files to import. These files must be tracked by patterns
  specified in the gitattributes.

I'll try to improve the documentation a little bit.

[goyzhang]

I am on branch update in this case, you are saying the following is expected?

Thanks, this clarification is helpful.

$ git lfs migrate import --no-rewrite LightingData.asset

fatal: could not rewrite "LightingData.asset": unable to find entry LightingData.asset in tree

Yes, this is expected. The --no-rewrite option takes a list of paths, not a list of pathspecs or patterns. In other words, we look them up literally, and they have to already be tracked by .gitattributes.

What you might want to do is something like the following to import all the asset files:

$ git lfs migrate import --no-rewrite $(find . -name '*.asset' | cut -c3-)

Do we have plan to make it support patterns? This is ugly and different with the implementation without --no-rewrite.

[bk2204]

At the moment, we don't have any plans to make it support patterns. Such a change would be incompatible, so we'd only make it at a 3.0 release. For example, it would modify the behavior of arguments with brackets.

[goyzhang]

At the moment, we don't have any plans to make it support patterns. Such a change would be incompatible, so we'd only make it at a 3.0 release. For example, it would modify the behavior of arguments with brackets.

OK, so maybe we should update the doc: https://github.com/git-lfs/git-lfs/blob/master/docs/man/git-lfs-migrate.1.ronn#migrate-without-rewriting-local-history

Without a specified commit message:

$ git lfs migrate import --no-rewrite test.zip *.mp3 *.psd
With a specified commit message:

$ git lfs migrate import --no-rewrite
-m "Import test.zip, .mp3, .psd files in root of repo"
test.zip *.mp3 *.psd

is a bad example.

[phobos2077]

The documentation is still lying about the format of this function.

Also I still don't understand how --no-rewrite mode is useful at all considering that git rm --cached *.png does exactly what is needed for a non-destructive migration to LFS (allows to add previously committed binary files to LFS), but it also supports file patterns.

[bk2204]

The patterns in the documentation case are expanded by the shell, not by Git LFS, since they are not quoted. That shouldn't be surprising to folks who are used to a standard POSIX shell.

It is true that there are multiple ways to do this, including git add --renormalize . && git commit -m message. You're welcome to use one of those ways if you like them better.

[seriema]

The patterns in the documentation case are expanded by the shell, not by Git LFS, since they are not quoted. That shouldn't be surprising to folks who are used to a standard POSIX shell.

That's super hard to pick up on. You're totally right but it's a very high expectation on users to catch that gotcha that there were no quotes in the example. Maybe remove --no-rewrite and turn git add --renormalize into the expected way, since that's used for .gitattributes changes overall. Or make the example much clearer by only using full filenames with no wildcards. If people want to use expansions in a POSIX shell then they'll figure out ways to pass lists to the command.

[vikhik]

Echoing that the documentation is awfully misleading - and I would argue that "That shouldn't be surprising to folks who are used to a standard POSIX shell." is an awful argument for defending technical writing (with a demonstration of multiple confused users).

It would be better to just remove the --no-rewrite option and update the docs to specify that git add --renormalize . is the preferred method to do a non-destructive migration.