[LFX Term 3]: Full storage path for KCL third-party dependencies

[zong-zhe]

1. Problem

• Dependencies from different registries may have the same name, and the current way kpm uses file names can cause conflicts that rely on duplicate names.
• The local saving path of the three-party library is case-sensitive, this has different effects on windows and unix systems.

2. Solution

In order to solve the problem, we redesigned the storage local system of the third-party dependencies with reference to rust

└── kpm
    ├── .kpm # all the configuration file for kpm client
    ├── git # all the kcl dependencies from git repo
    ├── oci # all the kcl dependencies from oci registry

2.1 Git

Under the subdir in kpm/git, the storage local system is

kpm/git
├── checkouts.   # checkout the specific version of git repository from cache bare repository 
│   ├── kcl-2a81898195a215f1
│   │   └── 33bb450. . # All the version of kcl package from git repository will be replaced with commit id
│   ├── kcl-578669463c900b87
│   │   └── 33bb450
└── db    # A bare git repository for cache git repo
    ├── kcl-2a81898195a215f1.      # <NAME>-<HASH> <NAME> is the name of git repo, 
    ├── kcl-578669463c900b87.   # <HASH> is calculated by the git full url.

NOTE:

• <NAME>-<HASH> <NAME> is the name of git repo, <HASH> is calculated by the git url. <HASH> is mainly used to distinguish between different git repositories, such as github and gitlab, and the reason for using a hash value is mainly to mask the inconsistency of the symbols in the url and local path.

2.2 Oci

Under the subdir in kpm/oci, the storage local system is

kpm/oci
├── cache # the cache for KCL dependencies tar
│   ├── ghcr.io-2a81898195a215f1    # <HOST>-<HASH> HOST is the name of oci registry,  <HASH> is calculated by the oci full url.
│   │   └── k8s_1.29.tar    # the tar for KCL dependencies
│   ├── docker.io-578669463c900b87
│   │   └── k8s_1.28.tar
└── src                                              	
│   ├── ghcr.io-2a81898195a215f1
│   │   └── k8s_1.29    # the KCL dependencies tar will untar here
│   ├── docker.io-578669463c900b87
│   │   └── k8s_1.28

4. Implementation

• The cache module for oci: feat: add OciCache to support the new local file structure for OCI #502

[zong-zhe]

PreTest

• Development of kpm current git module to expand its ability to download git bare repo.
• Development of kpm current git module to expand its ability to checkout a source code repo from a git bare repo.
• A unified dependency system is used to mask the underlying details, the file system only needs to provide the Get method externally. The unified file system automatically loads the cache and downloads.

[Manoramsharma]

HI @zong-zhe @Peefy I am excited to contribute to this issue. After going through the solution mentioned I have see following key take-aways:

1. By generating unique hash values based on repository URLs, we can distinguish dependencies with identical names. This method not only avoids conflicts but also provides a consistent way to reference dependencies.
2. Developing a system that abstracts the underlying details of fetching and caching dependencies. This system would expose a Get method, ensuring that the file system automatically manages caching and downloads. This would also standardize behavior across different operating systems.

In order to proceed forward with the pretest where I need to expand kpm's ability to download , chekout from a git bare repo. Which part of documentation I can refer for configuring this for kpm current git module?

[zong-zhe]

Hi @Manoramsharma 😃

Welcome! Let me provide more details.

kpm currently uses a three-party library go-getter to complete git related work, go-getter will call your local git tool to complete the function, which means that you need to install git tools in your local, and refer to the git tool documentation to complete the configuration. With kpm, you don't need to do any additional configuration, and if your git clone runs successfully, kpm can add dependency with the git repo.

kpm and git

• https://www.kcl-lang.io/docs/user_docs/guides/package-management/how-to/private_mod_repo
• https://www.kcl-lang.io/docs/tools/cli/package-management/command-reference/add#add-a-dependency-from-git-repository

kpm and go-getter

• https://github.com/hashicorp/go-getter
• feat: Support git private repo #275

[Manoramsharma]

Okay, so kpm initially supported only the public git repo, now after the implementation of go-getter kpm supports cloning options for performing different git related operations. Right?

Now for our case, development of kpm current git module to expand its ability to download git bare repo do I need to modify the current git implementation by adding specific functions to the go-getter or the related file that will ultimately supports the checkout of specific version of git repository from cache bare repository? For the checkouts we will mention here:

kpm/git
├── checkouts.   # checkout the specific version of git repository from cache bare repository 
│   ├── kcl-2a81898195a215f1
│   │   └── 33bb450. . # All the version of kcl package from git repository will be replaced with commit id
│   ├── kcl-578669463c900b87
│   │   └── 33bb450
└── db    # A bare git repository for cache git repo
    ├── kcl-2a81898195a215f1.      # <NAME>-<HASH> <NAME> is the name of git repo, 
    ├── kcl-578669463c900b87.   # <HASH> is calculated by the git full url.

@zong-zhe am I thinking in a right direction?

[Manoramsharma]

Proposal to Enhance kpm's current git module

To enhance the kpm current git module, I propose the following solutions to expand its capabilities:

1. Expand Ability to Download Git Bare Repositories

1.1 Define a Function for Cloning Bare Repositories

I will add a function to clone Git repositories as bare repositories. This function will use Git commands to perform the cloning and store the bare repository in the appropriate cache directory.

Example Implementation:

package git

import (
	"fmt"
	"os/exec"
	"path/filepath"
)

// CloneBareRepo clones a Git repository as a bare repository.
func CloneBareRepo(repoURL string, cacheDir string) (string, error) {
	repoHash := calculateHash(repoURL) // Function to calculate a unique hash for the repo URL
	repoName := extractRepoName(repoURL) // Function to extract repository name from URL
	cachePath := filepath.Join(cacheDir, fmt.Sprintf("%s-%s", repoName, repoHash))

	if _, err := exec.LookPath("git"); err != nil {
		return "", fmt.Errorf("git not found in PATH")
	}

	if err := exec.Command("git", "clone", "--bare", repoURL, cachePath).Run(); err != nil {
		return "", fmt.Errorf("error cloning repository: %v", err)
	}

	return cachePath, nil
}

// Helper functions
func calculateHash(url string) string {
	// Implement hashing function for URL
}

func extractRepoName(url string) string {
	// Implement repo name extraction
}

1.2 Integrate with the Unified Dependency System

I will ensure that this new function integrates with the unified dependency system. This may involve updating the system to manage and utilize the cache effectively.

package unified

// UnifiedDependencySystem is an abstraction for managing dependencies.
type UnifiedDependencySystem struct {
	cacheDir string
}

// Get handles fetching the Git repository and managing the cache.
func (uds *UnifiedDependencySystem) Get(repoURL string, commitID string, targetDir string) error {
	bareRepoPath, err := git.CloneBareRepo(repoURL, uds.cacheDir)
	if err != nil {
		return err
	}
	return git.CheckoutFromBareRepo(bareRepoPath, commitID, targetDir)
}

2. Expand Ability to Checkout Source Code from Bare Repositories

2.1 Define a Function for Checking Out Code from Bare Repositories

I will add a function to check out a specific commit or branch from a bare repository.

Example Implementation:

package git

import (
	"fmt"
	"os"
	"os/exec"
	"path/filepath"
)

// CheckoutFromBareRepo checks out a specific commit from a bare repository.
func CheckoutFromBareRepo(bareRepoPath string, commitID string, targetDir string) error {
	if err := os.MkdirAll(targetDir, 0755); err != nil {
		return fmt.Errorf("error creating target directory: %v", err)
	}

	cmd := exec.Command("git", "worktree", "add", targetDir, commitID)
	cmd.Dir = bareRepoPath
	if err := cmd.Run(); err != nil {
		return fmt.Errorf("error checking out commit: %v", err)
	}

	return nil
}

2.2 Update the Unified Dependency System

I will ensure that the unified dependency system correctly utilizes the new checkout functionality and integrates with the local cache.

package unified

// UnifiedDependencySystem is an abstraction for managing dependencies.
type UnifiedDependencySystem struct {
	cacheDir string
}

// Get handles fetching and checking out the Git repository.
func (uds *UnifiedDependencySystem) Get(repoURL string, commitID string, targetDir string) error {
	bareRepoPath, err := git.CloneBareRepo(repoURL, uds.cacheDir)
	if err != nil {
		return err
	}
	return git.CheckoutFromBareRepo(bareRepoPath, commitID, targetDir)
}

3. Handle Cache and Metadata Management

3.1 Implement Cache Management

I will add functionality to manage and update cached repositories, including metadata.

package cache

// CacheManager handles operations related to caching dependencies.
type CacheManager struct {
	cacheDir string
}

// UpdateCache updates the local cache with new metadata and repositories.
func (cm *CacheManager) UpdateCache(repoURL string) error {
	// Implement caching logic here
	return nil
}

3.2 Ensure Consistency

I will ensure that the local storage system (e.g., kpm/git/checkouts/, kpm/git/db/) remains consistent with the new functionalities and can handle different scenarios.

cc: @Peefy @zong-zhe

[zong-zhe]

Okay, so kpm initially supported only the public git repo, now after the implementation of go-getter kpm supports cloning options for performing different git related operations. Right?

Now for our case, development of kpm current git module to expand its ability to download git bare repo do I need to modify the current git implementation by adding specific functions to the go-getter or the related file that will ultimately supports the checkout of specific version of git repository from cache bare repository? For the checkouts we will mention here:

kpm/git
├── checkouts.   # checkout the specific version of git repository from cache bare repository 
│   ├── kcl-2a81898195a215f1
│   │   └── 33bb450. . # All the version of kcl package from git repository will be replaced with commit id
│   ├── kcl-578669463c900b87
│   │   └── 33bb450
└── db    # A bare git repository for cache git repo
    ├── kcl-2a81898195a215f1.      # <NAME>-<HASH> <NAME> is the name of git repo, 
    ├── kcl-578669463c900b87.   # <HASH> is calculated by the git full url.

@zong-zhe am I thinking in a right direction?

Hi @Manoramsharma 😃

You might need to learn if go-getter offers the ability to download bare repo, but if not, you can extend this file https://github.com/kcl-lang/kpm/blob/main/pkg/git/git.go to add kpm's basic ability to download a git bare repo and checkout the bare repo to src code repo. You can try to design your implementation with CloneOptions.

[zong-zhe]

2.2 Update the Unified Dependency System

I will ensure that the unified dependency system correctly utilizes the new checkout functionality and integrates with the local cache.

package unified

// UnifiedDependencySystem is an abstraction for managing dependencies.
type UnifiedDependencySystem struct {
	cacheDir string
}

// Get handles fetching and checking out the Git repository.
func (uds *UnifiedDependencySystem) Get(repoURL string, commitID string, targetDir string) error {
	bareRepoPath, err := git.CloneBareRepo(repoURL, uds.cacheDir)
	if err != nil {
		return err
	}
	return git.CheckoutFromBareRepo(bareRepoPath, commitID, targetDir)
}

You may need to add some detail here, because KCL's dependencies are not only those from git, you also need to consider the dependencies from oci together, abstracting them from the same behavior.

You also need to consider combining the downloader to support the download behavior of different dependencies.

https://github.com/kcl-lang/kpm/blob/main/pkg/downloader/downloader.go

[zong-zhe]

3. Handle Cache and Metadata Management

3.1 Implement Cache Management

I will add functionality to manage and update cached repositories, including metadata.

package cache

// CacheManager handles operations related to caching dependencies.
type CacheManager struct {
	cacheDir string
}

// UpdateCache updates the local cache with new metadata and repositories.
func (cm *CacheManager) UpdateCache(repoURL string) error {
	// Implement caching logic here
	return nil
}

This part I think may not need to be a separate module, it can be used as part of the unified file system.