Dependency Extractor

A comprehensive Node.js tool for extracting, analyzing, and downloading npm package dependencies and metadata.

Features

Package Fetcher: Extract package names from npm search URLs
Dependency Extractor: Analyze dependencies and devDependencies of npm packages
Module Downloader: Download npm packages and their tarballs for offline use
Interactive Mode: User-friendly command-line interface
Batch Processing: Process multiple packages from input files
Flexible Output: Choose between detailed or clean output formats

Installation

Prerequisites

Node.js (v14 or higher)
npm or yarn

Install Dependencies

npm install

Global Installation (Optional)

npm install -g .

Usage

1. Package Fetcher

Extract package names from npm search URLs.

Interactive Mode

npm run fetch
# or
node src/fetch-modules.js

Command Line Mode

npm run fetch -- <search-url> [output-file] [max-packages]
# or
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:front-end" packages.txt 500

Examples:

# Extract 1000 front-end packages
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:front-end&page=0&ranking=optimal"

# Extract 500 packages with custom output file
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 500

2. Dependency Extractor

Extract dependencies from npm packages.

Interactive Mode

npm run extract
# or
npm start
# or
node src/index.js
# or
node index.js

File Mode

# Full output with package information
npm run extract -- input.txt output.txt

# Clean output (dependencies only)
npm run extract -- input.txt output.txt clean

Examples:

# Extract dependencies from a list of packages
node src/index.js packages.txt dependencies.txt

# Get clean dependency list only
node src/index.js packages.txt clean-deps.txt clean

3. Module Downloader

Download npm packages and their tarballs for offline use.

npm run download
# or
node module-downloader.js

Configuration:

Creates npm_packages/ directory in project root
Downloads up to 200 versions per package (configurable)
Skips already downloaded packages
Generates package.json with metadata

📋 Complete Command Reference

Package Fetcher Commands

Interactive Mode

# Start interactive package fetcher
npm run fetch
# or
node src/fetch-modules.js

Command Line Mode

# Basic fetch with default output file
npm run fetch -- "https://www.npmjs.com/search?q=keywords:front-end"

# Fetch with custom output file
npm run fetch -- "https://www.npmjs.com/search?q=keywords:react" react-packages.txt

# Fetch with custom output file and max packages limit
npm run fetch -- "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 500

# Direct node commands
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:front-end"
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 500

Dependency Extractor Commands

Interactive Mode

# Start interactive dependency extractor
npm run extract
# or
npm start
# or
node src/index.js
# or
node index.js

File Mode (Full Output)

# Process packages from file with full information
npm run extract -- input.txt output.txt
# or
node src/index.js packages.txt dependencies.txt
# or
node index.js packages.txt dependencies.txt

File Mode (Clean Output - Dependencies Only)

# Process packages from file with clean output (dependencies only)
npm run extract -- input.txt output.txt clean
# or
node src/index.js packages.txt clean-deps.txt clean
# or
node index.js packages.txt clean-deps.txt clean

Module Downloader Commands

Download All Packages

# Download all packages from output-no-dup.txt
npm run download
# or
node module-downloader.js

Development Commands

Testing

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

Code Quality

# Run linting
npm run lint

# Fix linting issues automatically
npm run lint:fix

Documentation

# Generate documentation
npm run docs

Maintenance

# Clean downloaded packages directory
npm run clean

# Build project (lint + test)
npm run build

🌐 Complete Workflow Examples

Extract React Ecosystem Dependencies

# 1. Fetch React-related packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 100

# 2. Extract their dependencies (full output)
npm run extract -- react-packages.txt react-dependencies.txt

# 3. Extract dependencies only (clean output)
npm run extract -- react-packages.txt react-deps-clean.txt clean

# 4. Download the packages (optional)
npm run download

Analyze Frontend Packages

# 1. Fetch frontend packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:front-end" frontend-packages.txt 500

# 2. Extract dependencies
npm run extract -- frontend-packages.txt frontend-dependencies.txt clean

# 3. Download packages
npm run download

Single Package Analysis

# Interactive analysis of a single package
npm run extract
# Then enter: react

# Or analyze specific package dependencies
npm run extract -- packages.txt output.txt

Vue.js Ecosystem Analysis

# 1. Fetch Vue.js related packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:vue" vue-packages.txt 200

# 2. Extract dependencies with full information
npm run extract -- vue-packages.txt vue-dependencies-full.txt

# 3. Extract clean dependency list
npm run extract -- vue-packages.txt vue-dependencies-clean.txt clean

Node.js Backend Packages

# 1. Fetch backend packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:backend" backend-packages.txt 300

# 2. Extract dependencies
npm run extract -- backend-packages.txt backend-dependencies.txt clean

# 3. Download for offline use
npm run download

TypeScript Packages

# 1. Fetch TypeScript packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:typescript" typescript-packages.txt 150

# 2. Extract dependencies
npm run extract -- typescript-packages.txt typescript-dependencies.txt clean

📁 File Structure Examples

Input Files

# packages.txt example:
react
vue
angular
express
lodash
axios
moment

Output Files

Full Output (dependencies.txt):

Dependency Extractor Results
Generated on: 2024-01-15T10:30:00.000Z
=====================================

Package: react
Latest version: 18.2.0

Dependencies:
loose-envify
object-assign

Dev Dependencies:
@babel/core
@babel/preset-env
@babel/preset-react
=====================================

Package: vue
Latest version: 3.3.4

Dependencies:
@vue/shared
@vue/runtime-core
@vue/runtime-dom

Dev Dependencies:
@vue/compiler-sfc
@vue/test-utils
=====================================

Clean Output (clean-deps.txt):

loose-envify
object-assign
@babel/core
@babel/preset-env
@babel/preset-react
@vue/shared
@vue/runtime-core
@vue/runtime-dom
@vue/compiler-sfc
@vue/test-utils

🔧 Configuration Examples

Environment Variables

# Set custom download directory
export DOWNLOAD_DIR="/path/to/custom/directory"

# Set maximum versions per package
export MAX_VERSIONS=100

# Set custom npm registry
export NPM_REGISTRY="https://registry.npmjs.org"

# Then run commands
npm run download

Global Installation Usage

# Install globally for CLI access
npm install -g .

# Then use as CLI commands
dependency-extractor
fetch-modules
module-downloader

Project Structure

dependency-extractor/
├── src/
│   ├── fetch-modules.js      # Package fetcher
│   ├── index.js             # Dependency extractor
│   ├── module-downloader.js # Module downloader
│   └── utils/
│       ├── logger.js        # Logging utilities
│       └── file-utils.js    # File operations
├── config/
│   └── settings.js          # Configuration settings
├── examples/
│   ├── sample-packages.txt  # Sample package list
│   └── sample-output.txt    # Sample output
├── docs/
│   └── API.md              # API documentation
├── tests/
│   └── test-*.js           # Test files
├── package.json
├── README.md
└── .gitignore

Configuration

Environment Variables

DOWNLOAD_DIR: Custom download directory (default: ./npm_packages)
MAX_VERSIONS: Maximum versions to download per package (default: 200)
NPM_REGISTRY: Custom npm registry URL (default: https://registry.npmjs.org)

Settings File

Edit config/settings.js to customize behavior:

module.exports = {
  downloadDir: process.env.DOWNLOAD_DIR || "./npm_packages",
  maxVersions: parseInt(process.env.MAX_VERSIONS) || 200,
  npmRegistry: process.env.NPM_REGISTRY || "https://registry.npmjs.org",
  requestTimeout: 30000,
  retryAttempts: 3,
  retryDelay: 1000,
};

API Reference

Package Fetcher

const { fetchPackagesFromSearch } = require("./src/fetch-modules");

// Fetch packages from search URL
const packages = await fetchPackagesFromSearch(
  "https://www.npmjs.com/search?q=keywords:react",
  "output.txt",
  1000
);

Dependency Extractor

const { getDependencies, processPackages } = require("./src/index");

// Get dependencies for a single package
const deps = await getDependencies("react", true); // clean output

// Process multiple packages
await processPackages("input.txt", "output.txt", false); // full output

Module Downloader

const { processPackage } = require("./src/module-downloader");

// Download a single package
await processPackage("react");

Examples

Extract React Ecosystem Dependencies

# 1. Fetch React-related packages
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 100

# 2. Extract their dependencies
node src/index.js react-packages.txt react-dependencies.txt clean

# 3. Download the packages (optional)
node src/module-downloader.js

Analyze Package Dependencies

# Interactive mode
npm run extract
# Enter: react
# Output: Dependencies and devDependencies for React

Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Testing

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

License

This project is licensed under the MIT License - see the LICENSE file for details.

Changelog

v1.0.0

Initial release
Package fetcher functionality
Dependency extractor
Module downloader
Interactive and batch processing modes

Support

Issues: GitHub Issues
Documentation: API Documentation
Examples: Examples Directory

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
config		config
src		src
.env.example		.env.example
.gitignore		.gitignore
.prettierrc		.prettierrc
README.md		README.md
index.js		index.js
module-downloader.js		module-downloader.js
package-lock.json		package-lock.json
package.json		package.json

tim-syntax/dependency-extractor

Folders and files

Latest commit

History

Repository files navigation

Dependency Extractor

Features

Installation

Prerequisites

Install Dependencies

Global Installation (Optional)

Usage

1. Package Fetcher

Interactive Mode

Command Line Mode

2. Dependency Extractor

Interactive Mode

File Mode

3. Module Downloader

📋 Complete Command Reference

Package Fetcher Commands

Interactive Mode

Command Line Mode

Dependency Extractor Commands

Interactive Mode

File Mode (Full Output)

File Mode (Clean Output - Dependencies Only)

Module Downloader Commands

Download All Packages

Development Commands

Testing

Code Quality

Documentation

Maintenance

🌐 Complete Workflow Examples

Extract React Ecosystem Dependencies

Analyze Frontend Packages

Single Package Analysis

Vue.js Ecosystem Analysis

Node.js Backend Packages

TypeScript Packages

📁 File Structure Examples

Input Files

Output Files

Full Output (dependencies.txt):

Clean Output (clean-deps.txt):

🔧 Configuration Examples

Environment Variables

Global Installation Usage

Project Structure

Configuration

Environment Variables

Settings File

API Reference

Package Fetcher

Dependency Extractor

Module Downloader

Examples

Extract React Ecosystem Dependencies

Analyze Package Dependencies

Contributing

Testing

License

Changelog

v1.0.0

Support

Roadmap

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages