Skip to content

feat(datastore): add SQLite database backend support #1529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

feat(datastore): add SQLite database backend support #1529

wants to merge 11 commits into from

Conversation

@Chanakya-TS
Copy link
Contributor

@Chanakya-TS Chanakya-TS commented Aug 27, 2025 •
edited
Loading

feat(datastore): add SQLite database backend support

Description

This PR adds SQLite as a third database backend option for the Model Registry project, alongside the existing MySQL and PostgreSQL support. This implementation provides a lightweight, file-based database solution ideal for development environments, testing, and
single-instance deployments.

Key Features:

  • Complete SQLite connector with connection management and retry logic
  • Full migrator implementation using golang-migrate with sqlite3 driver
  • 21 database migrations converted from MySQL to SQLite syntax
  • Comprehensive test coverage for connections, migrations, and SQLite-specific features
  • Build system integration with Makefile targets (start/sqlite, stop/sqlite, gen/gorm/sqlite)
  • CI/CD integration with schema validation workflow
  • Management scripts for database setup and teardown

Technical Implementation:

  • Uses GORM SQLite driver (gorm.io/driver/sqlite v1.6.0)
  • Converts MySQL AUTO_INCREMENT to SQLite AUTOINCREMENT/INTEGER PRIMARY KEY syntax
  • Maps MySQL MEDIUMTEXT/LONGBLOB to SQLite TEXT/BLOB types
  • Supports boolean storage as INTEGER (0/1) following SQLite conventions
  • Maintains referential integrity and index coverage across all tables
  • Follows the same architectural patterns as existing MySQL/PostgreSQL implementations

Key Technical Fixes:

  • Schema Consistency: Fixed GORM generation to ensure consistent Go struct types across all database backends
  • Boolean Handling: Corrected SQLite boolean storage and retrieval compatibility with GORM
  • Primary Keys: Used INTEGER PRIMARY KEY instead of AUTOINCREMENT for better SQLite performance
  • Migration Scripts: Enhanced cleanup scripts to handle SQLite-specific schema generation

How Has This Been Tested?

Testing Environment:

  • Windows 10 with Go 1.24.4
  • WSL2 Ubuntu environment for cross-platform validation

Tests Performed:

  • ✅ Unit Tests: All SQLite-specific tests pass (go test ./internal/datastore/embedmd/sqlite/...)
  • ✅ Connection Tests: In-memory (:memory:) and file-based database connections
  • ✅ Migration Tests: Up/down migrations, step migrations, schema validation
  • ✅ CRUD Operations: Create, read, update, delete operations with various data types
  • ✅ Transaction Tests: Commit/rollback functionality
  • ✅ Concurrency Tests: Multiple database connections and concurrent access
  • ✅ SQLite-Specific Features: AUTOINCREMENT, boolean handling, TEXT/BLOB types
  • ✅ Build Integration: Successfully builds with make build and runs with SQLite backend
  • ✅ API Validation: Model Registry server starts and responds to API requests with SQLite database
  • ✅ GORM Generation: Code generation produces consistent struct types across all backends

@google-oss-prow
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign al-pragliola for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Chanakya-TS Chanakya-TS marked this pull request as ready for review August 27, 2025 00:18
@google-oss-prow google-oss-prow bot requested a review from andreyvelich August 27, 2025 00:18
@Chanakya-TS Chanakya-TS marked this pull request as draft August 27, 2025 00:27
@Chanakya-TS
feat(datastore): add SQLite database backend support
96562e2 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: add execute permissions to SQLite scripts
f8d4793 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: use temporary database for SQLite GORM generation in CI
7075fdc 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: remove SQLite schema validation from CI workflow
0854186 
SQLite generates different Go types (int32 vs int64, int32 vs bool)
compared to MySQL/PostgreSQL due to SQLite's type system. Since
the schema files are shared between database backends, SQLite
schema validation would always fail with type differences.

Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: update SQLite migrations to generate same Go types as MySQL/Post…
b71e30b 
…greSQL

Changes made to ensure GORM generates identical struct types across all database backends:
- Changed INTEGER to BIGINT for all timestamp fields (create_time_since_epoch, last_update_time_since_epoch, milliseconds_since_epoch)
- Changed INTEGER to BOOLEAN for all boolean fields (is_custom_property, bool_value, is_index_step)
- Restored SQLite schema validation in CI workflow

This ensures SQLite generates the same Go types:
- BIGINT maps to int64 (matching MySQL bigint)
- BOOLEAN maps to bool (matching MySQL tinyint(1))
- Maintains schema consistency across all database backends

Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: use INTEGER PRIMARY KEY instead of AUTOINCREMENT in SQLite migra…
bd5fc8e 
…tions

SQLite's INTEGER PRIMARY KEY (without AUTOINCREMENT) is the proper equivalent
to MySQL's AUTO_INCREMENT that generates the same GORM struct tags:
- INTEGER PRIMARY KEY maps to int32 with autoIncrement:true
- AUTOINCREMENT creates different GORM behavior (*int32 without autoIncrement tag)

This should now generate identical Go struct types as MySQL/PostgreSQL.

Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
feat: add SQLite-specific handling to GORM generator
b1f7172 
Adds custom dataTypeMap and field tag processing for SQLite to ensure
generated Go structs match MySQL/PostgreSQL format:

- INTEGER PRIMARY KEY fields generate int32 with autoIncrement:true tag
- Custom field processing adds autoIncrement tag for primary key fields
- Maintains type consistency across all database backends

This ensures SQLite schema validation passes in CI by generating
identical Go struct types as other database backends.

Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: update GORM cleanup script for SQLite schema consistency
8469553 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
feat: improve GORM generator consistency across database types
b941658 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: correct SQLite boolean handling test for GORM compatibility
f4e129c 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS
fix: check error return value in SQLite migrate test cleanup
39eb3c0 
Signed-off-by: Chanakya TS <[email protected]>
@Chanakya-TS Chanakya-TS marked this pull request as ready for review August 29, 2025 13:33
@Chanakya-TS
Copy link
Contributor Author

Hi, I would really appreciate it if somebody can review this PR. Thank you.

@Al-Pragliola
Copy link
Contributor

Hey @Chanakya-TS ! Sorry for the delay, adding support to a new db should be done with care as we still found issues with postgresql tests as of last week 😢

I would consider adding both e2e tests and (for now at least) try running fuzz tests locally before reviewing the PR

@Al-Pragliola Al-Pragliola mentioned this pull request Sep 22, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Al-Pragliola Al-Pragliola Awaiting requested review from Al-Pragliola

@pboyd pboyd Awaiting requested review from pboyd

@andreyvelich andreyvelich Awaiting requested review from andreyvelich

Assignees

No one assigned

Labels

Area/GitHub Area/Go REST server size/XXL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Chanakya-TS @Al-Pragliola