diff --git a/docs.json b/docs.json index 73f2943d..b83eed29 100644 --- a/docs.json +++ b/docs.json @@ -169,6 +169,7 @@ "usage/lifecycle-maintenance/handling-update-conflicts/custom-conflict-resolution" ] }, + "usage/lifecycle-maintenance/client-database-diagnostics", "usage/lifecycle-maintenance/handling-write-validation-errors", "usage/lifecycle-maintenance/upgrading-the-client-sdk", "usage/lifecycle-maintenance/postgres-maintenance", diff --git a/usage/lifecycle-maintenance/client-database-diagnostics.mdx b/usage/lifecycle-maintenance/client-database-diagnostics.mdx new file mode 100644 index 00000000..8ede93c8 --- /dev/null +++ b/usage/lifecycle-maintenance/client-database-diagnostics.mdx @@ -0,0 +1,165 @@ +--- +title: "Understanding The SQLite Database" +description: "Guide for analyzing and understanding the local SQLite database" +--- + +## Get the SQLite file + + +A SQLite database file can use any extension - .db, .sqlite, .sqlite3, etc. The extension doesn’t affect functionality; all contain the same SQLite format. We also recommend pulling the associated [Write-Ahead Log (WAL)](https://www.sqlite.org/wal.html) file (with the same name as the database plus the suffix `-wal`) to ensure no recent changes are lost. + + + + + Ensure your emulator is running, then replace `com.package-name` with your application's package name and `your-db-name.sqlite` with your database file name. + +```shell +adb exec-out run-as com.package-name cat databases/your-db-name.sqlite > "your/local/path/your-db-name.sqlite" +adb exec-out run-as com.package-name cat databases/your-db-name.sqlite-wal > "your/local/path/your-db-name.sqlite-wal" +``` + + **Common database locations:** + - [React Native Quick SQLite](/client-sdk-references/react-native-and-expo#react-native-quick-sqlite-2): `/data/data/com.package-name/files/` + - [OP-SQLite](/client-sdk-references/react-native-and-expo#op-sqlite): `/data/data/com.package-name/databases/` + + **Note:** If the database is in a different location, first find it with: +```shell +adb shell run-as com.package-name find /data/data/com.package-name -name "your-db-name.sqlite" +``` + + + Replace `your-db-name.sqlite` with your database file name and extension. +```shell +find ~/Library/Developer/CoreSimulator/Devices -type f -name 'your-db-name.sqlite' +find ~/Library/Developer/CoreSimulator/Devices -type f -name 'your-db-name.sqlite-wal' +``` + + **Common database location:** + - App sandbox: `Library/Application Support/` + + + + +Write-Ahead Log (WAL) file is not used in web environments. Browser-based SQLite implementations handle transactions differently. + + + Web applications use browser-based storage APIs. Database files are managed by the browser and not directly accessible via filesystem paths. + + **Storage options:** + - **OPFS (Origin Private File System)**: Native filesystem API with better performance (Chrome 102+, Firefox 111+, Safari 17.2+) + - **IndexedDB**: A key-value storage API. Unlike OPFS, IndexedDB doesn't store complete database files - it stores data in a structured format that cannot be directly downloaded as a SQLite file. + + +Run the JavaScript code in your browser's console (F12 → Console) while on your application's page. + + + **Export database to your computer (OPFS only):** +```javascript +// For OPFS +async function downloadDatabase() { +const root = await navigator.storage.getDirectory(); +const fileHandle = await root.getFileHandle('your-db-name.sqlite'); +const file = await fileHandle.getFile(); + +// Download the file +const url = URL.createObjectURL(file); +const a = document.createElement('a'); +a.href = url; +a.download = 'your-db-name.sqlite'; +a.click(); +URL.revokeObjectURL(url); +} + +downloadDatabase(); +``` + + **Browser DevTools (inspect only):** + - Chrome/Edge: `F12` → Application → Storage → IndexedDB or OPFS + - Firefox: `F12` → Storage → IndexedDB + - Safari: Develop → Show Web Inspector → Storage + + + +## Inspecting the SQLite file + +### 1. Open your SQLite file + +Use the `sqlite3` command-line tool or a GUI tool like [DB Browser for SQLite](https://sqlitebrowser.org/) to open your database file: +```shell +sqlite3 your-db-name.sqlite +``` + +### 2. Merge the WAL file + +Temporary changes are stored in a separate [Write-Ahead Log (WAL)](https://www.sqlite.org/wal.html) `.wal` file. To measure the database size accurately, merge these changes into the main database: +```sql +PRAGMA wal_checkpoint(TRUNCATE); +``` + +### 3. Get storage statistics + +Query the built-in `dbstat` virtual table to see how much space each table uses on disk: +```sql +SELECT name, pgsize AS storage_size, payload AS data_size +FROM dbstat +WHERE aggregate = true; +``` + +This returns: +- `name`: Table name +- `storage_size`: Total storage used on disk (in bytes, including SQLite overhead) +- `payload`: Actual data size (in bytes) + + +The `dbstat` table is automatically available in SQLite and provides low-level information about physical storage. Values represent on-disk usage including SQLite's internal structures (page headers, B-trees, indexes, free space), which is why they're larger than your logical data size. + + +## Understanding the size breakdown + +PowerSync databases contain more data than just your application tables to support the sync functionality: + +1. **Application data**: Your synced data in `ps_data__` tables +2. **Operation log (`ps_oplog`)**: A complete copy of all synced data required for offline conflict resolution and sync +3. **Indexes**: For efficient queries and lookups +4. **PowerSync metadata**: System tables and views for managing sync state (see [Client Architecture](https://docs.powersync.com/architecture/client-architecture#schema)) +5. **SQLite overhead**: Page structure, alignment, fragmentation, and internal bookkeeping + +The difference between `storage_size` and `payload` in the `dbstat` results shows SQLite's storage overhead. The `ps_oplog` table will typically be one of the largest tables since it maintains a full copy of your synced data. + +To see just the JSON data size in `ps_oplog` (excluding SQLite overhead), run: +```sql +SELECT sum(length(data)) / 1024.0 / 1024.0 AS size_mb FROM ps_oplog; +``` +This measures only the raw JSON payloads, which will be smaller than the on-disk storage reported by `dbstat`. + +## Reducing SQLite file size + +### VACUUM Command + +The `VACUUM` command reclaims unused space in the database: + +```sql +VACUUM; +``` + + +The `VACUUM` command requires enough free disk space to create a temporary copy of the database. Ensure sufficient space is available before running. + + +### Increase page size + + +This may cause issues when using `IndexedDB` on web - it is not supported there, and could corrupt the database. + + +Increasing the page size from the default **4KB** to **16KB** can reduce storage overhead significantly. + +**Trade-offs:** +- Reduces storage overhead substantially +- May increase overhead for many small writes + +The page size must be set as one of the first `PRAGMA` statements after opening the database: + +```sql +PRAGMA page_size = 16384; +``` \ No newline at end of file