Crystory

"Crystory" (Crystal + History) is a cross-platform USB/external drive indexing tool that builds a persistent and searchable record of all files on attached storage devices. Even after a device is removed, its mirrored data remains queryable locally or in the cloud.

🌟 Project Goals

• Identify USB and external drives by unique ID (UUID, serial, etc.)
• Cache full file/directory structure into local SQLite databases
• Allow fast query and lookup by device or file info
• Run as a background service to auto-scan newly attached storage
• Optionally sync device cache databases to the cloud
• CLI-first design, scriptable and extensible

🚀 Key Features

• Auto-detect new storage devices across OS platforms
• Scan and cache file metadata into per-device SQLite DBs
• Query file info without device being attached
• Track device metadata (label, UUID, mount path, etc.)
• Future: rclone-based cloud sync support (Dropbox, GDrive, S3)

⬇️ Installation

Currently, the most reliable way to install crystory is by compiling it from source via cargo.

1. Clone the repository:

   git clone https://github.com/yihua1218/crystory.git
   cd crystory
   

2. Install the binary globally:

   cargo install --path .
   

This command will compile the optimized release binary and place the crystory executable inside your cargo bin directory (typically ~/.cargo/bin). Make sure this directory is added to your system's PATH.

💻 Development & Building

Prerequisites

• Rust toolchain (cargo, rustc)

Compiling

To build the project for development, run:

cargo build

The compiled executable will be located at target/debug/crystory.

Running Locally

During development, you can use cargo run followed by -- to pass arguments to the CLI. This avoids the command not found error if crystory is not in your global system PATH:

cargo run -- list-devices
cargo run -- scan ./my_folder

Testing

Run the test suite to verify code correctness:

cargo test

📦 Example CLI Commands

Once compiled or installed globally (e.g. via cargo install --path .), you can invoke the program directly:

# List all known indexed devices
crystory list-devices

# Scan all auto-detected external storage devices (Default behavior)
crystory scan

# Scan a specific device or folder manually
crystory scan --mount /Volumes/USB_DRIVE

# Query a previously indexed device by UUID
crystory query --uuid 1234-5678 --filter ".pdf"

# Start or stop the background indexer service
crystory service start
crystory service stop

🗄️ Database & Storage

Crystory uses a local SQLite database (crystory.db) to cache all device metadata and file structures.

• Storage Location: The database is stored in the system's standard application data directory (e.g., ~/Library/Application Support/crystory/crystory.db on macOS, ~/.local/share/crystory/ on Linux, and %APPDATA%\crystory\ on Windows).
• Storage Format: It uses SQLite with two main tables: devices (storing storage capacity, UUID, and online status) and files (storing file paths, sizes, and timestamps mapping to the device UUID).

⚙️ Configuration

Crystory automatically creates a config.json file on first run. It is located in the system's standard configuration directory (e.g., ~/Library/Application Support/crystory/config.json on macOS, ~/.config/crystory/config.json on Linux, and %APPDATA%\crystory\config.json on Windows).

Configurable parameters:

• scan_strategy (string): The method used for scanning (default: "incremental").
• cpu_priority (string): Target CPU priority for the process (default: "low").
• throttle_ms_per_batch (number): Sleep duration in milliseconds between batch database transactions to prevent 100% CPU usage (default: 5).
• batch_size (number): Number of file entries processed in a single transaction (default: 100).

🛠️ To-do List (Prioritized)

✅ Phase 1 - Core Functionality

1. Device UUID and metadata detection (cross-platform)
2. Directory traversal and file metadata extraction
3. SQLite schema design for device + file cache
4. Basic CLI subcommands (scan, list-devices, query)
5. Local cache storage and lookup by UUID

🚧 Phase 2 - Platform Integration

6. Background service daemon for macOS / Linux / Windows
7. Auto-trigger scan on new device insertion
8. Logging and error handling

🌐 Phase 3 - Cloud and Extensibility

9. rclone or API-based sync to S3/Dropbox/GDrive
10. Web UI or TUI for browsing device contents
11. File hash support for duplication detection
12. Export/import index (for backup and sharing)

🏗️ Architecture

Crystory's core engine relies on:

• Rust rusqlite: Used to manage local cache databases efficiently.
• Cross-Platform UUID Tracking: Determines the unique identifiers of mounted generic devices using system-specific probes (e.g., system_profiler on macOS).
• Background Daemon (Planned): Will run quietly in the background, listening to system mount/unmount events to trigger scanner processes asynchronously.

🤝 Contributing

We heartily welcome community contributions! Please make sure to follow this checklist before submitting a pull request:

1. Format your code: Run cargo fmt to adhere to standard Rust formatting guidelines.
2. Linting: Ensure cargo clippy completes without firing any warnings.
3. Tests: Check that your changes don't break existing functionality by running cargo test.
4. Pre-Commit Checks: Verify your final commit using our standard AI workflow. Type /pre-commit to let the system automatically test and sanitize your code.

📄 License

This project is dedicated to the public domain under The Unlicense. For more licensing details, please refer to the LICENSE file in the project workspace.

---

Built with ❤️ in Rust. Inspired by the clarity of crystals and the persistence of memory.