diff --git a/README.md b/README.md index f6e9b61..26c37ba 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,9 @@ inherits its colors from your terminal theme. This repository currently contains the first local-calendar vertical slice: date/domain logic, guarded atomic iCalendar persistence, add/edit/delete forms, a responsive ANSI terminal UI with month and agenda views, explicit safe -calendar import/export, tests, and Linux launch integration. See +calendar import/export, tests, and Linux launch integration. A provider-neutral +SQLite cache now provides the durable boundary required before +Office 365 or any other remote provider is connected. See [the product specification](docs/PRODUCT.md), [architecture](docs/ARCHITECTURE.md), and [roadmap](docs/ROADMAP.md). @@ -105,3 +107,22 @@ backup itself is kept so recovery can be repeated. For a popup calendar binding and version-specific Hyprland rules, see [Hyprland integration](docs/HYPRLAND.md). + +## Remote-provider foundation + +The remote-provider foundation is a private local SQLite database in WAL mode, +separate from the interactive `.ics` file workflow. Its schema is migrated by +version in all-or-nothing transactions. It preserves raw provider payloads, +inactive calendars, and tombstoned events so synchronization cannot silently +discard data that the current application does not understand. Each opaque +provider cursor is scoped to a calendar and synchronization window and is +committed atomically with the pulled page it represents. Normalized event +instances provide immutable snapshot input, while transactional outbox and +conflict records are reserved for later write synchronization. + +This is a persistence prerequisite, not working Office 365 integration. Nocal +does not yet perform OAuth, make Graph requests, or synchronize a remote +calendar. OAuth tokens and other secrets will never be stored in SQLite; they +belong in the Freedesktop Secret Service. The next implementation step is an +injectable HTTP boundary with PKCE authentication and deterministic fake +transports, followed by read-only Microsoft Graph delta synchronization. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 6c91cde..128fcc4 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -6,7 +6,7 @@ and terminal rendering replaceable. ```text src/main.cpp ├── tui/ ANSI rendering, input decoding, terminal lifecycle - ├── storage/ iCalendar file adapter (later: SQLite cache) + ├── storage/ iCalendar file adapter and provider-neutral cache └── domain/ Event, Calendar, civil dates, month layout and queries future sync workers @@ -118,13 +118,27 @@ semantically. Browsing remains available for unsupported files. This conservative boundary is more important than partial editing because a successful-looking edit must not erase unrelated calendar data. -Provider synchronization will not write directly into this UI file. It will use -a transaction-capable local cache and preserve remote ETags/sync tokens in a -provider metadata table. +Provider synchronization does not write directly into this UI file. Its +prerequisite durable boundary is a provider-neutral SQLite database in WAL mode. +The database is a private local file, and schema changes run as versioned, +all-or-nothing migrations so a failed upgrade cannot leave a partially migrated +cache. -The planned cache schema has `calendars`, `events`, `event_instances`, and -`sync_state`. Raw provider payloads are retained alongside normalized fields so -an older Nocal cannot destroy fields it does not understand. +The cache separates provider records from normalized projections. It retains +raw provider payloads alongside normalized fields so an older Nocal cannot +destroy fields it does not understand. Inactive calendars and tombstoned events +remain recorded rather than disappearing from history. Normalized event +instances are materialized for immutable UI snapshots; provider payload details +do not cross into rendering or the domain model. + +Incremental progress is scoped by provider account, calendar, and synchronization +window. A provider's opaque cursor is committed in the same transaction as each +pulled page, preventing a crash from advancing the cursor beyond durable data. +Transactional outbox and conflict records are reserved in the boundary for +future local writes and deterministic conflict handling. This cache does not +itself implement networking, authentication, or Microsoft Graph synchronization. +OAuth tokens and other secrets are explicitly excluded from SQLite and belong +in the Freedesktop Secret Service. ## Terminal backend @@ -149,3 +163,10 @@ Service, never in the calendar database or command line. Conflict resolution is deterministic: unchanged side wins; otherwise retain both versions and mark a conflict for the user. Network work happens outside the render loop and publishes immutable snapshots to it. + +The first provider-facing step after the cache is verified is an injectable HTTP +boundary, browser-based authorization-code flow with PKCE, and fake transports +for deterministic authentication and failure tests. Read-only Microsoft Graph +calendar discovery and delta synchronization follows. Graph delta links remain +opaque and are persisted only through the calendar/window transaction described +above; remote writes and conflict resolution come later. diff --git a/docs/PRODUCT.md b/docs/PRODUCT.md index 143ff88..d4d9b04 100644 --- a/docs/PRODUCT.md +++ b/docs/PRODUCT.md @@ -112,6 +112,29 @@ values incompatible with the event's `DTSTART` time basis or `TZID`, detached read-only. Remaining local-data work includes detached overrides and rule editing. +## Remote calendar foundation + +Remote calendars must cross a durable local boundary before they are exposed to +the UI. That boundary is a private, provider-neutral SQLite cache in WAL mode, +with versioned migrations that either commit completely or leave the prior +schema intact. It keeps raw provider payloads, inactive calendars, and +tombstoned events so refreshes and older clients do not erase information. Each +opaque incremental cursor belongs to a particular calendar and time window and +is committed atomically with every pulled page. Normalized event instances feed +immutable snapshots rather than exposing provider response types to the month +view. + +The cache also reserves transactional outbox and conflict records for future +offline writes, but remote mutation behavior is not part of this phase. OAuth +tokens and other credentials never belong in the calendar database; account +secrets will be held by the Freedesktop Secret Service. + +This foundation does not make Office 365 usable yet: Nocal currently performs no +OAuth flow, HTTP request, or Microsoft Graph synchronization. The next product +slice introduces injectable HTTP and PKCE authentication with fake transports, +then adds read-only Graph delta synchronization. Remote writes remain gated on +the durability and conflict paths being exercised end to end. + ## Local file interchange Import and export are explicit command-line workflows. They never contact the diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 9fd705d..92ae337 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -24,12 +24,22 @@ - Embedded VTIMEZONE support - Screen-reader-friendly linear view and full Unicode display-width handling -## 0.2 — durable cache and provider boundary +## 0.2 — durable cache and provider boundary — in progress -- SQLite WAL cache and migration framework -- Background job queue, immutable UI snapshots, observable sync status -- Secret Service credentials and OAuth callback helper -- Generic `SyncProvider` contract plus a fake provider test suite +Completed cache foundation: + +- Provider-neutral SQLite WAL cache in a private local file +- Versioned, all-or-nothing schema migrations +- Raw provider payload retention, inactive calendars, and event tombstones +- Per-calendar/window opaque cursors committed atomically with pulled pages +- Normalized event instances for immutable snapshots +- Reserved transactional outbox and conflict records for later remote writes + +Next provider-boundary slices: + +- Injectable HTTP, PKCE authentication, and deterministic fake transports +- Secret Service credentials; OAuth secrets are never stored in SQLite +- Generic `SyncProvider` contract and observable sync status ## 0.3 — CalDAV @@ -40,9 +50,15 @@ ## 0.4 — hosted providers - Google Calendar adapter with incremental synchronization -- Microsoft 365 adapter over Microsoft Graph delta queries +- Microsoft 365 read-only discovery and Graph delta synchronization, followed by + ETag-aware remote writes after the conflict boundary is proven - Account/calendar management UI and per-calendar ANSI identity +The current cache work is only the prerequisite durable boundary. No network, +OAuth, or Graph synchronization is implemented in that phase. After the cache +passes its verification gates, the immediate sequence is injectable HTTP plus +PKCE and fake transports, then read-only Microsoft Graph delta synchronization. + ## 1.0 — distribution quality - Stable configuration/data format and documented recovery procedures diff --git a/include/nocal/storage/sync_cache.hpp b/include/nocal/storage/sync_cache.hpp new file mode 100644 index 0000000..f83be2e --- /dev/null +++ b/include/nocal/storage/sync_cache.hpp @@ -0,0 +1,122 @@ +#pragma once + +#include +#include +#include +#include +#include +#include + +namespace nocal::storage { + +struct CachedAccount { + std::string id; + std::string provider; + std::string remote_subject; + std::string display_name; + + bool operator==(const CachedAccount&) const = default; +}; + +struct CachedCalendar { + std::string id; + std::string account_id; + std::string remote_id; + std::string name; + std::string color; + bool read_only{false}; + bool active{true}; + std::string raw_payload; + + bool operator==(const CachedCalendar&) const = default; +}; + +struct CachedEvent { + std::string id; + std::string calendar_id; + std::string remote_id; + std::string uid; + std::string etag; + std::string change_key; + std::string raw_payload; + bool deleted{false}; + + bool operator==(const CachedEvent&) const = default; +}; + +struct CachedEventInstance { + std::string id; + std::string event_id; + std::int64_t start_epoch_microseconds{0}; + std::int64_t end_epoch_microseconds{0}; + bool all_day{false}; + std::string title; + std::string location; + std::string description; + std::string time_zone; + + bool operator==(const CachedEventInstance&) const = default; +}; + +struct SyncCheckpoint { + std::string calendar_id; + std::string window_start; + std::string window_end; + std::string cursor; + bool complete{false}; + + bool operator==(const SyncCheckpoint&) const = default; +}; + +struct PullPage { + std::vector events; + std::vector instances; + SyncCheckpoint checkpoint; + + bool operator==(const PullPage&) const = default; +}; + +struct CacheSnapshot { + std::vector accounts; + std::vector calendars; + std::vector events; + std::vector instances; + std::vector checkpoints; + + bool operator==(const CacheSnapshot&) const = default; +}; + +class SyncCache { +public: + static constexpr int current_schema_version = 1; + + explicit SyncCache(std::filesystem::path path); + ~SyncCache(); + + SyncCache(const SyncCache&) = delete; + SyncCache& operator=(const SyncCache&) = delete; + SyncCache(SyncCache&&) noexcept; + SyncCache& operator=(SyncCache&&) noexcept; + + [[nodiscard]] const std::filesystem::path& path() const noexcept; + + void upsert_account(const CachedAccount& account); + + // Call only after a provider calendar listing has completed. Calendars + // omitted from the completed listing are retained but marked inactive. + void replace_calendars_after_complete_listing( + std::string account_id, std::span calendars); + + // Applies one provider page and advances its opaque cursor in the same + // transaction. Instances for each changed event are replaced; tombstoned + // events retain their last raw payload and have no instances. + void apply_pull_page(const PullPage& page); + + [[nodiscard]] CacheSnapshot snapshot() const; + +private: + struct Impl; + std::unique_ptr impl_; +}; + +} // namespace nocal::storage diff --git a/meson.build b/meson.build index cbd8db8..71100b3 100644 --- a/meson.build +++ b/meson.build @@ -12,40 +12,48 @@ project( ) inc = include_directories('include') +sqlite = dependency('sqlite3') +threads = dependency('threads') nocal_sources = files( 'src/domain/calendar_transfer.cpp', 'src/domain/date.cpp', 'src/domain/event_query.cpp', 'src/domain/month_layout.cpp', 'src/storage/ics_store.cpp', + 'src/storage/sync_cache.cpp', 'src/tui/Screen.cpp', 'src/tui/EventEditor.cpp', 'src/tui/Terminal.cpp', 'src/tui/TuiApp.cpp', ) -nocal_lib = static_library('nocal-core', nocal_sources, include_directories: inc) +nocal_lib = static_library('nocal-core', nocal_sources, include_directories: inc, + dependencies: sqlite) +nocal_dep = declare_dependency(include_directories: inc, link_with: nocal_lib, + dependencies: sqlite) -nocal_executable = executable('nocal', 'src/main.cpp', include_directories: inc, - link_with: nocal_lib, install: true) +nocal_executable = executable('nocal', 'src/main.cpp', + dependencies: nocal_dep, install: true) domain_tests = executable('domain-tests', 'tests/domain_tests.cpp', - include_directories: inc, link_with: nocal_lib) + dependencies: nocal_dep) test('domain', domain_tests) calendar_transfer_tests = executable('calendar-transfer-tests', - 'tests/calendar_transfer_tests.cpp', include_directories: inc, - link_with: nocal_lib) + 'tests/calendar_transfer_tests.cpp', dependencies: nocal_dep) test('calendar-transfer', calendar_transfer_tests) ics_tests = executable('ics-tests', 'tests/ics_tests.cpp', - include_directories: inc, link_with: nocal_lib) + dependencies: nocal_dep) test('ics-storage', ics_tests) +sync_cache_tests = executable('sync-cache-tests', 'tests/sync_cache_tests.cpp', + dependencies: [nocal_dep, threads]) +test('sync-cache', sync_cache_tests) tui_tests = executable('tui-tests', 'tests/tui_tests.cpp', - include_directories: inc, link_with: nocal_lib) + dependencies: nocal_dep) test('tui', tui_tests) tui_focus_tests = executable('tui-focus-tests', 'tests/tui_focus_tests.cpp', - include_directories: inc, link_with: nocal_lib) + dependencies: nocal_dep) test('tui-focus', tui_focus_tests) editor_tests = executable('editor-tests', 'tests/editor_tests.cpp', - include_directories: inc, link_with: nocal_lib) + dependencies: nocal_dep) test('editor', editor_tests) shell = find_program('sh') diff --git a/shell.nix b/shell.nix index bab2e4d..5cb8441 100644 --- a/shell.nix +++ b/shell.nix @@ -9,5 +9,6 @@ pkgs.mkShell { buildInputs = with pkgs; [ gcc + sqlite ]; } diff --git a/src/storage/sync_cache.cpp b/src/storage/sync_cache.cpp new file mode 100644 index 0000000..9bf4986 --- /dev/null +++ b/src/storage/sync_cache.cpp @@ -0,0 +1,738 @@ +#include "nocal/storage/sync_cache.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include + +namespace nocal::storage { +namespace { + +[[noreturn]] void throw_sqlite(sqlite3* database, std::string_view operation) { + const char* detail = database == nullptr ? nullptr : sqlite3_errmsg(database); + throw std::runtime_error( + "sync cache " + std::string(operation) + " failed" + + (detail == nullptr ? std::string{} : ": " + std::string(detail))); +} + +void execute(sqlite3* database, const char* sql, std::string_view operation) { + char* detail = nullptr; + const int result = sqlite3_exec(database, sql, nullptr, nullptr, &detail); + if (result == SQLITE_OK) { + return; + } + const std::string message = detail == nullptr ? sqlite3_errmsg(database) : detail; + sqlite3_free(detail); + throw std::runtime_error( + "sync cache " + std::string(operation) + " failed: " + message); +} + +class Statement { +public: + Statement(sqlite3* database, const char* sql, std::string_view operation) + : database_(database), operation_(operation) { + if (sqlite3_prepare_v2(database_, sql, -1, &statement_, nullptr) != SQLITE_OK) { + throw_sqlite(database_, operation_); + } + } + + ~Statement() { sqlite3_finalize(statement_); } + + Statement(const Statement&) = delete; + Statement& operator=(const Statement&) = delete; + + void bind_text(int index, std::string_view value) { + if (value.size() > static_cast(std::numeric_limits::max()) + || sqlite3_bind_text(statement_, index, value.data(), static_cast(value.size()), + SQLITE_TRANSIENT) + != SQLITE_OK) { + throw_sqlite(database_, operation_); + } + } + + void bind_int64(int index, std::int64_t value) { + if (sqlite3_bind_int64(statement_, index, value) != SQLITE_OK) { + throw_sqlite(database_, operation_); + } + } + + void bind_bool(int index, bool value) { bind_int64(index, value ? 1 : 0); } + + [[nodiscard]] bool next() { + const int result = sqlite3_step(statement_); + if (result == SQLITE_ROW) { + return true; + } + if (result == SQLITE_DONE) { + return false; + } + throw_sqlite(database_, operation_); + } + + void run() { + if (next()) { + throw std::runtime_error( + "sync cache " + std::string(operation_) + " returned an unexpected row"); + } + } + + [[nodiscard]] std::string text_column(int index) const { + const unsigned char* value = sqlite3_column_text(statement_, index); + const int size = sqlite3_column_bytes(statement_, index); + if (value == nullptr || size == 0) { + return {}; + } + return {reinterpret_cast(value), static_cast(size)}; + } + + [[nodiscard]] std::int64_t int64_column(int index) const { + return sqlite3_column_int64(statement_, index); + } + + [[nodiscard]] bool bool_column(int index) const { + return sqlite3_column_int(statement_, index) != 0; + } + +private: + sqlite3* database_; + sqlite3_stmt* statement_{nullptr}; + std::string operation_; +}; + +class Transaction { +public: + explicit Transaction(sqlite3* database) : database_(database) { + execute(database_, "BEGIN IMMEDIATE", "starting transaction"); + } + + ~Transaction() { + if (!finished_) { + sqlite3_exec(database_, "ROLLBACK", nullptr, nullptr, nullptr); + } + } + + Transaction(const Transaction&) = delete; + Transaction& operator=(const Transaction&) = delete; + + void commit() { + execute(database_, "COMMIT", "committing transaction"); + finished_ = true; + } + +private: + sqlite3* database_; + bool finished_{false}; +}; + +[[nodiscard]] std::string path_error(const std::filesystem::path& path, std::string_view reason) { + return "invalid sync cache path '" + path.string() + "': " + std::string(reason); +} + +[[nodiscard]] bool prepare_database_file(const std::filesystem::path& path) { + if (path.empty() || !path.has_filename() || path.filename().empty() + || path.filename() == "." || path.filename() == ".." + || path.native().find('\0') != std::string::npos) { + throw std::invalid_argument(path_error(path, "a database filename is required")); + } + + struct stat link_status {}; + bool created = false; + int descriptor = -1; + if (::lstat(path.c_str(), &link_status) == 0) { + if (S_ISLNK(link_status.st_mode) || !S_ISREG(link_status.st_mode)) { + throw std::invalid_argument(path_error(path, "existing target must be a regular file")); + } + descriptor = ::open(path.c_str(), O_RDWR | O_CLOEXEC | O_NOFOLLOW); + } else if (errno == ENOENT) { + descriptor = ::open(path.c_str(), + O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, S_IRUSR | S_IWUSR); + created = descriptor >= 0; + } else { + throw std::runtime_error(path_error(path, std::strerror(errno))); + } + if (descriptor < 0) { + throw std::runtime_error(path_error(path, std::strerror(errno))); + } + struct stat opened_status {}; + if (::fstat(descriptor, &opened_status) != 0) { + const int error = errno; + ::close(descriptor); + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::runtime_error(path_error(path, std::strerror(error))); + } + if (!S_ISREG(opened_status.st_mode)) { + ::close(descriptor); + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::invalid_argument(path_error(path, "opened target is not a regular file")); + } + if (::fchmod(descriptor, S_IRUSR | S_IWUSR) != 0) { + const int error = errno; + ::close(descriptor); + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::runtime_error(path_error(path, std::strerror(error))); + } + if (::fstat(descriptor, &opened_status) != 0) { + const int error = errno; + ::close(descriptor); + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::runtime_error(path_error(path, std::strerror(error))); + } + if ((opened_status.st_mode & (S_IRWXU | S_IRWXG | S_IRWXO)) != (S_IRUSR | S_IWUSR)) { + ::close(descriptor); + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::runtime_error(path_error(path, "database permissions are not private")); + } + if (::close(descriptor) != 0) { + const int error = errno; + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::runtime_error(path_error(path, std::strerror(error))); + } + return created; +} + +[[nodiscard]] int user_version(sqlite3* database) { + Statement statement(database, "PRAGMA user_version", "reading schema version"); + if (!statement.next()) { + throw std::runtime_error("sync cache schema version is unavailable"); + } + const std::int64_t version = statement.int64_column(0); + if (version < 0 || version > std::numeric_limits::max()) { + throw std::runtime_error("sync cache schema version is invalid"); + } + return static_cast(version); +} + +void require_pragma_integer( + sqlite3* database, const char* sql, std::int64_t expected, std::string_view operation) { + Statement statement(database, sql, operation); + if (!statement.next() || statement.int64_column(0) != expected) { + throw std::runtime_error("sync cache " + std::string(operation) + " was not applied"); + } +} + +void require_pragma_text( + sqlite3* database, const char* sql, std::string_view expected, std::string_view operation) { + Statement statement(database, sql, operation); + if (!statement.next() || statement.text_column(0) != expected) { + throw std::runtime_error("sync cache " + std::string(operation) + " was not applied"); + } +} + +constexpr const char* schema_v1 = R"sql( +CREATE TABLE accounts ( + id TEXT PRIMARY KEY CHECK(length(id) > 0), + provider TEXT NOT NULL CHECK(length(provider) > 0), + remote_subject TEXT NOT NULL CHECK(length(remote_subject) > 0), + display_name TEXT NOT NULL, + UNIQUE(provider, remote_subject) +) STRICT; +CREATE TABLE calendars ( + id TEXT PRIMARY KEY CHECK(length(id) > 0), + account_id TEXT NOT NULL REFERENCES accounts(id) ON UPDATE RESTRICT ON DELETE RESTRICT, + remote_id TEXT NOT NULL CHECK(length(remote_id) > 0), + name TEXT NOT NULL, + color TEXT NOT NULL, + read_only INTEGER NOT NULL CHECK(read_only IN (0, 1)), + active INTEGER NOT NULL CHECK(active IN (0, 1)), + raw_payload TEXT NOT NULL, + UNIQUE(account_id, remote_id) +) STRICT; +CREATE INDEX calendars_account_id_idx ON calendars(account_id); +CREATE TABLE events ( + id TEXT PRIMARY KEY CHECK(length(id) > 0), + calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE RESTRICT, + remote_id TEXT NOT NULL CHECK(length(remote_id) > 0), + uid TEXT NOT NULL, + etag TEXT NOT NULL, + change_key TEXT NOT NULL, + raw_payload TEXT NOT NULL, + deleted INTEGER NOT NULL CHECK(deleted IN (0, 1)), + UNIQUE(calendar_id, remote_id) +) STRICT; +CREATE INDEX events_calendar_id_idx ON events(calendar_id); +CREATE TABLE event_instances ( + id TEXT PRIMARY KEY CHECK(length(id) > 0), + event_id TEXT NOT NULL REFERENCES events(id) ON UPDATE RESTRICT ON DELETE CASCADE, + start_epoch_microseconds INTEGER NOT NULL, + end_epoch_microseconds INTEGER NOT NULL, + all_day INTEGER NOT NULL CHECK(all_day IN (0, 1)), + title TEXT NOT NULL, + location TEXT NOT NULL, + description TEXT NOT NULL, + time_zone TEXT NOT NULL, + CHECK(start_epoch_microseconds <= end_epoch_microseconds), + CHECK(all_day = 0 OR start_epoch_microseconds < end_epoch_microseconds) +) STRICT; +CREATE INDEX event_instances_event_id_idx ON event_instances(event_id); +CREATE TABLE sync_state ( + calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE CASCADE, + window_start TEXT NOT NULL, + window_end TEXT NOT NULL, + cursor TEXT NOT NULL CHECK(length(cursor) > 0), + complete INTEGER NOT NULL CHECK(complete IN (0, 1)), + CHECK((length(window_start) = 0) = (length(window_end) = 0)), + PRIMARY KEY(calendar_id, window_start, window_end) +) STRICT; +CREATE TABLE outbox ( + id TEXT PRIMARY KEY CHECK(length(id) > 0), + calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE RESTRICT, + event_id TEXT NOT NULL REFERENCES events(id) ON UPDATE RESTRICT ON DELETE RESTRICT, + operation TEXT NOT NULL CHECK(length(operation) > 0), + payload TEXT NOT NULL, + base_etag TEXT NOT NULL, + idempotency_key TEXT NOT NULL UNIQUE CHECK(length(idempotency_key) > 0), + status TEXT NOT NULL CHECK(length(status) > 0), + attempt_count INTEGER NOT NULL DEFAULT 0 CHECK(attempt_count >= 0), + not_before_epoch_seconds INTEGER +) STRICT; +CREATE TABLE conflicts ( + id TEXT PRIMARY KEY CHECK(length(id) > 0), + calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE RESTRICT, + event_id TEXT NOT NULL REFERENCES events(id) ON UPDATE RESTRICT ON DELETE RESTRICT, + base_etag TEXT NOT NULL, + local_payload TEXT NOT NULL, + remote_payload TEXT NOT NULL, + status TEXT NOT NULL CHECK(length(status) > 0) +) STRICT; +)sql"; + +void validate_nonempty(std::string_view value, std::string_view field) { + if (value.empty()) { + throw std::invalid_argument(std::string(field) + " must not be empty"); + } +} + +void require_account(sqlite3* database, std::string_view account_id) { + Statement statement(database, "SELECT 1 FROM accounts WHERE id = ?", "checking account"); + statement.bind_text(1, account_id); + if (!statement.next()) { + throw std::invalid_argument("sync cache account does not exist"); + } +} + +void ensure_account_identity(sqlite3* database, const CachedAccount& account) { + Statement statement(database, + "SELECT provider, remote_subject FROM accounts WHERE id = ?", "checking account identity"); + statement.bind_text(1, account.id); + if (statement.next() + && (statement.text_column(0) != account.provider + || statement.text_column(1) != account.remote_subject)) { + throw std::invalid_argument("account identity cannot be reassigned"); + } +} + +void require_active_calendar(sqlite3* database, std::string_view calendar_id) { + Statement statement( + database, "SELECT active FROM calendars WHERE id = ?", "checking calendar"); + statement.bind_text(1, calendar_id); + if (!statement.next()) { + throw std::invalid_argument("sync cache calendar does not exist"); + } + if (!statement.bool_column(0)) { + throw std::invalid_argument("cannot apply a pull page to an inactive calendar"); + } +} + +void ensure_calendar_identity(sqlite3* database, const CachedCalendar& calendar) { + Statement statement(database, + "SELECT account_id, remote_id FROM calendars WHERE id = ?", + "checking calendar identity"); + statement.bind_text(1, calendar.id); + if (statement.next() + && (statement.text_column(0) != calendar.account_id + || statement.text_column(1) != calendar.remote_id)) { + throw std::invalid_argument("calendar identity cannot be reassigned"); + } +} + +void ensure_event_identity(sqlite3* database, const CachedEvent& event) { + Statement statement(database, + "SELECT calendar_id, remote_id FROM events WHERE id = ?", "checking event identity"); + statement.bind_text(1, event.id); + if (statement.next() + && (statement.text_column(0) != event.calendar_id + || statement.text_column(1) != event.remote_id)) { + throw std::invalid_argument("event identity cannot be reassigned"); + } +} + +} // namespace + +struct SyncCache::Impl { + explicit Impl(std::filesystem::path cache_path) : path(std::move(cache_path)) { + const bool created = prepare_database_file(path); + sqlite3* opened = nullptr; + const int result = sqlite3_open_v2(path.c_str(), &opened, + SQLITE_OPEN_READWRITE | SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_PRIVATECACHE + | SQLITE_OPEN_NOFOLLOW, + nullptr); + if (result != SQLITE_OK) { + const std::string detail = opened == nullptr ? sqlite3_errstr(result) : sqlite3_errmsg(opened); + if (opened != nullptr) { + sqlite3_close(opened); + } + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + } + throw std::runtime_error("unable to open sync cache '" + path.string() + "': " + detail); + } + database = opened; + + try { + if (sqlite3_extended_result_codes(database, 1) != SQLITE_OK + || sqlite3_busy_timeout(database, 5'000) != SQLITE_OK) { + throw_sqlite(database, "configuring connection error handling"); + } + execute(database, "PRAGMA trusted_schema = OFF", "disabling trusted schema"); + require_pragma_integer( + database, "PRAGMA trusted_schema", 0, "disabling trusted schema"); +#ifdef SQLITE_DBCONFIG_DEFENSIVE + if (sqlite3_db_config(database, SQLITE_DBCONFIG_DEFENSIVE, 1, nullptr) != SQLITE_OK) { + throw_sqlite(database, "enabling defensive mode"); + } +#endif + const int version = user_version(database); + if (version > SyncCache::current_schema_version) { + throw std::runtime_error("sync cache schema version " + std::to_string(version) + + " is newer than supported version " + + std::to_string(SyncCache::current_schema_version)); + } + execute(database, "PRAGMA foreign_keys = ON", "enabling foreign keys"); + require_pragma_integer(database, "PRAGMA foreign_keys", 1, "enabling foreign keys"); + if (version == 0) { + Transaction migration(database); + const int locked_version = user_version(database); + if (locked_version == 0) { + execute(database, schema_v1, "creating schema version 1"); + execute(database, "PRAGMA user_version = 1", "recording schema version 1"); + } else if (locked_version > SyncCache::current_schema_version) { + throw std::runtime_error("sync cache schema changed to an unsupported version"); + } + migration.commit(); + } + execute(database, "PRAGMA journal_mode = WAL", "enabling WAL mode"); + require_pragma_text(database, "PRAGMA journal_mode", "wal", "enabling WAL mode"); + execute(database, "PRAGMA synchronous = FULL", "setting full synchronization"); + require_pragma_integer( + database, "PRAGMA synchronous", 2, "setting full synchronization"); + } catch (...) { + sqlite3_close(database); + database = nullptr; + if (created) { + std::error_code ignored; + std::filesystem::remove(path, ignored); + std::filesystem::remove(path.string() + "-wal", ignored); + std::filesystem::remove(path.string() + "-shm", ignored); + } + throw; + } + } + + ~Impl() { + if (database != nullptr) { + sqlite3_close(database); + } + } + + std::filesystem::path path; + sqlite3* database{nullptr}; + mutable std::mutex mutex; +}; + +SyncCache::SyncCache(std::filesystem::path path) : impl_(std::make_unique(std::move(path))) {} + +SyncCache::~SyncCache() = default; +SyncCache::SyncCache(SyncCache&&) noexcept = default; +SyncCache& SyncCache::operator=(SyncCache&&) noexcept = default; + +const std::filesystem::path& SyncCache::path() const noexcept { return impl_->path; } + +void SyncCache::upsert_account(const CachedAccount& account) { + validate_nonempty(account.id, "account id"); + validate_nonempty(account.provider, "account provider"); + validate_nonempty(account.remote_subject, "account remote subject"); + + std::scoped_lock lock(impl_->mutex); + Transaction transaction(impl_->database); + ensure_account_identity(impl_->database, account); + Statement statement(impl_->database, R"sql( +INSERT INTO accounts(id, provider, remote_subject, display_name) VALUES(?, ?, ?, ?) +ON CONFLICT(id) DO UPDATE SET + display_name = excluded.display_name +)sql", "upserting account"); + statement.bind_text(1, account.id); + statement.bind_text(2, account.provider); + statement.bind_text(3, account.remote_subject); + statement.bind_text(4, account.display_name); + statement.run(); + transaction.commit(); +} + +void SyncCache::replace_calendars_after_complete_listing( + std::string account_id, std::span calendars) { + validate_nonempty(account_id, "account id"); + std::unordered_set ids; + std::unordered_set remote_ids; + for (const CachedCalendar& calendar : calendars) { + validate_nonempty(calendar.id, "calendar id"); + validate_nonempty(calendar.remote_id, "calendar remote id"); + validate_nonempty(calendar.raw_payload, "calendar raw payload"); + if (calendar.account_id != account_id) { + throw std::invalid_argument("calendar belongs to a different account"); + } + if (!calendar.active) { + throw std::invalid_argument("a completed calendar listing must contain active calendars"); + } + if (!ids.insert(calendar.id).second || !remote_ids.insert(calendar.remote_id).second) { + throw std::invalid_argument("completed calendar listing contains duplicate identities"); + } + } + + std::scoped_lock lock(impl_->mutex); + Transaction transaction(impl_->database); + require_account(impl_->database, account_id); + for (const CachedCalendar& calendar : calendars) { + ensure_calendar_identity(impl_->database, calendar); + } + { + Statement deactivate(impl_->database, + "UPDATE calendars SET active = 0 WHERE account_id = ?", "deactivating calendars"); + deactivate.bind_text(1, account_id); + deactivate.run(); + } + for (const CachedCalendar& calendar : calendars) { + Statement upsert(impl_->database, R"sql( +INSERT INTO calendars( + id, account_id, remote_id, name, color, read_only, active, raw_payload) +VALUES(?, ?, ?, ?, ?, ?, 1, ?) +ON CONFLICT(id) DO UPDATE SET + name = excluded.name, + color = excluded.color, + read_only = excluded.read_only, + active = 1, + raw_payload = excluded.raw_payload +)sql", "upserting calendar"); + upsert.bind_text(1, calendar.id); + upsert.bind_text(2, calendar.account_id); + upsert.bind_text(3, calendar.remote_id); + upsert.bind_text(4, calendar.name); + upsert.bind_text(5, calendar.color); + upsert.bind_bool(6, calendar.read_only); + upsert.bind_text(7, calendar.raw_payload); + upsert.run(); + } + transaction.commit(); +} + +void SyncCache::apply_pull_page(const PullPage& page) { + const SyncCheckpoint& checkpoint = page.checkpoint; + validate_nonempty(checkpoint.calendar_id, "checkpoint calendar id"); + validate_nonempty(checkpoint.cursor, "checkpoint cursor"); + if (checkpoint.window_start.empty() != checkpoint.window_end.empty()) { + throw std::invalid_argument("checkpoint window endpoints must both be present or both be empty"); + } + + std::unordered_set event_ids; + std::unordered_set event_remote_ids; + std::unordered_set live_event_ids; + for (const CachedEvent& event : page.events) { + validate_nonempty(event.id, "event id"); + validate_nonempty(event.remote_id, "event remote id"); + if (event.calendar_id != checkpoint.calendar_id) { + throw std::invalid_argument("changed event belongs to a different calendar"); + } + if (!event.deleted && event.raw_payload.empty()) { + throw std::invalid_argument("a live changed event must retain its raw provider payload"); + } + if (!event_ids.insert(event.id).second + || !event_remote_ids.insert(event.remote_id).second) { + throw std::invalid_argument("pull page contains duplicate event identities"); + } + if (!event.deleted) { + live_event_ids.insert(event.id); + } + } + + std::unordered_set instance_ids; + for (const CachedEventInstance& instance : page.instances) { + validate_nonempty(instance.id, "event instance id"); + validate_nonempty(instance.event_id, "event instance event id"); + if (instance.start_epoch_microseconds > instance.end_epoch_microseconds + || (instance.all_day + && instance.start_epoch_microseconds == instance.end_epoch_microseconds)) { + throw std::invalid_argument("event instance must have a valid half-open interval"); + } + if (!instance_ids.insert(instance.id).second) { + throw std::invalid_argument("pull page contains duplicate event instance ids"); + } + if (!live_event_ids.contains(instance.event_id)) { + throw std::invalid_argument( + "event instance must refer to a changed, nondeleted event in the same page"); + } + } + + std::scoped_lock lock(impl_->mutex); + Transaction transaction(impl_->database); + require_active_calendar(impl_->database, checkpoint.calendar_id); + for (const CachedEvent& event : page.events) { + ensure_event_identity(impl_->database, event); + } + for (const CachedEvent& event : page.events) { + Statement upsert(impl_->database, R"sql( +INSERT INTO events( + id, calendar_id, remote_id, uid, etag, change_key, raw_payload, deleted) +VALUES(?, ?, ?, ?, ?, ?, ?, ?) +ON CONFLICT(id) DO UPDATE SET + uid = excluded.uid, + etag = excluded.etag, + change_key = excluded.change_key, + raw_payload = CASE + WHEN excluded.deleted = 1 AND length(excluded.raw_payload) = 0 THEN events.raw_payload + ELSE excluded.raw_payload + END, + deleted = excluded.deleted +)sql", "upserting changed event"); + upsert.bind_text(1, event.id); + upsert.bind_text(2, event.calendar_id); + upsert.bind_text(3, event.remote_id); + upsert.bind_text(4, event.uid); + upsert.bind_text(5, event.etag); + upsert.bind_text(6, event.change_key); + upsert.bind_text(7, event.raw_payload); + upsert.bind_bool(8, event.deleted); + upsert.run(); + } + for (const CachedEvent& event : page.events) { + Statement remove(impl_->database, + "DELETE FROM event_instances WHERE event_id = ?", "replacing event instances"); + remove.bind_text(1, event.id); + remove.run(); + } + for (const CachedEventInstance& instance : page.instances) { + Statement insert(impl_->database, R"sql( +INSERT INTO event_instances( + id, event_id, start_epoch_microseconds, end_epoch_microseconds, all_day, + title, location, description, time_zone) +VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?) +)sql", "inserting event instance"); + insert.bind_text(1, instance.id); + insert.bind_text(2, instance.event_id); + insert.bind_int64(3, instance.start_epoch_microseconds); + insert.bind_int64(4, instance.end_epoch_microseconds); + insert.bind_bool(5, instance.all_day); + insert.bind_text(6, instance.title); + insert.bind_text(7, instance.location); + insert.bind_text(8, instance.description); + insert.bind_text(9, instance.time_zone); + insert.run(); + } + { + Statement upsert(impl_->database, R"sql( +INSERT INTO sync_state(calendar_id, window_start, window_end, cursor, complete) +VALUES(?, ?, ?, ?, ?) +ON CONFLICT(calendar_id, window_start, window_end) DO UPDATE SET + cursor = excluded.cursor, + complete = excluded.complete +)sql", "advancing sync checkpoint"); + upsert.bind_text(1, checkpoint.calendar_id); + upsert.bind_text(2, checkpoint.window_start); + upsert.bind_text(3, checkpoint.window_end); + upsert.bind_text(4, checkpoint.cursor); + upsert.bind_bool(5, checkpoint.complete); + upsert.run(); + } + transaction.commit(); +} + +CacheSnapshot SyncCache::snapshot() const { + std::scoped_lock lock(impl_->mutex); + CacheSnapshot result; + { + Statement query(impl_->database, + "SELECT id, provider, remote_subject, display_name FROM accounts ORDER BY id", + "reading accounts"); + while (query.next()) { + result.accounts.push_back({query.text_column(0), query.text_column(1), + query.text_column(2), query.text_column(3)}); + } + } + { + Statement query(impl_->database, R"sql( +SELECT id, account_id, remote_id, name, color, read_only, active, raw_payload +FROM calendars ORDER BY id +)sql", "reading calendars"); + while (query.next()) { + result.calendars.push_back({query.text_column(0), query.text_column(1), + query.text_column(2), query.text_column(3), query.text_column(4), + query.bool_column(5), query.bool_column(6), query.text_column(7)}); + } + } + { + Statement query(impl_->database, R"sql( +SELECT id, calendar_id, remote_id, uid, etag, change_key, raw_payload, deleted +FROM events ORDER BY id +)sql", "reading events"); + while (query.next()) { + result.events.push_back({query.text_column(0), query.text_column(1), + query.text_column(2), query.text_column(3), query.text_column(4), + query.text_column(5), query.text_column(6), query.bool_column(7)}); + } + } + { + Statement query(impl_->database, R"sql( +SELECT id, event_id, start_epoch_microseconds, end_epoch_microseconds, + all_day, title, location, description, time_zone +FROM event_instances ORDER BY id +)sql", "reading event instances"); + while (query.next()) { + result.instances.push_back({query.text_column(0), query.text_column(1), + query.int64_column(2), query.int64_column(3), query.bool_column(4), + query.text_column(5), query.text_column(6), query.text_column(7), + query.text_column(8)}); + } + } + { + Statement query(impl_->database, R"sql( +SELECT calendar_id, window_start, window_end, cursor, complete +FROM sync_state ORDER BY calendar_id, window_start, window_end +)sql", "reading sync checkpoints"); + while (query.next()) { + result.checkpoints.push_back({query.text_column(0), query.text_column(1), + query.text_column(2), query.text_column(3), query.bool_column(4)}); + } + } + return result; +} + +} // namespace nocal::storage diff --git a/tests/sync_cache_tests.cpp b/tests/sync_cache_tests.cpp new file mode 100644 index 0000000..a111bc9 --- /dev/null +++ b/tests/sync_cache_tests.cpp @@ -0,0 +1,820 @@ +#include "nocal/storage/sync_cache.hpp" + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(_WIN32) +#include +#include +#endif + +namespace { + +using nocal::storage::CacheSnapshot; +using nocal::storage::CachedAccount; +using nocal::storage::CachedCalendar; +using nocal::storage::CachedEvent; +using nocal::storage::CachedEventInstance; +using nocal::storage::PullPage; +using nocal::storage::SyncCache; +using nocal::storage::SyncCheckpoint; + +void require(bool condition, std::string_view message) { + if (!condition) { + throw std::runtime_error(std::string{message}); + } +} + +template +void require_throws(Function&& function, std::string_view message) { + try { + function(); + } catch (const std::exception&) { + return; + } + throw std::runtime_error(std::string{message}); +} + +class TempDirectory { +public: + TempDirectory() { +#if !defined(_WIN32) + std::string pattern = + (std::filesystem::temp_directory_path() / "nocal-sync-cache-tests.XXXXXX").string(); + std::vector writable(pattern.begin(), pattern.end()); + writable.push_back('\0'); + const char* created = ::mkdtemp(writable.data()); + if (created == nullptr) { + throw std::runtime_error("mkdtemp failed"); + } + path_ = created; +#else + path_ = std::filesystem::temp_directory_path() / "nocal-sync-cache-tests"; + std::filesystem::remove_all(path_); + std::filesystem::create_directories(path_); +#endif + } + + TempDirectory(const TempDirectory&) = delete; + TempDirectory& operator=(const TempDirectory&) = delete; + + ~TempDirectory() { + std::error_code ignored; + std::filesystem::remove_all(path_, ignored); + } + + [[nodiscard]] const std::filesystem::path& path() const noexcept { return path_; } + +private: + std::filesystem::path path_; +}; + +class SqliteConnection { +public: + explicit SqliteConnection(const std::filesystem::path& path) { + const int result = sqlite3_open_v2( + path.c_str(), &database_, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nullptr); + if (result != SQLITE_OK) { + const std::string message = database_ == nullptr ? "sqlite open failed" + : sqlite3_errmsg(database_); + if (database_ != nullptr) { + sqlite3_close(database_); + database_ = nullptr; + } + throw std::runtime_error(message); + } + } + + SqliteConnection(const SqliteConnection&) = delete; + SqliteConnection& operator=(const SqliteConnection&) = delete; + + ~SqliteConnection() { + if (database_ != nullptr) { + sqlite3_close(database_); + } + } + + void execute(std::string_view sql) { + char* error = nullptr; + const std::string statement{sql}; + const int result = sqlite3_exec(database_, statement.c_str(), nullptr, nullptr, &error); + if (result != SQLITE_OK) { + const std::string message = error == nullptr ? sqlite3_errmsg(database_) : error; + sqlite3_free(error); + throw std::runtime_error(message); + } + } + + [[nodiscard]] std::string scalar_text(std::string_view sql) const { + sqlite3_stmt* statement = nullptr; + const std::string query{sql}; + if (sqlite3_prepare_v2(database_, query.c_str(), -1, &statement, nullptr) != SQLITE_OK) { + throw std::runtime_error(sqlite3_errmsg(database_)); + } + struct Finalize { + sqlite3_stmt* statement; + ~Finalize() { sqlite3_finalize(statement); } + } finalize{statement}; + if (sqlite3_step(statement) != SQLITE_ROW) { + throw std::runtime_error(sqlite3_errmsg(database_)); + } + const unsigned char* value = sqlite3_column_text(statement, 0); + return value == nullptr ? std::string{} : reinterpret_cast(value); + } + + [[nodiscard]] int scalar_int(std::string_view sql) const { + sqlite3_stmt* statement = nullptr; + const std::string query{sql}; + if (sqlite3_prepare_v2(database_, query.c_str(), -1, &statement, nullptr) != SQLITE_OK) { + throw std::runtime_error(sqlite3_errmsg(database_)); + } + struct Finalize { + sqlite3_stmt* statement; + ~Finalize() { sqlite3_finalize(statement); } + } finalize{statement}; + if (sqlite3_step(statement) != SQLITE_ROW) { + throw std::runtime_error(sqlite3_errmsg(database_)); + } + return sqlite3_column_int(statement, 0); + } + + [[nodiscard]] std::vector first_column(std::string_view sql) const { + sqlite3_stmt* statement = nullptr; + const std::string query{sql}; + if (sqlite3_prepare_v2(database_, query.c_str(), -1, &statement, nullptr) != SQLITE_OK) { + throw std::runtime_error(sqlite3_errmsg(database_)); + } + struct Finalize { + sqlite3_stmt* statement; + ~Finalize() { sqlite3_finalize(statement); } + } finalize{statement}; + std::vector values; + int result = SQLITE_ROW; + while ((result = sqlite3_step(statement)) == SQLITE_ROW) { + const unsigned char* value = sqlite3_column_text(statement, 0); + values.emplace_back(value == nullptr ? "" : reinterpret_cast(value)); + } + if (result != SQLITE_DONE) { + throw std::runtime_error(sqlite3_errmsg(database_)); + } + return values; + } + +private: + sqlite3* database_{nullptr}; +}; + +CachedAccount account(std::string id = "account-a") { + return CachedAccount{std::move(id), "microsoft-graph", "subject-a", "Work account"}; +} + +CachedCalendar calendar( + std::string id = "calendar-a", std::string account_id = "account-a") { + return CachedCalendar{std::move(id), std::move(account_id), "remote-calendar-a", "Calendar A", + "blue", false, true, R"({"calendar":"a"})"}; +} + +CachedEvent event(std::string id = "event-a", std::string calendar_id = "calendar-a") { + return CachedEvent{std::move(id), std::move(calendar_id), "remote-event-a", "uid-a", "etag-a", + "change-a", R"({"event":"a"})", false}; +} + +CachedEventInstance instance( + std::string id = "instance-a", std::string event_id = "event-a") { + return CachedEventInstance{std::move(id), std::move(event_id), 1'000'000, 2'000'000, false, + "Event A", "Room", "Description", "Europe/London"}; +} + +SyncCheckpoint checkpoint(std::string cursor = "cursor-a", std::string calendar_id = "calendar-a", + std::string window_start = "opaque-start", std::string window_end = "opaque-end") { + return SyncCheckpoint{std::move(calendar_id), std::move(window_start), std::move(window_end), + std::move(cursor), false}; +} + +PullPage page(CachedEvent changed_event = event(), CachedEventInstance changed_instance = instance(), + SyncCheckpoint next = checkpoint()) { + return PullPage{{std::move(changed_event)}, {std::move(changed_instance)}, std::move(next)}; +} + +void seed_calendar(SyncCache& cache, const CachedAccount& cached_account = account(), + const CachedCalendar& cached_calendar = calendar()) { + cache.upsert_account(cached_account); + const std::vector calendars{cached_calendar}; + cache.replace_calendars_after_complete_listing(cached_account.id, calendars); +} + +[[nodiscard]] bool contains(const std::vector& values, std::string_view value) { + return std::ranges::find(values, value) != values.end(); +} + +#if !defined(_WIN32) +void require_private_file(const std::filesystem::path& path, std::string_view description) { + struct stat metadata {}; + require(::stat(path.c_str(), &metadata) == 0, + std::string{"could not stat "}.append(description)); + require(S_ISREG(metadata.st_mode), std::string{description}.append(" is not a regular file")); + require((metadata.st_mode & (S_IRWXG | S_IRWXO)) == 0, + std::string{description}.append(" grants group or other permissions")); + require((metadata.st_mode & S_IRUSR) != 0 && (metadata.st_mode & S_IWUSR) != 0, + std::string{description}.append(" is not owner-readable and writable")); +} +#endif + +void test_fresh_schema_permissions_and_secrets(const std::filesystem::path& root) { + const auto path = root / "fresh.db"; + SyncCache cache{path}; + require(cache.path() == path, "cache did not retain its path"); + require(std::filesystem::is_regular_file(path), "fresh cache database was not created"); + + SqliteConnection inspection{path}; + require(inspection.scalar_text("PRAGMA journal_mode") == "wal", "cache is not in WAL mode"); + require(inspection.scalar_int("PRAGMA user_version") == SyncCache::current_schema_version, + "fresh cache schema version is incorrect"); + + const auto tables = inspection.first_column( + "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"); + for (const std::string_view required : {"accounts", "calendars", "events", + "event_instances", "sync_state", "outbox", "conflicts"}) { + require(contains(tables, required), std::string{"missing cache table: "}.append(required)); + } + + for (const std::string_view child : {"calendars", "events", "event_instances", "sync_state"}) { + require(inspection.scalar_int( + std::string{"SELECT COUNT(*) FROM pragma_foreign_key_list('"} + .append(child) + .append("')")) + > 0, + std::string{"missing foreign key on "}.append(child)); + } + + const auto columns = inspection.first_column( + "SELECT lower(m.name || '.' || p.name) FROM sqlite_master AS m " + "JOIN pragma_table_info(m.name) AS p WHERE m.type='table'"); + for (const auto& column : columns) { + require(column.find("token") == std::string::npos + && column.find("secret") == std::string::npos + && column.find("credential") == std::string::npos, + "cache schema contains a token, secret, or credential column"); + } + +#if !defined(_WIN32) + require_private_file(path, "cache database"); + for (const auto suffix : {"-wal", "-shm"}) { + const std::filesystem::path sidecar = path.string() + suffix; + require(std::filesystem::exists(sidecar), "WAL sidecar was not created"); + require_private_file(sidecar, "cache sidecar"); + } +#endif +} + +void test_existing_permissions_are_tightened(const std::filesystem::path& root) { +#if !defined(_WIN32) + const auto path = root / "permissive.db"; + { + SqliteConnection database{path}; + database.execute("PRAGMA user_version=0"); + } + require(::chmod(path.c_str(), 0666) == 0, "could not make fixture permissive"); + SyncCache cache{path}; + require_private_file(path, "existing cache database"); +#else + static_cast(root); +#endif +} + +void test_account_calendars_and_deterministic_snapshot(const std::filesystem::path& root) { + SyncCache cache{root / "listing.db"}; + auto account_b = account("account-b"); + account_b.remote_subject = "subject-b"; + auto account_a = account(); + cache.upsert_account(account_b); + cache.upsert_account(account_a); + + auto calendar_b = calendar("calendar-b"); + calendar_b.remote_id = "remote-calendar-b"; + calendar_b.name = "Calendar B"; + auto calendar_a = calendar(); + cache.replace_calendars_after_complete_listing( + account_a.id, std::vector{calendar_b, calendar_a}); + + auto first = cache.snapshot(); + require(first.accounts.size() == 2 && first.accounts[0].id == "account-a" + && first.accounts[1].id == "account-b", + "accounts are not returned deterministically by ID"); + require(first.calendars.size() == 2 && first.calendars[0].id == "calendar-a" + && first.calendars[1].id == "calendar-b", + "calendars are not returned deterministically by ID"); + + calendar_a.name = "Renamed"; + cache.replace_calendars_after_complete_listing( + account_a.id, std::vector{calendar_a}); + const auto second = cache.snapshot(); + require(second.calendars.size() == 2, "omitted calendar was deleted"); + require(second.calendars[0].active && second.calendars[0].name == "Renamed", + "listed calendar was not updated and kept active"); + require(!second.calendars[1].active, "omitted calendar was not deactivated"); + require(second == cache.snapshot(), "unchanged snapshots are not deterministic"); +} + +void test_pull_pages_replacement_tombstone_and_windows(const std::filesystem::path& root) { + SyncCache cache{root / "pull.db"}; + seed_calendar(cache); + + auto event_b = event("event-b"); + event_b.remote_id = "remote-event-b"; + event_b.uid = "uid-b"; + event_b.raw_payload = R"({"event":"b"})"; + auto instance_b = instance("instance-b", "event-b"); + instance_b.title = "Event B"; + PullPage first{{event_b, event()}, {instance_b, instance()}, checkpoint("cursor-1")}; + cache.apply_pull_page(first); + const auto initial = cache.snapshot(); + require(initial.events.size() == 2 && initial.instances.size() == 2, + "pull page did not atomically add events and instances"); + require(initial.events[0].id == "event-a" && initial.events[1].id == "event-b" + && initial.instances[0].id == "instance-a" + && initial.instances[1].id == "instance-b", + "events and instances are not returned deterministically by ID"); + require(initial.checkpoints.size() == 1 && initial.checkpoints[0].cursor == "cursor-1", + "pull page did not advance its cursor"); + + cache.apply_pull_page(first); + require(cache.snapshot() == initial, "repeated pull page was not idempotent"); + + auto changed = event(); + changed.etag = "etag-a-2"; + changed.raw_payload = R"({"event":"a2"})"; + auto replacement = instance("instance-a-2"); + replacement.title = "Changed A"; + cache.apply_pull_page(page(changed, replacement, checkpoint("cursor-2"))); + auto replaced = cache.snapshot(); + require(replaced.events.size() == 2 && replaced.instances.size() == 2, + "changed event replacement affected an untouched event"); + require(contains(std::vector{replaced.instances[0].id, replaced.instances[1].id}, + "instance-a-2") + && contains(std::vector{ + replaced.instances[0].id, replaced.instances[1].id}, + "instance-b"), + "changed event instances were not replaced independently"); + + auto tombstone = event(); + tombstone.deleted = true; + tombstone.raw_payload.clear(); + cache.apply_pull_page(PullPage{{tombstone}, {}, checkpoint("cursor-3")}); + const auto deleted = cache.snapshot(); + const auto deleted_event = std::ranges::find_if( + deleted.events, [](const CachedEvent& value) { return value.id == "event-a"; }); + require(deleted_event != deleted.events.end() && deleted_event->deleted, + "tombstone did not retain a deleted event record"); + require(deleted_event->raw_payload == changed.raw_payload, + "empty tombstone did not retain the prior raw payload"); + require(std::ranges::none_of(deleted.instances, + [](const CachedEventInstance& value) { return value.event_id == "event-a"; }), + "tombstone retained event instances"); + + cache.apply_pull_page(PullPage{{}, {}, checkpoint("other-cursor", "calendar-a", + "second-window-start", "second-window-end")}); + const auto windows = cache.snapshot(); + require(windows.checkpoints.size() == 2, + "independent checkpoint windows for one calendar were overwritten"); + require(windows.checkpoints[0].window_start == "opaque-start" + && windows.checkpoints[1].window_start == "second-window-start", + "checkpoints are not returned in deterministic scope order"); +} + +void test_multiple_connections_serialize(const std::filesystem::path& root) { + const auto path = root / "connections.db"; + SyncCache first{path}; + SyncCache second{path}; + + std::exception_ptr first_error; + std::exception_ptr second_error; + std::atomic begin{false}; + auto write = [&begin](SyncCache& cache, CachedAccount value, std::exception_ptr& error) { + while (!begin.load(std::memory_order_acquire)) { + std::this_thread::yield(); + } + try { + cache.upsert_account(value); + } catch (...) { + error = std::current_exception(); + } + }; + + auto account_b = account("account-b"); + account_b.remote_subject = "subject-b"; + std::thread one{write, std::ref(first), account(), std::ref(first_error)}; + std::thread two{write, std::ref(second), account_b, std::ref(second_error)}; + begin.store(true, std::memory_order_release); + one.join(); + two.join(); + + require(first_error == nullptr && second_error == nullptr, + "ordinary concurrent writes leaked SQLITE_BUSY or another error"); + const auto snapshot = first.snapshot(); + require(snapshot.accounts.size() == 2, "serialized connections lost a write"); +} + +void test_input_validation_and_rollback(const std::filesystem::path& root) { + SyncCache cache{root / "validation.db"}; + seed_calendar(cache); + auto account_b = account("account-b"); + account_b.remote_subject = "subject-b"; + cache.upsert_account(account_b); + auto calendar_b = calendar("calendar-b", "account-b"); + calendar_b.remote_id = "remote-calendar-b"; + calendar_b.raw_payload = R"({"calendar":"b"})"; + cache.replace_calendars_after_complete_listing("account-b", std::vector{calendar_b}); + cache.apply_pull_page(page()); + const CacheSnapshot baseline = cache.snapshot(); + + auto expect_unchanged = [&](auto&& operation, std::string_view message) { + require_throws(std::forward(operation), message); + require(cache.snapshot() == baseline, "failed operation changed the cache snapshot"); + }; + + auto empty_account = account(); + empty_account.id.clear(); + expect_unchanged([&] { cache.upsert_account(empty_account); }, "empty account ID accepted"); + + auto reassigned = account(); + reassigned.remote_subject = "different-subject"; + expect_unchanged([&] { cache.upsert_account(reassigned); }, + "account local ID identity reassignment accepted"); + + auto provider_reassigned = account(); + provider_reassigned.provider = "different-provider"; + expect_unchanged([&] { cache.upsert_account(provider_reassigned); }, + "account provider identity reassignment accepted"); + + auto duplicate_calendar = calendar(); + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing( + "account-a", std::vector{duplicate_calendar, duplicate_calendar}); + }, + "duplicate calendar ID accepted"); + + auto duplicate_remote_calendar = calendar("calendar-new"); + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing("account-a", + std::vector{duplicate_calendar, duplicate_remote_calendar}); + }, + "duplicate calendar remote ID accepted"); + + auto empty_calendar = calendar(); + empty_calendar.id.clear(); + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing( + "account-a", std::vector{empty_calendar}); + }, + "empty calendar ID accepted"); + + auto wrong_account = calendar(); + wrong_account.account_id = "account-b"; + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing( + "account-a", std::vector{wrong_account}); + }, + "cross-account calendar accepted"); + + auto reused_calendar_id = calendar("calendar-a", "account-b"); + reused_calendar_id.remote_id = "different-remote-calendar"; + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing( + "account-b", std::vector{reused_calendar_id}); + }, + "calendar local ID was reassigned across accounts"); + + auto inactive_input = calendar(); + inactive_input.active = false; + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing( + "account-a", std::vector{inactive_input}); + }, + "inactive completed-listing input accepted"); + + auto missing_raw_calendar = calendar(); + missing_raw_calendar.raw_payload.clear(); + expect_unchanged( + [&] { + cache.replace_calendars_after_complete_listing( + "account-a", std::vector{missing_raw_calendar}); + }, + "calendar without raw payload accepted"); + + auto duplicate_event = event(); + expect_unchanged( + [&] { + cache.apply_pull_page(PullPage{{duplicate_event, duplicate_event}, {}, + checkpoint("duplicate-cursor")}); + }, + "duplicate event ID accepted"); + + auto duplicate_remote_event = event("event-new"); + expect_unchanged( + [&] { + cache.apply_pull_page(PullPage{{duplicate_event, duplicate_remote_event}, {}, + checkpoint("duplicate-remote-event")}); + }, + "duplicate event remote ID accepted"); + + auto empty_event = event(); + empty_event.id.clear(); + expect_unchanged( + [&] { cache.apply_pull_page(page(empty_event, instance(), checkpoint("empty-event"))); }, + "empty event ID accepted"); + + auto cross_calendar_event = event(); + cross_calendar_event.calendar_id = "calendar-b"; + expect_unchanged( + [&] { + cache.apply_pull_page( + PullPage{{cross_calendar_event}, {}, checkpoint("cross-calendar")}); + }, + "cross-calendar event accepted"); + + auto reassigned_event = event("event-a", "calendar-b"); + reassigned_event.remote_id = "different-remote-event"; + expect_unchanged( + [&] { + cache.apply_pull_page(PullPage{{reassigned_event}, {}, + checkpoint("reassigned-event", "calendar-b")}); + }, + "event local ID was reassigned across calendars"); + + auto missing_remote = event(); + missing_remote.remote_id.clear(); + expect_unchanged( + [&] { cache.apply_pull_page(page(missing_remote, instance(), checkpoint("remote"))); }, + "event without remote ID accepted"); + + auto missing_raw_event = event(); + missing_raw_event.raw_payload.clear(); + expect_unchanged( + [&] { cache.apply_pull_page(page(missing_raw_event, instance(), checkpoint("raw"))); }, + "nondeleted event without raw payload accepted"); + + auto duplicate_instance = instance(); + expect_unchanged( + [&] { + cache.apply_pull_page(PullPage{{event()}, {duplicate_instance, duplicate_instance}, + checkpoint("duplicate-instance")}); + }, + "duplicate instance ID accepted"); + + auto empty_instance = instance(); + empty_instance.id.clear(); + expect_unchanged( + [&] { cache.apply_pull_page(page(event(), empty_instance, checkpoint("empty-instance"))); }, + "empty instance ID accepted"); + + auto orphan = instance(); + orphan.event_id = "event-not-in-page"; + expect_unchanged( + [&] { cache.apply_pull_page(page(event(), orphan, checkpoint("orphan"))); }, + "orphan instance accepted"); + + auto backwards = instance(); + backwards.end_epoch_microseconds = backwards.start_epoch_microseconds - 1; + expect_unchanged( + [&] { cache.apply_pull_page(page(event(), backwards, checkpoint("backwards"))); }, + "backwards instance interval accepted"); + + auto empty_all_day = instance(); + empty_all_day.all_day = true; + empty_all_day.end_epoch_microseconds = empty_all_day.start_epoch_microseconds; + expect_unchanged( + [&] { cache.apply_pull_page(page(event(), empty_all_day, checkpoint("all-day"))); }, + "empty all-day interval accepted"); + + auto deleted_with_instance = event(); + deleted_with_instance.deleted = true; + deleted_with_instance.raw_payload.clear(); + expect_unchanged( + [&] { + cache.apply_pull_page( + page(deleted_with_instance, instance(), checkpoint("deleted-instance"))); + }, + "instance attached to tombstoned event accepted"); + + auto negative = event(); + negative.id = "event-negative"; + negative.remote_id = "remote-negative"; + negative.uid = "uid-negative"; + auto before_epoch = instance("instance-negative", "event-negative"); + before_epoch.start_epoch_microseconds = -2'000'000; + before_epoch.end_epoch_microseconds = -1'000'000; + cache.apply_pull_page(page(negative, before_epoch, checkpoint("negative"))); + require(std::ranges::any_of(cache.snapshot().instances, + [](const CachedEventInstance& value) { + return value.id == "instance-negative" + && value.start_epoch_microseconds < 0; + }), + "valid pre-1970 instance was rejected"); +} + +void test_sql_failure_rolls_back_snapshot_and_cursor(const std::filesystem::path& root) { + SyncCache cache{root / "sql-rollback.db"}; + seed_calendar(cache); + auto event_b = event("event-b"); + event_b.remote_id = "remote-event-b"; + event_b.uid = "uid-b"; + event_b.raw_payload = R"({"event":"b"})"; + auto instance_b = instance("instance-b", "event-b"); + cache.apply_pull_page( + PullPage{{event(), event_b}, {instance(), instance_b}, checkpoint("baseline-cursor")}); + const auto baseline = cache.snapshot(); + + auto changed = event(); + changed.etag = "must-roll-back"; + changed.raw_payload = R"({"event":"must-roll-back"})"; + auto colliding_instance = instance("instance-b", "event-a"); + require_throws( + [&] { + cache.apply_pull_page( + page(changed, colliding_instance, checkpoint("must-not-advance"))); + }, + "cross-event instance ID collision was accepted"); + require(cache.snapshot() == baseline, + "SQL failure did not roll back event, instance, and cursor state together"); +} + +void test_checkpoint_calendar_and_shape_validation(const std::filesystem::path& root) { + const auto path = root / "checkpoint-validation.db"; + SyncCache cache{path}; + seed_calendar(cache); + + auto expect_empty = [&](auto&& operation, std::string_view message) { + require_throws(std::forward(operation), message); + require(cache.snapshot().events.empty() && cache.snapshot().checkpoints.empty(), + "invalid pull partially changed event or checkpoint state"); + }; + + expect_empty( + [&] { cache.apply_pull_page(page(event(), instance(), checkpoint("", "calendar-a"))); }, + "empty checkpoint cursor accepted"); + expect_empty( + [&] { + cache.apply_pull_page(page(event(), instance(), + checkpoint("cursor", "calendar-a", "only-start", ""))); + }, + "half-specified checkpoint window accepted"); + expect_empty( + [&] { + cache.apply_pull_page(page(event(), instance(), + checkpoint("cursor", "calendar-a", "", "only-end"))); + }, + "half-specified checkpoint window accepted"); + expect_empty( + [&] { + cache.apply_pull_page( + page(event(), instance(), checkpoint("cursor", "missing-calendar"))); + }, + "missing checkpoint calendar accepted"); + + cache.apply_pull_page(PullPage{{}, {}, checkpoint("empty-window", "calendar-a", "", "")}); + require(cache.snapshot().checkpoints.size() == 1 + && cache.snapshot().checkpoints[0].window_start.empty() + && cache.snapshot().checkpoints[0].window_end.empty(), + "valid unwindowed opaque checkpoint was rejected"); + + cache.replace_calendars_after_complete_listing("account-a", {}); + require(!cache.snapshot().calendars[0].active, "fixture calendar was not deactivated"); + require_throws( + [&] { cache.apply_pull_page(page()); }, "inactive checkpoint calendar accepted"); + require(cache.snapshot().events.empty(), "inactive-calendar pull changed event state"); +} + +void test_foreign_key_failure_rolls_back(const std::filesystem::path& root) { + const auto path = root / "foreign-key.db"; + SyncCache cache{path}; + seed_calendar(cache); + cache.apply_pull_page(page()); + const auto baseline = cache.snapshot(); + + SqliteConnection external{path}; + external.execute("PRAGMA foreign_keys=ON"); + require_throws( + [&] { external.execute("DELETE FROM accounts WHERE id='account-a'"); }, + "foreign-key violating direct delete succeeded"); + require(cache.snapshot() == baseline, "foreign-key failure changed cached data"); +} + +void test_migration_failures_are_non_destructive(const std::filesystem::path& root) { + const auto collision_path = root / "collision.db"; + { + SqliteConnection database{collision_path}; + database.execute("CREATE TABLE accounts (sentinel TEXT NOT NULL)"); + database.execute("INSERT INTO accounts VALUES ('keep-me')"); + database.execute("PRAGMA user_version=0"); + } + require_throws([&] { SyncCache cache{collision_path}; }, + "incompatible migration collision was accepted"); + { + SqliteConnection database{collision_path}; + require(database.scalar_int("PRAGMA user_version") == 0, + "failed migration changed schema version"); + require(database.scalar_text("SELECT sentinel FROM accounts") == "keep-me", + "failed migration changed preexisting data"); + require(database.scalar_int( + "SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name!='accounts'") + == 0, + "failed migration left partially created v1 tables"); + } + + const auto future_path = root / "future.db"; + { + SqliteConnection database{future_path}; + database.execute("CREATE TABLE future_sentinel (value TEXT NOT NULL)"); + database.execute("INSERT INTO future_sentinel VALUES ('future-data')"); + database.execute("PRAGMA user_version=2"); + } + require_throws([&] { SyncCache cache{future_path}; }, "future schema version was accepted"); + { + SqliteConnection database{future_path}; + require(database.scalar_int("PRAGMA user_version") == 2, + "future schema rejection changed schema version"); + require(database.scalar_text("SELECT value FROM future_sentinel") == "future-data", + "future schema rejection changed data"); + require(database.scalar_int("SELECT COUNT(*) FROM sqlite_master WHERE type='table'") == 1, + "future schema rejection created v1 tables"); + } +} + +void test_invalid_files_are_refused(const std::filesystem::path& root) { + const auto corrupt_path = root / "corrupt.db"; + { + std::ofstream output(corrupt_path, std::ios::binary); + output << "this is not a SQLite database"; + require(static_cast(output), "failed to write corrupt database fixture"); + } + const std::string before = [&] { + std::ifstream input(corrupt_path, std::ios::binary); + return std::string{std::istreambuf_iterator{input}, {}}; + }(); + require_throws([&] { SyncCache cache{corrupt_path}; }, "non-SQLite cache file was accepted"); + std::ifstream after_input(corrupt_path, std::ios::binary); + const std::string after{std::istreambuf_iterator{after_input}, {}}; + require(after == before, "corrupt cache rejection changed the source file"); + +#if !defined(_WIN32) + const auto target_path = root / "symlink-target.db"; + { + SqliteConnection target{target_path}; + } + const auto symlink_path = root / "symlink.db"; + require(::symlink(target_path.c_str(), symlink_path.c_str()) == 0, + "failed to create symlink fixture"); + require_throws([&] { SyncCache cache{symlink_path}; }, "symlink cache path was accepted"); + + const auto directory_path = root / "directory.db"; + std::filesystem::create_directory(directory_path); + require_throws([&] { SyncCache cache{directory_path}; }, + "nonregular cache path was accepted"); +#endif +} + +} // namespace + +int main() { + try { + TempDirectory temporary; + test_fresh_schema_permissions_and_secrets(temporary.path()); + test_existing_permissions_are_tightened(temporary.path()); + test_account_calendars_and_deterministic_snapshot(temporary.path()); + test_pull_pages_replacement_tombstone_and_windows(temporary.path()); + test_multiple_connections_serialize(temporary.path()); + test_input_validation_and_rollback(temporary.path()); + test_sql_failure_rolls_back_snapshot_and_cursor(temporary.path()); + test_checkpoint_calendar_and_shape_validation(temporary.path()); + test_foreign_key_failure_rolls_back(temporary.path()); + test_migration_failures_are_non_destructive(temporary.path()); + test_invalid_files_are_refused(temporary.path()); + } catch (const std::exception& error) { + std::cerr << "sync cache test failure: " << error.what() << '\n'; + return 1; + } + std::cout << "sync cache tests passed\n"; + return 0; +}