bernardo/nocal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add durable provider cache

Browse Source 
Add a private provider-neutral SQLite WAL cache with transactional migrations, retained raw payloads and tombstones, per-window opaque checkpoints, and reserved outbox/conflict tables.

Verify migrations, permissions, identity invariants, concurrent access, atomic pull-page rollback, and future-schema rejection with deterministic tests. No OAuth, networking, secrets, or Graph synchronization is included yet.
This commit is contained in:
Bernardo Magri
2026-07-18 10:53:41 +01:00
parent cfca6ab6d0
commit 787daf00dd
9 changed files with 1794 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
README.md
View File

@@ -7,7 +7,9 @@ inherits its colors from your terminal theme.
This repository currently contains the first local-calendar vertical slice:
date/domain logic, guarded atomic iCalendar persistence, add/edit/delete forms,
a responsive ANSI terminal UI with month and agenda views, explicit safe
calendar import/export, tests, and Linux launch integration. See
calendar import/export, tests, and Linux launch integration. A provider-neutral
SQLite cache now provides the durable boundary required before
Office 365 or any other remote provider is connected. See
[the product specification](docs/PRODUCT.md),
[architecture](docs/ARCHITECTURE.md), and [roadmap](docs/ROADMAP.md).
@@ -105,3 +107,22 @@ backup itself is kept so recovery can be repeated.
For a popup calendar binding and version-specific Hyprland rules, see
[Hyprland integration](docs/HYPRLAND.md).
## Remote-provider foundation
The remote-provider foundation is a private local SQLite database in WAL mode,
separate from the interactive `.ics` file workflow. Its schema is migrated by
version in all-or-nothing transactions. It preserves raw provider payloads,
inactive calendars, and tombstoned events so synchronization cannot silently
discard data that the current application does not understand. Each opaque
provider cursor is scoped to a calendar and synchronization window and is
committed atomically with the pulled page it represents. Normalized event
instances provide immutable snapshot input, while transactional outbox and
conflict records are reserved for later write synchronization.
This is a persistence prerequisite, not working Office 365 integration. Nocal
does not yet perform OAuth, make Graph requests, or synchronize a remote
calendar. OAuth tokens and other secrets will never be stored in SQLite; they
belong in the Freedesktop Secret Service. The next implementation step is an
injectable HTTP boundary with PKCE authentication and deterministic fake
transports, followed by read-only Microsoft Graph delta synchronization.

35
docs/ARCHITECTURE.md
View File

@@ -6,7 +6,7 @@ and terminal rendering replaceable.
```text
src/main.cpp
├── tui/ ANSI rendering, input decoding, terminal lifecycle
├── storage/ iCalendar file adapter (later: SQLite cache)
├── storage/ iCalendar file adapter and provider-neutral cache
└── domain/ Event, Calendar, civil dates, month layout and queries
future sync workers
@@ -118,13 +118,27 @@ semantically. Browsing remains available for unsupported files. This
conservative boundary is more important than partial editing because a
successful-looking edit must not erase unrelated calendar data.
Provider synchronization will not write directly into this UI file. It will use
a transaction-capable local cache and preserve remote ETags/sync tokens in a
provider metadata table.
Provider synchronization does not write directly into this UI file. Its
prerequisite durable boundary is a provider-neutral SQLite database in WAL mode.
The database is a private local file, and schema changes run as versioned,
all-or-nothing migrations so a failed upgrade cannot leave a partially migrated
cache.
The planned cache schema has `calendars`, `events`, `event_instances`, and
`sync_state`. Raw provider payloads are retained alongside normalized fields so
an older Nocal cannot destroy fields it does not understand.
The cache separates provider records from normalized projections. It retains
raw provider payloads alongside normalized fields so an older Nocal cannot
destroy fields it does not understand. Inactive calendars and tombstoned events
remain recorded rather than disappearing from history. Normalized event
instances are materialized for immutable UI snapshots; provider payload details
do not cross into rendering or the domain model.
Incremental progress is scoped by provider account, calendar, and synchronization
window. A provider's opaque cursor is committed in the same transaction as each
pulled page, preventing a crash from advancing the cursor beyond durable data.
Transactional outbox and conflict records are reserved in the boundary for
future local writes and deterministic conflict handling. This cache does not
itself implement networking, authentication, or Microsoft Graph synchronization.
OAuth tokens and other secrets are explicitly excluded from SQLite and belong
in the Freedesktop Secret Service.
## Terminal backend
@@ -149,3 +163,10 @@ Service, never in the calendar database or command line.
Conflict resolution is deterministic: unchanged side wins; otherwise retain
both versions and mark a conflict for the user. Network work happens outside the
render loop and publishes immutable snapshots to it.
The first provider-facing step after the cache is verified is an injectable HTTP
boundary, browser-based authorization-code flow with PKCE, and fake transports
for deterministic authentication and failure tests. Read-only Microsoft Graph
calendar discovery and delta synchronization follows. Graph delta links remain
opaque and are persisted only through the calendar/window transaction described
above; remote writes and conflict resolution come later.

23
docs/PRODUCT.md
View File

@@ -112,6 +112,29 @@ values incompatible with the event's `DTSTART` time basis or `TZID`, detached
read-only. Remaining local-data work includes detached overrides and rule
editing.
## Remote calendar foundation
Remote calendars must cross a durable local boundary before they are exposed to
the UI. That boundary is a private, provider-neutral SQLite cache in WAL mode,
with versioned migrations that either commit completely or leave the prior
schema intact. It keeps raw provider payloads, inactive calendars, and
tombstoned events so refreshes and older clients do not erase information. Each
opaque incremental cursor belongs to a particular calendar and time window and
is committed atomically with every pulled page. Normalized event instances feed
immutable snapshots rather than exposing provider response types to the month
view.
The cache also reserves transactional outbox and conflict records for future
offline writes, but remote mutation behavior is not part of this phase. OAuth
tokens and other credentials never belong in the calendar database; account
secrets will be held by the Freedesktop Secret Service.
This foundation does not make Office 365 usable yet: Nocal currently performs no
OAuth flow, HTTP request, or Microsoft Graph synchronization. The next product
slice introduces injectable HTTP and PKCE authentication with fake transports,
then adds read-only Graph delta synchronization. Remote writes remain gated on
the durability and conflict paths being exercised end to end.
## Local file interchange
Import and export are explicit command-line workflows. They never contact the

28
docs/ROADMAP.md
View File

@@ -24,12 +24,22 @@
- Embedded VTIMEZONE support
- Screen-reader-friendly linear view and full Unicode display-width handling
## 0.2 — durable cache and provider boundary
## 0.2 — durable cache and provider boundary — in progress
- SQLite WAL cache and migration framework
- Background job queue, immutable UI snapshots, observable sync status
- Secret Service credentials and OAuth callback helper
- Generic `SyncProvider` contract plus a fake provider test suite
Completed cache foundation:
- Provider-neutral SQLite WAL cache in a private local file
- Versioned, all-or-nothing schema migrations
- Raw provider payload retention, inactive calendars, and event tombstones
- Per-calendar/window opaque cursors committed atomically with pulled pages
- Normalized event instances for immutable snapshots
- Reserved transactional outbox and conflict records for later remote writes
Next provider-boundary slices:
- Injectable HTTP, PKCE authentication, and deterministic fake transports
- Secret Service credentials; OAuth secrets are never stored in SQLite
- Generic `SyncProvider` contract and observable sync status
## 0.3 — CalDAV
@@ -40,9 +50,15 @@
## 0.4 — hosted providers
- Google Calendar adapter with incremental synchronization
- Microsoft 365 adapter over Microsoft Graph delta queries
- Microsoft 365 read-only discovery and Graph delta synchronization, followed by
ETag-aware remote writes after the conflict boundary is proven
- Account/calendar management UI and per-calendar ANSI identity
The current cache work is only the prerequisite durable boundary. No network,
OAuth, or Graph synchronization is implemented in that phase. After the cache
passes its verification gates, the immediate sequence is injectable HTTP plus
PKCE and fake transports, then read-only Microsoft Graph delta synchronization.
## 1.0 — distribution quality
- Stable configuration/data format and documented recovery procedures

122
include/nocal/storage/sync_cache.hpp Normal file
View File

@@ -0,0 +1,122 @@
#pragma once
#include <cstdint>
#include <filesystem>
#include <memory>
#include <span>
#include <string>
#include <vector>
namespace nocal::storage {
struct CachedAccount {
std::string id;
std::string provider;
std::string remote_subject;
std::string display_name;
bool operator==(const CachedAccount&) const = default;
};
struct CachedCalendar {
std::string id;
std::string account_id;
std::string remote_id;
std::string name;
std::string color;
bool read_only{false};
bool active{true};
std::string raw_payload;
bool operator==(const CachedCalendar&) const = default;
};
struct CachedEvent {
std::string id;
std::string calendar_id;
std::string remote_id;
std::string uid;
std::string etag;
std::string change_key;
std::string raw_payload;
bool deleted{false};
bool operator==(const CachedEvent&) const = default;
};
struct CachedEventInstance {
std::string id;
std::string event_id;
std::int64_t start_epoch_microseconds{0};
std::int64_t end_epoch_microseconds{0};
bool all_day{false};
std::string title;
std::string location;
std::string description;
std::string time_zone;
bool operator==(const CachedEventInstance&) const = default;
};
struct SyncCheckpoint {
std::string calendar_id;
std::string window_start;
std::string window_end;
std::string cursor;
bool complete{false};
bool operator==(const SyncCheckpoint&) const = default;
};
struct PullPage {
std::vector<CachedEvent> events;
std::vector<CachedEventInstance> instances;
SyncCheckpoint checkpoint;
bool operator==(const PullPage&) const = default;
};
struct CacheSnapshot {
std::vector<CachedAccount> accounts;
std::vector<CachedCalendar> calendars;
std::vector<CachedEvent> events;
std::vector<CachedEventInstance> instances;
std::vector<SyncCheckpoint> checkpoints;
bool operator==(const CacheSnapshot&) const = default;
};
class SyncCache {
public:
static constexpr int current_schema_version = 1;
explicit SyncCache(std::filesystem::path path);
~SyncCache();
SyncCache(const SyncCache&) = delete;
SyncCache& operator=(const SyncCache&) = delete;
SyncCache(SyncCache&&) noexcept;
SyncCache& operator=(SyncCache&&) noexcept;
[[nodiscard]] const std::filesystem::path& path() const noexcept;
void upsert_account(const CachedAccount& account);
// Call only after a provider calendar listing has completed. Calendars
// omitted from the completed listing are retained but marked inactive.
void replace_calendars_after_complete_listing(
std::string account_id, std::span<const CachedCalendar> calendars);
// Applies one provider page and advances its opaque cursor in the same
// transaction. Instances for each changed event are replaced; tombstoned
// events retain their last raw payload and have no instances.
void apply_pull_page(const PullPage& page);
[[nodiscard]] CacheSnapshot snapshot() const;
private:
struct Impl;
std::unique_ptr<Impl> impl_;
};
} // namespace nocal::storage

28
meson.build
View File

@@ -12,40 +12,48 @@ project(
)
inc = include_directories('include')
sqlite = dependency('sqlite3')
threads = dependency('threads')
nocal_sources = files(
'src/domain/calendar_transfer.cpp',
'src/domain/date.cpp',
'src/domain/event_query.cpp',
'src/domain/month_layout.cpp',
'src/storage/ics_store.cpp',
'src/storage/sync_cache.cpp',
'src/tui/Screen.cpp',
'src/tui/EventEditor.cpp',
'src/tui/Terminal.cpp',
'src/tui/TuiApp.cpp',
)
nocal_lib = static_library('nocal-core', nocal_sources, include_directories: inc)
nocal_lib = static_library('nocal-core', nocal_sources, include_directories: inc,
dependencies: sqlite)
nocal_dep = declare_dependency(include_directories: inc, link_with: nocal_lib,
dependencies: sqlite)
nocal_executable = executable('nocal', 'src/main.cpp', include_directories: inc,
link_with: nocal_lib, install: true)
nocal_executable = executable('nocal', 'src/main.cpp',
dependencies: nocal_dep, install: true)
domain_tests = executable('domain-tests', 'tests/domain_tests.cpp',
include_directories: inc, link_with: nocal_lib)
dependencies: nocal_dep)
test('domain', domain_tests)
calendar_transfer_tests = executable('calendar-transfer-tests',
'tests/calendar_transfer_tests.cpp', include_directories: inc,
link_with: nocal_lib)
'tests/calendar_transfer_tests.cpp', dependencies: nocal_dep)
test('calendar-transfer', calendar_transfer_tests)
ics_tests = executable('ics-tests', 'tests/ics_tests.cpp',
include_directories: inc, link_with: nocal_lib)
dependencies: nocal_dep)
test('ics-storage', ics_tests)
sync_cache_tests = executable('sync-cache-tests', 'tests/sync_cache_tests.cpp',
dependencies: [nocal_dep, threads])
test('sync-cache', sync_cache_tests)
tui_tests = executable('tui-tests', 'tests/tui_tests.cpp',
include_directories: inc, link_with: nocal_lib)
dependencies: nocal_dep)
test('tui', tui_tests)
tui_focus_tests = executable('tui-focus-tests', 'tests/tui_focus_tests.cpp',
include_directories: inc, link_with: nocal_lib)
dependencies: nocal_dep)
test('tui-focus', tui_focus_tests)
editor_tests = executable('editor-tests', 'tests/editor_tests.cpp',
include_directories: inc, link_with: nocal_lib)
dependencies: nocal_dep)
test('editor', editor_tests)
shell = find_program('sh')

1
shell.nix
View File

@@ -9,5 +9,6 @@ pkgs.mkShell {
buildInputs = with pkgs; [
gcc
sqlite
];
}

738
src/storage/sync_cache.cpp Normal file
View File

@@ -0,0 +1,738 @@
#include "nocal/storage/sync_cache.hpp"
#include <sqlite3.h>
#include <cerrno>
#include <cstring>
#include <filesystem>
#include <limits>
#include <mutex>
#include <stdexcept>
#include <string>
#include <string_view>
#include <unordered_set>
#include <utility>
#include <fcntl.h>
#include <sys/stat.h>
#include <unistd.h>
namespace nocal::storage {
namespace {
[[noreturn]] void throw_sqlite(sqlite3* database, std::string_view operation) {
const char* detail = database == nullptr ? nullptr : sqlite3_errmsg(database);
throw std::runtime_error(
"sync cache " + std::string(operation) + " failed"
+ (detail == nullptr ? std::string{} : ": " + std::string(detail)));
}
void execute(sqlite3* database, const char* sql, std::string_view operation) {
char* detail = nullptr;
const int result = sqlite3_exec(database, sql, nullptr, nullptr, &detail);
if (result == SQLITE_OK) {
return;
}
const std::string message = detail == nullptr ? sqlite3_errmsg(database) : detail;
sqlite3_free(detail);
throw std::runtime_error(
"sync cache " + std::string(operation) + " failed: " + message);
}
class Statement {
public:
Statement(sqlite3* database, const char* sql, std::string_view operation)
: database_(database), operation_(operation) {
if (sqlite3_prepare_v2(database_, sql, -1, &statement_, nullptr) != SQLITE_OK) {
throw_sqlite(database_, operation_);
}
}
~Statement() { sqlite3_finalize(statement_); }
Statement(const Statement&) = delete;
Statement& operator=(const Statement&) = delete;
void bind_text(int index, std::string_view value) {
if (value.size() > static_cast<std::size_t>(std::numeric_limits<int>::max())
|| sqlite3_bind_text(statement_, index, value.data(), static_cast<int>(value.size()),
SQLITE_TRANSIENT)
!= SQLITE_OK) {
throw_sqlite(database_, operation_);
}
}
void bind_int64(int index, std::int64_t value) {
if (sqlite3_bind_int64(statement_, index, value) != SQLITE_OK) {
throw_sqlite(database_, operation_);
}
}
void bind_bool(int index, bool value) { bind_int64(index, value ? 1 : 0); }
[[nodiscard]] bool next() {
const int result = sqlite3_step(statement_);
if (result == SQLITE_ROW) {
return true;
}
if (result == SQLITE_DONE) {
return false;
}
throw_sqlite(database_, operation_);
}
void run() {
if (next()) {
throw std::runtime_error(
"sync cache " + std::string(operation_) + " returned an unexpected row");
}
}
[[nodiscard]] std::string text_column(int index) const {
const unsigned char* value = sqlite3_column_text(statement_, index);
const int size = sqlite3_column_bytes(statement_, index);
if (value == nullptr || size == 0) {
return {};
}
return {reinterpret_cast<const char*>(value), static_cast<std::size_t>(size)};
}
[[nodiscard]] std::int64_t int64_column(int index) const {
return sqlite3_column_int64(statement_, index);
}
[[nodiscard]] bool bool_column(int index) const {
return sqlite3_column_int(statement_, index) != 0;
}
private:
sqlite3* database_;
sqlite3_stmt* statement_{nullptr};
std::string operation_;
};
class Transaction {
public:
explicit Transaction(sqlite3* database) : database_(database) {
execute(database_, "BEGIN IMMEDIATE", "starting transaction");
}
~Transaction() {
if (!finished_) {
sqlite3_exec(database_, "ROLLBACK", nullptr, nullptr, nullptr);
}
}
Transaction(const Transaction&) = delete;
Transaction& operator=(const Transaction&) = delete;
void commit() {
execute(database_, "COMMIT", "committing transaction");
finished_ = true;
}
private:
sqlite3* database_;
bool finished_{false};
};
[[nodiscard]] std::string path_error(const std::filesystem::path& path, std::string_view reason) {
return "invalid sync cache path '" + path.string() + "': " + std::string(reason);
}
[[nodiscard]] bool prepare_database_file(const std::filesystem::path& path) {
if (path.empty() || !path.has_filename() || path.filename().empty()
|| path.filename() == "." || path.filename() == ".."
|| path.native().find('\0') != std::string::npos) {
throw std::invalid_argument(path_error(path, "a database filename is required"));
}
struct stat link_status {};
bool created = false;
int descriptor = -1;
if (::lstat(path.c_str(), &link_status) == 0) {
if (S_ISLNK(link_status.st_mode) || !S_ISREG(link_status.st_mode)) {
throw std::invalid_argument(path_error(path, "existing target must be a regular file"));
}
descriptor = ::open(path.c_str(), O_RDWR | O_CLOEXEC | O_NOFOLLOW);
} else if (errno == ENOENT) {
descriptor = ::open(path.c_str(),
O_RDWR | O_CREAT | O_EXCL | O_CLOEXEC | O_NOFOLLOW, S_IRUSR | S_IWUSR);
created = descriptor >= 0;
} else {
throw std::runtime_error(path_error(path, std::strerror(errno)));
}
if (descriptor < 0) {
throw std::runtime_error(path_error(path, std::strerror(errno)));
}
struct stat opened_status {};
if (::fstat(descriptor, &opened_status) != 0) {
const int error = errno;
::close(descriptor);
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::runtime_error(path_error(path, std::strerror(error)));
}
if (!S_ISREG(opened_status.st_mode)) {
::close(descriptor);
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::invalid_argument(path_error(path, "opened target is not a regular file"));
}
if (::fchmod(descriptor, S_IRUSR | S_IWUSR) != 0) {
const int error = errno;
::close(descriptor);
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::runtime_error(path_error(path, std::strerror(error)));
}
if (::fstat(descriptor, &opened_status) != 0) {
const int error = errno;
::close(descriptor);
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::runtime_error(path_error(path, std::strerror(error)));
}
if ((opened_status.st_mode & (S_IRWXU | S_IRWXG | S_IRWXO)) != (S_IRUSR | S_IWUSR)) {
::close(descriptor);
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::runtime_error(path_error(path, "database permissions are not private"));
}
if (::close(descriptor) != 0) {
const int error = errno;
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::runtime_error(path_error(path, std::strerror(error)));
}
return created;
}
[[nodiscard]] int user_version(sqlite3* database) {
Statement statement(database, "PRAGMA user_version", "reading schema version");
if (!statement.next()) {
throw std::runtime_error("sync cache schema version is unavailable");
}
const std::int64_t version = statement.int64_column(0);
if (version < 0 || version > std::numeric_limits<int>::max()) {
throw std::runtime_error("sync cache schema version is invalid");
}
return static_cast<int>(version);
}
void require_pragma_integer(
sqlite3* database, const char* sql, std::int64_t expected, std::string_view operation) {
Statement statement(database, sql, operation);
if (!statement.next() || statement.int64_column(0) != expected) {
throw std::runtime_error("sync cache " + std::string(operation) + " was not applied");
}
}
void require_pragma_text(
sqlite3* database, const char* sql, std::string_view expected, std::string_view operation) {
Statement statement(database, sql, operation);
if (!statement.next() || statement.text_column(0) != expected) {
throw std::runtime_error("sync cache " + std::string(operation) + " was not applied");
}
}
constexpr const char* schema_v1 = R"sql(
CREATE TABLE accounts (
id TEXT PRIMARY KEY CHECK(length(id) > 0),
provider TEXT NOT NULL CHECK(length(provider) > 0),
remote_subject TEXT NOT NULL CHECK(length(remote_subject) > 0),
display_name TEXT NOT NULL,
UNIQUE(provider, remote_subject)
) STRICT;
CREATE TABLE calendars (
id TEXT PRIMARY KEY CHECK(length(id) > 0),
account_id TEXT NOT NULL REFERENCES accounts(id) ON UPDATE RESTRICT ON DELETE RESTRICT,
remote_id TEXT NOT NULL CHECK(length(remote_id) > 0),
name TEXT NOT NULL,
color TEXT NOT NULL,
read_only INTEGER NOT NULL CHECK(read_only IN (0, 1)),
active INTEGER NOT NULL CHECK(active IN (0, 1)),
raw_payload TEXT NOT NULL,
UNIQUE(account_id, remote_id)
) STRICT;
CREATE INDEX calendars_account_id_idx ON calendars(account_id);
CREATE TABLE events (
id TEXT PRIMARY KEY CHECK(length(id) > 0),
calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE RESTRICT,
remote_id TEXT NOT NULL CHECK(length(remote_id) > 0),
uid TEXT NOT NULL,
etag TEXT NOT NULL,
change_key TEXT NOT NULL,
raw_payload TEXT NOT NULL,
deleted INTEGER NOT NULL CHECK(deleted IN (0, 1)),
UNIQUE(calendar_id, remote_id)
) STRICT;
CREATE INDEX events_calendar_id_idx ON events(calendar_id);
CREATE TABLE event_instances (
id TEXT PRIMARY KEY CHECK(length(id) > 0),
event_id TEXT NOT NULL REFERENCES events(id) ON UPDATE RESTRICT ON DELETE CASCADE,
start_epoch_microseconds INTEGER NOT NULL,
end_epoch_microseconds INTEGER NOT NULL,
all_day INTEGER NOT NULL CHECK(all_day IN (0, 1)),
title TEXT NOT NULL,
location TEXT NOT NULL,
description TEXT NOT NULL,
time_zone TEXT NOT NULL,
CHECK(start_epoch_microseconds <= end_epoch_microseconds),
CHECK(all_day = 0 OR start_epoch_microseconds < end_epoch_microseconds)
) STRICT;
CREATE INDEX event_instances_event_id_idx ON event_instances(event_id);
CREATE TABLE sync_state (
calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE CASCADE,
window_start TEXT NOT NULL,
window_end TEXT NOT NULL,
cursor TEXT NOT NULL CHECK(length(cursor) > 0),
complete INTEGER NOT NULL CHECK(complete IN (0, 1)),
CHECK((length(window_start) = 0) = (length(window_end) = 0)),
PRIMARY KEY(calendar_id, window_start, window_end)
) STRICT;
CREATE TABLE outbox (
id TEXT PRIMARY KEY CHECK(length(id) > 0),
calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE RESTRICT,
event_id TEXT NOT NULL REFERENCES events(id) ON UPDATE RESTRICT ON DELETE RESTRICT,
operation TEXT NOT NULL CHECK(length(operation) > 0),
payload TEXT NOT NULL,
base_etag TEXT NOT NULL,
idempotency_key TEXT NOT NULL UNIQUE CHECK(length(idempotency_key) > 0),
status TEXT NOT NULL CHECK(length(status) > 0),
attempt_count INTEGER NOT NULL DEFAULT 0 CHECK(attempt_count >= 0),
not_before_epoch_seconds INTEGER
) STRICT;
CREATE TABLE conflicts (
id TEXT PRIMARY KEY CHECK(length(id) > 0),
calendar_id TEXT NOT NULL REFERENCES calendars(id) ON UPDATE RESTRICT ON DELETE RESTRICT,
event_id TEXT NOT NULL REFERENCES events(id) ON UPDATE RESTRICT ON DELETE RESTRICT,
base_etag TEXT NOT NULL,
local_payload TEXT NOT NULL,
remote_payload TEXT NOT NULL,
status TEXT NOT NULL CHECK(length(status) > 0)
) STRICT;
)sql";
void validate_nonempty(std::string_view value, std::string_view field) {
if (value.empty()) {
throw std::invalid_argument(std::string(field) + " must not be empty");
}
}
void require_account(sqlite3* database, std::string_view account_id) {
Statement statement(database, "SELECT 1 FROM accounts WHERE id = ?", "checking account");
statement.bind_text(1, account_id);
if (!statement.next()) {
throw std::invalid_argument("sync cache account does not exist");
}
}
void ensure_account_identity(sqlite3* database, const CachedAccount& account) {
Statement statement(database,
"SELECT provider, remote_subject FROM accounts WHERE id = ?", "checking account identity");
statement.bind_text(1, account.id);
if (statement.next()
&& (statement.text_column(0) != account.provider
|| statement.text_column(1) != account.remote_subject)) {
throw std::invalid_argument("account identity cannot be reassigned");
}
}
void require_active_calendar(sqlite3* database, std::string_view calendar_id) {
Statement statement(
database, "SELECT active FROM calendars WHERE id = ?", "checking calendar");
statement.bind_text(1, calendar_id);
if (!statement.next()) {
throw std::invalid_argument("sync cache calendar does not exist");
}
if (!statement.bool_column(0)) {
throw std::invalid_argument("cannot apply a pull page to an inactive calendar");
}
}
void ensure_calendar_identity(sqlite3* database, const CachedCalendar& calendar) {
Statement statement(database,
"SELECT account_id, remote_id FROM calendars WHERE id = ?",
"checking calendar identity");
statement.bind_text(1, calendar.id);
if (statement.next()
&& (statement.text_column(0) != calendar.account_id
|| statement.text_column(1) != calendar.remote_id)) {
throw std::invalid_argument("calendar identity cannot be reassigned");
}
}
void ensure_event_identity(sqlite3* database, const CachedEvent& event) {
Statement statement(database,
"SELECT calendar_id, remote_id FROM events WHERE id = ?", "checking event identity");
statement.bind_text(1, event.id);
if (statement.next()
&& (statement.text_column(0) != event.calendar_id
|| statement.text_column(1) != event.remote_id)) {
throw std::invalid_argument("event identity cannot be reassigned");
}
}
} // namespace
struct SyncCache::Impl {
explicit Impl(std::filesystem::path cache_path) : path(std::move(cache_path)) {
const bool created = prepare_database_file(path);
sqlite3* opened = nullptr;
const int result = sqlite3_open_v2(path.c_str(), &opened,
SQLITE_OPEN_READWRITE | SQLITE_OPEN_FULLMUTEX | SQLITE_OPEN_PRIVATECACHE
| SQLITE_OPEN_NOFOLLOW,
nullptr);
if (result != SQLITE_OK) {
const std::string detail = opened == nullptr ? sqlite3_errstr(result) : sqlite3_errmsg(opened);
if (opened != nullptr) {
sqlite3_close(opened);
}
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
}
throw std::runtime_error("unable to open sync cache '" + path.string() + "': " + detail);
}
database = opened;
try {
if (sqlite3_extended_result_codes(database, 1) != SQLITE_OK
|| sqlite3_busy_timeout(database, 5'000) != SQLITE_OK) {
throw_sqlite(database, "configuring connection error handling");
}
execute(database, "PRAGMA trusted_schema = OFF", "disabling trusted schema");
require_pragma_integer(
database, "PRAGMA trusted_schema", 0, "disabling trusted schema");
#ifdef SQLITE_DBCONFIG_DEFENSIVE
if (sqlite3_db_config(database, SQLITE_DBCONFIG_DEFENSIVE, 1, nullptr) != SQLITE_OK) {
throw_sqlite(database, "enabling defensive mode");
}
#endif
const int version = user_version(database);
if (version > SyncCache::current_schema_version) {
throw std::runtime_error("sync cache schema version " + std::to_string(version)
+ " is newer than supported version "
+ std::to_string(SyncCache::current_schema_version));
}
execute(database, "PRAGMA foreign_keys = ON", "enabling foreign keys");
require_pragma_integer(database, "PRAGMA foreign_keys", 1, "enabling foreign keys");
if (version == 0) {
Transaction migration(database);
const int locked_version = user_version(database);
if (locked_version == 0) {
execute(database, schema_v1, "creating schema version 1");
execute(database, "PRAGMA user_version = 1", "recording schema version 1");
} else if (locked_version > SyncCache::current_schema_version) {
throw std::runtime_error("sync cache schema changed to an unsupported version");
}
migration.commit();
}
execute(database, "PRAGMA journal_mode = WAL", "enabling WAL mode");
require_pragma_text(database, "PRAGMA journal_mode", "wal", "enabling WAL mode");
execute(database, "PRAGMA synchronous = FULL", "setting full synchronization");
require_pragma_integer(
database, "PRAGMA synchronous", 2, "setting full synchronization");
} catch (...) {
sqlite3_close(database);
database = nullptr;
if (created) {
std::error_code ignored;
std::filesystem::remove(path, ignored);
std::filesystem::remove(path.string() + "-wal", ignored);
std::filesystem::remove(path.string() + "-shm", ignored);
}
throw;
}
}
~Impl() {
if (database != nullptr) {
sqlite3_close(database);
}
}
std::filesystem::path path;
sqlite3* database{nullptr};
mutable std::mutex mutex;
};
SyncCache::SyncCache(std::filesystem::path path) : impl_(std::make_unique<Impl>(std::move(path))) {}
SyncCache::~SyncCache() = default;
SyncCache::SyncCache(SyncCache&&) noexcept = default;
SyncCache& SyncCache::operator=(SyncCache&&) noexcept = default;
const std::filesystem::path& SyncCache::path() const noexcept { return impl_->path; }
void SyncCache::upsert_account(const CachedAccount& account) {
validate_nonempty(account.id, "account id");
validate_nonempty(account.provider, "account provider");
validate_nonempty(account.remote_subject, "account remote subject");
std::scoped_lock lock(impl_->mutex);
Transaction transaction(impl_->database);
ensure_account_identity(impl_->database, account);
Statement statement(impl_->database, R"sql(
INSERT INTO accounts(id, provider, remote_subject, display_name) VALUES(?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
display_name = excluded.display_name
)sql", "upserting account");
statement.bind_text(1, account.id);
statement.bind_text(2, account.provider);
statement.bind_text(3, account.remote_subject);
statement.bind_text(4, account.display_name);
statement.run();
transaction.commit();
}
void SyncCache::replace_calendars_after_complete_listing(
std::string account_id, std::span<const CachedCalendar> calendars) {
validate_nonempty(account_id, "account id");
std::unordered_set<std::string> ids;
std::unordered_set<std::string> remote_ids;
for (const CachedCalendar& calendar : calendars) {
validate_nonempty(calendar.id, "calendar id");
validate_nonempty(calendar.remote_id, "calendar remote id");
validate_nonempty(calendar.raw_payload, "calendar raw payload");
if (calendar.account_id != account_id) {
throw std::invalid_argument("calendar belongs to a different account");
}
if (!calendar.active) {
throw std::invalid_argument("a completed calendar listing must contain active calendars");
}
if (!ids.insert(calendar.id).second || !remote_ids.insert(calendar.remote_id).second) {
throw std::invalid_argument("completed calendar listing contains duplicate identities");
}
}
std::scoped_lock lock(impl_->mutex);
Transaction transaction(impl_->database);
require_account(impl_->database, account_id);
for (const CachedCalendar& calendar : calendars) {
ensure_calendar_identity(impl_->database, calendar);
}
{
Statement deactivate(impl_->database,
"UPDATE calendars SET active = 0 WHERE account_id = ?", "deactivating calendars");
deactivate.bind_text(1, account_id);
deactivate.run();
}
for (const CachedCalendar& calendar : calendars) {
Statement upsert(impl_->database, R"sql(
INSERT INTO calendars(
id, account_id, remote_id, name, color, read_only, active, raw_payload)
VALUES(?, ?, ?, ?, ?, ?, 1, ?)
ON CONFLICT(id) DO UPDATE SET
name = excluded.name,
color = excluded.color,
read_only = excluded.read_only,
active = 1,
raw_payload = excluded.raw_payload
)sql", "upserting calendar");
upsert.bind_text(1, calendar.id);
upsert.bind_text(2, calendar.account_id);
upsert.bind_text(3, calendar.remote_id);
upsert.bind_text(4, calendar.name);
upsert.bind_text(5, calendar.color);
upsert.bind_bool(6, calendar.read_only);
upsert.bind_text(7, calendar.raw_payload);
upsert.run();
}
transaction.commit();
}
void SyncCache::apply_pull_page(const PullPage& page) {
const SyncCheckpoint& checkpoint = page.checkpoint;
validate_nonempty(checkpoint.calendar_id, "checkpoint calendar id");
validate_nonempty(checkpoint.cursor, "checkpoint cursor");
if (checkpoint.window_start.empty() != checkpoint.window_end.empty()) {
throw std::invalid_argument("checkpoint window endpoints must both be present or both be empty");
}
std::unordered_set<std::string> event_ids;
std::unordered_set<std::string> event_remote_ids;
std::unordered_set<std::string> live_event_ids;
for (const CachedEvent& event : page.events) {
validate_nonempty(event.id, "event id");
validate_nonempty(event.remote_id, "event remote id");
if (event.calendar_id != checkpoint.calendar_id) {
throw std::invalid_argument("changed event belongs to a different calendar");
}
if (!event.deleted && event.raw_payload.empty()) {
throw std::invalid_argument("a live changed event must retain its raw provider payload");
}
if (!event_ids.insert(event.id).second
|| !event_remote_ids.insert(event.remote_id).second) {
throw std::invalid_argument("pull page contains duplicate event identities");
}
if (!event.deleted) {
live_event_ids.insert(event.id);
}
}
std::unordered_set<std::string> instance_ids;
for (const CachedEventInstance& instance : page.instances) {
validate_nonempty(instance.id, "event instance id");
validate_nonempty(instance.event_id, "event instance event id");
if (instance.start_epoch_microseconds > instance.end_epoch_microseconds
|| (instance.all_day
&& instance.start_epoch_microseconds == instance.end_epoch_microseconds)) {
throw std::invalid_argument("event instance must have a valid half-open interval");
}
if (!instance_ids.insert(instance.id).second) {
throw std::invalid_argument("pull page contains duplicate event instance ids");
}
if (!live_event_ids.contains(instance.event_id)) {
throw std::invalid_argument(
"event instance must refer to a changed, nondeleted event in the same page");
}
}
std::scoped_lock lock(impl_->mutex);
Transaction transaction(impl_->database);
require_active_calendar(impl_->database, checkpoint.calendar_id);
for (const CachedEvent& event : page.events) {
ensure_event_identity(impl_->database, event);
}
for (const CachedEvent& event : page.events) {
Statement upsert(impl_->database, R"sql(
INSERT INTO events(
id, calendar_id, remote_id, uid, etag, change_key, raw_payload, deleted)
VALUES(?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(id) DO UPDATE SET
uid = excluded.uid,
etag = excluded.etag,
change_key = excluded.change_key,
raw_payload = CASE
WHEN excluded.deleted = 1 AND length(excluded.raw_payload) = 0 THEN events.raw_payload
ELSE excluded.raw_payload
END,
deleted = excluded.deleted
)sql", "upserting changed event");
upsert.bind_text(1, event.id);
upsert.bind_text(2, event.calendar_id);
upsert.bind_text(3, event.remote_id);
upsert.bind_text(4, event.uid);
upsert.bind_text(5, event.etag);
upsert.bind_text(6, event.change_key);
upsert.bind_text(7, event.raw_payload);
upsert.bind_bool(8, event.deleted);
upsert.run();
}
for (const CachedEvent& event : page.events) {
Statement remove(impl_->database,
"DELETE FROM event_instances WHERE event_id = ?", "replacing event instances");
remove.bind_text(1, event.id);
remove.run();
}
for (const CachedEventInstance& instance : page.instances) {
Statement insert(impl_->database, R"sql(
INSERT INTO event_instances(
id, event_id, start_epoch_microseconds, end_epoch_microseconds, all_day,
title, location, description, time_zone)
VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?)
)sql", "inserting event instance");
insert.bind_text(1, instance.id);
insert.bind_text(2, instance.event_id);
insert.bind_int64(3, instance.start_epoch_microseconds);
insert.bind_int64(4, instance.end_epoch_microseconds);
insert.bind_bool(5, instance.all_day);
insert.bind_text(6, instance.title);
insert.bind_text(7, instance.location);
insert.bind_text(8, instance.description);
insert.bind_text(9, instance.time_zone);
insert.run();
}
{
Statement upsert(impl_->database, R"sql(
INSERT INTO sync_state(calendar_id, window_start, window_end, cursor, complete)
VALUES(?, ?, ?, ?, ?)
ON CONFLICT(calendar_id, window_start, window_end) DO UPDATE SET
cursor = excluded.cursor,
complete = excluded.complete
)sql", "advancing sync checkpoint");
upsert.bind_text(1, checkpoint.calendar_id);
upsert.bind_text(2, checkpoint.window_start);
upsert.bind_text(3, checkpoint.window_end);
upsert.bind_text(4, checkpoint.cursor);
upsert.bind_bool(5, checkpoint.complete);
upsert.run();
}
transaction.commit();
}
CacheSnapshot SyncCache::snapshot() const {
std::scoped_lock lock(impl_->mutex);
CacheSnapshot result;
{
Statement query(impl_->database,
"SELECT id, provider, remote_subject, display_name FROM accounts ORDER BY id",
"reading accounts");
while (query.next()) {
result.accounts.push_back({query.text_column(0), query.text_column(1),
query.text_column(2), query.text_column(3)});
}
}
{
Statement query(impl_->database, R"sql(
SELECT id, account_id, remote_id, name, color, read_only, active, raw_payload
FROM calendars ORDER BY id
)sql", "reading calendars");
while (query.next()) {
result.calendars.push_back({query.text_column(0), query.text_column(1),
query.text_column(2), query.text_column(3), query.text_column(4),
query.bool_column(5), query.bool_column(6), query.text_column(7)});
}
}
{
Statement query(impl_->database, R"sql(
SELECT id, calendar_id, remote_id, uid, etag, change_key, raw_payload, deleted
FROM events ORDER BY id
)sql", "reading events");
while (query.next()) {
result.events.push_back({query.text_column(0), query.text_column(1),
query.text_column(2), query.text_column(3), query.text_column(4),
query.text_column(5), query.text_column(6), query.bool_column(7)});
}
}
{
Statement query(impl_->database, R"sql(
SELECT id, event_id, start_epoch_microseconds, end_epoch_microseconds,
all_day, title, location, description, time_zone
FROM event_instances ORDER BY id
)sql", "reading event instances");
while (query.next()) {
result.instances.push_back({query.text_column(0), query.text_column(1),
query.int64_column(2), query.int64_column(3), query.bool_column(4),
query.text_column(5), query.text_column(6), query.text_column(7),
query.text_column(8)});
}
}
{
Statement query(impl_->database, R"sql(
SELECT calendar_id, window_start, window_end, cursor, complete
FROM sync_state ORDER BY calendar_id, window_start, window_end
)sql", "reading sync checkpoints");
while (query.next()) {
result.checkpoints.push_back({query.text_column(0), query.text_column(1),
query.text_column(2), query.text_column(3), query.bool_column(4)});
}
}
return result;
}
} // namespace nocal::storage

820
tests/sync_cache_tests.cpp Normal file
View File

@@ -0,0 +1,820 @@
#include "nocal/storage/sync_cache.hpp"
#include <sqlite3.h>
#include <algorithm>
#include <atomic>
#include <cerrno>
#include <exception>
#include <filesystem>
#include <fstream>
#include <functional>
#include <iostream>
#include <iterator>
#include <stdexcept>
#include <string>
#include <string_view>
#include <thread>
#include <utility>
#include <vector>
#if !defined(_WIN32)
#include <sys/stat.h>
#include <unistd.h>
#endif
namespace {
using nocal::storage::CacheSnapshot;
using nocal::storage::CachedAccount;
using nocal::storage::CachedCalendar;
using nocal::storage::CachedEvent;
using nocal::storage::CachedEventInstance;
using nocal::storage::PullPage;
using nocal::storage::SyncCache;
using nocal::storage::SyncCheckpoint;
void require(bool condition, std::string_view message) {
if (!condition) {
throw std::runtime_error(std::string{message});
}
}
template <typename Function>
void require_throws(Function&& function, std::string_view message) {
try {
function();
} catch (const std::exception&) {
return;
}
throw std::runtime_error(std::string{message});
}
class TempDirectory {
public:
TempDirectory() {
#if !defined(_WIN32)
std::string pattern =
(std::filesystem::temp_directory_path() / "nocal-sync-cache-tests.XXXXXX").string();
std::vector<char> writable(pattern.begin(), pattern.end());
writable.push_back('\0');
const char* created = ::mkdtemp(writable.data());
if (created == nullptr) {
throw std::runtime_error("mkdtemp failed");
}
path_ = created;
#else
path_ = std::filesystem::temp_directory_path() / "nocal-sync-cache-tests";
std::filesystem::remove_all(path_);
std::filesystem::create_directories(path_);
#endif
}
TempDirectory(const TempDirectory&) = delete;
TempDirectory& operator=(const TempDirectory&) = delete;
~TempDirectory() {
std::error_code ignored;
std::filesystem::remove_all(path_, ignored);
}
[[nodiscard]] const std::filesystem::path& path() const noexcept { return path_; }
private:
std::filesystem::path path_;
};
class SqliteConnection {
public:
explicit SqliteConnection(const std::filesystem::path& path) {
const int result = sqlite3_open_v2(
path.c_str(), &database_, SQLITE_OPEN_READWRITE | SQLITE_OPEN_CREATE, nullptr);
if (result != SQLITE_OK) {
const std::string message = database_ == nullptr ? "sqlite open failed"
: sqlite3_errmsg(database_);
if (database_ != nullptr) {
sqlite3_close(database_);
database_ = nullptr;
}
throw std::runtime_error(message);
}
}
SqliteConnection(const SqliteConnection&) = delete;
SqliteConnection& operator=(const SqliteConnection&) = delete;
~SqliteConnection() {
if (database_ != nullptr) {
sqlite3_close(database_);
}
}
void execute(std::string_view sql) {
char* error = nullptr;
const std::string statement{sql};
const int result = sqlite3_exec(database_, statement.c_str(), nullptr, nullptr, &error);
if (result != SQLITE_OK) {
const std::string message = error == nullptr ? sqlite3_errmsg(database_) : error;
sqlite3_free(error);
throw std::runtime_error(message);
}
}
[[nodiscard]] std::string scalar_text(std::string_view sql) const {
sqlite3_stmt* statement = nullptr;
const std::string query{sql};
if (sqlite3_prepare_v2(database_, query.c_str(), -1, &statement, nullptr) != SQLITE_OK) {
throw std::runtime_error(sqlite3_errmsg(database_));
}
struct Finalize {
sqlite3_stmt* statement;
~Finalize() { sqlite3_finalize(statement); }
} finalize{statement};
if (sqlite3_step(statement) != SQLITE_ROW) {
throw std::runtime_error(sqlite3_errmsg(database_));
}
const unsigned char* value = sqlite3_column_text(statement, 0);
return value == nullptr ? std::string{} : reinterpret_cast<const char*>(value);
}
[[nodiscard]] int scalar_int(std::string_view sql) const {
sqlite3_stmt* statement = nullptr;
const std::string query{sql};
if (sqlite3_prepare_v2(database_, query.c_str(), -1, &statement, nullptr) != SQLITE_OK) {
throw std::runtime_error(sqlite3_errmsg(database_));
}
struct Finalize {
sqlite3_stmt* statement;
~Finalize() { sqlite3_finalize(statement); }
} finalize{statement};
if (sqlite3_step(statement) != SQLITE_ROW) {
throw std::runtime_error(sqlite3_errmsg(database_));
}
return sqlite3_column_int(statement, 0);
}
[[nodiscard]] std::vector<std::string> first_column(std::string_view sql) const {
sqlite3_stmt* statement = nullptr;
const std::string query{sql};
if (sqlite3_prepare_v2(database_, query.c_str(), -1, &statement, nullptr) != SQLITE_OK) {
throw std::runtime_error(sqlite3_errmsg(database_));
}
struct Finalize {
sqlite3_stmt* statement;
~Finalize() { sqlite3_finalize(statement); }
} finalize{statement};
std::vector<std::string> values;
int result = SQLITE_ROW;
while ((result = sqlite3_step(statement)) == SQLITE_ROW) {
const unsigned char* value = sqlite3_column_text(statement, 0);
values.emplace_back(value == nullptr ? "" : reinterpret_cast<const char*>(value));
}
if (result != SQLITE_DONE) {
throw std::runtime_error(sqlite3_errmsg(database_));
}
return values;
}
private:
sqlite3* database_{nullptr};
};
CachedAccount account(std::string id = "account-a") {
return CachedAccount{std::move(id), "microsoft-graph", "subject-a", "Work account"};
}
CachedCalendar calendar(
std::string id = "calendar-a", std::string account_id = "account-a") {
return CachedCalendar{std::move(id), std::move(account_id), "remote-calendar-a", "Calendar A",
"blue", false, true, R"({"calendar":"a"})"};
}
CachedEvent event(std::string id = "event-a", std::string calendar_id = "calendar-a") {
return CachedEvent{std::move(id), std::move(calendar_id), "remote-event-a", "uid-a", "etag-a",
"change-a", R"({"event":"a"})", false};
}
CachedEventInstance instance(
std::string id = "instance-a", std::string event_id = "event-a") {
return CachedEventInstance{std::move(id), std::move(event_id), 1'000'000, 2'000'000, false,
"Event A", "Room", "Description", "Europe/London"};
}
SyncCheckpoint checkpoint(std::string cursor = "cursor-a", std::string calendar_id = "calendar-a",
std::string window_start = "opaque-start", std::string window_end = "opaque-end") {
return SyncCheckpoint{std::move(calendar_id), std::move(window_start), std::move(window_end),
std::move(cursor), false};
}
PullPage page(CachedEvent changed_event = event(), CachedEventInstance changed_instance = instance(),
SyncCheckpoint next = checkpoint()) {
return PullPage{{std::move(changed_event)}, {std::move(changed_instance)}, std::move(next)};
}
void seed_calendar(SyncCache& cache, const CachedAccount& cached_account = account(),
const CachedCalendar& cached_calendar = calendar()) {
cache.upsert_account(cached_account);
const std::vector calendars{cached_calendar};
cache.replace_calendars_after_complete_listing(cached_account.id, calendars);
}
[[nodiscard]] bool contains(const std::vector<std::string>& values, std::string_view value) {
return std::ranges::find(values, value) != values.end();
}
#if !defined(_WIN32)
void require_private_file(const std::filesystem::path& path, std::string_view description) {
struct stat metadata {};
require(::stat(path.c_str(), &metadata) == 0,
std::string{"could not stat "}.append(description));
require(S_ISREG(metadata.st_mode), std::string{description}.append(" is not a regular file"));
require((metadata.st_mode & (S_IRWXG | S_IRWXO)) == 0,
std::string{description}.append(" grants group or other permissions"));
require((metadata.st_mode & S_IRUSR) != 0 && (metadata.st_mode & S_IWUSR) != 0,
std::string{description}.append(" is not owner-readable and writable"));
}
#endif
void test_fresh_schema_permissions_and_secrets(const std::filesystem::path& root) {
const auto path = root / "fresh.db";
SyncCache cache{path};
require(cache.path() == path, "cache did not retain its path");
require(std::filesystem::is_regular_file(path), "fresh cache database was not created");
SqliteConnection inspection{path};
require(inspection.scalar_text("PRAGMA journal_mode") == "wal", "cache is not in WAL mode");
require(inspection.scalar_int("PRAGMA user_version") == SyncCache::current_schema_version,
"fresh cache schema version is incorrect");
const auto tables = inspection.first_column(
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name");
for (const std::string_view required : {"accounts", "calendars", "events",
"event_instances", "sync_state", "outbox", "conflicts"}) {
require(contains(tables, required), std::string{"missing cache table: "}.append(required));
}
for (const std::string_view child : {"calendars", "events", "event_instances", "sync_state"}) {
require(inspection.scalar_int(
std::string{"SELECT COUNT(*) FROM pragma_foreign_key_list('"}
.append(child)
.append("')"))
> 0,
std::string{"missing foreign key on "}.append(child));
}
const auto columns = inspection.first_column(
"SELECT lower(m.name || '.' || p.name) FROM sqlite_master AS m "
"JOIN pragma_table_info(m.name) AS p WHERE m.type='table'");
for (const auto& column : columns) {
require(column.find("token") == std::string::npos
&& column.find("secret") == std::string::npos
&& column.find("credential") == std::string::npos,
"cache schema contains a token, secret, or credential column");
}
#if !defined(_WIN32)
require_private_file(path, "cache database");
for (const auto suffix : {"-wal", "-shm"}) {
const std::filesystem::path sidecar = path.string() + suffix;
require(std::filesystem::exists(sidecar), "WAL sidecar was not created");
require_private_file(sidecar, "cache sidecar");
}
#endif
}
void test_existing_permissions_are_tightened(const std::filesystem::path& root) {
#if !defined(_WIN32)
const auto path = root / "permissive.db";
{
SqliteConnection database{path};
database.execute("PRAGMA user_version=0");
}
require(::chmod(path.c_str(), 0666) == 0, "could not make fixture permissive");
SyncCache cache{path};
require_private_file(path, "existing cache database");
#else
static_cast<void>(root);
#endif
}
void test_account_calendars_and_deterministic_snapshot(const std::filesystem::path& root) {
SyncCache cache{root / "listing.db"};
auto account_b = account("account-b");
account_b.remote_subject = "subject-b";
auto account_a = account();
cache.upsert_account(account_b);
cache.upsert_account(account_a);
auto calendar_b = calendar("calendar-b");
calendar_b.remote_id = "remote-calendar-b";
calendar_b.name = "Calendar B";
auto calendar_a = calendar();
cache.replace_calendars_after_complete_listing(
account_a.id, std::vector<CachedCalendar>{calendar_b, calendar_a});
auto first = cache.snapshot();
require(first.accounts.size() == 2 && first.accounts[0].id == "account-a"
&& first.accounts[1].id == "account-b",
"accounts are not returned deterministically by ID");
require(first.calendars.size() == 2 && first.calendars[0].id == "calendar-a"
&& first.calendars[1].id == "calendar-b",
"calendars are not returned deterministically by ID");
calendar_a.name = "Renamed";
cache.replace_calendars_after_complete_listing(
account_a.id, std::vector<CachedCalendar>{calendar_a});
const auto second = cache.snapshot();
require(second.calendars.size() == 2, "omitted calendar was deleted");
require(second.calendars[0].active && second.calendars[0].name == "Renamed",
"listed calendar was not updated and kept active");
require(!second.calendars[1].active, "omitted calendar was not deactivated");
require(second == cache.snapshot(), "unchanged snapshots are not deterministic");
}
void test_pull_pages_replacement_tombstone_and_windows(const std::filesystem::path& root) {
SyncCache cache{root / "pull.db"};
seed_calendar(cache);
auto event_b = event("event-b");
event_b.remote_id = "remote-event-b";
event_b.uid = "uid-b";
event_b.raw_payload = R"({"event":"b"})";
auto instance_b = instance("instance-b", "event-b");
instance_b.title = "Event B";
PullPage first{{event_b, event()}, {instance_b, instance()}, checkpoint("cursor-1")};
cache.apply_pull_page(first);
const auto initial = cache.snapshot();
require(initial.events.size() == 2 && initial.instances.size() == 2,
"pull page did not atomically add events and instances");
require(initial.events[0].id == "event-a" && initial.events[1].id == "event-b"
&& initial.instances[0].id == "instance-a"
&& initial.instances[1].id == "instance-b",
"events and instances are not returned deterministically by ID");
require(initial.checkpoints.size() == 1 && initial.checkpoints[0].cursor == "cursor-1",
"pull page did not advance its cursor");
cache.apply_pull_page(first);
require(cache.snapshot() == initial, "repeated pull page was not idempotent");
auto changed = event();
changed.etag = "etag-a-2";
changed.raw_payload = R"({"event":"a2"})";
auto replacement = instance("instance-a-2");
replacement.title = "Changed A";
cache.apply_pull_page(page(changed, replacement, checkpoint("cursor-2")));
auto replaced = cache.snapshot();
require(replaced.events.size() == 2 && replaced.instances.size() == 2,
"changed event replacement affected an untouched event");
require(contains(std::vector<std::string>{replaced.instances[0].id, replaced.instances[1].id},
"instance-a-2")
&& contains(std::vector<std::string>{
replaced.instances[0].id, replaced.instances[1].id},
"instance-b"),
"changed event instances were not replaced independently");
auto tombstone = event();
tombstone.deleted = true;
tombstone.raw_payload.clear();
cache.apply_pull_page(PullPage{{tombstone}, {}, checkpoint("cursor-3")});
const auto deleted = cache.snapshot();
const auto deleted_event = std::ranges::find_if(
deleted.events, [](const CachedEvent& value) { return value.id == "event-a"; });
require(deleted_event != deleted.events.end() && deleted_event->deleted,
"tombstone did not retain a deleted event record");
require(deleted_event->raw_payload == changed.raw_payload,
"empty tombstone did not retain the prior raw payload");
require(std::ranges::none_of(deleted.instances,
[](const CachedEventInstance& value) { return value.event_id == "event-a"; }),
"tombstone retained event instances");
cache.apply_pull_page(PullPage{{}, {}, checkpoint("other-cursor", "calendar-a",
"second-window-start", "second-window-end")});
const auto windows = cache.snapshot();
require(windows.checkpoints.size() == 2,
"independent checkpoint windows for one calendar were overwritten");
require(windows.checkpoints[0].window_start == "opaque-start"
&& windows.checkpoints[1].window_start == "second-window-start",
"checkpoints are not returned in deterministic scope order");
}
void test_multiple_connections_serialize(const std::filesystem::path& root) {
const auto path = root / "connections.db";
SyncCache first{path};
SyncCache second{path};
std::exception_ptr first_error;
std::exception_ptr second_error;
std::atomic<bool> begin{false};
auto write = [&begin](SyncCache& cache, CachedAccount value, std::exception_ptr& error) {
while (!begin.load(std::memory_order_acquire)) {
std::this_thread::yield();
}
try {
cache.upsert_account(value);
} catch (...) {
error = std::current_exception();
}
};
auto account_b = account("account-b");
account_b.remote_subject = "subject-b";
std::thread one{write, std::ref(first), account(), std::ref(first_error)};
std::thread two{write, std::ref(second), account_b, std::ref(second_error)};
begin.store(true, std::memory_order_release);
one.join();
two.join();
require(first_error == nullptr && second_error == nullptr,
"ordinary concurrent writes leaked SQLITE_BUSY or another error");
const auto snapshot = first.snapshot();
require(snapshot.accounts.size() == 2, "serialized connections lost a write");
}
void test_input_validation_and_rollback(const std::filesystem::path& root) {
SyncCache cache{root / "validation.db"};
seed_calendar(cache);
auto account_b = account("account-b");
account_b.remote_subject = "subject-b";
cache.upsert_account(account_b);
auto calendar_b = calendar("calendar-b", "account-b");
calendar_b.remote_id = "remote-calendar-b";
calendar_b.raw_payload = R"({"calendar":"b"})";
cache.replace_calendars_after_complete_listing("account-b", std::vector{calendar_b});
cache.apply_pull_page(page());
const CacheSnapshot baseline = cache.snapshot();
auto expect_unchanged = [&](auto&& operation, std::string_view message) {
require_throws(std::forward<decltype(operation)>(operation), message);
require(cache.snapshot() == baseline, "failed operation changed the cache snapshot");
};
auto empty_account = account();
empty_account.id.clear();
expect_unchanged([&] { cache.upsert_account(empty_account); }, "empty account ID accepted");
auto reassigned = account();
reassigned.remote_subject = "different-subject";
expect_unchanged([&] { cache.upsert_account(reassigned); },
"account local ID identity reassignment accepted");
auto provider_reassigned = account();
provider_reassigned.provider = "different-provider";
expect_unchanged([&] { cache.upsert_account(provider_reassigned); },
"account provider identity reassignment accepted");
auto duplicate_calendar = calendar();
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing(
"account-a", std::vector<CachedCalendar>{duplicate_calendar, duplicate_calendar});
},
"duplicate calendar ID accepted");
auto duplicate_remote_calendar = calendar("calendar-new");
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing("account-a",
std::vector<CachedCalendar>{duplicate_calendar, duplicate_remote_calendar});
},
"duplicate calendar remote ID accepted");
auto empty_calendar = calendar();
empty_calendar.id.clear();
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing(
"account-a", std::vector<CachedCalendar>{empty_calendar});
},
"empty calendar ID accepted");
auto wrong_account = calendar();
wrong_account.account_id = "account-b";
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing(
"account-a", std::vector<CachedCalendar>{wrong_account});
},
"cross-account calendar accepted");
auto reused_calendar_id = calendar("calendar-a", "account-b");
reused_calendar_id.remote_id = "different-remote-calendar";
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing(
"account-b", std::vector<CachedCalendar>{reused_calendar_id});
},
"calendar local ID was reassigned across accounts");
auto inactive_input = calendar();
inactive_input.active = false;
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing(
"account-a", std::vector<CachedCalendar>{inactive_input});
},
"inactive completed-listing input accepted");
auto missing_raw_calendar = calendar();
missing_raw_calendar.raw_payload.clear();
expect_unchanged(
[&] {
cache.replace_calendars_after_complete_listing(
"account-a", std::vector<CachedCalendar>{missing_raw_calendar});
},
"calendar without raw payload accepted");
auto duplicate_event = event();
expect_unchanged(
[&] {
cache.apply_pull_page(PullPage{{duplicate_event, duplicate_event}, {},
checkpoint("duplicate-cursor")});
},
"duplicate event ID accepted");
auto duplicate_remote_event = event("event-new");
expect_unchanged(
[&] {
cache.apply_pull_page(PullPage{{duplicate_event, duplicate_remote_event}, {},
checkpoint("duplicate-remote-event")});
},
"duplicate event remote ID accepted");
auto empty_event = event();
empty_event.id.clear();
expect_unchanged(
[&] { cache.apply_pull_page(page(empty_event, instance(), checkpoint("empty-event"))); },
"empty event ID accepted");
auto cross_calendar_event = event();
cross_calendar_event.calendar_id = "calendar-b";
expect_unchanged(
[&] {
cache.apply_pull_page(
PullPage{{cross_calendar_event}, {}, checkpoint("cross-calendar")});
},
"cross-calendar event accepted");
auto reassigned_event = event("event-a", "calendar-b");
reassigned_event.remote_id = "different-remote-event";
expect_unchanged(
[&] {
cache.apply_pull_page(PullPage{{reassigned_event}, {},
checkpoint("reassigned-event", "calendar-b")});
},
"event local ID was reassigned across calendars");
auto missing_remote = event();
missing_remote.remote_id.clear();
expect_unchanged(
[&] { cache.apply_pull_page(page(missing_remote, instance(), checkpoint("remote"))); },
"event without remote ID accepted");
auto missing_raw_event = event();
missing_raw_event.raw_payload.clear();
expect_unchanged(
[&] { cache.apply_pull_page(page(missing_raw_event, instance(), checkpoint("raw"))); },
"nondeleted event without raw payload accepted");
auto duplicate_instance = instance();
expect_unchanged(
[&] {
cache.apply_pull_page(PullPage{{event()}, {duplicate_instance, duplicate_instance},
checkpoint("duplicate-instance")});
},
"duplicate instance ID accepted");
auto empty_instance = instance();
empty_instance.id.clear();
expect_unchanged(
[&] { cache.apply_pull_page(page(event(), empty_instance, checkpoint("empty-instance"))); },
"empty instance ID accepted");
auto orphan = instance();
orphan.event_id = "event-not-in-page";
expect_unchanged(
[&] { cache.apply_pull_page(page(event(), orphan, checkpoint("orphan"))); },
"orphan instance accepted");
auto backwards = instance();
backwards.end_epoch_microseconds = backwards.start_epoch_microseconds - 1;
expect_unchanged(
[&] { cache.apply_pull_page(page(event(), backwards, checkpoint("backwards"))); },
"backwards instance interval accepted");
auto empty_all_day = instance();
empty_all_day.all_day = true;
empty_all_day.end_epoch_microseconds = empty_all_day.start_epoch_microseconds;
expect_unchanged(
[&] { cache.apply_pull_page(page(event(), empty_all_day, checkpoint("all-day"))); },
"empty all-day interval accepted");
auto deleted_with_instance = event();
deleted_with_instance.deleted = true;
deleted_with_instance.raw_payload.clear();
expect_unchanged(
[&] {
cache.apply_pull_page(
page(deleted_with_instance, instance(), checkpoint("deleted-instance")));
},
"instance attached to tombstoned event accepted");
auto negative = event();
negative.id = "event-negative";
negative.remote_id = "remote-negative";
negative.uid = "uid-negative";
auto before_epoch = instance("instance-negative", "event-negative");
before_epoch.start_epoch_microseconds = -2'000'000;
before_epoch.end_epoch_microseconds = -1'000'000;
cache.apply_pull_page(page(negative, before_epoch, checkpoint("negative")));
require(std::ranges::any_of(cache.snapshot().instances,
[](const CachedEventInstance& value) {
return value.id == "instance-negative"
&& value.start_epoch_microseconds < 0;
}),
"valid pre-1970 instance was rejected");
}
void test_sql_failure_rolls_back_snapshot_and_cursor(const std::filesystem::path& root) {
SyncCache cache{root / "sql-rollback.db"};
seed_calendar(cache);
auto event_b = event("event-b");
event_b.remote_id = "remote-event-b";
event_b.uid = "uid-b";
event_b.raw_payload = R"({"event":"b"})";
auto instance_b = instance("instance-b", "event-b");
cache.apply_pull_page(
PullPage{{event(), event_b}, {instance(), instance_b}, checkpoint("baseline-cursor")});
const auto baseline = cache.snapshot();
auto changed = event();
changed.etag = "must-roll-back";
changed.raw_payload = R"({"event":"must-roll-back"})";
auto colliding_instance = instance("instance-b", "event-a");
require_throws(
[&] {
cache.apply_pull_page(
page(changed, colliding_instance, checkpoint("must-not-advance")));
},
"cross-event instance ID collision was accepted");
require(cache.snapshot() == baseline,
"SQL failure did not roll back event, instance, and cursor state together");
}
void test_checkpoint_calendar_and_shape_validation(const std::filesystem::path& root) {
const auto path = root / "checkpoint-validation.db";
SyncCache cache{path};
seed_calendar(cache);
auto expect_empty = [&](auto&& operation, std::string_view message) {
require_throws(std::forward<decltype(operation)>(operation), message);
require(cache.snapshot().events.empty() && cache.snapshot().checkpoints.empty(),
"invalid pull partially changed event or checkpoint state");
};
expect_empty(
[&] { cache.apply_pull_page(page(event(), instance(), checkpoint("", "calendar-a"))); },
"empty checkpoint cursor accepted");
expect_empty(
[&] {
cache.apply_pull_page(page(event(), instance(),
checkpoint("cursor", "calendar-a", "only-start", "")));
},
"half-specified checkpoint window accepted");
expect_empty(
[&] {
cache.apply_pull_page(page(event(), instance(),
checkpoint("cursor", "calendar-a", "", "only-end")));
},
"half-specified checkpoint window accepted");
expect_empty(
[&] {
cache.apply_pull_page(
page(event(), instance(), checkpoint("cursor", "missing-calendar")));
},
"missing checkpoint calendar accepted");
cache.apply_pull_page(PullPage{{}, {}, checkpoint("empty-window", "calendar-a", "", "")});
require(cache.snapshot().checkpoints.size() == 1
&& cache.snapshot().checkpoints[0].window_start.empty()
&& cache.snapshot().checkpoints[0].window_end.empty(),
"valid unwindowed opaque checkpoint was rejected");
cache.replace_calendars_after_complete_listing("account-a", {});
require(!cache.snapshot().calendars[0].active, "fixture calendar was not deactivated");
require_throws(
[&] { cache.apply_pull_page(page()); }, "inactive checkpoint calendar accepted");
require(cache.snapshot().events.empty(), "inactive-calendar pull changed event state");
}
void test_foreign_key_failure_rolls_back(const std::filesystem::path& root) {
const auto path = root / "foreign-key.db";
SyncCache cache{path};
seed_calendar(cache);
cache.apply_pull_page(page());
const auto baseline = cache.snapshot();
SqliteConnection external{path};
external.execute("PRAGMA foreign_keys=ON");
require_throws(
[&] { external.execute("DELETE FROM accounts WHERE id='account-a'"); },
"foreign-key violating direct delete succeeded");
require(cache.snapshot() == baseline, "foreign-key failure changed cached data");
}
void test_migration_failures_are_non_destructive(const std::filesystem::path& root) {
const auto collision_path = root / "collision.db";
{
SqliteConnection database{collision_path};
database.execute("CREATE TABLE accounts (sentinel TEXT NOT NULL)");
database.execute("INSERT INTO accounts VALUES ('keep-me')");
database.execute("PRAGMA user_version=0");
}
require_throws([&] { SyncCache cache{collision_path}; },
"incompatible migration collision was accepted");
{
SqliteConnection database{collision_path};
require(database.scalar_int("PRAGMA user_version") == 0,
"failed migration changed schema version");
require(database.scalar_text("SELECT sentinel FROM accounts") == "keep-me",
"failed migration changed preexisting data");
require(database.scalar_int(
"SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name!='accounts'")
== 0,
"failed migration left partially created v1 tables");
}
const auto future_path = root / "future.db";
{
SqliteConnection database{future_path};
database.execute("CREATE TABLE future_sentinel (value TEXT NOT NULL)");
database.execute("INSERT INTO future_sentinel VALUES ('future-data')");
database.execute("PRAGMA user_version=2");
}
require_throws([&] { SyncCache cache{future_path}; }, "future schema version was accepted");
{
SqliteConnection database{future_path};
require(database.scalar_int("PRAGMA user_version") == 2,
"future schema rejection changed schema version");
require(database.scalar_text("SELECT value FROM future_sentinel") == "future-data",
"future schema rejection changed data");
require(database.scalar_int("SELECT COUNT(*) FROM sqlite_master WHERE type='table'") == 1,
"future schema rejection created v1 tables");
}
}
void test_invalid_files_are_refused(const std::filesystem::path& root) {
const auto corrupt_path = root / "corrupt.db";
{
std::ofstream output(corrupt_path, std::ios::binary);
output << "this is not a SQLite database";
require(static_cast<bool>(output), "failed to write corrupt database fixture");
}
const std::string before = [&] {
std::ifstream input(corrupt_path, std::ios::binary);
return std::string{std::istreambuf_iterator<char>{input}, {}};
}();
require_throws([&] { SyncCache cache{corrupt_path}; }, "non-SQLite cache file was accepted");
std::ifstream after_input(corrupt_path, std::ios::binary);
const std::string after{std::istreambuf_iterator<char>{after_input}, {}};
require(after == before, "corrupt cache rejection changed the source file");
#if !defined(_WIN32)
const auto target_path = root / "symlink-target.db";
{
SqliteConnection target{target_path};
}
const auto symlink_path = root / "symlink.db";
require(::symlink(target_path.c_str(), symlink_path.c_str()) == 0,
"failed to create symlink fixture");
require_throws([&] { SyncCache cache{symlink_path}; }, "symlink cache path was accepted");
const auto directory_path = root / "directory.db";
std::filesystem::create_directory(directory_path);
require_throws([&] { SyncCache cache{directory_path}; },
"nonregular cache path was accepted");
#endif
}
} // namespace
int main() {
try {
TempDirectory temporary;
test_fresh_schema_permissions_and_secrets(temporary.path());
test_existing_permissions_are_tightened(temporary.path());
test_account_calendars_and_deterministic_snapshot(temporary.path());
test_pull_pages_replacement_tombstone_and_windows(temporary.path());
test_multiple_connections_serialize(temporary.path());
test_input_validation_and_rollback(temporary.path());
test_sql_failure_rolls_back_snapshot_and_cursor(temporary.path());
test_checkpoint_calendar_and_shape_validation(temporary.path());
test_foreign_key_failure_rolls_back(temporary.path());
test_migration_failures_are_non_destructive(temporary.path());
test_invalid_files_are_refused(temporary.path());
} catch (const std::exception& error) {
std::cerr << "sync cache test failure: " << error.what() << '\n';
return 1;
}
std::cout << "sync cache tests passed\n";
return 0;
}