0.1.0
This commit is contained in:
25
migration/khadhroony-bot2-reference/kb_store_pg/Cargo.toml
Normal file
25
migration/khadhroony-bot2-reference/kb_store_pg/Cargo.toml
Normal file
@@ -0,0 +1,25 @@
|
||||
# file: kb_store_pg/Cargo.toml
|
||||
# version: 5
|
||||
|
||||
[package]
|
||||
name = "kb_store_pg"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
license.workspace = true
|
||||
publish.workspace = true
|
||||
|
||||
[dependencies]
|
||||
async-trait.workspace = true
|
||||
chrono.workspace = true
|
||||
kb_config = { path = "../kb_config" }
|
||||
kb_core = { path = "../kb_core" }
|
||||
kb_model = { path = "../kb_model" }
|
||||
kb_store_core = { path = "../kb_store_core" }
|
||||
serde.workspace = true
|
||||
serde_json.workspace = true
|
||||
sqlx.workspace = true
|
||||
tokio.workspace = true
|
||||
tracing.workspace = true
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
246
migration/khadhroony-bot2-reference/kb_store_pg/README.md
Normal file
246
migration/khadhroony-bot2-reference/kb_store_pg/README.md
Normal file
@@ -0,0 +1,246 @@
|
||||
<!-- file: kb_store_pg/README.md -->
|
||||
<!-- version: 13 -->
|
||||
|
||||
# kb_store_pg
|
||||
|
||||
`kb_store_pg` implémente le backend PostgreSQL destiné à la production.
|
||||
|
||||
Ce crate dépend de `kb_store_core` pour les contrats et fournit les implémentations concrètes PostgreSQL : pool, healthcheck, migrations, requêtes SQL et repositories.
|
||||
|
||||
## Rôle exact
|
||||
|
||||
`kb_store_pg` contient :
|
||||
|
||||
- le handle de store PostgreSQL ;
|
||||
- la création effective du pool PostgreSQL ;
|
||||
- le healthcheck PostgreSQL minimal ;
|
||||
- les diagnostics backend : DSN masqué, schéma courant et version serveur ;
|
||||
- la stratégie de migrations PostgreSQL ;
|
||||
- les requêtes SQL dans `queries/` ;
|
||||
- les implémentations repository dans `repositories/`.
|
||||
|
||||
`kb_store_pg` ne contient pas :
|
||||
|
||||
- de contrat applicatif générique qui devrait vivre dans `kb_store_core` ;
|
||||
- de logique Tauri ;
|
||||
- de logique RPC ;
|
||||
- de décodeur ;
|
||||
- de matérialisateur métier ;
|
||||
- de création de schémas PostgreSQL applicatifs explicites ;
|
||||
- de création des tables `decode`, `mat`, `catalog`, `agg` ou `wallet` avant les jalons dédiés ; la seule table `ops` active en `0.3.4` est le ledger générique de traitement.
|
||||
|
||||
## Convention PostgreSQL
|
||||
|
||||
`kb_store_pg` utilise le schéma courant/default du profil PostgreSQL, généralement `public`.
|
||||
|
||||
Il est interdit de créer ou d'utiliser des schémas applicatifs explicites comme :
|
||||
|
||||
```text
|
||||
raw
|
||||
core
|
||||
obs
|
||||
decode
|
||||
mat
|
||||
catalog
|
||||
agg
|
||||
ops
|
||||
wallet
|
||||
```
|
||||
|
||||
La séparation logique se fait dans le nom de table :
|
||||
|
||||
```text
|
||||
kb_sol_<domain>_<name>
|
||||
```
|
||||
|
||||
Exemples valides :
|
||||
|
||||
```text
|
||||
kb_sol_raw_transactions
|
||||
kb_sol_obs_transaction_observations
|
||||
kb_sol_core_transactions
|
||||
kb_sol_core_account_keys
|
||||
kb_sol_core_instructions
|
||||
kb_sol_core_inner_instructions
|
||||
kb_sol_core_logs
|
||||
kb_sol_core_balance_changes
|
||||
kb_sol_obs_program_observations
|
||||
kb_sol_obs_instruction_observations
|
||||
kb_sol_decode_decoded_events
|
||||
kb_sol_mat_trade_events
|
||||
kb_sol_catalog_tokens
|
||||
kb_sol_catalog_pools
|
||||
kb_sol_catalog_pairs
|
||||
kb_sol_ops_processing_ledger
|
||||
```
|
||||
|
||||
Exemples interdits, écrits avec `DOT` pour que les audits textuels simples ne confondent pas documentation et usage réel :
|
||||
|
||||
```text
|
||||
raw DOT kb_sol_rpc_transactions
|
||||
raw DOT sol_transactions
|
||||
core DOT sol_instructions
|
||||
obs DOT program_observations
|
||||
decode DOT protocol_events
|
||||
ops DOT processing_ledger
|
||||
```
|
||||
|
||||
## Layout obligatoire
|
||||
|
||||
```text
|
||||
src/
|
||||
lib.rs
|
||||
pg_store.rs
|
||||
migrations.rs
|
||||
queries/
|
||||
repositories/
|
||||
```
|
||||
|
||||
## Conventions de dossiers
|
||||
|
||||
| Dossier | Rôle |
|
||||
|-----------------|--------------------------------------------------------|
|
||||
| `queries/` | SQL, bind et exécution SQL uniquement. |
|
||||
| `repositories/` | Implémentations PostgreSQL des traits `kb_store_core`. |
|
||||
|
||||
Aucune structure métier, DTO ou entity ne doit être créée dans `queries/`.
|
||||
|
||||
## Infrastructure `0.2.2`
|
||||
|
||||
Le jalon `0.2.2` introduit `PostgresStoreOptions` et `PostgresStore`.
|
||||
|
||||
Le store sait :
|
||||
|
||||
- construire ses options depuis `kb_config::PostgresConfig` ;
|
||||
- valider `url`, `max_connections` et `connect_timeout_ms` ;
|
||||
- créer un pool `sqlx::PgPool` ;
|
||||
- masquer le DSN pour les logs et l'interface ;
|
||||
- lire `current_schema()` ;
|
||||
- lire `version()` ;
|
||||
- exécuter un healthcheck `SELECT 1` ;
|
||||
- lire un snapshot de migrations non destructif.
|
||||
|
||||
Le store ne crée aucune table Solana lourde en `0.2.2`; raw est ajouté en `0.2.3` et core en `0.2.4`.
|
||||
|
||||
## Migrations
|
||||
|
||||
Le store applique un DDL idempotent contrôlé par le crate, dans le schéma courant du profil. Après validation réelle de la transition historique `0.2.x -> 0.3.1`, la baseline a été consolidée puis étendue par une migration additive dédiée au ledger :
|
||||
|
||||
```text
|
||||
0001_canonical_transaction_store.sql
|
||||
0002_core_store.sql
|
||||
0003_processing_ledger.sql
|
||||
```
|
||||
|
||||
Ces fichiers ne créent que les tables canoniques actuelles. Les anciens noms RPC/WS et la logique de conversion historique restent documentés dans `CHANGELOG.md`, mais ne font plus partie du DDL runtime.
|
||||
|
||||
Les futures migrations doivent :
|
||||
|
||||
- utiliser uniquement le schéma courant du profil ;
|
||||
- créer des tables non qualifiées comme `kb_sol_raw_transactions` ;
|
||||
- ajouter des contraintes strictes mais réversibles ;
|
||||
- privilégier des index minimaux par `signature`, `slot`, `program_id` et `created_at` selon table ;
|
||||
- éviter les contraintes métier prématurées.
|
||||
|
||||
## Règles locales
|
||||
|
||||
- Les commentaires de code restent en anglais.
|
||||
- La documentation Markdown reste en français.
|
||||
- Les exports publics sont contrôlés depuis `lib.rs`.
|
||||
- Les erreurs passent par `kb_core::Error` et `kb_core::Result`.
|
||||
- Les tests PostgreSQL réels sont reportables si aucun serveur local n'est garanti.
|
||||
|
||||
## Raw store `0.2.3`
|
||||
|
||||
Le jalon `0.2.3` ajoute les premières tables Solana réelles :
|
||||
|
||||
```text
|
||||
kb_sol_raw_rpc_transactions
|
||||
kb_sol_raw_ws_notifications
|
||||
```
|
||||
|
||||
Ces tables restent dans le schéma PostgreSQL courant du profil actif. Elles sont créées par une initialisation idempotente du store PostgreSQL et non par des schémas applicatifs explicites.
|
||||
|
||||
La table RPC est dédupliquée par `signature`. La table WebSocket est dédupliquée par `notification_key`, avec signature optionnelle et lien optionnel vers une transaction RPC canonique déjà stockée.
|
||||
|
||||
La méthode `PostgresStore::initialize_raw_store_schema()` applique uniquement le schéma raw minimal. Si `database.postgres.auto_initialize_schema` vaut `true`, cette initialisation est appelée après la connexion.
|
||||
|
||||
`0.2.3` ne supprime pas encore physiquement les payloads raw. Les colonnes `retention_state`, `processing_state`, `raw_json_hash` et `lifecycle_reason` préparent la compaction, l'archivage et la purge future.
|
||||
|
||||
## Core store `0.2.4`
|
||||
|
||||
Le jalon `0.2.4` ajoute les tables core minimales :
|
||||
|
||||
```text
|
||||
kb_sol_core_transactions
|
||||
kb_sol_core_account_keys
|
||||
kb_sol_core_instructions
|
||||
kb_sol_core_inner_instructions
|
||||
kb_sol_core_logs
|
||||
kb_sol_core_balance_changes
|
||||
```
|
||||
|
||||
`PostgresStore::initialize_core_store_schema()` applique le schéma raw puis le schéma core, car les transactions core peuvent référencer les transactions raw.
|
||||
|
||||
`PostgresStore::initialize_store_schema()` applique maintenant raw puis core. Si `database.postgres.auto_initialize_schema` vaut `true`, la connexion applique donc les deux familles de tables.
|
||||
|
||||
Le repository `CoreTransactionStore` est implémenté pour :
|
||||
|
||||
- insérer la transaction core canonique ;
|
||||
- insérer les account keys résolus ;
|
||||
- insérer les instructions replayables ;
|
||||
- insérer les inner instructions comme contexte d'arbre ;
|
||||
- insérer les logs ordonnés ;
|
||||
- insérer les balance changes ;
|
||||
- lister les instructions à rejouer ;
|
||||
- construire `CoreInstructionReplayInput` avec contexte, dont toutes les instructions outer de la signature ordonnées par index numérique ;
|
||||
- marquer le lifecycle d'une instruction.
|
||||
|
||||
## Baseline canonique `0.3.1`
|
||||
|
||||
`0.3.1` a validé sur PostgreSQL réel la transition des anciennes tables vers :
|
||||
|
||||
```text
|
||||
kb_sol_raw_transactions
|
||||
kb_sol_obs_transaction_observations
|
||||
kb_sol_core_transactions.raw_transaction_id
|
||||
```
|
||||
|
||||
La transaction raw est unique par signature et porte le document canonique versionné. Les observations sont légères et ne contiennent aucun payload transactionnel complet. Les repositories actifs exposent `RawTransactionInsert` et `TransactionObservationInsert`.
|
||||
|
||||
Après validation, les scripts SQL historiques ont été remplacés par la baseline courante `0001`/`0002`. L’initialiseur crate-managed conserve temporairement un chemin de compatibilité idempotent pour les workspaces encore issus de `pre.001`; il ne recrée pas les anciennes tables.
|
||||
|
||||
## Extraction core `0.3.4`
|
||||
|
||||
`kb_store_pg` implémente `CoreExtractionStore` avec une transaction PostgreSQL unique par signature. Une extraction réussie :
|
||||
|
||||
- remplace le graphe core de la signature par cascade ;
|
||||
- insère transaction, account keys, instructions, inner instructions, logs et balance changes ;
|
||||
- marque la ligne raw `core_extracted` ;
|
||||
- upsert le ledger `kb_sol_ops_processing_ledger` avec la version et le hash canonique ;
|
||||
- commit l’ensemble ou annule toutes les écritures.
|
||||
|
||||
Les échecs d’extraction sont enregistrés séparément dans le ledger et dans l’état raw afin de rester rejouables.
|
||||
|
||||
## Store decode et matérialisation `0.4.0`
|
||||
|
||||
La migration additive `0004_decode_materialization_store.sql` crée dans le schéma courant :
|
||||
|
||||
```text
|
||||
kb_sol_decode_events
|
||||
kb_sol_decode_coverage_declarations
|
||||
kb_sol_decode_coverage_observations
|
||||
kb_sol_mat_events
|
||||
```
|
||||
|
||||
Ces tables réutilisent `kb_sol_ops_processing_ledger`. Les observations decode, leur couverture, l’état lifecycle et le ledger sont écrits dans une même transaction PostgreSQL. Les sorties materialized et leur ledger suivent la même règle. Les colonnes JSONB portent le suffixe `_jsonb`.
|
||||
|
||||
## Tracing opérationnel
|
||||
|
||||
`kb_store_pg` utilise le target canonique `kb_store_pg`. Les décisions SQL de decode/mat/ledger et leurs rollbacks sont émises par le store lui-même. Un `mark_decode_failed`, une persistance de bundle matérialisation en statut `failed` ou une erreur transactionnelle sont écrits au niveau `error` avec les identifiants de corrélation disponibles.
|
||||
|
||||
Le store ne délègue pas ces détails à `kb_app_demo` et ne journalise jamais un DSN non masqué.
|
||||
|
||||
## Contexte outer du replay `0.4.1-pre.014`
|
||||
|
||||
La lecture contextualisée agrège directement `kb_sol_core_instructions` pour la signature ciblée. Elle expose les chemins outer purement numériques sous la forme stable `instructionIndex`, `instructionPath`, `programId`, `payloadJson`, `payloadHash`, avec un `ORDER BY instruction_path::BIGINT`. L’instruction cible reste présente dans le tableau. La requête est strictement en lecture et n’ajoute aucune migration.
|
||||
@@ -0,0 +1,21 @@
|
||||
-- file: kb_store_pg/maintenance/drop_raw_core_store.sql
|
||||
-- version: 4
|
||||
|
||||
-- Drop raw and core Solana storage tables in reverse dependency order.
|
||||
-- This script is intended for local development and test databases only.
|
||||
-- It does not drop unrelated schemas, migrations, roles or configuration state.
|
||||
-- It intentionally avoids CASCADE so unexpected dependencies fail loudly.
|
||||
-- No custom PostgreSQL enum/domain types are owned by these 0.2.3/0.2.4 tables.
|
||||
|
||||
BEGIN;
|
||||
|
||||
DROP TABLE IF EXISTS kb_sol_core_balance_changes;
|
||||
DROP TABLE IF EXISTS kb_sol_core_logs;
|
||||
DROP TABLE IF EXISTS kb_sol_core_inner_instructions;
|
||||
DROP TABLE IF EXISTS kb_sol_core_instructions;
|
||||
DROP TABLE IF EXISTS kb_sol_core_account_keys;
|
||||
DROP TABLE IF EXISTS kb_sol_core_transactions;
|
||||
DROP TABLE IF EXISTS kb_sol_obs_transaction_observations;
|
||||
DROP TABLE IF EXISTS kb_sol_raw_transactions;
|
||||
|
||||
COMMIT;
|
||||
@@ -0,0 +1,94 @@
|
||||
-- file: kb_store_pg/migrations/0001_canonical_transaction_store.sql
|
||||
-- version: 1
|
||||
|
||||
-- Current canonical transaction and acquisition observation store.
|
||||
-- This baseline contains only active tables and uses the current PostgreSQL schema.
|
||||
-- Historical 0.2.x upgrade logic was validated before this baseline was consolidated.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_raw_transactions (
|
||||
id BIGSERIAL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
canonical_json JSONB,
|
||||
canonical_json_hash TEXT,
|
||||
canonical_format_version INTEGER NOT NULL DEFAULT 1,
|
||||
retention_state TEXT NOT NULL DEFAULT 'full',
|
||||
processing_state TEXT NOT NULL DEFAULT 'received',
|
||||
lifecycle_reason TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_raw_transactions PRIMARY KEY (id),
|
||||
CONSTRAINT ck_kb_sol_raw_transactions_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_raw_transactions_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_raw_transactions_format_version_positive CHECK (canonical_format_version > 0),
|
||||
CONSTRAINT ck_kb_sol_raw_transactions_retention_state CHECK (retention_state IN ('full', 'compacted', 'archived', 'purged')),
|
||||
CONSTRAINT ck_kb_sol_raw_transactions_processing_state CHECK (processing_state IN ('received', 'core_extracted', 'decoded', 'materialized', 'failed')),
|
||||
CONSTRAINT ck_kb_sol_raw_transactions_full_has_json CHECK (retention_state <> 'full' OR canonical_json IS NOT NULL)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_raw_transactions_signature ON kb_sol_raw_transactions (signature);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_raw_transactions_slot ON kb_sol_raw_transactions (slot);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_raw_transactions_created_at ON kb_sol_raw_transactions (created_at);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_raw_transactions_processing ON kb_sol_raw_transactions (processing_state);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_raw_transactions_canonical_hash ON kb_sol_raw_transactions (canonical_json_hash) WHERE canonical_json_hash IS NOT NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_obs_transaction_observations (
|
||||
id BIGSERIAL,
|
||||
raw_transaction_id BIGINT,
|
||||
observation_key TEXT NOT NULL,
|
||||
signature TEXT,
|
||||
slot BIGINT,
|
||||
provider TEXT NOT NULL,
|
||||
endpoint_code TEXT,
|
||||
protocol TEXT NOT NULL,
|
||||
acquisition_method TEXT NOT NULL,
|
||||
origin TEXT NOT NULL,
|
||||
commitment TEXT,
|
||||
capture_session_id TEXT,
|
||||
filter_code TEXT,
|
||||
detected_at TIMESTAMPTZ,
|
||||
received_at TIMESTAMPTZ NOT NULL,
|
||||
normalized_at TIMESTAMPTZ,
|
||||
persisted_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
payload_size_bytes BIGINT,
|
||||
source_payload_hash TEXT,
|
||||
status TEXT NOT NULL DEFAULT 'received',
|
||||
error_code TEXT,
|
||||
error_message TEXT,
|
||||
CONSTRAINT pk_kb_sol_obs_transaction_observations PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_obs_transaction_observations_raw_transaction FOREIGN KEY (raw_transaction_id) REFERENCES kb_sol_raw_transactions(id) ON DELETE SET NULL,
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_key_not_empty CHECK (length(btrim(observation_key)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_signature_not_empty CHECK (signature IS NULL OR length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_slot_non_negative CHECK (slot IS NULL OR slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_provider_not_empty CHECK (length(btrim(provider)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_endpoint_not_empty CHECK (endpoint_code IS NULL OR length(btrim(endpoint_code)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_protocol_not_empty CHECK (length(btrim(protocol)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_method_not_empty CHECK (length(btrim(acquisition_method)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_origin CHECK (origin IN ('live', 'backfill', 'replay', 'repair', 'migration')),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_commitment_not_empty CHECK (commitment IS NULL OR length(btrim(commitment)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_session_not_empty CHECK (capture_session_id IS NULL OR length(btrim(capture_session_id)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_filter_not_empty CHECK (filter_code IS NULL OR length(btrim(filter_code)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_payload_size_non_negative CHECK (payload_size_bytes IS NULL OR payload_size_bytes >= 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_hash_not_empty CHECK (source_payload_hash IS NULL OR length(btrim(source_payload_hash)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_status CHECK (status IN ('detected', 'received', 'normalized', 'persisted', 'failed', 'missing')),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_error_code_not_empty CHECK (error_code IS NULL OR length(btrim(error_code)) > 0),
|
||||
CONSTRAINT ck_kb_sol_obs_transaction_observations_error_message_not_empty CHECK (error_message IS NULL OR length(btrim(error_message)) > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_obs_transaction_observations_key ON kb_sol_obs_transaction_observations (observation_key);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_obs_transaction_observations_signature ON kb_sol_obs_transaction_observations (signature) WHERE signature IS NOT NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_obs_transaction_observations_slot ON kb_sol_obs_transaction_observations (slot) WHERE slot IS NOT NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_obs_transaction_observations_provider_method ON kb_sol_obs_transaction_observations (provider, acquisition_method);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_obs_transaction_observations_received_at ON kb_sol_obs_transaction_observations (received_at);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_obs_transaction_observations_raw_transaction ON kb_sol_obs_transaction_observations (raw_transaction_id) WHERE raw_transaction_id IS NOT NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_obs_transaction_observations_status ON kb_sol_obs_transaction_observations (status);
|
||||
@@ -0,0 +1,196 @@
|
||||
-- file: kb_store_pg/migrations/0002_core_store.sql
|
||||
-- version: 1
|
||||
|
||||
-- Current normalized Solana core store.
|
||||
-- This migration uses the current PostgreSQL schema from the active profile.
|
||||
-- It must not create application schemas such as raw, core, obs, decode, mat or catalog.
|
||||
-- Object naming convention: pk_ for primary keys, fk_ for foreign keys, ux_ for unique indexes and ix_ for non-unique indexes.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_core_transactions (
|
||||
id BIGSERIAL,
|
||||
raw_transaction_id BIGINT,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
failed BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
err_json JSONB,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_core_transactions PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_core_transactions_raw_transaction FOREIGN KEY (raw_transaction_id) REFERENCES kb_sol_raw_transactions(id) ON DELETE SET NULL,
|
||||
CONSTRAINT ck_kb_sol_core_transactions_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_transactions_slot_non_negative CHECK (slot >= 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_core_transactions_signature
|
||||
ON kb_sol_core_transactions (signature);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_transactions_slot
|
||||
ON kb_sol_core_transactions (slot);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_transactions_created_at
|
||||
ON kb_sol_core_transactions (created_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_core_account_keys (
|
||||
id BIGSERIAL,
|
||||
transaction_id BIGINT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
account_index INTEGER NOT NULL,
|
||||
account_key TEXT NOT NULL,
|
||||
source TEXT NOT NULL,
|
||||
writable BOOLEAN NOT NULL,
|
||||
signer BOOLEAN NOT NULL,
|
||||
executable BOOLEAN,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_core_account_keys PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_core_account_keys_transaction FOREIGN KEY (transaction_id) REFERENCES kb_sol_core_transactions(id) ON DELETE CASCADE,
|
||||
CONSTRAINT ck_kb_sol_core_account_keys_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_account_keys_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_account_keys_index_non_negative CHECK (account_index >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_account_keys_key_not_empty CHECK (length(btrim(account_key)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_account_keys_source CHECK (source IN ('static', 'loaded_writable', 'loaded_readonly'))
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_core_account_keys_sig_index
|
||||
ON kb_sol_core_account_keys (signature, account_index);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_account_keys_key
|
||||
ON kb_sol_core_account_keys (account_key);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_core_instructions (
|
||||
id BIGSERIAL,
|
||||
transaction_id BIGINT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
instruction_path TEXT NOT NULL,
|
||||
program_id TEXT NOT NULL,
|
||||
accounts_json JSONB NOT NULL,
|
||||
payload_json JSONB,
|
||||
payload_json_hash TEXT,
|
||||
processing_state TEXT NOT NULL DEFAULT 'pending',
|
||||
processor_name TEXT,
|
||||
processor_version TEXT,
|
||||
lifecycle_reason TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_core_instructions PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_core_instructions_transaction FOREIGN KEY (transaction_id) REFERENCES kb_sol_core_transactions(id) ON DELETE CASCADE,
|
||||
CONSTRAINT ck_kb_sol_core_instructions_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_instructions_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_instructions_path_not_empty CHECK (length(btrim(instruction_path)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_instructions_program_not_empty CHECK (length(btrim(program_id)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_instructions_processing_state CHECK (processing_state IN ('pending', 'decoded', 'materialized', 'ignored', 'failed', 'replay_requested'))
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_core_instructions_sig_path
|
||||
ON kb_sol_core_instructions (signature, instruction_path);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_instructions_program
|
||||
ON kb_sol_core_instructions (program_id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_instructions_slot
|
||||
ON kb_sol_core_instructions (slot);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_instructions_processing
|
||||
ON kb_sol_core_instructions (processing_state);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_core_inner_instructions (
|
||||
id BIGSERIAL,
|
||||
transaction_id BIGINT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
parent_instruction_path TEXT NOT NULL,
|
||||
instruction_path TEXT NOT NULL,
|
||||
program_id TEXT NOT NULL,
|
||||
accounts_json JSONB NOT NULL,
|
||||
payload_json JSONB,
|
||||
payload_json_hash TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_core_inner_instructions PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_core_inner_instructions_transaction FOREIGN KEY (transaction_id) REFERENCES kb_sol_core_transactions(id) ON DELETE CASCADE,
|
||||
CONSTRAINT ck_kb_sol_core_inner_instructions_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_inner_instructions_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_inner_instructions_parent_not_empty CHECK (length(btrim(parent_instruction_path)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_inner_instructions_path_not_empty CHECK (length(btrim(instruction_path)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_inner_instructions_program_not_empty CHECK (length(btrim(program_id)) > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_core_inner_instructions_sig_path
|
||||
ON kb_sol_core_inner_instructions (signature, instruction_path);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_inner_instructions_parent
|
||||
ON kb_sol_core_inner_instructions (signature, parent_instruction_path);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_inner_instructions_program
|
||||
ON kb_sol_core_inner_instructions (program_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_core_logs (
|
||||
id BIGSERIAL,
|
||||
transaction_id BIGINT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
log_index INTEGER NOT NULL,
|
||||
instruction_path TEXT,
|
||||
program_id TEXT,
|
||||
log_text TEXT,
|
||||
log_text_hash TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_core_logs PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_core_logs_transaction FOREIGN KEY (transaction_id) REFERENCES kb_sol_core_transactions(id) ON DELETE CASCADE,
|
||||
CONSTRAINT ck_kb_sol_core_logs_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_logs_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_logs_index_non_negative CHECK (log_index >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_logs_text_not_empty CHECK (log_text IS NULL OR length(btrim(log_text)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_logs_path_not_empty CHECK (instruction_path IS NULL OR length(btrim(instruction_path)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_logs_program_not_empty CHECK (program_id IS NULL OR length(btrim(program_id)) > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_core_logs_sig_index
|
||||
ON kb_sol_core_logs (signature, log_index);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_logs_program
|
||||
ON kb_sol_core_logs (program_id)
|
||||
WHERE program_id IS NOT NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_logs_path
|
||||
ON kb_sol_core_logs (signature, instruction_path)
|
||||
WHERE instruction_path IS NOT NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_core_balance_changes (
|
||||
id BIGSERIAL,
|
||||
transaction_id BIGINT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
balance_change_index INTEGER NOT NULL,
|
||||
balance_kind TEXT NOT NULL,
|
||||
account_index INTEGER,
|
||||
account_key TEXT,
|
||||
mint TEXT,
|
||||
owner TEXT,
|
||||
pre_balance_json JSONB,
|
||||
post_balance_json JSONB,
|
||||
delta_json JSONB,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_core_balance_changes PRIMARY KEY (id),
|
||||
CONSTRAINT fk_kb_sol_core_balance_changes_transaction FOREIGN KEY (transaction_id) REFERENCES kb_sol_core_transactions(id) ON DELETE CASCADE,
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_signature_not_empty CHECK (length(btrim(signature)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_index_non_negative CHECK (balance_change_index >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_account_index_non_negative CHECK (account_index IS NULL OR account_index >= 0),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_kind CHECK (balance_kind IN ('native_lamports', 'token_amount')),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_account_not_empty CHECK (account_key IS NULL OR length(btrim(account_key)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_mint_not_empty CHECK (mint IS NULL OR length(btrim(mint)) > 0),
|
||||
CONSTRAINT ck_kb_sol_core_balance_changes_owner_not_empty CHECK (owner IS NULL OR length(btrim(owner)) > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_core_balance_changes_sig_index
|
||||
ON kb_sol_core_balance_changes (signature, balance_change_index);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_balance_changes_account
|
||||
ON kb_sol_core_balance_changes (account_key)
|
||||
WHERE account_key IS NOT NULL;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_core_balance_changes_mint
|
||||
ON kb_sol_core_balance_changes (mint)
|
||||
WHERE mint IS NOT NULL;
|
||||
|
||||
@@ -0,0 +1,41 @@
|
||||
-- file: kb_store_pg/migrations/0003_processing_ledger.sql
|
||||
-- version: 1
|
||||
|
||||
-- Current processing ledger for canonical to core extraction and later pipeline stages.
|
||||
-- This migration uses the current PostgreSQL schema from the active profile.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_ops_processing_ledger (
|
||||
id BIGSERIAL,
|
||||
stage TEXT NOT NULL,
|
||||
processor_name TEXT NOT NULL,
|
||||
processor_version TEXT NOT NULL,
|
||||
input_key TEXT NOT NULL,
|
||||
input_hash TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
started_at TIMESTAMPTZ,
|
||||
finished_at TIMESTAMPTZ,
|
||||
error_code TEXT,
|
||||
error_message TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_ops_processing_ledger PRIMARY KEY (id),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_stage_not_empty CHECK (length(btrim(stage)) > 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_processor_name_not_empty CHECK (length(btrim(processor_name)) > 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_processor_version_not_empty CHECK (length(btrim(processor_version)) > 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_input_key_not_empty CHECK (length(btrim(input_key)) > 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_input_hash_not_empty CHECK (length(btrim(input_hash)) > 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_status CHECK (status IN ('running', 'succeeded', 'failed')),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_attempt_count_non_negative CHECK (attempt_count >= 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_error_code_not_empty CHECK (error_code IS NULL OR length(btrim(error_code)) > 0),
|
||||
CONSTRAINT ck_kb_sol_ops_processing_ledger_error_message_not_empty CHECK (error_message IS NULL OR length(btrim(error_message)) > 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_ops_processing_ledger_identity
|
||||
ON kb_sol_ops_processing_ledger (stage, processor_name, processor_version, input_key);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_ops_processing_ledger_status
|
||||
ON kb_sol_ops_processing_ledger (stage, processor_name, status, updated_at);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_ops_processing_ledger_input_hash
|
||||
ON kb_sol_ops_processing_ledger (input_hash);
|
||||
@@ -0,0 +1,132 @@
|
||||
-- file: kb_store_pg/migrations/0004_decode_materialization_store.sql
|
||||
-- version: 4
|
||||
|
||||
-- Common versioned decode, coverage and materialization store.
|
||||
-- This migration uses the current PostgreSQL schema from the active profile.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_decode_events (
|
||||
id BIGSERIAL,
|
||||
processor_name TEXT NOT NULL,
|
||||
processor_version TEXT NOT NULL,
|
||||
input_key TEXT NOT NULL,
|
||||
input_hash TEXT NOT NULL,
|
||||
event_key TEXT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
instruction_path TEXT NOT NULL,
|
||||
program_id TEXT NOT NULL,
|
||||
protocol_code TEXT NOT NULL,
|
||||
surface_code TEXT NOT NULL,
|
||||
event_code TEXT NOT NULL,
|
||||
event_name TEXT NOT NULL,
|
||||
event_family TEXT NOT NULL,
|
||||
source_kind TEXT NOT NULL,
|
||||
confidence TEXT NOT NULL,
|
||||
proof_kind TEXT NOT NULL,
|
||||
proof_jsonb JSONB NOT NULL,
|
||||
payload_jsonb JSONB NOT NULL,
|
||||
transaction_failed BOOLEAN NOT NULL,
|
||||
transaction_error_jsonb JSONB,
|
||||
observation_committed BOOLEAN NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_decode_events PRIMARY KEY (id),
|
||||
CONSTRAINT ck_kb_sol_decode_events_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_decode_events_failed_not_committed CHECK (NOT transaction_failed OR NOT observation_committed)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_decode_events_processor_input_event
|
||||
ON kb_sol_decode_events (processor_name, processor_version, input_key, event_key);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_decode_events_signature_path
|
||||
ON kb_sol_decode_events (signature, instruction_path);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_decode_events_program_surface
|
||||
ON kb_sol_decode_events (program_id, surface_code, event_code);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_decode_events_family_commit
|
||||
ON kb_sol_decode_events (event_family, transaction_failed, observation_committed);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_decode_coverage_declarations (
|
||||
id BIGSERIAL,
|
||||
processor_name TEXT NOT NULL,
|
||||
processor_version TEXT NOT NULL,
|
||||
program_id TEXT NOT NULL,
|
||||
surface_code TEXT,
|
||||
entry_kind TEXT NOT NULL,
|
||||
entry_code TEXT NOT NULL,
|
||||
discriminator_hex TEXT,
|
||||
historical BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_decode_coverage_declarations PRIMARY KEY (id)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_decode_coverage_declarations_identity
|
||||
ON kb_sol_decode_coverage_declarations (
|
||||
processor_name,
|
||||
processor_version,
|
||||
program_id,
|
||||
COALESCE(surface_code, ''),
|
||||
entry_kind,
|
||||
entry_code,
|
||||
COALESCE(discriminator_hex, '')
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_decode_coverage_declarations_program
|
||||
ON kb_sol_decode_coverage_declarations (program_id, processor_name, processor_version);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_decode_coverage_observations (
|
||||
id BIGSERIAL,
|
||||
processor_name TEXT NOT NULL,
|
||||
processor_version TEXT NOT NULL,
|
||||
input_key TEXT NOT NULL,
|
||||
input_hash TEXT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
instruction_path TEXT NOT NULL,
|
||||
program_id TEXT NOT NULL,
|
||||
surface_code TEXT,
|
||||
entry_code TEXT,
|
||||
discriminator_hex TEXT,
|
||||
status TEXT NOT NULL,
|
||||
recognized BOOLEAN NOT NULL,
|
||||
decoded_count INTEGER NOT NULL DEFAULT 0,
|
||||
materialized_count INTEGER NOT NULL DEFAULT 0,
|
||||
error_count INTEGER NOT NULL DEFAULT 0,
|
||||
transaction_failed BOOLEAN NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_decode_coverage_observations PRIMARY KEY (id),
|
||||
CONSTRAINT ck_kb_sol_decode_coverage_observations_slot_non_negative CHECK (slot >= 0),
|
||||
CONSTRAINT ck_kb_sol_decode_coverage_observations_counts_non_negative CHECK (decoded_count >= 0 AND materialized_count >= 0 AND error_count >= 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_decode_coverage_observations_identity
|
||||
ON kb_sol_decode_coverage_observations (processor_name, processor_version, input_key);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_decode_coverage_observations_program_status
|
||||
ON kb_sol_decode_coverage_observations (program_id, status, transaction_failed);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_decode_coverage_observations_entry
|
||||
ON kb_sol_decode_coverage_observations (processor_name, processor_version, surface_code, entry_code);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS kb_sol_mat_events (
|
||||
id BIGSERIAL,
|
||||
processor_name TEXT NOT NULL,
|
||||
processor_version TEXT NOT NULL,
|
||||
input_key TEXT NOT NULL,
|
||||
input_hash TEXT NOT NULL,
|
||||
output_key TEXT NOT NULL,
|
||||
source_event_key TEXT NOT NULL,
|
||||
source_decoder_name TEXT NOT NULL,
|
||||
source_decoder_version TEXT NOT NULL,
|
||||
source_decode_input_key TEXT NOT NULL,
|
||||
signature TEXT NOT NULL,
|
||||
slot BIGINT NOT NULL,
|
||||
materialized_family TEXT NOT NULL,
|
||||
payload_jsonb JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
CONSTRAINT pk_kb_sol_mat_events PRIMARY KEY (id),
|
||||
CONSTRAINT ck_kb_sol_mat_events_slot_non_negative CHECK (slot >= 0)
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS ux_kb_sol_mat_events_processor_input_output
|
||||
ON kb_sol_mat_events (processor_name, processor_version, input_key, output_key);
|
||||
CREATE INDEX IF NOT EXISTS ix_kb_sol_mat_events_signature_family
|
||||
ON kb_sol_mat_events (source_decoder_name, source_decoder_version, source_decode_input_key, signature, materialized_family);
|
||||
@@ -0,0 +1,7 @@
|
||||
// file: kb_store_pg/src/constants.rs
|
||||
// version: 1
|
||||
|
||||
//! Local constants for the `kb_store_pg` crate.
|
||||
|
||||
/// Canonical tracing target for this crate.
|
||||
pub(crate) const TRACING_TARGET: &str = "kb_store_pg";
|
||||
118
migration/khadhroony-bot2-reference/kb_store_pg/src/lib.rs
Normal file
118
migration/khadhroony-bot2-reference/kb_store_pg/src/lib.rs
Normal file
@@ -0,0 +1,118 @@
|
||||
// file: kb_store_pg/src/lib.rs
|
||||
// version: 15
|
||||
|
||||
//! PostgreSQL storage implementation boundary.
|
||||
#![warn(missing_docs)]
|
||||
#![deny(unreachable_pub)]
|
||||
#![forbid(unsafe_code)]
|
||||
|
||||
mod constants;
|
||||
mod migrations;
|
||||
mod pg_store;
|
||||
mod queries;
|
||||
mod replay_candidates;
|
||||
mod repositories;
|
||||
#[cfg(test)]
|
||||
mod test_serial;
|
||||
|
||||
/// Canonical tracing target for this crate.
|
||||
pub(crate) use crate::constants::TRACING_TARGET;
|
||||
|
||||
/// Allowed Solana table domains encoded in table names.
|
||||
pub use crate::migrations::ALLOWED_SOLANA_TABLE_DOMAINS;
|
||||
/// Core account key table name.
|
||||
pub use crate::migrations::CORE_ACCOUNT_KEYS_TABLE_NAME;
|
||||
/// Core balance change table name.
|
||||
pub use crate::migrations::CORE_BALANCE_CHANGES_TABLE_NAME;
|
||||
/// Core inner instruction table name.
|
||||
pub use crate::migrations::CORE_INNER_INSTRUCTIONS_TABLE_NAME;
|
||||
/// Core instruction table name.
|
||||
pub use crate::migrations::CORE_INSTRUCTIONS_TABLE_NAME;
|
||||
/// Core log table name.
|
||||
pub use crate::migrations::CORE_LOGS_TABLE_NAME;
|
||||
/// Core Solana table names introduced by `0.2.4`.
|
||||
pub use crate::migrations::CORE_STORE_TABLE_NAMES;
|
||||
/// Core transaction table name.
|
||||
pub use crate::migrations::CORE_TRANSACTIONS_TABLE_NAME;
|
||||
/// Machine-readable decoder coverage declaration table name.
|
||||
pub use crate::migrations::DECODE_COVERAGE_DECLARATIONS_TABLE_NAME;
|
||||
/// Observed decoder coverage classification table name.
|
||||
pub use crate::migrations::DECODE_COVERAGE_OBSERVATIONS_TABLE_NAME;
|
||||
/// Versioned decoded event table name.
|
||||
pub use crate::migrations::DECODE_EVENTS_TABLE_NAME;
|
||||
/// Decode and materialization table names introduced by `0.4.0`.
|
||||
pub use crate::migrations::DECODE_STORE_TABLE_NAMES;
|
||||
/// Default PostgreSQL schema policy.
|
||||
pub use crate::migrations::DEFAULT_SCHEMA_POLICY;
|
||||
/// Versioned materialized output table name.
|
||||
pub use crate::migrations::MATERIALIZED_EVENTS_TABLE_NAME;
|
||||
/// PostgreSQL migration strategy used by this crate.
|
||||
pub use crate::migrations::MIGRATION_STRATEGY;
|
||||
/// Migration table name used by sqlx when migrations are enabled later.
|
||||
pub use crate::migrations::MIGRATION_TABLE_NAME;
|
||||
/// Processing ledger table name.
|
||||
pub use crate::migrations::PROCESSING_LEDGER_TABLE_NAME;
|
||||
/// Table diagnostic metadata.
|
||||
pub use crate::migrations::PostgresTableDiagnosticSpec;
|
||||
/// Canonical transaction acquisition table names active since `0.3.1`.
|
||||
pub use crate::migrations::RAW_STORE_TABLE_NAMES;
|
||||
/// Canonical raw transaction table name.
|
||||
pub use crate::migrations::RAW_TRANSACTIONS_TABLE_NAME;
|
||||
/// Solana application table prefix.
|
||||
pub use crate::migrations::SOLANA_TABLE_PREFIX;
|
||||
/// Advisory lock id used while initializing store schemas.
|
||||
pub use crate::migrations::STORE_SCHEMA_ADVISORY_LOCK_ID;
|
||||
/// Transaction acquisition observation table name.
|
||||
pub use crate::migrations::TRANSACTION_OBSERVATIONS_TABLE_NAME;
|
||||
/// Core store SQL statements applied by the idempotent initializer.
|
||||
pub use crate::migrations::core_store_schema_statements;
|
||||
/// Core table diagnostic specs.
|
||||
pub use crate::migrations::core_store_table_diagnostic_specs;
|
||||
/// Decode and materialization SQL statements applied by the idempotent initializer.
|
||||
pub use crate::migrations::decode_store_schema_statements;
|
||||
/// Decode and materialization table diagnostic specs.
|
||||
pub use crate::migrations::decode_store_table_diagnostic_specs;
|
||||
/// Returns true when a Solana table name follows the canonical prefix and domain rules.
|
||||
pub use crate::migrations::is_valid_solana_table_name;
|
||||
/// Canonical transaction acquisition SQL statements applied by the initializer.
|
||||
pub use crate::migrations::raw_store_schema_statements;
|
||||
/// Canonical transaction acquisition diagnostic specs.
|
||||
pub use crate::migrations::raw_store_table_diagnostic_specs;
|
||||
/// Validates core store table names.
|
||||
pub use crate::migrations::validate_core_store_table_names;
|
||||
/// Validates decode and materialization store table names.
|
||||
pub use crate::migrations::validate_decode_store_table_names;
|
||||
/// Validates canonical transaction acquisition table names.
|
||||
pub use crate::migrations::validate_raw_store_table_names;
|
||||
/// Validates a canonical Solana table name.
|
||||
pub use crate::migrations::validate_solana_table_name;
|
||||
/// PostgreSQL diagnostic snapshot.
|
||||
pub use crate::pg_store::PostgresBackendDiagnostics;
|
||||
/// Minimal PostgreSQL store handle.
|
||||
pub use crate::pg_store::PostgresStore;
|
||||
/// PostgreSQL store connection options.
|
||||
pub use crate::pg_store::PostgresStoreOptions;
|
||||
/// PostgreSQL table diagnostic snapshot.
|
||||
pub use crate::pg_store::PostgresTableDiagnostics;
|
||||
/// PostgreSQL table statistics snapshot.
|
||||
pub use crate::pg_store::PostgresTableStatistics;
|
||||
/// Returns a DSN masked for logs and UI diagnostics.
|
||||
pub use crate::pg_store::mask_postgres_dsn;
|
||||
/// Maximum number of rows returned by one replay candidate query.
|
||||
pub use crate::replay_candidates::MAX_REPLAY_CANDIDATE_ROWS;
|
||||
/// Bounded read-only filter for core entity summaries.
|
||||
pub use crate::replay_candidates::PostgresReplayEntityFilter;
|
||||
/// Core entity kind used while filtering replay candidates.
|
||||
pub use crate::replay_candidates::PostgresReplayEntityKind;
|
||||
/// Aggregated mint, owner or account-key occurrences from core tables.
|
||||
pub use crate::replay_candidates::PostgresReplayEntitySummary;
|
||||
/// Bounded read-only filter for program summaries.
|
||||
pub use crate::replay_candidates::PostgresReplayProgramFilter;
|
||||
/// Program occurrence scope used while filtering replay candidates.
|
||||
pub use crate::replay_candidates::PostgresReplayProgramScope;
|
||||
/// Aggregated program occurrences across outer, inner and linked logs.
|
||||
pub use crate::replay_candidates::PostgresReplayProgramSummary;
|
||||
/// One raw transaction candidate enriched with core and ledger diagnostics.
|
||||
pub use crate::replay_candidates::PostgresReplayTransactionCandidate;
|
||||
/// Bounded read-only filter for transaction replay candidates.
|
||||
pub use crate::replay_candidates::PostgresReplayTransactionFilter;
|
||||
1158
migration/khadhroony-bot2-reference/kb_store_pg/src/migrations.rs
Normal file
1158
migration/khadhroony-bot2-reference/kb_store_pg/src/migrations.rs
Normal file
File diff suppressed because it is too large
Load Diff
704
migration/khadhroony-bot2-reference/kb_store_pg/src/pg_store.rs
Normal file
704
migration/khadhroony-bot2-reference/kb_store_pg/src/pg_store.rs
Normal file
@@ -0,0 +1,704 @@
|
||||
// file: kb_store_pg/src/pg_store.rs
|
||||
// version: 15
|
||||
|
||||
//! Store implementation scaffold for the `kb_store_pg` crate.
|
||||
|
||||
/// PostgreSQL store connection options.
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct PostgresStoreOptions {
|
||||
/// Database URL or DSN.
|
||||
pub database_url: std::string::String,
|
||||
/// Maximum connection count.
|
||||
pub max_connections: u32,
|
||||
/// Connection timeout in milliseconds.
|
||||
pub connect_timeout_ms: u64,
|
||||
/// Enables idempotent raw schema initialization at startup.
|
||||
pub auto_initialize_schema: bool,
|
||||
}
|
||||
|
||||
impl PostgresStoreOptions {
|
||||
/// Builds options from typed PostgreSQL configuration.
|
||||
pub fn from_postgres_config(config: &kb_config::PostgresConfig) -> kb_core::Result<Self> {
|
||||
return crate::pg_store::PostgresStoreOptions::new(
|
||||
config.url.clone(),
|
||||
config.max_connections,
|
||||
config.connect_timeout_ms,
|
||||
config.auto_initialize_schema,
|
||||
);
|
||||
}
|
||||
|
||||
/// Builds options from the selected database backend configuration.
|
||||
pub fn from_database_config(config: &kb_config::DatabaseConfig) -> kb_core::Result<Self> {
|
||||
if config.backend != "postgres" {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"postgres store requires database.backend to be postgres",
|
||||
));
|
||||
}
|
||||
return crate::pg_store::PostgresStoreOptions::from_postgres_config(&config.postgres);
|
||||
}
|
||||
|
||||
/// Builds options from one resolved profile.
|
||||
pub fn from_profile_config(config: &kb_config::ProfileConfig) -> kb_core::Result<Self> {
|
||||
return crate::pg_store::PostgresStoreOptions::from_database_config(&config.database);
|
||||
}
|
||||
|
||||
/// Builds options from the active profile of an application configuration.
|
||||
pub fn from_active_app_config(config: &kb_config::AppConfig) -> kb_core::Result<Self> {
|
||||
let profile_result = kb_config::active_profile(config);
|
||||
return match profile_result {
|
||||
std::result::Result::Ok(profile) => {
|
||||
crate::pg_store::PostgresStoreOptions::from_profile_config(profile)
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Creates validated PostgreSQL store options.
|
||||
pub fn new(
|
||||
database_url: impl std::convert::Into<std::string::String>,
|
||||
max_connections: u32,
|
||||
connect_timeout_ms: u64,
|
||||
auto_initialize_schema: bool,
|
||||
) -> kb_core::Result<Self> {
|
||||
let database_url_value = database_url.into();
|
||||
if database_url_value.trim().is_empty() {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"postgres database url must not be empty",
|
||||
));
|
||||
}
|
||||
if max_connections == 0 {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"postgres max_connections must be greater than zero",
|
||||
));
|
||||
}
|
||||
if connect_timeout_ms == 0 {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"postgres connect_timeout_ms must be greater than zero",
|
||||
));
|
||||
}
|
||||
return std::result::Result::Ok(Self {
|
||||
database_url: database_url_value,
|
||||
max_connections,
|
||||
connect_timeout_ms,
|
||||
auto_initialize_schema,
|
||||
});
|
||||
}
|
||||
|
||||
/// Returns a DSN masked for diagnostics.
|
||||
pub fn masked_dsn(&self) -> std::string::String {
|
||||
return crate::mask_postgres_dsn(self.database_url.as_str());
|
||||
}
|
||||
}
|
||||
|
||||
/// PostgreSQL diagnostic snapshot.
|
||||
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
|
||||
pub struct PostgresBackendDiagnostics {
|
||||
/// Backend descriptor safe for UI display.
|
||||
pub descriptor: kb_store_core::StoreBackendDescriptor,
|
||||
/// Backend health snapshot.
|
||||
pub health: kb_store_core::StoreHealthSnapshot,
|
||||
/// Migration status snapshot.
|
||||
pub migrations: kb_store_core::StoreMigrationSnapshot,
|
||||
/// Full PostgreSQL server version string when available.
|
||||
pub server_version: std::option::Option<std::string::String>,
|
||||
}
|
||||
|
||||
/// Read-only statistics for one PostgreSQL table.
|
||||
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
|
||||
pub struct PostgresTableStatistics {
|
||||
/// Number of rows currently stored in the table.
|
||||
pub row_count: i64,
|
||||
/// Lowest observed Solana slot when the table contains a slot column and rows.
|
||||
pub min_slot: std::option::Option<i64>,
|
||||
/// Highest observed Solana slot when the table contains a slot column and rows.
|
||||
pub max_slot: std::option::Option<i64>,
|
||||
/// Latest insertion timestamp rendered by PostgreSQL for UI diagnostics.
|
||||
pub latest_created_at: std::option::Option<std::string::String>,
|
||||
}
|
||||
|
||||
/// Read-only diagnostics for one expected PostgreSQL table.
|
||||
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
|
||||
pub struct PostgresTableDiagnostics {
|
||||
/// Expected table name.
|
||||
pub table_name: std::string::String,
|
||||
/// Logical Solana domain encoded in the table name.
|
||||
pub domain: std::string::String,
|
||||
/// Human-readable role of the table.
|
||||
pub role: std::string::String,
|
||||
/// Whether the table exists in the current PostgreSQL search path.
|
||||
pub exists: bool,
|
||||
/// Table statistics when the table exists.
|
||||
pub statistics: std::option::Option<crate::PostgresTableStatistics>,
|
||||
}
|
||||
|
||||
/// PostgreSQL store handle.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct PostgresStore {
|
||||
options: crate::PostgresStoreOptions,
|
||||
pool: sqlx::PgPool,
|
||||
}
|
||||
|
||||
impl PostgresStore {
|
||||
/// Connects to PostgreSQL from typed store options.
|
||||
pub async fn connect(options: crate::PostgresStoreOptions) -> kb_core::Result<Self> {
|
||||
let pool_options = sqlx::postgres::PgPoolOptions::new()
|
||||
.max_connections(options.max_connections)
|
||||
.acquire_timeout(std::time::Duration::from_millis(options.connect_timeout_ms));
|
||||
let connect_result = pool_options.connect(options.database_url.as_str()).await;
|
||||
return match connect_result {
|
||||
std::result::Result::Ok(pool) => {
|
||||
let store = Self { options, pool };
|
||||
if store.options.auto_initialize_schema {
|
||||
let schema_result = store.initialize_store_schema().await;
|
||||
if let std::result::Result::Err(error) = schema_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
}
|
||||
std::result::Result::Ok(store)
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(
|
||||
format!("postgres connection failed: {error}"),
|
||||
)),
|
||||
};
|
||||
}
|
||||
|
||||
/// Connects to PostgreSQL from typed configuration.
|
||||
pub async fn connect_from_config(config: &kb_config::PostgresConfig) -> kb_core::Result<Self> {
|
||||
let options_result = crate::PostgresStoreOptions::from_postgres_config(config);
|
||||
return match options_result {
|
||||
std::result::Result::Ok(options) => crate::PostgresStore::connect(options).await,
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Connects to PostgreSQL from the selected database backend configuration.
|
||||
pub async fn connect_from_database_config(
|
||||
config: &kb_config::DatabaseConfig,
|
||||
) -> kb_core::Result<Self> {
|
||||
let options_result = crate::PostgresStoreOptions::from_database_config(config);
|
||||
return match options_result {
|
||||
std::result::Result::Ok(options) => crate::PostgresStore::connect(options).await,
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Connects to PostgreSQL from one resolved profile.
|
||||
pub async fn connect_from_profile_config(
|
||||
config: &kb_config::ProfileConfig,
|
||||
) -> kb_core::Result<Self> {
|
||||
let options_result = crate::PostgresStoreOptions::from_profile_config(config);
|
||||
return match options_result {
|
||||
std::result::Result::Ok(options) => crate::PostgresStore::connect(options).await,
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Connects to PostgreSQL from the active profile of an application configuration.
|
||||
pub async fn connect_from_active_app_config(
|
||||
config: &kb_config::AppConfig,
|
||||
) -> kb_core::Result<Self> {
|
||||
let options_result = crate::PostgresStoreOptions::from_active_app_config(config);
|
||||
return match options_result {
|
||||
std::result::Result::Ok(options) => crate::PostgresStore::connect(options).await,
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Creates a store handle from an existing PostgreSQL pool.
|
||||
pub fn from_pool(options: crate::PostgresStoreOptions, pool: sqlx::PgPool) -> Self {
|
||||
return Self { options, pool };
|
||||
}
|
||||
|
||||
/// Returns the underlying PostgreSQL pool.
|
||||
pub fn pool(&self) -> &sqlx::PgPool {
|
||||
return &self.pool;
|
||||
}
|
||||
|
||||
/// Returns the connection options used to create this store.
|
||||
pub fn options(&self) -> &crate::PostgresStoreOptions {
|
||||
return &self.options;
|
||||
}
|
||||
|
||||
/// Applies each idempotent store schema once per invocation in dependency order.
|
||||
pub async fn initialize_store_schema(&self) -> kb_core::Result<()> {
|
||||
let raw_result = crate::queries::apply_raw_store_schema(&self.pool).await;
|
||||
if let std::result::Result::Err(error) = raw_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let core_result = crate::queries::apply_core_store_schema(&self.pool).await;
|
||||
if let std::result::Result::Err(error) = core_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
return crate::queries::apply_decode_store_schema(&self.pool).await;
|
||||
}
|
||||
|
||||
/// Applies the idempotent minimal raw Solana store schema.
|
||||
pub async fn initialize_raw_store_schema(&self) -> kb_core::Result<()> {
|
||||
return crate::queries::apply_raw_store_schema(&self.pool).await;
|
||||
}
|
||||
|
||||
/// Applies the idempotent minimal core Solana store schema.
|
||||
pub async fn initialize_core_store_schema(&self) -> kb_core::Result<()> {
|
||||
let raw_result = self.initialize_raw_store_schema().await;
|
||||
if let std::result::Result::Err(error) = raw_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
return crate::queries::apply_core_store_schema(&self.pool).await;
|
||||
}
|
||||
|
||||
/// Applies the idempotent common decode and materialization store schema.
|
||||
pub async fn initialize_decode_store_schema(&self) -> kb_core::Result<()> {
|
||||
let core_result = self.initialize_core_store_schema().await;
|
||||
if let std::result::Result::Err(error) = core_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
return crate::queries::apply_decode_store_schema(&self.pool).await;
|
||||
}
|
||||
|
||||
/// Reads a UI-safe backend descriptor.
|
||||
pub async fn backend_descriptor(
|
||||
&self,
|
||||
) -> kb_core::Result<kb_store_core::StoreBackendDescriptor> {
|
||||
let schema_result = crate::queries::load_current_schema(&self.pool).await;
|
||||
return match schema_result {
|
||||
std::result::Result::Ok(schema) => kb_store_core::StoreBackendDescriptor::new(
|
||||
kb_store_core::StoreBackendKind::Postgres,
|
||||
"postgres",
|
||||
std::option::Option::Some(self.options.masked_dsn()),
|
||||
std::option::Option::Some(schema),
|
||||
),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Reads a PostgreSQL health snapshot.
|
||||
pub async fn health_snapshot(&self) -> kb_core::Result<kb_store_core::StoreHealthSnapshot> {
|
||||
let health_result = crate::queries::run_health_check(&self.pool).await;
|
||||
return match health_result {
|
||||
std::result::Result::Ok(()) => kb_store_core::StoreHealthSnapshot::new(
|
||||
"postgres",
|
||||
kb_store_core::StoreHealthStatus::Healthy,
|
||||
std::option::Option::Some(std::string::String::from("SELECT 1 succeeded")),
|
||||
),
|
||||
std::result::Result::Err(error) => kb_store_core::StoreHealthSnapshot::new(
|
||||
"postgres",
|
||||
kb_store_core::StoreHealthStatus::Unhealthy,
|
||||
std::option::Option::Some(error.to_string()),
|
||||
),
|
||||
};
|
||||
}
|
||||
|
||||
/// Reads a non-destructive migration snapshot.
|
||||
pub async fn migration_snapshot(
|
||||
&self,
|
||||
) -> kb_core::Result<kb_store_core::StoreMigrationSnapshot> {
|
||||
let migration_table_result = crate::queries::load_migration_table_name(&self.pool).await;
|
||||
return match migration_table_result {
|
||||
std::result::Result::Ok(std::option::Option::None) => {
|
||||
std::result::Result::Ok(kb_store_core::StoreMigrationSnapshot::new(
|
||||
kb_store_core::StoreMigrationStatus::NotInitialized,
|
||||
std::option::Option::None,
|
||||
std::vec::Vec::new(),
|
||||
std::option::Option::Some(std::string::String::from(
|
||||
"no sqlx migration table detected; 0.3.1 canonical acquisition/core schemas use idempotent crate-managed DDL",
|
||||
)),
|
||||
))
|
||||
},
|
||||
std::result::Result::Ok(std::option::Option::Some(_table_name)) => {
|
||||
self.migration_snapshot_from_existing_table().await
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
|
||||
/// Reads a complete PostgreSQL diagnostic snapshot.
|
||||
pub async fn backend_diagnostics(&self) -> kb_core::Result<crate::PostgresBackendDiagnostics> {
|
||||
let descriptor_result = self.backend_descriptor().await;
|
||||
let descriptor = match descriptor_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let health_result = self.health_snapshot().await;
|
||||
let health = match health_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let migrations_result = self.migration_snapshot().await;
|
||||
let migrations = match migrations_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let server_version = match crate::queries::load_server_version(&self.pool).await {
|
||||
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||
std::result::Result::Err(_error) => std::option::Option::None,
|
||||
};
|
||||
return std::result::Result::Ok(crate::PostgresBackendDiagnostics {
|
||||
descriptor,
|
||||
health,
|
||||
migrations,
|
||||
server_version,
|
||||
});
|
||||
}
|
||||
|
||||
/// Lists bounded raw transaction candidates enriched with core and ledger diagnostics.
|
||||
pub async fn replay_transaction_candidates(
|
||||
&self,
|
||||
filter: &crate::PostgresReplayTransactionFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresReplayTransactionCandidate>> {
|
||||
return crate::queries::list_replay_transaction_candidates(&self.pool, filter).await;
|
||||
}
|
||||
|
||||
/// Lists bounded program summaries across outer, inner and reliably linked logs.
|
||||
pub async fn replay_program_summaries(
|
||||
&self,
|
||||
filter: &crate::PostgresReplayProgramFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresReplayProgramSummary>> {
|
||||
return crate::queries::list_replay_program_summaries(&self.pool, filter).await;
|
||||
}
|
||||
|
||||
/// Lists bounded mint, owner or account-key summaries from core tables.
|
||||
pub async fn replay_entity_summaries(
|
||||
&self,
|
||||
filter: &crate::PostgresReplayEntityFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresReplayEntitySummary>> {
|
||||
return crate::queries::list_replay_entity_summaries(&self.pool, filter).await;
|
||||
}
|
||||
|
||||
/// Reads diagnostics for raw Solana store tables without changing the schema.
|
||||
pub async fn raw_table_diagnostics(
|
||||
&self,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresTableDiagnostics>> {
|
||||
let specs = crate::migrations::raw_store_table_diagnostic_specs();
|
||||
return self.table_diagnostics(&specs).await;
|
||||
}
|
||||
|
||||
/// Reads diagnostics for core Solana store tables without changing the schema.
|
||||
pub async fn core_table_diagnostics(
|
||||
&self,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresTableDiagnostics>> {
|
||||
let specs = crate::migrations::core_store_table_diagnostic_specs();
|
||||
return self.table_diagnostics(&specs).await;
|
||||
}
|
||||
|
||||
/// Reads diagnostics for decode and materialization store tables without changing the schema.
|
||||
pub async fn decode_table_diagnostics(
|
||||
&self,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresTableDiagnostics>> {
|
||||
let specs = crate::migrations::decode_store_table_diagnostic_specs();
|
||||
return self.table_diagnostics(&specs).await;
|
||||
}
|
||||
|
||||
/// Reads diagnostics for every known raw/core/decode Solana store table.
|
||||
pub async fn known_table_diagnostics(
|
||||
&self,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresTableDiagnostics>> {
|
||||
let mut diagnostics = std::vec::Vec::new();
|
||||
let raw_result = self.raw_table_diagnostics().await;
|
||||
let raw_tables = match raw_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
for table in raw_tables {
|
||||
diagnostics.push(table);
|
||||
}
|
||||
let core_result = self.core_table_diagnostics().await;
|
||||
let core_tables = match core_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
for table in core_tables {
|
||||
diagnostics.push(table);
|
||||
}
|
||||
let decode_result = self.decode_table_diagnostics().await;
|
||||
let decode_tables = match decode_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
for table in decode_tables {
|
||||
diagnostics.push(table);
|
||||
}
|
||||
return std::result::Result::Ok(diagnostics);
|
||||
}
|
||||
|
||||
async fn table_diagnostics(
|
||||
&self,
|
||||
specs: &[crate::migrations::PostgresTableDiagnosticSpec],
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresTableDiagnostics>> {
|
||||
let mut diagnostics = std::vec::Vec::new();
|
||||
for spec in specs {
|
||||
let exists_result = crate::queries::table_exists(&self.pool, spec.table_name).await;
|
||||
let exists = match exists_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let statistics = if exists {
|
||||
let statistics_result =
|
||||
crate::queries::load_table_statistics(&self.pool, spec.table_name).await;
|
||||
match statistics_result {
|
||||
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
}
|
||||
} else {
|
||||
std::option::Option::None
|
||||
};
|
||||
diagnostics.push(crate::PostgresTableDiagnostics {
|
||||
table_name: spec.table_name.to_string(),
|
||||
domain: spec.domain.to_string(),
|
||||
role: spec.role.to_string(),
|
||||
exists,
|
||||
statistics,
|
||||
});
|
||||
}
|
||||
return std::result::Result::Ok(diagnostics);
|
||||
}
|
||||
|
||||
async fn migration_snapshot_from_existing_table(
|
||||
&self,
|
||||
) -> kb_core::Result<kb_store_core::StoreMigrationSnapshot> {
|
||||
let version_result = crate::queries::load_latest_migration_version(&self.pool).await;
|
||||
return match version_result {
|
||||
std::result::Result::Ok(current_version) => {
|
||||
std::result::Result::Ok(kb_store_core::StoreMigrationSnapshot::new(
|
||||
kb_store_core::StoreMigrationStatus::Current,
|
||||
current_version,
|
||||
std::vec::Vec::new(),
|
||||
std::option::Option::Some(std::string::String::from(
|
||||
"sqlx migration table detected; canonical acquisition/core schema remains idempotent and crate-managed in 0.3.1",
|
||||
)),
|
||||
))
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns a DSN masked for logs and UI diagnostics.
|
||||
pub fn mask_postgres_dsn(dsn: &str) -> std::string::String {
|
||||
let trimmed_dsn = dsn.trim();
|
||||
if trimmed_dsn.is_empty() {
|
||||
return std::string::String::from("");
|
||||
}
|
||||
let queryless = crate::pg_store::strip_query(trimmed_dsn);
|
||||
return match queryless.split_once("://") {
|
||||
std::option::Option::Some((scheme, remainder)) => {
|
||||
crate::pg_store::mask_scheme_remainder(scheme, remainder, trimmed_dsn.contains('?'))
|
||||
},
|
||||
std::option::Option::None => crate::pg_store::mask_plain_dsn(queryless.as_str()),
|
||||
};
|
||||
}
|
||||
|
||||
fn strip_query(dsn: &str) -> std::string::String {
|
||||
return match dsn.split_once('?') {
|
||||
std::option::Option::Some((prefix, _query)) => std::string::String::from(prefix),
|
||||
std::option::Option::None => std::string::String::from(dsn),
|
||||
};
|
||||
}
|
||||
|
||||
fn mask_scheme_remainder(scheme: &str, remainder: &str, had_query: bool) -> std::string::String {
|
||||
let suffix = crate::pg_store::query_suffix(had_query);
|
||||
return match remainder.rsplit_once('@') {
|
||||
std::option::Option::Some((_userinfo, host_path)) => {
|
||||
format!("{scheme}://***:***@{host_path}{suffix}")
|
||||
},
|
||||
std::option::Option::None => format!("{scheme}://{remainder}{suffix}"),
|
||||
};
|
||||
}
|
||||
|
||||
fn mask_plain_dsn(dsn: &str) -> std::string::String {
|
||||
if dsn.contains("password=") {
|
||||
return std::string::String::from("<postgres-dsn-redacted>");
|
||||
}
|
||||
return std::string::String::from(dsn);
|
||||
}
|
||||
|
||||
fn query_suffix(had_query: bool) -> std::string::String {
|
||||
if had_query {
|
||||
return std::string::String::from("?<redacted>");
|
||||
}
|
||||
return std::string::String::from("");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn options_reject_empty_database_url() {
|
||||
let result = crate::PostgresStoreOptions::new(" ", 1, 1000, false);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn options_reject_zero_max_connections() {
|
||||
let result = crate::PostgresStoreOptions::new("postgres://localhost/db", 0, 1000, false);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn options_reject_zero_connect_timeout() {
|
||||
let result = crate::PostgresStoreOptions::new("postgres://localhost/db", 1, 0, false);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn options_from_database_config_rejects_sqlite_backend() {
|
||||
let config = sample_database_config("sqlite");
|
||||
let result = crate::PostgresStoreOptions::from_database_config(&config);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn options_from_database_config_accepts_postgres_backend() {
|
||||
let config = sample_database_config("postgres");
|
||||
let result = crate::PostgresStoreOptions::from_database_config(&config);
|
||||
let options = match result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected error: {error}"),
|
||||
};
|
||||
assert_eq!(options.database_url, "postgres://localhost/solana");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn options_from_active_app_config_uses_active_profile() {
|
||||
let app_config = kb_config::AppConfig {
|
||||
active_profile: std::string::String::from("selected"),
|
||||
profiles: std::vec![
|
||||
sample_profile_config("ignored", "sqlite"),
|
||||
sample_profile_config("selected", "postgres"),
|
||||
],
|
||||
};
|
||||
let result = crate::PostgresStoreOptions::from_active_app_config(&app_config);
|
||||
let options = match result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected error: {error}"),
|
||||
};
|
||||
assert_eq!(options.database_url, "postgres://localhost/solana");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn optional_postgres_healthcheck_from_env() {
|
||||
let database_url = match std::env::var("KB_POSTGRES_TEST_URL") {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_error) => return,
|
||||
};
|
||||
let _postgres_guard = crate::test_serial::postgres_test_guard().await;
|
||||
let options_result = crate::PostgresStoreOptions::new(database_url, 1, 5000, false);
|
||||
let options = match options_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected options error: {error}"),
|
||||
};
|
||||
let store_result = crate::PostgresStore::connect(options).await;
|
||||
let store = match store_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected connect error: {error}"),
|
||||
};
|
||||
let health_result = store.health_snapshot().await;
|
||||
let health = match health_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected health error: {error}"),
|
||||
};
|
||||
assert_eq!(health.status, kb_store_core::StoreHealthStatus::Healthy);
|
||||
return;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mask_postgres_dsn_masks_userinfo() {
|
||||
let masked = crate::mask_postgres_dsn("postgres://user:secret@localhost:5432/db");
|
||||
assert_eq!(masked, "postgres://***:***@localhost:5432/db");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mask_postgres_dsn_masks_query_string() {
|
||||
let masked =
|
||||
crate::mask_postgres_dsn("postgres://localhost/db?sslmode=require&password=secret");
|
||||
assert_eq!(masked, "postgres://localhost/db?<redacted>");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mask_postgres_dsn_masks_plain_password_dsn() {
|
||||
let masked = crate::mask_postgres_dsn("host=localhost password=secret dbname=khadhroony");
|
||||
assert_eq!(masked, "<postgres-dsn-redacted>");
|
||||
}
|
||||
|
||||
fn sample_database_config(backend: &str) -> kb_config::DatabaseConfig {
|
||||
return kb_config::DatabaseConfig {
|
||||
enabled: true,
|
||||
backend: std::string::String::from(backend),
|
||||
postgres: kb_config::PostgresConfig {
|
||||
url: std::string::String::from("postgres://localhost/solana"),
|
||||
max_connections: 2,
|
||||
connect_timeout_ms: 1000,
|
||||
auto_initialize_schema: false,
|
||||
},
|
||||
sqlite: kb_config::SqliteConfig {
|
||||
path: std::string::String::from("data/test.db"),
|
||||
create_if_missing: true,
|
||||
busy_timeout_ms: 1000,
|
||||
max_connections: 1,
|
||||
auto_initialize_schema: false,
|
||||
use_wal: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
fn sample_profile_config(name: &str, backend: &str) -> kb_config::ProfileConfig {
|
||||
return kb_config::ProfileConfig {
|
||||
name: std::string::String::from(name),
|
||||
app: kb_config::AppSectionConfig {
|
||||
name: std::string::String::from("test"),
|
||||
environment: std::string::String::from("test"),
|
||||
auto_reconnect_default: false,
|
||||
},
|
||||
logging: kb_config::LoggingConfig {
|
||||
default_level: std::string::String::from("info"),
|
||||
targets: std::vec::Vec::new(),
|
||||
target_filters: std::vec::Vec::new(),
|
||||
},
|
||||
database: sample_database_config(backend),
|
||||
data: kb_config::DataConfig {
|
||||
wallets_directory: std::string::String::from("wallets"),
|
||||
logs_directory: std::string::String::from("logs"),
|
||||
},
|
||||
solana: kb_config::SolanaConfig {
|
||||
http_endpoints: std::vec::Vec::new(),
|
||||
ws_endpoints: std::vec::Vec::new(),
|
||||
listeners: kb_config::ListenerConfig {
|
||||
enabled: false,
|
||||
default_commitment: std::string::String::from("confirmed"),
|
||||
log_listeners: std::vec::Vec::new(),
|
||||
program_listeners: std::vec::Vec::new(),
|
||||
account_listeners: std::vec::Vec::new(),
|
||||
},
|
||||
},
|
||||
demo: kb_config::DemoConfig {
|
||||
live_demo_enabled: false,
|
||||
trading_demo_enabled: false,
|
||||
},
|
||||
wallet: kb_config::WalletConfig {
|
||||
wallet_dir: std::string::String::from("wallets"),
|
||||
cluster: std::string::String::from("mainnet-beta"),
|
||||
temporary_wallet_enabled: false,
|
||||
temporary_wallet_alias: std::string::String::from("test-disabled"),
|
||||
temporary_wallet_persist: false,
|
||||
localnet_send_enabled: false,
|
||||
devnet_send_enabled: false,
|
||||
testnet_send_enabled: false,
|
||||
mainnet_send_enabled: false,
|
||||
},
|
||||
execution: kb_config::ExecutionConfig {
|
||||
dry_run_default: true,
|
||||
require_simulation: true,
|
||||
require_operator_confirmation: true,
|
||||
localnet_max_spend_lamports: 0,
|
||||
devnet_max_spend_lamports: 0,
|
||||
testnet_max_spend_lamports: 0,
|
||||
mainnet_max_spend_lamports: 0,
|
||||
max_fee_lamports: 10_000,
|
||||
max_compute_unit_price_micro_lamports: 0,
|
||||
recent_blockhash_max_age_slots: 150,
|
||||
send_max_retries: 3,
|
||||
confirmation_poll_interval_ms: 500,
|
||||
confirmation_max_attempts: 120,
|
||||
devnet_airdrop_max_lamports: 0,
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
// file: kb_store_pg/src/queries.rs
|
||||
// version: 9
|
||||
|
||||
//! PostgreSQL query modules.
|
||||
|
||||
mod core_extraction_queries;
|
||||
mod core_queries;
|
||||
mod decode_pipeline_queries;
|
||||
mod health_queries;
|
||||
mod raw_queries;
|
||||
mod replay_candidate_queries;
|
||||
mod table_diagnostics_queries;
|
||||
|
||||
pub(crate) use crate::queries::core_extraction_queries::is_core_extraction_current;
|
||||
pub(crate) use crate::queries::core_extraction_queries::list_raw_transactions_for_core_extraction;
|
||||
pub(crate) use crate::queries::core_extraction_queries::mark_core_extraction_failed;
|
||||
pub(crate) use crate::queries::core_extraction_queries::persist_core_extraction;
|
||||
pub(crate) use crate::queries::core_queries::apply_core_store_schema;
|
||||
pub(crate) use crate::queries::core_queries::insert_core_account_keys;
|
||||
pub(crate) use crate::queries::core_queries::insert_core_balance_changes;
|
||||
pub(crate) use crate::queries::core_queries::insert_core_inner_instructions;
|
||||
pub(crate) use crate::queries::core_queries::insert_core_instructions;
|
||||
pub(crate) use crate::queries::core_queries::insert_core_logs;
|
||||
pub(crate) use crate::queries::core_queries::insert_core_transaction;
|
||||
pub(crate) use crate::queries::core_queries::list_core_instruction_replay_inputs;
|
||||
pub(crate) use crate::queries::core_queries::list_core_instructions_for_replay;
|
||||
pub(crate) use crate::queries::core_queries::update_core_instruction_lifecycle;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::apply_decode_store_schema;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::is_decode_current;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::list_decode_coverage_summary;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::list_decode_inputs;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::list_materialized_events;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::mark_decode_failed;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::persist_decode_coverage_declarations;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::persist_decode_result;
|
||||
pub(crate) use crate::queries::decode_pipeline_queries::persist_materialization_result;
|
||||
pub(crate) use crate::queries::health_queries::load_current_schema;
|
||||
pub(crate) use crate::queries::health_queries::load_latest_migration_version;
|
||||
pub(crate) use crate::queries::health_queries::load_migration_table_name;
|
||||
pub(crate) use crate::queries::health_queries::load_server_version;
|
||||
pub(crate) use crate::queries::health_queries::run_health_check;
|
||||
pub(crate) use crate::queries::raw_queries::apply_raw_store_schema;
|
||||
pub(crate) use crate::queries::raw_queries::has_raw_transaction_signature;
|
||||
pub(crate) use crate::queries::raw_queries::has_transaction_observation_key;
|
||||
pub(crate) use crate::queries::raw_queries::insert_raw_transaction;
|
||||
pub(crate) use crate::queries::raw_queries::insert_transaction_observation;
|
||||
pub(crate) use crate::queries::raw_queries::update_raw_payload_lifecycle;
|
||||
pub(crate) use crate::queries::replay_candidate_queries::list_replay_entity_summaries;
|
||||
pub(crate) use crate::queries::replay_candidate_queries::list_replay_program_summaries;
|
||||
pub(crate) use crate::queries::replay_candidate_queries::list_replay_transaction_candidates;
|
||||
pub(crate) use crate::queries::table_diagnostics_queries::load_table_statistics;
|
||||
pub(crate) use crate::queries::table_diagnostics_queries::table_exists;
|
||||
@@ -0,0 +1,18 @@
|
||||
<!-- file: kb_store_pg/src/queries/README.md -->
|
||||
<!-- version: 4 -->
|
||||
|
||||
# Queries PostgreSQL
|
||||
|
||||
Ce dossier contient uniquement le SQL, les binds et l'exécution SQL PostgreSQL.
|
||||
|
||||
Interdiction : ne pas placer de struct métier, entity ou DTO dans ce dossier.
|
||||
|
||||
## Modules actuels
|
||||
|
||||
| Module | Rôle |
|
||||
|--------------------------------|---------------------------------------------------------------------------------------|
|
||||
| `health_queries.rs` | Healthcheck, schéma courant, version serveur et diagnostic migrations. |
|
||||
| `raw_queries.rs` | Schéma raw minimal, insertions raw RPC/WS, déduplication et lifecycle raw. |
|
||||
| `core_queries.rs` | Schéma core minimal, insertions core, lifecycle instruction et reconstruction replay. |
|
||||
| `table_diagnostics_queries.rs` | Diagnostics read-only des tables raw/core connues. |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,77 @@
|
||||
// file: kb_store_pg/src/queries/health_queries.rs
|
||||
// version: 2
|
||||
|
||||
//! PostgreSQL health and diagnostic SQL queries.
|
||||
|
||||
pub(crate) async fn run_health_check(pool: &sqlx::PgPool) -> kb_core::Result<()> {
|
||||
let query_result = sqlx::query_scalar::<sqlx::Postgres, i32>("SELECT 1").fetch_one(pool).await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(_value) => std::result::Result::Ok(()),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres healthcheck failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn load_current_schema(
|
||||
pool: &sqlx::PgPool,
|
||||
) -> kb_core::Result<std::string::String> {
|
||||
let query_result =
|
||||
sqlx::query_scalar::<sqlx::Postgres, std::string::String>("SELECT current_schema()")
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(schema) => std::result::Result::Ok(schema),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres current schema query failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn load_server_version(
|
||||
pool: &sqlx::PgPool,
|
||||
) -> kb_core::Result<std::string::String> {
|
||||
let query_result =
|
||||
sqlx::query_scalar::<sqlx::Postgres, std::string::String>("SELECT version()")
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(version) => std::result::Result::Ok(version),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres version query failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn load_migration_table_name(
|
||||
pool: &sqlx::PgPool,
|
||||
) -> kb_core::Result<std::option::Option<std::string::String>> {
|
||||
let query_result =
|
||||
sqlx::query_scalar::<sqlx::Postgres, std::option::Option<std::string::String>>(
|
||||
"SELECT to_regclass('_sqlx_migrations')::text",
|
||||
)
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(table_name) => std::result::Result::Ok(table_name),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres migration table query failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn load_latest_migration_version(
|
||||
pool: &sqlx::PgPool,
|
||||
) -> kb_core::Result<std::option::Option<std::string::String>> {
|
||||
let query_result = sqlx::query_scalar::<sqlx::Postgres, std::option::Option<std::string::String>>(
|
||||
"SELECT version::text FROM _sqlx_migrations WHERE success = true ORDER BY version DESC LIMIT 1",
|
||||
)
|
||||
.fetch_optional(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(version) => std::result::Result::Ok(version.flatten()),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres latest migration query failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,503 @@
|
||||
// file: kb_store_pg/src/queries/raw_queries.rs
|
||||
// version: 7
|
||||
|
||||
//! PostgreSQL canonical transaction and acquisition observation SQL queries.
|
||||
|
||||
pub(crate) async fn apply_raw_store_schema(pool: &sqlx::PgPool) -> kb_core::Result<()> {
|
||||
let validation_result = crate::validate_raw_store_table_names();
|
||||
if let std::result::Result::Err(error) = validation_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let transaction_result = pool.begin().await;
|
||||
let mut transaction = match transaction_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres canonical raw store schema transaction failed: {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let lock_result = sqlx::query("SELECT pg_advisory_xact_lock($1)")
|
||||
.bind(crate::migrations::STORE_SCHEMA_ADVISORY_LOCK_ID)
|
||||
.execute(&mut *transaction)
|
||||
.await;
|
||||
if let std::result::Result::Err(error) = lock_result {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres canonical raw store schema advisory lock failed: {error}"
|
||||
)));
|
||||
}
|
||||
for statement in crate::migrations::raw_store_schema_statements() {
|
||||
let execution_result = sqlx::query(statement).execute(&mut *transaction).await;
|
||||
if let std::result::Result::Err(error) = execution_result {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres canonical raw store schema statement failed: {error}"
|
||||
)));
|
||||
}
|
||||
}
|
||||
let commit_result = transaction.commit().await;
|
||||
if let std::result::Result::Err(error) = commit_result {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres canonical raw store schema commit failed: {error}"
|
||||
)));
|
||||
}
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
|
||||
pub(crate) async fn has_raw_transaction_signature(
|
||||
pool: &sqlx::PgPool,
|
||||
signature: &str,
|
||||
) -> kb_core::Result<bool> {
|
||||
let validation_result = crate::queries::raw_queries::validate_required_text(
|
||||
signature,
|
||||
"canonical raw transaction signature must not be empty",
|
||||
);
|
||||
if let std::result::Result::Err(error) = validation_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let query_result = sqlx::query_scalar::<sqlx::Postgres, bool>(
|
||||
"SELECT EXISTS(SELECT 1 FROM kb_sol_raw_transactions WHERE signature = $1)",
|
||||
)
|
||||
.bind(signature)
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres canonical transaction signature lookup failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn has_transaction_observation_key(
|
||||
pool: &sqlx::PgPool,
|
||||
observation_key: &str,
|
||||
) -> kb_core::Result<bool> {
|
||||
let validation_result = crate::queries::raw_queries::validate_required_text(
|
||||
observation_key,
|
||||
"transaction observation key must not be empty",
|
||||
);
|
||||
if let std::result::Result::Err(error) = validation_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let query_result = sqlx::query_scalar::<sqlx::Postgres, bool>(
|
||||
"SELECT EXISTS(SELECT 1 FROM kb_sol_obs_transaction_observations WHERE observation_key = $1)",
|
||||
)
|
||||
.bind(observation_key)
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres transaction observation lookup failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_raw_transaction(
|
||||
pool: &sqlx::PgPool,
|
||||
input: &kb_store_core::RawTransactionInsert,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
let slot_result = crate::queries::raw_queries::sql_slot_from_u64(input.slot);
|
||||
let slot = match slot_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let version_result = i32::try_from(input.canonical_format_version);
|
||||
let canonical_format_version = match version_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"canonical transaction format version does not fit into SQL INTEGER: {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let query_result = sqlx::query_scalar::<sqlx::Postgres, i64>(
|
||||
"INSERT INTO kb_sol_raw_transactions (signature, slot, canonical_json, canonical_json_hash, canonical_format_version, retention_state, processing_state) VALUES ($1, $2, $3, $4, $5, 'full', 'received') ON CONFLICT (signature) DO NOTHING RETURNING id",
|
||||
)
|
||||
.bind(input.signature.as_str())
|
||||
.bind(slot)
|
||||
.bind(&input.canonical_json)
|
||||
.bind(input.canonical_json_hash.as_deref())
|
||||
.bind(canonical_format_version)
|
||||
.fetch_optional(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(std::option::Option::Some(_id)) => {
|
||||
std::result::Result::Ok(kb_store_core::InsertOutcome::new(1, 0, 0))
|
||||
},
|
||||
std::result::Result::Ok(std::option::Option::None) => {
|
||||
std::result::Result::Ok(kb_store_core::InsertOutcome::new(0, 0, 1))
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres canonical transaction insert failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_transaction_observation(
|
||||
pool: &sqlx::PgPool,
|
||||
input: &kb_store_core::TransactionObservationInsert,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
let slot_result = crate::queries::raw_queries::optional_sql_slot_from_u64(input.slot);
|
||||
let slot = match slot_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let payload_size_result =
|
||||
crate::queries::raw_queries::optional_sql_bigint_from_u64(input.payload_size_bytes);
|
||||
let payload_size_bytes = match payload_size_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let origin = crate::queries::raw_queries::transaction_observation_origin_to_sql(input.origin);
|
||||
let status = crate::queries::raw_queries::transaction_observation_status_to_sql(input.status);
|
||||
let query_result = sqlx::query_scalar::<sqlx::Postgres, i64>(
|
||||
"INSERT INTO kb_sol_obs_transaction_observations (raw_transaction_id, observation_key, signature, slot, provider, endpoint_code, protocol, acquisition_method, origin, commitment, capture_session_id, filter_code, detected_at, received_at, normalized_at, payload_size_bytes, source_payload_hash, status, error_code, error_message) VALUES (COALESCE($1, (SELECT id FROM kb_sol_raw_transactions WHERE signature = $3 LIMIT 1)), $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20) ON CONFLICT (observation_key) DO NOTHING RETURNING id",
|
||||
)
|
||||
.bind(input.raw_transaction_id)
|
||||
.bind(input.observation_key.as_str())
|
||||
.bind(input.signature.as_deref())
|
||||
.bind(slot)
|
||||
.bind(input.provider.as_str())
|
||||
.bind(input.endpoint_code.as_deref())
|
||||
.bind(input.protocol.as_str())
|
||||
.bind(input.acquisition_method.as_str())
|
||||
.bind(origin)
|
||||
.bind(input.commitment.as_deref())
|
||||
.bind(input.capture_session_id.as_deref())
|
||||
.bind(input.filter_code.as_deref())
|
||||
.bind(input.detected_at.as_ref())
|
||||
.bind(input.received_at)
|
||||
.bind(input.normalized_at.as_ref())
|
||||
.bind(payload_size_bytes)
|
||||
.bind(input.source_payload_hash.as_deref())
|
||||
.bind(status)
|
||||
.bind(input.error_code.as_deref())
|
||||
.bind(input.error_message.as_deref())
|
||||
.fetch_optional(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(std::option::Option::Some(_id)) => {
|
||||
std::result::Result::Ok(kb_store_core::InsertOutcome::new(1, 0, 0))
|
||||
},
|
||||
std::result::Result::Ok(std::option::Option::None) => {
|
||||
std::result::Result::Ok(kb_store_core::InsertOutcome::new(0, 0, 1))
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres transaction observation insert failed: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn update_raw_payload_lifecycle(
|
||||
pool: &sqlx::PgPool,
|
||||
mark: &kb_store_core::RawPayloadLifecycleMark,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
if mark.raw_table_name != crate::RAW_TRANSACTIONS_TABLE_NAME {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"raw lifecycle table name is not supported by the PostgreSQL canonical raw store",
|
||||
));
|
||||
}
|
||||
return crate::queries::raw_queries::update_raw_transaction_lifecycle(pool, mark).await;
|
||||
}
|
||||
|
||||
fn raw_retention_state_to_sql(state: kb_store_core::RawPayloadRetentionState) -> &'static str {
|
||||
return match state {
|
||||
kb_store_core::RawPayloadRetentionState::Full => "full",
|
||||
kb_store_core::RawPayloadRetentionState::Compacted => "compacted",
|
||||
kb_store_core::RawPayloadRetentionState::Archived => "archived",
|
||||
kb_store_core::RawPayloadRetentionState::Purged => "purged",
|
||||
};
|
||||
}
|
||||
|
||||
fn raw_processing_state_to_sql(state: kb_store_core::RawPayloadProcessingState) -> &'static str {
|
||||
return match state {
|
||||
kb_store_core::RawPayloadProcessingState::Received => "received",
|
||||
kb_store_core::RawPayloadProcessingState::CoreExtracted => "core_extracted",
|
||||
kb_store_core::RawPayloadProcessingState::Decoded => "decoded",
|
||||
kb_store_core::RawPayloadProcessingState::Materialized => "materialized",
|
||||
kb_store_core::RawPayloadProcessingState::Failed => "failed",
|
||||
};
|
||||
}
|
||||
|
||||
fn transaction_observation_origin_to_sql(
|
||||
origin: kb_store_core::TransactionObservationOrigin,
|
||||
) -> &'static str {
|
||||
return match origin {
|
||||
kb_store_core::TransactionObservationOrigin::Live => "live",
|
||||
kb_store_core::TransactionObservationOrigin::Backfill => "backfill",
|
||||
kb_store_core::TransactionObservationOrigin::Replay => "replay",
|
||||
kb_store_core::TransactionObservationOrigin::Repair => "repair",
|
||||
kb_store_core::TransactionObservationOrigin::Migration => "migration",
|
||||
};
|
||||
}
|
||||
|
||||
fn transaction_observation_status_to_sql(
|
||||
status: kb_store_core::TransactionObservationStatus,
|
||||
) -> &'static str {
|
||||
return match status {
|
||||
kb_store_core::TransactionObservationStatus::Detected => "detected",
|
||||
kb_store_core::TransactionObservationStatus::Received => "received",
|
||||
kb_store_core::TransactionObservationStatus::Normalized => "normalized",
|
||||
kb_store_core::TransactionObservationStatus::Persisted => "persisted",
|
||||
kb_store_core::TransactionObservationStatus::Failed => "failed",
|
||||
kb_store_core::TransactionObservationStatus::Missing => "missing",
|
||||
};
|
||||
}
|
||||
|
||||
fn sql_slot_from_u64(slot: u64) -> kb_core::Result<i64> {
|
||||
let conversion_result = i64::try_from(slot);
|
||||
return match conversion_result {
|
||||
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"Solana slot does not fit into PostgreSQL BIGINT: {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
fn optional_sql_slot_from_u64(
|
||||
slot: std::option::Option<u64>,
|
||||
) -> kb_core::Result<std::option::Option<i64>> {
|
||||
return crate::queries::raw_queries::optional_sql_bigint_from_u64(slot);
|
||||
}
|
||||
|
||||
fn optional_sql_bigint_from_u64(
|
||||
value: std::option::Option<u64>,
|
||||
) -> kb_core::Result<std::option::Option<i64>> {
|
||||
return match value {
|
||||
std::option::Option::Some(raw_value) => {
|
||||
let conversion_result = i64::try_from(raw_value);
|
||||
match conversion_result {
|
||||
std::result::Result::Ok(converted) => {
|
||||
std::result::Result::Ok(std::option::Option::Some(converted))
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(
|
||||
format!("unsigned value does not fit into PostgreSQL BIGINT: {error}"),
|
||||
)),
|
||||
}
|
||||
},
|
||||
std::option::Option::None => std::result::Result::Ok(std::option::Option::None),
|
||||
};
|
||||
}
|
||||
|
||||
fn validate_required_text(value: &str, message: &str) -> kb_core::Result<()> {
|
||||
if value.trim().is_empty() {
|
||||
return std::result::Result::Err(kb_core::Error::db(message));
|
||||
}
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
|
||||
async fn update_raw_transaction_lifecycle(
|
||||
pool: &sqlx::PgPool,
|
||||
mark: &kb_store_core::RawPayloadLifecycleMark,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
let retention_state =
|
||||
crate::queries::raw_queries::raw_retention_state_to_sql(mark.retention_state);
|
||||
let processing_state =
|
||||
crate::queries::raw_queries::raw_processing_state_to_sql(mark.processing_state);
|
||||
let query_result = sqlx::query(
|
||||
"UPDATE kb_sol_raw_transactions SET retention_state = $1, processing_state = $2, lifecycle_reason = $3, updated_at = NOW() WHERE signature = $4",
|
||||
)
|
||||
.bind(retention_state)
|
||||
.bind(processing_state)
|
||||
.bind(mark.reason.as_deref())
|
||||
.bind(mark.raw_row_key.as_str())
|
||||
.execute(pool)
|
||||
.await;
|
||||
return crate::queries::raw_queries::outcome_from_update_result(
|
||||
query_result,
|
||||
"postgres canonical raw transaction lifecycle update failed",
|
||||
);
|
||||
}
|
||||
|
||||
fn outcome_from_update_result(
|
||||
query_result: std::result::Result<sqlx::postgres::PgQueryResult, sqlx::Error>,
|
||||
error_prefix: &str,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return match query_result {
|
||||
std::result::Result::Ok(result) => {
|
||||
let rows_affected = result.rows_affected();
|
||||
if rows_affected == 0 {
|
||||
return std::result::Result::Ok(kb_store_core::InsertOutcome::new(0, 0, 1));
|
||||
}
|
||||
std::result::Result::Ok(kb_store_core::InsertOutcome::new(0, rows_affected, 0))
|
||||
},
|
||||
std::result::Result::Err(error) => {
|
||||
std::result::Result::Err(kb_core::Error::db(format!("{error_prefix}: {error}")))
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn retention_state_serializes_to_lower_snake_case() {
|
||||
let value = crate::queries::raw_queries::raw_retention_state_to_sql(
|
||||
kb_store_core::RawPayloadRetentionState::Compacted,
|
||||
);
|
||||
assert_eq!(value, "compacted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn processing_state_serializes_to_lower_snake_case() {
|
||||
let value = crate::queries::raw_queries::raw_processing_state_to_sql(
|
||||
kb_store_core::RawPayloadProcessingState::CoreExtracted,
|
||||
);
|
||||
assert_eq!(value, "core_extracted");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn observation_origin_serializes_to_lower_snake_case() {
|
||||
let value = crate::queries::raw_queries::transaction_observation_origin_to_sql(
|
||||
kb_store_core::TransactionObservationOrigin::Backfill,
|
||||
);
|
||||
assert_eq!(value, "backfill");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn observation_status_serializes_to_lower_snake_case() {
|
||||
let value = crate::queries::raw_queries::transaction_observation_status_to_sql(
|
||||
kb_store_core::TransactionObservationStatus::Normalized,
|
||||
);
|
||||
assert_eq!(value, "normalized");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sql_slot_rejects_values_above_bigint() {
|
||||
let result = crate::queries::raw_queries::sql_slot_from_u64(u64::MAX);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn optional_postgres_canonical_store_roundtrip_from_env() {
|
||||
let database_url = match std::env::var("KB_POSTGRES_TEST_URL") {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_error) => return,
|
||||
};
|
||||
let _postgres_guard = crate::test_serial::postgres_test_guard().await;
|
||||
let options_result = crate::PostgresStoreOptions::new(database_url, 1, 5000, false);
|
||||
let options = match options_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected options error: {error}"),
|
||||
};
|
||||
let store_result = crate::PostgresStore::connect(options).await;
|
||||
let store = match store_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected connect error: {error}"),
|
||||
};
|
||||
let schema_result = store.initialize_raw_store_schema().await;
|
||||
if let std::result::Result::Err(error) = schema_result {
|
||||
panic!("unexpected schema error: {error}");
|
||||
}
|
||||
let signature = test_signature();
|
||||
let raw_input_result = kb_store_core::RawTransactionInsert::new(
|
||||
signature.clone(),
|
||||
1,
|
||||
serde_json::json!({"source": "test"}),
|
||||
1,
|
||||
);
|
||||
let raw_input = match raw_input_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected raw input error: {error}"),
|
||||
};
|
||||
let first_result =
|
||||
kb_store_core::RawTransactionStore::insert_raw_transaction(&store, &raw_input).await;
|
||||
let first_outcome = match first_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected first insert error: {error}"),
|
||||
};
|
||||
assert_eq!(first_outcome.inserted_count, 1);
|
||||
let duplicate_result =
|
||||
kb_store_core::RawTransactionStore::insert_raw_transaction(&store, &raw_input).await;
|
||||
let duplicate_outcome = match duplicate_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected duplicate insert error: {error}"),
|
||||
};
|
||||
assert_eq!(duplicate_outcome.skipped_count, 1);
|
||||
let has_result = kb_store_core::RawTransactionStore::has_raw_transaction_signature(
|
||||
&store,
|
||||
&kb_model::Signature(signature.clone()),
|
||||
)
|
||||
.await;
|
||||
let has_signature = match has_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected signature lookup error: {error}"),
|
||||
};
|
||||
assert!(has_signature);
|
||||
let observation_key = std::string::String::from("test:http:") + signature.as_str();
|
||||
let observation_result = kb_store_core::TransactionObservationInsert::new(
|
||||
observation_key.clone(),
|
||||
"test_provider",
|
||||
"solana_http",
|
||||
"getTransaction",
|
||||
kb_store_core::TransactionObservationOrigin::Backfill,
|
||||
chrono::Utc::now(),
|
||||
);
|
||||
let observation = match observation_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected observation error: {error}"),
|
||||
};
|
||||
let identity_result =
|
||||
observation.with_transaction_identity(signature.clone(), std::option::Option::Some(1));
|
||||
let observation_with_identity = match identity_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected identity error: {error}"),
|
||||
};
|
||||
let observation_insert_result =
|
||||
kb_store_core::RawTransactionStore::insert_transaction_observation(
|
||||
&store,
|
||||
&observation_with_identity,
|
||||
)
|
||||
.await;
|
||||
let observation_outcome = match observation_insert_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
panic!("unexpected observation insert error: {error}")
|
||||
},
|
||||
};
|
||||
assert_eq!(observation_outcome.inserted_count, 1);
|
||||
let observation_lookup_result =
|
||||
kb_store_core::RawTransactionStore::has_transaction_observation_key(
|
||||
&store,
|
||||
observation_key.as_str(),
|
||||
)
|
||||
.await;
|
||||
let has_observation = match observation_lookup_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
panic!("unexpected observation lookup error: {error}")
|
||||
},
|
||||
};
|
||||
assert!(has_observation);
|
||||
let mark_result = kb_store_core::RawPayloadLifecycleMark::new(
|
||||
crate::RAW_TRANSACTIONS_TABLE_NAME,
|
||||
signature,
|
||||
kb_store_core::RawPayloadRetentionState::Full,
|
||||
kb_store_core::RawPayloadProcessingState::CoreExtracted,
|
||||
std::option::Option::Some(std::string::String::from("test extraction")),
|
||||
);
|
||||
let mark = match mark_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected mark error: {error}"),
|
||||
};
|
||||
let mark_outcome_result =
|
||||
kb_store_core::RawTransactionStore::mark_raw_payload_lifecycle(&store, &mark).await;
|
||||
let mark_outcome = match mark_outcome_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected lifecycle update error: {error}"),
|
||||
};
|
||||
assert_eq!(mark_outcome.updated_count, 1);
|
||||
return;
|
||||
}
|
||||
|
||||
fn test_signature() -> std::string::String {
|
||||
let now_result = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH);
|
||||
let duration = match now_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_error) => {
|
||||
return std::string::String::from("test_signature_fallback");
|
||||
},
|
||||
};
|
||||
return format!("test_signature_{}", duration.as_nanos());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,474 @@
|
||||
// file: kb_store_pg/src/queries/replay_candidate_queries.rs
|
||||
// version: 2
|
||||
|
||||
//! Read-only PostgreSQL queries for replay candidate discovery.
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct ReplayTransactionCandidateRow {
|
||||
signature: std::string::String,
|
||||
slot: i64,
|
||||
raw_processing_state: std::string::String,
|
||||
retention_state: std::string::String,
|
||||
has_core_transaction: bool,
|
||||
transaction_failed: std::option::Option<bool>,
|
||||
ledger_status: std::string::String,
|
||||
processor_version: std::option::Option<std::string::String>,
|
||||
attempt_count: i32,
|
||||
outer_instruction_count: i64,
|
||||
inner_instruction_count: i64,
|
||||
outer_program_count: i64,
|
||||
inner_program_count: i64,
|
||||
updated_at: std::string::String,
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct ReplayProgramSummaryRow {
|
||||
program_id: std::string::String,
|
||||
transaction_count: i64,
|
||||
outer_instruction_count: i64,
|
||||
inner_instruction_count: i64,
|
||||
log_count: i64,
|
||||
min_slot: i64,
|
||||
max_slot: i64,
|
||||
}
|
||||
|
||||
#[derive(sqlx::FromRow)]
|
||||
struct ReplayEntitySummaryRow {
|
||||
entity_kind: std::string::String,
|
||||
entity_value: std::string::String,
|
||||
transaction_count: i64,
|
||||
occurrence_count: i64,
|
||||
min_slot: i64,
|
||||
max_slot: i64,
|
||||
}
|
||||
|
||||
pub(crate) async fn list_replay_transaction_candidates(
|
||||
pool: &sqlx::PgPool,
|
||||
filter: &crate::PostgresReplayTransactionFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresReplayTransactionCandidate>> {
|
||||
let min_slot_result =
|
||||
crate::queries::replay_candidate_queries::optional_sql_bigint(filter.min_slot);
|
||||
let min_slot = match min_slot_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let max_slot_result =
|
||||
crate::queries::replay_candidate_queries::optional_sql_bigint(filter.max_slot);
|
||||
let max_slot = match max_slot_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||
};
|
||||
let entity_kind = match filter.entity_kind {
|
||||
std::option::Option::Some(value) => std::option::Option::Some(value.as_sql()),
|
||||
std::option::Option::None => std::option::Option::None,
|
||||
};
|
||||
let sql = if filter.newest_first {
|
||||
crate::queries::replay_candidate_queries::transaction_candidate_sql_desc()
|
||||
} else {
|
||||
crate::queries::replay_candidate_queries::transaction_candidate_sql_asc()
|
||||
};
|
||||
let query_result = sqlx::query_as::<
|
||||
sqlx::Postgres,
|
||||
crate::queries::replay_candidate_queries::ReplayTransactionCandidateRow,
|
||||
>(sql)
|
||||
.bind(filter.signature_contains.as_deref())
|
||||
.bind(min_slot)
|
||||
.bind(max_slot)
|
||||
.bind(filter.raw_processing_state.as_deref())
|
||||
.bind(filter.ledger_status.as_deref())
|
||||
.bind(filter.program_id.as_deref())
|
||||
.bind(filter.program_scope.as_sql())
|
||||
.bind(entity_kind)
|
||||
.bind(filter.entity_value.as_deref())
|
||||
.bind(i64::from(filter.limit))
|
||||
.fetch_all(pool)
|
||||
.await;
|
||||
let rows = match query_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres replay transaction candidate query failed: {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let mut output = std::vec::Vec::with_capacity(rows.len());
|
||||
for row in rows {
|
||||
output.push(crate::PostgresReplayTransactionCandidate {
|
||||
signature: row.signature,
|
||||
slot: row.slot,
|
||||
raw_processing_state: row.raw_processing_state,
|
||||
retention_state: row.retention_state,
|
||||
has_core_transaction: row.has_core_transaction,
|
||||
transaction_failed: row.transaction_failed,
|
||||
ledger_status: row.ledger_status,
|
||||
processor_version: row.processor_version,
|
||||
attempt_count: row.attempt_count,
|
||||
outer_instruction_count: row.outer_instruction_count,
|
||||
inner_instruction_count: row.inner_instruction_count,
|
||||
outer_program_count: row.outer_program_count,
|
||||
inner_program_count: row.inner_program_count,
|
||||
updated_at: row.updated_at,
|
||||
});
|
||||
}
|
||||
return std::result::Result::Ok(output);
|
||||
}
|
||||
|
||||
pub(crate) async fn list_replay_program_summaries(
|
||||
pool: &sqlx::PgPool,
|
||||
filter: &crate::PostgresReplayProgramFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresReplayProgramSummary>> {
|
||||
let query_result = sqlx::query_as::<sqlx::Postgres, crate::queries::replay_candidate_queries::ReplayProgramSummaryRow>(
|
||||
r#"WITH occurrences AS (
|
||||
SELECT program_id, signature, slot, 'outer'::TEXT AS scope FROM kb_sol_core_instructions
|
||||
UNION ALL
|
||||
SELECT program_id, signature, slot, 'inner'::TEXT AS scope FROM kb_sol_core_inner_instructions
|
||||
UNION ALL
|
||||
SELECT program_id, signature, slot, 'logs'::TEXT AS scope FROM kb_sol_core_logs WHERE program_id IS NOT NULL
|
||||
)
|
||||
SELECT program_id,
|
||||
COUNT(DISTINCT signature)::BIGINT AS transaction_count,
|
||||
COUNT(*) FILTER (WHERE scope = 'outer')::BIGINT AS outer_instruction_count,
|
||||
COUNT(*) FILTER (WHERE scope = 'inner')::BIGINT AS inner_instruction_count,
|
||||
COUNT(*) FILTER (WHERE scope = 'logs')::BIGINT AS log_count,
|
||||
MIN(slot)::BIGINT AS min_slot,
|
||||
MAX(slot)::BIGINT AS max_slot
|
||||
FROM occurrences
|
||||
WHERE ($1::TEXT IS NULL OR program_id ILIKE '%' || $1 || '%')
|
||||
GROUP BY program_id
|
||||
ORDER BY transaction_count DESC, program_id ASC
|
||||
LIMIT $2"#,
|
||||
)
|
||||
.bind(filter.program_id_contains.as_deref())
|
||||
.bind(i64::from(filter.limit))
|
||||
.fetch_all(pool)
|
||||
.await;
|
||||
let rows = match query_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres replay program summary query failed: {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let mut output = std::vec::Vec::with_capacity(rows.len());
|
||||
for row in rows {
|
||||
output.push(crate::PostgresReplayProgramSummary {
|
||||
program_id: row.program_id,
|
||||
transaction_count: row.transaction_count,
|
||||
outer_instruction_count: row.outer_instruction_count,
|
||||
inner_instruction_count: row.inner_instruction_count,
|
||||
log_count: row.log_count,
|
||||
min_slot: row.min_slot,
|
||||
max_slot: row.max_slot,
|
||||
});
|
||||
}
|
||||
return std::result::Result::Ok(output);
|
||||
}
|
||||
|
||||
pub(crate) async fn list_replay_entity_summaries(
|
||||
pool: &sqlx::PgPool,
|
||||
filter: &crate::PostgresReplayEntityFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<crate::PostgresReplayEntitySummary>> {
|
||||
let entity_kind = filter.entity_kind.as_sql();
|
||||
let query_result = sqlx::query_as::<
|
||||
sqlx::Postgres,
|
||||
crate::queries::replay_candidate_queries::ReplayEntitySummaryRow,
|
||||
>(
|
||||
r#"WITH entities AS (
|
||||
SELECT 'mint'::TEXT AS entity_kind, mint AS entity_value, signature, slot
|
||||
FROM kb_sol_core_balance_changes
|
||||
WHERE mint IS NOT NULL
|
||||
UNION ALL
|
||||
SELECT 'owner'::TEXT AS entity_kind, owner AS entity_value, signature, slot
|
||||
FROM kb_sol_core_balance_changes
|
||||
WHERE owner IS NOT NULL
|
||||
UNION ALL
|
||||
SELECT 'account_key'::TEXT AS entity_kind, account_key AS entity_value, signature, slot
|
||||
FROM kb_sol_core_account_keys
|
||||
)
|
||||
SELECT entity_kind,
|
||||
entity_value,
|
||||
COUNT(DISTINCT signature)::BIGINT AS transaction_count,
|
||||
COUNT(*)::BIGINT AS occurrence_count,
|
||||
MIN(slot)::BIGINT AS min_slot,
|
||||
MAX(slot)::BIGINT AS max_slot
|
||||
FROM entities
|
||||
WHERE entity_kind = $1
|
||||
AND ($2::TEXT IS NULL OR entity_value ILIKE '%' || $2 || '%')
|
||||
GROUP BY entity_kind, entity_value
|
||||
ORDER BY transaction_count DESC, entity_value ASC
|
||||
LIMIT $3"#,
|
||||
)
|
||||
.bind(entity_kind)
|
||||
.bind(filter.entity_value_contains.as_deref())
|
||||
.bind(i64::from(filter.limit))
|
||||
.fetch_all(pool)
|
||||
.await;
|
||||
let rows = match query_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres replay entity summary query failed: {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let mut output = std::vec::Vec::with_capacity(rows.len());
|
||||
for row in rows {
|
||||
output.push(crate::PostgresReplayEntitySummary {
|
||||
entity_kind: row.entity_kind,
|
||||
entity_value: row.entity_value,
|
||||
transaction_count: row.transaction_count,
|
||||
occurrence_count: row.occurrence_count,
|
||||
min_slot: row.min_slot,
|
||||
max_slot: row.max_slot,
|
||||
});
|
||||
}
|
||||
return std::result::Result::Ok(output);
|
||||
}
|
||||
|
||||
fn transaction_candidate_sql_desc() -> &'static str {
|
||||
return crate::queries::replay_candidate_queries::transaction_candidate_sql("DESC");
|
||||
}
|
||||
|
||||
fn transaction_candidate_sql_asc() -> &'static str {
|
||||
return crate::queries::replay_candidate_queries::transaction_candidate_sql("ASC");
|
||||
}
|
||||
|
||||
fn transaction_candidate_sql(order: &str) -> &'static str {
|
||||
if order == "DESC" {
|
||||
return r#"SELECT raw.signature,
|
||||
raw.slot,
|
||||
raw.processing_state AS raw_processing_state,
|
||||
raw.retention_state,
|
||||
(core.id IS NOT NULL) AS has_core_transaction,
|
||||
core.failed AS transaction_failed,
|
||||
COALESCE(ledger.status, 'not_started') AS ledger_status,
|
||||
ledger.processor_version,
|
||||
COALESCE(ledger.attempt_count, 0)::INTEGER AS attempt_count,
|
||||
COALESCE(outer_stats.instruction_count, 0)::BIGINT AS outer_instruction_count,
|
||||
COALESCE(inner_stats.instruction_count, 0)::BIGINT AS inner_instruction_count,
|
||||
COALESCE(outer_stats.program_count, 0)::BIGINT AS outer_program_count,
|
||||
COALESCE(inner_stats.program_count, 0)::BIGINT AS inner_program_count,
|
||||
raw.updated_at::TEXT AS updated_at
|
||||
FROM kb_sol_raw_transactions raw
|
||||
LEFT JOIN kb_sol_core_transactions core ON core.signature = raw.signature
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT status, processor_version, attempt_count
|
||||
FROM kb_sol_ops_processing_ledger
|
||||
WHERE stage = 'core_extraction'
|
||||
AND processor_name = 'canonical_to_core'
|
||||
AND input_key = raw.signature
|
||||
ORDER BY updated_at DESC, id DESC
|
||||
LIMIT 1
|
||||
) ledger ON TRUE
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT COUNT(*)::BIGINT AS instruction_count, COUNT(DISTINCT program_id)::BIGINT AS program_count
|
||||
FROM kb_sol_core_instructions
|
||||
WHERE signature = raw.signature
|
||||
) outer_stats ON TRUE
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT COUNT(*)::BIGINT AS instruction_count, COUNT(DISTINCT program_id)::BIGINT AS program_count
|
||||
FROM kb_sol_core_inner_instructions
|
||||
WHERE signature = raw.signature
|
||||
) inner_stats ON TRUE
|
||||
WHERE ($1::TEXT IS NULL OR raw.signature ILIKE '%' || $1 || '%')
|
||||
AND ($2::BIGINT IS NULL OR raw.slot >= $2)
|
||||
AND ($3::BIGINT IS NULL OR raw.slot <= $3)
|
||||
AND ($4::TEXT IS NULL OR raw.processing_state = $4)
|
||||
AND ($5::TEXT IS NULL OR ($5 = 'not_started' AND ledger.status IS NULL) OR ledger.status = $5)
|
||||
AND ($6::TEXT IS NULL OR
|
||||
($7 = 'any' AND (
|
||||
EXISTS (SELECT 1 FROM kb_sol_core_instructions candidate_outer WHERE candidate_outer.signature = raw.signature AND candidate_outer.program_id = $6) OR
|
||||
EXISTS (SELECT 1 FROM kb_sol_core_inner_instructions candidate_inner WHERE candidate_inner.signature = raw.signature AND candidate_inner.program_id = $6) OR
|
||||
EXISTS (SELECT 1 FROM kb_sol_core_logs candidate_log WHERE candidate_log.signature = raw.signature AND candidate_log.program_id = $6)
|
||||
)) OR
|
||||
($7 = 'outer' AND EXISTS (SELECT 1 FROM kb_sol_core_instructions candidate_outer WHERE candidate_outer.signature = raw.signature AND candidate_outer.program_id = $6)) OR
|
||||
($7 = 'inner' AND EXISTS (SELECT 1 FROM kb_sol_core_inner_instructions candidate_inner WHERE candidate_inner.signature = raw.signature AND candidate_inner.program_id = $6)) OR
|
||||
($7 = 'logs' AND EXISTS (SELECT 1 FROM kb_sol_core_logs candidate_log WHERE candidate_log.signature = raw.signature AND candidate_log.program_id = $6)))
|
||||
AND ($8::TEXT IS NULL OR
|
||||
($8 = 'mint' AND EXISTS (
|
||||
SELECT 1 FROM kb_sol_core_balance_changes candidate_balance
|
||||
WHERE candidate_balance.signature = raw.signature
|
||||
AND candidate_balance.mint = $9
|
||||
)) OR
|
||||
($8 = 'owner' AND EXISTS (
|
||||
SELECT 1 FROM kb_sol_core_balance_changes candidate_balance
|
||||
WHERE candidate_balance.signature = raw.signature
|
||||
AND candidate_balance.owner = $9
|
||||
)) OR
|
||||
($8 = 'account_key' AND EXISTS (
|
||||
SELECT 1 FROM kb_sol_core_account_keys candidate_account
|
||||
WHERE candidate_account.signature = raw.signature
|
||||
AND candidate_account.account_key = $9
|
||||
)))
|
||||
ORDER BY raw.slot DESC, raw.signature DESC
|
||||
LIMIT $10"#;
|
||||
}
|
||||
return r#"SELECT raw.signature,
|
||||
raw.slot,
|
||||
raw.processing_state AS raw_processing_state,
|
||||
raw.retention_state,
|
||||
(core.id IS NOT NULL) AS has_core_transaction,
|
||||
core.failed AS transaction_failed,
|
||||
COALESCE(ledger.status, 'not_started') AS ledger_status,
|
||||
ledger.processor_version,
|
||||
COALESCE(ledger.attempt_count, 0)::INTEGER AS attempt_count,
|
||||
COALESCE(outer_stats.instruction_count, 0)::BIGINT AS outer_instruction_count,
|
||||
COALESCE(inner_stats.instruction_count, 0)::BIGINT AS inner_instruction_count,
|
||||
COALESCE(outer_stats.program_count, 0)::BIGINT AS outer_program_count,
|
||||
COALESCE(inner_stats.program_count, 0)::BIGINT AS inner_program_count,
|
||||
raw.updated_at::TEXT AS updated_at
|
||||
FROM kb_sol_raw_transactions raw
|
||||
LEFT JOIN kb_sol_core_transactions core ON core.signature = raw.signature
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT status, processor_version, attempt_count
|
||||
FROM kb_sol_ops_processing_ledger
|
||||
WHERE stage = 'core_extraction'
|
||||
AND processor_name = 'canonical_to_core'
|
||||
AND input_key = raw.signature
|
||||
ORDER BY updated_at DESC, id DESC
|
||||
LIMIT 1
|
||||
) ledger ON TRUE
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT COUNT(*)::BIGINT AS instruction_count, COUNT(DISTINCT program_id)::BIGINT AS program_count
|
||||
FROM kb_sol_core_instructions
|
||||
WHERE signature = raw.signature
|
||||
) outer_stats ON TRUE
|
||||
LEFT JOIN LATERAL (
|
||||
SELECT COUNT(*)::BIGINT AS instruction_count, COUNT(DISTINCT program_id)::BIGINT AS program_count
|
||||
FROM kb_sol_core_inner_instructions
|
||||
WHERE signature = raw.signature
|
||||
) inner_stats ON TRUE
|
||||
WHERE ($1::TEXT IS NULL OR raw.signature ILIKE '%' || $1 || '%')
|
||||
AND ($2::BIGINT IS NULL OR raw.slot >= $2)
|
||||
AND ($3::BIGINT IS NULL OR raw.slot <= $3)
|
||||
AND ($4::TEXT IS NULL OR raw.processing_state = $4)
|
||||
AND ($5::TEXT IS NULL OR ($5 = 'not_started' AND ledger.status IS NULL) OR ledger.status = $5)
|
||||
AND ($6::TEXT IS NULL OR
|
||||
($7 = 'any' AND (
|
||||
EXISTS (SELECT 1 FROM kb_sol_core_instructions candidate_outer WHERE candidate_outer.signature = raw.signature AND candidate_outer.program_id = $6) OR
|
||||
EXISTS (SELECT 1 FROM kb_sol_core_inner_instructions candidate_inner WHERE candidate_inner.signature = raw.signature AND candidate_inner.program_id = $6) OR
|
||||
EXISTS (SELECT 1 FROM kb_sol_core_logs candidate_log WHERE candidate_log.signature = raw.signature AND candidate_log.program_id = $6)
|
||||
)) OR
|
||||
($7 = 'outer' AND EXISTS (SELECT 1 FROM kb_sol_core_instructions candidate_outer WHERE candidate_outer.signature = raw.signature AND candidate_outer.program_id = $6)) OR
|
||||
($7 = 'inner' AND EXISTS (SELECT 1 FROM kb_sol_core_inner_instructions candidate_inner WHERE candidate_inner.signature = raw.signature AND candidate_inner.program_id = $6)) OR
|
||||
($7 = 'logs' AND EXISTS (SELECT 1 FROM kb_sol_core_logs candidate_log WHERE candidate_log.signature = raw.signature AND candidate_log.program_id = $6)))
|
||||
AND ($8::TEXT IS NULL OR
|
||||
($8 = 'mint' AND EXISTS (
|
||||
SELECT 1 FROM kb_sol_core_balance_changes candidate_balance
|
||||
WHERE candidate_balance.signature = raw.signature
|
||||
AND candidate_balance.mint = $9
|
||||
)) OR
|
||||
($8 = 'owner' AND EXISTS (
|
||||
SELECT 1 FROM kb_sol_core_balance_changes candidate_balance
|
||||
WHERE candidate_balance.signature = raw.signature
|
||||
AND candidate_balance.owner = $9
|
||||
)) OR
|
||||
($8 = 'account_key' AND EXISTS (
|
||||
SELECT 1 FROM kb_sol_core_account_keys candidate_account
|
||||
WHERE candidate_account.signature = raw.signature
|
||||
AND candidate_account.account_key = $9
|
||||
)))
|
||||
ORDER BY raw.slot ASC, raw.signature ASC
|
||||
LIMIT $10"#;
|
||||
}
|
||||
|
||||
fn optional_sql_bigint(
|
||||
value: std::option::Option<u64>,
|
||||
) -> kb_core::Result<std::option::Option<i64>> {
|
||||
return match value {
|
||||
std::option::Option::Some(raw_value) => {
|
||||
let conversion_result = i64::try_from(raw_value);
|
||||
match conversion_result {
|
||||
std::result::Result::Ok(converted) => {
|
||||
std::result::Result::Ok(std::option::Option::Some(converted))
|
||||
},
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(
|
||||
format!("replay candidate slot does not fit into PostgreSQL BIGINT: {error}"),
|
||||
)),
|
||||
}
|
||||
},
|
||||
std::option::Option::None => std::result::Result::Ok(std::option::Option::None),
|
||||
};
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[tokio::test]
|
||||
async fn optional_postgres_replay_candidate_queries_from_env() {
|
||||
let database_url = match std::env::var("KB_POSTGRES_TEST_URL") {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(_error) => return,
|
||||
};
|
||||
let _postgres_guard = crate::test_serial::postgres_test_guard().await;
|
||||
let options_result = crate::PostgresStoreOptions::new(database_url, 1, 5000, false);
|
||||
let options = match options_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected options error: {error}"),
|
||||
};
|
||||
let store_result = crate::PostgresStore::connect(options).await;
|
||||
let store = match store_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected connect error: {error}"),
|
||||
};
|
||||
let raw_schema_result = store.initialize_raw_store_schema().await;
|
||||
if let std::result::Result::Err(error) = raw_schema_result {
|
||||
panic!("unexpected raw schema error: {error}");
|
||||
}
|
||||
let core_schema_result = store.initialize_core_store_schema().await;
|
||||
if let std::result::Result::Err(error) = core_schema_result {
|
||||
panic!("unexpected core schema error: {error}");
|
||||
}
|
||||
let transaction_filter_result = crate::PostgresReplayTransactionFilter::new(
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
crate::PostgresReplayProgramScope::Any,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
10,
|
||||
true,
|
||||
);
|
||||
let transaction_filter = match transaction_filter_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
panic!("unexpected transaction filter error: {error}")
|
||||
},
|
||||
};
|
||||
let transaction_result = store.replay_transaction_candidates(&transaction_filter).await;
|
||||
if let std::result::Result::Err(error) = transaction_result {
|
||||
panic!("unexpected transaction candidate query error: {error}");
|
||||
}
|
||||
let program_filter_result =
|
||||
crate::PostgresReplayProgramFilter::new(std::option::Option::None, 10);
|
||||
let program_filter = match program_filter_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => panic!("unexpected program filter error: {error}"),
|
||||
};
|
||||
let program_result = store.replay_program_summaries(&program_filter).await;
|
||||
if let std::result::Result::Err(error) = program_result {
|
||||
panic!("unexpected program summary query error: {error}");
|
||||
}
|
||||
for entity_kind in [
|
||||
crate::PostgresReplayEntityKind::Mint,
|
||||
crate::PostgresReplayEntityKind::Owner,
|
||||
crate::PostgresReplayEntityKind::AccountKey,
|
||||
] {
|
||||
let entity_filter_result =
|
||||
crate::PostgresReplayEntityFilter::new(entity_kind, std::option::Option::None, 10);
|
||||
let entity_filter = match entity_filter_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
panic!("unexpected entity filter error: {error}")
|
||||
},
|
||||
};
|
||||
let entity_result = store.replay_entity_summaries(&entity_filter).await;
|
||||
if let std::result::Result::Err(error) = entity_result {
|
||||
panic!("unexpected entity summary query error: {error}");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
// file: kb_store_pg/src/queries/table_diagnostics_queries.rs
|
||||
// version: 5
|
||||
|
||||
//! Read-only PostgreSQL diagnostics for known Solana store tables.
|
||||
|
||||
use sqlx::Row; // rust-rules: trait-import
|
||||
|
||||
pub(crate) async fn table_exists(pool: &sqlx::PgPool, table_name: &str) -> kb_core::Result<bool> {
|
||||
let query_result =
|
||||
sqlx::query_scalar::<sqlx::Postgres, bool>("SELECT to_regclass($1)::text IS NOT NULL")
|
||||
.bind(table_name)
|
||||
.fetch_one(pool)
|
||||
.await;
|
||||
return match query_result {
|
||||
std::result::Result::Ok(value) => std::result::Result::Ok(value),
|
||||
std::result::Result::Err(error) => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table existence diagnostic failed for '{table_name}': {error}"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
pub(crate) async fn load_table_statistics(
|
||||
pool: &sqlx::PgPool,
|
||||
table_name: &str,
|
||||
) -> kb_core::Result<crate::PostgresTableStatistics> {
|
||||
return match table_name {
|
||||
crate::RAW_TRANSACTIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_raw_transactions_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::TRANSACTION_OBSERVATIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_obs_transaction_observations_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::CORE_TRANSACTIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_core_transactions_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::CORE_ACCOUNT_KEYS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_core_account_keys_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::CORE_INSTRUCTIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_core_instructions_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::CORE_INNER_INSTRUCTIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_core_inner_instructions_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::CORE_LOGS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_core_logs_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::CORE_BALANCE_CHANGES_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_core_balance_changes_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::PROCESSING_LEDGER_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_ops_processing_ledger_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::DECODE_EVENTS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_decode_events_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::DECODE_COVERAGE_DECLARATIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_decode_coverage_declarations_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::DECODE_COVERAGE_OBSERVATIONS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_decode_coverage_observations_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
crate::MATERIALIZED_EVENTS_TABLE_NAME => {
|
||||
crate::queries::table_diagnostics_queries::load_table_statistics_from_sql(
|
||||
pool,
|
||||
crate::migrations::table_stats_kb_sol_mat_events_sql(),
|
||||
table_name,
|
||||
)
|
||||
.await
|
||||
},
|
||||
_ => std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table statistics are not supported for '{table_name}'"
|
||||
))),
|
||||
};
|
||||
}
|
||||
|
||||
async fn load_table_statistics_from_sql(
|
||||
pool: &sqlx::PgPool,
|
||||
sql: &'static str,
|
||||
table_name: &str,
|
||||
) -> kb_core::Result<crate::PostgresTableStatistics> {
|
||||
let query_result = sqlx::query(sql).fetch_one(pool).await;
|
||||
let row = match query_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table statistics failed for '{table_name}': {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let row_count_result = row.try_get::<i64, _>("row_count");
|
||||
let row_count = match row_count_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table row_count mapping failed for '{table_name}': {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let min_slot_result = row.try_get::<std::option::Option<i64>, _>("min_slot");
|
||||
let min_slot = match min_slot_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table min_slot mapping failed for '{table_name}': {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let max_slot_result = row.try_get::<std::option::Option<i64>, _>("max_slot");
|
||||
let max_slot = match max_slot_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table max_slot mapping failed for '{table_name}': {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
let latest_created_at_result =
|
||||
row.try_get::<std::option::Option<std::string::String>, _>("latest_created_at");
|
||||
let latest_created_at = match latest_created_at_result {
|
||||
std::result::Result::Ok(value) => value,
|
||||
std::result::Result::Err(error) => {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"postgres table latest_created_at mapping failed for '{table_name}': {error}"
|
||||
)));
|
||||
},
|
||||
};
|
||||
return std::result::Result::Ok(crate::PostgresTableStatistics {
|
||||
row_count,
|
||||
min_slot,
|
||||
max_slot,
|
||||
latest_created_at,
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,380 @@
|
||||
// file: kb_store_pg/src/replay_candidates.rs
|
||||
// version: 2
|
||||
|
||||
//! Read-only replay candidate filters and PostgreSQL result rows.
|
||||
|
||||
/// Maximum number of rows returned by one replay candidate query.
|
||||
pub const MAX_REPLAY_CANDIDATE_ROWS: u32 = 5_000;
|
||||
|
||||
/// Program occurrence scope used while filtering replay candidates.
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum PostgresReplayProgramScope {
|
||||
/// Match outer instructions, inner instructions or reliably linked logs.
|
||||
Any,
|
||||
/// Match only top-level instructions.
|
||||
Outer,
|
||||
/// Match only inner instructions.
|
||||
Inner,
|
||||
/// Match only logs with a reliably linked program id.
|
||||
Logs,
|
||||
}
|
||||
|
||||
impl PostgresReplayProgramScope {
|
||||
/// Returns the stable SQL code for this scope.
|
||||
pub fn as_sql(self) -> &'static str {
|
||||
return match self {
|
||||
Self::Any => "any",
|
||||
Self::Outer => "outer",
|
||||
Self::Inner => "inner",
|
||||
Self::Logs => "logs",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/// Core entity kind used while filtering replay candidates.
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum PostgresReplayEntityKind {
|
||||
/// SPL or Token-2022 mint address.
|
||||
Mint,
|
||||
/// Token account owner address.
|
||||
Owner,
|
||||
/// Native or token account address.
|
||||
AccountKey,
|
||||
}
|
||||
|
||||
impl PostgresReplayEntityKind {
|
||||
/// Returns the stable SQL code for this entity kind.
|
||||
pub fn as_sql(self) -> &'static str {
|
||||
return match self {
|
||||
Self::Mint => "mint",
|
||||
Self::Owner => "owner",
|
||||
Self::AccountKey => "account_key",
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/// Bounded read-only filter for transaction replay candidates.
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct PostgresReplayTransactionFilter {
|
||||
/// Optional partial signature search.
|
||||
pub signature_contains: std::option::Option<std::string::String>,
|
||||
/// Optional inclusive minimum slot.
|
||||
pub min_slot: std::option::Option<u64>,
|
||||
/// Optional inclusive maximum slot.
|
||||
pub max_slot: std::option::Option<u64>,
|
||||
/// Optional raw processing state.
|
||||
pub raw_processing_state: std::option::Option<std::string::String>,
|
||||
/// Optional latest core extraction ledger status, including `not_started`.
|
||||
pub ledger_status: std::option::Option<std::string::String>,
|
||||
/// Optional exact program id.
|
||||
pub program_id: std::option::Option<std::string::String>,
|
||||
/// Program occurrence scope.
|
||||
pub program_scope: crate::PostgresReplayProgramScope,
|
||||
/// Optional core entity kind.
|
||||
pub entity_kind: std::option::Option<crate::PostgresReplayEntityKind>,
|
||||
/// Optional exact entity value.
|
||||
pub entity_value: std::option::Option<std::string::String>,
|
||||
/// Maximum returned rows.
|
||||
pub limit: u32,
|
||||
/// Orders newest slots first when true.
|
||||
pub newest_first: bool,
|
||||
}
|
||||
|
||||
impl PostgresReplayTransactionFilter {
|
||||
/// Creates and validates a bounded transaction candidate filter.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
signature_contains: std::option::Option<std::string::String>,
|
||||
min_slot: std::option::Option<u64>,
|
||||
max_slot: std::option::Option<u64>,
|
||||
raw_processing_state: std::option::Option<std::string::String>,
|
||||
ledger_status: std::option::Option<std::string::String>,
|
||||
program_id: std::option::Option<std::string::String>,
|
||||
program_scope: crate::PostgresReplayProgramScope,
|
||||
entity_kind: std::option::Option<crate::PostgresReplayEntityKind>,
|
||||
entity_value: std::option::Option<std::string::String>,
|
||||
limit: u32,
|
||||
newest_first: bool,
|
||||
) -> kb_core::Result<Self> {
|
||||
let signature_contains_value = trim_optional_text(signature_contains);
|
||||
let raw_processing_state_value = trim_optional_text(raw_processing_state);
|
||||
let ledger_status_value = trim_optional_text(ledger_status);
|
||||
let program_id_value = trim_optional_text(program_id);
|
||||
let entity_value_value = trim_optional_text(entity_value);
|
||||
let slot_result = validate_slot_range(min_slot, max_slot);
|
||||
if let std::result::Result::Err(error) = slot_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let limit_result = validate_limit(limit);
|
||||
if let std::result::Result::Err(error) = limit_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let raw_state_result = validate_optional_code(
|
||||
raw_processing_state_value.as_deref(),
|
||||
&["received", "core_extracted", "decoded", "materialized", "failed"],
|
||||
"raw processing state",
|
||||
);
|
||||
if let std::result::Result::Err(error) = raw_state_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
let ledger_status_result = validate_optional_code(
|
||||
ledger_status_value.as_deref(),
|
||||
&["not_started", "running", "succeeded", "failed"],
|
||||
"ledger status",
|
||||
);
|
||||
if let std::result::Result::Err(error) = ledger_status_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
if entity_kind.is_some() != entity_value_value.is_some() {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"replay entity kind and value must either both be present or both be absent",
|
||||
));
|
||||
}
|
||||
return std::result::Result::Ok(Self {
|
||||
signature_contains: signature_contains_value,
|
||||
min_slot,
|
||||
max_slot,
|
||||
raw_processing_state: raw_processing_state_value,
|
||||
ledger_status: ledger_status_value,
|
||||
program_id: program_id_value,
|
||||
program_scope,
|
||||
entity_kind,
|
||||
entity_value: entity_value_value,
|
||||
limit,
|
||||
newest_first,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// One raw transaction candidate enriched with core and ledger diagnostics.
|
||||
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
|
||||
pub struct PostgresReplayTransactionCandidate {
|
||||
/// Canonical transaction signature.
|
||||
pub signature: std::string::String,
|
||||
/// Transaction slot.
|
||||
pub slot: i64,
|
||||
/// Current raw processing state.
|
||||
pub raw_processing_state: std::string::String,
|
||||
/// Current raw retention state.
|
||||
pub retention_state: std::string::String,
|
||||
/// Whether a core transaction row exists.
|
||||
pub has_core_transaction: bool,
|
||||
/// Core transaction failure flag when a core row exists.
|
||||
pub transaction_failed: std::option::Option<bool>,
|
||||
/// Latest core extraction ledger status or `not_started`.
|
||||
pub ledger_status: std::string::String,
|
||||
/// Latest core extraction processor version when available.
|
||||
pub processor_version: std::option::Option<std::string::String>,
|
||||
/// Latest core extraction attempt count.
|
||||
pub attempt_count: i32,
|
||||
/// Number of top-level instructions.
|
||||
pub outer_instruction_count: i64,
|
||||
/// Number of inner instructions.
|
||||
pub inner_instruction_count: i64,
|
||||
/// Number of distinct top-level programs.
|
||||
pub outer_program_count: i64,
|
||||
/// Number of distinct inner programs.
|
||||
pub inner_program_count: i64,
|
||||
/// Raw row update timestamp rendered by PostgreSQL.
|
||||
pub updated_at: std::string::String,
|
||||
}
|
||||
|
||||
/// Bounded read-only filter for program summaries.
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct PostgresReplayProgramFilter {
|
||||
/// Optional partial program id search.
|
||||
pub program_id_contains: std::option::Option<std::string::String>,
|
||||
/// Maximum returned rows.
|
||||
pub limit: u32,
|
||||
}
|
||||
|
||||
impl PostgresReplayProgramFilter {
|
||||
/// Creates and validates a bounded program summary filter.
|
||||
pub fn new(
|
||||
program_id_contains: std::option::Option<std::string::String>,
|
||||
limit: u32,
|
||||
) -> kb_core::Result<Self> {
|
||||
let limit_result = validate_limit(limit);
|
||||
if let std::result::Result::Err(error) = limit_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
return std::result::Result::Ok(Self {
|
||||
program_id_contains: trim_optional_text(program_id_contains),
|
||||
limit,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// Aggregated program occurrences across outer, inner and linked logs.
|
||||
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
|
||||
pub struct PostgresReplayProgramSummary {
|
||||
/// Program id.
|
||||
pub program_id: std::string::String,
|
||||
/// Number of distinct transactions containing the program.
|
||||
pub transaction_count: i64,
|
||||
/// Number of top-level instruction occurrences.
|
||||
pub outer_instruction_count: i64,
|
||||
/// Number of inner instruction occurrences.
|
||||
pub inner_instruction_count: i64,
|
||||
/// Number of reliably linked log occurrences.
|
||||
pub log_count: i64,
|
||||
/// Lowest observed slot.
|
||||
pub min_slot: i64,
|
||||
/// Highest observed slot.
|
||||
pub max_slot: i64,
|
||||
}
|
||||
|
||||
/// Bounded read-only filter for core entity summaries.
|
||||
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||
pub struct PostgresReplayEntityFilter {
|
||||
/// Entity kind to aggregate.
|
||||
pub entity_kind: crate::PostgresReplayEntityKind,
|
||||
/// Optional partial entity value search.
|
||||
pub entity_value_contains: std::option::Option<std::string::String>,
|
||||
/// Maximum returned rows.
|
||||
pub limit: u32,
|
||||
}
|
||||
|
||||
impl PostgresReplayEntityFilter {
|
||||
/// Creates and validates a bounded entity summary filter.
|
||||
pub fn new(
|
||||
entity_kind: crate::PostgresReplayEntityKind,
|
||||
entity_value_contains: std::option::Option<std::string::String>,
|
||||
limit: u32,
|
||||
) -> kb_core::Result<Self> {
|
||||
let limit_result = validate_limit(limit);
|
||||
if let std::result::Result::Err(error) = limit_result {
|
||||
return std::result::Result::Err(error);
|
||||
}
|
||||
return std::result::Result::Ok(Self {
|
||||
entity_kind,
|
||||
entity_value_contains: trim_optional_text(entity_value_contains),
|
||||
limit,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
/// Aggregated mint, owner or account-key occurrences from core tables.
|
||||
#[derive(Clone, Debug, Eq, PartialEq, serde::Deserialize, serde::Serialize)]
|
||||
pub struct PostgresReplayEntitySummary {
|
||||
/// Stable entity kind code.
|
||||
pub entity_kind: std::string::String,
|
||||
/// Mint, owner or account-key address.
|
||||
pub entity_value: std::string::String,
|
||||
/// Number of distinct transactions containing the entity.
|
||||
pub transaction_count: i64,
|
||||
/// Total number of core-table occurrences.
|
||||
pub occurrence_count: i64,
|
||||
/// Lowest observed slot.
|
||||
pub min_slot: i64,
|
||||
/// Highest observed slot.
|
||||
pub max_slot: i64,
|
||||
}
|
||||
|
||||
fn trim_optional_text(
|
||||
value: std::option::Option<std::string::String>,
|
||||
) -> std::option::Option<std::string::String> {
|
||||
return match value {
|
||||
std::option::Option::Some(text) => {
|
||||
let trimmed = text.trim();
|
||||
if trimmed.is_empty() {
|
||||
return std::option::Option::None;
|
||||
}
|
||||
std::option::Option::Some(trimmed.to_string())
|
||||
},
|
||||
std::option::Option::None => std::option::Option::None,
|
||||
};
|
||||
}
|
||||
|
||||
fn validate_slot_range(
|
||||
min_slot: std::option::Option<u64>,
|
||||
max_slot: std::option::Option<u64>,
|
||||
) -> kb_core::Result<()> {
|
||||
if let (std::option::Option::Some(minimum), std::option::Option::Some(maximum)) =
|
||||
(min_slot, max_slot)
|
||||
{
|
||||
if minimum > maximum {
|
||||
return std::result::Result::Err(kb_core::Error::db(
|
||||
"replay candidate minimum slot must not exceed maximum slot",
|
||||
));
|
||||
}
|
||||
}
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
|
||||
fn validate_limit(limit: u32) -> kb_core::Result<()> {
|
||||
if limit == 0 || limit > crate::MAX_REPLAY_CANDIDATE_ROWS {
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"replay candidate limit must be between 1 and {}",
|
||||
crate::MAX_REPLAY_CANDIDATE_ROWS
|
||||
)));
|
||||
}
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
|
||||
fn validate_optional_code(
|
||||
value: std::option::Option<&str>,
|
||||
allowed: &[&str],
|
||||
label: &str,
|
||||
) -> kb_core::Result<()> {
|
||||
let selected = match value {
|
||||
std::option::Option::Some(code) => code,
|
||||
std::option::Option::None => return std::result::Result::Ok(()),
|
||||
};
|
||||
for allowed_code in allowed {
|
||||
if selected == *allowed_code {
|
||||
return std::result::Result::Ok(());
|
||||
}
|
||||
}
|
||||
return std::result::Result::Err(kb_core::Error::db(format!(
|
||||
"unsupported {label}: {selected}"
|
||||
)));
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
#[test]
|
||||
fn transaction_filter_rejects_inverted_slots() {
|
||||
let result = crate::PostgresReplayTransactionFilter::new(
|
||||
std::option::Option::None,
|
||||
std::option::Option::Some(20),
|
||||
std::option::Option::Some(10),
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
crate::PostgresReplayProgramScope::Any,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
100,
|
||||
true,
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn transaction_filter_requires_complete_entity_pair() {
|
||||
let result = crate::PostgresReplayTransactionFilter::new(
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
std::option::Option::None,
|
||||
crate::PostgresReplayProgramScope::Any,
|
||||
std::option::Option::Some(crate::PostgresReplayEntityKind::Mint),
|
||||
std::option::Option::None,
|
||||
100,
|
||||
true,
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn program_filter_rejects_limit_above_maximum() {
|
||||
let result = crate::PostgresReplayProgramFilter::new(
|
||||
std::option::Option::None,
|
||||
crate::MAX_REPLAY_CANDIDATE_ROWS + 1,
|
||||
);
|
||||
assert!(result.is_err());
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
// file: kb_store_pg/src/repositories.rs
|
||||
// version: 5
|
||||
|
||||
//! PostgreSQL repository implementations.
|
||||
|
||||
mod core_extraction_repository;
|
||||
mod core_transaction_repository;
|
||||
mod decode_pipeline_repository;
|
||||
mod raw_transaction_repository;
|
||||
mod store_health_repository;
|
||||
@@ -0,0 +1,17 @@
|
||||
<!-- file: kb_store_pg/src/repositories/README.md -->
|
||||
<!-- version: 3 -->
|
||||
|
||||
# Repositories PostgreSQL
|
||||
|
||||
Ce dossier contient les implémentations PostgreSQL concrètes des traits définis dans `kb_store_core`.
|
||||
|
||||
Les repositories peuvent appeler `queries/`, mais ne doivent pas exposer directement les détails SQL aux applications.
|
||||
|
||||
## Modules actuels
|
||||
|
||||
| Module | Rôle |
|
||||
|----------------------------------|------------------------------------------------------|
|
||||
| `store_health_repository.rs` | Implémentation PostgreSQL de `StoreHealthStore`. |
|
||||
| `raw_transaction_repository.rs` | Implémentation PostgreSQL de `RawTransactionStore`. |
|
||||
| `core_transaction_repository.rs` | Implémentation PostgreSQL de `CoreTransactionStore`. |
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
// file: kb_store_pg/src/repositories/core_extraction_repository.rs
|
||||
// version: 2
|
||||
|
||||
//! PostgreSQL atomic canonical transaction to core extraction repository.
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl kb_store_core::CoreExtractionStore for crate::PostgresStore {
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn list_raw_transactions_for_core_extraction(
|
||||
&self,
|
||||
filter: &kb_store_core::CoreExtractionSelectionFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<kb_store_core::RawTransactionRow>> {
|
||||
return crate::queries::list_raw_transactions_for_core_extraction(self.pool(), filter)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn is_core_extraction_current(
|
||||
&self,
|
||||
identity: &kb_store_core::ProcessingLedgerIdentity,
|
||||
) -> kb_core::Result<bool> {
|
||||
return crate::queries::is_core_extraction_current(self.pool(), identity).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn persist_core_extraction(
|
||||
&self,
|
||||
bundle: &kb_store_core::CoreExtractionBundle,
|
||||
force_replay: bool,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::persist_core_extraction(self.pool(), bundle, force_replay).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn mark_core_extraction_failed(
|
||||
&self,
|
||||
failure: &kb_store_core::CoreExtractionFailure,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::mark_core_extraction_failed(self.pool(), failure).await;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
// file: kb_store_pg/src/repositories/core_transaction_repository.rs
|
||||
// version: 2
|
||||
|
||||
//! PostgreSQL core Solana repository implementation.
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl kb_store_core::CoreTransactionStore for crate::PostgresStore {
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_core_transaction(
|
||||
&self,
|
||||
input: &kb_store_core::CoreTransactionInsert,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_core_transaction(self.pool(), input).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_core_account_keys(
|
||||
&self,
|
||||
inputs: &[kb_store_core::CoreAccountKeyInsert],
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_core_account_keys(self.pool(), inputs).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_core_instructions(
|
||||
&self,
|
||||
inputs: &[kb_store_core::CoreInstructionInsert],
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_core_instructions(self.pool(), inputs).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_core_inner_instructions(
|
||||
&self,
|
||||
inputs: &[kb_store_core::CoreInnerInstructionInsert],
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_core_inner_instructions(self.pool(), inputs).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_core_logs(
|
||||
&self,
|
||||
inputs: &[kb_store_core::CoreLogInsert],
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_core_logs(self.pool(), inputs).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_core_balance_changes(
|
||||
&self,
|
||||
inputs: &[kb_store_core::CoreBalanceChangeInsert],
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_core_balance_changes(self.pool(), inputs).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn list_core_instructions_for_replay(
|
||||
&self,
|
||||
filter: &kb_store_core::CoreInstructionReplayFilter,
|
||||
page_request: &kb_store_core::PageRequest,
|
||||
) -> kb_core::Result<std::vec::Vec<kb_store_core::CoreInstructionRow>> {
|
||||
return crate::queries::list_core_instructions_for_replay(
|
||||
self.pool(),
|
||||
filter,
|
||||
page_request,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn list_core_instruction_replay_inputs(
|
||||
&self,
|
||||
filter: &kb_store_core::CoreInstructionReplayFilter,
|
||||
page_request: &kb_store_core::PageRequest,
|
||||
) -> kb_core::Result<std::vec::Vec<kb_store_core::CoreInstructionReplayInput>> {
|
||||
return crate::queries::list_core_instruction_replay_inputs(
|
||||
self.pool(),
|
||||
filter,
|
||||
page_request,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn mark_core_instruction_lifecycle(
|
||||
&self,
|
||||
mark: &kb_store_core::CoreInstructionLifecycleMark,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::update_core_instruction_lifecycle(self.pool(), mark).await;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
// file: kb_store_pg/src/repositories/decode_pipeline_repository.rs
|
||||
// version: 3
|
||||
|
||||
//! PostgreSQL contextual decode and materialization repository.
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl kb_store_core::DecodePipelineStore for crate::PostgresStore {
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn list_decode_inputs(
|
||||
&self,
|
||||
filter: &kb_store_core::DecodeSelectionFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<kb_store_core::CoreInstructionReplayInput>> {
|
||||
return crate::queries::list_decode_inputs(self.pool(), filter).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn is_decode_current(
|
||||
&self,
|
||||
identity: &kb_store_core::ProcessingLedgerIdentity,
|
||||
) -> kb_core::Result<bool> {
|
||||
return crate::queries::is_decode_current(self.pool(), identity).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn persist_decode_coverage_declarations(
|
||||
&self,
|
||||
declarations: &[kb_store_core::DecodeCoverageDeclarationInsert],
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::persist_decode_coverage_declarations(self.pool(), declarations)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn persist_decode_result(
|
||||
&self,
|
||||
bundle: &kb_store_core::DecodePersistenceBundle,
|
||||
force_replay: bool,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::persist_decode_result(self.pool(), bundle, force_replay).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn mark_decode_failed(
|
||||
&self,
|
||||
failure: &kb_store_core::DecodeFailure,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::mark_decode_failed(self.pool(), failure).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn persist_materialization_result(
|
||||
&self,
|
||||
bundle: &kb_store_core::MaterializationPersistenceBundle,
|
||||
force_replay: bool,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::persist_materialization_result(self.pool(), bundle, force_replay)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn list_decode_coverage_summary(
|
||||
&self,
|
||||
processor_name: std::option::Option<&str>,
|
||||
processor_version: std::option::Option<&str>,
|
||||
limit: u32,
|
||||
) -> kb_core::Result<std::vec::Vec<kb_store_core::DecodeCoverageSummaryRow>> {
|
||||
return crate::queries::list_decode_coverage_summary(
|
||||
self.pool(),
|
||||
processor_name,
|
||||
processor_version,
|
||||
limit,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn list_materialized_events(
|
||||
&self,
|
||||
filter: &kb_store_core::MaterializedEventFilter,
|
||||
) -> kb_core::Result<std::vec::Vec<kb_store_core::MaterializedEventQueryRow>> {
|
||||
return crate::queries::list_materialized_events(self.pool(), filter).await;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
// file: kb_store_pg/src/repositories/raw_transaction_repository.rs
|
||||
// version: 3
|
||||
|
||||
//! PostgreSQL canonical transaction and acquisition observation repository implementation.
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl kb_store_core::RawTransactionStore for crate::PostgresStore {
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn has_raw_transaction_signature(
|
||||
&self,
|
||||
signature: &kb_model::Signature,
|
||||
) -> kb_core::Result<bool> {
|
||||
return crate::queries::has_raw_transaction_signature(self.pool(), signature.0.as_str())
|
||||
.await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn has_transaction_observation_key(
|
||||
&self,
|
||||
observation_key: &str,
|
||||
) -> kb_core::Result<bool> {
|
||||
return crate::queries::has_transaction_observation_key(self.pool(), observation_key).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_raw_transaction(
|
||||
&self,
|
||||
input: &kb_store_core::RawTransactionInsert,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_raw_transaction(self.pool(), input).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn insert_transaction_observation(
|
||||
&self,
|
||||
input: &kb_store_core::TransactionObservationInsert,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::insert_transaction_observation(self.pool(), input).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn mark_raw_payload_lifecycle(
|
||||
&self,
|
||||
mark: &kb_store_core::RawPayloadLifecycleMark,
|
||||
) -> kb_core::Result<kb_store_core::InsertOutcome> {
|
||||
return crate::queries::update_raw_payload_lifecycle(self.pool(), mark).await;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
// file: kb_store_pg/src/repositories/store_health_repository.rs
|
||||
// version: 4
|
||||
|
||||
//! PostgreSQL store health repository implementation.
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl kb_store_core::StoreHealthStore for crate::PostgresStore {
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn backend_descriptor(&self) -> kb_core::Result<kb_store_core::StoreBackendDescriptor> {
|
||||
return crate::PostgresStore::backend_descriptor(self).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn health_snapshot(&self) -> kb_core::Result<kb_store_core::StoreHealthSnapshot> {
|
||||
return crate::PostgresStore::health_snapshot(self).await;
|
||||
}
|
||||
|
||||
#[expect(
|
||||
clippy::implicit_return,
|
||||
reason = "async_trait expansion triggers implicit_return on generated async trait methods."
|
||||
)]
|
||||
async fn migration_snapshot(&self) -> kb_core::Result<kb_store_core::StoreMigrationSnapshot> {
|
||||
return crate::PostgresStore::migration_snapshot(self).await;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
// file: kb_store_pg/src/test_serial.rs
|
||||
// version: 2
|
||||
|
||||
//! Test-only serialization helpers for optional real PostgreSQL tests.
|
||||
|
||||
static POSTGRES_TEST_MUTEX: std::sync::OnceLock<std::sync::Arc<tokio::sync::Mutex<()>>> =
|
||||
std::sync::OnceLock::new();
|
||||
|
||||
/// Acquires the process-local guard shared by optional real PostgreSQL tests.
|
||||
pub(crate) async fn postgres_test_guard() -> tokio::sync::OwnedMutexGuard<()> {
|
||||
let mutex = POSTGRES_TEST_MUTEX
|
||||
.get_or_init(|| return std::sync::Arc::new(tokio::sync::Mutex::new(())))
|
||||
.clone();
|
||||
return mutex.lock_owned().await;
|
||||
}
|
||||
Reference in New Issue
Block a user