Compare commits
36 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| dd48ed9140 | |||
| 5a65c28898 | |||
| 42374115c7 | |||
| 9506df9487 | |||
| 7f8ce987a5 | |||
| 8dbd86c297 | |||
| 75b2d7e7f1 | |||
| b861a1e3b8 | |||
| d89aee9910 | |||
| 75cbf9df09 | |||
| 138395ac35 | |||
| be0032e76b | |||
| 8096e6f07b | |||
| 032919fa07 | |||
| ea5f756d17 | |||
| 4fdeab4d67 | |||
| 478548b7f4 | |||
| 20cf700f22 | |||
| f92c22d55a | |||
| cb670952e8 | |||
| a25d8dd55a | |||
| ecfc30a94b | |||
| 6f2d0a092a | |||
| 3de0488cab | |||
| 2460162243 | |||
| 1b9104d64b | |||
| 80be36bfdd | |||
| e8d2382ac3 | |||
| 749e53edc2 | |||
| 7377a6f6cd | |||
| 45601ce386 | |||
| 7d490eb5d8 | |||
| ca018095bf | |||
| 0dd722ffca | |||
| 75e8030b07 | |||
| c24e4c48f8 |
28
CHANGELOG.md
28
CHANGELOG.md
@@ -1,8 +1,34 @@
|
|||||||
<!-- file: CHANGELOG.md -->
|
<!-- file: CHANGELOG.md -->
|
||||||
<!-- version: 23 -->
|
<!-- version: 26 -->
|
||||||
|
|
||||||
# Changelog KSP
|
# Changelog KSP
|
||||||
|
|
||||||
|
## 0.3.6 — Job API + backfill RAW historique observable — 2026-09-01
|
||||||
|
|
||||||
|
`0.3.6` introduit `ksp-job-api` comme contrat passif et runtime-neutral pour les traitements bornés/terminables : identité `JobId`/`JobKindCode`, lifecycle explicite, annulation coopérative partagée, snapshots typés et notifications latest-value consommables par plusieurs listeners sans transformer l'API en scheduler, runtime Tokio ou bus d'événements. La surface reste Core-only et sert immédiatement au premier consumer concret `ksp-job-backfill-lib`.
|
||||||
|
|
||||||
|
`ksp-job-backfill-lib` ferme une première verticale historique `RawTransaction` sur les abstractions KSP existantes. Les scopes `LatestAddress`, `BeforeAddress`, `AfterAddress` et `ExplicitSignatures` sont bornés par pages/candidats/concurrence, dédupliqués de manière stable et liés à un fingerprint sémantique réseau/scope qui exclut volontairement provider, endpoint, rôle Transport et `JobId`. La découverte utilise les wrappers typés de `ksp-onchain-transport-lib`; l'hydratation passe exclusivement par `get_transaction_observed`, conserve la provenance du provider/endpoint gagnant et transforme la réponse en RAW v1 canonique avec hash SHA-256 exact, sans SDK provider ni modèle wire dupliqué.
|
||||||
|
|
||||||
|
La persistance passe uniquement par `ksp-store-lib` et son écriture atomique `RawTransaction + RawTransactionObservation` en mode normal. Les résultats distinguent insertion, présence idempotente, tombstone purgé, transaction manquante et conflit de contenu ; aucun read-before-write, `ForceRehydrate`, SQL, backend PostgreSQL direct ou décision de rétention n'est déplacé dans Job. Le réseau Store est vérifié avant écriture et l'identité logique de transaction reste `(RawNetworkId, Signature)` indépendamment de la source d'acquisition.
|
||||||
|
|
||||||
|
Le runtime Backfill ajoute concurrence bornée, réconciliation des terminaisons hors ordre, frontier strictement contiguë et checkpoint caller-owned. Les reprises `Latest`, `Before`, `After` et `ExplicitSignatures` ne sautent que le préfixe durable réellement prouvé. `BackfillJobRuntime` expose un `BackfillJobHandle` clonable avec annulation coopérative et `BackfillSnapshotSource` latest-value ; les opérations pré-Store peuvent être abandonnées, tandis qu'une persistance déjà soumise est toujours drainée avant la décision terminale. Les snapshots conservent phase, compteurs, holes/conflicts/missing, maximum in-flight, frontier/checkpoint et code d'échec sans projeter payload RAW, secret, URL ou état runtime interne.
|
||||||
|
|
||||||
|
Le hardening externe verrouille les codes d'erreur, bornes adversariales, redaction `Debug`, indépendance du fingerprint, absence de fuite Tokio/Futures/backend dans la façade, dépendances backend-neutral et discipline d'import/visibilité crate-root. La réconciliation documentaire finale ajoute les README/USAGE durables de Job API et Backfill et aligne les architectures Jobs/Workers/Apps sans transformer les guides d'utilisation en journal de release.
|
||||||
|
|
||||||
|
Les gates de clôture passent `cargo fmt --all -- --check`, audits Rust/Markdown, `cargo check --workspace`, Clippy workspace `--all-targets --all-features -- -D warnings`, `cargo test --workspace --all-targets --all-features`, tests ciblés Job/Backfill et graphes Cargo. Les smokes réseau/Tauri déjà explicitement opt-in restent ignorés par défaut ; aucune preuve non exécutée n'est déclarée PASS.
|
||||||
|
|
||||||
|
## 0.3.5 — Interface acquisition events partagés — 2026-08-31
|
||||||
|
|
||||||
|
`0.3.5` étend `ksp-interface-lib` avec deux familles passives d'acquisition réellement partagées, sans transformer Interface en runtime, en event bus ou en seconde couche RAW. `SlotLifecycleEvent` expose un `slot` et un `SlotLifecycleStage` non exhaustif limité à `Processed`, `FirstShredReceived`, `Completed`, `CreatedBank`, `Dead`, `OptimisticallyConfirmed` et `Rooted`. La normalisation conserve la sémantique commune : les notifications Solana `optimisticConfirmation` et Yellowstone `Confirmed` convergent vers `OptimisticallyConfirmed`, tandis que Solana `root` et Yellowstone `Finalized` convergent vers `Rooted`; les différences d'ordre, de complétude, de replay et de transport restent la responsabilité du producteur/Transport.
|
||||||
|
|
||||||
|
La seconde famille matérialise le fait minimal d'exécution transactionnelle partagé par plusieurs sources. `TransactionSignature` possède exactement 64 octets et un `Debug` redacted, `TransactionExecutionOutcome` distingue seulement `Succeeded` et `Failed`, et `TransactionExecutionEvent` transporte uniquement `slot + signature + outcome`. Les logs, erreurs provider détaillées, commitments, indexes, timestamps, filtres et payloads complets restent Transport-owned. Les snapshots HTTP `getSignatureStatuses`, les transitions one-shot `signatureSubscribe`, un éventuel `TransactionLogEvent`, les votes et les entrées Yellowstone ne sont pas fusionnés artificiellement dans ce contrat.
|
||||||
|
|
||||||
|
La frontière d'ownership reste stricte : les DTOs wire/provider demeurent dans `ksp-onchain-transport-lib`, les événements passifs provider-neutral sont Interface-owned, et les modèles persistants/replayables `RawTransaction` / `RawAccountState` restent `ksp-store-api`. `ksp-interface-lib` conserve exactement `ksp-core-lib` comme seule dépendance normale, sans feature propre, dev/build dependency, serde, codec, logging ou runtime. Les canaris publics, external consumer, inventaires exacts et hardening vérifient également qu'aucun second RAW, metadata source ou payload hostile n'entre dans la surface Interface.
|
||||||
|
|
||||||
|
Les gates de clôture passent audits Rust/Markdown, `cargo check --workspace`, Clippy all-targets, tests ciblés Interface/Program API, `cargo test --workspace` et graphes Cargo. `cargo tree -p ksp-interface-lib --edges normal` confirme le chemin `ksp-interface-lib -> ksp-core-lib -> solana-pubkey -> solana-address`; le graphe features ne montre aucune feature propre Interface et les doublons éventuels restent ceux du workspace global. La documentation durable a été réconciliée et `TransactionLogEvent` est conservé comme idée différée soumise à un nouveau gate consumer/bornes.
|
||||||
|
|
||||||
|
`prompts/025-V0_3_6_START_PROMPT.md` ouvre `0.3.6` sur le développement parallèle de `ksp-job-api` et `ksp-job-backfill-lib`. La release doit reprendre fonctionnellement le backfill historique kbot3 sans en copier le code : audit obligatoire de l'archive historique, `RawTransaction` par adresse via `getSignaturesForAddress` + `getTransaction`, directions/anchors et déduplication, hydratation/persistence via les façades KSP, frontier/checkpoint/reprise, cancellation/concurrency bornées, idempotence et distinctions missing/conflit. `ksp-job-api` doit en parallèle stabiliser lifecycle/progress/outcome et un contrat de notifications/listeners borné pour qu'une couche supérieure puisse visualiser l'état sans parser les logs. Le ROADMAP enchaîne ensuite l'app de backfill/inspection `0.3.7`, `ksp-worker-api` `0.3.8`, `ksp-worker-live-transactions-retriever-lib` `0.3.9` puis une application de monitoring/visualisation Jobs + Workers `0.3.10`, avec reprise du même pattern d'observabilité côté Worker sans fusionner les sémantiques Job/Worker/Store.
|
||||||
|
|
||||||
## 0.3.4 — Store/PostgreSQL RawAccountState + complétude RAW — 2026-08-31
|
## 0.3.4 — Store/PostgreSQL RawAccountState + complétude RAW — 2026-08-31
|
||||||
|
|
||||||
`0.3.4` complète la seconde vertical slice RAW physique sur le couple `ksp-store-lib` / `ksp-store-postgres-lib` et ferme la conformance PostgreSQL des **10 capabilities** backend-agnostic de `ksp-store-api` : les six capabilities `RawTransaction*` acquises en `0.3.3` restent intactes et les quatre capabilities `RawAccountStateRead`, `RawAccountStateWrite`, `RawAccountObservationRead` et `RawAccountObservationWrite` sont désormais implémentées par `PostgresBackend` puis dispatchées par la façade `Store`. La séparation reste stricte : les consommateurs ordinaires passent par `ksp-store-lib`, le backend PostgreSQL conserve SQL/driver/pool/TLS/migrations privés, et la façade reste compilable/testable sans backend via `--no-default-features`.
|
`0.3.4` complète la seconde vertical slice RAW physique sur le couple `ksp-store-lib` / `ksp-store-postgres-lib` et ferme la conformance PostgreSQL des **10 capabilities** backend-agnostic de `ksp-store-api` : les six capabilities `RawTransaction*` acquises en `0.3.3` restent intactes et les quatre capabilities `RawAccountStateRead`, `RawAccountStateWrite`, `RawAccountObservationRead` et `RawAccountObservationWrite` sont désormais implémentées par `PostgresBackend` puis dispatchées par la façade `Store`. La séparation reste stricte : les consommateurs ordinaires passent par `ksp-store-lib`, le backend PostgreSQL conserve SQL/driver/pool/TLS/migrations privés, et la façade reste compilable/testable sans backend via `--no-default-features`.
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
# file: Cargo.toml
|
# file: Cargo.toml
|
||||||
# version: 382
|
# version: 415
|
||||||
|
|
||||||
[workspace]
|
[workspace]
|
||||||
resolver = "3"
|
resolver = "3"
|
||||||
members = ["crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib"]
|
members = ["crates/ksp-app-config-desk", "crates/ksp-app-solprices-desk", "crates/ksp-app-wallet-desk", "crates/ksp-config-lib", "crates/ksp-core-lib", "crates/ksp-interface-lib", "crates/ksp-job-api", "crates/ksp-job-backfill-lib", "crates/ksp-logging-lib", "crates/ksp-offchain-transport-lib", "crates/ksp-onchain-transport-lib", "crates/ksp-program-api", "crates/ksp-store-api", "crates/ksp-store-lib", "crates/ksp-store-postgres-lib", "crates/ksp-wallet-lib"]
|
||||||
|
|
||||||
[workspace.package]
|
[workspace.package]
|
||||||
version = "0.3.4"
|
version = "0.3.6"
|
||||||
edition = "2024"
|
edition = "2024"
|
||||||
license = "MIT"
|
license = "MIT"
|
||||||
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
repository = "https://git.sasedev.com/Sasedev/khadhroony-solana-project"
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: README.md -->
|
<!-- file: README.md -->
|
||||||
<!-- version: 7 -->
|
<!-- version: 8 -->
|
||||||
|
|
||||||
# Khadhroony Solana Project
|
# Khadhroony Solana Project
|
||||||
|
|
||||||
@@ -47,6 +47,12 @@ Les besoins du trading constituent une priorité produit à court terme mais ne
|
|||||||
- Aucun `rust-toolchain.toml` n'est utilisé.
|
- Aucun `rust-toolchain.toml` n'est utilisé.
|
||||||
- Les environnements et contraintes des démonstrations doivent être visibles dans leur nomenclature lorsqu'ils ne sont pas sélectionnables.
|
- Les environnements et contraintes des démonstrations doivent être visibles dans leur nomenclature lorsqu'ils ne sont pas sélectionnables.
|
||||||
|
|
||||||
|
## Fondations opérationnelles actuelles
|
||||||
|
|
||||||
|
La couche RAW dispose d'une façade Store backend-neutral et d'un premier job historique concret. `ksp-job-api` porte les contrats passifs communs des jobs bornés ; `ksp-job-backfill-lib` compose Transport observé et Store pour découvrir, hydrater et persister des transactions historiques RAW avec concurrence bornée, checkpoint contigu caller-owned, annulation coopérative et snapshots latest-value.
|
||||||
|
|
||||||
|
Ces contrats restent distincts des futurs workers continus : un job borné n'est ni un service worker ni un pipeline générique imposé aux autres couches.
|
||||||
|
|
||||||
## Points d'entrée
|
## Points d'entrée
|
||||||
|
|
||||||
- [`RULES.md`](RULES.md) — index des règles normatives ;
|
- [`RULES.md`](RULES.md) — index des règles normatives ;
|
||||||
|
|||||||
28
ROADMAP.md
28
ROADMAP.md
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: ROADMAP.md -->
|
<!-- file: ROADMAP.md -->
|
||||||
<!-- version: 98 -->
|
<!-- version: 101 -->
|
||||||
|
|
||||||
# Roadmap KSP
|
# Roadmap KSP
|
||||||
|
|
||||||
@@ -97,18 +97,27 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
- [X] `0.3.2` — `ksp-store-lib` + `ksp-store-postgres-lib` stables comme fondation runtime/backend PostgreSQL : feature `postgres` par défaut, `Store` lié à un unique `RawNetworkId`, Config `std.store` avec targets/bases `devnet`/`mainnet`/`testnet`, pool Deadpool borné, `tokio-postgres`, TLS Rustls `Disabled`/`VerifyFull`, moteur de migrations privé `V000` + SHA-256/advisory lock, health/readiness portable et close borné. Gate complet + PostgreSQL réel major 17 verts ; aucune table/capability `RawTransaction`/`RawAccountState` métier n'est encore ajoutée.
|
- [X] `0.3.2` — `ksp-store-lib` + `ksp-store-postgres-lib` stables comme fondation runtime/backend PostgreSQL : feature `postgres` par défaut, `Store` lié à un unique `RawNetworkId`, Config `std.store` avec targets/bases `devnet`/`mainnet`/`testnet`, pool Deadpool borné, `tokio-postgres`, TLS Rustls `Disabled`/`VerifyFull`, moteur de migrations privé `V000` + SHA-256/advisory lock, health/readiness portable et close borné. Gate complet + PostgreSQL réel major 17 verts ; aucune table/capability `RawTransaction`/`RawAccountState` métier n'est encore ajoutée.
|
||||||
- [X] `0.3.3` — Vertical slice PostgreSQL `RawTransaction` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : six capabilities transaction/observation/rétention, V001 physique liée à un réseau, acquisition canonical+observation atomique, idempotence/conflit, get/list keyset cursorisé, archive/purge/tombstone/ForceRehydrate, hardening des erreurs et du schéma, concurrence et rollback validés sur PostgreSQL 17.
|
- [X] `0.3.3` — Vertical slice PostgreSQL `RawTransaction` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : six capabilities transaction/observation/rétention, V001 physique liée à un réseau, acquisition canonical+observation atomique, idempotence/conflit, get/list keyset cursorisé, archive/purge/tombstone/ForceRehydrate, hardening des erreurs et du schéma, concurrence et rollback validés sur PostgreSQL 17.
|
||||||
- [X] `0.3.4` — Vertical slice PostgreSQL `RawAccountState` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : quatre capabilities account ajoutées aux six transaction pour une conformance RAW 10/10, V002 additive de 32 ressources au-dessus de V000/V001 immuables, state+observation atomiques, idempotence/conflit exacts, metadata Yellowstone observation-only, get/list keyset `(slot,pubkey,state_hash)` avec cursor KSPA anti-replay, hardening cross-family et live validé sur PostgreSQL 17 sans rétention destructive account.
|
- [X] `0.3.4` — Vertical slice PostgreSQL `RawAccountState` complète sur `ksp-store-lib` + `ksp-store-postgres-lib` : quatre capabilities account ajoutées aux six transaction pour une conformance RAW 10/10, V002 additive de 32 ressources au-dessus de V000/V001 immuables, state+observation atomiques, idempotence/conflit exacts, metadata Yellowstone observation-only, get/list keyset `(slot,pubkey,state_hash)` avec cursor KSPA anti-replay, hardening cross-family et live validé sur PostgreSQL 17 sans rétention destructive account.
|
||||||
- [ ] `0.3.5` — Étendre `ksp-interface-lib` uniquement avec les modèles passifs/event-only dont une matrice des surfaces HTTP/WS/Yellowstone/Helius démontre la sémantique réellement partagée et le besoin consumer ; auditer notamment logs, slot/root/slotsUpdates, transaction status et vote, sans dupliquer `RawTransaction`/`RawAccountState`, sans event bus et sans dépendance Interface vers Transport/Store.
|
- [X] `0.3.5` — `ksp-interface-lib` étendu avec deux familles passives réellement partagées : `SlotLifecycleEvent` (`Processed`, `FirstShredReceived`, `Completed`, `CreatedBank`, `Dead`, `OptimisticallyConfirmed`, `Rooted`) et `TransactionExecutionEvent` (`slot + TransactionSignature[64] + Succeeded/Failed`). Interface reste Core-only, provider-neutral, sans serde/codec/runtime/event bus et sans duplication de `RawTransaction`/`RawAccountState`; les DTOs riches restent Transport-owned et les candidats non convergents restent différés.
|
||||||
- [ ] `0.3.6` — Introduire `ksp-job-api` et un premier job de backfill historique concret consommant `ksp-store-lib`, avec policy/batch-size/progression possédés par le job et non par Store.
|
- [X] `0.3.6` — `ksp-job-api` + `ksp-job-backfill-lib` stables pour la première verticale historique `RawTransaction` : lifecycle/cancellation/latest-value runtime-neutral, scopes `Latest`/`Before`/`After`/signatures explicites, découverte/hydratation Transport observée, RAW v1 canonique, persistance Store atomique/idempotente, concurrence bornée, frontier contiguë, checkpoint/reprise caller-owned, snapshots sûrs et hardening externe sans dépendance backend/provider directe.
|
||||||
- [ ] `0.3.7` — Introduire une application spécialisée de backfill/inspection RAW.
|
- [ ] `0.3.7` — Introduire `ksp-app-backfill-desk`, application Tauri spécialisée de contrôle et monitoring de `ksp-job-backfill-lib`, composée via Config : sélection de composition/profil, lancement/annulation, scopes/anchors, limites, progression latest-value, compteurs, frontier/checkpoint/reprise et outcome terminal. Elle ne devient ni navigateur Store généraliste, ni scheduler, ni propriétaire des retries/providers/backends.
|
||||||
- [ ] Compléter ensuite la couche RAW avec le worker/service live, son contrôle et les outils d’exploitation réellement nécessaires avant de passer à la couche de normalisation générique suivante.
|
- [ ] `0.3.8` — Introduire `ksp-app-store-desk` V1, application Tauri backend-agnostique de diagnostic, consultation et requêtage du Store via `ksp-store-lib` + Config : health/capabilities, `RawTransaction`, `RawAccountState`, observations, rétention/tombstones et pagination sûre. Cette application évoluera ensuite avec les couches STRUCTURAL, DECODED, processing/materialization et DOMAIN réellement ajoutées au Store, sans SQL ni backend physique dans l'UI.
|
||||||
|
- [ ] `0.3.9` — Introduire `ksp-worker-api` comme API générique de lifecycle/health/progression pour services continus, en reprenant le pattern latest-value stabilisé par Job tout en gardant les sémantiques Worker distinctes des jobs terminables et des wake-ups Store post-commit.
|
||||||
|
- [ ] `0.3.10` — Introduire `ksp-worker-raw-transaction-ingest-lib` pour l'acquisition continue de `RawTransaction` via les surfaces live de `ksp-onchain-transport-lib`, persistance atomique par `ksp-store-lib`, reprise/backpressure/idempotence et notifications `ksp-worker-api`, sans decode Program ni dépendance backend/provider directe.
|
||||||
|
- [ ] `0.3.11` — Introduire `ksp-app-raw-transaction-ingest-desk`, application Tauri spécialisée de contrôle et monitoring du worker live : lifecycle, health, rates, backpressure, reconnect/recovery et compteurs sûrs ; la consultation détaillée des données persistées reste la responsabilité de `ksp-app-store-desk`.
|
||||||
|
|
||||||
|
### TODO/IDEAS — applications spécialisées et control plane
|
||||||
|
|
||||||
|
- [ ] **TODO** — faire évoluer `ksp-app-store-desk` avec chaque nouvelle couche réellement persistée : RAW d'abord, puis STRUCTURAL, DECODED, journal de processing/materialization et projections DOMAIN selon les contrats effectivement disponibles. Les vues avancées restent backend-agnostiques et passent uniquement par `ksp-store-lib`.
|
||||||
|
- [ ] **TODO** — différer une future `ksp-app-control-desk` jusqu'à ce que KSP dispose au minimum d'un niveau N3/D3 de processing/materialization exploitable et de plusieurs decoders réels. Cette application sera une surface **end-user simplifiée** : démarrer/arrêter les flux autorisés, lancer une recherche courante, voir l'état global et les erreurs importantes. Elle ne recopiera pas le diagnostic détaillé, les réglages avancés, les tables complètes ni toutes les fonctions des `ksp-app-*-desk` spécialisées, qui resteront les outils d'administration, développement et investigation approfondie.
|
||||||
|
|
||||||
### TODO/IDEAS — taxonomie N1, processing et rétention
|
### TODO/IDEAS — taxonomie N1, processing et rétention
|
||||||
|
|
||||||
- [ ] **TODO** — maintenir la matrice d’admission HTTP/WS/gRPC/provider lors de toute nouvelle famille N1 : plusieurs sources ne convergent vers un même struct que si elles satisfont la même sémantique sans perte.
|
- [ ] **TODO** — maintenir la matrice d’admission HTTP/WS/gRPC/provider lors de toute nouvelle famille N1 : plusieurs sources ne convergent vers un même struct que si elles satisfont la même sémantique sans perte.
|
||||||
- [X] `RawAccountState` + observation — contrat commun stabilisé en `0.3.1` avec bytes complets + slot, provenance séparée et enrichissements source-specific optionnels ; la persistence PostgreSQL physique est complétée en `0.3.4` avec les quatre capabilities account et la conformance RAW 10/10.
|
- [X] `RawAccountState` + observation — contrat commun stabilisé en `0.3.1` avec bytes complets + slot, provenance séparée et enrichissements source-specific optionnels ; la persistence PostgreSQL physique est complétée en `0.3.4` avec les quatre capabilities account et la conformance RAW 10/10.
|
||||||
- [ ] **TODO** — `TransactionStatusObservation` : réauditer `signatureSubscribe`, `getSignatureStatuses`, Yellowstone TransactionStatus et extensions provider lorsqu’un consumer réel apparaît ; ne pas fusionner snapshot, transition et update dans un modèle Option-soup.
|
- [ ] **TODO** — statut/commitment transactionnel restant : `0.3.5` couvre uniquement le fait passif d’exécution `slot + signature + outcome`; réauditer séparément `signatureSubscribe` et `getSignatureStatuses` lorsqu’un consumer de commitment/snapshot réel apparaît, sans fusionner snapshot, transition et execution update dans un modèle Option-soup.
|
||||||
- [ ] **TODO** — logs realtime : conserver `logMessages` dans `RawTransaction` jusqu’à la décomposition STRUCTURAL ; traiter `logsSubscribe` comme event-only candidat et décider son contrat passif dans `ksp-interface-lib`, sans table Store par défaut. Le format canonique d’un wake-up « donnée persistée disponible » reste distinct et appartient à `ksp-store-api` conformément à `KSP-NOTIFY-*`, mais ne sera matérialisé qu’avec un publisher/consumer réel.
|
- [ ] **IDEA** — logs realtime enrichis : `logsSubscribe` alimente déjà la projection minimale `TransactionExecutionEvent`, mais un éventuel `TransactionLogEvent` portant les lignes de log reste différé dans `docs/IDEAS.md` jusqu’à démonstration d’un consumer et de bornes explicites. `logMessages` reste dans `RawTransaction` jusqu’à STRUCTURAL ; le wake-up post-commit reste distinct et Store API-owned conformément à `KSP-NOTIFY-*`.
|
||||||
- [ ] **TODO** — slot/root/slotsUpdates et vote : ne créer un modèle passif commun que si un consumer realtime réel et une sémantique cross-ledger/provider justifient le contrat ; aucune persistence Store par défaut.
|
- [X] slot/root/slotsUpdates — `0.3.5` stabilise `SlotLifecycleEvent` pour l’intersection réellement partagée, sans persistence Store par défaut et sans promettre l’ordre/complétude du flux.
|
||||||
|
- [ ] **TODO** — vote realtime : reste hors Interface tant qu’aucun consumer transversal et aucune sémantique provider-neutral suffisamment précise ne justifient un contrat partagé.
|
||||||
- [ ] **IDEA** — `RawBlock` : ne rouvrir que si une information block-level non reconstructible devient nécessaire ; `getBlock` doit d’abord être traité comme source de `RawTransaction`, pas comme invitation à recopier le ledger en blocs.
|
- [ ] **IDEA** — `RawBlock` : ne rouvrir que si une information block-level non reconstructible devient nécessaire ; `getBlock` doit d’abord être traité comme source de `RawTransaction`, pas comme invitation à recopier le ledger en blocs.
|
||||||
- [ ] **REJET ACTUEL** — Yellowstone `Entry` : trop bas niveau et aucune destination replay/decomposition/event métier justifiant un modèle KSP n’est identifiée.
|
- [ ] **REJET ACTUEL** — Yellowstone `Entry` : trop bas niveau et aucune destination replay/decomposition/event métier justifiant un modèle KSP n’est identifiée.
|
||||||
- [ ] **TODO** — processing ledger : reprendre l’idée kbot2/kbot3 `stage + processor identity/version + input identity/hash + terminal status`, sans faire d’un `processed: bool` la preuve durable unique ; prévoir force replay/version upgrades lorsque les processors seront ouverts.
|
- [ ] **TODO** — processing ledger : reprendre l’idée kbot2/kbot3 `stage + processor identity/version + input identity/hash + terminal status`, sans faire d’un `processed: bool` la preuve durable unique ; prévoir force replay/version upgrades lorsque les processors seront ouverts.
|
||||||
@@ -124,7 +133,8 @@ RAW -> STRUCTURAL -> DECODED -> DOMAIN
|
|||||||
- [ ] Implémenter en priorité `RawTransaction -> STRUCTURAL` sans decoder Program : transaction/message, comptes/références, instructions top-level, CPI/inner instructions, logs/meta/balances/return data et relations structurelles.
|
- [ ] Implémenter en priorité `RawTransaction -> STRUCTURAL` sans decoder Program : transaction/message, comptes/références, instructions top-level, CPI/inner instructions, logs/meta/balances/return data et relations structurelles.
|
||||||
- [ ] Vérifier avant extension si d'autres familles N1 possèdent une vraie décomposition STRUCTURAL utile ; ne pas créer de niveau vide par convention.
|
- [ ] Vérifier avant extension si d'autres familles N1 possèdent une vraie décomposition STRUCTURAL utile ; ne pas créer de niveau vide par convention.
|
||||||
- [ ] Ajouter replay/backfill RAW -> STRUCTURAL avec processing versionné.
|
- [ ] Ajouter replay/backfill RAW -> STRUCTURAL avec processing versionné.
|
||||||
- [ ] Ajouter worker/service STRUCTURAL et l'application de contrôle/inspection utile.
|
- [ ] Ajouter le worker/service STRUCTURAL utile sans le coupler à un worker RAW concret.
|
||||||
|
- [ ] Faire évoluer `ksp-app-store-desk` avec des vues/requêtes STRUCTURAL lorsque cette couche est réellement persistée ; ne pas créer une seconde application de browsing des mêmes données.
|
||||||
|
|
||||||
## Séries DECODED/DOMAIN/EXECUTION — progression verticale
|
## Séries DECODED/DOMAIN/EXECUTION — progression verticale
|
||||||
|
|
||||||
|
|||||||
@@ -1,9 +1,11 @@
|
|||||||
<!-- file: crates/ksp-interface-lib/README.md -->
|
<!-- file: crates/ksp-interface-lib/README.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# ksp-interface-lib
|
# ksp-interface-lib
|
||||||
|
|
||||||
`ksp-interface-lib` est la façade wire officielle KSP destinée aux contrats passifs partagés par les implémentations Program Solana officielles ou externes. La crate expose uniquement des structures de représentation/admission ; elle ne possède ni transport, ni exécution, ni persistence, ni comportement métier Program.
|
`ksp-interface-lib` possède les contrats passifs KSP qui doivent être partagés entre plusieurs composants sans imposer leur runtime d'origine. Sa surface couvre actuellement deux familles distinctes : les contrats wire Program génériques et un petit ensemble de faits d'acquisition provider-neutral dont la sémantique commune a été démontrée.
|
||||||
|
|
||||||
|
La crate reste une façade de représentation. Elle ne possède ni transport réseau, ni event bus, ni worker/job, ni persistence, ni exécution, ni comportement métier Program.
|
||||||
|
|
||||||
## Ownership
|
## Ownership
|
||||||
|
|
||||||
@@ -15,9 +17,9 @@ Error / ErrorCode / Result
|
|||||||
Program IDs fondamentaux
|
Program IDs fondamentaux
|
||||||
```
|
```
|
||||||
|
|
||||||
`Pubkey` est réexporté depuis le crate-root Interface afin qu'un consumer wire n'introduise aucun wrapper d'identité parallèle. Les Program IDs restent possédés et répertoriés par Core.
|
`Pubkey` est réexporté depuis le crate-root Interface afin qu'un consumer n'introduise aucun wrapper d'identité parallèle. Les Program IDs restent possédés et répertoriés par Core.
|
||||||
|
|
||||||
La dependency direction candidate `0.2.13` reste strictement :
|
Le graphe normal reste strictement :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
ksp-interface-lib
|
ksp-interface-lib
|
||||||
@@ -25,9 +27,11 @@ ksp-interface-lib
|
|||||||
└── solana-pubkey
|
└── solana-pubkey
|
||||||
```
|
```
|
||||||
|
|
||||||
## Surface publique `0.2.13`
|
La crate ne possède aucune feature Cargo, aucune `dev-dependency` et aucune `build-dependency` runtime propre.
|
||||||
|
|
||||||
La façade crate-root expose exactement :
|
## Surface publique
|
||||||
|
|
||||||
|
La façade crate-root expose :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
Pubkey
|
Pubkey
|
||||||
@@ -36,10 +40,17 @@ MAX_PROGRAM_INSTRUCTION_ACCOUNTS
|
|||||||
ProgramInstruction
|
ProgramInstruction
|
||||||
MAX_PROGRAM_INSTRUCTION_DATA_LEN
|
MAX_PROGRAM_INSTRUCTION_DATA_LEN
|
||||||
ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED
|
ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED
|
||||||
|
SlotLifecycleStage
|
||||||
|
SlotLifecycleEvent
|
||||||
|
TransactionSignature
|
||||||
|
TransactionExecutionOutcome
|
||||||
|
TransactionExecutionEvent
|
||||||
```
|
```
|
||||||
|
|
||||||
Aucun module interne n'est public.
|
Aucun module interne n'est public.
|
||||||
|
|
||||||
|
## Contrats Program passifs
|
||||||
|
|
||||||
### `ProgramAccountMeta`
|
### `ProgramAccountMeta`
|
||||||
|
|
||||||
`ProgramAccountMeta` représente un compte ordonné d'une instruction avec :
|
`ProgramAccountMeta` représente un compte ordonné d'une instruction avec :
|
||||||
@@ -68,16 +79,14 @@ data: Vec<u8>
|
|||||||
|
|
||||||
Les cas vides sont valides et une `program_id` inconnue du registry KSP reste admissible.
|
Les cas vides sont valides et une `program_id` inconnue du registry KSP reste admissible.
|
||||||
|
|
||||||
## Bornes d'admission
|
### Bornes d'admission
|
||||||
|
|
||||||
Interface applique deux limites locales :
|
|
||||||
|
|
||||||
| Limite | Valeur |
|
| Limite | Valeur |
|
||||||
|------------------------------------|----------|
|
|------------------------------------|----------|
|
||||||
| `MAX_PROGRAM_INSTRUCTION_ACCOUNTS` | `255` |
|
| `MAX_PROGRAM_INSTRUCTION_ACCOUNTS` | `255` |
|
||||||
| `MAX_PROGRAM_INSTRUCTION_DATA_LEN` | `10_240` |
|
| `MAX_PROGRAM_INSTRUCTION_DATA_LEN` | `10_240` |
|
||||||
|
|
||||||
Ces valeurs sont des **bornes d'admission Interface**. Elles ne constituent pas une garantie qu'une instruction donnée tient dans toutes les contraintes d'une transaction Solana top-level. La limite CPI de comptes uniques n'est notamment pas transformée en règle artificielle sur la liste d'account metas.
|
Ces valeurs sont des **bornes d'admission Interface**. Elles ne constituent pas une garantie qu'une instruction donnée respecte à elle seule toutes les contraintes d'une transaction Solana complète.
|
||||||
|
|
||||||
Un dépassement utilise le code commun :
|
Un dépassement utilise le code commun :
|
||||||
|
|
||||||
@@ -89,21 +98,74 @@ Le contexte d'erreur est limité aux métadonnées sûres `field`, `actual_len`
|
|||||||
|
|
||||||
Le `Debug` de `ProgramInstruction` est volontairement borné : il affiche `program_id`, `account_count` et `data_len`, jamais les accounts complets ni les octets du payload.
|
Le `Debug` de `ProgramInstruction` est volontairement borné : il affiche `program_id`, `account_count` et `data_len`, jamais les accounts complets ni les octets du payload.
|
||||||
|
|
||||||
|
## Faits passifs d'acquisition
|
||||||
|
|
||||||
|
Les événements Interface sont des **projections provider-neutral supplémentaires**. Ils ne remplacent jamais les DTOs riches de `ksp-onchain-transport-lib` et ne deviennent jamais la source de vérité durable d'un backlog.
|
||||||
|
|
||||||
|
La conversion depuis un DTO HTTP/WS/gRPC/provider appartient à la composition ou au consumer qui connaît les deux contrats. `ksp-interface-lib` ne dépend donc pas de Transport.
|
||||||
|
|
||||||
|
### `SlotLifecycleEvent`
|
||||||
|
|
||||||
|
`SlotLifecycleEvent` conserve exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
slot: u64
|
||||||
|
stage: SlotLifecycleStage
|
||||||
|
```
|
||||||
|
|
||||||
|
Les stages actuellement représentés sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Processed
|
||||||
|
FirstShredReceived
|
||||||
|
Completed
|
||||||
|
CreatedBank
|
||||||
|
Dead
|
||||||
|
OptimisticallyConfirmed
|
||||||
|
Rooted
|
||||||
|
```
|
||||||
|
|
||||||
|
`SlotLifecycleStage` est `#[non_exhaustive]` afin qu'un consumer externe traite explicitement l'évolution future de l'enum.
|
||||||
|
|
||||||
|
Parent, timestamp, diagnostics de slot mort, source/provider, filter/session metadata et autres détails Transport ne sont pas copiés dans ce contrat minimal.
|
||||||
|
|
||||||
|
### `TransactionExecutionEvent`
|
||||||
|
|
||||||
|
`TransactionExecutionEvent` conserve exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
slot: u64
|
||||||
|
signature: TransactionSignature
|
||||||
|
outcome: TransactionExecutionOutcome
|
||||||
|
```
|
||||||
|
|
||||||
|
`TransactionSignature` contient exactement les 64 bytes canoniques déjà décodés d'une signature Solana. Son `Debug` ne rend pas les bytes.
|
||||||
|
|
||||||
|
`TransactionExecutionOutcome` distingue uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Succeeded
|
||||||
|
Failed
|
||||||
|
```
|
||||||
|
|
||||||
|
L'enum est `#[non_exhaustive]`. Le contrat ne contient aucun log, payload RAW, erreur provider, commitment, provenance, network id, source metadata ni détail d'exécution arbitraire. Une source ambiguë ou insuffisante doit rester dans son owner Transport au lieu de forcer un événement Interface.
|
||||||
|
|
||||||
## Codecs et runtime
|
## Codecs et runtime
|
||||||
|
|
||||||
La foundation `0.2.13` n'ajoute aucun codec par réflexe :
|
La crate n'ajoute aucun codec ou runtime par réflexe :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
serde / serde_json absents
|
serde / serde_json absents
|
||||||
borsh absent
|
borsh absent tant qu'aucun wire réel ne le requiert
|
||||||
wincode absent
|
wincode absent tant qu'aucun wire réel ne le requiert
|
||||||
bincode absent
|
bincode interdit pour les codecs wire KSP
|
||||||
solana-instruction absent
|
transport/runtime réseau absent
|
||||||
|
logging runtime absent
|
||||||
```
|
```
|
||||||
|
|
||||||
Des codecs/layouts/discriminants spécifiques pourront être ajoutés ultérieurement uniquement lorsqu'un vertical Program réel en démontre le besoin et que leur ownership wire appartient bien à Interface.
|
Des codecs/layouts/discriminants spécifiques peuvent être ajoutés uniquement lorsqu'un protocole réel en démontre le besoin et que leur ownership wire appartient bien à Interface.
|
||||||
|
|
||||||
La crate ne produit aucun événement runtime. Elle ne dépend donc pas de `ksp-logging-lib` et ne possède ni `constants.rs` ni `TRACING_TARGET`. Si un futur comportement Interface exige réellement du logging, le runtime devra passer par la façade Logging KSP plutôt que par une dépendance directe à Tracing.
|
Les événements passifs ne constituent pas un comportement runtime. La crate ne dépend donc pas de `ksp-logging-lib` et ne possède ni `constants.rs` ni `TRACING_TARGET`.
|
||||||
|
|
||||||
## Frontières
|
## Frontières
|
||||||
|
|
||||||
@@ -112,20 +174,20 @@ La crate ne produit aucun événement runtime. Elle ne dépend donc pas de `ksp-
|
|||||||
```text
|
```text
|
||||||
RPC / WebSocket / gRPC
|
RPC / WebSocket / gRPC
|
||||||
provider DTOs Transport
|
provider DTOs Transport
|
||||||
wallet / signature
|
sessions / reconnect / backpressure
|
||||||
Config / environnement
|
Config / environnement
|
||||||
persistence / Store
|
persistence / Store / cursor / retention
|
||||||
|
notifications post-commit Store
|
||||||
|
worker / job / scheduler / event bus
|
||||||
Program decoding / recognition / proofs
|
Program decoding / recognition / proofs
|
||||||
execution policy / signers
|
execution policy / signers
|
||||||
transaction replay / CPI path / runtime logs
|
RawTransaction / RawAccountState
|
||||||
lifecycle réseau
|
|
||||||
```
|
```
|
||||||
|
|
||||||
La foundation Program API est reportée à `0.2.14`. Les wires génériques d'acquisition/CORE restent reportés à `0.3.2+`.
|
Les données persistantes/replayables restent dans `ksp-store-api`. Les notifications post-commit de données persistées restent un contrat Store API lorsqu'un publisher/consumer réel les justifie. Les faits réseau riches restent Transport-owned.
|
||||||
|
|
||||||
## Références
|
## Références
|
||||||
|
|
||||||
- [Usage public](USAGE.md)
|
- [Usage public](USAGE.md)
|
||||||
- [Plan `0.2.13`](../../docs/plans/020-V0_2_13_INTERFACE_PLAN.md)
|
|
||||||
- [Validation `0.2.13`](../../docs/validation/016-V0_2_13_INTERFACE.md)
|
|
||||||
- [Architecture Wire + Program](../../docs/architecture/006-WIRE_AND_PROGRAM.md)
|
- [Architecture Wire + Program](../../docs/architecture/006-WIRE_AND_PROGRAM.md)
|
||||||
|
- [Architecture Acquisition/Workers/Jobs](../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md)
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
<!-- file: crates/ksp-interface-lib/USAGE.md -->
|
<!-- file: crates/ksp-interface-lib/USAGE.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# Usage de ksp-interface-lib
|
# Usage de ksp-interface-lib
|
||||||
|
|
||||||
Cette page décrit la façade publique matérialisée par `0.2.13`. Les modules internes ne font pas partie du contrat consommable : utiliser uniquement les exports du crate-root.
|
Cette page décrit l'utilisation durable de la façade publique. Les modules internes ne font pas partie du contrat consommable : utiliser uniquement les exports du crate-root.
|
||||||
|
|
||||||
## Construire des account metas
|
## Construire des account metas
|
||||||
|
|
||||||
@@ -52,9 +52,9 @@ match result {
|
|||||||
|
|
||||||
L'ordre et les doublons des accounts sont conservés. Les octets `data` restent opaques : `ProgramInstruction` ne les sérialise, désérialise ni interprète.
|
L'ordre et les doublons des accounts sont conservés. Les octets `data` restent opaques : `ProgramInstruction` ne les sérialise, désérialise ni interprète.
|
||||||
|
|
||||||
Les `Vec` fournis à `try_new` sont consommés par la structure après validation des bornes ; aucun clone ou reformatage interne n'est requis par le contrat actuel.
|
Les `Vec` fournis à `try_new` sont consommés par la structure après validation des bornes ; aucun clone ou reformatage interne n'est requis par le contrat.
|
||||||
|
|
||||||
## Bornes
|
## Respecter les bornes Program
|
||||||
|
|
||||||
Les limites publiques sont :
|
Les limites publiques sont :
|
||||||
|
|
||||||
@@ -63,9 +63,9 @@ assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS, 255);
|
|||||||
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN, 10_240);
|
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN, 10_240);
|
||||||
```
|
```
|
||||||
|
|
||||||
`255` account metas et `10_240` bytes de data sont admis. `256` account metas ou `10_241` bytes sont refusés par `try_new` avant création d'un `ProgramInstruction` valide.
|
`255` account metas et `10_240` bytes de data sont admis. `256` account metas ou `10_241` bytes sont refusés par `ProgramInstruction::try_new`.
|
||||||
|
|
||||||
Ces limites sont des bornes locales Interface et ne promettent pas qu'une instruction admise respecte à elle seule toutes les contraintes de taille/account-set d'une transaction Solana complète.
|
Ces limites sont locales à Interface et ne promettent pas qu'une instruction admise respecte à elle seule toutes les contraintes d'une transaction Solana complète.
|
||||||
|
|
||||||
## Observer une erreur de limite
|
## Observer une erreur de limite
|
||||||
|
|
||||||
@@ -92,31 +92,108 @@ maximum_len
|
|||||||
|
|
||||||
Le contenu du payload et les account metas arbitraires ne sont pas projetés dans le diagnostic.
|
Le contenu du payload et les account metas arbitraires ne sont pas projetés dans le diagnostic.
|
||||||
|
|
||||||
## Debug borné
|
## Construire un événement de lifecycle de slot
|
||||||
|
|
||||||
Le `Debug` de `ProgramInstruction` contient uniquement :
|
Un producer/composer qui a déjà établi la correspondance sémantique avec son DTO Transport peut construire le fait passif partagé :
|
||||||
|
|
||||||
```text
|
```rust
|
||||||
program_id
|
let event = ksp_interface_lib::SlotLifecycleEvent::new(
|
||||||
account_count
|
42,
|
||||||
data_len
|
ksp_interface_lib::SlotLifecycleStage::Processed,
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(event.slot(), 42);
|
||||||
|
assert_eq!(event.stage(), ksp_interface_lib::SlotLifecycleStage::Processed);
|
||||||
```
|
```
|
||||||
|
|
||||||
Il ne faut donc pas attendre de ce rendu une sérialisation wire ou un dump du payload.
|
Les stages actuellement disponibles sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Processed
|
||||||
|
FirstShredReceived
|
||||||
|
Completed
|
||||||
|
CreatedBank
|
||||||
|
Dead
|
||||||
|
OptimisticallyConfirmed
|
||||||
|
Rooted
|
||||||
|
```
|
||||||
|
|
||||||
|
Un consumer externe doit traiter `SlotLifecycleStage` comme une enum évolutive `#[non_exhaustive]` et prévoir un fallback dans ses `match`.
|
||||||
|
|
||||||
|
Le type ne contient volontairement ni timestamp, ni parent, ni diagnostic, ni source/provider, ni identifiant de subscription.
|
||||||
|
|
||||||
|
## Construire un événement d'exécution de transaction
|
||||||
|
|
||||||
|
Une signature doit d'abord être disponible sous sa forme canonique de 64 bytes :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let signature = ksp_interface_lib::TransactionSignature::new([7_u8; 64]);
|
||||||
|
|
||||||
|
let event = ksp_interface_lib::TransactionExecutionEvent::new(
|
||||||
|
123,
|
||||||
|
signature,
|
||||||
|
ksp_interface_lib::TransactionExecutionOutcome::Succeeded,
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(event.slot(), 123);
|
||||||
|
assert_eq!(event.signature().as_bytes(), &[7_u8; 64]);
|
||||||
|
assert_eq!(
|
||||||
|
event.outcome(),
|
||||||
|
ksp_interface_lib::TransactionExecutionOutcome::Succeeded,
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
`TransactionExecutionOutcome` distingue uniquement `Succeeded` et `Failed` et reste `#[non_exhaustive]`.
|
||||||
|
|
||||||
|
Le `Debug` de `TransactionSignature` et de `TransactionExecutionEvent` n'affiche pas les bytes de signature.
|
||||||
|
|
||||||
|
## Convertir depuis Transport
|
||||||
|
|
||||||
|
Ne pas ajouter `ksp-onchain-transport-lib` comme dépendance de `ksp-interface-lib` pour fournir des `From<TransportDto>`.
|
||||||
|
|
||||||
|
La conversion appartient au composant qui connaît les deux côtés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Transport DTO riche
|
||||||
|
|
|
||||||
|
| conversion explicite dans composition/consumer
|
||||||
|
v
|
||||||
|
Interface event passif minimal
|
||||||
|
```
|
||||||
|
|
||||||
|
Ne construire un événement Interface que si la source fournit suffisamment d'information pour le fait commun exact. Si l'état est ambigu, conserver le DTO dans son owner Transport ou déclencher une hydratation adaptée ; ne pas inventer de valeur par défaut.
|
||||||
|
|
||||||
|
## Ne pas utiliser les événements comme stockage durable
|
||||||
|
|
||||||
|
`SlotLifecycleEvent` et `TransactionExecutionEvent` sont des faits passifs, pas des modèles RAW replayables.
|
||||||
|
|
||||||
|
Ils ne remplacent pas :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransaction
|
||||||
|
RawTransactionObservation
|
||||||
|
RawAccountState
|
||||||
|
RawAccountObservation
|
||||||
|
Store backlog / cursor / retention
|
||||||
|
```
|
||||||
|
|
||||||
|
Les consumers qui ont besoin de reprise après crash ou de replay doivent s'appuyer sur `ksp-store-lib`/`ksp-store-api` selon leur responsabilité, pas sur un event Interface en mémoire.
|
||||||
|
|
||||||
## Dépendances à ne pas ajouter côté consumer
|
## Dépendances à ne pas ajouter côté consumer
|
||||||
|
|
||||||
Un consumer de la façade Interface n'a pas besoin d'ajouter un SDK Program Solana uniquement pour reconstruire `ProgramInstruction`. La crate utilise le `Pubkey` canonique partagé avec Core et conserve son propre contrat passif.
|
Un consumer de la façade Interface n'a pas besoin d'ajouter un SDK Program Solana uniquement pour reconstruire `ProgramInstruction`, ni un runtime Transport pour manipuler les événements passifs déjà normalisés.
|
||||||
|
|
||||||
La foundation ne fournit volontairement pas :
|
La crate ne fournit volontairement pas :
|
||||||
|
|
||||||
```text
|
```text
|
||||||
serde générique
|
serde générique
|
||||||
Borsh / Wincode générique
|
Borsh / Wincode générique
|
||||||
solana-instruction interop automatique
|
solana-instruction interop automatique
|
||||||
transport réseau
|
transport réseau
|
||||||
|
converters provider automatiques
|
||||||
Program decoder/preparer
|
Program decoder/preparer
|
||||||
signing/execution
|
signing/execution
|
||||||
|
persistence/event bus
|
||||||
```
|
```
|
||||||
|
|
||||||
Ces surfaces doivent être introduites dans leur owner respectif lorsqu'un cas réel le justifie, pas comme dépendances implicites d'un consumer Interface.
|
Ces surfaces doivent être introduites dans leur owner respectif lorsqu'un cas réel le justifie.
|
||||||
|
|||||||
@@ -1,20 +1,22 @@
|
|||||||
// file: crates/ksp-interface-lib/src/lib.rs
|
// file: crates/ksp-interface-lib/src/lib.rs
|
||||||
// version: 3
|
// version: 5
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
#![forbid(unsafe_code)]
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
//! Passive wire contracts shared by KSP Program implementations.
|
//! Passive contracts shared across KSP component boundaries.
|
||||||
//!
|
//!
|
||||||
//! The foundation reuses the canonical Solana [`Pubkey`] owned by
|
//! The crate reuses canonical Solana primitives owned by `ksp-core-lib` and
|
||||||
//! `ksp-core-lib` and exposes only bounded, passive Program-facing structures.
|
//! exposes only bounded, passive Program-facing and provider-neutral
|
||||||
//! Runtime, transport, persistence and Program behavior remain outside this
|
//! acquisition structures. Runtime, transport, persistence and Program
|
||||||
//! crate.
|
//! behavior remain outside this crate.
|
||||||
|
|
||||||
mod error;
|
mod error;
|
||||||
mod program_account_meta;
|
mod program_account_meta;
|
||||||
mod program_instruction;
|
mod program_instruction;
|
||||||
|
mod slot_lifecycle;
|
||||||
|
mod transaction_execution;
|
||||||
|
|
||||||
/// Error code used when an Interface-owned Program instruction admission limit is exceeded.
|
/// Error code used when an Interface-owned Program instruction admission limit is exceeded.
|
||||||
pub use self::error::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED;
|
pub use self::error::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED;
|
||||||
@@ -26,5 +28,15 @@ pub use self::program_account_meta::ProgramAccountMeta;
|
|||||||
pub use self::program_instruction::MAX_PROGRAM_INSTRUCTION_DATA_LEN;
|
pub use self::program_instruction::MAX_PROGRAM_INSTRUCTION_DATA_LEN;
|
||||||
/// Passive, bounded Program instruction wire contract.
|
/// Passive, bounded Program instruction wire contract.
|
||||||
pub use self::program_instruction::ProgramInstruction;
|
pub use self::program_instruction::ProgramInstruction;
|
||||||
|
/// Passive provider-neutral occurrence of one slot lifecycle stage.
|
||||||
|
pub use self::slot_lifecycle::SlotLifecycleEvent;
|
||||||
|
/// Provider-neutral stage in the lifecycle of an observed Solana slot.
|
||||||
|
pub use self::slot_lifecycle::SlotLifecycleStage;
|
||||||
|
/// Passive provider-neutral observation of one transaction execution result.
|
||||||
|
pub use self::transaction_execution::TransactionExecutionEvent;
|
||||||
|
/// Provider-neutral outcome of one observed transaction execution.
|
||||||
|
pub use self::transaction_execution::TransactionExecutionOutcome;
|
||||||
|
/// Canonical 64-byte Solana transaction signature used by passive Interface events.
|
||||||
|
pub use self::transaction_execution::TransactionSignature;
|
||||||
/// Canonical Solana account address primitive owned by `ksp-core-lib`.
|
/// Canonical Solana account address primitive owned by `ksp-core-lib`.
|
||||||
pub use ksp_core_lib::Pubkey;
|
pub use ksp_core_lib::Pubkey;
|
||||||
|
|||||||
53
crates/ksp-interface-lib/src/slot_lifecycle.rs
Normal file
53
crates/ksp-interface-lib/src/slot_lifecycle.rs
Normal file
@@ -0,0 +1,53 @@
|
|||||||
|
// file: crates/ksp-interface-lib/src/slot_lifecycle.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Provider-neutral stage in the lifecycle of an observed Solana slot.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum SlotLifecycleStage {
|
||||||
|
/// The slot has been processed.
|
||||||
|
Processed,
|
||||||
|
/// The first shred for the slot has been received.
|
||||||
|
FirstShredReceived,
|
||||||
|
/// Slot ingestion has completed.
|
||||||
|
Completed,
|
||||||
|
/// A bank has been created for the slot.
|
||||||
|
CreatedBank,
|
||||||
|
/// The slot has been marked dead.
|
||||||
|
Dead,
|
||||||
|
/// The slot has reached optimistic confirmation.
|
||||||
|
OptimisticallyConfirmed,
|
||||||
|
/// The slot has become rooted.
|
||||||
|
Rooted,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Passive provider-neutral occurrence of one slot lifecycle stage.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
pub struct SlotLifecycleEvent {
|
||||||
|
slot: u64,
|
||||||
|
stage: SlotLifecycleStage,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SlotLifecycleEvent {
|
||||||
|
/// Creates a lifecycle event for `slot` and `stage`.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(slot: u64, stage: SlotLifecycleStage) -> Self {
|
||||||
|
return Self { slot, stage };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the observed slot exactly as supplied.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn slot(&self) -> u64 {
|
||||||
|
return self.slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the provider-neutral lifecycle stage.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn stage(&self) -> SlotLifecycleStage {
|
||||||
|
return self.stage;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/slot_lifecycle.rs"]
|
||||||
|
mod tests;
|
||||||
85
crates/ksp-interface-lib/src/transaction_execution.rs
Normal file
85
crates/ksp-interface-lib/src/transaction_execution.rs
Normal file
@@ -0,0 +1,85 @@
|
|||||||
|
// file: crates/ksp-interface-lib/src/transaction_execution.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Canonical 64-byte Solana transaction signature used by passive Interface events.
|
||||||
|
#[derive(Clone, Copy, Eq, Hash, Ord, PartialEq, PartialOrd)]
|
||||||
|
pub struct TransactionSignature([u8; 64]);
|
||||||
|
|
||||||
|
impl TransactionSignature {
|
||||||
|
/// Creates one transaction signature from already-decoded canonical bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(bytes: [u8; 64]) -> Self {
|
||||||
|
return Self(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the exact canonical signature bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn as_bytes(&self) -> &[u8; 64] {
|
||||||
|
return &self.0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for TransactionSignature {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.write_str("TransactionSignature(..)");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Provider-neutral outcome of one observed transaction execution.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum TransactionExecutionOutcome {
|
||||||
|
/// The observed transaction execution completed without a transaction error.
|
||||||
|
Succeeded,
|
||||||
|
/// The observed transaction execution completed with a transaction error.
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Passive provider-neutral observation of one transaction execution result.
|
||||||
|
#[derive(Clone, Copy, Eq, PartialEq)]
|
||||||
|
pub struct TransactionExecutionEvent {
|
||||||
|
slot: u64,
|
||||||
|
signature: TransactionSignature,
|
||||||
|
outcome: TransactionExecutionOutcome,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TransactionExecutionEvent {
|
||||||
|
/// Creates one transaction execution event from its common provider-neutral fields.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(slot: u64, signature: TransactionSignature, outcome: TransactionExecutionOutcome) -> Self {
|
||||||
|
return Self { slot, signature, outcome };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the containing slot exactly as observed by the producer.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn slot(&self) -> u64 {
|
||||||
|
return self.slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the canonical transaction signature.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn signature(&self) -> TransactionSignature {
|
||||||
|
return self.signature;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the normalized execution outcome.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn outcome(&self) -> TransactionExecutionOutcome {
|
||||||
|
return self.outcome;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for TransactionExecutionEvent {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("TransactionExecutionEvent")
|
||||||
|
.field("slot", &self.slot)
|
||||||
|
.field("signature", &"<omitted>")
|
||||||
|
.field("outcome", &self.outcome)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/transaction_execution.rs"]
|
||||||
|
mod tests;
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-interface-lib/tests/dependency_boundary.rs
|
// file: crates/ksp-interface-lib/tests/dependency_boundary.rs
|
||||||
// version: 5
|
// version: 8
|
||||||
|
|
||||||
//! Dependency and passive-surface canaries for the Interface foundation.
|
//! Dependency and passive-surface canaries for the Interface foundation.
|
||||||
|
|
||||||
@@ -91,6 +91,8 @@ fn pre_005_all_production_sources_preserve_the_dependency_firewall() {
|
|||||||
include_str!("../src/lib.rs"),
|
include_str!("../src/lib.rs"),
|
||||||
include_str!("../src/program_account_meta.rs"),
|
include_str!("../src/program_account_meta.rs"),
|
||||||
include_str!("../src/program_instruction.rs"),
|
include_str!("../src/program_instruction.rs"),
|
||||||
|
include_str!("../src/slot_lifecycle.rs"),
|
||||||
|
include_str!("../src/transaction_execution.rs"),
|
||||||
];
|
];
|
||||||
for source in production_sources {
|
for source in production_sources {
|
||||||
for forbidden in [
|
for forbidden in [
|
||||||
@@ -129,3 +131,52 @@ fn pre_005_instruction_source_has_no_narrowing_cast_or_hidden_codec_entry_point(
|
|||||||
}
|
}
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_004_acquisition_sources_remain_provider_neutral_and_runtime_free() {
|
||||||
|
let acquisition_sources = [include_str!("../src/slot_lifecycle.rs"), include_str!("../src/transaction_execution.rs")];
|
||||||
|
for source in acquisition_sources {
|
||||||
|
for forbidden in [
|
||||||
|
"borsh::",
|
||||||
|
"bincode::",
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_logging_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ksp_store_api::",
|
||||||
|
"ksp_store_lib::",
|
||||||
|
"reqwest::",
|
||||||
|
"serde::",
|
||||||
|
"serde_json::",
|
||||||
|
"solana_instruction::",
|
||||||
|
"tauri::",
|
||||||
|
"tokio::",
|
||||||
|
"tonic::",
|
||||||
|
concat!("tracing", "::"),
|
||||||
|
"wincode::",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "forbidden acquisition dependency path detected: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_005_manifest_keeps_exact_core_only_graph_without_features_or_hidden_dependency_sections() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
assert!(!manifest.contains("[features]"));
|
||||||
|
assert!(!manifest.contains("[dev-dependencies]"));
|
||||||
|
assert!(!manifest.contains("[build-dependencies]"));
|
||||||
|
assert_eq!(manifest.matches("[dependencies]").count(), 1);
|
||||||
|
let dependencies_tail = manifest.split("[dependencies]").nth(1);
|
||||||
|
assert!(dependencies_tail.is_some(), "Interface dependencies section must exist");
|
||||||
|
let dependencies_tail = match dependencies_tail {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let dependencies = match dependencies_tail.split("[lints]").next() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(manifest_dependency_names(dependencies), std::vec!["ksp-core-lib"]);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,12 +1,22 @@
|
|||||||
// file: crates/ksp-interface-lib/tests/external_consumer.rs
|
// file: crates/ksp-interface-lib/tests/external_consumer.rs
|
||||||
// version: 1
|
// version: 2
|
||||||
|
|
||||||
//! Downstream-style consumer canary for the public Interface facade.
|
//! Downstream-style consumer canaries for the public Interface facade.
|
||||||
|
|
||||||
fn consume_instruction(instruction: ksp_interface_lib::ProgramInstruction) -> (ksp_interface_lib::Pubkey, usize, usize) {
|
fn consume_instruction(instruction: ksp_interface_lib::ProgramInstruction) -> (ksp_interface_lib::Pubkey, usize, usize) {
|
||||||
return (*instruction.program_id(), instruction.accounts().len(), instruction.data().len());
|
return (*instruction.program_id(), instruction.accounts().len(), instruction.data().len());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn consume_slot_lifecycle(event: ksp_interface_lib::SlotLifecycleEvent) -> (u64, ksp_interface_lib::SlotLifecycleStage) {
|
||||||
|
return (event.slot(), event.stage());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn consume_transaction_execution(
|
||||||
|
event: ksp_interface_lib::TransactionExecutionEvent,
|
||||||
|
) -> (u64, ksp_interface_lib::TransactionSignature, ksp_interface_lib::TransactionExecutionOutcome) {
|
||||||
|
return (event.slot(), event.signature(), event.outcome());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_005_external_consumer_uses_only_the_crate_root_facade() {
|
fn pre_005_external_consumer_uses_only_the_crate_root_facade() {
|
||||||
let program_id = ksp_interface_lib::Pubkey::new_from_array([0xA1_u8; 32]);
|
let program_id = ksp_interface_lib::Pubkey::new_from_array([0xA1_u8; 32]);
|
||||||
@@ -25,3 +35,13 @@ fn pre_005_external_consumer_uses_only_the_crate_root_facade() {
|
|||||||
assert_eq!(ksp_interface_lib::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED.domain(), "interface");
|
assert_eq!(ksp_interface_lib::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED.domain(), "interface");
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_004_external_consumer_uses_only_crate_root_for_both_acquisition_families() {
|
||||||
|
let lifecycle = ksp_interface_lib::SlotLifecycleEvent::new(u64::MAX, ksp_interface_lib::SlotLifecycleStage::OptimisticallyConfirmed);
|
||||||
|
assert_eq!(consume_slot_lifecycle(lifecycle), (u64::MAX, ksp_interface_lib::SlotLifecycleStage::OptimisticallyConfirmed));
|
||||||
|
let signature = ksp_interface_lib::TransactionSignature::new([0xA3_u8; 64]);
|
||||||
|
let execution = ksp_interface_lib::TransactionExecutionEvent::new(u64::MAX - 1, signature, ksp_interface_lib::TransactionExecutionOutcome::Failed);
|
||||||
|
assert_eq!(consume_transaction_execution(execution), (u64::MAX - 1, signature, ksp_interface_lib::TransactionExecutionOutcome::Failed));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,75 +1,75 @@
|
|||||||
// file: crates/ksp-interface-lib/tests/release_completeness.rs
|
// file: crates/ksp-interface-lib/tests/release_completeness.rs
|
||||||
// version: 1
|
// version: 5
|
||||||
|
|
||||||
//! Release-level completeness canaries for the `0.2.13` Interface foundation.
|
//! Release-completeness canaries for the public `ksp-interface-lib` surface.
|
||||||
|
|
||||||
#[test]
|
fn crate_root_source() -> std::result::Result<std::string::String, std::io::Error> {
|
||||||
fn pre_005_exact_crate_root_export_inventory_is_stable() {
|
let path = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src/lib.rs");
|
||||||
let crate_root = include_str!("../src/lib.rs");
|
return std::fs::read_to_string(path);
|
||||||
let mut actual = std::vec::Vec::new();
|
|
||||||
for line in crate_root.lines() {
|
|
||||||
let trimmed = line.trim();
|
|
||||||
if trimmed.starts_with("pub use ") {
|
|
||||||
actual.push(trimmed);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
actual.sort_unstable();
|
|
||||||
let mut expected = std::vec![
|
|
||||||
"pub use self::error::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED;",
|
|
||||||
"pub use self::program_account_meta::MAX_PROGRAM_INSTRUCTION_ACCOUNTS;",
|
|
||||||
"pub use self::program_account_meta::ProgramAccountMeta;",
|
|
||||||
"pub use self::program_instruction::MAX_PROGRAM_INSTRUCTION_DATA_LEN;",
|
|
||||||
"pub use self::program_instruction::ProgramInstruction;",
|
|
||||||
"pub use ksp_core_lib::Pubkey;",
|
|
||||||
];
|
|
||||||
expected.sort_unstable();
|
|
||||||
assert_eq!(actual, expected);
|
|
||||||
assert!(!crate_root.contains("pub mod "));
|
|
||||||
return;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn pre_005_production_module_inventory_contains_no_second_wire_domain() -> std::io::Result<()> {
|
fn v0_3_5_pre_004_foundation_keeps_exactly_two_passive_acquisition_families() {
|
||||||
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
|
||||||
let entries = match std::fs::read_dir(source_root) {
|
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let mut names = std::vec::Vec::new();
|
|
||||||
for entry in entries {
|
|
||||||
let entry = match entry {
|
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
let file_type = match entry.file_type() {
|
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
if !file_type.is_file() {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
let name = match entry.file_name().into_string() {
|
|
||||||
std::result::Result::Ok(value) => value,
|
|
||||||
std::result::Result::Err(_) => continue,
|
|
||||||
};
|
|
||||||
if name.ends_with(".rs") {
|
|
||||||
names.push(name);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
names.sort_unstable();
|
|
||||||
assert_eq!(names, std::vec!["error.rs", "lib.rs", "program_account_meta.rs", "program_instruction.rs"]);
|
|
||||||
return std::result::Result::Ok(());
|
|
||||||
}
|
|
||||||
|
|
||||||
#[test]
|
|
||||||
fn pre_005_foundation_has_one_error_code_and_two_bounded_passive_types() {
|
|
||||||
assert_eq!(ksp_interface_lib::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED.domain(), "interface");
|
assert_eq!(ksp_interface_lib::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED.domain(), "interface");
|
||||||
assert_eq!(ksp_interface_lib::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED.code(), "program_instruction_limit_exceeded");
|
assert_eq!(ksp_interface_lib::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED.code(), "program_instruction_limit_exceeded");
|
||||||
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS, 255);
|
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS, 255);
|
||||||
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN, 10_240);
|
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN, 10_240);
|
||||||
let program_id = ksp_interface_lib::Pubkey::new_from_array([0xB1_u8; 32]);
|
let lifecycle = ksp_interface_lib::SlotLifecycleEvent::new(u64::MAX, ksp_interface_lib::SlotLifecycleStage::Rooted);
|
||||||
let account = ksp_interface_lib::ProgramAccountMeta::readonly(ksp_interface_lib::Pubkey::new_from_array([0xB2_u8; 32]), false);
|
assert_eq!(lifecycle.slot(), u64::MAX);
|
||||||
let instruction = ksp_interface_lib::ProgramInstruction::try_new(program_id, std::vec![account], std::vec![0xB3_u8]);
|
assert_eq!(lifecycle.stage(), ksp_interface_lib::SlotLifecycleStage::Rooted);
|
||||||
assert!(instruction.is_ok());
|
let signature = ksp_interface_lib::TransactionSignature::new([0xB4_u8; 64]);
|
||||||
|
let execution = ksp_interface_lib::TransactionExecutionEvent::new(u64::MAX, signature, ksp_interface_lib::TransactionExecutionOutcome::Succeeded);
|
||||||
|
assert_eq!(execution.slot(), u64::MAX);
|
||||||
|
assert_eq!(execution.signature(), signature);
|
||||||
|
assert_eq!(execution.outcome(), ksp_interface_lib::TransactionExecutionOutcome::Succeeded);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_004_exact_crate_root_export_inventory_is_stable() -> std::result::Result<(), std::io::Error> {
|
||||||
|
let source = match crate_root_source() {
|
||||||
|
Ok(source) => source,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let public_use_count = source.lines().filter(|line| return line.starts_with("pub use ")).count();
|
||||||
|
assert_eq!(public_use_count, 11);
|
||||||
|
assert!(source.contains("pub use self::error::ERROR_CODE_PROGRAM_INSTRUCTION_LIMIT_EXCEEDED;"));
|
||||||
|
assert!(source.contains("pub use self::program_account_meta::MAX_PROGRAM_INSTRUCTION_ACCOUNTS;"));
|
||||||
|
assert!(source.contains("pub use self::program_account_meta::ProgramAccountMeta;"));
|
||||||
|
assert!(source.contains("pub use self::program_instruction::MAX_PROGRAM_INSTRUCTION_DATA_LEN;"));
|
||||||
|
assert!(source.contains("pub use self::program_instruction::ProgramInstruction;"));
|
||||||
|
assert!(source.contains("pub use self::slot_lifecycle::SlotLifecycleEvent;"));
|
||||||
|
assert!(source.contains("pub use self::slot_lifecycle::SlotLifecycleStage;"));
|
||||||
|
assert!(source.contains("pub use self::transaction_execution::TransactionExecutionEvent;"));
|
||||||
|
assert!(source.contains("pub use self::transaction_execution::TransactionExecutionOutcome;"));
|
||||||
|
assert!(source.contains("pub use self::transaction_execution::TransactionSignature;"));
|
||||||
|
assert!(source.contains("pub use ksp_core_lib::Pubkey;"));
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_004_production_module_inventory_is_exact() -> std::result::Result<(), std::io::Error> {
|
||||||
|
let source = match crate_root_source() {
|
||||||
|
Ok(source) => source,
|
||||||
|
Err(error) => return Err(error),
|
||||||
|
};
|
||||||
|
let modules = source
|
||||||
|
.lines()
|
||||||
|
.filter_map(|line| return line.strip_prefix("mod ").and_then(|module| return module.strip_suffix(';')))
|
||||||
|
.collect::<std::vec::Vec<_>>();
|
||||||
|
assert_eq!(modules, std::vec!["error", "program_account_meta", "program_instruction", "slot_lifecycle", "transaction_execution"]);
|
||||||
|
assert!(!source.contains("serde"));
|
||||||
|
assert!(!source.contains("tracing"));
|
||||||
|
assert!(!source.contains("ksp_onchain_transport"));
|
||||||
|
assert!(!source.contains("ksp_store"));
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_004_evolutive_acquisition_enums_remain_non_exhaustive() {
|
||||||
|
let lifecycle_source = include_str!("../src/slot_lifecycle.rs");
|
||||||
|
let execution_source = include_str!("../src/transaction_execution.rs");
|
||||||
|
assert!(lifecycle_source.contains("#[non_exhaustive]\npub enum SlotLifecycleStage"));
|
||||||
|
assert!(execution_source.contains("#[non_exhaustive]\npub enum TransactionExecutionOutcome"));
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|||||||
100
crates/ksp-interface-lib/tests/security_hardening.rs
Normal file
100
crates/ksp-interface-lib/tests/security_hardening.rs
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
// file: crates/ksp-interface-lib/tests/security_hardening.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! Adversarial and ownership-hardening canaries for passive Interface acquisition events.
|
||||||
|
|
||||||
|
const HOSTILE_MARKER: &str = "INTERFACE-ACQUISITION-SECRET-CANARY";
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_005_signature_and_event_debug_never_render_signature_bytes() {
|
||||||
|
let mut bytes = [0x41_u8; 64];
|
||||||
|
bytes[..HOSTILE_MARKER.len()].copy_from_slice(HOSTILE_MARKER.as_bytes());
|
||||||
|
let signature = ksp_interface_lib::TransactionSignature::new(bytes);
|
||||||
|
assert_eq!(signature.as_bytes(), &bytes);
|
||||||
|
let signature_debug = std::format!("{signature:?}");
|
||||||
|
assert_eq!(signature_debug, "TransactionSignature(..)");
|
||||||
|
assert!(!signature_debug.contains(HOSTILE_MARKER));
|
||||||
|
let event = ksp_interface_lib::TransactionExecutionEvent::new(u64::MAX, signature, ksp_interface_lib::TransactionExecutionOutcome::Failed);
|
||||||
|
let event_debug = std::format!("{event:?}");
|
||||||
|
assert!(event_debug.contains("TransactionExecutionEvent"));
|
||||||
|
assert!(event_debug.contains("18446744073709551615"));
|
||||||
|
assert!(event_debug.contains("Failed"));
|
||||||
|
assert!(!event_debug.contains(HOSTILE_MARKER));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_005_acquisition_structs_keep_exact_minimal_fields_without_payload_or_source_metadata() {
|
||||||
|
let lifecycle_source = include_str!("../src/slot_lifecycle.rs");
|
||||||
|
let execution_source = include_str!("../src/transaction_execution.rs");
|
||||||
|
assert!(lifecycle_source.contains("pub struct SlotLifecycleEvent {\n slot: u64,\n stage: SlotLifecycleStage,\n}"));
|
||||||
|
assert!(execution_source.contains("pub struct TransactionSignature([u8; 64]);"));
|
||||||
|
assert!(execution_source.contains(
|
||||||
|
"pub struct TransactionExecutionEvent {\n slot: u64,\n signature: TransactionSignature,\n outcome: TransactionExecutionOutcome,\n}"
|
||||||
|
));
|
||||||
|
for source in [lifecycle_source, execution_source] {
|
||||||
|
for forbidden in [
|
||||||
|
"std::vec::Vec<",
|
||||||
|
"std::string::String",
|
||||||
|
"Box<",
|
||||||
|
"serde_json::Value",
|
||||||
|
"timestamp:",
|
||||||
|
"provider_id:",
|
||||||
|
"endpoint:",
|
||||||
|
"filter:",
|
||||||
|
"network:",
|
||||||
|
"payload:",
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "forbidden acquisition field/surface detected: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_005_interface_exports_no_second_raw_or_deferred_event_family() {
|
||||||
|
let crate_root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in [
|
||||||
|
"RawTransaction",
|
||||||
|
"RawAccount",
|
||||||
|
"RawPayload",
|
||||||
|
"RawPageCursor",
|
||||||
|
"RawRetention",
|
||||||
|
"RawObservation",
|
||||||
|
"TransactionLogEvent",
|
||||||
|
"TransactionCommitmentEvent",
|
||||||
|
"LedgerEntryEvent",
|
||||||
|
"VoteEvent",
|
||||||
|
"BlockEvent",
|
||||||
|
] {
|
||||||
|
assert!(!crate_root.contains(forbidden), "forbidden Interface export detected: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn v0_3_5_pre_005_program_instruction_foundation_remains_bounded_and_unchanged() {
|
||||||
|
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS, 255);
|
||||||
|
assert_eq!(ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN, 10_240);
|
||||||
|
let program_id = ksp_interface_lib::Pubkey::new_from_array([0xD1_u8; 32]);
|
||||||
|
let account = ksp_interface_lib::ProgramAccountMeta::readonly(ksp_interface_lib::Pubkey::new_from_array([0xD2_u8; 32]), false);
|
||||||
|
let admitted = ksp_interface_lib::ProgramInstruction::try_new(
|
||||||
|
program_id,
|
||||||
|
std::vec![account; ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS],
|
||||||
|
std::vec![0x5A_u8; ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN],
|
||||||
|
);
|
||||||
|
assert!(admitted.is_ok());
|
||||||
|
let admitted = match admitted {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(admitted.accounts().len(), ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS);
|
||||||
|
assert_eq!(admitted.data().len(), ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN);
|
||||||
|
let too_many_accounts =
|
||||||
|
ksp_interface_lib::ProgramInstruction::try_new(program_id, std::vec![account; ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_ACCOUNTS + 1], std::vec![]);
|
||||||
|
assert!(too_many_accounts.is_err());
|
||||||
|
let oversized_data =
|
||||||
|
ksp_interface_lib::ProgramInstruction::try_new(program_id, std::vec![], std::vec![0x5A_u8; ksp_interface_lib::MAX_PROGRAM_INSTRUCTION_DATA_LEN + 1]);
|
||||||
|
assert!(oversized_data.is_err());
|
||||||
|
return;
|
||||||
|
}
|
||||||
29
crates/ksp-interface-lib/tests/slot_lifecycle_public_api.rs
Normal file
29
crates/ksp-interface-lib/tests/slot_lifecycle_public_api.rs
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
// file: crates/ksp-interface-lib/tests/slot_lifecycle_public_api.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Public API canaries for the shared slot lifecycle event contract.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_5_pre_002_slot_lifecycle_contract_is_available_from_crate_root() {
|
||||||
|
let event = ksp_interface_lib::SlotLifecycleEvent::new(u64::MAX, ksp_interface_lib::SlotLifecycleStage::Rooted);
|
||||||
|
assert_eq!(event.slot(), u64::MAX);
|
||||||
|
assert_eq!(event.stage(), ksp_interface_lib::SlotLifecycleStage::Rooted);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_5_pre_002_slot_lifecycle_stage_remains_downstream_evolvable() {
|
||||||
|
fn stage_label(stage: ksp_interface_lib::SlotLifecycleStage) -> &'static str {
|
||||||
|
return match stage {
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::Processed => "processed",
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::FirstShredReceived => "first_shred_received",
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::Completed => "completed",
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::CreatedBank => "created_bank",
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::Dead => "dead",
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::OptimisticallyConfirmed => "optimistically_confirmed",
|
||||||
|
ksp_interface_lib::SlotLifecycleStage::Rooted => "rooted",
|
||||||
|
_ => "future",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
assert_eq!(stage_label(ksp_interface_lib::SlotLifecycleStage::Processed), "processed");
|
||||||
|
assert_eq!(stage_label(ksp_interface_lib::SlotLifecycleStage::Rooted), "rooted");
|
||||||
|
}
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
// file: crates/ksp-interface-lib/tests/transaction_execution_public_api.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! Public API canaries for the provider-neutral transaction execution contract.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_5_pre_003_transaction_execution_contract_is_available_from_crate_root() {
|
||||||
|
let signature = ksp_interface_lib::TransactionSignature::new([0xC7_u8; 64]);
|
||||||
|
let event = ksp_interface_lib::TransactionExecutionEvent::new(u64::MAX, signature, ksp_interface_lib::TransactionExecutionOutcome::Succeeded);
|
||||||
|
assert_eq!(signature.as_bytes(), &[0xC7_u8; 64]);
|
||||||
|
assert_eq!(event.slot(), u64::MAX);
|
||||||
|
assert_eq!(event.signature(), signature);
|
||||||
|
assert_eq!(event.outcome(), ksp_interface_lib::TransactionExecutionOutcome::Succeeded);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn outcome_label(outcome: ksp_interface_lib::TransactionExecutionOutcome) -> &'static str {
|
||||||
|
return match outcome {
|
||||||
|
ksp_interface_lib::TransactionExecutionOutcome::Succeeded => "succeeded",
|
||||||
|
ksp_interface_lib::TransactionExecutionOutcome::Failed => "failed",
|
||||||
|
_ => "future",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_5_pre_003_transaction_execution_outcome_remains_downstream_evolvable() {
|
||||||
|
assert_eq!(outcome_label(ksp_interface_lib::TransactionExecutionOutcome::Succeeded), "succeeded");
|
||||||
|
assert_eq!(outcome_label(ksp_interface_lib::TransactionExecutionOutcome::Failed), "failed");
|
||||||
|
}
|
||||||
45
crates/ksp-interface-lib/unit_tests/slot_lifecycle.rs
Normal file
45
crates/ksp-interface-lib/unit_tests/slot_lifecycle.rs
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
// file: crates/ksp-interface-lib/unit_tests/slot_lifecycle.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn slot_lifecycle_stages_are_distinct_copy_and_complete_for_the_admitted_family() {
|
||||||
|
fn assert_copy<T: Copy>() {}
|
||||||
|
assert_copy::<crate::SlotLifecycleStage>();
|
||||||
|
let stages = [
|
||||||
|
crate::SlotLifecycleStage::Processed,
|
||||||
|
crate::SlotLifecycleStage::FirstShredReceived,
|
||||||
|
crate::SlotLifecycleStage::Completed,
|
||||||
|
crate::SlotLifecycleStage::CreatedBank,
|
||||||
|
crate::SlotLifecycleStage::Dead,
|
||||||
|
crate::SlotLifecycleStage::OptimisticallyConfirmed,
|
||||||
|
crate::SlotLifecycleStage::Rooted,
|
||||||
|
];
|
||||||
|
for (index, stage) in stages.iter().enumerate() {
|
||||||
|
for other in stages.iter().skip(index + 1) {
|
||||||
|
assert_ne!(stage, other);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn slot_lifecycle_event_preserves_full_u64_slot_and_stage() {
|
||||||
|
fn assert_copy<T: Copy>() {}
|
||||||
|
assert_copy::<crate::SlotLifecycleEvent>();
|
||||||
|
let event = crate::SlotLifecycleEvent::new(u64::MAX, crate::SlotLifecycleStage::Rooted);
|
||||||
|
assert_eq!(event.slot(), u64::MAX);
|
||||||
|
assert_eq!(event.stage(), crate::SlotLifecycleStage::Rooted);
|
||||||
|
let copied = event;
|
||||||
|
assert_eq!(event, copied);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn slot_lifecycle_debug_is_bounded_and_contains_only_shared_fields() {
|
||||||
|
let event = crate::SlotLifecycleEvent::new(42, crate::SlotLifecycleStage::OptimisticallyConfirmed);
|
||||||
|
let debug = std::format!("{event:?}");
|
||||||
|
assert!(debug.len() <= 128);
|
||||||
|
assert!(debug.contains("slot: 42"));
|
||||||
|
assert!(debug.contains("OptimisticallyConfirmed"));
|
||||||
|
assert!(!debug.contains("provider"));
|
||||||
|
assert!(!debug.contains("yellowstone"));
|
||||||
|
assert!(!debug.contains("websocket"));
|
||||||
|
}
|
||||||
28
crates/ksp-interface-lib/unit_tests/transaction_execution.rs
Normal file
28
crates/ksp-interface-lib/unit_tests/transaction_execution.rs
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
// file: crates/ksp-interface-lib/unit_tests/transaction_execution.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn transaction_signature_preserves_exact_bytes_and_redacts_debug() {
|
||||||
|
let signature = crate::TransactionSignature::new([0xA5_u8; 64]);
|
||||||
|
assert_eq!(signature.as_bytes(), &[0xA5_u8; 64]);
|
||||||
|
assert_eq!(std::format!("{signature:?}"), "TransactionSignature(..)");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn transaction_execution_outcomes_are_distinct_and_copy() {
|
||||||
|
let succeeded = crate::TransactionExecutionOutcome::Succeeded;
|
||||||
|
let failed = crate::TransactionExecutionOutcome::Failed;
|
||||||
|
assert_ne!(succeeded, failed);
|
||||||
|
let copied = succeeded;
|
||||||
|
assert_eq!(copied, crate::TransactionExecutionOutcome::Succeeded);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn transaction_execution_event_preserves_full_slot_signature_and_outcome() {
|
||||||
|
let signature = crate::TransactionSignature::new([0xB6_u8; 64]);
|
||||||
|
let event = crate::TransactionExecutionEvent::new(u64::MAX, signature, crate::TransactionExecutionOutcome::Failed);
|
||||||
|
assert_eq!(event.slot(), u64::MAX);
|
||||||
|
assert_eq!(event.signature(), signature);
|
||||||
|
assert_eq!(event.outcome(), crate::TransactionExecutionOutcome::Failed);
|
||||||
|
assert_eq!(std::format!("{event:?}"), "TransactionExecutionEvent { slot: 18446744073709551615, signature: \"<omitted>\", outcome: Failed }");
|
||||||
|
}
|
||||||
14
crates/ksp-job-api/Cargo.toml
Normal file
14
crates/ksp-job-api/Cargo.toml
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
# file: crates/ksp-job-api/Cargo.toml
|
||||||
|
# version: 1
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "ksp-job-api"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
72
crates/ksp-job-api/README.md
Normal file
72
crates/ksp-job-api/README.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
<!-- file: crates/ksp-job-api/README.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# ksp-job-api
|
||||||
|
|
||||||
|
`ksp-job-api` fournit les contrats passifs et runtime-neutral communs aux jobs bornés de KSP.
|
||||||
|
|
||||||
|
La crate possède l'identité logique d'un job, son lifecycle terminable, l'intention d'annulation coopérative et le contrat latest-value utilisé pour observer un snapshot complet. Elle ne possède aucun runtime concret, aucune politique métier de backfill, aucun Transport, aucun Store et aucun Worker.
|
||||||
|
|
||||||
|
## Responsabilités
|
||||||
|
|
||||||
|
La façade crate-root expose :
|
||||||
|
|
||||||
|
- `JobId` et `JobKindCode`, bornés et validés ;
|
||||||
|
- `JobState` et `JobCompletion` ;
|
||||||
|
- `JobLifecycle`, propriétaire des transitions admises ;
|
||||||
|
- `JobCancellationToken`, cloneable et idempotent ;
|
||||||
|
- `JobNotificationSequence`, strictement monotone et sans wrap silencieux ;
|
||||||
|
- `JobNotification<S>`, valeur observable complète à une position donnée ;
|
||||||
|
- `JobSnapshotSource`, contrat runtime-neutral de lecture courante et attente d'une valeur plus récente ;
|
||||||
|
- les codes d'erreur Job et les types `Error`/`Result` communs de Core.
|
||||||
|
|
||||||
|
## Lifecycle
|
||||||
|
|
||||||
|
Le lifecycle admis reste explicitement borné :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Created
|
||||||
|
| | +----------------------> Cancelled
|
||||||
|
v
|
||||||
|
Running
|
||||||
|
| | +----> Cancelling -----> Cancelled
|
||||||
|
| | | | +---------> Failed
|
||||||
|
| +-------------> Completed(Complete|Partial)
|
||||||
|
|
|
||||||
|
+------------------------> Completed(Complete|Partial)
|
||||||
|
+------------------------> Failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Tout état terminal est immuable. Une transition invalide retourne `ERROR_CODE_JOB_TRANSITION_INVALID` sans modifier l'état source.
|
||||||
|
|
||||||
|
## Observation latest-value
|
||||||
|
|
||||||
|
`JobSnapshotSource` n'impose ni callback, ni queue d'événements, ni runtime async particulier. Un listener peut :
|
||||||
|
|
||||||
|
1. lire la valeur complète courante ;
|
||||||
|
2. mémoriser sa `JobNotificationSequence` ;
|
||||||
|
3. attendre une valeur plus récente ;
|
||||||
|
4. recevoir directement la dernière valeur complète, même si plusieurs mises à jour intermédiaires ont été coalescées.
|
||||||
|
|
||||||
|
Le snapshot concret appartient au job qui implémente la source. `ksp-job-api` ne connaît pas son contenu.
|
||||||
|
|
||||||
|
## Annulation
|
||||||
|
|
||||||
|
`JobCancellationToken` représente uniquement une intention coopérative partagée. Il ne tue pas une tâche, n'annule pas une I/O par lui-même et ne décide pas du résultat terminal. Le runtime concret reste responsable d'observer le token aux frontières sûres et de publier son état final.
|
||||||
|
|
||||||
|
## Firewall
|
||||||
|
|
||||||
|
La dépendance normale est volontairement minimale :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-job-api
|
||||||
|
-> ksp-core-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate ne dépend pas de Tokio, Futures, Transport, Store, Config, Logging, serde ni d'une crate de job concret.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [`USAGE.md`](USAGE.md) — utilisation durable des contrats Job ;
|
||||||
|
- [`../../docs/architecture/003-COMPONENT_CONTRACTS.md`](../../docs/architecture/003-COMPONENT_CONTRACTS.md) — contrats de composants ;
|
||||||
|
- [`../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md`](../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md) — séparation workers/jobs et premier backfill RAW.
|
||||||
141
crates/ksp-job-api/USAGE.md
Normal file
141
crates/ksp-job-api/USAGE.md
Normal file
@@ -0,0 +1,141 @@
|
|||||||
|
<!-- file: crates/ksp-job-api/USAGE.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Utilisation de ksp-job-api
|
||||||
|
|
||||||
|
Cette page décrit la façade publique durable de `ksp-job-api`. Les consumers utilisent uniquement les exports du crate-root.
|
||||||
|
|
||||||
|
## Construire une identité de job
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn job_identity() -> ksp_job_api::Result<(ksp_job_api::JobId, ksp_job_api::JobKindCode)> {
|
||||||
|
let id = match ksp_job_api::JobId::new("raw-backfill-mainnet-0001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let kind = match ksp_job_api::JobKindCode::new("raw_transaction_backfill") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok((id, kind));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`JobId` identifie un job logique et ses reprises contrôlées. `JobKindCode` identifie une famille de jobs. Les deux sont bornés et utilisent un alphabet sûr ; leur `Debug` n'est pas une surface destinée à transporter un payload métier.
|
||||||
|
|
||||||
|
## Piloter un lifecycle passif
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn lifecycle(id: ksp_job_api::JobId, kind: ksp_job_api::JobKindCode) -> ksp_job_api::Result<ksp_job_api::JobLifecycle> {
|
||||||
|
let mut lifecycle = ksp_job_api::JobLifecycle::new(id, kind);
|
||||||
|
if let std::result::Result::Err(error) = lifecycle.start() {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = lifecycle.complete(ksp_job_api::JobCompletion::Complete) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(lifecycle);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le producer ne doit pas forcer un état directement. Il utilise les opérations de `JobLifecycle`, qui refusent les transitions hors matrice.
|
||||||
|
|
||||||
|
Pour une annulation observée pendant l'exécution :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
if let std::result::Result::Err(error) = lifecycle.mark_cancelling() {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if let std::result::Result::Err(error) = lifecycle.mark_cancelled() {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Partager une intention d'annulation
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let token = ksp_job_api::JobCancellationToken::new();
|
||||||
|
let listener = token.clone();
|
||||||
|
|
||||||
|
assert!(!listener.is_cancellation_requested());
|
||||||
|
assert!(token.request_cancellation());
|
||||||
|
assert!(listener.is_cancellation_requested());
|
||||||
|
assert!(!token.request_cancellation());
|
||||||
|
```
|
||||||
|
|
||||||
|
Le premier appel qui change l'état retourne `true`. Les demandes suivantes sont idempotentes et retournent `false`.
|
||||||
|
|
||||||
|
Le token ne doit pas être interprété comme une primitive de kill : le runtime concret décide où l'annulation peut interrompre l'admission ou une attente et quelles opérations déjà engagées doivent être drainées.
|
||||||
|
|
||||||
|
## Publier une valeur latest-value
|
||||||
|
|
||||||
|
Un producer concret peut construire une notification complète :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[derive(Clone)]
|
||||||
|
struct Snapshot {
|
||||||
|
completed: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
let id = ksp_job_api::JobId::new("job-0001")?;
|
||||||
|
let kind = ksp_job_api::JobKindCode::new("example")?;
|
||||||
|
let sequence = ksp_job_api::JobNotificationSequence::initial();
|
||||||
|
let notification = ksp_job_api::JobNotification::new(
|
||||||
|
id,
|
||||||
|
kind,
|
||||||
|
sequence,
|
||||||
|
ksp_job_api::JobState::Running,
|
||||||
|
Snapshot { completed: 0 },
|
||||||
|
);
|
||||||
|
|
||||||
|
assert_eq!(notification.sequence().value(), 0);
|
||||||
|
assert_eq!(notification.state(), ksp_job_api::JobState::Running);
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `Debug` de `JobNotification<S>` masque volontairement le snapshot. Le type concret `S` doit lui-même rester sûr à exposer lorsque le consumer accède explicitement à `snapshot()`.
|
||||||
|
|
||||||
|
## Implémenter une source de snapshots
|
||||||
|
|
||||||
|
Une implémentation concrète possède son runtime et expose seulement le contrat `JobSnapshotSource` :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
async fn observe<S>(source: &S)
|
||||||
|
where
|
||||||
|
S: ksp_job_api::JobSnapshotSource,
|
||||||
|
{
|
||||||
|
let current = source.current();
|
||||||
|
let observed = current.sequence();
|
||||||
|
let newer = source.wait_for_change(observed).await;
|
||||||
|
assert!(newer.sequence().is_after(observed));
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`wait_for_change` retourne la dernière valeur complète connue après coalescence éventuelle. Un consumer ne doit donc pas supposer qu'il recevra chaque mise à jour intermédiaire.
|
||||||
|
|
||||||
|
## Faire avancer une séquence
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let first = ksp_job_api::JobNotificationSequence::initial();
|
||||||
|
let second = match first.next() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
assert!(second.is_after(first));
|
||||||
|
```
|
||||||
|
|
||||||
|
L'épuisement de `u64` est une erreur explicite ; la séquence ne wrappe jamais silencieusement.
|
||||||
|
|
||||||
|
## Frontières à respecter
|
||||||
|
|
||||||
|
Ne pas ajouter à `ksp-job-api` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Tokio/Futures runtime concret
|
||||||
|
Transport ou Store
|
||||||
|
Config/Logging
|
||||||
|
DTO métier d'un job précis
|
||||||
|
scheduler, worker ou control plane
|
||||||
|
persistence de checkpoint
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces responsabilités appartiennent aux crates concrètes et à la composition supérieure.
|
||||||
44
crates/ksp-job-api/src/cancellation.rs
Normal file
44
crates/ksp-job-api/src/cancellation.rs
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
// file: crates/ksp-job-api/src/cancellation.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Runtime-neutral cloneable token carrying cooperative cancellation intent.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct JobCancellationToken {
|
||||||
|
requested: std::sync::Arc<std::sync::atomic::AtomicBool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl JobCancellationToken {
|
||||||
|
/// Creates a token with no cancellation request.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new() -> Self {
|
||||||
|
return Self { requested: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)) };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Requests cancellation and returns `true` only for the first request shared by all clones.
|
||||||
|
#[must_use]
|
||||||
|
pub fn cancel(&self) -> bool {
|
||||||
|
return !self.requested.swap(true, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reports whether cancellation has been requested through any clone.
|
||||||
|
#[must_use]
|
||||||
|
pub fn is_cancellation_requested(&self) -> bool {
|
||||||
|
return self.requested.load(std::sync::atomic::Ordering::Acquire);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::default::Default for JobCancellationToken {
|
||||||
|
fn default() -> Self {
|
||||||
|
return Self::new();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for JobCancellationToken {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("JobCancellationToken").field("cancellation_requested", &self.is_cancellation_requested()).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/cancellation.rs"]
|
||||||
|
mod tests;
|
||||||
12
crates/ksp-job-api/src/error.rs
Normal file
12
crates/ksp-job-api/src/error.rs
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
// file: crates/ksp-job-api/src/error.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
/// Error code used when a Job identifier violates its bounded safe-code contract.
|
||||||
|
pub const ERROR_CODE_JOB_ID_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_api", "job_id_invalid");
|
||||||
|
/// Error code used when a Job kind code violates its bounded safe-code contract.
|
||||||
|
pub const ERROR_CODE_JOB_KIND_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_api", "job_kind_invalid");
|
||||||
|
/// Error code used when a Job notification sequence cannot advance without wrapping.
|
||||||
|
pub const ERROR_CODE_JOB_NOTIFICATION_SEQUENCE_EXHAUSTED: ksp_core_lib::ErrorCode =
|
||||||
|
ksp_core_lib::ErrorCode::new("job_api", "job_notification_sequence_exhausted");
|
||||||
|
/// Error code used when a requested Job lifecycle transition is not allowed.
|
||||||
|
pub const ERROR_CODE_JOB_TRANSITION_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_api", "job_transition_invalid");
|
||||||
76
crates/ksp-job-api/src/identity.rs
Normal file
76
crates/ksp-job-api/src/identity.rs
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
// file: crates/ksp-job-api/src/identity.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Maximum UTF-8 byte length admitted for one Job identifier.
|
||||||
|
pub const MAX_JOB_ID_BYTES: usize = 128;
|
||||||
|
/// Maximum UTF-8 byte length admitted for one Job kind code.
|
||||||
|
pub const MAX_JOB_KIND_CODE_BYTES: usize = 128;
|
||||||
|
|
||||||
|
/// Bounded caller-supplied identity of one logical Job and its controlled resumptions.
|
||||||
|
#[derive(Clone, Eq, Hash, Ord, PartialEq, PartialOrd)]
|
||||||
|
pub struct JobId(std::string::String);
|
||||||
|
|
||||||
|
impl JobId {
|
||||||
|
/// Creates one non-empty Job identifier using the KSP safe-code alphabet.
|
||||||
|
pub fn new(value: impl std::convert::Into<std::string::String>) -> crate::Result<Self> {
|
||||||
|
let value = value.into();
|
||||||
|
if !valid_job_code(value.as_str(), crate::MAX_JOB_ID_BYTES) {
|
||||||
|
return std::result::Result::Err(identity_error(crate::ERROR_CODE_JOB_ID_INVALID, "job_id"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated Job identifier.
|
||||||
|
#[must_use]
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
return self.0.as_str();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for JobId {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.write_str("JobId(..)");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Bounded stable code identifying one concrete family of Jobs.
|
||||||
|
#[derive(Clone, Eq, Hash, Ord, PartialEq, PartialOrd)]
|
||||||
|
pub struct JobKindCode(std::string::String);
|
||||||
|
|
||||||
|
impl JobKindCode {
|
||||||
|
/// Creates one non-empty Job kind using the KSP safe-code alphabet.
|
||||||
|
pub fn new(value: impl std::convert::Into<std::string::String>) -> crate::Result<Self> {
|
||||||
|
let value = value.into();
|
||||||
|
if !valid_job_code(value.as_str(), crate::MAX_JOB_KIND_CODE_BYTES) {
|
||||||
|
return std::result::Result::Err(identity_error(crate::ERROR_CODE_JOB_KIND_INVALID, "job_kind"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated stable Job kind code.
|
||||||
|
#[must_use]
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
return self.0.as_str();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for JobKindCode {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_tuple("JobKindCode").field(&self.0).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn identity_error(code: crate::ErrorCode, field: &'static str) -> crate::Error {
|
||||||
|
return crate::Error::new(code, "invalid bounded Job identity").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn valid_job_code(value: &str, maximum_len: usize) -> bool {
|
||||||
|
if value.is_empty() || value.len() > maximum_len {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return value.bytes().all(|byte| return byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-' | b'.' | b':'));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/identity.rs"]
|
||||||
|
mod tests;
|
||||||
60
crates/ksp-job-api/src/lib.rs
Normal file
60
crates/ksp-job-api/src/lib.rs
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
// file: crates/ksp-job-api/src/lib.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
//! Passive runtime-neutral lifecycle contracts for bounded KSP Jobs.
|
||||||
|
//!
|
||||||
|
//! This foundation owns validated Job identity, explicit lifecycle transitions
|
||||||
|
//! cooperative cancellation intent and runtime-neutral latest-value observation.
|
||||||
|
//! Concrete Job behavior, runtime spawning, Transport, Store and Worker contracts
|
||||||
|
//! remain outside this crate.
|
||||||
|
|
||||||
|
mod cancellation;
|
||||||
|
mod error;
|
||||||
|
mod identity;
|
||||||
|
mod lifecycle;
|
||||||
|
mod notification;
|
||||||
|
|
||||||
|
/// Runtime-neutral cloneable token carrying cooperative cancellation intent.
|
||||||
|
pub use self::cancellation::JobCancellationToken;
|
||||||
|
/// Error code used when a Job identifier violates its bounded safe-code contract.
|
||||||
|
pub use self::error::ERROR_CODE_JOB_ID_INVALID;
|
||||||
|
/// Error code used when a Job kind code violates its bounded safe-code contract.
|
||||||
|
pub use self::error::ERROR_CODE_JOB_KIND_INVALID;
|
||||||
|
/// Error code used when a Job notification sequence cannot advance without wrapping.
|
||||||
|
pub use self::error::ERROR_CODE_JOB_NOTIFICATION_SEQUENCE_EXHAUSTED;
|
||||||
|
/// Error code used when a requested Job lifecycle transition is not allowed.
|
||||||
|
pub use self::error::ERROR_CODE_JOB_TRANSITION_INVALID;
|
||||||
|
/// Bounded caller-supplied identity of one logical Job and its controlled resumptions.
|
||||||
|
pub use self::identity::JobId;
|
||||||
|
/// Bounded stable code identifying one concrete family of Jobs.
|
||||||
|
pub use self::identity::JobKindCode;
|
||||||
|
/// Maximum UTF-8 byte length admitted for one Job identifier.
|
||||||
|
pub use self::identity::MAX_JOB_ID_BYTES;
|
||||||
|
/// Maximum UTF-8 byte length admitted for one Job kind code.
|
||||||
|
pub use self::identity::MAX_JOB_KIND_CODE_BYTES;
|
||||||
|
/// Completion classification of a Job that reached its normal terminal state.
|
||||||
|
pub use self::lifecycle::JobCompletion;
|
||||||
|
/// Passive owner of one Job identity and its validated lifecycle state.
|
||||||
|
pub use self::lifecycle::JobLifecycle;
|
||||||
|
/// Current lifecycle state of one bounded Job.
|
||||||
|
pub use self::lifecycle::JobState;
|
||||||
|
/// Latest complete observable value for one Job at a monotone sequence position.
|
||||||
|
pub use self::notification::JobNotification;
|
||||||
|
/// Monotone sequence attached to one latest-value Job notification stream.
|
||||||
|
pub use self::notification::JobNotificationSequence;
|
||||||
|
/// Runtime-neutral future returned while observing a latest-value Job snapshot source.
|
||||||
|
pub use self::notification::JobSnapshotFuture;
|
||||||
|
/// Runtime-neutral read and change-wait contract for one latest-value Job snapshot stream.
|
||||||
|
pub use self::notification::JobSnapshotSource;
|
||||||
|
/// Common KSP error type used by Job-facing contracts.
|
||||||
|
pub use ksp_core_lib::Error;
|
||||||
|
/// Stable structured code identifying a KSP error category and condition.
|
||||||
|
pub use ksp_core_lib::ErrorCode;
|
||||||
|
/// Structured contextual field attached to a KSP error.
|
||||||
|
pub use ksp_core_lib::ErrorContext;
|
||||||
|
/// Common KSP result alias using [`Error`].
|
||||||
|
pub use ksp_core_lib::Result;
|
||||||
168
crates/ksp-job-api/src/lifecycle.rs
Normal file
168
crates/ksp-job-api/src/lifecycle.rs
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
// file: crates/ksp-job-api/src/lifecycle.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
/// Completion classification of a Job that reached its normal terminal state.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum JobCompletion {
|
||||||
|
/// Every selected candidate or unit of work reached a durable complete result.
|
||||||
|
Complete,
|
||||||
|
/// The Job terminated normally while retaining one or more explicitly observable gaps.
|
||||||
|
Partial,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl JobCompletion {
|
||||||
|
/// Returns the stable safe code for this completion classification.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn code(&self) -> &'static str {
|
||||||
|
return match self {
|
||||||
|
Self::Complete => "complete",
|
||||||
|
Self::Partial => "partial",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Current lifecycle state of one bounded Job.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum JobState {
|
||||||
|
/// The Job was admitted but has not started producing work.
|
||||||
|
Created,
|
||||||
|
/// The Job is actively producing or draining normal work.
|
||||||
|
Running,
|
||||||
|
/// Cancellation was observed and new work must no longer be admitted.
|
||||||
|
Cancelling,
|
||||||
|
/// The Job reached a normal terminal state with an explicit completion classification.
|
||||||
|
Completed(crate::JobCompletion),
|
||||||
|
/// The Job reached its cooperative cancellation terminal state.
|
||||||
|
Cancelled,
|
||||||
|
/// The Job reached a terminal failure state.
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl JobState {
|
||||||
|
/// Returns the stable safe lifecycle code without rendering Job data.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn code(&self) -> &'static str {
|
||||||
|
return match self {
|
||||||
|
Self::Created => "created",
|
||||||
|
Self::Running => "running",
|
||||||
|
Self::Cancelling => "cancelling",
|
||||||
|
Self::Completed(_) => "completed",
|
||||||
|
Self::Cancelled => "cancelled",
|
||||||
|
Self::Failed => "failed",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the normal completion classification when the state is [`Self::Completed`].
|
||||||
|
#[must_use]
|
||||||
|
pub const fn completion(&self) -> std::option::Option<crate::JobCompletion> {
|
||||||
|
return match self {
|
||||||
|
Self::Completed(completion) => std::option::Option::Some(*completion),
|
||||||
|
_ => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reports whether no later lifecycle transition is permitted.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn is_terminal(&self) -> bool {
|
||||||
|
return matches!(self, Self::Completed(_) | Self::Cancelled | Self::Failed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Passive owner of one Job identity and its validated lifecycle state.
|
||||||
|
#[derive(Eq, PartialEq)]
|
||||||
|
pub struct JobLifecycle {
|
||||||
|
id: crate::JobId,
|
||||||
|
kind: crate::JobKindCode,
|
||||||
|
state: crate::JobState,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl JobLifecycle {
|
||||||
|
/// Creates one lifecycle in [`JobState::Created`] state.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(id: crate::JobId, kind: crate::JobKindCode) -> Self {
|
||||||
|
return Self { id, kind, state: crate::JobState::Created };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the logical Job identity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn id(&self) -> &crate::JobId {
|
||||||
|
return &self.id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the stable Job family code.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn kind(&self) -> &crate::JobKindCode {
|
||||||
|
return &self.kind;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the current lifecycle state.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn state(&self) -> crate::JobState {
|
||||||
|
return self.state;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Transitions a newly created Job into its running state.
|
||||||
|
pub fn start(&mut self) -> crate::Result<()> {
|
||||||
|
return self.transition(crate::JobState::Running);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Records that a running Job observed cooperative cancellation intent.
|
||||||
|
pub fn mark_cancelling(&mut self) -> crate::Result<()> {
|
||||||
|
return self.transition(crate::JobState::Cancelling);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Completes a running or cancelling Job with its explicit normal outcome.
|
||||||
|
pub fn complete(&mut self, completion: crate::JobCompletion) -> crate::Result<()> {
|
||||||
|
return self.transition(crate::JobState::Completed(completion));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Marks a created or cancelling Job as cooperatively cancelled.
|
||||||
|
pub fn mark_cancelled(&mut self) -> crate::Result<()> {
|
||||||
|
return self.transition(crate::JobState::Cancelled);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Marks a running or cancelling Job as failed.
|
||||||
|
pub fn fail(&mut self) -> crate::Result<()> {
|
||||||
|
return self.transition(crate::JobState::Failed);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn transition(&mut self, target: crate::JobState) -> crate::Result<()> {
|
||||||
|
if !allowed_transition(self.state, target) {
|
||||||
|
return std::result::Result::Err(transition_error(self.state, target));
|
||||||
|
}
|
||||||
|
self.state = target;
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for JobLifecycle {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.debug_struct("JobLifecycle").field("id", &self.id).field("kind", &self.kind).field("state", &self.state).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn allowed_transition(source: crate::JobState, target: crate::JobState) -> bool {
|
||||||
|
return matches!(
|
||||||
|
(source, target),
|
||||||
|
(crate::JobState::Created, crate::JobState::Running)
|
||||||
|
| (crate::JobState::Created, crate::JobState::Cancelled)
|
||||||
|
| (crate::JobState::Running, crate::JobState::Cancelling)
|
||||||
|
| (crate::JobState::Running, crate::JobState::Completed(_))
|
||||||
|
| (crate::JobState::Running, crate::JobState::Failed)
|
||||||
|
| (crate::JobState::Cancelling, crate::JobState::Completed(_))
|
||||||
|
| (crate::JobState::Cancelling, crate::JobState::Cancelled)
|
||||||
|
| (crate::JobState::Cancelling, crate::JobState::Failed)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn transition_error(source: crate::JobState, target: crate::JobState) -> crate::Error {
|
||||||
|
return crate::Error::new(crate::ERROR_CODE_JOB_TRANSITION_INVALID, "invalid Job lifecycle transition")
|
||||||
|
.with_context("source_state", source.code())
|
||||||
|
.with_context("target_state", target.code());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/lifecycle.rs"]
|
||||||
|
mod tests;
|
||||||
126
crates/ksp-job-api/src/notification.rs
Normal file
126
crates/ksp-job-api/src/notification.rs
Normal file
@@ -0,0 +1,126 @@
|
|||||||
|
// file: crates/ksp-job-api/src/notification.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
/// Monotone sequence attached to one latest-value Job notification stream.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd)]
|
||||||
|
pub struct JobNotificationSequence(u64);
|
||||||
|
|
||||||
|
impl JobNotificationSequence {
|
||||||
|
/// Creates the initial sequence position for one Job snapshot stream.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn initial() -> Self {
|
||||||
|
return Self(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the opaque numeric position carried by this sequence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn value(&self) -> u64 {
|
||||||
|
return self.0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Advances the sequence exactly once or reports exhaustion without wrapping.
|
||||||
|
pub fn next(&self) -> crate::Result<Self> {
|
||||||
|
let next = match self.0.checked_add(1) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => {
|
||||||
|
return std::result::Result::Err(
|
||||||
|
crate::Error::new(crate::ERROR_CODE_JOB_NOTIFICATION_SEQUENCE_EXHAUSTED, "Job notification sequence exhausted")
|
||||||
|
.with_context("sequence", self.0.to_string()),
|
||||||
|
);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(Self(next));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reports whether this sequence is strictly newer than an observed sequence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn is_after(&self, observed: Self) -> bool {
|
||||||
|
return self.0 > observed.0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Latest complete observable value for one Job at a monotone sequence position.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct JobNotification<S> {
|
||||||
|
id: crate::JobId,
|
||||||
|
kind: crate::JobKindCode,
|
||||||
|
sequence: crate::JobNotificationSequence,
|
||||||
|
snapshot: S,
|
||||||
|
state: crate::JobState,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<S> JobNotification<S> {
|
||||||
|
/// Creates one immutable latest-value notification from an already validated Job identity and snapshot.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn new(id: crate::JobId, kind: crate::JobKindCode, sequence: crate::JobNotificationSequence, state: crate::JobState, snapshot: S) -> Self {
|
||||||
|
return Self { id, kind, sequence, snapshot, state };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the logical Job identity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn id(&self) -> &crate::JobId {
|
||||||
|
return &self.id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the stable Job family code.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn kind(&self) -> &crate::JobKindCode {
|
||||||
|
return &self.kind;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the monotone sequence of this latest value.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn sequence(&self) -> crate::JobNotificationSequence {
|
||||||
|
return self.sequence;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the complete safe snapshot owned by the concrete Job contract.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn snapshot(&self) -> &S {
|
||||||
|
return &self.snapshot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the lifecycle state represented by this snapshot.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn state(&self) -> crate::JobState {
|
||||||
|
return self.state;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<S> std::fmt::Debug for JobNotification<S> {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("JobNotification")
|
||||||
|
.field("id", &self.id)
|
||||||
|
.field("kind", &self.kind)
|
||||||
|
.field("sequence", &self.sequence)
|
||||||
|
.field("state", &self.state)
|
||||||
|
.field("snapshot", &"<redacted>")
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runtime-neutral future returned while observing a latest-value Job snapshot source.
|
||||||
|
pub type JobSnapshotFuture<'a, S> = std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = crate::JobNotification<S>> + std::marker::Send + 'a>>;
|
||||||
|
|
||||||
|
/// Runtime-neutral read and change-wait contract for one latest-value Job snapshot stream.
|
||||||
|
pub trait JobSnapshotSource: std::marker::Send + std::marker::Sync {
|
||||||
|
/// Complete snapshot type retained by the concrete source.
|
||||||
|
type Snapshot: std::clone::Clone + std::marker::Send + std::marker::Sync + 'static;
|
||||||
|
|
||||||
|
/// Returns the complete current value without requiring replay of prior notifications.
|
||||||
|
#[must_use]
|
||||||
|
fn current(&self) -> crate::JobNotification<Self::Snapshot>;
|
||||||
|
|
||||||
|
/// Waits for a value newer than `observed`, returning the complete current snapshot after coalescing any intermediate updates.
|
||||||
|
fn wait_for_change(&self, observed: crate::JobNotificationSequence) -> crate::JobSnapshotFuture<'_, Self::Snapshot>;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
fn exhausted_notification_sequence() -> crate::JobNotificationSequence {
|
||||||
|
return JobNotificationSequence(u64::MAX);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/notification.rs"]
|
||||||
|
mod tests;
|
||||||
82
crates/ksp-job-api/tests/dependency_boundary.rs
Normal file
82
crates/ksp-job-api/tests/dependency_boundary.rs
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
// file: crates/ksp-job-api/tests/dependency_boundary.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Dependency and runtime-neutrality canaries for the Job API foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_manifest_has_exact_core_only_dependency_graph() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
assert!(!manifest.contains("[features]"));
|
||||||
|
assert!(!manifest.contains("[dev-dependencies]"));
|
||||||
|
assert!(!manifest.contains("[build-dependencies]"));
|
||||||
|
assert_eq!(manifest.matches("[dependencies]").count(), 1);
|
||||||
|
let dependencies_tail = manifest.split("[dependencies]").nth(1);
|
||||||
|
assert!(dependencies_tail.is_some());
|
||||||
|
let dependencies_tail = match dependencies_tail {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let dependencies = match dependencies_tail.split("[lints]").next() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(manifest_dependency_names(dependencies), std::vec!["ksp-core-lib"]);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_production_sources_forbid_runtime_domain_and_wire_dependencies() {
|
||||||
|
let sources = [
|
||||||
|
include_str!("../src/cancellation.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/lifecycle.rs"),
|
||||||
|
include_str!("../src/notification.rs"),
|
||||||
|
];
|
||||||
|
for source in sources {
|
||||||
|
for forbidden in [
|
||||||
|
"ksp_config_lib::",
|
||||||
|
"ksp_interface_lib::",
|
||||||
|
"ksp_logging_lib::",
|
||||||
|
"ksp_offchain_transport_lib::",
|
||||||
|
"ksp_onchain_transport_lib::",
|
||||||
|
"ksp_store_api::",
|
||||||
|
"ksp_store_lib::",
|
||||||
|
"ksp_worker",
|
||||||
|
"reqwest::",
|
||||||
|
"serde::",
|
||||||
|
"serde_json::",
|
||||||
|
"solana_",
|
||||||
|
"tauri::",
|
||||||
|
"tokio::",
|
||||||
|
"tonic::",
|
||||||
|
concat!("tracing", "::"),
|
||||||
|
] {
|
||||||
|
assert!(!source.contains(forbidden), "forbidden Job API dependency path detected: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn manifest_dependency_names(section: &str) -> std::vec::Vec<&str> {
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for line in section.lines() {
|
||||||
|
let content = match line.split('#').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if content.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match content.split('=').next() {
|
||||||
|
std::option::Option::Some(value) => value.trim().trim_end_matches(".workspace"),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if !name.is_empty() {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
return names;
|
||||||
|
}
|
||||||
148
crates/ksp-job-api/tests/notifications.rs
Normal file
148
crates/ksp-job-api/tests/notifications.rs
Normal file
@@ -0,0 +1,148 @@
|
|||||||
|
// file: crates/ksp-job-api/tests/notifications.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! External-consumer canaries for latest-value Job observation.
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
struct TestSnapshot {
|
||||||
|
completed: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone)]
|
||||||
|
struct TestSnapshotSource {
|
||||||
|
current: std::sync::Arc<ksp_job_api::JobNotification<TestSnapshot>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl TestSnapshotSource {
|
||||||
|
fn new(current: ksp_job_api::JobNotification<TestSnapshot>) -> Self {
|
||||||
|
return Self { current: std::sync::Arc::new(current) };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ksp_job_api::JobSnapshotSource for TestSnapshotSource {
|
||||||
|
type Snapshot = TestSnapshot;
|
||||||
|
|
||||||
|
fn current(&self) -> ksp_job_api::JobNotification<Self::Snapshot> {
|
||||||
|
return self.current.as_ref().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wait_for_change(&self, observed: ksp_job_api::JobNotificationSequence) -> ksp_job_api::JobSnapshotFuture<'_, Self::Snapshot> {
|
||||||
|
let current = ksp_job_api::JobSnapshotSource::current(self);
|
||||||
|
assert!(current.sequence().is_after(observed));
|
||||||
|
return std::boxed::Box::pin(std::future::ready(current));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct TestWake;
|
||||||
|
|
||||||
|
impl std::task::Wake for TestWake {
|
||||||
|
fn wake(self: std::sync::Arc<Self>) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn poll_ready<S>(mut future: ksp_job_api::JobSnapshotFuture<'_, S>) -> std::option::Option<ksp_job_api::JobNotification<S>> {
|
||||||
|
let waker = std::task::Waker::from(std::sync::Arc::new(TestWake));
|
||||||
|
let mut context = std::task::Context::from_waker(&waker);
|
||||||
|
return match std::future::Future::poll(future.as_mut(), &mut context) {
|
||||||
|
std::task::Poll::Ready(value) => std::option::Option::Some(value),
|
||||||
|
std::task::Poll::Pending => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn notification_sequence(value: u64) -> std::option::Option<ksp_job_api::JobNotificationSequence> {
|
||||||
|
let mut sequence = ksp_job_api::JobNotificationSequence::initial();
|
||||||
|
for _ in 0..value {
|
||||||
|
sequence = match sequence.next() {
|
||||||
|
std::result::Result::Ok(next) => next,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(sequence);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn notification(sequence_value: u64, state: ksp_job_api::JobState, completed: u64) -> std::option::Option<ksp_job_api::JobNotification<TestSnapshot>> {
|
||||||
|
let id = match ksp_job_api::JobId::new("external-job-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let kind = match ksp_job_api::JobKindCode::new("backfill_raw") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let sequence = match notification_sequence(sequence_value) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_job_api::JobNotification::new(id, kind, sequence, state, TestSnapshot { completed }));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_external_notification_contract_is_consumable_from_crate_root() {
|
||||||
|
let notification = notification(3, ksp_job_api::JobState::Running, 2);
|
||||||
|
assert!(notification.is_some());
|
||||||
|
let notification = match notification {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(notification.id().as_str(), "external-job-001");
|
||||||
|
assert_eq!(notification.kind().as_str(), "backfill_raw");
|
||||||
|
assert_eq!(notification.sequence().value(), 3);
|
||||||
|
assert_eq!(notification.state(), ksp_job_api::JobState::Running);
|
||||||
|
assert_eq!(notification.snapshot().completed, 2);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_slow_and_independent_listeners_resynchronize_to_latest_value() {
|
||||||
|
let latest = notification(8, ksp_job_api::JobState::Running, 7);
|
||||||
|
assert!(latest.is_some());
|
||||||
|
let source = match latest {
|
||||||
|
std::option::Option::Some(value) => TestSnapshotSource::new(value),
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let listener_a = source.clone();
|
||||||
|
let listener_b = source.clone();
|
||||||
|
let observed_a = match notification_sequence(2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let observed_b = match notification_sequence(6) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let changed_a = poll_ready(ksp_job_api::JobSnapshotSource::wait_for_change(&listener_a, observed_a));
|
||||||
|
let changed_b = poll_ready(ksp_job_api::JobSnapshotSource::wait_for_change(&listener_b, observed_b));
|
||||||
|
assert!(changed_a.is_some());
|
||||||
|
assert!(changed_b.is_some());
|
||||||
|
let changed_a = match changed_a {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let changed_b = match changed_b {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(changed_a.sequence().value(), 8);
|
||||||
|
assert_eq!(changed_b.sequence().value(), 8);
|
||||||
|
assert_eq!(changed_a.snapshot().completed, 7);
|
||||||
|
assert_eq!(changed_b.snapshot().completed, 7);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_terminal_snapshot_remains_readable_from_shared_source() {
|
||||||
|
let terminal = notification(9, ksp_job_api::JobState::Completed(ksp_job_api::JobCompletion::Partial), 8);
|
||||||
|
assert!(terminal.is_some());
|
||||||
|
let source = match terminal {
|
||||||
|
std::option::Option::Some(value) => TestSnapshotSource::new(value),
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let cloned = source.clone();
|
||||||
|
let current = ksp_job_api::JobSnapshotSource::current(&cloned);
|
||||||
|
assert_eq!(current.sequence().value(), 9);
|
||||||
|
assert_eq!(current.state(), ksp_job_api::JobState::Completed(ksp_job_api::JobCompletion::Partial));
|
||||||
|
assert!(current.state().is_terminal());
|
||||||
|
assert_eq!(current.snapshot().completed, 8);
|
||||||
|
return;
|
||||||
|
}
|
||||||
53
crates/ksp-job-api/tests/public_api.rs
Normal file
53
crates/ksp-job-api/tests/public_api.rs
Normal file
@@ -0,0 +1,53 @@
|
|||||||
|
// file: crates/ksp-job-api/tests/public_api.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! External-consumer canaries for the public Job API foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_identity_lifecycle_and_cancellation_are_consumable_from_crate_root() {
|
||||||
|
let id = ksp_job_api::JobId::new("campaign-2026.09.01:001");
|
||||||
|
let kind = ksp_job_api::JobKindCode::new("backfill_raw");
|
||||||
|
assert!(id.is_ok());
|
||||||
|
assert!(kind.is_ok());
|
||||||
|
let id = match id {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let kind = match kind {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_job_api::JobLifecycle::new(id.clone(), kind.clone());
|
||||||
|
assert_eq!(lifecycle.id(), &id);
|
||||||
|
assert_eq!(lifecycle.kind(), &kind);
|
||||||
|
assert_eq!(lifecycle.state(), ksp_job_api::JobState::Created);
|
||||||
|
assert!(lifecycle.start().is_ok());
|
||||||
|
assert!(lifecycle.complete(ksp_job_api::JobCompletion::Partial).is_ok());
|
||||||
|
assert_eq!(lifecycle.state().completion(), std::option::Option::Some(ksp_job_api::JobCompletion::Partial));
|
||||||
|
let token = ksp_job_api::JobCancellationToken::new();
|
||||||
|
let clone = token.clone();
|
||||||
|
assert!(token.cancel());
|
||||||
|
assert!(clone.is_cancellation_requested());
|
||||||
|
assert!(!clone.cancel());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_error_codes_are_stable_and_core_owned() {
|
||||||
|
let codes: [ksp_core_lib::ErrorCode; 3] =
|
||||||
|
[ksp_job_api::ERROR_CODE_JOB_ID_INVALID, ksp_job_api::ERROR_CODE_JOB_KIND_INVALID, ksp_job_api::ERROR_CODE_JOB_TRANSITION_INVALID];
|
||||||
|
assert_eq!(codes[0].domain(), "job_api");
|
||||||
|
assert_eq!(codes[0].code(), "job_id_invalid");
|
||||||
|
assert_eq!(codes[1].domain(), "job_api");
|
||||||
|
assert_eq!(codes[1].code(), "job_kind_invalid");
|
||||||
|
assert_eq!(codes[2].domain(), "job_api");
|
||||||
|
assert_eq!(codes[2].code(), "job_transition_invalid");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_public_bounds_are_exact() {
|
||||||
|
assert_eq!(ksp_job_api::MAX_JOB_ID_BYTES, 128);
|
||||||
|
assert_eq!(ksp_job_api::MAX_JOB_KIND_CODE_BYTES, 128);
|
||||||
|
return;
|
||||||
|
}
|
||||||
101
crates/ksp-job-api/tests/release_completeness.rs
Normal file
101
crates/ksp-job-api/tests/release_completeness.rs
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
// file: crates/ksp-job-api/tests/release_completeness.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
//! Completeness canaries for the current Job API foundation.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_crate_root_export_inventory_is_exact() {
|
||||||
|
let crate_root = include_str!("../src/lib.rs");
|
||||||
|
let mut actual = std::vec::Vec::new();
|
||||||
|
for line in crate_root.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with("pub use ") {
|
||||||
|
actual.push(trimmed);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
actual.sort_unstable();
|
||||||
|
let mut expected = std::vec![
|
||||||
|
"pub use self::cancellation::JobCancellationToken;",
|
||||||
|
"pub use self::error::ERROR_CODE_JOB_ID_INVALID;",
|
||||||
|
"pub use self::error::ERROR_CODE_JOB_KIND_INVALID;",
|
||||||
|
"pub use self::error::ERROR_CODE_JOB_NOTIFICATION_SEQUENCE_EXHAUSTED;",
|
||||||
|
"pub use self::error::ERROR_CODE_JOB_TRANSITION_INVALID;",
|
||||||
|
"pub use self::identity::JobId;",
|
||||||
|
"pub use self::identity::JobKindCode;",
|
||||||
|
"pub use self::identity::MAX_JOB_ID_BYTES;",
|
||||||
|
"pub use self::identity::MAX_JOB_KIND_CODE_BYTES;",
|
||||||
|
"pub use self::lifecycle::JobCompletion;",
|
||||||
|
"pub use self::lifecycle::JobLifecycle;",
|
||||||
|
"pub use self::lifecycle::JobState;",
|
||||||
|
"pub use self::notification::JobNotification;",
|
||||||
|
"pub use self::notification::JobNotificationSequence;",
|
||||||
|
"pub use self::notification::JobSnapshotFuture;",
|
||||||
|
"pub use self::notification::JobSnapshotSource;",
|
||||||
|
"pub use ksp_core_lib::Error;",
|
||||||
|
"pub use ksp_core_lib::ErrorCode;",
|
||||||
|
"pub use ksp_core_lib::ErrorContext;",
|
||||||
|
"pub use ksp_core_lib::Result;",
|
||||||
|
];
|
||||||
|
expected.sort_unstable();
|
||||||
|
assert_eq!(actual, expected);
|
||||||
|
assert!(!crate_root.contains("pub mod "));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_production_module_inventory_is_exact() -> std::io::Result<()> {
|
||||||
|
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let entries = match std::fs::read_dir(source_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(names, std::vec!["cancellation.rs", "error.rs", "identity.rs", "lib.rs", "lifecycle.rs", "notification.rs"]);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_surface_opens_only_generic_notifications_without_backfill_or_worker_contracts() {
|
||||||
|
let sources = [
|
||||||
|
include_str!("../src/cancellation.rs"),
|
||||||
|
include_str!("../src/identity.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/lifecycle.rs"),
|
||||||
|
include_str!("../src/notification.rs"),
|
||||||
|
];
|
||||||
|
for source in sources {
|
||||||
|
for forbidden in ["BackfillRequest", "WorkerControl", "WorkerState", "spawn(", "RawTransaction", "provider", "endpoint"] {
|
||||||
|
assert!(!source.contains(forbidden), "future or domain-specific Job contract leaked early: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let notification_source = include_str!("../src/notification.rs");
|
||||||
|
assert!(notification_source.contains("pub struct JobNotification<S>"));
|
||||||
|
assert!(notification_source.contains("pub trait JobSnapshotSource"));
|
||||||
|
assert!(notification_source.contains("std::future::Future"));
|
||||||
|
assert!(!notification_source.contains("tokio::"));
|
||||||
|
let lifecycle_source = include_str!("../src/lifecycle.rs");
|
||||||
|
assert!(lifecycle_source.contains("#[derive(Eq, PartialEq)]\npub struct JobLifecycle"));
|
||||||
|
assert!(!lifecycle_source.contains("#[derive(Clone, Eq, PartialEq)]\npub struct JobLifecycle"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
72
crates/ksp-job-api/tests/security_hardening.rs
Normal file
72
crates/ksp-job-api/tests/security_hardening.rs
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
// file: crates/ksp-job-api/tests/security_hardening.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! Adversarial lifecycle, cancellation and redaction canaries.
|
||||||
|
|
||||||
|
const HOSTILE_MARKER: &str = "JOB-IDENTITY-SECRET-CANARY";
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_job_id_and_lifecycle_debug_redact_caller_identity() {
|
||||||
|
let id = ksp_job_api::JobId::new(HOSTILE_MARKER);
|
||||||
|
let kind = ksp_job_api::JobKindCode::new("backfill_raw");
|
||||||
|
assert!(id.is_ok());
|
||||||
|
assert!(kind.is_ok());
|
||||||
|
let id = match id {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let kind = match kind {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(std::format!("{id:?}"), "JobId(..)");
|
||||||
|
let lifecycle = ksp_job_api::JobLifecycle::new(id, kind);
|
||||||
|
let debug = std::format!("{lifecycle:?}");
|
||||||
|
assert!(debug.contains("JobLifecycle"));
|
||||||
|
assert!(debug.contains("backfill_raw"));
|
||||||
|
assert!(debug.contains("Created"));
|
||||||
|
assert!(!debug.contains(HOSTILE_MARKER));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_terminal_states_are_immutable_under_all_public_mutators() {
|
||||||
|
let id = ksp_job_api::JobId::new("terminal-job");
|
||||||
|
let kind = ksp_job_api::JobKindCode::new("test_job");
|
||||||
|
assert!(id.is_ok());
|
||||||
|
assert!(kind.is_ok());
|
||||||
|
let id = match id {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let kind = match kind {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let mut lifecycle = ksp_job_api::JobLifecycle::new(id, kind);
|
||||||
|
assert!(lifecycle.start().is_ok());
|
||||||
|
assert!(lifecycle.complete(ksp_job_api::JobCompletion::Complete).is_ok());
|
||||||
|
assert!(lifecycle.start().is_err());
|
||||||
|
assert!(lifecycle.mark_cancelling().is_err());
|
||||||
|
assert!(lifecycle.complete(ksp_job_api::JobCompletion::Partial).is_err());
|
||||||
|
assert!(lifecycle.mark_cancelled().is_err());
|
||||||
|
assert!(lifecycle.fail().is_err());
|
||||||
|
assert_eq!(lifecycle.state(), ksp_job_api::JobState::Completed(ksp_job_api::JobCompletion::Complete));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_cancellation_token_is_send_sync_and_cross_thread_visible() {
|
||||||
|
fn require_send_sync<T: std::marker::Send + std::marker::Sync>() {}
|
||||||
|
require_send_sync::<ksp_job_api::JobCancellationToken>();
|
||||||
|
let token = ksp_job_api::JobCancellationToken::new();
|
||||||
|
let worker_token = token.clone();
|
||||||
|
let thread = std::thread::spawn(move || return worker_token.cancel());
|
||||||
|
let request = match thread.join() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert!(request);
|
||||||
|
assert!(token.is_cancellation_requested());
|
||||||
|
return;
|
||||||
|
}
|
||||||
22
crates/ksp-job-api/unit_tests/cancellation.rs
Normal file
22
crates/ksp-job-api/unit_tests/cancellation.rs
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
// file: crates/ksp-job-api/unit_tests/cancellation.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_cancellation_is_shared_and_idempotent() {
|
||||||
|
let token = crate::JobCancellationToken::new();
|
||||||
|
let clone = token.clone();
|
||||||
|
assert!(!token.is_cancellation_requested());
|
||||||
|
assert!(clone.cancel());
|
||||||
|
assert!(token.is_cancellation_requested());
|
||||||
|
assert!(!token.cancel());
|
||||||
|
assert!(!clone.cancel());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_default_token_starts_without_cancellation() {
|
||||||
|
let token = crate::JobCancellationToken::default();
|
||||||
|
assert!(!token.is_cancellation_requested());
|
||||||
|
assert_eq!(std::format!("{token:?}"), "JobCancellationToken { cancellation_requested: false }");
|
||||||
|
return;
|
||||||
|
}
|
||||||
57
crates/ksp-job-api/unit_tests/identity.rs
Normal file
57
crates/ksp-job-api/unit_tests/identity.rs
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
// file: crates/ksp-job-api/unit_tests/identity.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_identity_accepts_safe_codes_at_exact_bounds() {
|
||||||
|
let job_id = crate::JobId::new("a".repeat(crate::MAX_JOB_ID_BYTES));
|
||||||
|
let kind = crate::JobKindCode::new("backfill.raw:solana-1");
|
||||||
|
assert!(job_id.is_ok());
|
||||||
|
assert!(kind.is_ok());
|
||||||
|
let job_id = match job_id {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let kind = match kind {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(job_id.as_str().len(), crate::MAX_JOB_ID_BYTES);
|
||||||
|
assert_eq!(kind.as_str(), "backfill.raw:solana-1");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_identity_rejects_empty_oversized_and_unsafe_values() {
|
||||||
|
for value in [std::string::String::new(), "a".repeat(crate::MAX_JOB_ID_BYTES + 1), "unsafe/value".to_string(), "unicode-é".to_string()] {
|
||||||
|
let rejected = crate::JobId::new(value);
|
||||||
|
assert!(rejected.is_err());
|
||||||
|
let error = match rejected {
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_JOB_ID_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "field");
|
||||||
|
assert_eq!(error.context()[0].value(), "job_id");
|
||||||
|
}
|
||||||
|
for value in [std::string::String::new(), "b".repeat(crate::MAX_JOB_KIND_CODE_BYTES + 1), "backfill raw".to_string()] {
|
||||||
|
let rejected = crate::JobKindCode::new(value);
|
||||||
|
assert!(rejected.is_err());
|
||||||
|
let error = match rejected {
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
std::result::Result::Ok(_) => continue,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_JOB_KIND_INVALID);
|
||||||
|
assert_eq!(error.context()[0].value(), "job_kind");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_private_validator_uses_exact_safe_code_alphabet() {
|
||||||
|
assert!(super::valid_job_code("AZaz09_.:-", 10));
|
||||||
|
for rejected in ["slash/value", "space value", "line\nbreak", "é"] {
|
||||||
|
assert!(!super::valid_job_code(rejected, crate::MAX_JOB_ID_BYTES));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
153
crates/ksp-job-api/unit_tests/lifecycle.rs
Normal file
153
crates/ksp-job-api/unit_tests/lifecycle.rs
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
// file: crates/ksp-job-api/unit_tests/lifecycle.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn new_lifecycle() -> std::option::Option<crate::JobLifecycle> {
|
||||||
|
let id = match crate::JobId::new("job-001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let kind = match crate::JobKindCode::new("backfill_raw") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(crate::JobLifecycle::new(id, kind));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_lifecycle_accepts_every_planned_terminal_path() {
|
||||||
|
for completion in [crate::JobCompletion::Complete, crate::JobCompletion::Partial] {
|
||||||
|
let lifecycle = new_lifecycle();
|
||||||
|
assert!(lifecycle.is_some());
|
||||||
|
let mut lifecycle = match lifecycle {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
assert!(lifecycle.start().is_ok());
|
||||||
|
assert!(lifecycle.complete(completion).is_ok());
|
||||||
|
assert_eq!(lifecycle.state(), crate::JobState::Completed(completion));
|
||||||
|
assert!(lifecycle.state().is_terminal());
|
||||||
|
assert_eq!(lifecycle.state().completion(), std::option::Option::Some(completion));
|
||||||
|
}
|
||||||
|
let lifecycle = new_lifecycle();
|
||||||
|
assert!(lifecycle.is_some());
|
||||||
|
let mut lifecycle = match lifecycle {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(lifecycle.start().is_ok());
|
||||||
|
assert!(lifecycle.fail().is_ok());
|
||||||
|
assert_eq!(lifecycle.state(), crate::JobState::Failed);
|
||||||
|
let lifecycle = new_lifecycle();
|
||||||
|
assert!(lifecycle.is_some());
|
||||||
|
let mut lifecycle = match lifecycle {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert!(lifecycle.mark_cancelled().is_ok());
|
||||||
|
assert_eq!(lifecycle.state(), crate::JobState::Cancelled);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_cancelling_allows_cancel_complete_or_fail() {
|
||||||
|
for terminal in [crate::JobState::Cancelled, crate::JobState::Completed(crate::JobCompletion::Complete), crate::JobState::Failed] {
|
||||||
|
let lifecycle = new_lifecycle();
|
||||||
|
assert!(lifecycle.is_some());
|
||||||
|
let mut lifecycle = match lifecycle {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
assert!(lifecycle.start().is_ok());
|
||||||
|
assert!(lifecycle.mark_cancelling().is_ok());
|
||||||
|
let result = match terminal {
|
||||||
|
crate::JobState::Cancelled => lifecycle.mark_cancelled(),
|
||||||
|
crate::JobState::Completed(completion) => lifecycle.complete(completion),
|
||||||
|
crate::JobState::Failed => lifecycle.fail(),
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert!(result.is_ok());
|
||||||
|
assert_eq!(lifecycle.state(), terminal);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_invalid_transition_preserves_source_state_and_reports_safe_context() {
|
||||||
|
let lifecycle = new_lifecycle();
|
||||||
|
assert!(lifecycle.is_some());
|
||||||
|
let mut lifecycle = match lifecycle {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let rejected = lifecycle.complete(crate::JobCompletion::Complete);
|
||||||
|
assert!(rejected.is_err());
|
||||||
|
assert_eq!(lifecycle.state(), crate::JobState::Created);
|
||||||
|
let error = match rejected {
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_JOB_TRANSITION_INVALID);
|
||||||
|
assert_eq!(error.context().len(), 2);
|
||||||
|
assert_eq!(error.context()[0].key(), "source_state");
|
||||||
|
assert_eq!(error.context()[0].value(), "created");
|
||||||
|
assert_eq!(error.context()[1].key(), "target_state");
|
||||||
|
assert_eq!(error.context()[1].value(), "completed");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_every_terminal_state_rejects_later_mutation() {
|
||||||
|
let terminals = [crate::JobState::Completed(crate::JobCompletion::Complete), crate::JobState::Cancelled, crate::JobState::Failed];
|
||||||
|
for terminal in terminals {
|
||||||
|
let lifecycle = new_lifecycle();
|
||||||
|
assert!(lifecycle.is_some());
|
||||||
|
let mut lifecycle = match lifecycle {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if terminal == crate::JobState::Cancelled {
|
||||||
|
assert!(lifecycle.mark_cancelled().is_ok());
|
||||||
|
} else {
|
||||||
|
assert!(lifecycle.start().is_ok());
|
||||||
|
let terminal_result = match terminal {
|
||||||
|
crate::JobState::Completed(completion) => lifecycle.complete(completion),
|
||||||
|
crate::JobState::Failed => lifecycle.fail(),
|
||||||
|
_ => return,
|
||||||
|
};
|
||||||
|
assert!(terminal_result.is_ok());
|
||||||
|
}
|
||||||
|
let rejected = lifecycle.start();
|
||||||
|
assert!(rejected.is_err());
|
||||||
|
assert_eq!(lifecycle.state(), terminal);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_002_transition_matrix_is_exact() {
|
||||||
|
let states = [
|
||||||
|
crate::JobState::Created,
|
||||||
|
crate::JobState::Running,
|
||||||
|
crate::JobState::Cancelling,
|
||||||
|
crate::JobState::Completed(crate::JobCompletion::Complete),
|
||||||
|
crate::JobState::Cancelled,
|
||||||
|
crate::JobState::Failed,
|
||||||
|
];
|
||||||
|
for source in states {
|
||||||
|
for target in states {
|
||||||
|
let expected = matches!(
|
||||||
|
(source, target),
|
||||||
|
(crate::JobState::Created, crate::JobState::Running)
|
||||||
|
| (crate::JobState::Created, crate::JobState::Cancelled)
|
||||||
|
| (crate::JobState::Running, crate::JobState::Cancelling)
|
||||||
|
| (crate::JobState::Running, crate::JobState::Completed(_))
|
||||||
|
| (crate::JobState::Running, crate::JobState::Failed)
|
||||||
|
| (crate::JobState::Cancelling, crate::JobState::Completed(_))
|
||||||
|
| (crate::JobState::Cancelling, crate::JobState::Cancelled)
|
||||||
|
| (crate::JobState::Cancelling, crate::JobState::Failed)
|
||||||
|
);
|
||||||
|
assert_eq!(super::allowed_transition(source, target), expected, "unexpected transition matrix result for {source:?} -> {target:?}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
81
crates/ksp-job-api/unit_tests/notification.rs
Normal file
81
crates/ksp-job-api/unit_tests/notification.rs
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
// file: crates/ksp-job-api/unit_tests/notification.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
fn sequence(value: u64) -> crate::JobNotificationSequence {
|
||||||
|
let mut sequence = crate::JobNotificationSequence::initial();
|
||||||
|
for _ in 0..value {
|
||||||
|
sequence = match sequence.next() {
|
||||||
|
std::result::Result::Ok(next) => next,
|
||||||
|
std::result::Result::Err(_) => return sequence,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
return sequence;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_notification_sequence_advances_strictly_and_orders_positions() {
|
||||||
|
let mut first = crate::JobNotificationSequence::initial();
|
||||||
|
for _ in 0..41 {
|
||||||
|
let next = first.next();
|
||||||
|
assert!(next.is_ok());
|
||||||
|
first = match next {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
let second = first.next();
|
||||||
|
assert!(second.is_ok());
|
||||||
|
let second = match second {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.value(), 41);
|
||||||
|
assert_eq!(second.value(), 42);
|
||||||
|
assert!(second.is_after(first));
|
||||||
|
assert!(!first.is_after(second));
|
||||||
|
assert!(second > first);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_notification_sequence_exhaustion_is_explicit_and_non_wrapping() {
|
||||||
|
let exhausted = super::exhausted_notification_sequence().next();
|
||||||
|
assert!(exhausted.is_err());
|
||||||
|
let error = match exhausted {
|
||||||
|
std::result::Result::Err(value) => value,
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_JOB_NOTIFICATION_SEQUENCE_EXHAUSTED);
|
||||||
|
assert_eq!(error.context().len(), 1);
|
||||||
|
assert_eq!(error.context()[0].key(), "sequence");
|
||||||
|
assert_eq!(error.context()[0].value(), u64::MAX.to_string());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_003_notification_preserves_complete_value_and_redacts_snapshot_debug() {
|
||||||
|
let id = crate::JobId::new("job-notify-001");
|
||||||
|
let kind = crate::JobKindCode::new("backfill_raw");
|
||||||
|
assert!(id.is_ok());
|
||||||
|
assert!(kind.is_ok());
|
||||||
|
let id = match id {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let kind = match kind {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let notification =
|
||||||
|
crate::JobNotification::new(id.clone(), kind.clone(), sequence(7), crate::JobState::Running, "RAW-PAYLOAD-MUST-NOT-APPEAR-IN-DEBUG".to_string());
|
||||||
|
assert_eq!(notification.id(), &id);
|
||||||
|
assert_eq!(notification.kind(), &kind);
|
||||||
|
assert_eq!(notification.sequence().value(), 7);
|
||||||
|
assert_eq!(notification.state(), crate::JobState::Running);
|
||||||
|
assert_eq!(notification.snapshot(), "RAW-PAYLOAD-MUST-NOT-APPEAR-IN-DEBUG");
|
||||||
|
let debug = std::format!("{notification:?}");
|
||||||
|
assert!(debug.contains("JobNotification"));
|
||||||
|
assert!(debug.contains("<redacted>"));
|
||||||
|
assert!(!debug.contains("RAW-PAYLOAD-MUST-NOT-APPEAR-IN-DEBUG"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
25
crates/ksp-job-backfill-lib/Cargo.toml
Normal file
25
crates/ksp-job-backfill-lib/Cargo.toml
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
# file: crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
# version: 4
|
||||||
|
|
||||||
|
[package]
|
||||||
|
name = "ksp-job-backfill-lib"
|
||||||
|
version.workspace = true
|
||||||
|
edition.workspace = true
|
||||||
|
repository.workspace = true
|
||||||
|
|
||||||
|
[dependencies]
|
||||||
|
futures-util = { workspace = true, features = ["std"] }
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
ksp-job-api = { path = "../ksp-job-api" }
|
||||||
|
ksp-logging-lib = { path = "../ksp-logging-lib" }
|
||||||
|
ksp-onchain-transport-lib = { path = "../ksp-onchain-transport-lib" }
|
||||||
|
ksp-store-lib = { path = "../ksp-store-lib", default-features = false }
|
||||||
|
serde_json.workspace = true
|
||||||
|
sha2.workspace = true
|
||||||
|
tokio = { workspace = true, features = ["macros", "sync"] }
|
||||||
|
|
||||||
|
[dev-dependencies]
|
||||||
|
tokio = { workspace = true, features = ["macros", "rt-multi-thread"] }
|
||||||
|
|
||||||
|
[lints]
|
||||||
|
workspace = true
|
||||||
123
crates/ksp-job-backfill-lib/README.md
Normal file
123
crates/ksp-job-backfill-lib/README.md
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
<!-- file: crates/ksp-job-backfill-lib/README.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# ksp-job-backfill-lib
|
||||||
|
|
||||||
|
`ksp-job-backfill-lib` implémente le premier job historique concret de KSP : un backfill borné de transactions Solana vers la couche RAW durable.
|
||||||
|
|
||||||
|
La crate compose les contrats Job, Transport observé et Store backend-neutral sans posséder leurs politiques internes. Elle couvre l'admission, la découverte historique, l'hydratation `getTransaction`, la conversion RAW v1, la persistance atomique, la concurrence bornée, la frontier contiguë, le checkpoint caller-owned, l'annulation coopérative et les snapshots latest-value.
|
||||||
|
|
||||||
|
## Identité
|
||||||
|
|
||||||
|
L'identité logique d'une transaction reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
(RawNetworkId, RawTransactionSignature)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le rôle HTTP, le provider, l'endpoint et le protocole ne font pas partie de cette identité. Ils décrivent la sélection et/ou la provenance d'acquisition.
|
||||||
|
|
||||||
|
Le fingerprint d'un scope est déterministe sur sa sémantique de découverte et son réseau ; il n'intègre pas `JobId`, provider, endpoint, protocole ou rôle Transport.
|
||||||
|
|
||||||
|
## Scopes
|
||||||
|
|
||||||
|
Quatre scopes sont exposés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
LatestAddress
|
||||||
|
BeforeAddress
|
||||||
|
AfterAddress
|
||||||
|
ExplicitSignatures
|
||||||
|
```
|
||||||
|
|
||||||
|
La requête borne explicitement page size, nombre de pages, nombre de candidats et concurrence d'hydratation. Les signatures explicites sont dédupliquées de façon stable à la première occurrence.
|
||||||
|
|
||||||
|
## Hydratation et RAW v1
|
||||||
|
|
||||||
|
L'hydratation passe uniquement par la voie observée de `ksp-onchain-transport-lib`.
|
||||||
|
|
||||||
|
Un résultat disponible produit en mémoire :
|
||||||
|
|
||||||
|
- un `RawTransaction` canonique ;
|
||||||
|
- une `RawTransactionObservation` liée à la même référence logique ;
|
||||||
|
- une provenance contenant le provider et l'endpoint réellement gagnants ;
|
||||||
|
- un payload JSON canonique `ksp.solana.raw_transaction`, version `1` ;
|
||||||
|
- un digest SHA-256 des bytes canoniques.
|
||||||
|
|
||||||
|
Un `getTransaction = null` devient `BackfillHydrationOutcome::Missing` et ne fabrique ni payload ni provenance.
|
||||||
|
|
||||||
|
## Persistance
|
||||||
|
|
||||||
|
La persistance utilise exclusivement `ksp-store-lib` avec `default-features = false` côté dépendance de crate :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
-> ksp-store-lib
|
||||||
|
-X-> backend imposé par le job
|
||||||
|
```
|
||||||
|
|
||||||
|
Le chemin d'écriture est l'acquisition atomique `RawTransaction + RawTransactionObservation` en mode normal. La crate distingue insertion, déjà présent, tombstone purgé, observation nouvelle/existante, missing et conflit. Un conflit de contenu n'est jamais converti en succès idempotent.
|
||||||
|
|
||||||
|
## Concurrence et checkpoint
|
||||||
|
|
||||||
|
L'exécution maintient au plus `hydration_concurrency` candidats actifs. Les terminaisons hors ordre sont réconciliées par index stable ; le checkpoint n'avance que sur un préfixe **contigu** d'outcomes durablement sûrs.
|
||||||
|
|
||||||
|
`BackfillCheckpoint` est opaque et caller-owned. Il lie :
|
||||||
|
|
||||||
|
```text
|
||||||
|
JobId
|
||||||
|
scope fingerprint
|
||||||
|
completed contiguous prefix
|
||||||
|
private Before resume cursor
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate ne persiste pas elle-même ce checkpoint. Une reprise crash-safe durable nécessite donc que le caller choisisse explicitement où conserver le checkpoint retourné.
|
||||||
|
|
||||||
|
## Runtime et annulation
|
||||||
|
|
||||||
|
`BackfillJobRuntime` est un coordinator single-run. `BackfillJobHandle` peut être cloné pour :
|
||||||
|
|
||||||
|
- demander une annulation coopérative ;
|
||||||
|
- lire l'état de cette demande ;
|
||||||
|
- obtenir une source latest-value indépendante.
|
||||||
|
|
||||||
|
L'annulation arrête l'admission de nouveau travail et peut interrompre certaines attentes pré-Store. Une persistance Store déjà soumise est toujours drainée avant publication terminale.
|
||||||
|
|
||||||
|
Les snapshots exposent uniquement des compteurs, phases, boundary, checkpoint et code d'échec sûrs ; ils ne contiennent ni payload RAW, ni URL/credential Transport.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
Les dépendances normales sont :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-logging-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-store-lib (default-features = false)
|
||||||
|
futures-util
|
||||||
|
serde_json
|
||||||
|
sha2
|
||||||
|
tokio (macros + sync, détail runtime privé)
|
||||||
|
```
|
||||||
|
|
||||||
|
La crate ne dépend pas de Config, `ksp-store-api` directement, `ksp-store-postgres-lib` ni d'un SDK provider.
|
||||||
|
|
||||||
|
## Hors périmètre
|
||||||
|
|
||||||
|
La crate ne possède pas :
|
||||||
|
|
||||||
|
- application desktop ou CLI ;
|
||||||
|
- Worker API / worker live ;
|
||||||
|
- retry, pacing ou sélection d'endpoint Transport ;
|
||||||
|
- backend Store concret ;
|
||||||
|
- decoding Program ou matérialisation CORE/DECODE/SPECIALIZED ;
|
||||||
|
- table dédiée de checkpoint ;
|
||||||
|
- control plane Job générique.
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [`USAGE.md`](USAGE.md) — construction des scopes/requêtes, runtime, snapshots et reprise ;
|
||||||
|
- [`../ksp-job-api/README.md`](../ksp-job-api/README.md) — contrats Job runtime-neutral ;
|
||||||
|
- [`../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md`](../../docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md) — architecture durable acquisition/jobs ;
|
||||||
|
- [`../../docs/architecture/008-DATA_MATERIALIZATION_AND_STORE.md`](../../docs/architecture/008-DATA_MATERIALIZATION_AND_STORE.md) — frontière RAW/Store.
|
||||||
206
crates/ksp-job-backfill-lib/USAGE.md
Normal file
206
crates/ksp-job-backfill-lib/USAGE.md
Normal file
@@ -0,0 +1,206 @@
|
|||||||
|
<!-- file: crates/ksp-job-backfill-lib/USAGE.md -->
|
||||||
|
<!-- version: 2 -->
|
||||||
|
|
||||||
|
# Utilisation de ksp-job-backfill-lib
|
||||||
|
|
||||||
|
Cette page décrit l'utilisation durable de la façade publique de `ksp-job-backfill-lib`. Elle suppose qu'un caller a déjà construit un `HttpTransportPool` et un `Store` compatibles avec le même réseau logique.
|
||||||
|
|
||||||
|
## Construire une signature et un scope
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let anchor = match ksp_job_backfill_lib::BackfillSignature::new(
|
||||||
|
"1111111111111111111111111111111111111111111111111111111111111111",
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
|
||||||
|
let address = ksp_core_lib::Pubkey::new_from_array([7_u8; 32]);
|
||||||
|
let scope = ksp_job_backfill_lib::BackfillScope::before_address(address, anchor);
|
||||||
|
```
|
||||||
|
|
||||||
|
Les autres formes sont :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let latest = ksp_job_backfill_lib::BackfillScope::latest_address(address);
|
||||||
|
let before = ksp_job_backfill_lib::BackfillScope::before_address(address, anchor.clone());
|
||||||
|
let after = ksp_job_backfill_lib::BackfillScope::after_address(address, anchor.clone());
|
||||||
|
let explicit = ksp_job_backfill_lib::BackfillScope::explicit_signatures(vec![anchor]);
|
||||||
|
```
|
||||||
|
|
||||||
|
`ExplicitSignatures` déduplique la liste en conservant la première occurrence. Les scopes address utilisent `getSignaturesForAddress` ; le scope explicite n'effectue aucune découverte address.
|
||||||
|
|
||||||
|
## Construire une requête bornée
|
||||||
|
|
||||||
|
```rust
|
||||||
|
fn request(scope: ksp_job_backfill_lib::BackfillScope) -> ksp_core_lib::Result<ksp_job_backfill_lib::BackfillRequest> {
|
||||||
|
let job_id = match ksp_job_api::JobId::new("raw-backfill-0001") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("mainnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let role = ksp_onchain_transport_lib::HttpRoleName::new("historical");
|
||||||
|
|
||||||
|
return ksp_job_backfill_lib::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
role,
|
||||||
|
ksp_job_backfill_lib::BackfillCommitment::Finalized,
|
||||||
|
scope,
|
||||||
|
500,
|
||||||
|
20,
|
||||||
|
5_000,
|
||||||
|
16,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Bornes publiques :
|
||||||
|
|
||||||
|
```text
|
||||||
|
page_size 1 ..= 1_000
|
||||||
|
max_pages 1 ..= 10_000
|
||||||
|
max_candidates 1 ..= 10_000
|
||||||
|
hydration_concurrency 1 ..= 64
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `min_context_slot` n'est pas admis pour `ExplicitSignatures`.
|
||||||
|
|
||||||
|
## Comprendre le fingerprint
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let fingerprint = request.scope_fingerprint();
|
||||||
|
let bytes: &[u8; 32] = fingerprint.as_bytes();
|
||||||
|
```
|
||||||
|
|
||||||
|
Le fingerprint représente le scope sémantique et le réseau. Il ne change pas uniquement parce que le rôle Transport, le provider ou l'endpoint d'acquisition change.
|
||||||
|
|
||||||
|
Ne pas utiliser le fingerprint comme identité de transaction : l'identité durable reste `(network, signature)`.
|
||||||
|
|
||||||
|
## Exécuter le runtime complet
|
||||||
|
|
||||||
|
```rust
|
||||||
|
async fn run_backfill(
|
||||||
|
request: ksp_job_backfill_lib::BackfillRequest,
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
store: &ksp_store_lib::Store,
|
||||||
|
) -> ksp_core_lib::Result<ksp_job_backfill_lib::BackfillJobSnapshot> {
|
||||||
|
let runtime = match ksp_job_backfill_lib::BackfillJobRuntime::new(request) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return runtime.run(transport, store).await;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le Store et la requête doivent cibler le même `RawNetworkId`. Un mismatch est rejeté avant l'écriture.
|
||||||
|
|
||||||
|
## Observer la progression
|
||||||
|
|
||||||
|
Obtenir le handle avant de déplacer le runtime dans `run` :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let runtime = ksp_job_backfill_lib::BackfillJobRuntime::new(request)?;
|
||||||
|
let handle = runtime.handle();
|
||||||
|
let snapshots = handle.snapshots();
|
||||||
|
|
||||||
|
let current = ksp_job_api::JobSnapshotSource::current(&snapshots);
|
||||||
|
let observed = current.sequence();
|
||||||
|
let newer = ksp_job_api::JobSnapshotSource::wait_for_change(&snapshots, observed).await;
|
||||||
|
|
||||||
|
assert!(newer.sequence().is_after(observed));
|
||||||
|
```
|
||||||
|
|
||||||
|
Chaque listener peut cloner sa propre `BackfillSnapshotSource`. La source est latest-value : plusieurs mises à jour intermédiaires peuvent être coalescées, mais la valeur retournée est toujours un snapshot complet.
|
||||||
|
|
||||||
|
Les compteurs publics du snapshot couvrent notamment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
candidates_selected / admitted / finished
|
||||||
|
entities_inserted / existing / purged
|
||||||
|
observations_inserted / existing
|
||||||
|
missing / conflicts / holes
|
||||||
|
maximum_in_flight
|
||||||
|
contiguous_completed
|
||||||
|
checkpoint
|
||||||
|
failure_code
|
||||||
|
```
|
||||||
|
|
||||||
|
## Demander une annulation
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let accepted = handle.cancel();
|
||||||
|
if accepted {
|
||||||
|
assert!(handle.is_cancellation_requested());
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
L'annulation est coopérative. Elle peut stopper de nouvelles admissions et certaines attentes avant persistance. Une écriture Store déjà soumise est drainée ; le caller ne doit donc pas supposer qu'une demande d'annulation rend immédiatement toutes les opérations in-flight inexistantes.
|
||||||
|
|
||||||
|
Une demande faite après publication terminale est rejetée (`false`).
|
||||||
|
|
||||||
|
## Reprendre avec un checkpoint
|
||||||
|
|
||||||
|
Le snapshot ou le batch d'exécution peut fournir un `BackfillCheckpoint` sûr. Pour une reprise contrôlée :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
let resumed = match request.with_checkpoint(checkpoint) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Le checkpoint doit appartenir au même `JobId` et au même fingerprint de scope.
|
||||||
|
|
||||||
|
Sémantique de reprise :
|
||||||
|
|
||||||
|
```text
|
||||||
|
LatestAddress repart du latest courant ; frontier contiguë conservée dans le nouveau run
|
||||||
|
BeforeAddress reprend avec le dernier cursor before prouvé
|
||||||
|
AfterAddress rejoue le scope et saute seulement le préfixe contigu prouvé
|
||||||
|
ExplicitSignatures rejoue la liste et saute seulement le préfixe contigu prouvé
|
||||||
|
```
|
||||||
|
|
||||||
|
`BackfillCheckpoint` n'est pas persisté automatiquement. Si le caller exige une reprise après crash/process restart, il doit stocker ce checkpoint dans une surface durable appropriée puis le réinjecter explicitement.
|
||||||
|
|
||||||
|
## Utiliser les primitives séparément
|
||||||
|
|
||||||
|
La façade expose aussi les étapes pour des compositions/tests spécialisés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
discover_backfill_candidates
|
||||||
|
hydrate_backfill_candidate
|
||||||
|
persist_backfill_hydration
|
||||||
|
execute_backfill_discovery
|
||||||
|
```
|
||||||
|
|
||||||
|
`hydrate_backfill_candidate` ne persiste rien. `persist_backfill_hydration` n'effectue aucun appel Transport. `execute_backfill_discovery` combine hydratation/persistance sur un résultat de découverte déjà validé.
|
||||||
|
|
||||||
|
Pour un flux applicatif normal qui veut lifecycle + snapshots + annulation, préférer `BackfillJobRuntime::run`.
|
||||||
|
|
||||||
|
## Interpréter les outcomes de persistance
|
||||||
|
|
||||||
|
Les outcomes distinguent explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
entity: Inserted | AlreadyPresent | SkippedPurged | Conflict
|
||||||
|
observation: Inserted | AlreadyPresent | NotRecorded | NotApplicable
|
||||||
|
```
|
||||||
|
|
||||||
|
`Missing` ne provoque aucune écriture Store. Un conflit de contenu reste observable comme conflit et ne doit pas être traité comme une relance idempotente réussie.
|
||||||
|
|
||||||
|
## Frontières à respecter
|
||||||
|
|
||||||
|
Le caller ne doit pas :
|
||||||
|
|
||||||
|
- pré-lire le Store pour décider s'il faut hydrater une transaction ;
|
||||||
|
- appeler directement `ksp-store-postgres-lib` depuis le job ;
|
||||||
|
- ajouter une politique de retry/pacing qui concurrence Transport ;
|
||||||
|
- utiliser provider/endpoint comme identité transactionnelle ;
|
||||||
|
- inventer une provenance pour `getTransaction = null` ;
|
||||||
|
- interpréter un checkpoint caller-owned comme une garantie de persistence crash-safe automatique ;
|
||||||
|
- utiliser le job RAW comme decoder Program ou processor CORE.
|
||||||
218
crates/ksp-job-backfill-lib/src/checkpoint.rs
Normal file
218
crates/ksp-job-backfill-lib/src/checkpoint.rs
Normal file
@@ -0,0 +1,218 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/checkpoint.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
/// Opaque caller-owned checkpoint for one controlled Backfill resumption.
|
||||||
|
///
|
||||||
|
/// The checkpoint carries no payload, endpoint, provider, URL or secret. Persistence of this
|
||||||
|
/// value is deliberately external to Store in v0.3.6; the Backfill library only validates and
|
||||||
|
/// consumes checkpoints supplied back by its caller.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct BackfillCheckpoint {
|
||||||
|
job_id: ksp_job_api::JobId,
|
||||||
|
scope_fingerprint: crate::BackfillScopeFingerprint,
|
||||||
|
completed_prefix: usize,
|
||||||
|
resume_before: std::option::Option<crate::BackfillSignature>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillCheckpoint {
|
||||||
|
/// Returns the logical Job identity that owns this checkpoint.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn job_id(&self) -> &ksp_job_api::JobId {
|
||||||
|
return &self.job_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the semantic scope fingerprint bound to this checkpoint.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn scope_fingerprint(&self) -> crate::BackfillScopeFingerprint {
|
||||||
|
return self.scope_fingerprint;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of candidates proven durable in one contiguous prefix.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn completed_prefix(&self) -> usize {
|
||||||
|
return self.completed_prefix;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates one internally proven checkpoint.
|
||||||
|
pub(crate) fn new(
|
||||||
|
job_id: ksp_job_api::JobId,
|
||||||
|
scope_fingerprint: crate::BackfillScopeFingerprint,
|
||||||
|
completed_prefix: usize,
|
||||||
|
resume_before: std::option::Option<crate::BackfillSignature>,
|
||||||
|
) -> Self {
|
||||||
|
return Self { job_id, scope_fingerprint, completed_prefix, resume_before };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the internal exclusive `before` cursor used only by controlled Before resumption.
|
||||||
|
pub(crate) const fn resume_before(&self) -> std::option::Option<&crate::BackfillSignature> {
|
||||||
|
return self.resume_before.as_ref();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillCheckpoint {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("BackfillCheckpoint")
|
||||||
|
.field("job_id", &self.job_id)
|
||||||
|
.field("scope_fingerprint", &self.scope_fingerprint)
|
||||||
|
.field("completed_prefix", &self.completed_prefix)
|
||||||
|
.field("has_resume_before", &self.resume_before.is_some())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Private bounded bitmap tracking durable completions and their contiguous prefix.
|
||||||
|
pub(crate) struct CompletionFrontier {
|
||||||
|
durable: std::vec::Vec<bool>,
|
||||||
|
contiguous_completed: usize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::CompletionFrontier {
|
||||||
|
/// Creates one empty frontier for the exact bounded candidate count.
|
||||||
|
pub(crate) fn new(candidate_count: usize) -> Self {
|
||||||
|
return Self { durable: vec![false; candidate_count], contiguous_completed: 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Seeds a previously proven replay prefix before processing the remaining candidates.
|
||||||
|
pub(crate) fn seed_prefix(&mut self, completed_prefix: usize) -> ksp_core_lib::Result<()> {
|
||||||
|
if completed_prefix > self.durable.len() {
|
||||||
|
return std::result::Result::Err(checkpoint_error("checkpoint.completed_prefix"));
|
||||||
|
}
|
||||||
|
for index in 0..completed_prefix {
|
||||||
|
self.durable[index] = true;
|
||||||
|
}
|
||||||
|
self.contiguous_completed = completed_prefix;
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Marks one candidate durable and advances only through the now-contiguous prefix.
|
||||||
|
pub(crate) fn mark_durable(&mut self, index: usize) -> ksp_core_lib::Result<()> {
|
||||||
|
let slot = match self.durable.get_mut(index) {
|
||||||
|
std::option::Option::Some(slot) => slot,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(checkpoint_error("frontier.index")),
|
||||||
|
};
|
||||||
|
*slot = true;
|
||||||
|
while self.contiguous_completed < self.durable.len() && self.durable[self.contiguous_completed] {
|
||||||
|
self.contiguous_completed += 1;
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable candidates in the contiguous prefix.
|
||||||
|
pub(crate) const fn contiguous_completed(&self) -> usize {
|
||||||
|
return self.contiguous_completed;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Validates one checkpoint against the Job, semantic scope and scope-specific cursor shape.
|
||||||
|
pub(crate) fn validate_request_checkpoint(
|
||||||
|
request_job_id: &ksp_job_api::JobId,
|
||||||
|
scope_fingerprint: crate::BackfillScopeFingerprint,
|
||||||
|
scope_kind: crate::BackfillScopeKind,
|
||||||
|
checkpoint: &crate::BackfillCheckpoint,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
if checkpoint.job_id() != request_job_id {
|
||||||
|
return std::result::Result::Err(checkpoint_error("checkpoint.job_id"));
|
||||||
|
}
|
||||||
|
if checkpoint.scope_fingerprint() != scope_fingerprint {
|
||||||
|
return std::result::Result::Err(checkpoint_error("checkpoint.scope_fingerprint"));
|
||||||
|
}
|
||||||
|
if scope_kind != crate::BackfillScopeKind::BeforeAddress && checkpoint.resume_before().is_some() {
|
||||||
|
return std::result::Result::Err(checkpoint_error("checkpoint.resume_before"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resolves the exclusive Before cursor from a validated checkpoint or the original scope anchor.
|
||||||
|
pub(crate) fn resume_before_cursor(request: &crate::BackfillRequest) -> std::option::Option<std::string::String> {
|
||||||
|
if request.scope().kind() != crate::BackfillScopeKind::BeforeAddress {
|
||||||
|
return std::option::Option::None;
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(checkpoint) = request.checkpoint()
|
||||||
|
&& let std::option::Option::Some(cursor) = checkpoint.resume_before()
|
||||||
|
{
|
||||||
|
return std::option::Option::Some(cursor.as_str().to_owned());
|
||||||
|
}
|
||||||
|
return request.scope().anchor().map(|anchor| return anchor.as_str().to_owned());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Resolves the replay prefix skipped only by After and Explicit execution.
|
||||||
|
pub(crate) fn execution_resume_offset(request: &crate::BackfillRequest, discovery: &crate::BackfillDiscovery) -> ksp_core_lib::Result<usize> {
|
||||||
|
let checkpoint = match request.checkpoint() {
|
||||||
|
std::option::Option::Some(checkpoint) => checkpoint,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(0),
|
||||||
|
};
|
||||||
|
return match request.scope().kind() {
|
||||||
|
crate::BackfillScopeKind::AfterAddress | crate::BackfillScopeKind::ExplicitSignatures => {
|
||||||
|
if checkpoint.completed_prefix() > discovery.candidates().len() {
|
||||||
|
return std::result::Result::Err(checkpoint_error("checkpoint.completed_prefix"));
|
||||||
|
}
|
||||||
|
std::result::Result::Ok(checkpoint.completed_prefix())
|
||||||
|
},
|
||||||
|
crate::BackfillScopeKind::LatestAddress | crate::BackfillScopeKind::BeforeAddress => std::result::Result::Ok(0),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds the safe next checkpoint from one drained contiguous completion frontier.
|
||||||
|
pub(crate) fn checkpoint_from_frontier(
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
discovery: &crate::BackfillDiscovery,
|
||||||
|
frontier: &crate::CompletionFrontier,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillCheckpoint> {
|
||||||
|
let validation = crate::validate_discovery_identity(request, discovery);
|
||||||
|
if let std::result::Result::Err(error) = validation {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let local_prefix = frontier.contiguous_completed();
|
||||||
|
if request.scope().kind() == crate::BackfillScopeKind::AfterAddress && discovery.boundary() == crate::BackfillDiscoveryBoundary::AfterAnchorNotReached {
|
||||||
|
if let std::option::Option::Some(checkpoint) = request.checkpoint() {
|
||||||
|
return std::result::Result::Ok(checkpoint.clone());
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(crate::BackfillCheckpoint::new(request.job_id().clone(), request.scope_fingerprint(), 0, std::option::Option::None));
|
||||||
|
}
|
||||||
|
let (completed_prefix, resume_before) = match request.scope().kind() {
|
||||||
|
crate::BackfillScopeKind::LatestAddress => (local_prefix, std::option::Option::None),
|
||||||
|
crate::BackfillScopeKind::BeforeAddress => {
|
||||||
|
let previous = request.checkpoint().map_or(0, crate::BackfillCheckpoint::completed_prefix);
|
||||||
|
let completed_prefix = match previous.checked_add(local_prefix) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(checkpoint_error("checkpoint.completed_prefix")),
|
||||||
|
};
|
||||||
|
let resume_before = if local_prefix == 0 {
|
||||||
|
match request.checkpoint() {
|
||||||
|
std::option::Option::Some(checkpoint) => checkpoint.resume_before().cloned(),
|
||||||
|
std::option::Option::None => request.scope().anchor().cloned(),
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
let index = local_prefix - 1;
|
||||||
|
let candidate = match discovery.candidates().get(index) {
|
||||||
|
std::option::Option::Some(candidate) => candidate,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(checkpoint_error("frontier.index")),
|
||||||
|
};
|
||||||
|
std::option::Option::Some(candidate.identity().signature().clone())
|
||||||
|
};
|
||||||
|
(completed_prefix, resume_before)
|
||||||
|
},
|
||||||
|
crate::BackfillScopeKind::AfterAddress | crate::BackfillScopeKind::ExplicitSignatures => (local_prefix, std::option::Option::None),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(crate::BackfillCheckpoint::new(request.job_id().clone(), request.scope_fingerprint(), completed_prefix, resume_before));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Validates that one discovery result belongs to the request network and semantic scope.
|
||||||
|
pub(crate) fn validate_discovery_identity(request: &crate::BackfillRequest, discovery: &crate::BackfillDiscovery) -> ksp_core_lib::Result<()> {
|
||||||
|
if discovery.network() != request.network() {
|
||||||
|
return std::result::Result::Err(checkpoint_error("discovery.network"));
|
||||||
|
}
|
||||||
|
if discovery.scope_fingerprint() != request.scope_fingerprint() {
|
||||||
|
return std::result::Result::Err(checkpoint_error("discovery.scope_fingerprint"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checkpoint_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_CHECKPOINT_INVALID, "invalid Backfill checkpoint/frontier state").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/checkpoint.rs"]
|
||||||
|
mod tests;
|
||||||
7
crates/ksp-job-backfill-lib/src/constants.rs
Normal file
7
crates/ksp-job-backfill-lib/src/constants.rs
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/constants.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
//! Backfill runtime-owned constants.
|
||||||
|
|
||||||
|
/// Owning tracing target for the concrete bounded RAW backfill runtime.
|
||||||
|
pub(crate) const TRACING_TARGET: &str = "ksp-job-backfill-lib";
|
||||||
485
crates/ksp-job-backfill-lib/src/conversion.rs
Normal file
485
crates/ksp-job-backfill-lib/src/conversion.rs
Normal file
@@ -0,0 +1,485 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/conversion.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
|
/// KSP-owned source-independent RAW transaction format identifier produced by this Backfill vertical.
|
||||||
|
pub const RAW_TRANSACTION_FORMAT_ID: &str = "ksp.solana.raw_transaction";
|
||||||
|
/// Initial KSP-owned RAW transaction format version produced by this Backfill vertical.
|
||||||
|
pub const RAW_TRANSACTION_FORMAT_VERSION: u32 = 1;
|
||||||
|
|
||||||
|
const RAW_TRANSACTION_METHOD_CODE: &str = "getTransaction";
|
||||||
|
const RAW_TRANSACTION_OBSERVATION_CONTRACT_VERSION: u32 = 1;
|
||||||
|
const RAW_TRANSACTION_PROTOCOL_CODE: &str = "solana.http.json_rpc";
|
||||||
|
|
||||||
|
/// Complete in-memory RAW transaction acquisition ready for the later Store persistence tranche.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub struct BackfillRawAcquisition {
|
||||||
|
inner: Box<BackfillRawAcquisitionInner>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug)]
|
||||||
|
struct BackfillRawAcquisitionInner {
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillRawAcquisition {
|
||||||
|
/// Returns the canonical RAW transaction produced from the typed Transport response.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn transaction(&self) -> &ksp_store_lib::RawTransaction {
|
||||||
|
return &self.inner.transaction;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the acquisition observation whose provenance records the actual successful endpoint.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation(&self) -> &ksp_store_lib::RawTransactionObservation {
|
||||||
|
return &self.inner.observation;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Consumes the in-memory acquisition into the canonical transaction and its observation.
|
||||||
|
#[must_use]
|
||||||
|
pub fn into_parts(self) -> (ksp_store_lib::RawTransaction, ksp_store_lib::RawTransactionObservation) {
|
||||||
|
let inner = *self.inner;
|
||||||
|
return (inner.transaction, inner.observation);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Result of hydrating one deterministic Backfill candidate through observed `getTransaction`.
|
||||||
|
#[derive(Debug)]
|
||||||
|
pub enum BackfillHydrationOutcome {
|
||||||
|
/// The RPC returned one complete transaction and conversion produced canonical RAW plus provenance.
|
||||||
|
Available(crate::BackfillRawAcquisition),
|
||||||
|
/// The RPC returned JSON `null`; only the canonical transaction identity exists and no provenance is fabricated.
|
||||||
|
Missing(ksp_store_lib::RawTransactionReference),
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillHydrationOutcome {
|
||||||
|
/// Returns the network-scoped transaction identity represented by this hydration outcome.
|
||||||
|
#[must_use]
|
||||||
|
pub fn reference(&self) -> &ksp_store_lib::RawTransactionReference {
|
||||||
|
return match self {
|
||||||
|
Self::Available(acquisition) => acquisition.transaction().reference(),
|
||||||
|
Self::Missing(reference) => reference,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether `getTransaction` returned JSON `null`.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn is_missing(&self) -> bool {
|
||||||
|
return matches!(self, Self::Missing(_));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Hydrates one candidate with the typed observed Transport path and converts it to canonical RAW v1.
|
||||||
|
///
|
||||||
|
/// The caller supplies the local receipt timestamp because wall-clock ownership remains outside this
|
||||||
|
/// pure conversion tranche. Transport retains endpoint selection and retry. This function never
|
||||||
|
/// persists to Store; persistence begins in `pre.007`.
|
||||||
|
pub async fn hydrate_backfill_candidate(
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
candidate: &crate::BackfillCandidate,
|
||||||
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillHydrationOutcome> {
|
||||||
|
let reference = canonical_reference(request, candidate);
|
||||||
|
let reference = match reference {
|
||||||
|
std::result::Result::Ok(reference) => reference,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let config = ksp_onchain_transport_lib::SolanaGetTransactionConfig::new(
|
||||||
|
std::option::Option::Some(request.commitment().transport()),
|
||||||
|
std::option::Option::Some(ksp_onchain_transport_lib::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
);
|
||||||
|
let observed = transport.get_transaction_observed(request.role(), candidate.identity().signature().as_str(), std::option::Option::Some(&config)).await;
|
||||||
|
let observed = match observed {
|
||||||
|
std::result::Result::Ok(observed) => observed,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let provider = observed.provider().as_str().to_owned();
|
||||||
|
let endpoint = observed.endpoint_name().to_owned();
|
||||||
|
let transaction = observed.into_value();
|
||||||
|
let transaction = match transaction {
|
||||||
|
std::option::Option::Some(transaction) => transaction,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(crate::BackfillHydrationOutcome::Missing(reference)),
|
||||||
|
};
|
||||||
|
let fields = CanonicalTransactionFields {
|
||||||
|
slot: transaction.slot(),
|
||||||
|
block_time: transaction.block_time(),
|
||||||
|
transaction: transaction.transaction(),
|
||||||
|
meta: transaction.meta(),
|
||||||
|
version: transaction.version(),
|
||||||
|
transaction_index: transaction.transaction_index(),
|
||||||
|
};
|
||||||
|
let acquisition = convert_available_fields(request, reference, fields, provider.as_str(), endpoint.as_str(), received_at);
|
||||||
|
return match acquisition {
|
||||||
|
std::result::Result::Ok(acquisition) => std::result::Result::Ok(crate::BackfillHydrationOutcome::Available(acquisition)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Decodes one validated Base58 signature to exactly 64 canonical bytes without a Solana SDK dependency.
|
||||||
|
pub(crate) fn decode_backfill_signature(signature: &crate::BackfillSignature) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionSignature> {
|
||||||
|
let text = signature.as_str().as_bytes();
|
||||||
|
let mut decoded = [0_u8; 64];
|
||||||
|
let mut leading_zeroes = 0_usize;
|
||||||
|
for byte in text {
|
||||||
|
if *byte != b'1' {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
leading_zeroes += 1;
|
||||||
|
}
|
||||||
|
for byte in text {
|
||||||
|
let digit = match base58_digit(*byte) {
|
||||||
|
std::option::Option::Some(digit) => digit,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(conversion_error("signature")),
|
||||||
|
};
|
||||||
|
let mut carry = u32::from(digit);
|
||||||
|
for output in decoded.iter_mut().rev() {
|
||||||
|
let value = (u32::from(*output) * 58) + carry;
|
||||||
|
*output = (value & 0xff) as u8;
|
||||||
|
carry = value >> 8;
|
||||||
|
}
|
||||||
|
if carry != 0 {
|
||||||
|
return std::result::Result::Err(conversion_error("signature"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let significant_len = match decoded.iter().position(|byte| return *byte != 0) {
|
||||||
|
std::option::Option::Some(index) => decoded.len() - index,
|
||||||
|
std::option::Option::None => 0,
|
||||||
|
};
|
||||||
|
if leading_zeroes + significant_len != decoded.len() {
|
||||||
|
return std::result::Result::Err(conversion_error("signature"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawTransactionSignature::new(decoded));
|
||||||
|
}
|
||||||
|
|
||||||
|
struct CanonicalTransactionFields<'a> {
|
||||||
|
slot: u64,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
transaction: &'a ksp_onchain_transport_lib::SolanaEncodedTransaction,
|
||||||
|
meta: &'a ksp_onchain_transport_lib::SolanaWireField<serde_json::Value>,
|
||||||
|
version: &'a ksp_onchain_transport_lib::SolanaWireField<ksp_onchain_transport_lib::SolanaTransactionVersion>,
|
||||||
|
transaction_index: &'a ksp_onchain_transport_lib::SolanaWireField<u32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonical_reference(request: &crate::BackfillRequest, candidate: &crate::BackfillCandidate) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionReference> {
|
||||||
|
if candidate.identity().network() != request.network() {
|
||||||
|
return std::result::Result::Err(conversion_error("candidate.network"));
|
||||||
|
}
|
||||||
|
let signature = crate::decode_backfill_signature(candidate.identity().signature());
|
||||||
|
let signature = match signature {
|
||||||
|
std::result::Result::Ok(signature) => signature,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(ksp_store_lib::RawTransactionReference::new(candidate.identity().network().clone(), signature));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn convert_available_fields(
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
reference: ksp_store_lib::RawTransactionReference,
|
||||||
|
fields: CanonicalTransactionFields<'_>,
|
||||||
|
provider: &str,
|
||||||
|
endpoint: &str,
|
||||||
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillRawAcquisition> {
|
||||||
|
let block_time = convert_block_time(fields.block_time);
|
||||||
|
let block_time = match block_time {
|
||||||
|
std::result::Result::Ok(block_time) => block_time,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let bytes = canonical_payload_bytes(&fields);
|
||||||
|
let bytes = match bytes {
|
||||||
|
std::result::Result::Ok(bytes) => bytes,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let hash: [u8; 32] = sha2::Sha256::digest(bytes.as_slice()).into();
|
||||||
|
let format_id = ksp_store_lib::RawFormatId::new(crate::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
let format_id = match format_id {
|
||||||
|
std::result::Result::Ok(format_id) => format_id,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(conversion_error("payload.format_id")),
|
||||||
|
};
|
||||||
|
let payload = ksp_store_lib::RawPayload::try_new(
|
||||||
|
format_id,
|
||||||
|
crate::RAW_TRANSACTION_FORMAT_VERSION,
|
||||||
|
bytes.into_boxed_slice(),
|
||||||
|
ksp_store_lib::RawContentHash::new(hash),
|
||||||
|
);
|
||||||
|
let payload = match payload {
|
||||||
|
std::result::Result::Ok(payload) => payload,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let provenance = build_provenance(request, provider, endpoint, received_at);
|
||||||
|
let provenance = match provenance {
|
||||||
|
std::result::Result::Ok(provenance) => provenance,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let observation_key = observation_key(request, &reference, provider, endpoint);
|
||||||
|
let transaction = ksp_store_lib::RawTransaction::new(reference.clone(), fields.slot, block_time, payload);
|
||||||
|
let observation = ksp_store_lib::RawTransactionObservation::new(observation_key, reference, provenance);
|
||||||
|
return std::result::Result::Ok(crate::BackfillRawAcquisition { inner: Box::new(BackfillRawAcquisitionInner { transaction, observation }) });
|
||||||
|
}
|
||||||
|
|
||||||
|
fn convert_block_time(value: std::option::Option<i64>) -> ksp_core_lib::Result<std::option::Option<ksp_store_lib::RawTimestamp>> {
|
||||||
|
let seconds = match value {
|
||||||
|
std::option::Option::Some(seconds) => seconds,
|
||||||
|
std::option::Option::None => return std::result::Result::Ok(std::option::Option::None),
|
||||||
|
};
|
||||||
|
let seconds = match u64::try_from(seconds) {
|
||||||
|
std::result::Result::Ok(seconds) => seconds,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(conversion_error("block_time")),
|
||||||
|
};
|
||||||
|
let millis = match seconds.checked_mul(1_000) {
|
||||||
|
std::option::Option::Some(millis) => millis,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(conversion_error("block_time")),
|
||||||
|
};
|
||||||
|
let timestamp = ksp_store_lib::RawTimestamp::from_unix_millis(millis);
|
||||||
|
return match timestamp {
|
||||||
|
std::result::Result::Ok(timestamp) => std::result::Result::Ok(std::option::Option::Some(timestamp)),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(conversion_error("block_time")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn canonical_payload_bytes(fields: &CanonicalTransactionFields<'_>) -> ksp_core_lib::Result<std::vec::Vec<u8>> {
|
||||||
|
let (transaction_data, transaction_encoding) = match fields.transaction {
|
||||||
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary { data, encoding } => {
|
||||||
|
if *encoding != ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64 {
|
||||||
|
return std::result::Result::Err(conversion_error("transaction.encoding"));
|
||||||
|
}
|
||||||
|
(data.as_str(), "base64")
|
||||||
|
},
|
||||||
|
ksp_onchain_transport_lib::SolanaEncodedTransaction::LegacyBinary(_) | ksp_onchain_transport_lib::SolanaEncodedTransaction::Json(_) => {
|
||||||
|
return std::result::Result::Err(conversion_error("transaction.encoding"));
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let mut bytes = std::vec::Vec::new();
|
||||||
|
bytes.extend_from_slice(b"{\"transaction\":[");
|
||||||
|
if let std::result::Result::Err(error) = append_json_string(&mut bytes, transaction_data) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
bytes.push(b',');
|
||||||
|
if let std::result::Result::Err(error) = append_json_string(&mut bytes, transaction_encoding) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
bytes.push(b']');
|
||||||
|
if let std::result::Result::Err(error) = append_wire_value(&mut bytes, "meta", fields.meta, |output, value| return append_canonical_json(output, value)) {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let version_result = append_wire_value(&mut bytes, "version", fields.version, |output, value| {
|
||||||
|
return match value {
|
||||||
|
ksp_onchain_transport_lib::SolanaTransactionVersion::Legacy => append_json_string(output, "legacy"),
|
||||||
|
ksp_onchain_transport_lib::SolanaTransactionVersion::Number(number) => {
|
||||||
|
output.extend_from_slice(number.to_string().as_bytes());
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
};
|
||||||
|
});
|
||||||
|
if let std::result::Result::Err(error) = version_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let transaction_index_result = append_wire_value(&mut bytes, "transactionIndex", fields.transaction_index, |output, value| {
|
||||||
|
output.extend_from_slice(value.to_string().as_bytes());
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
});
|
||||||
|
if let std::result::Result::Err(error) = transaction_index_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
bytes.push(b'}');
|
||||||
|
return std::result::Result::Ok(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_wire_value<T, F>(
|
||||||
|
output: &mut std::vec::Vec<u8>,
|
||||||
|
key: &str,
|
||||||
|
field: &ksp_onchain_transport_lib::SolanaWireField<T>,
|
||||||
|
mut append_value: F,
|
||||||
|
) -> ksp_core_lib::Result<()>
|
||||||
|
where
|
||||||
|
F: FnMut(&mut std::vec::Vec<u8>, &T) -> ksp_core_lib::Result<()>,
|
||||||
|
{
|
||||||
|
return match field {
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Omitted => std::result::Result::Ok(()),
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Null => {
|
||||||
|
output.push(b',');
|
||||||
|
let key_result = append_json_string(output, key);
|
||||||
|
if let std::result::Result::Err(error) = key_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.extend_from_slice(b":null");
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
ksp_onchain_transport_lib::SolanaWireField::Value(value) => {
|
||||||
|
output.push(b',');
|
||||||
|
let key_result = append_json_string(output, key);
|
||||||
|
if let std::result::Result::Err(error) = key_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.push(b':');
|
||||||
|
append_value(output, value)
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_canonical_json(output: &mut std::vec::Vec<u8>, value: &serde_json::Value) -> ksp_core_lib::Result<()> {
|
||||||
|
return match value {
|
||||||
|
serde_json::Value::Null => {
|
||||||
|
output.extend_from_slice(b"null");
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::Bool(value) => {
|
||||||
|
if *value {
|
||||||
|
output.extend_from_slice(b"true");
|
||||||
|
} else {
|
||||||
|
output.extend_from_slice(b"false");
|
||||||
|
}
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::Number(value) => {
|
||||||
|
output.extend_from_slice(value.to_string().as_bytes());
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::String(value) => append_json_string(output, value.as_str()),
|
||||||
|
serde_json::Value::Array(values) => {
|
||||||
|
output.push(b'[');
|
||||||
|
for (index, item) in values.iter().enumerate() {
|
||||||
|
if index != 0 {
|
||||||
|
output.push(b',');
|
||||||
|
}
|
||||||
|
let item_result = append_canonical_json(output, item);
|
||||||
|
if let std::result::Result::Err(error) = item_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output.push(b']');
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
serde_json::Value::Object(values) => {
|
||||||
|
output.push(b'{');
|
||||||
|
let mut keys = values.keys().collect::<std::vec::Vec<_>>();
|
||||||
|
keys.sort_unstable();
|
||||||
|
for (index, key) in keys.iter().enumerate() {
|
||||||
|
if index != 0 {
|
||||||
|
output.push(b',');
|
||||||
|
}
|
||||||
|
let key_result = append_json_string(output, key.as_str());
|
||||||
|
if let std::result::Result::Err(error) = key_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
output.push(b':');
|
||||||
|
let item = values.get(key.as_str());
|
||||||
|
let item = match item {
|
||||||
|
std::option::Option::Some(item) => item,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(conversion_error("payload.meta")),
|
||||||
|
};
|
||||||
|
let item_result = append_canonical_json(output, item);
|
||||||
|
if let std::result::Result::Err(error) = item_result {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
output.push(b'}');
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn append_json_string(output: &mut std::vec::Vec<u8>, value: &str) -> ksp_core_lib::Result<()> {
|
||||||
|
let encoded = serde_json::to_vec(value);
|
||||||
|
return match encoded {
|
||||||
|
std::result::Result::Ok(encoded) => {
|
||||||
|
output.extend_from_slice(encoded.as_slice());
|
||||||
|
std::result::Result::Ok(())
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(conversion_error("payload.json")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn build_provenance(
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
provider: &str,
|
||||||
|
endpoint: &str,
|
||||||
|
received_at: ksp_store_lib::RawTimestamp,
|
||||||
|
) -> ksp_core_lib::Result<ksp_store_lib::RawAcquisitionProvenance> {
|
||||||
|
let provider = match provenance_code(provider, "provenance.provider") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let protocol = match provenance_code(RAW_TRANSACTION_PROTOCOL_CODE, "provenance.protocol") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let method = match provenance_code(RAW_TRANSACTION_METHOD_CODE, "provenance.method") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let endpoint = match provenance_code(endpoint, "provenance.endpoint") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let commitment = match provenance_code(request.commitment().code(), "provenance.commitment") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let capture_session = match provenance_code(request.job_id().as_str(), "provenance.capture_session") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Backfill, received_at)
|
||||||
|
.with_endpoint_id(endpoint)
|
||||||
|
.with_commitment(commitment)
|
||||||
|
.with_capture_session_id(capture_session);
|
||||||
|
return std::result::Result::Ok(provenance);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn provenance_code(value: &str, field: &'static str) -> ksp_core_lib::Result<ksp_store_lib::RawProvenanceCode> {
|
||||||
|
let code = ksp_store_lib::RawProvenanceCode::new(value);
|
||||||
|
return match code {
|
||||||
|
std::result::Result::Ok(code) => std::result::Result::Ok(code),
|
||||||
|
std::result::Result::Err(_) => std::result::Result::Err(conversion_error(field)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn observation_key(
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
reference: &ksp_store_lib::RawTransactionReference,
|
||||||
|
provider: &str,
|
||||||
|
endpoint: &str,
|
||||||
|
) -> ksp_store_lib::RawObservationKey {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(b"ksp.job.backfill.raw_transaction_observation.v1\0");
|
||||||
|
hash_bytes(&mut hasher, request.job_id().as_str().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, request.scope_fingerprint().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, reference.signature().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, provider.as_bytes());
|
||||||
|
hash_bytes(&mut hasher, endpoint.as_bytes());
|
||||||
|
hash_bytes(&mut hasher, request.commitment().code().as_bytes());
|
||||||
|
hasher.update(RAW_TRANSACTION_OBSERVATION_CONTRACT_VERSION.to_be_bytes());
|
||||||
|
let bytes: [u8; 32] = hasher.finalize().into();
|
||||||
|
return ksp_store_lib::RawObservationKey::new(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_bytes(hasher: &mut sha2::Sha256, value: &[u8]) {
|
||||||
|
hasher.update((value.len() as u64).to_be_bytes());
|
||||||
|
hasher.update(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn base58_digit(byte: u8) -> std::option::Option<u8> {
|
||||||
|
return match byte {
|
||||||
|
b'1'..=b'9' => std::option::Option::Some(byte - b'1'),
|
||||||
|
b'A'..=b'H' => std::option::Option::Some((byte - b'A') + 9),
|
||||||
|
b'J'..=b'N' => std::option::Option::Some((byte - b'J') + 17),
|
||||||
|
b'P'..=b'Z' => std::option::Option::Some((byte - b'P') + 22),
|
||||||
|
b'a'..=b'k' => std::option::Option::Some((byte - b'a') + 33),
|
||||||
|
b'm'..=b'z' => std::option::Option::Some((byte - b'm') + 44),
|
||||||
|
_ => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn conversion_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID, "invalid deterministic Backfill RAW conversion")
|
||||||
|
.with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/conversion.rs"]
|
||||||
|
mod tests;
|
||||||
492
crates/ksp-job-backfill-lib/src/discovery.rs
Normal file
492
crates/ksp-job-backfill-lib/src/discovery.rs
Normal file
@@ -0,0 +1,492 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/discovery.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
/// Network-scoped identity of one discovered transaction candidate before canonical signature decoding.
|
||||||
|
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub struct BackfillCandidateIdentity {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
signature: crate::BackfillSignature,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillCandidateIdentity {
|
||||||
|
/// Creates one candidate identity from its logical network and encoded transaction signature.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new(network: ksp_store_lib::RawNetworkId, signature: crate::BackfillSignature) -> Self {
|
||||||
|
return Self { network, signature };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the logical network that scopes signature uniqueness.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
|
return &self.network;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated encoded transaction signature.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn signature(&self) -> &crate::BackfillSignature {
|
||||||
|
return &self.signature;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One deterministic transaction candidate produced by bounded discovery.
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
pub struct BackfillCandidate {
|
||||||
|
identity: crate::BackfillCandidateIdentity,
|
||||||
|
discovered_slot: std::option::Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillCandidate {
|
||||||
|
/// Creates one candidate from a network-scoped identity and optional discovery slot.
|
||||||
|
#[must_use]
|
||||||
|
pub fn new(identity: crate::BackfillCandidateIdentity, discovered_slot: std::option::Option<u64>) -> Self {
|
||||||
|
return Self { identity, discovered_slot };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the network-scoped candidate identity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn identity(&self) -> &crate::BackfillCandidateIdentity {
|
||||||
|
return &self.identity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the slot reported by address discovery, when one was available.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn discovered_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.discovered_slot;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reason bounded candidate discovery stopped.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum BackfillDiscoveryBoundary {
|
||||||
|
/// Explicit signatures required no RPC pagination.
|
||||||
|
ExplicitInput,
|
||||||
|
/// The caller's requested candidate limit was satisfied.
|
||||||
|
CandidateLimit,
|
||||||
|
/// The RPC returned a short or empty page, reaching the bounded remote history boundary.
|
||||||
|
RpcBoundary,
|
||||||
|
/// The maximum page count was exhausted before the requested address window completed.
|
||||||
|
PageLimit,
|
||||||
|
/// An `AfterAddress` scan exhausted its page bound before the exclusive anchor boundary was reached.
|
||||||
|
AfterAnchorNotReached,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillDiscoveryBoundary {
|
||||||
|
/// Returns the stable diagnostic code for this boundary.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn code(self) -> &'static str {
|
||||||
|
return match self {
|
||||||
|
Self::ExplicitInput => "explicit_input",
|
||||||
|
Self::CandidateLimit => "candidate_limit",
|
||||||
|
Self::RpcBoundary => "rpc_boundary",
|
||||||
|
Self::PageLimit => "page_limit",
|
||||||
|
Self::AfterAnchorNotReached => "after_anchor_not_reached",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether the discovery result is partial and must not advance a future checkpoint beyond the unresolved gap.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn is_partial(self) -> bool {
|
||||||
|
return matches!(self, Self::PageLimit | Self::AfterAnchorNotReached);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete bounded output of one candidate discovery pass.
|
||||||
|
pub struct BackfillDiscovery {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
scope_fingerprint: crate::BackfillScopeFingerprint,
|
||||||
|
candidates: std::vec::Vec<crate::BackfillCandidate>,
|
||||||
|
pages_fetched: usize,
|
||||||
|
boundary: crate::BackfillDiscoveryBoundary,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillDiscovery {
|
||||||
|
/// Creates one internally validated bounded discovery value.
|
||||||
|
pub(crate) fn new(
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
scope_fingerprint: crate::BackfillScopeFingerprint,
|
||||||
|
candidates: std::vec::Vec<crate::BackfillCandidate>,
|
||||||
|
pages_fetched: usize,
|
||||||
|
boundary: crate::BackfillDiscoveryBoundary,
|
||||||
|
) -> Self {
|
||||||
|
return Self { network, scope_fingerprint, candidates, pages_fetched, boundary };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the logical network shared by every candidate identity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
|
return &self.network;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the semantic scope fingerprint copied from the validated request.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn scope_fingerprint(&self) -> crate::BackfillScopeFingerprint {
|
||||||
|
return self.scope_fingerprint;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns discovered candidates in deterministic processing order.
|
||||||
|
#[must_use]
|
||||||
|
pub fn candidates(&self) -> &[crate::BackfillCandidate] {
|
||||||
|
return self.candidates.as_slice();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of `getSignaturesForAddress` pages fetched by this pass.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn pages_fetched(&self) -> usize {
|
||||||
|
return self.pages_fetched;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the reason discovery stopped.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn boundary(&self) -> crate::BackfillDiscoveryBoundary {
|
||||||
|
return self.boundary;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether the bounded pass stopped before a complete requested discovery window was proven.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn is_partial(&self) -> bool {
|
||||||
|
return self.boundary.is_partial();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillDiscovery {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("BackfillDiscovery")
|
||||||
|
.field("network", &self.network)
|
||||||
|
.field("scope_fingerprint", &self.scope_fingerprint)
|
||||||
|
.field("candidate_count", &self.candidates.len())
|
||||||
|
.field("pages_fetched", &self.pages_fetched)
|
||||||
|
.field("boundary", &self.boundary)
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
struct SignaturePageEntry {
|
||||||
|
signature: std::string::String,
|
||||||
|
slot: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
type SignaturePageFuture<'a> = std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = ksp_core_lib::Result<std::vec::Vec<SignaturePageEntry>>> + 'a>>;
|
||||||
|
|
||||||
|
trait SignaturePageSource {
|
||||||
|
fn fetch_signature_page<'a>(
|
||||||
|
&'a self,
|
||||||
|
role: &'a ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
address: &'a ksp_core_lib::Pubkey,
|
||||||
|
config: ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig,
|
||||||
|
) -> SignaturePageFuture<'a>;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl SignaturePageSource for ksp_onchain_transport_lib::HttpTransportPool {
|
||||||
|
fn fetch_signature_page<'a>(
|
||||||
|
&'a self,
|
||||||
|
role: &'a ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
address: &'a ksp_core_lib::Pubkey,
|
||||||
|
config: ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig,
|
||||||
|
) -> SignaturePageFuture<'a> {
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
let result = self.get_signatures_for_address(role, address, std::option::Option::Some(&config)).await;
|
||||||
|
let infos = match result {
|
||||||
|
std::result::Result::Ok(infos) => infos,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut entries = std::vec::Vec::with_capacity(infos.len());
|
||||||
|
for info in infos {
|
||||||
|
entries.push(SignaturePageEntry { signature: info.signature().to_owned(), slot: info.slot() });
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(entries);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Discovers one bounded deterministic candidate set using only the typed KSP Transport wrapper.
|
||||||
|
///
|
||||||
|
/// Provider, endpoint and protocol selection remain entirely owned by Transport. Candidate identity
|
||||||
|
/// is scoped only by the request network plus transaction signature.
|
||||||
|
pub async fn discover_backfill_candidates(
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillDiscovery> {
|
||||||
|
ksp_logging_lib::debug!(
|
||||||
|
target: crate::TRACING_TARGET,
|
||||||
|
job_id = request.job_id().as_str(),
|
||||||
|
network = request.network().as_str(),
|
||||||
|
scope = request.scope().kind().code(),
|
||||||
|
page_size = request.page_size(),
|
||||||
|
max_pages = request.max_pages(),
|
||||||
|
max_candidates = request.max_candidates(),
|
||||||
|
"starting bounded Backfill candidate discovery"
|
||||||
|
);
|
||||||
|
let result = discover_with_source(transport, request, std::option::Option::None).await;
|
||||||
|
if let std::result::Result::Ok(discovery) = &result {
|
||||||
|
ksp_logging_lib::debug!(
|
||||||
|
target: crate::TRACING_TARGET,
|
||||||
|
job_id = request.job_id().as_str(),
|
||||||
|
network = request.network().as_str(),
|
||||||
|
scope = request.scope().kind().code(),
|
||||||
|
pages_fetched = discovery.pages_fetched(),
|
||||||
|
candidate_count = discovery.candidates().len(),
|
||||||
|
boundary = discovery.boundary().code(),
|
||||||
|
partial = discovery.is_partial(),
|
||||||
|
"completed bounded Backfill candidate discovery"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Discovers one bounded candidate set while allowing cooperative cancellation of in-flight RPC futures.
|
||||||
|
pub(crate) async fn discover_backfill_candidates_cancellable(
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
cancellation: &crate::BackfillCancellationSignal,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillDiscovery> {
|
||||||
|
return discover_with_source(transport, request, std::option::Option::Some(cancellation)).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn discover_with_source<S>(
|
||||||
|
source: &S,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
cancellation: std::option::Option<&crate::BackfillCancellationSignal>,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillDiscovery>
|
||||||
|
where
|
||||||
|
S: SignaturePageSource,
|
||||||
|
{
|
||||||
|
if cancellation.is_some_and(crate::BackfillCancellationSignal::is_requested) {
|
||||||
|
return std::result::Result::Err(cancelled_error());
|
||||||
|
}
|
||||||
|
return match request.scope().kind() {
|
||||||
|
crate::BackfillScopeKind::ExplicitSignatures => discover_explicit(request),
|
||||||
|
crate::BackfillScopeKind::LatestAddress | crate::BackfillScopeKind::BeforeAddress => discover_older(source, request, cancellation).await,
|
||||||
|
crate::BackfillScopeKind::AfterAddress => discover_after(source, request, cancellation).await,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn discover_explicit(request: &crate::BackfillRequest) -> ksp_core_lib::Result<crate::BackfillDiscovery> {
|
||||||
|
let signatures = match request.scope().signatures() {
|
||||||
|
std::option::Option::Some(signatures) => signatures,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(discovery_invalid("scope.signatures")),
|
||||||
|
};
|
||||||
|
let mut candidates = std::vec::Vec::with_capacity(signatures.len());
|
||||||
|
for signature in signatures {
|
||||||
|
let identity = crate::BackfillCandidateIdentity::new(request.network().clone(), signature.clone());
|
||||||
|
candidates.push(crate::BackfillCandidate::new(identity, std::option::Option::None));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(crate::BackfillDiscovery::new(
|
||||||
|
request.network().clone(),
|
||||||
|
request.scope_fingerprint(),
|
||||||
|
candidates,
|
||||||
|
0,
|
||||||
|
crate::BackfillDiscoveryBoundary::ExplicitInput,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn discover_older<S>(
|
||||||
|
source: &S,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
cancellation: std::option::Option<&crate::BackfillCancellationSignal>,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillDiscovery>
|
||||||
|
where
|
||||||
|
S: SignaturePageSource,
|
||||||
|
{
|
||||||
|
let address = match request.scope().address() {
|
||||||
|
std::option::Option::Some(address) => address,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(discovery_invalid("scope.address")),
|
||||||
|
};
|
||||||
|
let mut before = match request.scope().kind() {
|
||||||
|
crate::BackfillScopeKind::BeforeAddress => crate::resume_before_cursor(request),
|
||||||
|
crate::BackfillScopeKind::LatestAddress => std::option::Option::None,
|
||||||
|
crate::BackfillScopeKind::AfterAddress | crate::BackfillScopeKind::ExplicitSignatures => {
|
||||||
|
return std::result::Result::Err(discovery_invalid("scope.kind"));
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let mut candidates = std::vec::Vec::with_capacity(request.max_candidates());
|
||||||
|
let mut seen = std::collections::HashSet::<crate::BackfillSignature>::with_capacity(request.max_candidates());
|
||||||
|
let mut pages_fetched = 0_usize;
|
||||||
|
let boundary = loop {
|
||||||
|
if candidates.len() >= request.max_candidates() {
|
||||||
|
break crate::BackfillDiscoveryBoundary::CandidateLimit;
|
||||||
|
}
|
||||||
|
if pages_fetched >= request.max_pages() {
|
||||||
|
break crate::BackfillDiscoveryBoundary::PageLimit;
|
||||||
|
}
|
||||||
|
let remaining = request.max_candidates() - candidates.len();
|
||||||
|
let page_limit = std::cmp::min(request.page_size(), remaining);
|
||||||
|
let config = ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig::new(
|
||||||
|
before.clone(),
|
||||||
|
std::option::Option::None,
|
||||||
|
std::option::Option::Some(page_limit),
|
||||||
|
std::option::Option::Some(request.commitment().transport()),
|
||||||
|
request.min_context_slot(),
|
||||||
|
);
|
||||||
|
let page_result = fetch_signature_page(source, request.role(), address, config, cancellation).await;
|
||||||
|
let page = match page_result {
|
||||||
|
std::result::Result::Ok(page) => page,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
pages_fetched += 1;
|
||||||
|
if page.len() > page_limit {
|
||||||
|
return std::result::Result::Err(discovery_invalid("page.length"));
|
||||||
|
}
|
||||||
|
let page_len = page.len();
|
||||||
|
let next_before = match page.last() {
|
||||||
|
std::option::Option::Some(entry) => {
|
||||||
|
let signature_result = validated_signature(entry.signature.as_str());
|
||||||
|
match signature_result {
|
||||||
|
std::result::Result::Ok(signature) => std::option::Option::Some(signature),
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
for entry in page {
|
||||||
|
let signature_result = validated_signature(entry.signature.as_str());
|
||||||
|
let signature = match signature_result {
|
||||||
|
std::result::Result::Ok(signature) => signature,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if seen.insert(signature.clone()) {
|
||||||
|
let identity = crate::BackfillCandidateIdentity::new(request.network().clone(), signature);
|
||||||
|
candidates.push(crate::BackfillCandidate::new(identity, std::option::Option::Some(entry.slot)));
|
||||||
|
if candidates.len() >= request.max_candidates() {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if page_len < page_limit {
|
||||||
|
break crate::BackfillDiscoveryBoundary::RpcBoundary;
|
||||||
|
}
|
||||||
|
let next_before = match next_before {
|
||||||
|
std::option::Option::Some(next_before) => next_before,
|
||||||
|
std::option::Option::None => break crate::BackfillDiscoveryBoundary::RpcBoundary,
|
||||||
|
};
|
||||||
|
if before.as_deref() == std::option::Option::Some(next_before.as_str()) {
|
||||||
|
return std::result::Result::Err(discovery_stalled());
|
||||||
|
}
|
||||||
|
before = std::option::Option::Some(next_before.as_str().to_owned());
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(crate::BackfillDiscovery::new(request.network().clone(), request.scope_fingerprint(), candidates, pages_fetched, boundary));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn discover_after<S>(
|
||||||
|
source: &S,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
cancellation: std::option::Option<&crate::BackfillCancellationSignal>,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillDiscovery>
|
||||||
|
where
|
||||||
|
S: SignaturePageSource,
|
||||||
|
{
|
||||||
|
let address = match request.scope().address() {
|
||||||
|
std::option::Option::Some(address) => address,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(discovery_invalid("scope.address")),
|
||||||
|
};
|
||||||
|
let anchor = match request.scope().anchor() {
|
||||||
|
std::option::Option::Some(anchor) => anchor,
|
||||||
|
std::option::Option::None => return std::result::Result::Err(discovery_invalid("scope.anchor")),
|
||||||
|
};
|
||||||
|
let until = anchor.as_str().to_owned();
|
||||||
|
let mut before = std::option::Option::<std::string::String>::None;
|
||||||
|
let mut nearest = std::collections::VecDeque::<crate::BackfillCandidate>::with_capacity(request.max_candidates());
|
||||||
|
let mut seen = std::collections::HashSet::<crate::BackfillSignature>::new();
|
||||||
|
let mut pages_fetched = 0_usize;
|
||||||
|
let boundary = loop {
|
||||||
|
if pages_fetched >= request.max_pages() {
|
||||||
|
break crate::BackfillDiscoveryBoundary::AfterAnchorNotReached;
|
||||||
|
}
|
||||||
|
let config = ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig::new(
|
||||||
|
before.clone(),
|
||||||
|
std::option::Option::Some(until.clone()),
|
||||||
|
std::option::Option::Some(request.page_size()),
|
||||||
|
std::option::Option::Some(request.commitment().transport()),
|
||||||
|
request.min_context_slot(),
|
||||||
|
);
|
||||||
|
let page_result = fetch_signature_page(source, request.role(), address, config, cancellation).await;
|
||||||
|
let page = match page_result {
|
||||||
|
std::result::Result::Ok(page) => page,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
pages_fetched += 1;
|
||||||
|
if page.len() > request.page_size() {
|
||||||
|
return std::result::Result::Err(discovery_invalid("page.length"));
|
||||||
|
}
|
||||||
|
let page_len = page.len();
|
||||||
|
let next_before = match page.last() {
|
||||||
|
std::option::Option::Some(entry) => {
|
||||||
|
let signature_result = validated_signature(entry.signature.as_str());
|
||||||
|
match signature_result {
|
||||||
|
std::result::Result::Ok(signature) => std::option::Option::Some(signature),
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::option::Option::None => std::option::Option::None,
|
||||||
|
};
|
||||||
|
for entry in page {
|
||||||
|
let signature_result = validated_signature(entry.signature.as_str());
|
||||||
|
let signature = match signature_result {
|
||||||
|
std::result::Result::Ok(signature) => signature,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if seen.insert(signature.clone()) {
|
||||||
|
let identity = crate::BackfillCandidateIdentity::new(request.network().clone(), signature);
|
||||||
|
nearest.push_back(crate::BackfillCandidate::new(identity, std::option::Option::Some(entry.slot)));
|
||||||
|
if nearest.len() > request.max_candidates() {
|
||||||
|
nearest.pop_front();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if page_len < request.page_size() {
|
||||||
|
break crate::BackfillDiscoveryBoundary::RpcBoundary;
|
||||||
|
}
|
||||||
|
let next_before = match next_before {
|
||||||
|
std::option::Option::Some(next_before) => next_before,
|
||||||
|
std::option::Option::None => break crate::BackfillDiscoveryBoundary::RpcBoundary,
|
||||||
|
};
|
||||||
|
if before.as_deref() == std::option::Option::Some(next_before.as_str()) {
|
||||||
|
return std::result::Result::Err(discovery_stalled());
|
||||||
|
}
|
||||||
|
before = std::option::Option::Some(next_before.as_str().to_owned());
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(crate::BackfillDiscovery::new(
|
||||||
|
request.network().clone(),
|
||||||
|
request.scope_fingerprint(),
|
||||||
|
nearest.into_iter().collect(),
|
||||||
|
pages_fetched,
|
||||||
|
boundary,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn fetch_signature_page<S>(
|
||||||
|
source: &S,
|
||||||
|
role: &ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
address: &ksp_core_lib::Pubkey,
|
||||||
|
config: ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig,
|
||||||
|
cancellation: std::option::Option<&crate::BackfillCancellationSignal>,
|
||||||
|
) -> ksp_core_lib::Result<std::vec::Vec<SignaturePageEntry>>
|
||||||
|
where
|
||||||
|
S: SignaturePageSource,
|
||||||
|
{
|
||||||
|
let future = source.fetch_signature_page(role, address, config);
|
||||||
|
return match cancellation {
|
||||||
|
std::option::Option::Some(signal) => signal.run_cancellable(future).await,
|
||||||
|
std::option::Option::None => future.await,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cancelled_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_CANCELLED, "Backfill discovery cancelled before RPC completion");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validated_signature(value: &str) -> ksp_core_lib::Result<crate::BackfillSignature> {
|
||||||
|
return crate::BackfillSignature::new(value.to_owned());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn discovery_invalid(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_DISCOVERY_INVALID, "invalid bounded Backfill discovery state").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn discovery_stalled() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_DISCOVERY_STALLED, "Backfill discovery cursor did not advance").with_context("field", "before");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/discovery.rs"]
|
||||||
|
mod tests;
|
||||||
24
crates/ksp-job-backfill-lib/src/error.rs
Normal file
24
crates/ksp-job-backfill-lib/src/error.rs
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/error.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
/// Error code used when one Backfill checkpoint/frontier is incompatible with the current Job or semantic scope.
|
||||||
|
pub const ERROR_CODE_BACKFILL_CHECKPOINT_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "checkpoint_invalid");
|
||||||
|
/// Error code used when a signature page violates a bounded discovery invariant.
|
||||||
|
pub const ERROR_CODE_BACKFILL_DISCOVERY_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "discovery_invalid");
|
||||||
|
/// Error code used when paginated discovery cannot advance its exclusive RPC cursor safely.
|
||||||
|
pub const ERROR_CODE_BACKFILL_DISCOVERY_STALLED: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "discovery_stalled");
|
||||||
|
/// Error code used when bounded concurrent candidate execution violates its internal admission or clock invariants.
|
||||||
|
pub const ERROR_CODE_BACKFILL_EXECUTION_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "execution_invalid");
|
||||||
|
/// Error code used when Store persistence returns an impossible Backfill state or targets a different network.
|
||||||
|
pub const ERROR_CODE_BACKFILL_PERSISTENCE_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "persistence_invalid");
|
||||||
|
/// Error code used when deterministic Transport-to-RAW conversion violates the v1 contract.
|
||||||
|
pub const ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "raw_conversion_invalid");
|
||||||
|
/// Error code used when one Backfill request violates its bounded admission contract.
|
||||||
|
pub const ERROR_CODE_BACKFILL_REQUEST_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "request_invalid");
|
||||||
|
/// Error code used when the concrete Backfill runtime reaches an impossible lifecycle or notification state.
|
||||||
|
pub const ERROR_CODE_BACKFILL_RUNTIME_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "runtime_invalid");
|
||||||
|
/// Error code used when one transaction signature text violates the bounded Base58-shape contract.
|
||||||
|
pub const ERROR_CODE_BACKFILL_SIGNATURE_INVALID: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "signature_invalid");
|
||||||
|
|
||||||
|
/// Error code used internally when cooperative cancellation wins before durable Store submission.
|
||||||
|
pub(crate) const ERROR_CODE_BACKFILL_CANCELLED: ksp_core_lib::ErrorCode = ksp_core_lib::ErrorCode::new("job_backfill", "cancelled");
|
||||||
660
crates/ksp-job-backfill-lib/src/execution.rs
Normal file
660
crates/ksp-job-backfill-lib/src/execution.rs
Normal file
@@ -0,0 +1,660 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/execution.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
use futures_util::StreamExt; // rust-rules: trait-import
|
||||||
|
|
||||||
|
/// Bounded result of one concurrent Backfill candidate execution pass.
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
pub struct BackfillExecutionBatch {
|
||||||
|
candidate_count: usize,
|
||||||
|
admitted_count: usize,
|
||||||
|
finished_count: usize,
|
||||||
|
durable_count: usize,
|
||||||
|
inserted_count: usize,
|
||||||
|
already_present_count: usize,
|
||||||
|
purged_count: usize,
|
||||||
|
missing_count: usize,
|
||||||
|
conflict_count: usize,
|
||||||
|
observation_inserted_count: usize,
|
||||||
|
observation_already_present_count: usize,
|
||||||
|
cancelled_count: usize,
|
||||||
|
hole_count: usize,
|
||||||
|
maximum_in_flight: usize,
|
||||||
|
local_contiguous_completed: usize,
|
||||||
|
discovery_partial: bool,
|
||||||
|
cancelled: bool,
|
||||||
|
checkpoint: crate::BackfillCheckpoint,
|
||||||
|
failure_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillExecutionBatch {
|
||||||
|
/// Returns the number of candidates present in the bounded discovery result.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn candidate_count(&self) -> usize {
|
||||||
|
return self.candidate_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of candidates admitted into hydration during this execution pass.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn admitted_count(&self) -> usize {
|
||||||
|
return self.admitted_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted candidates whose hydration/persistence future reached a known outcome.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn finished_count(&self) -> usize {
|
||||||
|
return self.finished_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of candidates whose Store outcome is durable for checkpoint advancement.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn durable_count(&self) -> usize {
|
||||||
|
return self.durable_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted canonical RAW entities.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn inserted_count(&self) -> usize {
|
||||||
|
return self.inserted_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of canonical RAW entities already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn already_present_count(&self) -> usize {
|
||||||
|
return self.already_present_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of purge tombstones respected by normal persistence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn purged_count(&self) -> usize {
|
||||||
|
return self.purged_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of candidates whose `getTransaction` result was missing.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn missing_count(&self) -> usize {
|
||||||
|
return self.missing_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of Store content conflicts.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn conflict_count(&self) -> usize {
|
||||||
|
return self.conflict_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted acquisition observations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_inserted_count(&self) -> usize {
|
||||||
|
return self.observation_inserted_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of acquisition observations already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation_already_present_count(&self) -> usize {
|
||||||
|
return self.observation_already_present_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted candidates cancelled before Store submission.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn cancelled_count(&self) -> usize {
|
||||||
|
return self.cancelled_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of known candidate outcomes that intentionally block the contiguous frontier.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn hole_count(&self) -> usize {
|
||||||
|
return self.hole_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the greatest number of candidate futures simultaneously in flight.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn maximum_in_flight(&self) -> usize {
|
||||||
|
return self.maximum_in_flight;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the durable contiguous prefix completed inside the current discovery result.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn local_contiguous_completed(&self) -> usize {
|
||||||
|
return self.local_contiguous_completed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the safe caller-owned checkpoint after draining all work already admitted.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn checkpoint(&self) -> &crate::BackfillCheckpoint {
|
||||||
|
return &self.checkpoint;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the first stable fatal error code that stopped new admissions, when one occurred.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn failure_code(&self) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
return self.failure_code;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether cooperative cancellation stopped admissions or cancelled an admitted pre-Store operation.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn was_cancelled(&self) -> bool {
|
||||||
|
return self.cancelled;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether discovery or candidate outcomes left the pass incomplete.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn is_partial(&self) -> bool {
|
||||||
|
return self.discovery_partial || self.hole_count != 0 || self.failure_code.is_some() || self.cancelled;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal latest-value execution facts published after each known candidate completion.
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
pub(crate) struct BackfillExecutionProgress {
|
||||||
|
admitted_count: usize,
|
||||||
|
finished_count: usize,
|
||||||
|
inserted_count: usize,
|
||||||
|
already_present_count: usize,
|
||||||
|
purged_count: usize,
|
||||||
|
missing_count: usize,
|
||||||
|
conflict_count: usize,
|
||||||
|
observation_inserted_count: usize,
|
||||||
|
observation_already_present_count: usize,
|
||||||
|
cancelled_count: usize,
|
||||||
|
hole_count: usize,
|
||||||
|
maximum_in_flight: usize,
|
||||||
|
contiguous_completed: usize,
|
||||||
|
checkpoint: crate::BackfillCheckpoint,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillExecutionProgress {
|
||||||
|
/// Returns the cumulative admission count.
|
||||||
|
pub(crate) const fn admitted_count(&self) -> usize {
|
||||||
|
return self.admitted_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative finished count.
|
||||||
|
pub(crate) const fn finished_count(&self) -> usize {
|
||||||
|
return self.finished_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative inserted entity count.
|
||||||
|
pub(crate) const fn inserted_count(&self) -> usize {
|
||||||
|
return self.inserted_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative already-present entity count.
|
||||||
|
pub(crate) const fn already_present_count(&self) -> usize {
|
||||||
|
return self.already_present_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative purged entity count.
|
||||||
|
pub(crate) const fn purged_count(&self) -> usize {
|
||||||
|
return self.purged_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative missing count.
|
||||||
|
pub(crate) const fn missing_count(&self) -> usize {
|
||||||
|
return self.missing_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative conflict count.
|
||||||
|
pub(crate) const fn conflict_count(&self) -> usize {
|
||||||
|
return self.conflict_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative inserted observation count.
|
||||||
|
pub(crate) const fn observation_inserted_count(&self) -> usize {
|
||||||
|
return self.observation_inserted_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative already-present observation count.
|
||||||
|
pub(crate) const fn observation_already_present_count(&self) -> usize {
|
||||||
|
return self.observation_already_present_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative cancelled candidate count.
|
||||||
|
pub(crate) const fn cancelled_count(&self) -> usize {
|
||||||
|
return self.cancelled_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative hole count.
|
||||||
|
pub(crate) const fn hole_count(&self) -> usize {
|
||||||
|
return self.hole_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the maximum observed in-flight count.
|
||||||
|
pub(crate) const fn maximum_in_flight(&self) -> usize {
|
||||||
|
return self.maximum_in_flight;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative safe contiguous prefix represented by the progress checkpoint.
|
||||||
|
pub(crate) const fn contiguous_completed(&self) -> usize {
|
||||||
|
return self.contiguous_completed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the safe checkpoint proven at this progress position.
|
||||||
|
pub(crate) const fn checkpoint(&self) -> &crate::BackfillCheckpoint {
|
||||||
|
return &self.checkpoint;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Executes one bounded discovered candidate set with request-owned hydration concurrency.
|
||||||
|
///
|
||||||
|
/// This compatibility entry point has no external cancellation handle. Transport owns retries and
|
||||||
|
/// endpoint selection; Store submissions are always awaited to completion.
|
||||||
|
pub async fn execute_backfill_discovery(
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
store: &ksp_store_lib::Store,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
discovery: &crate::BackfillDiscovery,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillExecutionBatch> {
|
||||||
|
let validation = crate::validate_discovery_identity(request, discovery);
|
||||||
|
if let std::result::Result::Err(error) = validation {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let processor = RuntimeCandidateProcessor { transport, store, request, cancellation: std::option::Option::None };
|
||||||
|
return execute_with_processor(&processor, request, discovery, std::option::Option::None, std::option::Option::None).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Executes one bounded discovery with cooperative cancellation and concrete progress publication.
|
||||||
|
pub(crate) async fn execute_backfill_discovery_cancellable(
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
store: &ksp_store_lib::Store,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
discovery: &crate::BackfillDiscovery,
|
||||||
|
cancellation: &crate::BackfillCancellationSignal,
|
||||||
|
publisher: &crate::BackfillRuntimePublisher,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillExecutionBatch> {
|
||||||
|
let processor = RuntimeCandidateProcessor { transport, store, request, cancellation: std::option::Option::Some(cancellation) };
|
||||||
|
return execute_with_processor(&processor, request, discovery, std::option::Option::Some(cancellation), std::option::Option::Some(publisher)).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
type CandidateProcessFuture<'a> =
|
||||||
|
std::pin::Pin<std::boxed::Box<dyn std::future::Future<Output = ksp_core_lib::Result<crate::BackfillPersistenceOutcome>> + 'a>>;
|
||||||
|
|
||||||
|
trait CandidateProcessor {
|
||||||
|
fn process<'a>(&'a self, candidate: &'a crate::BackfillCandidate) -> CandidateProcessFuture<'a>;
|
||||||
|
}
|
||||||
|
|
||||||
|
struct RuntimeCandidateProcessor<'a> {
|
||||||
|
transport: &'a ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
store: &'a ksp_store_lib::Store,
|
||||||
|
request: &'a crate::BackfillRequest,
|
||||||
|
cancellation: std::option::Option<&'a crate::BackfillCancellationSignal>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CandidateProcessor for RuntimeCandidateProcessor<'_> {
|
||||||
|
fn process<'a>(&'a self, candidate: &'a crate::BackfillCandidate) -> CandidateProcessFuture<'a> {
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
let received_at = match current_raw_timestamp() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let hydration_future = crate::hydrate_backfill_candidate(self.transport, self.request, candidate, received_at);
|
||||||
|
let hydration = match self.cancellation {
|
||||||
|
std::option::Option::Some(signal) => signal.run_cancellable(hydration_future).await,
|
||||||
|
std::option::Option::None => hydration_future.await,
|
||||||
|
};
|
||||||
|
let hydration = match hydration {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if self.cancellation.is_some_and(crate::BackfillCancellationSignal::is_requested) {
|
||||||
|
return std::result::Result::Err(cancelled_error());
|
||||||
|
}
|
||||||
|
// Deliberately not cancellable once Store submission begins: a submitted durable write is drained.
|
||||||
|
return crate::persist_backfill_hydration(self.store, hydration).await;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_lines)]
|
||||||
|
async fn execute_with_processor<P>(
|
||||||
|
processor: &P,
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
discovery: &crate::BackfillDiscovery,
|
||||||
|
cancellation: std::option::Option<&crate::BackfillCancellationSignal>,
|
||||||
|
publisher: std::option::Option<&crate::BackfillRuntimePublisher>,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillExecutionBatch>
|
||||||
|
where
|
||||||
|
P: CandidateProcessor,
|
||||||
|
{
|
||||||
|
let validation = crate::validate_discovery_identity(request, discovery);
|
||||||
|
if let std::result::Result::Err(error) = validation {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let resume_offset = match crate::execution_resume_offset(request, discovery) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut frontier = crate::CompletionFrontier::new(discovery.candidates().len());
|
||||||
|
if matches!(request.scope().kind(), crate::BackfillScopeKind::AfterAddress | crate::BackfillScopeKind::ExplicitSignatures) {
|
||||||
|
let seeded = frontier.seed_prefix(resume_offset);
|
||||||
|
if let std::result::Result::Err(error) = seeded {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let mut next_index = resume_offset;
|
||||||
|
let mut admitted_count = 0_usize;
|
||||||
|
let mut finished_count = 0_usize;
|
||||||
|
let mut durable_count = 0_usize;
|
||||||
|
let mut inserted_count = 0_usize;
|
||||||
|
let mut already_present_count = 0_usize;
|
||||||
|
let mut purged_count = 0_usize;
|
||||||
|
let mut missing_count = 0_usize;
|
||||||
|
let mut conflict_count = 0_usize;
|
||||||
|
let mut observation_inserted_count = 0_usize;
|
||||||
|
let mut observation_already_present_count = 0_usize;
|
||||||
|
let mut cancelled_count = 0_usize;
|
||||||
|
let mut hole_count = 0_usize;
|
||||||
|
let mut maximum_in_flight = 0_usize;
|
||||||
|
let mut failure_code = std::option::Option::<ksp_core_lib::ErrorCode>::None;
|
||||||
|
let mut cancellation_requested = false;
|
||||||
|
let mut in_flight = futures_util::stream::FuturesUnordered::new();
|
||||||
|
loop {
|
||||||
|
if !cancellation_requested && cancellation.is_some_and(crate::BackfillCancellationSignal::is_requested) {
|
||||||
|
cancellation_requested = true;
|
||||||
|
let progress = progress_from_state(
|
||||||
|
request,
|
||||||
|
discovery,
|
||||||
|
&frontier,
|
||||||
|
admitted_count,
|
||||||
|
finished_count,
|
||||||
|
inserted_count,
|
||||||
|
already_present_count,
|
||||||
|
purged_count,
|
||||||
|
missing_count,
|
||||||
|
conflict_count,
|
||||||
|
observation_inserted_count,
|
||||||
|
observation_already_present_count,
|
||||||
|
cancelled_count,
|
||||||
|
hole_count,
|
||||||
|
maximum_in_flight,
|
||||||
|
);
|
||||||
|
let progress = match progress {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let published = publish_progress(publisher, &progress, true, true);
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
while failure_code.is_none()
|
||||||
|
&& !cancellation_requested
|
||||||
|
&& next_index < discovery.candidates().len()
|
||||||
|
&& in_flight.len() < request.hydration_concurrency()
|
||||||
|
{
|
||||||
|
let index = next_index;
|
||||||
|
let candidate = &discovery.candidates()[index];
|
||||||
|
let future = processor.process(candidate);
|
||||||
|
in_flight.push(async move {
|
||||||
|
return (index, future.await);
|
||||||
|
});
|
||||||
|
next_index += 1;
|
||||||
|
admitted_count = match checked_increment(admitted_count, "admitted_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
maximum_in_flight = std::cmp::max(maximum_in_flight, in_flight.len());
|
||||||
|
}
|
||||||
|
let completed = in_flight.next().await;
|
||||||
|
let (index, result) = match completed {
|
||||||
|
std::option::Option::Some(completed) => completed,
|
||||||
|
std::option::Option::None => break,
|
||||||
|
};
|
||||||
|
finished_count = match checked_increment(finished_count, "finished_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
match result {
|
||||||
|
std::result::Result::Ok(outcome) => {
|
||||||
|
let classified = classify_persistence(
|
||||||
|
&outcome,
|
||||||
|
&mut inserted_count,
|
||||||
|
&mut already_present_count,
|
||||||
|
&mut purged_count,
|
||||||
|
&mut missing_count,
|
||||||
|
&mut conflict_count,
|
||||||
|
&mut observation_inserted_count,
|
||||||
|
&mut observation_already_present_count,
|
||||||
|
);
|
||||||
|
if let std::result::Result::Err(error) = classified {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
if persistence_advances_frontier(&outcome) {
|
||||||
|
let marked = frontier.mark_durable(index);
|
||||||
|
if let std::result::Result::Err(error) = marked {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
durable_count = match checked_increment(durable_count, "durable_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
} else {
|
||||||
|
hole_count = match checked_increment(hole_count, "hole_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if outcome.entity() == crate::BackfillEntityPersistence::Conflict && failure_code.is_none() {
|
||||||
|
failure_code = std::option::Option::Some(ksp_store_lib::ERROR_CODE_RAW_CONFLICT);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
hole_count = match checked_increment(hole_count, "hole_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(increment_error) => return std::result::Result::Err(increment_error),
|
||||||
|
};
|
||||||
|
if error.code() == crate::ERROR_CODE_BACKFILL_CANCELLED {
|
||||||
|
cancelled_count = match checked_increment(cancelled_count, "cancelled_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(increment_error) => return std::result::Result::Err(increment_error),
|
||||||
|
};
|
||||||
|
cancellation_requested = true;
|
||||||
|
} else if failure_code.is_none() {
|
||||||
|
failure_code = std::option::Option::Some(error.code());
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
let progress = progress_from_state(
|
||||||
|
request,
|
||||||
|
discovery,
|
||||||
|
&frontier,
|
||||||
|
admitted_count,
|
||||||
|
finished_count,
|
||||||
|
inserted_count,
|
||||||
|
already_present_count,
|
||||||
|
purged_count,
|
||||||
|
missing_count,
|
||||||
|
conflict_count,
|
||||||
|
observation_inserted_count,
|
||||||
|
observation_already_present_count,
|
||||||
|
cancelled_count,
|
||||||
|
hole_count,
|
||||||
|
maximum_in_flight,
|
||||||
|
);
|
||||||
|
let progress = match progress {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let draining = cancellation_requested || failure_code.is_some();
|
||||||
|
let published = publish_progress(publisher, &progress, cancellation_requested, draining);
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let checkpoint = match crate::checkpoint_from_frontier(request, discovery, &frontier) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(crate::BackfillExecutionBatch {
|
||||||
|
candidate_count: discovery.candidates().len(),
|
||||||
|
admitted_count,
|
||||||
|
finished_count,
|
||||||
|
durable_count,
|
||||||
|
inserted_count,
|
||||||
|
already_present_count,
|
||||||
|
purged_count,
|
||||||
|
missing_count,
|
||||||
|
conflict_count,
|
||||||
|
observation_inserted_count,
|
||||||
|
observation_already_present_count,
|
||||||
|
cancelled_count,
|
||||||
|
hole_count,
|
||||||
|
maximum_in_flight,
|
||||||
|
local_contiguous_completed: frontier.contiguous_completed(),
|
||||||
|
discovery_partial: discovery.is_partial(),
|
||||||
|
cancelled: cancellation_requested,
|
||||||
|
checkpoint,
|
||||||
|
failure_code,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
fn progress_from_state(
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
discovery: &crate::BackfillDiscovery,
|
||||||
|
frontier: &crate::CompletionFrontier,
|
||||||
|
admitted_count: usize,
|
||||||
|
finished_count: usize,
|
||||||
|
inserted_count: usize,
|
||||||
|
already_present_count: usize,
|
||||||
|
purged_count: usize,
|
||||||
|
missing_count: usize,
|
||||||
|
conflict_count: usize,
|
||||||
|
observation_inserted_count: usize,
|
||||||
|
observation_already_present_count: usize,
|
||||||
|
cancelled_count: usize,
|
||||||
|
hole_count: usize,
|
||||||
|
maximum_in_flight: usize,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillExecutionProgress> {
|
||||||
|
let checkpoint = match crate::checkpoint_from_frontier(request, discovery, frontier) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(crate::BackfillExecutionProgress {
|
||||||
|
admitted_count,
|
||||||
|
finished_count,
|
||||||
|
inserted_count,
|
||||||
|
already_present_count,
|
||||||
|
purged_count,
|
||||||
|
missing_count,
|
||||||
|
conflict_count,
|
||||||
|
observation_inserted_count,
|
||||||
|
observation_already_present_count,
|
||||||
|
cancelled_count,
|
||||||
|
hole_count,
|
||||||
|
maximum_in_flight,
|
||||||
|
contiguous_completed: checkpoint.completed_prefix(),
|
||||||
|
checkpoint,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish_progress(
|
||||||
|
publisher: std::option::Option<&crate::BackfillRuntimePublisher>,
|
||||||
|
progress: &crate::BackfillExecutionProgress,
|
||||||
|
cancelling: bool,
|
||||||
|
draining: bool,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
if let std::option::Option::Some(publisher) = publisher {
|
||||||
|
let published = publisher.publish_execution_progress(progress, cancelling, draining);
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
fn classify_persistence(
|
||||||
|
outcome: &crate::BackfillPersistenceOutcome,
|
||||||
|
inserted_count: &mut usize,
|
||||||
|
already_present_count: &mut usize,
|
||||||
|
purged_count: &mut usize,
|
||||||
|
missing_count: &mut usize,
|
||||||
|
conflict_count: &mut usize,
|
||||||
|
observation_inserted_count: &mut usize,
|
||||||
|
observation_already_present_count: &mut usize,
|
||||||
|
) -> ksp_core_lib::Result<()> {
|
||||||
|
match outcome.entity() {
|
||||||
|
crate::BackfillEntityPersistence::Inserted => {
|
||||||
|
*inserted_count = match checked_increment(*inserted_count, "inserted_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::BackfillEntityPersistence::AlreadyPresent => {
|
||||||
|
*already_present_count = match checked_increment(*already_present_count, "already_present_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::BackfillEntityPersistence::SkippedPurged => {
|
||||||
|
*purged_count = match checked_increment(*purged_count, "purged_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::BackfillEntityPersistence::Missing => {
|
||||||
|
*missing_count = match checked_increment(*missing_count, "missing_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::BackfillEntityPersistence::Conflict => {
|
||||||
|
*conflict_count = match checked_increment(*conflict_count, "conflict_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
}
|
||||||
|
match outcome.observation() {
|
||||||
|
crate::BackfillObservationPersistence::Inserted => {
|
||||||
|
*observation_inserted_count = match checked_increment(*observation_inserted_count, "observation_inserted_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::BackfillObservationPersistence::AlreadyPresent => {
|
||||||
|
*observation_already_present_count = match checked_increment(*observation_already_present_count, "observation_already_present_count") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
},
|
||||||
|
crate::BackfillObservationPersistence::NotRecorded | crate::BackfillObservationPersistence::NotApplicable => {},
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persistence_advances_frontier(outcome: &crate::BackfillPersistenceOutcome) -> bool {
|
||||||
|
return matches!(
|
||||||
|
outcome.entity(),
|
||||||
|
crate::BackfillEntityPersistence::Inserted | crate::BackfillEntityPersistence::AlreadyPresent | crate::BackfillEntityPersistence::SkippedPurged
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn checked_increment(value: usize, field: &'static str) -> ksp_core_lib::Result<usize> {
|
||||||
|
return value.checked_add(1).ok_or_else(|| return execution_error(field));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn current_raw_timestamp() -> ksp_core_lib::Result<ksp_store_lib::RawTimestamp> {
|
||||||
|
let duration = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH);
|
||||||
|
let duration = match duration {
|
||||||
|
std::result::Result::Ok(duration) => duration,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(execution_error("clock.before_epoch")),
|
||||||
|
};
|
||||||
|
let millis = match u64::try_from(duration.as_millis()) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::result::Result::Err(execution_error("clock.millis")),
|
||||||
|
};
|
||||||
|
return ksp_store_lib::RawTimestamp::from_unix_millis(millis);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cancelled_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_CANCELLED, "Backfill candidate cancelled before durable Store submission");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn execution_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_EXECUTION_INVALID, "invalid bounded Backfill execution state").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/execution.rs"]
|
||||||
|
mod tests;
|
||||||
143
crates/ksp-job-backfill-lib/src/lib.rs
Normal file
143
crates/ksp-job-backfill-lib/src/lib.rs
Normal file
@@ -0,0 +1,143 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
#![warn(missing_docs)]
|
||||||
|
#![deny(unreachable_pub)]
|
||||||
|
#![forbid(unsafe_code)]
|
||||||
|
|
||||||
|
//! Concrete bounded historical RAW transaction Backfill foundation.
|
||||||
|
//!
|
||||||
|
//! This tranche owns explicit admission, network-scoped candidate identity, deterministic
|
||||||
|
//! `getSignaturesForAddress` pagination and canonical RAW v1 conversion through observed
|
||||||
|
//! `getTransaction`. Transport retains provider/endpoint selection and retry; Store retains
|
||||||
|
//! durable idempotence through the atomic Store facade. This tranche also owns bounded concurrent
|
||||||
|
//! candidate execution, caller-owned contiguous checkpoints, cooperative cancellation and concrete
|
||||||
|
//! latest-value snapshots for external listeners.
|
||||||
|
|
||||||
|
mod checkpoint;
|
||||||
|
mod constants;
|
||||||
|
mod conversion;
|
||||||
|
mod discovery;
|
||||||
|
mod error;
|
||||||
|
mod execution;
|
||||||
|
mod persistence;
|
||||||
|
mod request;
|
||||||
|
mod runtime;
|
||||||
|
|
||||||
|
/// Opaque caller-owned checkpoint for one controlled Backfill resumption.
|
||||||
|
pub use self::checkpoint::BackfillCheckpoint;
|
||||||
|
/// Result of hydrating one deterministic candidate through observed `getTransaction`.
|
||||||
|
pub use self::conversion::BackfillHydrationOutcome;
|
||||||
|
/// Complete in-memory RAW transaction acquisition ready for later Store persistence.
|
||||||
|
pub use self::conversion::BackfillRawAcquisition;
|
||||||
|
/// KSP-owned source-independent RAW transaction format identifier produced by this Backfill vertical.
|
||||||
|
pub use self::conversion::RAW_TRANSACTION_FORMAT_ID;
|
||||||
|
/// Initial KSP-owned RAW transaction format version produced by this Backfill vertical.
|
||||||
|
pub use self::conversion::RAW_TRANSACTION_FORMAT_VERSION;
|
||||||
|
/// Hydrates one candidate through observed Transport and converts a non-null response to canonical RAW v1.
|
||||||
|
pub use self::conversion::hydrate_backfill_candidate;
|
||||||
|
/// One deterministic transaction candidate produced by bounded discovery.
|
||||||
|
pub use self::discovery::BackfillCandidate;
|
||||||
|
/// Network-scoped identity of one discovered transaction candidate before canonical signature decoding.
|
||||||
|
pub use self::discovery::BackfillCandidateIdentity;
|
||||||
|
/// Complete bounded output of one candidate discovery pass.
|
||||||
|
pub use self::discovery::BackfillDiscovery;
|
||||||
|
/// Reason bounded candidate discovery stopped.
|
||||||
|
pub use self::discovery::BackfillDiscoveryBoundary;
|
||||||
|
/// Discovers one bounded deterministic candidate set through the typed KSP Transport wrapper.
|
||||||
|
pub use self::discovery::discover_backfill_candidates;
|
||||||
|
/// Error code used when one checkpoint/frontier is incompatible with the current Job or semantic scope.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_CHECKPOINT_INVALID;
|
||||||
|
/// Error code used when a signature page violates a bounded discovery invariant.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_DISCOVERY_INVALID;
|
||||||
|
/// Error code used when paginated discovery cannot advance its exclusive RPC cursor safely.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_DISCOVERY_STALLED;
|
||||||
|
/// Error code used when bounded concurrent execution violates an internal admission or clock invariant.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_EXECUTION_INVALID;
|
||||||
|
/// Error code used when Store persistence returns an impossible Backfill state or targets a different network.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_PERSISTENCE_INVALID;
|
||||||
|
/// Error code used when deterministic Transport-to-RAW conversion violates the v1 contract.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID;
|
||||||
|
/// Error code used when one Backfill request violates its bounded admission contract.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_REQUEST_INVALID;
|
||||||
|
/// Error code used when the concrete Backfill runtime reaches an impossible lifecycle or notification state.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_RUNTIME_INVALID;
|
||||||
|
/// Error code used when one transaction signature text violates the bounded Base58-shape contract.
|
||||||
|
pub use self::error::ERROR_CODE_BACKFILL_SIGNATURE_INVALID;
|
||||||
|
/// Bounded result of one concurrent Backfill candidate execution pass.
|
||||||
|
pub use self::execution::BackfillExecutionBatch;
|
||||||
|
/// Executes one bounded discovered candidate set with request-owned hydration concurrency.
|
||||||
|
pub use self::execution::execute_backfill_discovery;
|
||||||
|
/// Canonical entity disposition produced by one Backfill Store persistence attempt.
|
||||||
|
pub use self::persistence::BackfillEntityPersistence;
|
||||||
|
/// Observation disposition produced by one Backfill Store persistence attempt.
|
||||||
|
pub use self::persistence::BackfillObservationPersistence;
|
||||||
|
/// Stable Backfill projection of one hydration persistence result.
|
||||||
|
pub use self::persistence::BackfillPersistenceOutcome;
|
||||||
|
/// Persists one hydrated result through the backend-neutral atomic Store contract.
|
||||||
|
pub use self::persistence::persist_backfill_hydration;
|
||||||
|
/// Commitment levels intentionally admitted by the historical Backfill vertical.
|
||||||
|
pub use self::request::BackfillCommitment;
|
||||||
|
/// Fully explicit bounded request for one historical transaction Backfill Job.
|
||||||
|
pub use self::request::BackfillRequest;
|
||||||
|
/// Validated bounded discovery scope for one historical Backfill Job.
|
||||||
|
pub use self::request::BackfillScope;
|
||||||
|
/// Opaque deterministic fingerprint of one semantic Backfill scope.
|
||||||
|
pub use self::request::BackfillScopeFingerprint;
|
||||||
|
/// Stable category of one bounded Backfill discovery scope.
|
||||||
|
pub use self::request::BackfillScopeKind;
|
||||||
|
/// Bounded Base58-shaped transaction signature text with exact RAW signature conversion.
|
||||||
|
pub use self::request::BackfillSignature;
|
||||||
|
/// Maximum number of transaction candidates admitted by one bounded Backfill Job.
|
||||||
|
pub use self::request::MAX_BACKFILL_CANDIDATES;
|
||||||
|
/// Maximum number of concurrent transaction hydrations admitted by one Backfill request.
|
||||||
|
pub use self::request::MAX_BACKFILL_HYDRATION_CONCURRENCY;
|
||||||
|
/// Maximum page size admitted for one `getSignaturesForAddress` request.
|
||||||
|
pub use self::request::MAX_BACKFILL_PAGE_SIZE;
|
||||||
|
/// Maximum number of `getSignaturesForAddress` pages admitted by one address Backfill request.
|
||||||
|
pub use self::request::MAX_BACKFILL_PAGES;
|
||||||
|
/// Maximum Base58 text length possible for one canonical 64-byte Solana signature.
|
||||||
|
pub use self::request::MAX_BACKFILL_SIGNATURE_TEXT_BYTES;
|
||||||
|
/// Minimum Base58 text length possible for one canonical 64-byte Solana signature.
|
||||||
|
pub use self::request::MIN_BACKFILL_SIGNATURE_TEXT_BYTES;
|
||||||
|
/// Stable Job kind code used by the concrete historical RAW transaction Backfill runtime.
|
||||||
|
pub use self::runtime::BACKFILL_JOB_KIND_CODE;
|
||||||
|
/// Cloneable external control handle for one concrete Backfill runtime.
|
||||||
|
pub use self::runtime::BackfillJobHandle;
|
||||||
|
/// Current concrete phase of one historical RAW transaction Backfill Job.
|
||||||
|
pub use self::runtime::BackfillJobPhase;
|
||||||
|
/// Concrete single-run Backfill coordinator paired with a cloneable control/snapshot handle.
|
||||||
|
pub use self::runtime::BackfillJobRuntime;
|
||||||
|
/// Complete safe latest-value snapshot of one concrete historical RAW transaction Backfill Job.
|
||||||
|
pub use self::runtime::BackfillJobSnapshot;
|
||||||
|
/// Cloneable runtime-neutral-facing latest-value source for concrete Backfill snapshots.
|
||||||
|
pub use self::runtime::BackfillSnapshotSource;
|
||||||
|
|
||||||
|
/// Internal contiguous completion frontier used by bounded execution.
|
||||||
|
pub(crate) use self::checkpoint::CompletionFrontier;
|
||||||
|
/// Builds one safe caller-owned checkpoint from the current contiguous frontier.
|
||||||
|
pub(crate) use self::checkpoint::checkpoint_from_frontier;
|
||||||
|
/// Resolves the candidate offset skipped by one validated replay checkpoint.
|
||||||
|
pub(crate) use self::checkpoint::execution_resume_offset;
|
||||||
|
/// Resolves the internal exclusive Before cursor for discovery resumption.
|
||||||
|
pub(crate) use self::checkpoint::resume_before_cursor;
|
||||||
|
/// Validates that one discovery result belongs to the current request identity.
|
||||||
|
pub(crate) use self::checkpoint::validate_discovery_identity;
|
||||||
|
/// Validates one caller-owned checkpoint against Job and semantic scope identity.
|
||||||
|
pub(crate) use self::checkpoint::validate_request_checkpoint;
|
||||||
|
/// Owning tracing target used by the concrete Backfill runtime.
|
||||||
|
pub(crate) use self::constants::TRACING_TARGET;
|
||||||
|
/// Exact private Base58 decoder shared by the public signature wrapper and hydration path.
|
||||||
|
pub(crate) use self::conversion::decode_backfill_signature;
|
||||||
|
/// Internal cancellable discovery path used by the concrete runtime.
|
||||||
|
pub(crate) use self::discovery::discover_backfill_candidates_cancellable;
|
||||||
|
/// Internal cooperative cancellation code used to distinguish cancellation from failure.
|
||||||
|
pub(crate) use self::error::ERROR_CODE_BACKFILL_CANCELLED;
|
||||||
|
/// Internal execution progress facts fed into the concrete latest-value source.
|
||||||
|
pub(crate) use self::execution::BackfillExecutionProgress;
|
||||||
|
/// Internal cancellable execution path used by the concrete runtime.
|
||||||
|
pub(crate) use self::execution::execute_backfill_discovery_cancellable;
|
||||||
|
/// Internal concrete cancellation signal shared by discovery and execution.
|
||||||
|
pub(crate) use self::runtime::BackfillCancellationSignal;
|
||||||
|
/// Internal execution progress publisher feeding the latest-value snapshot source.
|
||||||
|
pub(crate) use self::runtime::BackfillRuntimePublisher;
|
||||||
210
crates/ksp-job-backfill-lib/src/persistence.rs
Normal file
210
crates/ksp-job-backfill-lib/src/persistence.rs
Normal file
@@ -0,0 +1,210 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/persistence.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
/// Canonical entity disposition produced by one Backfill Store persistence attempt.
|
||||||
|
#[non_exhaustive]
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum BackfillEntityPersistence {
|
||||||
|
/// The canonical RAW transaction was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// Identical canonical RAW transaction content was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// A durable purge tombstone prevented normal Backfill rehydration.
|
||||||
|
SkippedPurged,
|
||||||
|
/// `getTransaction` returned JSON `null`, so no canonical Store write was attempted.
|
||||||
|
Missing,
|
||||||
|
/// Store reported divergent canonical content for the same network-scoped transaction identity.
|
||||||
|
Conflict,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Observation disposition produced by one Backfill Store persistence attempt.
|
||||||
|
#[non_exhaustive]
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum BackfillObservationPersistence {
|
||||||
|
/// The deterministic acquisition observation was inserted for the first time.
|
||||||
|
Inserted,
|
||||||
|
/// The same deterministic acquisition observation was already durable.
|
||||||
|
AlreadyPresent,
|
||||||
|
/// Store intentionally recorded no observation because normal persistence skipped a purged entity or detected a conflict.
|
||||||
|
NotRecorded,
|
||||||
|
/// No observation existed because hydration returned `Missing` before persistence.
|
||||||
|
NotApplicable,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Stable Backfill projection of one hydration persistence result.
|
||||||
|
///
|
||||||
|
/// The canonical transaction identity remains `(network, signature)`. Provider, endpoint and
|
||||||
|
/// transport details can affect the persisted observation but never the transaction identity.
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
pub struct BackfillPersistenceOutcome {
|
||||||
|
reference: ksp_store_lib::RawTransactionReference,
|
||||||
|
entity: crate::BackfillEntityPersistence,
|
||||||
|
observation: crate::BackfillObservationPersistence,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillPersistenceOutcome {
|
||||||
|
/// Creates one internally classified persistence result.
|
||||||
|
pub(crate) fn new(
|
||||||
|
reference: ksp_store_lib::RawTransactionReference,
|
||||||
|
entity: crate::BackfillEntityPersistence,
|
||||||
|
observation: crate::BackfillObservationPersistence,
|
||||||
|
) -> Self {
|
||||||
|
return Self { reference, entity, observation };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the network-scoped canonical transaction identity classified by this result.
|
||||||
|
#[must_use]
|
||||||
|
pub fn reference(&self) -> &ksp_store_lib::RawTransactionReference {
|
||||||
|
return &self.reference;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the canonical transaction persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entity(&self) -> crate::BackfillEntityPersistence {
|
||||||
|
return self.entity;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the acquisition-observation persistence disposition.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observation(&self) -> crate::BackfillObservationPersistence {
|
||||||
|
return self.observation;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Persists one hydrated Backfill result through the backend-neutral Store facade.
|
||||||
|
///
|
||||||
|
/// Available acquisitions use the existing atomic transaction-plus-observation Store contract in
|
||||||
|
/// `Normal` mode. A durable purge is therefore respected and never rehydrated implicitly.
|
||||||
|
/// `Missing` performs no Store write. Stable Store content conflicts are projected explicitly as
|
||||||
|
/// [`BackfillEntityPersistence::Conflict`] rather than being silently treated as idempotent skips.
|
||||||
|
/// Other Store failures remain errors.
|
||||||
|
pub async fn persist_backfill_hydration(
|
||||||
|
store: &ksp_store_lib::Store,
|
||||||
|
hydration: crate::BackfillHydrationOutcome,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillPersistenceOutcome> {
|
||||||
|
return persist_hydration_with_port(store, hydration).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
trait RawTransactionPersistencePort: std::marker::Send + std::marker::Sync {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool;
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>>;
|
||||||
|
}
|
||||||
|
|
||||||
|
impl RawTransactionPersistencePort for ksp_store_lib::Store {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
let snapshot = self.runtime_snapshot();
|
||||||
|
return snapshot.network() == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
return ksp_store_lib::RawTransactionWrite::persist_raw_transaction_acquisition(self, transaction, observation, mode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn persist_hydration_with_port<P>(port: &P, hydration: crate::BackfillHydrationOutcome) -> ksp_core_lib::Result<crate::BackfillPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: RawTransactionPersistencePort,
|
||||||
|
{
|
||||||
|
let reference = hydration.reference().clone();
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(persistence_error("store.network"));
|
||||||
|
}
|
||||||
|
return match hydration {
|
||||||
|
crate::BackfillHydrationOutcome::Missing(_) => std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(
|
||||||
|
reference,
|
||||||
|
crate::BackfillEntityPersistence::Missing,
|
||||||
|
crate::BackfillObservationPersistence::NotApplicable,
|
||||||
|
)),
|
||||||
|
crate::BackfillHydrationOutcome::Available(acquisition) => {
|
||||||
|
let (transaction, observation) = acquisition.into_parts();
|
||||||
|
persist_available_with_port(port, reference, transaction, observation).await
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn persist_available_with_port<P>(
|
||||||
|
port: &P,
|
||||||
|
reference: ksp_store_lib::RawTransactionReference,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillPersistenceOutcome>
|
||||||
|
where
|
||||||
|
P: RawTransactionPersistencePort,
|
||||||
|
{
|
||||||
|
if !port.network_matches(reference.network()) {
|
||||||
|
return std::result::Result::Err(persistence_error("store.network"));
|
||||||
|
}
|
||||||
|
if transaction.reference() != &reference || observation.transaction() != &reference {
|
||||||
|
return std::result::Result::Err(persistence_error("acquisition.reference"));
|
||||||
|
}
|
||||||
|
let result = port.persist_acquisition(transaction, observation, ksp_store_lib::RawTransactionAcquisitionMode::Normal).await;
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(outcome) => map_store_outcome(reference, outcome),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
if error.code() == ksp_store_lib::ERROR_CODE_RAW_CONFLICT {
|
||||||
|
return std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(
|
||||||
|
reference,
|
||||||
|
crate::BackfillEntityPersistence::Conflict,
|
||||||
|
crate::BackfillObservationPersistence::NotRecorded,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
std::result::Result::Err(error)
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn map_store_outcome(
|
||||||
|
reference: ksp_store_lib::RawTransactionReference,
|
||||||
|
outcome: ksp_store_lib::RawAcquisitionWriteOutcome,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillPersistenceOutcome> {
|
||||||
|
let entity = outcome.entity();
|
||||||
|
let observation = outcome.observation();
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::Inserted && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(
|
||||||
|
reference,
|
||||||
|
crate::BackfillEntityPersistence::Inserted,
|
||||||
|
crate::BackfillObservationPersistence::Inserted,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::Inserted {
|
||||||
|
return std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(
|
||||||
|
reference,
|
||||||
|
crate::BackfillEntityPersistence::AlreadyPresent,
|
||||||
|
crate::BackfillObservationPersistence::Inserted,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent && observation == ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent {
|
||||||
|
return std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(
|
||||||
|
reference,
|
||||||
|
crate::BackfillEntityPersistence::AlreadyPresent,
|
||||||
|
crate::BackfillObservationPersistence::AlreadyPresent,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if entity == ksp_store_lib::RawEntityWriteOutcome::SkippedPurged && observation == ksp_store_lib::RawObservationWriteOutcome::NotRecorded {
|
||||||
|
return std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(
|
||||||
|
reference,
|
||||||
|
crate::BackfillEntityPersistence::SkippedPurged,
|
||||||
|
crate::BackfillObservationPersistence::NotRecorded,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(persistence_error("store.outcome"));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persistence_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_PERSISTENCE_INVALID, "invalid Backfill Store persistence state").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/persistence.rs"]
|
||||||
|
mod tests;
|
||||||
494
crates/ksp-job-backfill-lib/src/request.rs
Normal file
494
crates/ksp-job-backfill-lib/src/request.rs
Normal file
@@ -0,0 +1,494 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/request.rs
|
||||||
|
// version: 5
|
||||||
|
|
||||||
|
use sha2::Digest; // rust-rules: trait-import
|
||||||
|
|
||||||
|
/// Maximum number of transaction candidates admitted by one bounded Backfill Job.
|
||||||
|
pub const MAX_BACKFILL_CANDIDATES: usize = 10_000;
|
||||||
|
/// Maximum number of concurrent transaction hydrations admitted by one Backfill request.
|
||||||
|
pub const MAX_BACKFILL_HYDRATION_CONCURRENCY: usize = 64;
|
||||||
|
/// Maximum number of `getSignaturesForAddress` pages admitted by one address Backfill request.
|
||||||
|
pub const MAX_BACKFILL_PAGES: usize = 10_000;
|
||||||
|
/// Maximum page size admitted for one `getSignaturesForAddress` request.
|
||||||
|
pub const MAX_BACKFILL_PAGE_SIZE: usize = 1_000;
|
||||||
|
/// Maximum Base58 text length possible for one canonical 64-byte Solana signature.
|
||||||
|
pub const MAX_BACKFILL_SIGNATURE_TEXT_BYTES: usize = 88;
|
||||||
|
/// Minimum Base58 text length possible for one canonical 64-byte Solana signature.
|
||||||
|
pub const MIN_BACKFILL_SIGNATURE_TEXT_BYTES: usize = 64;
|
||||||
|
|
||||||
|
/// Commitment levels intentionally admitted by the historical Backfill vertical.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum BackfillCommitment {
|
||||||
|
/// Read history at Solana `confirmed` commitment.
|
||||||
|
Confirmed,
|
||||||
|
/// Read history at Solana `finalized` commitment.
|
||||||
|
Finalized,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillCommitment {
|
||||||
|
/// Returns the stable Backfill commitment code.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn code(self) -> &'static str {
|
||||||
|
return match self {
|
||||||
|
Self::Confirmed => "confirmed",
|
||||||
|
Self::Finalized => "finalized",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Maps the narrowed Backfill commitment to the Transport-owned Solana commitment.
|
||||||
|
#[must_use]
|
||||||
|
pub(crate) const fn transport(self) -> ksp_onchain_transport_lib::SolanaCommitment {
|
||||||
|
return match self {
|
||||||
|
Self::Confirmed => ksp_onchain_transport_lib::SolanaCommitment::Confirmed,
|
||||||
|
Self::Finalized => ksp_onchain_transport_lib::SolanaCommitment::Finalized,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Bounded Base58-shaped transaction signature text used by discovery and RAW conversion.
|
||||||
|
///
|
||||||
|
/// Construction validates the encoded shape required by discovery. Exact conversion to the
|
||||||
|
/// Store-owned 64-byte signature is available through [`Self::to_raw_transaction_signature`].
|
||||||
|
#[derive(Clone, Eq, Hash, Ord, PartialEq, PartialOrd)]
|
||||||
|
pub struct BackfillSignature(std::string::String);
|
||||||
|
|
||||||
|
impl crate::BackfillSignature {
|
||||||
|
/// Creates one bounded Base58-shaped signature text.
|
||||||
|
pub fn new(value: impl std::convert::Into<std::string::String>) -> ksp_core_lib::Result<Self> {
|
||||||
|
let value = value.into();
|
||||||
|
if !valid_signature_text(value.as_str()) {
|
||||||
|
return std::result::Result::Err(signature_error());
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self(value));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated encoded signature text.
|
||||||
|
#[must_use]
|
||||||
|
pub fn as_str(&self) -> &str {
|
||||||
|
return self.0.as_str();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Decodes this Base58 text to the exact Store-owned 64-byte Solana signature.
|
||||||
|
pub fn to_raw_transaction_signature(&self) -> ksp_core_lib::Result<ksp_store_lib::RawTransactionSignature> {
|
||||||
|
return crate::decode_backfill_signature(self);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillSignature {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.write_str("BackfillSignature(..)");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Stable category of one bounded Backfill discovery scope.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
pub enum BackfillScopeKind {
|
||||||
|
/// Read the current newest address window.
|
||||||
|
LatestAddress,
|
||||||
|
/// Read address history older than one exclusive anchor.
|
||||||
|
BeforeAddress,
|
||||||
|
/// Read the bounded newer window closest to one exclusive anchor.
|
||||||
|
AfterAddress,
|
||||||
|
/// Hydrate an explicit bounded signature list without address discovery.
|
||||||
|
ExplicitSignatures,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillScopeKind {
|
||||||
|
/// Returns the stable scope code used by diagnostics and scope fingerprinting.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn code(self) -> &'static str {
|
||||||
|
return match self {
|
||||||
|
Self::LatestAddress => "latest_address",
|
||||||
|
Self::BeforeAddress => "before_address",
|
||||||
|
Self::AfterAddress => "after_address",
|
||||||
|
Self::ExplicitSignatures => "explicit_signatures",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
enum BackfillScopeValue {
|
||||||
|
LatestAddress { address: ksp_core_lib::Pubkey },
|
||||||
|
BeforeAddress { address: ksp_core_lib::Pubkey, anchor: crate::BackfillSignature },
|
||||||
|
AfterAddress { address: ksp_core_lib::Pubkey, anchor: crate::BackfillSignature },
|
||||||
|
ExplicitSignatures { signatures: std::vec::Vec<crate::BackfillSignature> },
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Validated bounded discovery scope for one historical Backfill Job.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct BackfillScope {
|
||||||
|
value: BackfillScopeValue,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillScope {
|
||||||
|
/// Creates a scope starting at the newest known history for one address.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn latest_address(address: ksp_core_lib::Pubkey) -> Self {
|
||||||
|
return Self { value: BackfillScopeValue::LatestAddress { address } };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates a scope reading history older than one exclusive address anchor.
|
||||||
|
#[must_use]
|
||||||
|
pub fn before_address(address: ksp_core_lib::Pubkey, anchor: crate::BackfillSignature) -> Self {
|
||||||
|
return Self { value: BackfillScopeValue::BeforeAddress { address, anchor } };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates a scope reading the bounded newer history closest to one exclusive address anchor.
|
||||||
|
#[must_use]
|
||||||
|
pub fn after_address(address: ksp_core_lib::Pubkey, anchor: crate::BackfillSignature) -> Self {
|
||||||
|
return Self { value: BackfillScopeValue::AfterAddress { address, anchor } };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Creates an explicit signature scope with stable first-occurrence deduplication.
|
||||||
|
pub fn explicit_signatures(signatures: std::vec::Vec<crate::BackfillSignature>) -> ksp_core_lib::Result<Self> {
|
||||||
|
if signatures.is_empty() || signatures.len() > crate::MAX_BACKFILL_CANDIDATES {
|
||||||
|
return std::result::Result::Err(request_error("scope.signatures"));
|
||||||
|
}
|
||||||
|
let mut unique = std::vec::Vec::with_capacity(signatures.len());
|
||||||
|
let mut seen = std::collections::HashSet::with_capacity(signatures.len());
|
||||||
|
for signature in signatures {
|
||||||
|
if seen.insert(signature.clone()) {
|
||||||
|
unique.push(signature);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if unique.is_empty() {
|
||||||
|
return std::result::Result::Err(request_error("scope.signatures"));
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(Self { value: BackfillScopeValue::ExplicitSignatures { signatures: unique } });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the stable category of this scope.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn kind(&self) -> crate::BackfillScopeKind {
|
||||||
|
return match &self.value {
|
||||||
|
BackfillScopeValue::LatestAddress { .. } => crate::BackfillScopeKind::LatestAddress,
|
||||||
|
BackfillScopeValue::BeforeAddress { .. } => crate::BackfillScopeKind::BeforeAddress,
|
||||||
|
BackfillScopeValue::AfterAddress { .. } => crate::BackfillScopeKind::AfterAddress,
|
||||||
|
BackfillScopeValue::ExplicitSignatures { .. } => crate::BackfillScopeKind::ExplicitSignatures,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the address used by address scopes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn address(&self) -> std::option::Option<&ksp_core_lib::Pubkey> {
|
||||||
|
return match &self.value {
|
||||||
|
BackfillScopeValue::LatestAddress { address }
|
||||||
|
| BackfillScopeValue::BeforeAddress { address, .. }
|
||||||
|
| BackfillScopeValue::AfterAddress { address, .. } => std::option::Option::Some(address),
|
||||||
|
BackfillScopeValue::ExplicitSignatures { .. } => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the exclusive anchor used by before/after scopes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn anchor(&self) -> std::option::Option<&crate::BackfillSignature> {
|
||||||
|
return match &self.value {
|
||||||
|
BackfillScopeValue::BeforeAddress { anchor, .. } | BackfillScopeValue::AfterAddress { anchor, .. } => std::option::Option::Some(anchor),
|
||||||
|
BackfillScopeValue::LatestAddress { .. } | BackfillScopeValue::ExplicitSignatures { .. } => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the stable deduplicated explicit signature list, when this is an explicit scope.
|
||||||
|
#[must_use]
|
||||||
|
pub fn signatures(&self) -> std::option::Option<&[crate::BackfillSignature]> {
|
||||||
|
return match &self.value {
|
||||||
|
BackfillScopeValue::ExplicitSignatures { signatures } => std::option::Option::Some(signatures.as_slice()),
|
||||||
|
BackfillScopeValue::LatestAddress { .. } | BackfillScopeValue::BeforeAddress { .. } | BackfillScopeValue::AfterAddress { .. } => {
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillScope {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let mut debug = formatter.debug_struct("BackfillScope");
|
||||||
|
debug.field("kind", &self.kind());
|
||||||
|
match &self.value {
|
||||||
|
BackfillScopeValue::LatestAddress { address } => {
|
||||||
|
debug.field("address", address);
|
||||||
|
},
|
||||||
|
BackfillScopeValue::BeforeAddress { address, .. } | BackfillScopeValue::AfterAddress { address, .. } => {
|
||||||
|
debug.field("address", address).field("anchor", &"<redacted>");
|
||||||
|
},
|
||||||
|
BackfillScopeValue::ExplicitSignatures { signatures } => {
|
||||||
|
debug.field("signature_count", &signatures.len());
|
||||||
|
},
|
||||||
|
}
|
||||||
|
return debug.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Opaque deterministic fingerprint of one semantic Backfill scope.
|
||||||
|
#[derive(Clone, Copy, Eq, Hash, PartialEq)]
|
||||||
|
pub struct BackfillScopeFingerprint([u8; 32]);
|
||||||
|
|
||||||
|
impl crate::BackfillScopeFingerprint {
|
||||||
|
const fn from_bytes(bytes: [u8; 32]) -> Self {
|
||||||
|
return Self(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the exact deterministic fingerprint bytes.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn as_bytes(&self) -> &[u8; 32] {
|
||||||
|
return &self.0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillScopeFingerprint {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter.write_str("BackfillScopeFingerprint(..)");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Fully explicit bounded request for one historical transaction Backfill Job.
|
||||||
|
#[derive(Clone, Eq, PartialEq)]
|
||||||
|
pub struct BackfillRequest {
|
||||||
|
job_id: ksp_job_api::JobId,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
commitment: crate::BackfillCommitment,
|
||||||
|
scope: crate::BackfillScope,
|
||||||
|
page_size: usize,
|
||||||
|
max_pages: usize,
|
||||||
|
max_candidates: usize,
|
||||||
|
hydration_concurrency: usize,
|
||||||
|
min_context_slot: std::option::Option<u64>,
|
||||||
|
scope_fingerprint: crate::BackfillScopeFingerprint,
|
||||||
|
checkpoint: std::option::Option<crate::BackfillCheckpoint>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillRequest {
|
||||||
|
/// Creates and validates one fully explicit bounded Backfill request.
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
|
pub fn new(
|
||||||
|
job_id: ksp_job_api::JobId,
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
role: ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
commitment: crate::BackfillCommitment,
|
||||||
|
scope: crate::BackfillScope,
|
||||||
|
page_size: usize,
|
||||||
|
max_pages: usize,
|
||||||
|
max_candidates: usize,
|
||||||
|
hydration_concurrency: usize,
|
||||||
|
min_context_slot: std::option::Option<u64>,
|
||||||
|
) -> ksp_core_lib::Result<Self> {
|
||||||
|
if page_size == 0 || page_size > crate::MAX_BACKFILL_PAGE_SIZE {
|
||||||
|
return std::result::Result::Err(request_error("page_size"));
|
||||||
|
}
|
||||||
|
if max_pages == 0 || max_pages > crate::MAX_BACKFILL_PAGES {
|
||||||
|
return std::result::Result::Err(request_error("max_pages"));
|
||||||
|
}
|
||||||
|
if max_candidates == 0 || max_candidates > crate::MAX_BACKFILL_CANDIDATES {
|
||||||
|
return std::result::Result::Err(request_error("max_candidates"));
|
||||||
|
}
|
||||||
|
if hydration_concurrency == 0 || hydration_concurrency > crate::MAX_BACKFILL_HYDRATION_CONCURRENCY {
|
||||||
|
return std::result::Result::Err(request_error("hydration_concurrency"));
|
||||||
|
}
|
||||||
|
if role.as_str().is_empty() || role.as_str().trim() != role.as_str() {
|
||||||
|
return std::result::Result::Err(request_error("role"));
|
||||||
|
}
|
||||||
|
if scope.kind() == crate::BackfillScopeKind::ExplicitSignatures {
|
||||||
|
if min_context_slot.is_some() {
|
||||||
|
return std::result::Result::Err(request_error("min_context_slot"));
|
||||||
|
}
|
||||||
|
let signature_count = match scope.signatures() {
|
||||||
|
std::option::Option::Some(signatures) => signatures.len(),
|
||||||
|
std::option::Option::None => return std::result::Result::Err(request_error("scope.signatures")),
|
||||||
|
};
|
||||||
|
if signature_count > max_candidates {
|
||||||
|
return std::result::Result::Err(request_error("max_candidates"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let scope_fingerprint = fingerprint_scope(&network, commitment, &scope, page_size, max_pages, max_candidates, min_context_slot);
|
||||||
|
return std::result::Result::Ok(Self {
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
role,
|
||||||
|
commitment,
|
||||||
|
scope,
|
||||||
|
page_size,
|
||||||
|
max_pages,
|
||||||
|
max_candidates,
|
||||||
|
hydration_concurrency,
|
||||||
|
min_context_slot,
|
||||||
|
scope_fingerprint,
|
||||||
|
checkpoint: std::option::Option::None,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the caller-owned logical Job identity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn job_id(&self) -> &ksp_job_api::JobId {
|
||||||
|
return &self.job_id;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the Store-scoped logical network identity for every candidate in this Job.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn network(&self) -> &ksp_store_lib::RawNetworkId {
|
||||||
|
return &self.network;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the logical HTTP role used only for Transport selection.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn role(&self) -> &ksp_onchain_transport_lib::HttpRoleName {
|
||||||
|
return &self.role;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the narrowed commitment used by discovery and hydration.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn commitment(&self) -> crate::BackfillCommitment {
|
||||||
|
return self.commitment;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated discovery scope.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn scope(&self) -> &crate::BackfillScope {
|
||||||
|
return &self.scope;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the per-request signature page size.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn page_size(&self) -> usize {
|
||||||
|
return self.page_size;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the maximum number of address pages admitted by this Job.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn max_pages(&self) -> usize {
|
||||||
|
return self.max_pages;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the maximum number of transaction candidates admitted by this Job.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn max_candidates(&self) -> usize {
|
||||||
|
return self.max_candidates;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the future hydration concurrency bound.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn hydration_concurrency(&self) -> usize {
|
||||||
|
return self.hydration_concurrency;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the optional Solana minimum context slot used by address discovery.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn min_context_slot(&self) -> std::option::Option<u64> {
|
||||||
|
return self.min_context_slot;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the deterministic semantic scope fingerprint.
|
||||||
|
///
|
||||||
|
/// Transport role, provider, endpoint and protocol are deliberately excluded. They describe
|
||||||
|
/// acquisition provenance, not transaction or scope identity.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn scope_fingerprint(&self) -> crate::BackfillScopeFingerprint {
|
||||||
|
return self.scope_fingerprint;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Attaches one caller-owned checkpoint after validating Job and semantic scope identity.
|
||||||
|
pub fn with_checkpoint(mut self, checkpoint: crate::BackfillCheckpoint) -> ksp_core_lib::Result<Self> {
|
||||||
|
let validation = crate::validate_request_checkpoint(&self.job_id, self.scope_fingerprint, self.scope.kind(), &checkpoint);
|
||||||
|
if let std::result::Result::Err(error) = validation {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
self.checkpoint = std::option::Option::Some(checkpoint);
|
||||||
|
return std::result::Result::Ok(self);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the validated optional checkpoint supplied for controlled resumption.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn checkpoint(&self) -> std::option::Option<&crate::BackfillCheckpoint> {
|
||||||
|
return self.checkpoint.as_ref();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillRequest {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("BackfillRequest")
|
||||||
|
.field("job_id", &self.job_id)
|
||||||
|
.field("network", &self.network)
|
||||||
|
.field("role", &self.role)
|
||||||
|
.field("commitment", &self.commitment)
|
||||||
|
.field("scope", &self.scope)
|
||||||
|
.field("page_size", &self.page_size)
|
||||||
|
.field("max_pages", &self.max_pages)
|
||||||
|
.field("max_candidates", &self.max_candidates)
|
||||||
|
.field("hydration_concurrency", &self.hydration_concurrency)
|
||||||
|
.field("min_context_slot", &self.min_context_slot)
|
||||||
|
.field("scope_fingerprint", &self.scope_fingerprint)
|
||||||
|
.field("has_checkpoint", &self.checkpoint.is_some())
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fingerprint_scope(
|
||||||
|
network: &ksp_store_lib::RawNetworkId,
|
||||||
|
commitment: crate::BackfillCommitment,
|
||||||
|
scope: &crate::BackfillScope,
|
||||||
|
page_size: usize,
|
||||||
|
max_pages: usize,
|
||||||
|
max_candidates: usize,
|
||||||
|
min_context_slot: std::option::Option<u64>,
|
||||||
|
) -> crate::BackfillScopeFingerprint {
|
||||||
|
let mut hasher = sha2::Sha256::new();
|
||||||
|
hasher.update(b"ksp.job.backfill.scope.v1\0");
|
||||||
|
hash_bytes(&mut hasher, network.as_str().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, commitment.code().as_bytes());
|
||||||
|
hash_bytes(&mut hasher, scope.kind().code().as_bytes());
|
||||||
|
if let std::option::Option::Some(address) = scope.address() {
|
||||||
|
hash_bytes(&mut hasher, address.as_ref());
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(anchor) = scope.anchor() {
|
||||||
|
hash_bytes(&mut hasher, anchor.as_str().as_bytes());
|
||||||
|
}
|
||||||
|
if let std::option::Option::Some(signatures) = scope.signatures() {
|
||||||
|
for signature in signatures {
|
||||||
|
hash_bytes(&mut hasher, signature.as_str().as_bytes());
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hash_u64(&mut hasher, page_size as u64);
|
||||||
|
hash_u64(&mut hasher, max_pages as u64);
|
||||||
|
hash_u64(&mut hasher, max_candidates as u64);
|
||||||
|
match min_context_slot {
|
||||||
|
std::option::Option::Some(slot) => {
|
||||||
|
hasher.update([1_u8]);
|
||||||
|
hash_u64(&mut hasher, slot);
|
||||||
|
},
|
||||||
|
std::option::Option::None => hasher.update([0_u8]),
|
||||||
|
}
|
||||||
|
let bytes: [u8; 32] = hasher.finalize().into();
|
||||||
|
return crate::BackfillScopeFingerprint::from_bytes(bytes);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_bytes(hasher: &mut sha2::Sha256, value: &[u8]) {
|
||||||
|
hash_u64(hasher, value.len() as u64);
|
||||||
|
hasher.update(value);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn hash_u64(hasher: &mut sha2::Sha256, value: u64) {
|
||||||
|
hasher.update(value.to_be_bytes());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn request_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_REQUEST_INVALID, "invalid bounded Backfill request").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn signature_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_SIGNATURE_INVALID, "invalid bounded Backfill transaction signature")
|
||||||
|
.with_context("field", "signature");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn valid_signature_text(value: &str) -> bool {
|
||||||
|
if value.len() < crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES || value.len() > crate::MAX_BACKFILL_SIGNATURE_TEXT_BYTES {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return value.bytes().all(|byte| {
|
||||||
|
return matches!(byte, b'1'..=b'9' | b'A'..=b'H' | b'J'..=b'N' | b'P'..=b'Z' | b'a'..=b'k' | b'm'..=b'z');
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/request.rs"]
|
||||||
|
mod tests;
|
||||||
715
crates/ksp-job-backfill-lib/src/runtime.rs
Normal file
715
crates/ksp-job-backfill-lib/src/runtime.rs
Normal file
@@ -0,0 +1,715 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/src/runtime.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
/// Stable Job kind code used by the concrete historical RAW transaction Backfill runtime.
|
||||||
|
pub const BACKFILL_JOB_KIND_CODE: &str = "solana.raw_transaction.backfill";
|
||||||
|
|
||||||
|
const CONTROL_ACTIVE: u8 = 0;
|
||||||
|
const CONTROL_CANCELLATION_REQUESTED: u8 = 1;
|
||||||
|
const CONTROL_CANCELLED: u8 = 3;
|
||||||
|
const CONTROL_COMPLETED: u8 = 2;
|
||||||
|
const CONTROL_FAILED: u8 = 4;
|
||||||
|
|
||||||
|
/// Current concrete phase of one historical RAW transaction Backfill Job.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
|
||||||
|
#[non_exhaustive]
|
||||||
|
pub enum BackfillJobPhase {
|
||||||
|
/// Runtime was created but execution has not started.
|
||||||
|
Created,
|
||||||
|
/// Candidate discovery is in progress.
|
||||||
|
Discovering,
|
||||||
|
/// Candidate hydration and persistence are being admitted and completed.
|
||||||
|
Executing,
|
||||||
|
/// Cancellation or a fatal result stopped admission while already submitted work is draining.
|
||||||
|
Draining,
|
||||||
|
/// No more work can be admitted and the Job is terminal.
|
||||||
|
Finished,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillJobPhase {
|
||||||
|
/// Returns the stable safe code for this concrete runtime phase.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn code(self) -> &'static str {
|
||||||
|
return match self {
|
||||||
|
Self::Created => "created",
|
||||||
|
Self::Discovering => "discovering",
|
||||||
|
Self::Executing => "executing",
|
||||||
|
Self::Draining => "draining",
|
||||||
|
Self::Finished => "finished",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Complete safe latest-value snapshot of one concrete historical RAW transaction Backfill Job.
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
pub struct BackfillJobSnapshot {
|
||||||
|
phase: crate::BackfillJobPhase,
|
||||||
|
scope_kind: crate::BackfillScopeKind,
|
||||||
|
discovery_boundary: std::option::Option<crate::BackfillDiscoveryBoundary>,
|
||||||
|
candidates_selected: usize,
|
||||||
|
candidates_admitted: usize,
|
||||||
|
candidates_finished: usize,
|
||||||
|
entities_inserted: usize,
|
||||||
|
entities_existing: usize,
|
||||||
|
entities_purged: usize,
|
||||||
|
missing: usize,
|
||||||
|
conflicts: usize,
|
||||||
|
observations_inserted: usize,
|
||||||
|
observations_existing: usize,
|
||||||
|
cancelled_candidates: usize,
|
||||||
|
holes: usize,
|
||||||
|
maximum_in_flight: usize,
|
||||||
|
contiguous_completed: usize,
|
||||||
|
checkpoint: std::option::Option<crate::BackfillCheckpoint>,
|
||||||
|
failure_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillJobSnapshot {
|
||||||
|
fn initial(request: &crate::BackfillRequest) -> Self {
|
||||||
|
return Self {
|
||||||
|
phase: crate::BackfillJobPhase::Created,
|
||||||
|
scope_kind: request.scope().kind(),
|
||||||
|
discovery_boundary: std::option::Option::None,
|
||||||
|
candidates_selected: 0,
|
||||||
|
candidates_admitted: 0,
|
||||||
|
candidates_finished: 0,
|
||||||
|
entities_inserted: 0,
|
||||||
|
entities_existing: 0,
|
||||||
|
entities_purged: 0,
|
||||||
|
missing: 0,
|
||||||
|
conflicts: 0,
|
||||||
|
observations_inserted: 0,
|
||||||
|
observations_existing: 0,
|
||||||
|
cancelled_candidates: 0,
|
||||||
|
holes: 0,
|
||||||
|
maximum_in_flight: 0,
|
||||||
|
contiguous_completed: request.checkpoint().map_or(0, crate::BackfillCheckpoint::completed_prefix),
|
||||||
|
checkpoint: request.checkpoint().cloned(),
|
||||||
|
failure_code: std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the concrete execution phase represented by this snapshot.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn phase(&self) -> crate::BackfillJobPhase {
|
||||||
|
return self.phase;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the safe scope category without rendering address or signature payloads.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn scope_kind(&self) -> crate::BackfillScopeKind {
|
||||||
|
return self.scope_kind;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the discovery boundary once candidate selection has completed.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn discovery_boundary(&self) -> std::option::Option<crate::BackfillDiscoveryBoundary> {
|
||||||
|
return self.discovery_boundary;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the bounded candidate count selected by discovery.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn candidates_selected(&self) -> usize {
|
||||||
|
return self.candidates_selected;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of candidates admitted into hydration.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn candidates_admitted(&self) -> usize {
|
||||||
|
return self.candidates_admitted;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted candidates that reached a known coordinator result.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn candidates_finished(&self) -> usize {
|
||||||
|
return self.candidates_finished;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted canonical RAW transactions.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entities_inserted(&self) -> usize {
|
||||||
|
return self.entities_inserted;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of identical canonical RAW transactions already present.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entities_existing(&self) -> usize {
|
||||||
|
return self.entities_existing;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of durable purge tombstones respected by normal Backfill persistence.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn entities_purged(&self) -> usize {
|
||||||
|
return self.entities_purged;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of `getTransaction = null` candidates.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn missing(&self) -> usize {
|
||||||
|
return self.missing;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of Store content conflicts observed by this run.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn conflicts(&self) -> usize {
|
||||||
|
return self.conflicts;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of newly inserted acquisition observations.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observations_inserted(&self) -> usize {
|
||||||
|
return self.observations_inserted;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of acquisition observations already durable.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn observations_existing(&self) -> usize {
|
||||||
|
return self.observations_existing;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of admitted candidates cancelled before Store submission.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn cancelled_candidates(&self) -> usize {
|
||||||
|
return self.cancelled_candidates;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the number of known candidate outcomes blocking the contiguous frontier.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn holes(&self) -> usize {
|
||||||
|
return self.holes;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the greatest observed concurrent candidate count.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn maximum_in_flight(&self) -> usize {
|
||||||
|
return self.maximum_in_flight;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the cumulative safe contiguous completion prefix represented by the current checkpoint.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn contiguous_completed(&self) -> usize {
|
||||||
|
return self.contiguous_completed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the latest caller-owned safe checkpoint, when one exists.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn checkpoint(&self) -> std::option::Option<&crate::BackfillCheckpoint> {
|
||||||
|
return self.checkpoint.as_ref();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the stable fatal error code retained by a failed Job, when one exists.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn failure_code(&self) -> std::option::Option<ksp_core_lib::ErrorCode> {
|
||||||
|
return self.failure_code;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cloneable runtime-neutral-facing latest-value source for concrete Backfill snapshots.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct BackfillSnapshotSource {
|
||||||
|
receiver: tokio::sync::watch::Receiver<ksp_job_api::JobNotification<crate::BackfillJobSnapshot>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ksp_job_api::JobSnapshotSource for crate::BackfillSnapshotSource {
|
||||||
|
type Snapshot = crate::BackfillJobSnapshot;
|
||||||
|
|
||||||
|
fn current(&self) -> ksp_job_api::JobNotification<Self::Snapshot> {
|
||||||
|
return self.receiver.borrow().clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn wait_for_change(&self, observed: ksp_job_api::JobNotificationSequence) -> ksp_job_api::JobSnapshotFuture<'_, Self::Snapshot> {
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
loop {
|
||||||
|
let current = receiver.borrow().clone();
|
||||||
|
if current.sequence().is_after(observed) || current.state().is_terminal() {
|
||||||
|
return current;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return receiver.borrow().clone();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillSnapshotSource {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
let current = self.receiver.borrow();
|
||||||
|
return formatter.debug_struct("BackfillSnapshotSource").field("sequence", ¤t.sequence()).field("state", ¤t.state()).finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Cloneable external control handle for one concrete Backfill runtime.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub struct BackfillJobHandle {
|
||||||
|
control: BackfillRuntimeControl,
|
||||||
|
snapshots: crate::BackfillSnapshotSource,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillJobHandle {
|
||||||
|
/// Requests cooperative cancellation and returns `true` only when accepted before terminal publication.
|
||||||
|
#[must_use]
|
||||||
|
pub fn cancel(&self) -> bool {
|
||||||
|
return self.control.request_cancellation();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns an independent latest-value snapshot source for one listener.
|
||||||
|
#[must_use]
|
||||||
|
pub fn snapshots(&self) -> crate::BackfillSnapshotSource {
|
||||||
|
return self.snapshots.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether cooperative cancellation has been accepted for this non-terminal Job.
|
||||||
|
#[must_use]
|
||||||
|
pub fn is_cancellation_requested(&self) -> bool {
|
||||||
|
return self.control.is_cancellation_requested();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl std::fmt::Debug for crate::BackfillJobHandle {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("BackfillJobHandle")
|
||||||
|
.field("cancellation_requested", &self.is_cancellation_requested())
|
||||||
|
.field("snapshot", &<crate::BackfillSnapshotSource as ksp_job_api::JobSnapshotSource>::current(&self.snapshots))
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Concrete single-run Backfill coordinator paired with a cloneable control/snapshot handle.
|
||||||
|
pub struct BackfillJobRuntime {
|
||||||
|
request: crate::BackfillRequest,
|
||||||
|
control: BackfillRuntimeControl,
|
||||||
|
cancellation: crate::BackfillCancellationSignal,
|
||||||
|
publisher: crate::BackfillRuntimePublisher,
|
||||||
|
handle: crate::BackfillJobHandle,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillJobRuntime {
|
||||||
|
/// Creates one runtime in `Created` state and its stable latest-value channel.
|
||||||
|
pub fn new(request: crate::BackfillRequest) -> ksp_core_lib::Result<Self> {
|
||||||
|
let kind = match ksp_job_api::JobKindCode::new(crate::BACKFILL_JOB_KIND_CODE) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let initial_snapshot = crate::BackfillJobSnapshot::initial(&request);
|
||||||
|
let initial = ksp_job_api::JobNotification::new(
|
||||||
|
request.job_id().clone(),
|
||||||
|
kind.clone(),
|
||||||
|
ksp_job_api::JobNotificationSequence::initial(),
|
||||||
|
ksp_job_api::JobState::Created,
|
||||||
|
initial_snapshot,
|
||||||
|
);
|
||||||
|
let (snapshot_sender, snapshot_receiver) = tokio::sync::watch::channel(initial);
|
||||||
|
let (cancel_sender, cancel_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let control = BackfillRuntimeControl::new(cancel_sender);
|
||||||
|
let cancellation = crate::BackfillCancellationSignal::new(control.token(), cancel_receiver);
|
||||||
|
let snapshots = crate::BackfillSnapshotSource { receiver: snapshot_receiver };
|
||||||
|
let handle = crate::BackfillJobHandle { control: control.clone(), snapshots: snapshots.clone() };
|
||||||
|
let publisher = crate::BackfillRuntimePublisher { sender: snapshot_sender, id: request.job_id().clone(), kind };
|
||||||
|
return std::result::Result::Ok(Self { request, control, cancellation, publisher, handle });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns a cloneable control and latest-value observation handle before the runtime is moved into execution.
|
||||||
|
#[must_use]
|
||||||
|
pub fn handle(&self) -> crate::BackfillJobHandle {
|
||||||
|
return self.handle.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs discovery, bounded execution, cooperative cancellation and terminal snapshot publication.
|
||||||
|
pub async fn run(
|
||||||
|
self,
|
||||||
|
transport: &ksp_onchain_transport_lib::HttpTransportPool,
|
||||||
|
store: &ksp_store_lib::Store,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
if self.cancellation.is_requested() {
|
||||||
|
let claimed = self.control.claim_normal_terminal();
|
||||||
|
if claimed != TerminalClaim::Cancelled {
|
||||||
|
return std::result::Result::Err(runtime_error("terminal.pre_start"));
|
||||||
|
}
|
||||||
|
return self.publisher.publish_cancelled_from_created();
|
||||||
|
}
|
||||||
|
let started = self.publisher.publish_running(crate::BackfillJobPhase::Discovering);
|
||||||
|
if let std::result::Result::Err(error) = started {
|
||||||
|
self.control.claim_failed();
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let discovery = crate::discover_backfill_candidates_cancellable(transport, &self.request, &self.cancellation).await;
|
||||||
|
let discovery = match discovery {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
if error.code() == crate::ERROR_CODE_BACKFILL_CANCELLED {
|
||||||
|
return self.finish_cancelled();
|
||||||
|
}
|
||||||
|
self.control.claim_failed();
|
||||||
|
let published = self.publisher.publish_failed(error.code());
|
||||||
|
if let std::result::Result::Err(publish_error) = published {
|
||||||
|
return std::result::Result::Err(publish_error);
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
let published = self.publisher.publish_discovery(&discovery);
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
self.control.claim_failed();
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
let batch = crate::execute_backfill_discovery_cancellable(transport, store, &self.request, &discovery, &self.cancellation, &self.publisher).await;
|
||||||
|
let batch = match batch {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
self.control.claim_failed();
|
||||||
|
let published = self.publisher.publish_failed(error.code());
|
||||||
|
if let std::result::Result::Err(publish_error) = published {
|
||||||
|
return std::result::Result::Err(publish_error);
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
},
|
||||||
|
};
|
||||||
|
if let std::option::Option::Some(code) = batch.failure_code() {
|
||||||
|
self.control.claim_failed();
|
||||||
|
let published = self.publisher.publish_batch_terminal(&batch, ksp_job_api::JobState::Failed, std::option::Option::Some(code));
|
||||||
|
if let std::result::Result::Err(error) = published {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(code, "Backfill execution reached a fatal candidate result"));
|
||||||
|
}
|
||||||
|
let terminal = self.control.claim_normal_terminal();
|
||||||
|
return match terminal {
|
||||||
|
TerminalClaim::Cancelled => {
|
||||||
|
let cancelling = self.publisher.publish_batch_cancelling(&batch);
|
||||||
|
if let std::result::Result::Err(error) = cancelling {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
self.publisher.publish_batch_terminal(&batch, ksp_job_api::JobState::Cancelled, std::option::Option::None)
|
||||||
|
},
|
||||||
|
TerminalClaim::Completed => {
|
||||||
|
let completion = if batch.is_partial() { ksp_job_api::JobCompletion::Partial } else { ksp_job_api::JobCompletion::Complete };
|
||||||
|
self.publisher.publish_batch_terminal(&batch, ksp_job_api::JobState::Completed(completion), std::option::Option::None)
|
||||||
|
},
|
||||||
|
TerminalClaim::Failed => std::result::Result::Err(runtime_error("terminal.failed")),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn finish_cancelled(&self) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
let terminal = self.control.claim_normal_terminal();
|
||||||
|
if terminal != TerminalClaim::Cancelled {
|
||||||
|
return std::result::Result::Err(runtime_error("terminal.cancelled"));
|
||||||
|
}
|
||||||
|
let cancelling = self.publisher.publish_cancelling();
|
||||||
|
if let std::result::Result::Err(error) = cancelling {
|
||||||
|
return std::result::Result::Err(error);
|
||||||
|
}
|
||||||
|
return self.publisher.publish_terminal(ksp_job_api::JobState::Cancelled, std::option::Option::None);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal atomic terminal/cancellation arbitration shared by runtime and external handle.
|
||||||
|
#[derive(Clone)]
|
||||||
|
struct BackfillRuntimeControl {
|
||||||
|
state: std::sync::Arc<std::sync::atomic::AtomicU8>,
|
||||||
|
token: ksp_job_api::JobCancellationToken,
|
||||||
|
cancel_sender: tokio::sync::watch::Sender<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BackfillRuntimeControl {
|
||||||
|
/// Creates one active control state paired with the cancellation wake channel.
|
||||||
|
fn new(cancel_sender: tokio::sync::watch::Sender<bool>) -> Self {
|
||||||
|
return Self {
|
||||||
|
state: std::sync::Arc::new(std::sync::atomic::AtomicU8::new(CONTROL_ACTIVE)),
|
||||||
|
token: ksp_job_api::JobCancellationToken::new(),
|
||||||
|
cancel_sender,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the runtime-neutral cancellation token mirrored by this control.
|
||||||
|
fn token(&self) -> ksp_job_api::JobCancellationToken {
|
||||||
|
return self.token.clone();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Atomically accepts the first pre-terminal cancellation request.
|
||||||
|
fn request_cancellation(&self) -> bool {
|
||||||
|
let accepted = self
|
||||||
|
.state
|
||||||
|
.compare_exchange(CONTROL_ACTIVE, CONTROL_CANCELLATION_REQUESTED, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire)
|
||||||
|
.is_ok();
|
||||||
|
if accepted && self.token.cancel() {
|
||||||
|
let _previous = self.cancel_sender.send_replace(true);
|
||||||
|
}
|
||||||
|
return accepted;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether cooperative cancellation was accepted.
|
||||||
|
fn is_cancellation_requested(&self) -> bool {
|
||||||
|
return self.token.is_cancellation_requested();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Atomically resolves the completion-versus-cancellation terminal race.
|
||||||
|
fn claim_normal_terminal(&self) -> TerminalClaim {
|
||||||
|
let completed =
|
||||||
|
self.state.compare_exchange(CONTROL_ACTIVE, CONTROL_COMPLETED, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire);
|
||||||
|
if completed.is_ok() {
|
||||||
|
return TerminalClaim::Completed;
|
||||||
|
}
|
||||||
|
let cancelled = self.state.compare_exchange(
|
||||||
|
CONTROL_CANCELLATION_REQUESTED,
|
||||||
|
CONTROL_CANCELLED,
|
||||||
|
std::sync::atomic::Ordering::AcqRel,
|
||||||
|
std::sync::atomic::Ordering::Acquire,
|
||||||
|
);
|
||||||
|
if cancelled.is_ok() {
|
||||||
|
return TerminalClaim::Cancelled;
|
||||||
|
}
|
||||||
|
return TerminalClaim::Failed;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Marks a non-terminal control as failed, overriding a pending cancellation request.
|
||||||
|
fn claim_failed(&self) {
|
||||||
|
loop {
|
||||||
|
let state = self.state.load(std::sync::atomic::Ordering::Acquire);
|
||||||
|
if matches!(state, CONTROL_COMPLETED | CONTROL_CANCELLED | CONTROL_FAILED) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let changed = self.state.compare_exchange(state, CONTROL_FAILED, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire);
|
||||||
|
if changed.is_ok() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal wakeable cancellation signal used only around abandonable pre-Store futures.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub(crate) struct BackfillCancellationSignal {
|
||||||
|
token: ksp_job_api::JobCancellationToken,
|
||||||
|
receiver: tokio::sync::watch::Receiver<bool>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillCancellationSignal {
|
||||||
|
/// Creates one signal from the runtime-neutral token and Tokio wake receiver.
|
||||||
|
pub(crate) fn new(token: ksp_job_api::JobCancellationToken, receiver: tokio::sync::watch::Receiver<bool>) -> Self {
|
||||||
|
return Self { token, receiver };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns whether cancellation has been requested.
|
||||||
|
pub(crate) fn is_requested(&self) -> bool {
|
||||||
|
return self.token.is_cancellation_requested();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs one abandonable operation until it completes or cancellation wins.
|
||||||
|
pub(crate) async fn run_cancellable<F, T>(&self, operation: F) -> ksp_core_lib::Result<T>
|
||||||
|
where
|
||||||
|
F: std::future::Future<Output = ksp_core_lib::Result<T>>,
|
||||||
|
{
|
||||||
|
if self.is_requested() {
|
||||||
|
return std::result::Result::Err(cancelled_error());
|
||||||
|
}
|
||||||
|
let mut receiver = self.receiver.clone();
|
||||||
|
return tokio::select! {
|
||||||
|
biased;
|
||||||
|
_ = wait_for_cancellation(&self.token, &mut receiver) => std::result::Result::Err(cancelled_error()),
|
||||||
|
result = operation => result,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn wait_for_cancellation(token: &ksp_job_api::JobCancellationToken, receiver: &mut tokio::sync::watch::Receiver<bool>) {
|
||||||
|
loop {
|
||||||
|
if token.is_cancellation_requested() || *receiver.borrow() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let changed = receiver.changed().await;
|
||||||
|
if changed.is_err() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal result of atomically claiming a normal terminal state.
|
||||||
|
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||||
|
enum TerminalClaim {
|
||||||
|
Completed,
|
||||||
|
Cancelled,
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Internal latest-value publisher owning the concrete Backfill notification stream.
|
||||||
|
#[derive(Clone)]
|
||||||
|
pub(crate) struct BackfillRuntimePublisher {
|
||||||
|
sender: tokio::sync::watch::Sender<ksp_job_api::JobNotification<crate::BackfillJobSnapshot>>,
|
||||||
|
id: ksp_job_api::JobId,
|
||||||
|
kind: ksp_job_api::JobKindCode,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl crate::BackfillRuntimePublisher {
|
||||||
|
/// Publishes one non-terminal running phase.
|
||||||
|
pub(crate) fn publish_running(&self, phase: crate::BackfillJobPhase) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(ksp_job_api::JobState::Running, |snapshot| {
|
||||||
|
snapshot.phase = phase;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes the complete bounded discovery result as the current execution snapshot.
|
||||||
|
pub(crate) fn publish_discovery(&self, discovery: &crate::BackfillDiscovery) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(ksp_job_api::JobState::Running, |snapshot| {
|
||||||
|
snapshot.phase = crate::BackfillJobPhase::Executing;
|
||||||
|
snapshot.discovery_boundary = std::option::Option::Some(discovery.boundary());
|
||||||
|
snapshot.candidates_selected = discovery.candidates().len();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one coalescable execution progress value.
|
||||||
|
pub(crate) fn publish_execution_progress(
|
||||||
|
&self,
|
||||||
|
progress: &crate::BackfillExecutionProgress,
|
||||||
|
cancelling: bool,
|
||||||
|
draining: bool,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
let state = if cancelling { ksp_job_api::JobState::Cancelling } else { ksp_job_api::JobState::Running };
|
||||||
|
return self.publish_with(state, |snapshot| {
|
||||||
|
snapshot.phase = if draining { crate::BackfillJobPhase::Draining } else { crate::BackfillJobPhase::Executing };
|
||||||
|
apply_progress(snapshot, progress);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes cancellation observation before terminal cancellation.
|
||||||
|
pub(crate) fn publish_cancelling(&self) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(ksp_job_api::JobState::Cancelling, |snapshot| {
|
||||||
|
snapshot.phase = crate::BackfillJobPhase::Draining;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes the drained batch state while cancellation is terminalizing.
|
||||||
|
pub(crate) fn publish_batch_cancelling(&self, batch: &crate::BackfillExecutionBatch) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(ksp_job_api::JobState::Cancelling, |snapshot| {
|
||||||
|
snapshot.phase = crate::BackfillJobPhase::Draining;
|
||||||
|
apply_batch(snapshot, batch);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes one terminal state together with the fully drained execution batch.
|
||||||
|
pub(crate) fn publish_batch_terminal(
|
||||||
|
&self,
|
||||||
|
batch: &crate::BackfillExecutionBatch,
|
||||||
|
state: ksp_job_api::JobState,
|
||||||
|
failure_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(state, |snapshot| {
|
||||||
|
snapshot.phase = crate::BackfillJobPhase::Finished;
|
||||||
|
snapshot.failure_code = failure_code;
|
||||||
|
apply_batch(snapshot, batch);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes a terminal failure before a batch exists.
|
||||||
|
pub(crate) fn publish_failed(&self, code: ksp_core_lib::ErrorCode) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_terminal(ksp_job_api::JobState::Failed, std::option::Option::Some(code));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes a terminal state without a completed execution batch.
|
||||||
|
pub(crate) fn publish_terminal(
|
||||||
|
&self,
|
||||||
|
state: ksp_job_api::JobState,
|
||||||
|
failure_code: std::option::Option<ksp_core_lib::ErrorCode>,
|
||||||
|
) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(state, |snapshot| {
|
||||||
|
snapshot.phase = crate::BackfillJobPhase::Finished;
|
||||||
|
snapshot.failure_code = failure_code;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Publishes direct Created-to-Cancelled termination before execution starts.
|
||||||
|
pub(crate) fn publish_cancelled_from_created(&self) -> ksp_core_lib::Result<crate::BackfillJobSnapshot> {
|
||||||
|
return self.publish_with(ksp_job_api::JobState::Cancelled, |snapshot| {
|
||||||
|
snapshot.phase = crate::BackfillJobPhase::Finished;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
fn publish_with<F>(&self, state: ksp_job_api::JobState, update: F) -> ksp_core_lib::Result<crate::BackfillJobSnapshot>
|
||||||
|
where
|
||||||
|
F: FnOnce(&mut crate::BackfillJobSnapshot),
|
||||||
|
{
|
||||||
|
let current = self.sender.borrow().clone();
|
||||||
|
if current.state().is_terminal() {
|
||||||
|
return std::result::Result::Err(runtime_error("notification.terminal"));
|
||||||
|
}
|
||||||
|
if !valid_snapshot_transition(current.state(), state) {
|
||||||
|
return std::result::Result::Err(runtime_error("notification.transition"));
|
||||||
|
}
|
||||||
|
let sequence = match current.sequence().next() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut snapshot = current.snapshot().clone();
|
||||||
|
update(&mut snapshot);
|
||||||
|
let notification = ksp_job_api::JobNotification::new(self.id.clone(), self.kind.clone(), sequence, state, snapshot.clone());
|
||||||
|
let _previous = self.sender.send_replace(notification);
|
||||||
|
return std::result::Result::Ok(snapshot);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn valid_snapshot_transition(source: ksp_job_api::JobState, target: ksp_job_api::JobState) -> bool {
|
||||||
|
if source.is_terminal() {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
return matches!(
|
||||||
|
(source, target),
|
||||||
|
(ksp_job_api::JobState::Created, ksp_job_api::JobState::Running)
|
||||||
|
| (ksp_job_api::JobState::Created, ksp_job_api::JobState::Cancelled)
|
||||||
|
| (ksp_job_api::JobState::Running, ksp_job_api::JobState::Running)
|
||||||
|
| (ksp_job_api::JobState::Running, ksp_job_api::JobState::Cancelling)
|
||||||
|
| (ksp_job_api::JobState::Running, ksp_job_api::JobState::Completed(_))
|
||||||
|
| (ksp_job_api::JobState::Running, ksp_job_api::JobState::Failed)
|
||||||
|
| (ksp_job_api::JobState::Cancelling, ksp_job_api::JobState::Cancelling)
|
||||||
|
| (ksp_job_api::JobState::Cancelling, ksp_job_api::JobState::Cancelled)
|
||||||
|
| (ksp_job_api::JobState::Cancelling, ksp_job_api::JobState::Failed)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn apply_progress(snapshot: &mut crate::BackfillJobSnapshot, progress: &crate::BackfillExecutionProgress) {
|
||||||
|
snapshot.candidates_admitted = progress.admitted_count();
|
||||||
|
snapshot.candidates_finished = progress.finished_count();
|
||||||
|
snapshot.entities_inserted = progress.inserted_count();
|
||||||
|
snapshot.entities_existing = progress.already_present_count();
|
||||||
|
snapshot.entities_purged = progress.purged_count();
|
||||||
|
snapshot.missing = progress.missing_count();
|
||||||
|
snapshot.conflicts = progress.conflict_count();
|
||||||
|
snapshot.observations_inserted = progress.observation_inserted_count();
|
||||||
|
snapshot.observations_existing = progress.observation_already_present_count();
|
||||||
|
snapshot.cancelled_candidates = progress.cancelled_count();
|
||||||
|
snapshot.holes = progress.hole_count();
|
||||||
|
snapshot.maximum_in_flight = progress.maximum_in_flight();
|
||||||
|
snapshot.contiguous_completed = progress.contiguous_completed();
|
||||||
|
snapshot.checkpoint = std::option::Option::Some(progress.checkpoint().clone());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn apply_batch(snapshot: &mut crate::BackfillJobSnapshot, batch: &crate::BackfillExecutionBatch) {
|
||||||
|
snapshot.candidates_admitted = batch.admitted_count();
|
||||||
|
snapshot.candidates_finished = batch.finished_count();
|
||||||
|
snapshot.entities_inserted = batch.inserted_count();
|
||||||
|
snapshot.entities_existing = batch.already_present_count();
|
||||||
|
snapshot.entities_purged = batch.purged_count();
|
||||||
|
snapshot.missing = batch.missing_count();
|
||||||
|
snapshot.conflicts = batch.conflict_count();
|
||||||
|
snapshot.observations_inserted = batch.observation_inserted_count();
|
||||||
|
snapshot.observations_existing = batch.observation_already_present_count();
|
||||||
|
snapshot.cancelled_candidates = batch.cancelled_count();
|
||||||
|
snapshot.holes = batch.hole_count();
|
||||||
|
snapshot.maximum_in_flight = batch.maximum_in_flight();
|
||||||
|
snapshot.contiguous_completed = batch.checkpoint().completed_prefix();
|
||||||
|
snapshot.checkpoint = std::option::Option::Some(batch.checkpoint().clone());
|
||||||
|
}
|
||||||
|
|
||||||
|
fn cancelled_error() -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_CANCELLED, "Backfill operation cancelled before durable Store submission");
|
||||||
|
}
|
||||||
|
|
||||||
|
fn runtime_error(field: &'static str) -> ksp_core_lib::Error {
|
||||||
|
return ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_RUNTIME_INVALID, "invalid concrete Backfill runtime state").with_context("field", field);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
#[path = "../unit_tests/runtime.rs"]
|
||||||
|
mod tests;
|
||||||
93
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
Normal file
93
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
Normal file
@@ -0,0 +1,93 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
//! Dependency firewall canaries through the concrete cancellation and latest-value runtime tranche.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_manifest_uses_only_planned_ksp_edges_and_private_tokio_runtime() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
for required in [
|
||||||
|
"futures-util = { workspace = true, features = [\"std\"] }",
|
||||||
|
"ksp-core-lib = { path = \"../ksp-core-lib\" }",
|
||||||
|
"ksp-job-api = { path = \"../ksp-job-api\" }",
|
||||||
|
"ksp-logging-lib = { path = \"../ksp-logging-lib\" }",
|
||||||
|
"ksp-onchain-transport-lib = { path = \"../ksp-onchain-transport-lib\" }",
|
||||||
|
"ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }",
|
||||||
|
"serde_json.workspace = true",
|
||||||
|
"sha2.workspace = true",
|
||||||
|
"tokio = { workspace = true, features = [\"macros\", \"sync\"] }",
|
||||||
|
] {
|
||||||
|
assert!(manifest.contains(required), "required Backfill dependency missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in [
|
||||||
|
"ksp-config-lib",
|
||||||
|
"ksp-interface-lib",
|
||||||
|
"ksp-program-api",
|
||||||
|
"ksp-store-api",
|
||||||
|
"ksp-store-postgres-lib",
|
||||||
|
"ksp-wallet-lib",
|
||||||
|
"solana-",
|
||||||
|
"reqwest",
|
||||||
|
"serde.workspace = true",
|
||||||
|
"serde = {",
|
||||||
|
"tonic",
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains(forbidden), "forbidden Backfill dependency present: {forbidden}");
|
||||||
|
}
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("tokio::"), "Tokio implementation types must not leak through the public crate root");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_production_sources_keep_transport_store_and_scheduler_ownership_separate() {
|
||||||
|
let neutral_sources = [
|
||||||
|
include_str!("../src/checkpoint.rs"),
|
||||||
|
include_str!("../src/constants.rs"),
|
||||||
|
include_str!("../src/discovery.rs"),
|
||||||
|
include_str!("../src/error.rs"),
|
||||||
|
include_str!("../src/lib.rs"),
|
||||||
|
include_str!("../src/request.rs"),
|
||||||
|
include_str!("../src/runtime.rs"),
|
||||||
|
];
|
||||||
|
for source in neutral_sources {
|
||||||
|
for forbidden in ["ksp_config_lib::", "ksp_interface_lib::", "ksp_store_api::", "ksp_store_postgres_lib::", "reqwest::", "std::env", "tonic::"] {
|
||||||
|
assert!(!source.contains(forbidden), "forbidden concrete Backfill path detected: {forbidden}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let conversion = include_str!("../src/conversion.rs");
|
||||||
|
assert!(conversion.contains("serde_json::"));
|
||||||
|
assert!(conversion.contains("get_transaction_observed"));
|
||||||
|
assert!(conversion.contains("SolanaTransactionEncoding::Base64"));
|
||||||
|
assert!(conversion.contains("std::option::Option::Some(0)"));
|
||||||
|
assert!(conversion.contains("BackfillHydrationOutcome::Missing(reference)"));
|
||||||
|
assert!(!conversion.contains("execute_standard_rpc"));
|
||||||
|
assert!(!conversion.contains("persist_raw_transaction_acquisition"));
|
||||||
|
let persistence = include_str!("../src/persistence.rs");
|
||||||
|
assert!(persistence.contains("persist_raw_transaction_acquisition"));
|
||||||
|
assert!(persistence.contains("RawTransactionAcquisitionMode::Normal"));
|
||||||
|
assert!(persistence.contains("ERROR_CODE_RAW_CONFLICT"));
|
||||||
|
assert!(!persistence.contains("record_raw_transaction_observation"));
|
||||||
|
assert!(!persistence.contains("RawTransactionAcquisitionMode::ForceRehydrate"));
|
||||||
|
let discovery = include_str!("../src/discovery.rs");
|
||||||
|
assert!(discovery.contains("get_signatures_for_address"));
|
||||||
|
assert!(!discovery.contains("execute_standard_rpc"));
|
||||||
|
assert!(!discovery.contains("retry"));
|
||||||
|
assert!(!discovery.contains("endpoint_name"));
|
||||||
|
let execution = include_str!("../src/execution.rs");
|
||||||
|
assert!(execution.contains("FuturesUnordered"));
|
||||||
|
assert!(execution.contains("hydrate_backfill_candidate"));
|
||||||
|
assert!(execution.contains("persist_backfill_hydration"));
|
||||||
|
assert!(execution.contains("request.hydration_concurrency()"));
|
||||||
|
for forbidden in ["tokio::spawn", "tokio::time", "retry", "endpoint_name", "provider()", "ForceRehydrate", "get_raw_transaction"] {
|
||||||
|
assert!(!execution.contains(forbidden), "forbidden scheduler/policy ownership detected: {forbidden}");
|
||||||
|
}
|
||||||
|
let runtime = include_str!("../src/runtime.rs");
|
||||||
|
assert!(runtime.contains("tokio::sync::watch"));
|
||||||
|
assert!(runtime.contains("JobSnapshotSource"));
|
||||||
|
assert!(runtime.contains("JobCancellationToken"));
|
||||||
|
assert!(!runtime.contains("tokio::spawn"));
|
||||||
|
assert!(!runtime.contains("tokio::time"));
|
||||||
|
assert!(!runtime.contains("reqwest::"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
350
crates/ksp-job-backfill-lib/tests/hardening.rs
Normal file
350
crates/ksp-job-backfill-lib/tests/hardening.rs
Normal file
@@ -0,0 +1,350 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/hardening.rs
|
||||||
|
// version: 4
|
||||||
|
|
||||||
|
//! Adversarial, security, visibility and external-boundary hardening canaries for `pre.010`.
|
||||||
|
|
||||||
|
fn signature_text() -> std::string::String {
|
||||||
|
return "2".repeat(ksp_job_backfill_lib::MIN_BACKFILL_SIGNATURE_TEXT_BYTES);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn signature() -> std::option::Option<ksp_job_backfill_lib::BackfillSignature> {
|
||||||
|
let result = ksp_job_backfill_lib::BackfillSignature::new(signature_text());
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_job_backfill_lib::ERROR_CODE_BACKFILL_SIGNATURE_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn network(value: &'static str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
let result = ksp_store_lib::RawNetworkId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code().domain(), "store_api");
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn job_id(value: &'static str) -> std::option::Option<ksp_job_api::JobId> {
|
||||||
|
let result = ksp_job_api::JobId::new(value);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code().domain(), "job_api");
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn explicit_scope() -> std::option::Option<ksp_job_backfill_lib::BackfillScope> {
|
||||||
|
let signature = match signature() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let result = ksp_job_backfill_lib::BackfillScope::explicit_signatures(vec![signature]);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_job_backfill_lib::ERROR_CODE_BACKFILL_REQUEST_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn request(
|
||||||
|
id: &'static str,
|
||||||
|
network_name: &'static str,
|
||||||
|
role_name: &'static str,
|
||||||
|
scope: ksp_job_backfill_lib::BackfillScope,
|
||||||
|
) -> std::option::Option<ksp_job_backfill_lib::BackfillRequest> {
|
||||||
|
let id = match job_id(id) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let network = match network(network_name) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let result = ksp_job_backfill_lib::BackfillRequest::new(
|
||||||
|
id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new(role_name),
|
||||||
|
ksp_job_backfill_lib::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
500,
|
||||||
|
8,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(error) => {
|
||||||
|
assert_eq!(error.code(), ksp_job_backfill_lib::ERROR_CODE_BACKFILL_REQUEST_INVALID);
|
||||||
|
std::option::Option::None
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_error_codes_are_stable_unique_and_domain_scoped() {
|
||||||
|
let codes = [
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_CHECKPOINT_INVALID, "checkpoint_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_DISCOVERY_INVALID, "discovery_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_DISCOVERY_STALLED, "discovery_stalled"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_EXECUTION_INVALID, "execution_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_PERSISTENCE_INVALID, "persistence_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID, "raw_conversion_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_REQUEST_INVALID, "request_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_RUNTIME_INVALID, "runtime_invalid"),
|
||||||
|
(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_SIGNATURE_INVALID, "signature_invalid"),
|
||||||
|
];
|
||||||
|
for (index, (code, expected)) in codes.iter().enumerate() {
|
||||||
|
assert_eq!(code.domain(), "job_backfill");
|
||||||
|
assert_eq!(code.code(), *expected);
|
||||||
|
for (other_index, (other, _)) in codes.iter().enumerate() {
|
||||||
|
if index != other_index {
|
||||||
|
assert_ne!(code, other);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_request_adversarial_bounds_return_stable_codes() {
|
||||||
|
let invalid_signature = ksp_job_backfill_lib::BackfillSignature::new("0".repeat(ksp_job_backfill_lib::MIN_BACKFILL_SIGNATURE_TEXT_BYTES));
|
||||||
|
let error = match invalid_signature {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), ksp_job_backfill_lib::ERROR_CODE_BACKFILL_SIGNATURE_INVALID);
|
||||||
|
let id = match job_id("backfill:hardening:bounds") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match network("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = match explicit_scope() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = ksp_job_backfill_lib::BackfillRequest::new(
|
||||||
|
id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
ksp_job_backfill_lib::BackfillCommitment::Finalized,
|
||||||
|
scope,
|
||||||
|
0,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), ksp_job_backfill_lib::ERROR_CODE_BACKFILL_REQUEST_INVALID);
|
||||||
|
assert!(error.context().iter().any(|context| return context.key() == "field" && context.value() == "page_size"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_scope_fingerprint_is_network_sensitive_and_transport_source_independent() {
|
||||||
|
let scope = match explicit_scope() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first = match request("backfill:hardening:fingerprint:a", "devnet", "history-primary", scope.clone()) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match request("backfill:hardening:fingerprint:b", "devnet", "history-secondary", scope.clone()) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let other_network = match request("backfill:hardening:fingerprint:c", "mainnet", "history-primary", scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.scope_fingerprint(), second.scope_fingerprint());
|
||||||
|
assert_ne!(first.scope_fingerprint(), other_network.scope_fingerprint());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_debug_surfaces_redact_signature_fingerprint_and_runtime_snapshot_payloads() {
|
||||||
|
let raw_signature = signature_text();
|
||||||
|
let scope = match explicit_scope() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let backfill_request = match request("backfill:hardening:debug", "devnet", "history", scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let request_debug = format!("{backfill_request:?}");
|
||||||
|
assert!(!request_debug.contains(raw_signature.as_str()));
|
||||||
|
assert!(request_debug.contains("BackfillScopeFingerprint(..)"));
|
||||||
|
assert!(request_debug.contains("signature_count"));
|
||||||
|
let fingerprint_debug = format!("{:?}", backfill_request.scope_fingerprint());
|
||||||
|
assert_eq!(fingerprint_debug, "BackfillScopeFingerprint(..)");
|
||||||
|
let runtime = ksp_job_backfill_lib::BackfillJobRuntime::new(backfill_request);
|
||||||
|
let runtime = match runtime {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let handle_debug = format!("{:?}", runtime.handle());
|
||||||
|
assert!(!handle_debug.contains(raw_signature.as_str()));
|
||||||
|
assert!(handle_debug.contains("snapshot"));
|
||||||
|
assert!(handle_debug.contains("<redacted>"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_runtime_handle_starts_created_and_cancellation_is_idempotent() {
|
||||||
|
let scope = match explicit_scope() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let backfill_request = match request("backfill:hardening:control", "devnet", "history", scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let runtime = ksp_job_backfill_lib::BackfillJobRuntime::new(backfill_request);
|
||||||
|
let runtime = match runtime {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let handle = runtime.handle();
|
||||||
|
let source = handle.snapshots();
|
||||||
|
let current = <ksp_job_backfill_lib::BackfillSnapshotSource as ksp_job_api::JobSnapshotSource>::current(&source);
|
||||||
|
assert_eq!(current.state(), ksp_job_api::JobState::Created);
|
||||||
|
assert_eq!(current.snapshot().phase(), ksp_job_backfill_lib::BackfillJobPhase::Created);
|
||||||
|
assert!(!handle.is_cancellation_requested());
|
||||||
|
assert!(handle.cancel());
|
||||||
|
assert!(handle.is_cancellation_requested());
|
||||||
|
assert!(!handle.cancel());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_manifest_dependency_surface_is_exact_and_backend_neutral() {
|
||||||
|
let manifest = include_str!("../Cargo.toml");
|
||||||
|
let mut section = "";
|
||||||
|
let mut normal = std::collections::BTreeSet::new();
|
||||||
|
let mut dev = std::collections::BTreeSet::new();
|
||||||
|
for line in manifest.lines() {
|
||||||
|
let trimmed = line.trim();
|
||||||
|
if trimmed.starts_with('[') && trimmed.ends_with(']') {
|
||||||
|
section = trimmed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if trimmed.is_empty() || trimmed.starts_with('#') {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match trimmed.split_once('=') {
|
||||||
|
std::option::Option::Some((name, _)) => name.trim(),
|
||||||
|
std::option::Option::None => continue,
|
||||||
|
};
|
||||||
|
if section == "[dependencies]" {
|
||||||
|
normal.insert(name);
|
||||||
|
} else if section == "[dev-dependencies]" {
|
||||||
|
dev.insert(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assert_eq!(
|
||||||
|
normal,
|
||||||
|
std::collections::BTreeSet::from([
|
||||||
|
"futures-util",
|
||||||
|
"ksp-core-lib",
|
||||||
|
"ksp-job-api",
|
||||||
|
"ksp-logging-lib",
|
||||||
|
"ksp-onchain-transport-lib",
|
||||||
|
"ksp-store-lib",
|
||||||
|
"serde_json.workspace",
|
||||||
|
"sha2.workspace",
|
||||||
|
"tokio",
|
||||||
|
])
|
||||||
|
);
|
||||||
|
assert_eq!(dev, std::collections::BTreeSet::from(["tokio"]));
|
||||||
|
assert!(manifest.contains("ksp-store-lib = { path = \"../ksp-store-lib\", default-features = false }"));
|
||||||
|
assert!(manifest.contains("tokio = { workspace = true, features = [\"macros\", \"sync\"] }"));
|
||||||
|
assert!(!manifest.contains("ksp-store-postgres-lib"));
|
||||||
|
assert!(!manifest.contains("ksp-store-api"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_source_visibility_contract_uses_crate_root_for_shared_items() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
assert!(!root.contains("BackfillRuntimeControl"));
|
||||||
|
assert!(!root.contains("TerminalClaim"));
|
||||||
|
let source_contracts: [(&str, &[&str]); 3] = [
|
||||||
|
(include_str!("../src/checkpoint.rs"), &["BackfillCheckpoint", "CompletionFrontier"]),
|
||||||
|
(
|
||||||
|
include_str!("../src/request.rs"),
|
||||||
|
&["BackfillCommitment", "BackfillSignature", "BackfillScopeKind", "BackfillScope", "BackfillScopeFingerprint", "BackfillRequest"],
|
||||||
|
),
|
||||||
|
(include_str!("../src/runtime.rs"), &["BackfillJobPhase", "BackfillJobSnapshot", "BackfillJobHandle", "BackfillJobRuntime"]),
|
||||||
|
];
|
||||||
|
for (source, symbols) in source_contracts {
|
||||||
|
for symbol in symbols {
|
||||||
|
let required = format!("impl crate::{symbol}");
|
||||||
|
assert!(source.contains(required.as_str()), "shared item must use crate-root impl path: {symbol}");
|
||||||
|
let forbidden = format!("impl {symbol}");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "shared item must not use bare impl path: {symbol}");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let request = include_str!("../src/request.rs");
|
||||||
|
assert!(request.contains("crate::BackfillScopeFingerprint::from_bytes(bytes)"));
|
||||||
|
assert!(!request.contains("crate::BackfillScopeFingerprint(bytes)"));
|
||||||
|
assert!(!request.contains("return BackfillScopeFingerprint(bytes)"));
|
||||||
|
for (module, source) in [
|
||||||
|
("checkpoint", include_str!("../src/checkpoint.rs")),
|
||||||
|
("conversion", include_str!("../src/conversion.rs")),
|
||||||
|
("discovery", include_str!("../src/discovery.rs")),
|
||||||
|
("execution", include_str!("../src/execution.rs")),
|
||||||
|
("persistence", include_str!("../src/persistence.rs")),
|
||||||
|
("request", include_str!("../src/request.rs")),
|
||||||
|
("runtime", include_str!("../src/runtime.rs")),
|
||||||
|
] {
|
||||||
|
let forbidden = format!("crate::{module}::");
|
||||||
|
assert!(!source.contains(forbidden.as_str()), "internal module path bypasses crate-root façade: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_lower_layers_have_no_dependency_return_to_job() {
|
||||||
|
for manifest in [
|
||||||
|
include_str!("../../ksp-core-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-logging-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-onchain-transport-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-api/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-lib/Cargo.toml"),
|
||||||
|
include_str!("../../ksp-store-postgres-lib/Cargo.toml"),
|
||||||
|
] {
|
||||||
|
assert!(!manifest.contains("ksp-job-api"));
|
||||||
|
assert!(!manifest.contains("ksp-job-backfill-lib"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_public_root_exposes_no_runtime_or_backend_implementation_types() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for forbidden in
|
||||||
|
["pub mod ", "tokio::", "FuturesUnordered", "serde_json::", "sha2::", "ksp_store_api::", "ksp_store_postgres_lib::", "reqwest::", "tonic::"]
|
||||||
|
{
|
||||||
|
assert!(!root.contains(forbidden), "implementation/backend detail leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
125
crates/ksp-job-backfill-lib/tests/public_api.rs
Normal file
125
crates/ksp-job-backfill-lib/tests/public_api.rs
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/public_api.rs
|
||||||
|
// version: 7
|
||||||
|
|
||||||
|
//! Public API canaries through the concrete Backfill cancellation and latest-value runtime tranche.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_request_scope_and_discovery_contracts_are_available_from_crate_root() {
|
||||||
|
let _discover = ksp_job_backfill_lib::discover_backfill_candidates;
|
||||||
|
let address = ksp_core_lib::Pubkey::new_from_array([21_u8; 32]);
|
||||||
|
let scope = ksp_job_backfill_lib::BackfillScope::latest_address(address);
|
||||||
|
let network = ksp_store_lib::RawNetworkId::new("devnet");
|
||||||
|
assert!(network.is_ok());
|
||||||
|
let network = match network {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let job_id = ksp_job_api::JobId::new("backfill:public-api");
|
||||||
|
assert!(job_id.is_ok());
|
||||||
|
let job_id = match job_id {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let request = ksp_job_backfill_lib::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
ksp_job_backfill_lib::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
500,
|
||||||
|
8,
|
||||||
|
std::option::Option::Some(100),
|
||||||
|
);
|
||||||
|
assert!(request.is_ok());
|
||||||
|
let request = match request {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(request.scope().kind(), ksp_job_backfill_lib::BackfillScopeKind::LatestAddress);
|
||||||
|
assert_eq!(request.page_size(), 100);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_candidate_identity_is_network_plus_signature_not_transport_source() {
|
||||||
|
let signature = ksp_job_backfill_lib::BackfillSignature::new("1".repeat(ksp_job_backfill_lib::MIN_BACKFILL_SIGNATURE_TEXT_BYTES));
|
||||||
|
let signature = match signature {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let mainnet = ksp_store_lib::RawNetworkId::new("mainnet");
|
||||||
|
let devnet = ksp_store_lib::RawNetworkId::new("devnet");
|
||||||
|
let mainnet = match mainnet {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let devnet = match devnet {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let mainnet_identity = ksp_job_backfill_lib::BackfillCandidateIdentity::new(mainnet, signature.clone());
|
||||||
|
let devnet_identity = ksp_job_backfill_lib::BackfillCandidateIdentity::new(devnet, signature);
|
||||||
|
assert_ne!(mainnet_identity, devnet_identity);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_raw_conversion_contract_is_available_from_crate_root() {
|
||||||
|
assert_eq!(ksp_job_backfill_lib::RAW_TRANSACTION_FORMAT_ID, "ksp.solana.raw_transaction");
|
||||||
|
assert_eq!(ksp_job_backfill_lib::RAW_TRANSACTION_FORMAT_VERSION, 1);
|
||||||
|
let signature = ksp_job_backfill_lib::BackfillSignature::new("1".repeat(64));
|
||||||
|
assert!(signature.is_ok());
|
||||||
|
let signature = match signature {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let decoded = signature.to_raw_transaction_signature();
|
||||||
|
assert!(decoded.is_ok());
|
||||||
|
if let std::result::Result::Ok(decoded) = decoded {
|
||||||
|
assert_eq!(decoded.as_bytes(), &[0_u8; 64]);
|
||||||
|
}
|
||||||
|
let _hydrate = ksp_job_backfill_lib::hydrate_backfill_candidate;
|
||||||
|
let _outcome: std::option::Option<ksp_job_backfill_lib::BackfillHydrationOutcome> = std::option::Option::None;
|
||||||
|
let _acquisition: std::option::Option<ksp_job_backfill_lib::BackfillRawAcquisition> = std::option::Option::None;
|
||||||
|
assert_eq!(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID, ksp_core_lib::ErrorCode::new("job_backfill", "raw_conversion_invalid"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_store_persistence_contract_is_available_from_crate_root() {
|
||||||
|
let _persist = ksp_job_backfill_lib::persist_backfill_hydration;
|
||||||
|
let _outcome: std::option::Option<ksp_job_backfill_lib::BackfillPersistenceOutcome> = std::option::Option::None;
|
||||||
|
let _entity = ksp_job_backfill_lib::BackfillEntityPersistence::AlreadyPresent;
|
||||||
|
let _observation = ksp_job_backfill_lib::BackfillObservationPersistence::AlreadyPresent;
|
||||||
|
assert_eq!(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_PERSISTENCE_INVALID, ksp_core_lib::ErrorCode::new("job_backfill", "persistence_invalid"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_checkpoint_and_bounded_execution_contracts_are_available_from_crate_root() {
|
||||||
|
let _execute = ksp_job_backfill_lib::execute_backfill_discovery;
|
||||||
|
let _checkpoint: std::option::Option<ksp_job_backfill_lib::BackfillCheckpoint> = std::option::Option::None;
|
||||||
|
let _batch: std::option::Option<ksp_job_backfill_lib::BackfillExecutionBatch> = std::option::Option::None;
|
||||||
|
assert_eq!(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_CHECKPOINT_INVALID, ksp_core_lib::ErrorCode::new("job_backfill", "checkpoint_invalid"));
|
||||||
|
assert_eq!(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_EXECUTION_INVALID, ksp_core_lib::ErrorCode::new("job_backfill", "execution_invalid"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_concrete_runtime_snapshot_and_control_contracts_are_available_from_crate_root() {
|
||||||
|
fn assert_source<T>(_: std::marker::PhantomData<T>)
|
||||||
|
where
|
||||||
|
T: ksp_job_api::JobSnapshotSource<Snapshot = ksp_job_backfill_lib::BackfillJobSnapshot>,
|
||||||
|
{
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
assert_source(std::marker::PhantomData::<ksp_job_backfill_lib::BackfillSnapshotSource>);
|
||||||
|
let _runtime_new = ksp_job_backfill_lib::BackfillJobRuntime::new;
|
||||||
|
let _handle: std::option::Option<ksp_job_backfill_lib::BackfillJobHandle> = std::option::Option::None;
|
||||||
|
let _phase = ksp_job_backfill_lib::BackfillJobPhase::Discovering;
|
||||||
|
assert_eq!(ksp_job_backfill_lib::BACKFILL_JOB_KIND_CODE, "solana.raw_transaction.backfill");
|
||||||
|
assert_eq!(ksp_job_backfill_lib::ERROR_CODE_BACKFILL_RUNTIME_INVALID, ksp_core_lib::ErrorCode::new("job_backfill", "runtime_invalid"));
|
||||||
|
return;
|
||||||
|
}
|
||||||
113
crates/ksp-job-backfill-lib/tests/release_completeness.rs
Normal file
113
crates/ksp-job-backfill-lib/tests/release_completeness.rs
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/tests/release_completeness.rs
|
||||||
|
// version: 9
|
||||||
|
|
||||||
|
//! Completeness canaries through the `pre.010` hardening and external-canary tranche.
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_production_module_inventory_is_exact() -> std::io::Result<()> {
|
||||||
|
let source_root = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("src");
|
||||||
|
let entries = match std::fs::read_dir(source_root) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let mut names = std::vec::Vec::new();
|
||||||
|
for entry in entries {
|
||||||
|
let entry = match entry {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let file_type = match entry.file_type() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
if !file_type.is_file() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
let name = match entry.file_name().into_string() {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => continue,
|
||||||
|
};
|
||||||
|
if name.ends_with(".rs") {
|
||||||
|
names.push(name);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
names.sort_unstable();
|
||||||
|
assert_eq!(
|
||||||
|
names,
|
||||||
|
std::vec![
|
||||||
|
"checkpoint.rs",
|
||||||
|
"constants.rs",
|
||||||
|
"conversion.rs",
|
||||||
|
"discovery.rs",
|
||||||
|
"error.rs",
|
||||||
|
"execution.rs",
|
||||||
|
"lib.rs",
|
||||||
|
"persistence.rs",
|
||||||
|
"request.rs",
|
||||||
|
"runtime.rs",
|
||||||
|
]
|
||||||
|
);
|
||||||
|
return std::result::Result::Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_surface_closes_concrete_cancellation_and_latest_value_runtime() {
|
||||||
|
let root = include_str!("../src/lib.rs");
|
||||||
|
for required in [
|
||||||
|
"BackfillCandidate",
|
||||||
|
"BackfillCandidateIdentity",
|
||||||
|
"BackfillCommitment",
|
||||||
|
"BackfillDiscovery",
|
||||||
|
"BackfillDiscoveryBoundary",
|
||||||
|
"BackfillRequest",
|
||||||
|
"BackfillScope",
|
||||||
|
"BackfillScopeFingerprint",
|
||||||
|
"BackfillSignature",
|
||||||
|
"discover_backfill_candidates",
|
||||||
|
"BackfillRawAcquisition",
|
||||||
|
"BackfillHydrationOutcome",
|
||||||
|
"hydrate_backfill_candidate",
|
||||||
|
"BackfillPersistenceOutcome",
|
||||||
|
"BackfillEntityPersistence",
|
||||||
|
"BackfillObservationPersistence",
|
||||||
|
"persist_backfill_hydration",
|
||||||
|
"BackfillCheckpoint",
|
||||||
|
"BackfillExecutionBatch",
|
||||||
|
"execute_backfill_discovery",
|
||||||
|
"ERROR_CODE_BACKFILL_CHECKPOINT_INVALID",
|
||||||
|
"ERROR_CODE_BACKFILL_EXECUTION_INVALID",
|
||||||
|
"BackfillJobHandle",
|
||||||
|
"BackfillJobPhase",
|
||||||
|
"BackfillJobRuntime",
|
||||||
|
"BackfillJobSnapshot",
|
||||||
|
"BackfillSnapshotSource",
|
||||||
|
"BACKFILL_JOB_KIND_CODE",
|
||||||
|
"ERROR_CODE_BACKFILL_RUNTIME_INVALID",
|
||||||
|
] {
|
||||||
|
assert!(root.contains(required), "required pre.010 public contract missing: {required}");
|
||||||
|
}
|
||||||
|
for forbidden in ["tokio::", "FuturesUnordered", "watch::Receiver", "watch::Sender"] {
|
||||||
|
assert!(!root.contains(forbidden), "runtime implementation detail leaked into public root: {forbidden}");
|
||||||
|
}
|
||||||
|
assert!(!root.contains("pub mod "));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_010_external_hardening_suite_is_present_and_scoped() {
|
||||||
|
let hardening = include_str!("hardening.rs");
|
||||||
|
for required in [
|
||||||
|
"pre_010_external_error_codes_are_stable_unique_and_domain_scoped",
|
||||||
|
"pre_010_external_request_adversarial_bounds_return_stable_codes",
|
||||||
|
"pre_010_scope_fingerprint_is_network_sensitive_and_transport_source_independent",
|
||||||
|
"pre_010_debug_surfaces_redact_signature_fingerprint_and_runtime_snapshot_payloads",
|
||||||
|
"pre_010_external_runtime_handle_starts_created_and_cancellation_is_idempotent",
|
||||||
|
"pre_010_manifest_dependency_surface_is_exact_and_backend_neutral",
|
||||||
|
"pre_010_source_visibility_contract_uses_crate_root_for_shared_items",
|
||||||
|
"pre_010_lower_layers_have_no_dependency_return_to_job",
|
||||||
|
"pre_010_public_root_exposes_no_runtime_or_backend_implementation_types",
|
||||||
|
] {
|
||||||
|
assert!(hardening.contains(required), "required pre.010 hardening canary missing: {required}");
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
168
crates/ksp-job-backfill-lib/unit_tests/checkpoint.rs
Normal file
168
crates/ksp-job-backfill-lib/unit_tests/checkpoint.rs
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/checkpoint.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn signature(character: char) -> std::option::Option<crate::BackfillSignature> {
|
||||||
|
return crate::BackfillSignature::new(character.to_string().repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES)).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn request(scope: crate::BackfillScope) -> std::option::Option<crate::BackfillRequest> {
|
||||||
|
let job_id = match ksp_job_api::JobId::new("backfill:checkpoint-test") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return crate::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
10,
|
||||||
|
4,
|
||||||
|
std::option::Option::None,
|
||||||
|
)
|
||||||
|
.ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn candidate(request: &crate::BackfillRequest, character: char) -> std::option::Option<crate::BackfillCandidate> {
|
||||||
|
let signature = match signature(character) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let identity = crate::BackfillCandidateIdentity::new(request.network().clone(), signature);
|
||||||
|
return std::option::Option::Some(crate::BackfillCandidate::new(identity, std::option::Option::None));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn discovery(
|
||||||
|
request: &crate::BackfillRequest,
|
||||||
|
characters: &[char],
|
||||||
|
boundary: crate::BackfillDiscoveryBoundary,
|
||||||
|
) -> std::option::Option<crate::BackfillDiscovery> {
|
||||||
|
let mut candidates = std::vec::Vec::with_capacity(characters.len());
|
||||||
|
for character in characters {
|
||||||
|
let value = match candidate(request, *character) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
candidates.push(value);
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(crate::BackfillDiscovery::new(request.network().clone(), request.scope_fingerprint(), candidates, 1, boundary));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_frontier_advances_only_across_contiguous_durable_results() {
|
||||||
|
let mut frontier = crate::CompletionFrontier::new(4);
|
||||||
|
assert!(frontier.mark_durable(1).is_ok());
|
||||||
|
assert_eq!(frontier.contiguous_completed(), 0);
|
||||||
|
assert!(frontier.mark_durable(0).is_ok());
|
||||||
|
assert_eq!(frontier.contiguous_completed(), 2);
|
||||||
|
assert!(frontier.mark_durable(3).is_ok());
|
||||||
|
assert_eq!(frontier.contiguous_completed(), 2);
|
||||||
|
assert!(frontier.mark_durable(2).is_ok());
|
||||||
|
assert_eq!(frontier.contiguous_completed(), 4);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_checkpoint_is_bound_to_job_and_scope_fingerprint() {
|
||||||
|
let scope = crate::BackfillScope::latest_address(ksp_core_lib::Pubkey::new_from_array([1_u8; 32]));
|
||||||
|
let primary_request = match request(scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let other_job = match ksp_job_api::JobId::new("backfill:other") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let wrong_job = crate::BackfillCheckpoint::new(other_job, primary_request.scope_fingerprint(), 0, std::option::Option::None);
|
||||||
|
let result = primary_request.clone().with_checkpoint(wrong_job);
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_CHECKPOINT_INVALID);
|
||||||
|
}
|
||||||
|
let other_scope = crate::BackfillScope::latest_address(ksp_core_lib::Pubkey::new_from_array([2_u8; 32]));
|
||||||
|
let other_request = match request(other_scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let wrong_scope = crate::BackfillCheckpoint::new(primary_request.job_id().clone(), other_request.scope_fingerprint(), 0, std::option::Option::None);
|
||||||
|
assert!(primary_request.with_checkpoint(wrong_scope).is_err());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_before_checkpoint_is_cumulative_and_tracks_last_contiguous_candidate() {
|
||||||
|
let anchor = match signature('8') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = crate::BackfillScope::before_address(ksp_core_lib::Pubkey::new_from_array([3_u8; 32]), anchor);
|
||||||
|
let request = match request(scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_discovery = match discovery(&request, &['7', '6', '5'], crate::BackfillDiscoveryBoundary::RpcBoundary) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let mut first_frontier = crate::CompletionFrontier::new(3);
|
||||||
|
assert!(first_frontier.mark_durable(1).is_ok());
|
||||||
|
assert_eq!(first_frontier.contiguous_completed(), 0);
|
||||||
|
assert!(first_frontier.mark_durable(0).is_ok());
|
||||||
|
let first = match crate::checkpoint_from_frontier(&request, &first_discovery, &first_frontier) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.completed_prefix(), 2);
|
||||||
|
let expected_first = "6".repeat(64);
|
||||||
|
assert_eq!(first.resume_before().map(crate::BackfillSignature::as_str), std::option::Option::Some(expected_first.as_str()));
|
||||||
|
let resumed = match request.with_checkpoint(first) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let second_discovery = match discovery(&resumed, &['5', '4'], crate::BackfillDiscoveryBoundary::RpcBoundary) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let mut second_frontier = crate::CompletionFrontier::new(2);
|
||||||
|
assert!(second_frontier.mark_durable(0).is_ok());
|
||||||
|
let second = match crate::checkpoint_from_frontier(&resumed, &second_discovery, &second_frontier) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(second.completed_prefix(), 3);
|
||||||
|
let expected_second = "5".repeat(64);
|
||||||
|
assert_eq!(second.resume_before().map(crate::BackfillSignature::as_str), std::option::Option::Some(expected_second.as_str()));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_008_after_anchor_not_reached_never_advances_checkpoint() {
|
||||||
|
let anchor = match signature('1') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = crate::BackfillScope::after_address(ksp_core_lib::Pubkey::new_from_array([4_u8; 32]), anchor);
|
||||||
|
let request = match request(scope) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let partial = match discovery(&request, &['7', '6', '5'], crate::BackfillDiscoveryBoundary::AfterAnchorNotReached) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let mut frontier = crate::CompletionFrontier::new(3);
|
||||||
|
assert!(frontier.mark_durable(0).is_ok());
|
||||||
|
assert!(frontier.mark_durable(1).is_ok());
|
||||||
|
let checkpoint = match crate::checkpoint_from_frontier(&request, &partial, &frontier) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(checkpoint.completed_prefix(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
301
crates/ksp-job-backfill-lib/unit_tests/conversion.rs
Normal file
301
crates/ksp-job-backfill-lib/unit_tests/conversion.rs
Normal file
@@ -0,0 +1,301 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/conversion.rs
|
||||||
|
// version: 2
|
||||||
|
|
||||||
|
fn signature_text() -> std::option::Option<crate::BackfillSignature> {
|
||||||
|
return match crate::BackfillSignature::new("1".repeat(64)) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn request(network: &str, signature: crate::BackfillSignature) -> std::option::Option<crate::BackfillRequest> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let job_id = match ksp_job_api::JobId::new("backfill:pre006") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let scope = match crate::BackfillScope::explicit_signatures(std::vec![signature]) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return match crate::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
crate::BackfillCommitment::Finalized,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
100,
|
||||||
|
4,
|
||||||
|
std::option::Option::None,
|
||||||
|
) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn candidate(network: &str, signature: crate::BackfillSignature) -> std::option::Option<crate::BackfillCandidate> {
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new(network) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(crate::BackfillCandidate::new(crate::BackfillCandidateIdentity::new(network, signature), std::option::Option::Some(42)));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn received_at() -> std::option::Option<ksp_store_lib::RawTimestamp> {
|
||||||
|
return match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_001_000) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn fields<'a>(
|
||||||
|
transaction: &'a ksp_onchain_transport_lib::SolanaEncodedTransaction,
|
||||||
|
meta: &'a ksp_onchain_transport_lib::SolanaWireField<serde_json::Value>,
|
||||||
|
version: &'a ksp_onchain_transport_lib::SolanaWireField<ksp_onchain_transport_lib::SolanaTransactionVersion>,
|
||||||
|
transaction_index: &'a ksp_onchain_transport_lib::SolanaWireField<u32>,
|
||||||
|
block_time: std::option::Option<i64>,
|
||||||
|
) -> super::CanonicalTransactionFields<'a> {
|
||||||
|
return super::CanonicalTransactionFields { slot: 123, block_time, transaction, meta, version, transaction_index };
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_signature_decoding_requires_exactly_sixty_four_bytes() {
|
||||||
|
let signature = match signature_text() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let decoded = signature.to_raw_transaction_signature();
|
||||||
|
assert!(decoded.is_ok());
|
||||||
|
if let std::result::Result::Ok(decoded) = decoded {
|
||||||
|
assert_eq!(decoded.as_bytes(), &[0_u8; 64]);
|
||||||
|
}
|
||||||
|
let too_many_zeroes = crate::BackfillSignature::new("1".repeat(65));
|
||||||
|
assert!(too_many_zeroes.is_ok());
|
||||||
|
if let std::result::Result::Ok(too_many_zeroes) = too_many_zeroes {
|
||||||
|
assert!(too_many_zeroes.to_raw_transaction_signature().is_err());
|
||||||
|
}
|
||||||
|
let overflow = crate::BackfillSignature::new("z".repeat(88));
|
||||||
|
assert!(overflow.is_ok());
|
||||||
|
if let std::result::Result::Ok(overflow) = overflow {
|
||||||
|
assert!(overflow.to_raw_transaction_signature().is_err());
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_canonical_payload_golden_bytes_hash_and_provenance_are_exact() {
|
||||||
|
let signature = match signature_text() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let request = match request("devnet", signature.clone()) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let candidate = match candidate("devnet", signature) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let reference = match super::canonical_reference(&request, &candidate) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let transaction = ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary {
|
||||||
|
data: "AQID".to_owned(),
|
||||||
|
encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64,
|
||||||
|
};
|
||||||
|
let meta = ksp_onchain_transport_lib::SolanaWireField::Value(serde_json::json!({"z": 1, "a": {"y": true, "x": null}}));
|
||||||
|
let version = ksp_onchain_transport_lib::SolanaWireField::Value(ksp_onchain_transport_lib::SolanaTransactionVersion::Legacy);
|
||||||
|
let transaction_index = ksp_onchain_transport_lib::SolanaWireField::Value(7_u32);
|
||||||
|
let received_at = match received_at() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let acquisition = super::convert_available_fields(
|
||||||
|
&request,
|
||||||
|
reference,
|
||||||
|
fields(&transaction, &meta, &version, &transaction_index, std::option::Option::Some(1_700_000_000)),
|
||||||
|
"publicnode",
|
||||||
|
"devnet-primary",
|
||||||
|
received_at,
|
||||||
|
);
|
||||||
|
assert!(acquisition.is_ok());
|
||||||
|
let acquisition = match acquisition {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let expected = b"{\"transaction\":[\"AQID\",\"base64\"],\"meta\":{\"a\":{\"x\":null,\"y\":true},\"z\":1},\"version\":\"legacy\",\"transactionIndex\":7}";
|
||||||
|
assert_eq!(acquisition.transaction().payload().bytes(), expected);
|
||||||
|
assert_eq!(acquisition.transaction().payload().byte_len(), 112);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_id().as_str(), crate::RAW_TRANSACTION_FORMAT_ID);
|
||||||
|
assert_eq!(acquisition.transaction().payload().format_version(), crate::RAW_TRANSACTION_FORMAT_VERSION);
|
||||||
|
assert_eq!(
|
||||||
|
acquisition.transaction().payload().content_hash().as_bytes(),
|
||||||
|
&[
|
||||||
|
34, 7, 146, 210, 177, 93, 38, 47, 218, 36, 44, 178, 32, 119, 78, 233, 221, 239, 254, 191, 4, 220, 250, 218, 188, 248, 239, 118, 169, 177, 167, 195
|
||||||
|
]
|
||||||
|
);
|
||||||
|
assert_eq!(acquisition.transaction().slot(), 123);
|
||||||
|
assert_eq!(acquisition.transaction().block_time().map(|value| return value.unix_millis()), std::option::Option::Some(1_700_000_000_000));
|
||||||
|
assert_eq!(acquisition.transaction().reference().network().as_str(), "devnet");
|
||||||
|
assert_eq!(acquisition.transaction().reference().signature().as_bytes(), &[0_u8; 64]);
|
||||||
|
let provenance = acquisition.observation().provenance();
|
||||||
|
assert_eq!(provenance.provider().as_str(), "publicnode");
|
||||||
|
assert_eq!(provenance.protocol().as_str(), "solana.http.json_rpc");
|
||||||
|
assert_eq!(provenance.acquisition_method().as_str(), "getTransaction");
|
||||||
|
assert_eq!(provenance.endpoint_id().map(ksp_store_lib::RawProvenanceCode::as_str), std::option::Option::Some("devnet-primary"));
|
||||||
|
assert_eq!(provenance.commitment().map(ksp_store_lib::RawProvenanceCode::as_str), std::option::Option::Some("finalized"));
|
||||||
|
assert_eq!(provenance.capture_session_id().map(ksp_store_lib::RawProvenanceCode::as_str), std::option::Option::Some("backfill:pre006"));
|
||||||
|
assert_eq!(provenance.received_at(), received_at);
|
||||||
|
assert!(provenance.source_payload_hash().is_none());
|
||||||
|
assert!(provenance.source_payload_size_bytes().is_none());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_wire_omission_and_null_produce_distinct_canonical_bytes() {
|
||||||
|
let transaction = ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary {
|
||||||
|
data: "AQID".to_owned(),
|
||||||
|
encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base64,
|
||||||
|
};
|
||||||
|
let meta_omitted = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let meta_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
||||||
|
let version_omitted = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let version_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
||||||
|
let transaction_index_omitted = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let transaction_index_null = ksp_onchain_transport_lib::SolanaWireField::Null;
|
||||||
|
let omitted = super::canonical_payload_bytes(&fields(&transaction, &meta_omitted, &version_omitted, &transaction_index_omitted, std::option::Option::None));
|
||||||
|
assert!(omitted.is_ok());
|
||||||
|
let omitted = match omitted {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(omitted, b"{\"transaction\":[\"AQID\",\"base64\"]}");
|
||||||
|
let nulls = super::canonical_payload_bytes(&fields(&transaction, &meta_null, &version_null, &transaction_index_null, std::option::Option::None));
|
||||||
|
assert!(nulls.is_ok());
|
||||||
|
let nulls = match nulls {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(nulls, b"{\"transaction\":[\"AQID\",\"base64\"],\"meta\":null,\"version\":null,\"transactionIndex\":null}");
|
||||||
|
assert_ne!(omitted, nulls);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_non_base64_transaction_shapes_are_rejected() {
|
||||||
|
let base58 = ksp_onchain_transport_lib::SolanaEncodedTransaction::Binary {
|
||||||
|
data: "1111".to_owned(),
|
||||||
|
encoding: ksp_onchain_transport_lib::SolanaTransactionBinaryEncoding::Base58,
|
||||||
|
};
|
||||||
|
let legacy = ksp_onchain_transport_lib::SolanaEncodedTransaction::LegacyBinary("1111".to_owned());
|
||||||
|
let json = ksp_onchain_transport_lib::SolanaEncodedTransaction::Json(serde_json::json!({"message": {}}));
|
||||||
|
let meta = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let version = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
let transaction_index = ksp_onchain_transport_lib::SolanaWireField::Omitted;
|
||||||
|
for transaction in [&base58, &legacy, &json] {
|
||||||
|
let result = super::canonical_payload_bytes(&fields(transaction, &meta, &version, &transaction_index, std::option::Option::None));
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_negative_and_unrepresentable_block_times_are_terminal_conversion_errors() {
|
||||||
|
let negative = super::convert_block_time(std::option::Option::Some(-1));
|
||||||
|
assert!(negative.is_err());
|
||||||
|
let oversized = super::convert_block_time(std::option::Option::Some(i64::MAX));
|
||||||
|
assert!(oversized.is_err());
|
||||||
|
let absent = super::convert_block_time(std::option::Option::None);
|
||||||
|
assert!(matches!(absent, std::result::Result::Ok(std::option::Option::None)));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_observation_key_is_deterministic_and_endpoint_specific() {
|
||||||
|
let signature = match signature_text() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let request = match request("devnet", signature.clone()) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let candidate = match candidate("devnet", signature) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let reference = match super::canonical_reference(&request, &candidate) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let first = super::observation_key(&request, &reference, "provider", "endpoint-a");
|
||||||
|
let same = super::observation_key(&request, &reference, "provider", "endpoint-a");
|
||||||
|
let other_endpoint = super::observation_key(&request, &reference, "provider", "endpoint-b");
|
||||||
|
let other_provider = super::observation_key(&request, &reference, "provider-2", "endpoint-a");
|
||||||
|
assert_eq!(first, same);
|
||||||
|
assert_ne!(first, other_endpoint);
|
||||||
|
assert_ne!(first, other_provider);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_candidate_network_mismatch_is_rejected_before_transport() {
|
||||||
|
let signature = match signature_text() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let request = match request("devnet", signature.clone()) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let candidate = match candidate("mainnet", signature) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let reference = super::canonical_reference(&request, &candidate);
|
||||||
|
assert!(reference.is_err());
|
||||||
|
if let std::result::Result::Err(error) = reference {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_missing_outcome_contains_only_network_scoped_reference() {
|
||||||
|
let signature = match signature_text() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let request = match request("devnet", signature.clone()) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let candidate = match candidate("devnet", signature) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let reference = match super::canonical_reference(&request, &candidate) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let outcome = crate::BackfillHydrationOutcome::Missing(reference);
|
||||||
|
assert!(outcome.is_missing());
|
||||||
|
assert_eq!(outcome.reference().network().as_str(), "devnet");
|
||||||
|
assert_eq!(outcome.reference().signature().as_bytes(), &[0_u8; 64]);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_006_fix_001_raw_acquisition_uses_one_private_indirection() {
|
||||||
|
assert_eq!(std::mem::size_of::<crate::BackfillRawAcquisition>(), std::mem::size_of::<usize>(),);
|
||||||
|
return;
|
||||||
|
}
|
||||||
412
crates/ksp-job-backfill-lib/unit_tests/discovery.rs
Normal file
412
crates/ksp-job-backfill-lib/unit_tests/discovery.rs
Normal file
@@ -0,0 +1,412 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/discovery.rs
|
||||||
|
// version: 6
|
||||||
|
|
||||||
|
#[derive(Clone, Debug, Eq, PartialEq)]
|
||||||
|
struct PageCall {
|
||||||
|
before: std::option::Option<std::string::String>,
|
||||||
|
until: std::option::Option<std::string::String>,
|
||||||
|
limit: std::option::Option<usize>,
|
||||||
|
commitment: std::option::Option<ksp_onchain_transport_lib::SolanaCommitment>,
|
||||||
|
min_context_slot: std::option::Option<u64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FakeSource {
|
||||||
|
pages: std::sync::Mutex<std::collections::VecDeque<std::vec::Vec<super::SignaturePageEntry>>>,
|
||||||
|
calls: std::sync::Mutex<std::vec::Vec<PageCall>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct PendingSource {
|
||||||
|
calls: std::sync::atomic::AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl PendingSource {
|
||||||
|
fn new() -> Self {
|
||||||
|
return Self { calls: std::sync::atomic::AtomicUsize::new(0) };
|
||||||
|
}
|
||||||
|
|
||||||
|
fn calls(&self) -> usize {
|
||||||
|
return self.calls.load(std::sync::atomic::Ordering::Acquire);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl super::SignaturePageSource for PendingSource {
|
||||||
|
fn fetch_signature_page<'a>(
|
||||||
|
&'a self,
|
||||||
|
_role: &'a ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
_address: &'a ksp_core_lib::Pubkey,
|
||||||
|
_config: ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig,
|
||||||
|
) -> super::SignaturePageFuture<'a> {
|
||||||
|
self.calls.fetch_add(1, std::sync::atomic::Ordering::AcqRel);
|
||||||
|
return std::boxed::Box::pin(async {
|
||||||
|
return std::future::pending::<ksp_core_lib::Result<std::vec::Vec<super::SignaturePageEntry>>>().await;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakeSource {
|
||||||
|
fn new(pages: std::vec::Vec<std::vec::Vec<super::SignaturePageEntry>>) -> Self {
|
||||||
|
return Self { pages: std::sync::Mutex::new(pages.into()), calls: std::sync::Mutex::new(std::vec::Vec::new()) };
|
||||||
|
}
|
||||||
|
|
||||||
|
fn calls(&self) -> std::vec::Vec<PageCall> {
|
||||||
|
let guard = self.calls.lock();
|
||||||
|
return match guard {
|
||||||
|
std::result::Result::Ok(value) => value.clone(),
|
||||||
|
std::result::Result::Err(_) => std::vec::Vec::new(),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl super::SignaturePageSource for FakeSource {
|
||||||
|
fn fetch_signature_page<'a>(
|
||||||
|
&'a self,
|
||||||
|
_role: &'a ksp_onchain_transport_lib::HttpRoleName,
|
||||||
|
_address: &'a ksp_core_lib::Pubkey,
|
||||||
|
config: ksp_onchain_transport_lib::SolanaSignaturesForAddressConfig,
|
||||||
|
) -> super::SignaturePageFuture<'a> {
|
||||||
|
let call = PageCall {
|
||||||
|
before: config.before().map(str::to_owned),
|
||||||
|
until: config.until().map(str::to_owned),
|
||||||
|
limit: config.limit(),
|
||||||
|
commitment: config.commitment(),
|
||||||
|
min_context_slot: config.min_context_slot(),
|
||||||
|
};
|
||||||
|
let calls_result = self.calls.lock();
|
||||||
|
match calls_result {
|
||||||
|
std::result::Result::Ok(mut calls) => calls.push(call),
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
return std::boxed::Box::pin(async {
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
crate::ERROR_CODE_BACKFILL_DISCOVERY_INVALID,
|
||||||
|
"test call recorder lock poisoned",
|
||||||
|
));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
}
|
||||||
|
let pages_result = self.pages.lock();
|
||||||
|
let page = match pages_result {
|
||||||
|
std::result::Result::Ok(mut pages) => match pages.pop_front() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => std::vec::Vec::new(),
|
||||||
|
},
|
||||||
|
std::result::Result::Err(_) => {
|
||||||
|
return std::boxed::Box::pin(async {
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_DISCOVERY_INVALID, "test page queue lock poisoned"));
|
||||||
|
});
|
||||||
|
},
|
||||||
|
};
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
return std::result::Result::Ok(page);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn page_entry(character: char, slot: u64) -> super::SignaturePageEntry {
|
||||||
|
return super::SignaturePageEntry { signature: character.to_string().repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES), slot };
|
||||||
|
}
|
||||||
|
|
||||||
|
fn signature(character: char) -> std::option::Option<crate::BackfillSignature> {
|
||||||
|
return match crate::BackfillSignature::new(character.to_string().repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES)) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn request(scope: crate::BackfillScope, page_size: usize, max_pages: usize, max_candidates: usize) -> std::option::Option<crate::BackfillRequest> {
|
||||||
|
let job_id = match ksp_job_api::JobId::new("backfill:discovery-test") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let result = crate::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
page_size,
|
||||||
|
max_pages,
|
||||||
|
max_candidates,
|
||||||
|
1,
|
||||||
|
std::option::Option::Some(42),
|
||||||
|
);
|
||||||
|
return match result {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn candidate_signatures(discovery: &crate::BackfillDiscovery) -> std::vec::Vec<std::string::String> {
|
||||||
|
let mut signatures = std::vec::Vec::with_capacity(discovery.candidates().len());
|
||||||
|
for candidate in discovery.candidates() {
|
||||||
|
signatures.push(candidate.identity().signature().as_str().to_owned());
|
||||||
|
}
|
||||||
|
return signatures;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_005_latest_paginates_newest_first_and_deduplicates_pages_stably() {
|
||||||
|
let scope = crate::BackfillScope::latest_address(ksp_core_lib::Pubkey::new_from_array([1_u8; 32]));
|
||||||
|
let request = match request(scope, 3, 4, 10) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![
|
||||||
|
std::vec![page_entry('6', 60), page_entry('5', 50), page_entry('5', 50)],
|
||||||
|
std::vec![page_entry('4', 40), page_entry('3', 30)],
|
||||||
|
]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
let discovery = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(candidate_signatures(&discovery), std::vec!["6".repeat(64), "5".repeat(64), "4".repeat(64), "3".repeat(64)]);
|
||||||
|
assert_eq!(discovery.pages_fetched(), 2);
|
||||||
|
assert_eq!(discovery.boundary(), crate::BackfillDiscoveryBoundary::RpcBoundary);
|
||||||
|
assert!(!discovery.is_partial());
|
||||||
|
let calls = source.calls();
|
||||||
|
assert_eq!(calls.len(), 2);
|
||||||
|
assert_eq!(calls[0].before, std::option::Option::None);
|
||||||
|
let expected_cursor = "5".repeat(64);
|
||||||
|
assert_eq!(calls[1].before.as_deref(), std::option::Option::Some(expected_cursor.as_str()));
|
||||||
|
assert_eq!(calls[0].until, std::option::Option::None);
|
||||||
|
assert_eq!(calls[0].limit, std::option::Option::Some(3));
|
||||||
|
assert_eq!(calls[0].commitment, std::option::Option::Some(ksp_onchain_transport_lib::SolanaCommitment::Confirmed));
|
||||||
|
assert_eq!(calls[0].min_context_slot, std::option::Option::Some(42));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_005_before_uses_exclusive_anchor_then_advances_rpc_cursor() {
|
||||||
|
let anchor = match signature('7') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = crate::BackfillScope::before_address(ksp_core_lib::Pubkey::new_from_array([2_u8; 32]), anchor.clone());
|
||||||
|
let request = match request(scope, 2, 3, 5) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![std::vec![page_entry('6', 60), page_entry('5', 50)], std::vec![page_entry('4', 40)]]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
let discovery = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(candidate_signatures(&discovery), std::vec!["6".repeat(64), "5".repeat(64), "4".repeat(64)]);
|
||||||
|
let calls = source.calls();
|
||||||
|
assert_eq!(calls.len(), 2);
|
||||||
|
assert_eq!(calls[0].before.as_deref(), std::option::Option::Some(anchor.as_str()));
|
||||||
|
let expected_cursor = "5".repeat(64);
|
||||||
|
assert_eq!(calls[1].before.as_deref(), std::option::Option::Some(expected_cursor.as_str()));
|
||||||
|
assert_eq!(calls[0].until, std::option::Option::None);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_005_after_keeps_only_nearest_newer_window_and_preserves_rpc_order() {
|
||||||
|
let anchor = match signature('1') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = crate::BackfillScope::after_address(ksp_core_lib::Pubkey::new_from_array([3_u8; 32]), anchor.clone());
|
||||||
|
let request = match request(scope, 3, 3, 3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![
|
||||||
|
std::vec![page_entry('7', 70), page_entry('6', 60), page_entry('5', 50)],
|
||||||
|
std::vec![page_entry('4', 40), page_entry('3', 30)],
|
||||||
|
]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
let discovery = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(candidate_signatures(&discovery), std::vec!["5".repeat(64), "4".repeat(64), "3".repeat(64)]);
|
||||||
|
assert_eq!(discovery.boundary(), crate::BackfillDiscoveryBoundary::RpcBoundary);
|
||||||
|
assert!(!discovery.is_partial());
|
||||||
|
let calls = source.calls();
|
||||||
|
assert_eq!(calls.len(), 2);
|
||||||
|
assert_eq!(calls[0].until.as_deref(), std::option::Option::Some(anchor.as_str()));
|
||||||
|
assert_eq!(calls[1].until.as_deref(), std::option::Option::Some(anchor.as_str()));
|
||||||
|
let expected_cursor = "5".repeat(64);
|
||||||
|
assert_eq!(calls[1].before.as_deref(), std::option::Option::Some(expected_cursor.as_str()));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_005_after_page_bound_is_partial_and_does_not_claim_anchor_completion() {
|
||||||
|
let anchor = match signature('1') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = crate::BackfillScope::after_address(ksp_core_lib::Pubkey::new_from_array([4_u8; 32]), anchor);
|
||||||
|
let request = match request(scope, 2, 2, 3) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![std::vec![page_entry('7', 70), page_entry('6', 60)], std::vec![page_entry('5', 50), page_entry('4', 40)],]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
let discovery = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(candidate_signatures(&discovery), std::vec!["6".repeat(64), "5".repeat(64), "4".repeat(64)]);
|
||||||
|
assert_eq!(discovery.boundary(), crate::BackfillDiscoveryBoundary::AfterAnchorNotReached);
|
||||||
|
assert!(discovery.is_partial());
|
||||||
|
assert_eq!(discovery.pages_fetched(), 2);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_005_latest_page_bound_is_partial_when_full_pages_leave_more_history_possible() {
|
||||||
|
let scope = crate::BackfillScope::latest_address(ksp_core_lib::Pubkey::new_from_array([5_u8; 32]));
|
||||||
|
let request = match request(scope, 2, 1, 5) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![std::vec![page_entry('7', 70), page_entry('6', 60)]]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
let discovery = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(discovery.boundary(), crate::BackfillDiscoveryBoundary::PageLimit);
|
||||||
|
assert!(discovery.is_partial());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_005_explicit_scope_never_calls_transport_and_preserves_network_scoped_identity() {
|
||||||
|
let first = match signature('2') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match signature('3') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope_result = crate::BackfillScope::explicit_signatures(std::vec![first.clone(), second.clone(), first]);
|
||||||
|
let scope = match scope_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let job_id = match ksp_job_api::JobId::new("backfill:explicit") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("synthetic") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let request_result = crate::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network.clone(),
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("unused"),
|
||||||
|
crate::BackfillCommitment::Finalized,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
10,
|
||||||
|
1,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
let request = match request_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec::Vec::new());
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
let discovery = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(discovery.boundary(), crate::BackfillDiscoveryBoundary::ExplicitInput);
|
||||||
|
assert_eq!(discovery.pages_fetched(), 0);
|
||||||
|
assert_eq!(discovery.candidates().len(), 2);
|
||||||
|
assert_eq!(discovery.candidates()[0].identity().network(), &network);
|
||||||
|
assert!(source.calls().is_empty());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_008_before_resume_uses_checkpoint_cursor_instead_of_original_anchor() {
|
||||||
|
let anchor = match signature('8') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let resume = match signature('5') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope = crate::BackfillScope::before_address(ksp_core_lib::Pubkey::new_from_array([8_u8; 32]), anchor);
|
||||||
|
let request = match request(scope, 2, 2, 4) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let checkpoint = crate::BackfillCheckpoint::new(request.job_id().clone(), request.scope_fingerprint(), 2, std::option::Option::Some(resume.clone()));
|
||||||
|
let request = match request.with_checkpoint(checkpoint) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![std::vec![page_entry('4', 40)]]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
let calls = source.calls();
|
||||||
|
assert_eq!(calls.len(), 1);
|
||||||
|
assert_eq!(calls[0].before.as_deref(), std::option::Option::Some(resume.as_str()));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_008_latest_resume_restarts_from_current_latest_without_rpc_cursor() {
|
||||||
|
let scope = crate::BackfillScope::latest_address(ksp_core_lib::Pubkey::new_from_array([9_u8; 32]));
|
||||||
|
let request = match request(scope, 2, 2, 4) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let checkpoint = crate::BackfillCheckpoint::new(request.job_id().clone(), request.scope_fingerprint(), 2, std::option::Option::None);
|
||||||
|
let request = match request.with_checkpoint(checkpoint) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let source = FakeSource::new(std::vec![std::vec![page_entry('7', 70)]]);
|
||||||
|
let result = super::discover_with_source(&source, &request, std::option::Option::None).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
let calls = source.calls();
|
||||||
|
assert_eq!(calls.len(), 1);
|
||||||
|
assert_eq!(calls[0].before, std::option::Option::None);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_009_discovery_rpc_wait_is_cancelled_cooperatively() {
|
||||||
|
let scope = crate::BackfillScope::latest_address(ksp_core_lib::Pubkey::new_from_array([9_u8; 32]));
|
||||||
|
let request = match request(scope, 2, 2, 5) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let source = PendingSource::new();
|
||||||
|
let token = ksp_job_api::JobCancellationToken::new();
|
||||||
|
let (cancel_sender, cancel_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let cancellation = crate::BackfillCancellationSignal::new(token.clone(), cancel_receiver);
|
||||||
|
let discovery = super::discover_with_source(&source, &request, std::option::Option::Some(&cancellation));
|
||||||
|
let cancel = async {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert!(token.cancel());
|
||||||
|
assert!(cancel_sender.send(true).is_ok());
|
||||||
|
};
|
||||||
|
let (result, ()) = tokio::join!(discovery, cancel);
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_CANCELLED);
|
||||||
|
assert_eq!(source.calls(), 1);
|
||||||
|
return;
|
||||||
|
}
|
||||||
321
crates/ksp-job-backfill-lib/unit_tests/execution.rs
Normal file
321
crates/ksp-job-backfill-lib/unit_tests/execution.rs
Normal file
@@ -0,0 +1,321 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/execution.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum FakeDisposition {
|
||||||
|
Durable,
|
||||||
|
Missing,
|
||||||
|
Conflict,
|
||||||
|
Failure,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
struct FakePlan {
|
||||||
|
pending_polls: usize,
|
||||||
|
disposition: FakeDisposition,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FakeProcessor {
|
||||||
|
plans: std::vec::Vec<FakePlan>,
|
||||||
|
active: std::sync::atomic::AtomicUsize,
|
||||||
|
maximum: std::sync::atomic::AtomicUsize,
|
||||||
|
calls: std::sync::atomic::AtomicUsize,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakeProcessor {
|
||||||
|
fn new(plans: std::vec::Vec<FakePlan>) -> Self {
|
||||||
|
return Self {
|
||||||
|
plans,
|
||||||
|
active: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
maximum: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn maximum(&self) -> usize {
|
||||||
|
return self.maximum.load(std::sync::atomic::Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn calls(&self) -> usize {
|
||||||
|
return self.calls.load(std::sync::atomic::Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl super::CandidateProcessor for FakeProcessor {
|
||||||
|
fn process<'a>(&'a self, candidate: &'a crate::BackfillCandidate) -> super::CandidateProcessFuture<'a> {
|
||||||
|
self.calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
|
||||||
|
let first_byte = candidate.identity().signature().as_str().as_bytes()[0];
|
||||||
|
let plan_index = usize::from(first_byte.saturating_sub(b'1'));
|
||||||
|
let plan = match self.plans.get(plan_index) {
|
||||||
|
std::option::Option::Some(plan) => *plan,
|
||||||
|
std::option::Option::None => FakePlan { pending_polls: 0, disposition: FakeDisposition::Failure },
|
||||||
|
};
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
let mut pending_polls = plan.pending_polls;
|
||||||
|
let mut started = false;
|
||||||
|
std::future::poll_fn(|context| {
|
||||||
|
if !started {
|
||||||
|
started = true;
|
||||||
|
let active = self.active.fetch_add(1, std::sync::atomic::Ordering::SeqCst) + 1;
|
||||||
|
self.maximum.fetch_max(active, std::sync::atomic::Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
if pending_polls != 0 {
|
||||||
|
pending_polls -= 1;
|
||||||
|
context.waker().wake_by_ref();
|
||||||
|
return std::task::Poll::Pending;
|
||||||
|
}
|
||||||
|
self.active.fetch_sub(1, std::sync::atomic::Ordering::SeqCst);
|
||||||
|
return std::task::Poll::Ready(());
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
return planned_outcome(candidate, plan.disposition);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn planned_outcome(candidate: &crate::BackfillCandidate, disposition: FakeDisposition) -> ksp_core_lib::Result<crate::BackfillPersistenceOutcome> {
|
||||||
|
if matches!(disposition, FakeDisposition::Failure) {
|
||||||
|
return std::result::Result::Err(ksp_core_lib::Error::new(crate::ERROR_CODE_BACKFILL_RAW_CONVERSION_INVALID, "planned candidate failure"));
|
||||||
|
}
|
||||||
|
let fill = candidate.identity().signature().as_str().as_bytes()[0];
|
||||||
|
let reference =
|
||||||
|
ksp_store_lib::RawTransactionReference::new(candidate.identity().network().clone(), ksp_store_lib::RawTransactionSignature::new([fill; 64]));
|
||||||
|
let (entity, observation) = match disposition {
|
||||||
|
FakeDisposition::Durable => (crate::BackfillEntityPersistence::AlreadyPresent, crate::BackfillObservationPersistence::AlreadyPresent),
|
||||||
|
FakeDisposition::Missing => (crate::BackfillEntityPersistence::Missing, crate::BackfillObservationPersistence::NotApplicable),
|
||||||
|
FakeDisposition::Conflict => (crate::BackfillEntityPersistence::Conflict, crate::BackfillObservationPersistence::NotRecorded),
|
||||||
|
FakeDisposition::Failure => return std::result::Result::Err(super::execution_error("test.disposition")),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(crate::BackfillPersistenceOutcome::new(reference, entity, observation));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn signature(character: char) -> std::option::Option<crate::BackfillSignature> {
|
||||||
|
return crate::BackfillSignature::new(character.to_string().repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES)).ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn explicit_request(characters: &[char], concurrency: usize) -> std::option::Option<crate::BackfillRequest> {
|
||||||
|
let mut signatures = std::vec::Vec::with_capacity(characters.len());
|
||||||
|
for character in characters {
|
||||||
|
let value = match signature(*character) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
signatures.push(value);
|
||||||
|
}
|
||||||
|
let scope = match crate::BackfillScope::explicit_signatures(signatures) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let job_id = match ksp_job_api::JobId::new("backfill:execution-test") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return crate::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
characters.len(),
|
||||||
|
concurrency,
|
||||||
|
std::option::Option::None,
|
||||||
|
)
|
||||||
|
.ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
fn discovery(request: &crate::BackfillRequest, characters: &[char]) -> std::option::Option<crate::BackfillDiscovery> {
|
||||||
|
let mut candidates = std::vec::Vec::with_capacity(characters.len());
|
||||||
|
for character in characters {
|
||||||
|
let signature = match signature(*character) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let identity = crate::BackfillCandidateIdentity::new(request.network().clone(), signature);
|
||||||
|
candidates.push(crate::BackfillCandidate::new(identity, std::option::Option::None));
|
||||||
|
}
|
||||||
|
return std::option::Option::Some(crate::BackfillDiscovery::new(
|
||||||
|
request.network().clone(),
|
||||||
|
request.scope_fingerprint(),
|
||||||
|
candidates,
|
||||||
|
0,
|
||||||
|
crate::BackfillDiscoveryBoundary::ExplicitInput,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_008_execution_is_bounded_and_reconciles_out_of_order_durable_completions() {
|
||||||
|
let request = match explicit_request(&['1', '2', '3', '4'], 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let discovery = match discovery(&request, &['1', '2', '3', '4']) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let processor = FakeProcessor::new(std::vec![
|
||||||
|
FakePlan { pending_polls: 6, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 2, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
]);
|
||||||
|
let result = super::execute_with_processor(&processor, &request, &discovery, std::option::Option::None, std::option::Option::None).await;
|
||||||
|
let batch = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(batch.admitted_count(), 4);
|
||||||
|
assert_eq!(batch.finished_count(), 4);
|
||||||
|
assert_eq!(batch.durable_count(), 4);
|
||||||
|
assert_eq!(batch.hole_count(), 0);
|
||||||
|
assert_eq!(batch.maximum_in_flight(), 2);
|
||||||
|
assert_eq!(processor.maximum(), 2);
|
||||||
|
assert_eq!(batch.local_contiguous_completed(), 4);
|
||||||
|
assert_eq!(batch.checkpoint().completed_prefix(), 4);
|
||||||
|
assert!(!batch.is_partial());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_008_missing_is_non_fatal_but_blocks_frontier_while_later_candidates_continue() {
|
||||||
|
let request = match explicit_request(&['1', '2', '3'], 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let discovery = match discovery(&request, &['1', '2', '3']) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let processor = FakeProcessor::new(std::vec![
|
||||||
|
FakePlan { pending_polls: 1, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Missing },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
]);
|
||||||
|
let batch = match super::execute_with_processor(&processor, &request, &discovery, std::option::Option::None, std::option::Option::None).await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(batch.admitted_count(), 3);
|
||||||
|
assert_eq!(batch.finished_count(), 3);
|
||||||
|
assert_eq!(batch.durable_count(), 2);
|
||||||
|
assert_eq!(batch.hole_count(), 1);
|
||||||
|
assert_eq!(batch.local_contiguous_completed(), 1);
|
||||||
|
assert_eq!(batch.checkpoint().completed_prefix(), 1);
|
||||||
|
assert_eq!(batch.failure_code(), std::option::Option::None);
|
||||||
|
assert!(batch.is_partial());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_008_conflict_stops_new_admissions_and_drains_already_in_flight_work() {
|
||||||
|
let request = match explicit_request(&['1', '2', '3', '4'], 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let discovery = match discovery(&request, &['1', '2', '3', '4']) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let processor = FakeProcessor::new(std::vec![
|
||||||
|
FakePlan { pending_polls: 6, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Conflict },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
]);
|
||||||
|
let batch = match super::execute_with_processor(&processor, &request, &discovery, std::option::Option::None, std::option::Option::None).await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(processor.calls(), 2);
|
||||||
|
assert_eq!(batch.admitted_count(), 2);
|
||||||
|
assert_eq!(batch.finished_count(), 2);
|
||||||
|
assert_eq!(batch.durable_count(), 1);
|
||||||
|
assert_eq!(batch.hole_count(), 1);
|
||||||
|
assert_eq!(batch.local_contiguous_completed(), 1);
|
||||||
|
assert_eq!(batch.failure_code(), std::option::Option::Some(ksp_store_lib::ERROR_CODE_RAW_CONFLICT));
|
||||||
|
assert!(batch.is_partial());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_008_explicit_resume_skips_only_the_checkpointed_contiguous_prefix() {
|
||||||
|
let request = match explicit_request(&['1', '2', '3', '4'], 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let checkpoint = crate::BackfillCheckpoint::new(request.job_id().clone(), request.scope_fingerprint(), 2, std::option::Option::None);
|
||||||
|
let request = match request.with_checkpoint(checkpoint) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let discovery = match discovery(&request, &['1', '2', '3', '4']) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let processor = FakeProcessor::new(std::vec![
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Failure },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Failure },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
]);
|
||||||
|
let batch = match super::execute_with_processor(&processor, &request, &discovery, std::option::Option::None, std::option::Option::None).await {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(processor.calls(), 2);
|
||||||
|
assert_eq!(batch.admitted_count(), 2);
|
||||||
|
assert_eq!(batch.local_contiguous_completed(), 4);
|
||||||
|
assert_eq!(batch.checkpoint().completed_prefix(), 4);
|
||||||
|
assert_eq!(batch.failure_code(), std::option::Option::None);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_009_cancellation_stops_admission_and_drains_already_admitted_candidate_work() {
|
||||||
|
let request = match explicit_request(&['1', '2', '3', '4'], 2) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let discovery = match discovery(&request, &['1', '2', '3', '4']) {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let processor = FakeProcessor::new(std::vec![
|
||||||
|
FakePlan { pending_polls: 12, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 12, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
FakePlan { pending_polls: 0, disposition: FakeDisposition::Durable },
|
||||||
|
]);
|
||||||
|
let token = ksp_job_api::JobCancellationToken::new();
|
||||||
|
let (cancel_sender, cancel_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let signal = crate::BackfillCancellationSignal::new(token.clone(), cancel_receiver);
|
||||||
|
let execution = super::execute_with_processor(&processor, &request, &discovery, std::option::Option::Some(&signal), std::option::Option::None);
|
||||||
|
let cancellation = async {
|
||||||
|
loop {
|
||||||
|
if processor.calls() >= 2 {
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
}
|
||||||
|
assert!(token.cancel());
|
||||||
|
assert!(cancel_sender.send(true).is_ok());
|
||||||
|
};
|
||||||
|
let (result, ()) = tokio::join!(execution, cancellation);
|
||||||
|
let batch = match result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(processor.calls(), 2);
|
||||||
|
assert_eq!(batch.admitted_count(), 2);
|
||||||
|
assert_eq!(batch.finished_count(), 2);
|
||||||
|
assert_eq!(batch.durable_count(), 2);
|
||||||
|
assert_eq!(batch.cancelled_count(), 0);
|
||||||
|
assert!(batch.was_cancelled());
|
||||||
|
assert_eq!(batch.checkpoint().completed_prefix(), 2);
|
||||||
|
return;
|
||||||
|
}
|
||||||
348
crates/ksp-job-backfill-lib/unit_tests/persistence.rs
Normal file
348
crates/ksp-job-backfill-lib/unit_tests/persistence.rs
Normal file
@@ -0,0 +1,348 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/persistence.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
#[derive(Clone, Copy)]
|
||||||
|
enum FakeResponse {
|
||||||
|
Outcome(ksp_store_lib::RawAcquisitionWriteOutcome),
|
||||||
|
Conflict,
|
||||||
|
Failure,
|
||||||
|
}
|
||||||
|
|
||||||
|
struct FakePersistencePort {
|
||||||
|
network: ksp_store_lib::RawNetworkId,
|
||||||
|
responses: std::sync::Mutex<std::collections::VecDeque<FakeResponse>>,
|
||||||
|
calls: std::sync::atomic::AtomicUsize,
|
||||||
|
normal_mode_only: std::sync::atomic::AtomicBool,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakePersistencePort {
|
||||||
|
fn new(network: ksp_store_lib::RawNetworkId, responses: &[FakeResponse]) -> Self {
|
||||||
|
return Self {
|
||||||
|
network,
|
||||||
|
responses: std::sync::Mutex::new(responses.iter().copied().collect()),
|
||||||
|
calls: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
normal_mode_only: std::sync::atomic::AtomicBool::new(true),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn calls(&self) -> usize {
|
||||||
|
return self.calls.load(std::sync::atomic::Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn used_only_normal_mode(&self) -> bool {
|
||||||
|
return self.normal_mode_only.load(std::sync::atomic::Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl super::RawTransactionPersistencePort for FakePersistencePort {
|
||||||
|
fn network_matches(&self, network: &ksp_store_lib::RawNetworkId) -> bool {
|
||||||
|
return &self.network == network;
|
||||||
|
}
|
||||||
|
|
||||||
|
fn persist_acquisition<'a>(
|
||||||
|
&'a self,
|
||||||
|
transaction: ksp_store_lib::RawTransaction,
|
||||||
|
observation: ksp_store_lib::RawTransactionObservation,
|
||||||
|
mode: ksp_store_lib::RawTransactionAcquisitionMode,
|
||||||
|
) -> ksp_store_lib::StoreApiFuture<'a, ksp_store_lib::Result<ksp_store_lib::RawAcquisitionWriteOutcome>> {
|
||||||
|
self.calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
|
||||||
|
if mode != ksp_store_lib::RawTransactionAcquisitionMode::Normal {
|
||||||
|
self.normal_mode_only.store(false, std::sync::atomic::Ordering::SeqCst);
|
||||||
|
}
|
||||||
|
let result = if transaction.reference() != observation.transaction() {
|
||||||
|
std::result::Result::Err(ksp_core_lib::Error::new(
|
||||||
|
ksp_core_lib::ErrorCode::new("test", "reference_mismatch"),
|
||||||
|
"fake persistence reference mismatch",
|
||||||
|
))
|
||||||
|
} else {
|
||||||
|
let response = match self.responses.lock() {
|
||||||
|
std::result::Result::Ok(mut responses) => responses.pop_front(),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
match response {
|
||||||
|
std::option::Option::Some(FakeResponse::Outcome(outcome)) => std::result::Result::Ok(outcome),
|
||||||
|
std::option::Option::Some(FakeResponse::Conflict) => {
|
||||||
|
std::result::Result::Err(ksp_core_lib::Error::new(ksp_store_lib::ERROR_CODE_RAW_CONFLICT, "fake canonical content conflict"))
|
||||||
|
},
|
||||||
|
std::option::Option::Some(FakeResponse::Failure) | std::option::Option::None => {
|
||||||
|
std::result::Result::Err(ksp_core_lib::Error::new(ksp_core_lib::ErrorCode::new("test", "store_failure"), "fake Store failure"))
|
||||||
|
},
|
||||||
|
}
|
||||||
|
};
|
||||||
|
return std::boxed::Box::pin(async move {
|
||||||
|
return result;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn raw_network(value: &str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
return match ksp_store_lib::RawNetworkId::new(value) {
|
||||||
|
std::result::Result::Ok(network) => std::option::Option::Some(network),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn raw_reference(network: &str, signature_byte: u8) -> std::option::Option<ksp_store_lib::RawTransactionReference> {
|
||||||
|
let network = match raw_network(network) {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return std::option::Option::Some(ksp_store_lib::RawTransactionReference::new(network, ksp_store_lib::RawTransactionSignature::new([signature_byte; 64])));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn raw_acquisition_parts(
|
||||||
|
network: &str,
|
||||||
|
signature_byte: u8,
|
||||||
|
observation_byte: u8,
|
||||||
|
) -> std::option::Option<(ksp_store_lib::RawTransactionReference, ksp_store_lib::RawTransaction, ksp_store_lib::RawTransactionObservation)> {
|
||||||
|
let reference = match raw_reference(network, signature_byte) {
|
||||||
|
std::option::Option::Some(reference) => reference,
|
||||||
|
std::option::Option::None => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let format_id = match ksp_store_lib::RawFormatId::new(crate::RAW_TRANSACTION_FORMAT_ID) {
|
||||||
|
std::result::Result::Ok(format_id) => format_id,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let payload = ksp_store_lib::RawPayload::try_new(
|
||||||
|
format_id,
|
||||||
|
crate::RAW_TRANSACTION_FORMAT_VERSION,
|
||||||
|
std::vec![signature_byte].into_boxed_slice(),
|
||||||
|
ksp_store_lib::RawContentHash::new([signature_byte; 32]),
|
||||||
|
);
|
||||||
|
let payload = match payload {
|
||||||
|
std::result::Result::Ok(payload) => payload,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let received_at = match ksp_store_lib::RawTimestamp::from_unix_millis(1_700_000_000_000) {
|
||||||
|
std::result::Result::Ok(received_at) => received_at,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provider = match ksp_store_lib::RawProvenanceCode::new("provider") {
|
||||||
|
std::result::Result::Ok(provider) => provider,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let protocol = match ksp_store_lib::RawProvenanceCode::new("solana.http.json_rpc") {
|
||||||
|
std::result::Result::Ok(protocol) => protocol,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let method = match ksp_store_lib::RawProvenanceCode::new("getTransaction") {
|
||||||
|
std::result::Result::Ok(method) => method,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let provenance = ksp_store_lib::RawAcquisitionProvenance::new(provider, protocol, method, ksp_store_lib::RawAcquisitionOrigin::Backfill, received_at);
|
||||||
|
let transaction = ksp_store_lib::RawTransaction::new(reference.clone(), 42, std::option::Option::None, payload);
|
||||||
|
let observation =
|
||||||
|
ksp_store_lib::RawTransactionObservation::new(ksp_store_lib::RawObservationKey::new([observation_byte; 32]), reference.clone(), provenance);
|
||||||
|
return std::option::Option::Some((reference, transaction, observation));
|
||||||
|
}
|
||||||
|
|
||||||
|
fn store_outcome(
|
||||||
|
entity: ksp_store_lib::RawEntityWriteOutcome,
|
||||||
|
observation: ksp_store_lib::RawObservationWriteOutcome,
|
||||||
|
) -> ksp_store_lib::RawAcquisitionWriteOutcome {
|
||||||
|
return ksp_store_lib::RawAcquisitionWriteOutcome::new(entity, observation);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_missing_skips_store_and_preserves_network_scoped_identity() {
|
||||||
|
let reference = match raw_reference("devnet", 1) {
|
||||||
|
std::option::Option::Some(reference) => reference,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, &[]);
|
||||||
|
let result = super::persist_hydration_with_port(&port, crate::BackfillHydrationOutcome::Missing(reference.clone())).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
if let std::result::Result::Ok(result) = result {
|
||||||
|
assert_eq!(result.reference(), &reference);
|
||||||
|
assert_eq!(result.entity(), crate::BackfillEntityPersistence::Missing);
|
||||||
|
assert_eq!(result.observation(), crate::BackfillObservationPersistence::NotApplicable);
|
||||||
|
}
|
||||||
|
assert_eq!(port.calls(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_store_network_mismatch_is_rejected_before_any_write() {
|
||||||
|
let reference = match raw_reference("devnet", 2) {
|
||||||
|
std::option::Option::Some(reference) => reference,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("mainnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, &[]);
|
||||||
|
let result = super::persist_hydration_with_port(&port, crate::BackfillHydrationOutcome::Missing(reference)).await;
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_PERSISTENCE_INVALID);
|
||||||
|
}
|
||||||
|
assert_eq!(port.calls(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_atomic_insert_maps_entity_and_observation_without_second_write() {
|
||||||
|
let (reference, transaction, observation) = match raw_acquisition_parts("devnet", 3, 13) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let response = store_outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::Inserted);
|
||||||
|
let port = FakePersistencePort::new(network, &[FakeResponse::Outcome(response)]);
|
||||||
|
let result = super::persist_available_with_port(&port, reference.clone(), transaction, observation).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
if let std::result::Result::Ok(result) = result {
|
||||||
|
assert_eq!(result.reference(), &reference);
|
||||||
|
assert_eq!(result.entity(), crate::BackfillEntityPersistence::Inserted);
|
||||||
|
assert_eq!(result.observation(), crate::BackfillObservationPersistence::Inserted);
|
||||||
|
}
|
||||||
|
assert_eq!(port.calls(), 1);
|
||||||
|
assert!(port.used_only_normal_mode());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_existing_entity_distinguishes_new_from_idempotent_observation() {
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first = store_outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::Inserted);
|
||||||
|
let second = store_outcome(ksp_store_lib::RawEntityWriteOutcome::AlreadyPresent, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent);
|
||||||
|
let port = FakePersistencePort::new(network, &[FakeResponse::Outcome(first), FakeResponse::Outcome(second)]);
|
||||||
|
let (reference, transaction, observation) = match raw_acquisition_parts("devnet", 4, 14) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let result = super::persist_available_with_port(&port, reference, transaction, observation).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
if let std::result::Result::Ok(result) = result {
|
||||||
|
assert_eq!(result.entity(), crate::BackfillEntityPersistence::AlreadyPresent);
|
||||||
|
assert_eq!(result.observation(), crate::BackfillObservationPersistence::Inserted);
|
||||||
|
}
|
||||||
|
let (reference, transaction, observation) = match raw_acquisition_parts("devnet", 4, 14) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let rerun = super::persist_available_with_port(&port, reference, transaction, observation).await;
|
||||||
|
assert!(rerun.is_ok());
|
||||||
|
if let std::result::Result::Ok(rerun) = rerun {
|
||||||
|
assert_eq!(rerun.entity(), crate::BackfillEntityPersistence::AlreadyPresent);
|
||||||
|
assert_eq!(rerun.observation(), crate::BackfillObservationPersistence::AlreadyPresent);
|
||||||
|
}
|
||||||
|
assert_eq!(port.calls(), 2);
|
||||||
|
assert!(port.used_only_normal_mode());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_normal_backfill_respects_purged_tombstone_without_observation() {
|
||||||
|
let (reference, transaction, observation) = match raw_acquisition_parts("devnet", 5, 15) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let response = store_outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::NotRecorded);
|
||||||
|
let port = FakePersistencePort::new(network, &[FakeResponse::Outcome(response)]);
|
||||||
|
let result = super::persist_available_with_port(&port, reference, transaction, observation).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
if let std::result::Result::Ok(result) = result {
|
||||||
|
assert_eq!(result.entity(), crate::BackfillEntityPersistence::SkippedPurged);
|
||||||
|
assert_eq!(result.observation(), crate::BackfillObservationPersistence::NotRecorded);
|
||||||
|
}
|
||||||
|
assert!(port.used_only_normal_mode());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_store_content_conflict_is_explicit_and_not_idempotent_success() {
|
||||||
|
let (reference, transaction, observation) = match raw_acquisition_parts("devnet", 6, 16) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, &[FakeResponse::Conflict]);
|
||||||
|
let result = super::persist_available_with_port(&port, reference, transaction, observation).await;
|
||||||
|
assert!(result.is_ok());
|
||||||
|
if let std::result::Result::Ok(result) = result {
|
||||||
|
assert_eq!(result.entity(), crate::BackfillEntityPersistence::Conflict);
|
||||||
|
assert_eq!(result.observation(), crate::BackfillObservationPersistence::NotRecorded);
|
||||||
|
assert_ne!(result.entity(), crate::BackfillEntityPersistence::AlreadyPresent);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_non_conflict_store_failure_propagates_unchanged() {
|
||||||
|
let (reference, transaction, observation) = match raw_acquisition_parts("devnet", 7, 17) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, &[FakeResponse::Failure]);
|
||||||
|
let result = super::persist_available_with_port(&port, reference, transaction, observation).await;
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), ksp_core_lib::ErrorCode::new("test", "store_failure"));
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_007_normal_mode_rejects_impossible_store_outcome_combinations() {
|
||||||
|
let reference = match raw_reference("devnet", 8) {
|
||||||
|
std::option::Option::Some(reference) => reference,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let impossible = [
|
||||||
|
store_outcome(ksp_store_lib::RawEntityWriteOutcome::Rehydrated, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
store_outcome(ksp_store_lib::RawEntityWriteOutcome::Inserted, ksp_store_lib::RawObservationWriteOutcome::AlreadyPresent),
|
||||||
|
store_outcome(ksp_store_lib::RawEntityWriteOutcome::SkippedPurged, ksp_store_lib::RawObservationWriteOutcome::Inserted),
|
||||||
|
];
|
||||||
|
for outcome in impossible {
|
||||||
|
let result = super::map_store_outcome(reference.clone(), outcome);
|
||||||
|
assert!(result.is_err());
|
||||||
|
if let std::result::Result::Err(error) = result {
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_PERSISTENCE_INVALID);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_007_mismatched_transaction_observation_reference_is_rejected_before_store() {
|
||||||
|
let (reference, transaction, _) = match raw_acquisition_parts("devnet", 9, 19) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let (_, _, observation) = match raw_acquisition_parts("devnet", 10, 20) {
|
||||||
|
std::option::Option::Some(parts) => parts,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let network = match raw_network("devnet") {
|
||||||
|
std::option::Option::Some(network) => network,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let port = FakePersistencePort::new(network, &[]);
|
||||||
|
let result = super::persist_available_with_port(&port, reference, transaction, observation).await;
|
||||||
|
assert!(result.is_err());
|
||||||
|
assert_eq!(port.calls(), 0);
|
||||||
|
return;
|
||||||
|
}
|
||||||
234
crates/ksp-job-backfill-lib/unit_tests/request.rs
Normal file
234
crates/ksp-job-backfill-lib/unit_tests/request.rs
Normal file
@@ -0,0 +1,234 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/request.rs
|
||||||
|
// version: 1
|
||||||
|
|
||||||
|
fn signature(character: char) -> std::option::Option<crate::BackfillSignature> {
|
||||||
|
return match crate::BackfillSignature::new(character.to_string().repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES)) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn network(value: &str) -> std::option::Option<ksp_store_lib::RawNetworkId> {
|
||||||
|
return match ksp_store_lib::RawNetworkId::new(value) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn job_id(value: &str) -> std::option::Option<ksp_job_api::JobId> {
|
||||||
|
return match ksp_job_api::JobId::new(value) {
|
||||||
|
std::result::Result::Ok(value) => std::option::Option::Some(value),
|
||||||
|
std::result::Result::Err(_) => std::option::Option::None,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_signature_shape_is_bounded_and_debug_redacted() {
|
||||||
|
let minimum = crate::BackfillSignature::new("1".repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES));
|
||||||
|
assert!(minimum.is_ok());
|
||||||
|
let maximum = crate::BackfillSignature::new("z".repeat(crate::MAX_BACKFILL_SIGNATURE_TEXT_BYTES));
|
||||||
|
assert!(maximum.is_ok());
|
||||||
|
assert!(crate::BackfillSignature::new("1".repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES - 1)).is_err());
|
||||||
|
assert!(crate::BackfillSignature::new("1".repeat(crate::MAX_BACKFILL_SIGNATURE_TEXT_BYTES + 1)).is_err());
|
||||||
|
assert!(crate::BackfillSignature::new(format!("{}0", "1".repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES - 1))).is_err());
|
||||||
|
let minimum = match minimum {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let debug = format!("{minimum:?}");
|
||||||
|
assert_eq!(debug, "BackfillSignature(..)");
|
||||||
|
assert!(!debug.contains(minimum.as_str()));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_explicit_scope_deduplicates_at_first_occurrence_without_reordering() {
|
||||||
|
let first = match signature('1') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second = match signature('2') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope_result = crate::BackfillScope::explicit_signatures(std::vec![first.clone(), second.clone(), first.clone(), second.clone()]);
|
||||||
|
assert!(scope_result.is_ok());
|
||||||
|
let scope = match scope_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let signatures = match scope.signatures() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
assert_eq!(signatures, &[first, second]);
|
||||||
|
let debug = format!("{scope:?}");
|
||||||
|
assert!(debug.contains("signature_count"));
|
||||||
|
assert!(!debug.contains(signatures[0].as_str()));
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_request_bounds_are_exact_and_explicit_context_is_rejected() {
|
||||||
|
let network = match network("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let job_id = match job_id("backfill:test") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let role = ksp_onchain_transport_lib::HttpRoleName::new("history");
|
||||||
|
let address = ksp_core_lib::Pubkey::new_from_array([7_u8; 32]);
|
||||||
|
let scope = crate::BackfillScope::latest_address(address);
|
||||||
|
let valid = crate::BackfillRequest::new(
|
||||||
|
job_id.clone(),
|
||||||
|
network.clone(),
|
||||||
|
role.clone(),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope.clone(),
|
||||||
|
crate::MAX_BACKFILL_PAGE_SIZE,
|
||||||
|
crate::MAX_BACKFILL_PAGES,
|
||||||
|
crate::MAX_BACKFILL_CANDIDATES,
|
||||||
|
crate::MAX_BACKFILL_HYDRATION_CONCURRENCY,
|
||||||
|
std::option::Option::Some(1),
|
||||||
|
);
|
||||||
|
assert!(valid.is_ok());
|
||||||
|
for (page_size, max_pages, max_candidates, concurrency) in [
|
||||||
|
(0, 1, 1, 1),
|
||||||
|
(crate::MAX_BACKFILL_PAGE_SIZE + 1, 1, 1, 1),
|
||||||
|
(1, 0, 1, 1),
|
||||||
|
(1, crate::MAX_BACKFILL_PAGES + 1, 1, 1),
|
||||||
|
(1, 1, 0, 1),
|
||||||
|
(1, 1, crate::MAX_BACKFILL_CANDIDATES + 1, 1),
|
||||||
|
(1, 1, 1, 0),
|
||||||
|
(1, 1, 1, crate::MAX_BACKFILL_HYDRATION_CONCURRENCY + 1),
|
||||||
|
] {
|
||||||
|
let result = crate::BackfillRequest::new(
|
||||||
|
job_id.clone(),
|
||||||
|
network.clone(),
|
||||||
|
role.clone(),
|
||||||
|
crate::BackfillCommitment::Finalized,
|
||||||
|
scope.clone(),
|
||||||
|
page_size,
|
||||||
|
max_pages,
|
||||||
|
max_candidates,
|
||||||
|
concurrency,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
assert!(result.is_err());
|
||||||
|
}
|
||||||
|
let explicit_signature = match signature('3') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let explicit_scope = crate::BackfillScope::explicit_signatures(std::vec![explicit_signature]);
|
||||||
|
let explicit_scope = match explicit_scope {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let explicit_with_context =
|
||||||
|
crate::BackfillRequest::new(job_id, network, role, crate::BackfillCommitment::Confirmed, explicit_scope, 1, 1, 1, 1, std::option::Option::Some(10));
|
||||||
|
assert!(explicit_with_context.is_err());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_scope_fingerprint_is_network_semantic_and_transport_source_independent() {
|
||||||
|
let signature = match signature('4') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let scope_result = crate::BackfillScope::explicit_signatures(std::vec![signature]);
|
||||||
|
let scope = match scope_result {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let mainnet = match network("mainnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let devnet = match network("devnet") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first_job = match job_id("backfill:first") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let second_job = match job_id("backfill:second") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let first = crate::BackfillRequest::new(
|
||||||
|
first_job,
|
||||||
|
devnet.clone(),
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("primary-http"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope.clone(),
|
||||||
|
100,
|
||||||
|
5,
|
||||||
|
10,
|
||||||
|
1,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
let second = crate::BackfillRequest::new(
|
||||||
|
second_job,
|
||||||
|
devnet,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("fallback-http"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope.clone(),
|
||||||
|
100,
|
||||||
|
5,
|
||||||
|
10,
|
||||||
|
64,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
let other_network = crate::BackfillRequest::new(
|
||||||
|
match job_id("backfill:third") {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
},
|
||||||
|
mainnet,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("primary-http"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
5,
|
||||||
|
10,
|
||||||
|
1,
|
||||||
|
std::option::Option::None,
|
||||||
|
);
|
||||||
|
let first = match first {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let second = match second {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let other_network = match other_network {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
assert_eq!(first.scope_fingerprint(), second.scope_fingerprint());
|
||||||
|
assert_ne!(first.scope_fingerprint(), other_network.scope_fingerprint());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_005_scope_kind_and_anchor_are_distinct_semantics() {
|
||||||
|
let anchor = match signature('5') {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let address = ksp_core_lib::Pubkey::new_from_array([9_u8; 32]);
|
||||||
|
let before = crate::BackfillScope::before_address(address, anchor.clone());
|
||||||
|
let after = crate::BackfillScope::after_address(address, anchor);
|
||||||
|
assert_eq!(before.kind(), crate::BackfillScopeKind::BeforeAddress);
|
||||||
|
assert_eq!(after.kind(), crate::BackfillScopeKind::AfterAddress);
|
||||||
|
assert!(before.anchor().is_some());
|
||||||
|
assert!(after.anchor().is_some());
|
||||||
|
assert_ne!(before, after);
|
||||||
|
return;
|
||||||
|
}
|
||||||
158
crates/ksp-job-backfill-lib/unit_tests/runtime.rs
Normal file
158
crates/ksp-job-backfill-lib/unit_tests/runtime.rs
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
// file: crates/ksp-job-backfill-lib/unit_tests/runtime.rs
|
||||||
|
// version: 3
|
||||||
|
|
||||||
|
use ksp_job_api::JobSnapshotSource; // rust-rules: trait-import
|
||||||
|
|
||||||
|
fn request() -> std::option::Option<crate::BackfillRequest> {
|
||||||
|
let signature = match crate::BackfillSignature::new("1".repeat(crate::MIN_BACKFILL_SIGNATURE_TEXT_BYTES)) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let scope = match crate::BackfillScope::explicit_signatures(std::vec![signature]) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let job_id = match ksp_job_api::JobId::new("backfill:runtime-test") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
let network = match ksp_store_lib::RawNetworkId::new("devnet") {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return std::option::Option::None,
|
||||||
|
};
|
||||||
|
return crate::BackfillRequest::new(
|
||||||
|
job_id,
|
||||||
|
network,
|
||||||
|
ksp_onchain_transport_lib::HttpRoleName::new("history"),
|
||||||
|
crate::BackfillCommitment::Confirmed,
|
||||||
|
scope,
|
||||||
|
100,
|
||||||
|
10,
|
||||||
|
1,
|
||||||
|
1,
|
||||||
|
std::option::Option::None,
|
||||||
|
)
|
||||||
|
.ok();
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_009_latest_value_source_coalesces_progress_for_slow_independent_listeners() {
|
||||||
|
let request = match request() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let runtime = match crate::BackfillJobRuntime::new(request) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let handle = runtime.handle();
|
||||||
|
let listener_a = handle.snapshots();
|
||||||
|
let listener_b = handle.snapshots();
|
||||||
|
let initial = listener_a.current();
|
||||||
|
assert_eq!(initial.sequence().value(), 0);
|
||||||
|
assert_eq!(initial.state(), ksp_job_api::JobState::Created);
|
||||||
|
let first = runtime.publisher.publish_running(crate::BackfillJobPhase::Discovering);
|
||||||
|
assert!(first.is_ok());
|
||||||
|
let second = runtime.publisher.publish_running(crate::BackfillJobPhase::Executing);
|
||||||
|
assert!(second.is_ok());
|
||||||
|
let coalesced = listener_a.wait_for_change(initial.sequence()).await;
|
||||||
|
assert_eq!(coalesced.sequence().value(), 2);
|
||||||
|
assert_eq!(coalesced.snapshot().phase(), crate::BackfillJobPhase::Executing);
|
||||||
|
let listener_b_current = listener_b.current();
|
||||||
|
assert_eq!(listener_b_current.sequence().value(), 2);
|
||||||
|
assert_eq!(listener_b_current.snapshot().phase(), crate::BackfillJobPhase::Executing);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_009_terminal_snapshot_is_retained_and_late_cancellation_is_rejected() {
|
||||||
|
let request = match request() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let runtime = match crate::BackfillJobRuntime::new(request) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let handle = runtime.handle();
|
||||||
|
let listener = handle.snapshots();
|
||||||
|
let started = runtime.publisher.publish_running(crate::BackfillJobPhase::Discovering);
|
||||||
|
assert!(started.is_ok());
|
||||||
|
assert_eq!(runtime.control.claim_normal_terminal(), super::TerminalClaim::Completed);
|
||||||
|
let terminal = runtime.publisher.publish_terminal(ksp_job_api::JobState::Completed(ksp_job_api::JobCompletion::Complete), std::option::Option::None);
|
||||||
|
assert!(terminal.is_ok());
|
||||||
|
assert!(!handle.cancel());
|
||||||
|
let current = listener.current();
|
||||||
|
assert!(current.state().is_terminal());
|
||||||
|
assert_eq!(current.snapshot().phase(), crate::BackfillJobPhase::Finished);
|
||||||
|
let retained = listener.wait_for_change(current.sequence()).await;
|
||||||
|
assert_eq!(retained.sequence(), current.sequence());
|
||||||
|
assert_eq!(retained.state(), current.state());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_terminal_race_is_first_decision_wins_for_cancellation_vs_completion() {
|
||||||
|
let (cancel_sender, _) = tokio::sync::watch::channel(false);
|
||||||
|
let cancellation_first = super::BackfillRuntimeControl::new(cancel_sender);
|
||||||
|
assert!(cancellation_first.request_cancellation());
|
||||||
|
assert_eq!(cancellation_first.claim_normal_terminal(), super::TerminalClaim::Cancelled);
|
||||||
|
assert!(!cancellation_first.request_cancellation());
|
||||||
|
let (cancel_sender, _) = tokio::sync::watch::channel(false);
|
||||||
|
let completion_first = super::BackfillRuntimeControl::new(cancel_sender);
|
||||||
|
assert_eq!(completion_first.claim_normal_terminal(), super::TerminalClaim::Completed);
|
||||||
|
assert!(!completion_first.request_cancellation());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_fatal_failure_overrides_pending_cancellation_before_terminal_publication() {
|
||||||
|
let (cancel_sender, _) = tokio::sync::watch::channel(false);
|
||||||
|
let control = super::BackfillRuntimeControl::new(cancel_sender);
|
||||||
|
assert!(control.request_cancellation());
|
||||||
|
control.claim_failed();
|
||||||
|
assert_eq!(control.claim_normal_terminal(), super::TerminalClaim::Failed);
|
||||||
|
assert!(!control.request_cancellation());
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn pre_009_long_running_pre_store_future_is_cancelled_cooperatively() {
|
||||||
|
let (cancel_sender, cancel_receiver) = tokio::sync::watch::channel(false);
|
||||||
|
let control = super::BackfillRuntimeControl::new(cancel_sender);
|
||||||
|
let signal = crate::BackfillCancellationSignal::new(control.token(), cancel_receiver);
|
||||||
|
let operation = std::future::pending::<ksp_core_lib::Result<usize>>();
|
||||||
|
let wait = signal.run_cancellable(operation);
|
||||||
|
let cancel = async {
|
||||||
|
tokio::task::yield_now().await;
|
||||||
|
assert!(control.request_cancellation());
|
||||||
|
};
|
||||||
|
let (result, ()) = tokio::join!(wait, cancel);
|
||||||
|
let error = match result {
|
||||||
|
std::result::Result::Ok(_) => return,
|
||||||
|
std::result::Result::Err(error) => error,
|
||||||
|
};
|
||||||
|
assert_eq!(error.code(), crate::ERROR_CODE_BACKFILL_CANCELLED);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn pre_009_snapshot_debug_and_public_shape_do_not_include_transport_or_raw_payloads() {
|
||||||
|
let request = match request() {
|
||||||
|
std::option::Option::Some(value) => value,
|
||||||
|
std::option::Option::None => return,
|
||||||
|
};
|
||||||
|
let runtime = match crate::BackfillJobRuntime::new(request) {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(_) => return,
|
||||||
|
};
|
||||||
|
let snapshot = runtime.handle().snapshots().current();
|
||||||
|
let debug = format!("{:?}", snapshot.snapshot());
|
||||||
|
for forbidden in ["http://", "https://", "endpoint", "provider", "raw_payload", "transaction_data"] {
|
||||||
|
assert!(!debug.contains(forbidden), "unsafe snapshot diagnostic leaked: {forbidden}");
|
||||||
|
}
|
||||||
|
assert_eq!(snapshot.snapshot().scope_kind(), crate::BackfillScopeKind::ExplicitSignatures);
|
||||||
|
assert_eq!(snapshot.snapshot().candidates_selected(), 0);
|
||||||
|
assert!(snapshot.snapshot().failure_code().is_none());
|
||||||
|
return;
|
||||||
|
}
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/http_executor.rs
|
// file: crates/ksp-onchain-transport-lib/src/http_executor.rs
|
||||||
// version: 4
|
// version: 7
|
||||||
|
|
||||||
const HTTP_BAD_GATEWAY: u16 = 502;
|
const HTTP_BAD_GATEWAY: u16 = 502;
|
||||||
const HTTP_GATEWAY_TIMEOUT: u16 = 504;
|
const HTTP_GATEWAY_TIMEOUT: u16 = 504;
|
||||||
@@ -8,6 +8,63 @@ const HTTP_REQUEST_TIMEOUT: u16 = 408;
|
|||||||
const HTTP_SERVICE_UNAVAILABLE: u16 = 503;
|
const HTTP_SERVICE_UNAVAILABLE: u16 = 503;
|
||||||
const HTTP_TOO_MANY_REQUESTS: u16 = 429;
|
const HTTP_TOO_MANY_REQUESTS: u16 = 429;
|
||||||
|
|
||||||
|
/// Typed value returned by an observed HTTP RPC path together with the safe identity of the endpoint that produced the successful response.
|
||||||
|
///
|
||||||
|
/// Endpoint URLs, headers and raw HTTP bodies are intentionally absent.
|
||||||
|
#[derive(Clone, PartialEq)]
|
||||||
|
pub struct HttpObservedValue<T> {
|
||||||
|
value: T,
|
||||||
|
endpoint_name: std::string::String,
|
||||||
|
provider: crate::HttpProviderName,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T> HttpObservedValue<T> {
|
||||||
|
/// Returns the typed RPC value.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn value(&self) -> &T {
|
||||||
|
return &self.value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the safe configured identity of the endpoint that produced the successful response.
|
||||||
|
#[must_use]
|
||||||
|
pub fn endpoint_name(&self) -> &str {
|
||||||
|
return self.endpoint_name.as_str();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Returns the safe provider descriptor attached to the successful endpoint.
|
||||||
|
#[must_use]
|
||||||
|
pub const fn provider(&self) -> &crate::HttpProviderName {
|
||||||
|
return &self.provider;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Consumes the observation and returns only the typed value.
|
||||||
|
#[must_use]
|
||||||
|
pub fn into_value(self) -> T {
|
||||||
|
return self.value;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Builds an observed value from a successful Transport attempt and its safe routing identity.
|
||||||
|
pub(crate) fn new(value: T, endpoint_name: std::string::String, provider: crate::HttpProviderName) -> Self {
|
||||||
|
return Self { value, endpoint_name, provider };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Consumes the observation into its typed value and safe routing identity for crate-internal typed decoding.
|
||||||
|
pub(crate) fn into_parts(self) -> (T, std::string::String, crate::HttpProviderName) {
|
||||||
|
return (self.value, self.endpoint_name, self.provider);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl<T> std::fmt::Debug for HttpObservedValue<T> {
|
||||||
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
|
return formatter
|
||||||
|
.debug_struct("HttpObservedValue")
|
||||||
|
.field("endpoint_name", &self.endpoint_name)
|
||||||
|
.field("provider", &self.provider)
|
||||||
|
.field("value", &"<available>")
|
||||||
|
.finish();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
impl crate::HttpTransportPool {
|
impl crate::HttpTransportPool {
|
||||||
/// Executes one audited standard Solana HTTP JSON-RPC method through KSP routing, admission and bounded retry policy.
|
/// Executes one audited standard Solana HTTP JSON-RPC method through KSP routing, admission and bounded retry policy.
|
||||||
///
|
///
|
||||||
@@ -19,6 +76,33 @@ impl crate::HttpTransportPool {
|
|||||||
method: &crate::HttpRpcMethodDescriptor,
|
method: &crate::HttpRpcMethodDescriptor,
|
||||||
params: std::vec::Vec<serde_json::Value>,
|
params: std::vec::Vec<serde_json::Value>,
|
||||||
) -> ksp_core_lib::Result<serde_json::Value> {
|
) -> ksp_core_lib::Result<serde_json::Value> {
|
||||||
|
return self.execute_standard_rpc_with(role, method, params, |value, _permit| return value).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Executes one audited standard Solana HTTP JSON-RPC method and retains only safe routing identity for the successful attempt.
|
||||||
|
pub(crate) async fn execute_standard_rpc_observed(
|
||||||
|
&self,
|
||||||
|
role: &crate::HttpRoleName,
|
||||||
|
method: &crate::HttpRpcMethodDescriptor,
|
||||||
|
params: std::vec::Vec<serde_json::Value>,
|
||||||
|
) -> ksp_core_lib::Result<crate::HttpObservedValue<serde_json::Value>> {
|
||||||
|
return self
|
||||||
|
.execute_standard_rpc_with(role, method, params, |value, permit| {
|
||||||
|
return crate::HttpObservedValue::new(value, permit.selection().endpoint_name().to_owned(), permit.client().provider().clone());
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn execute_standard_rpc_with<T, F>(
|
||||||
|
&self,
|
||||||
|
role: &crate::HttpRoleName,
|
||||||
|
method: &crate::HttpRpcMethodDescriptor,
|
||||||
|
params: std::vec::Vec<serde_json::Value>,
|
||||||
|
on_success: F,
|
||||||
|
) -> ksp_core_lib::Result<T>
|
||||||
|
where
|
||||||
|
F: std::ops::FnOnce(serde_json::Value, &crate::HttpRequestPermit) -> T,
|
||||||
|
{
|
||||||
let support = method.ensure_runtime_supported();
|
let support = method.ensure_runtime_supported();
|
||||||
if let std::result::Result::Err(error) = support {
|
if let std::result::Result::Err(error) = support {
|
||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
@@ -156,7 +240,12 @@ impl crate::HttpTransportPool {
|
|||||||
http_status = status,
|
http_status = status,
|
||||||
"completed Solana HTTP JSON-RPC request"
|
"completed Solana HTTP JSON-RPC request"
|
||||||
);
|
);
|
||||||
return parsed.into_result();
|
let value = parsed.into_result();
|
||||||
|
let value = match value {
|
||||||
|
std::result::Result::Ok(value) => value,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return std::result::Result::Ok(on_success(value, &permit));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -203,14 +292,11 @@ async fn wait_retry_delay(delay: std::time::Duration, deadline: std::time::Insta
|
|||||||
return std::time::Instant::now() < deadline;
|
return std::time::Instant::now() < deadline;
|
||||||
}
|
}
|
||||||
|
|
||||||
fn execution_timeout(method: &crate::HttpRpcMethodDescriptor, message: &str) -> ksp_core_lib::Result<serde_json::Value> {
|
fn execution_timeout<T>(method: &crate::HttpRpcMethodDescriptor, message: &str) -> ksp_core_lib::Result<T> {
|
||||||
return std::result::Result::Err(ksp_core_lib::Error::new(crate::ERROR_CODE_TIMEOUT, message).with_context("rpc_method", method.method()));
|
return std::result::Result::Err(ksp_core_lib::Error::new(crate::ERROR_CODE_TIMEOUT, message).with_context("rpc_method", method.method()));
|
||||||
}
|
}
|
||||||
|
|
||||||
fn rate_limited_error(
|
fn rate_limited_error<T>(method: &crate::HttpRpcMethodDescriptor, provider_retry_after: std::option::Option<std::time::Duration>) -> ksp_core_lib::Result<T> {
|
||||||
method: &crate::HttpRpcMethodDescriptor,
|
|
||||||
provider_retry_after: std::option::Option<std::time::Duration>,
|
|
||||||
) -> ksp_core_lib::Result<serde_json::Value> {
|
|
||||||
let mut error = ksp_core_lib::Error::new(crate::ERROR_CODE_RATE_LIMITED, "Solana HTTP endpoint rate-limited the JSON-RPC request")
|
let mut error = ksp_core_lib::Error::new(crate::ERROR_CODE_RATE_LIMITED, "Solana HTTP endpoint rate-limited the JSON-RPC request")
|
||||||
.with_context("rpc_method", method.method());
|
.with_context("rpc_method", method.method());
|
||||||
if let std::option::Option::Some(delay) = provider_retry_after {
|
if let std::option::Option::Some(delay) = provider_retry_after {
|
||||||
@@ -219,7 +305,7 @@ fn rate_limited_error(
|
|||||||
return std::result::Result::Err(error);
|
return std::result::Result::Err(error);
|
||||||
}
|
}
|
||||||
|
|
||||||
fn http_status_error(method: &crate::HttpRpcMethodDescriptor, status: u16) -> ksp_core_lib::Result<serde_json::Value> {
|
fn http_status_error<T>(method: &crate::HttpRpcMethodDescriptor, status: u16) -> ksp_core_lib::Result<T> {
|
||||||
return std::result::Result::Err(
|
return std::result::Result::Err(
|
||||||
ksp_core_lib::Error::new(crate::ERROR_CODE_HTTP_REQUEST_FAILED, "Solana HTTP endpoint returned an unsuccessful status")
|
ksp_core_lib::Error::new(crate::ERROR_CODE_HTTP_REQUEST_FAILED, "Solana HTTP endpoint returned an unsuccessful status")
|
||||||
.with_context("rpc_method", method.method())
|
.with_context("rpc_method", method.method())
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/lib.rs
|
// file: crates/ksp-onchain-transport-lib/src/lib.rs
|
||||||
// version: 44
|
// version: 45
|
||||||
|
|
||||||
#![warn(missing_docs)]
|
#![warn(missing_docs)]
|
||||||
#![deny(unreachable_pub)]
|
#![deny(unreachable_pub)]
|
||||||
@@ -271,6 +271,8 @@ pub use self::http_client::HttpEndpointClient;
|
|||||||
pub use self::http_client::HttpEndpointRoleSnapshot;
|
pub use self::http_client::HttpEndpointRoleSnapshot;
|
||||||
/// Safe metadata snapshot for one logical HTTP endpoint.
|
/// Safe metadata snapshot for one logical HTTP endpoint.
|
||||||
pub use self::http_client::HttpEndpointSnapshot;
|
pub use self::http_client::HttpEndpointSnapshot;
|
||||||
|
/// Typed RPC value paired with the safe identity of the HTTP endpoint that produced the successful response.
|
||||||
|
pub use self::http_executor::HttpObservedValue;
|
||||||
/// Result of one logical endpoint selection.
|
/// Result of one logical endpoint selection.
|
||||||
pub use self::http_pool::HttpEndpointSelection;
|
pub use self::http_pool::HttpEndpointSelection;
|
||||||
/// Runtime admission permit for one HTTP request.
|
/// Runtime admission permit for one HTTP request.
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/src/rpc_transactions.rs
|
// file: crates/ksp-onchain-transport-lib/src/rpc_transactions.rs
|
||||||
// version: 10
|
// version: 11
|
||||||
|
|
||||||
const MAX_RECENT_PRIORITIZATION_FEE_ACCOUNTS: usize = 128;
|
const MAX_RECENT_PRIORITIZATION_FEE_ACCOUNTS: usize = 128;
|
||||||
const MAX_SIGNATURES_FOR_ADDRESS_LIMIT: usize = 1_000;
|
const MAX_SIGNATURES_FOR_ADDRESS_LIMIT: usize = 1_000;
|
||||||
@@ -1249,25 +1249,42 @@ impl crate::HttpTransportPool {
|
|||||||
signature: &str,
|
signature: &str,
|
||||||
config: std::option::Option<&crate::SolanaGetTransactionConfig>,
|
config: std::option::Option<&crate::SolanaGetTransactionConfig>,
|
||||||
) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedTransaction>> {
|
) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedTransaction>> {
|
||||||
if let std::option::Option::Some(config) = config
|
let params = get_transaction_params(signature, config);
|
||||||
&& config.commitment() == std::option::Option::Some(crate::SolanaCommitment::Processed)
|
let params = match params {
|
||||||
{
|
std::result::Result::Ok(params) => params,
|
||||||
return std::result::Result::Err(
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
ksp_core_lib::Error::new(
|
};
|
||||||
crate::ERROR_CODE_INVALID_RPC_PARAMETERS,
|
|
||||||
"getTransaction commitment must be confirmed or finalized when explicitly provided",
|
|
||||||
)
|
|
||||||
.with_context("rpc_method", "getTransaction")
|
|
||||||
.with_context("commitment", "processed"),
|
|
||||||
);
|
|
||||||
}
|
|
||||||
let mut params = std::vec![serde_json::Value::String(signature.to_owned())];
|
|
||||||
if let std::option::Option::Some(config) = config {
|
|
||||||
params.push((*config).to_json_value());
|
|
||||||
}
|
|
||||||
return self.execute_get_transaction(role, params).await;
|
return self.execute_get_transaction(role, params).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Executes the current object-form `getTransaction` request and reports the safe identity of the endpoint that produced the successful response.
|
||||||
|
///
|
||||||
|
/// Routing, admission, timeout and retry behavior are identical to [`Self::get_transaction`]. The returned observation never contains an endpoint URL,
|
||||||
|
/// HTTP headers or a raw HTTP body.
|
||||||
|
pub async fn get_transaction_observed(
|
||||||
|
&self,
|
||||||
|
role: &crate::HttpRoleName,
|
||||||
|
signature: &str,
|
||||||
|
config: std::option::Option<&crate::SolanaGetTransactionConfig>,
|
||||||
|
) -> ksp_core_lib::Result<crate::HttpObservedValue<std::option::Option<crate::SolanaConfirmedTransaction>>> {
|
||||||
|
let params = get_transaction_params(signature, config);
|
||||||
|
let params = match params {
|
||||||
|
std::result::Result::Ok(params) => params,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let observed = self.execute_transaction_rpc_observed("getTransaction", role, params).await;
|
||||||
|
let observed = match observed {
|
||||||
|
std::result::Result::Ok(observed) => observed,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
let (value, endpoint_name, provider) = observed.into_parts();
|
||||||
|
let transaction = decode_get_transaction(value);
|
||||||
|
return match transaction {
|
||||||
|
std::result::Result::Ok(transaction) => std::result::Result::Ok(crate::HttpObservedValue::new(transaction, endpoint_name, provider)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/// Executes the deprecated bare-encoding `getTransaction` request form retained by Solana RPC for backwards compatibility.
|
/// Executes the deprecated bare-encoding `getTransaction` request form retained by Solana RPC for backwards compatibility.
|
||||||
#[deprecated(note = "use HttpTransportPool::get_transaction with SolanaGetTransactionConfig; the bare encoding request form is deprecated")]
|
#[deprecated(note = "use HttpTransportPool::get_transaction with SolanaGetTransactionConfig; the bare encoding request form is deprecated")]
|
||||||
pub async fn get_transaction_legacy(
|
pub async fn get_transaction_legacy(
|
||||||
@@ -1297,14 +1314,7 @@ impl crate::HttpTransportPool {
|
|||||||
std::result::Result::Ok(value) => value,
|
std::result::Result::Ok(value) => value,
|
||||||
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
};
|
};
|
||||||
if value.is_null() {
|
return decode_get_transaction(value);
|
||||||
return std::result::Result::Ok(std::option::Option::None);
|
|
||||||
}
|
|
||||||
let transaction = crate::SolanaConfirmedTransaction::decode_wire("getTransaction", value);
|
|
||||||
return match transaction {
|
|
||||||
std::result::Result::Ok(transaction) => std::result::Result::Ok(std::option::Option::Some(transaction)),
|
|
||||||
std::result::Result::Err(error) => std::result::Result::Err(error),
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Executes typed `requestAirdrop` through the common KSP HTTP transport path.
|
/// Executes typed `requestAirdrop` through the common KSP HTTP transport path.
|
||||||
@@ -1466,6 +1476,54 @@ impl crate::HttpTransportPool {
|
|||||||
};
|
};
|
||||||
return self.execute_standard_rpc(role, method, params).await;
|
return self.execute_standard_rpc(role, method, params).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async fn execute_transaction_rpc_observed(
|
||||||
|
&self,
|
||||||
|
method_name: &'static str,
|
||||||
|
role: &crate::HttpRoleName,
|
||||||
|
params: std::vec::Vec<serde_json::Value>,
|
||||||
|
) -> ksp_core_lib::Result<crate::HttpObservedValue<serde_json::Value>> {
|
||||||
|
let method = transaction_descriptor(method_name);
|
||||||
|
let method = match method {
|
||||||
|
std::result::Result::Ok(method) => method,
|
||||||
|
std::result::Result::Err(error) => return std::result::Result::Err(error),
|
||||||
|
};
|
||||||
|
return self.execute_standard_rpc_observed(role, method, params).await;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn get_transaction_params(
|
||||||
|
signature: &str,
|
||||||
|
config: std::option::Option<&crate::SolanaGetTransactionConfig>,
|
||||||
|
) -> ksp_core_lib::Result<std::vec::Vec<serde_json::Value>> {
|
||||||
|
if let std::option::Option::Some(config) = config
|
||||||
|
&& config.commitment() == std::option::Option::Some(crate::SolanaCommitment::Processed)
|
||||||
|
{
|
||||||
|
return std::result::Result::Err(
|
||||||
|
ksp_core_lib::Error::new(
|
||||||
|
crate::ERROR_CODE_INVALID_RPC_PARAMETERS,
|
||||||
|
"getTransaction commitment must be confirmed or finalized when explicitly provided",
|
||||||
|
)
|
||||||
|
.with_context("rpc_method", "getTransaction")
|
||||||
|
.with_context("commitment", "processed"),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
let mut params = std::vec![serde_json::Value::String(signature.to_owned())];
|
||||||
|
if let std::option::Option::Some(config) = config {
|
||||||
|
params.push((*config).to_json_value());
|
||||||
|
}
|
||||||
|
return std::result::Result::Ok(params);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn decode_get_transaction(value: serde_json::Value) -> ksp_core_lib::Result<std::option::Option<crate::SolanaConfirmedTransaction>> {
|
||||||
|
if value.is_null() {
|
||||||
|
return std::result::Result::Ok(std::option::Option::None);
|
||||||
|
}
|
||||||
|
let transaction = crate::SolanaConfirmedTransaction::decode_wire("getTransaction", value);
|
||||||
|
return match transaction {
|
||||||
|
std::result::Result::Ok(transaction) => std::result::Result::Ok(std::option::Option::Some(transaction)),
|
||||||
|
std::result::Result::Err(error) => std::result::Result::Err(error),
|
||||||
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(serde::Deserialize)]
|
#[derive(serde::Deserialize)]
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/tests/public_api.rs
|
// file: crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
// version: 48
|
// version: 49
|
||||||
|
|
||||||
//! Integration tests for the public `ksp-onchain-transport-lib` consumer contract.
|
//! Integration tests for the public `ksp-onchain-transport-lib` consumer contract.
|
||||||
|
|
||||||
@@ -199,6 +199,15 @@ fn public_pre_002_shared_rpc_types_are_constructible_from_crate_root() {
|
|||||||
assert_eq!(vote.vote_pubkey(), std::option::Option::Some(&pubkey));
|
assert_eq!(vote.vote_pubkey(), std::option::Option::Some(&pubkey));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn public_v0_3_6_pre_004_observed_get_transaction_surface_is_available_from_crate_root() {
|
||||||
|
let _get_transaction_observed = ksp_onchain_transport_lib::HttpTransportPool::get_transaction_observed;
|
||||||
|
let observed: std::option::Option<
|
||||||
|
ksp_onchain_transport_lib::HttpObservedValue<std::option::Option<ksp_onchain_transport_lib::SolanaConfirmedTransaction>>,
|
||||||
|
> = std::option::Option::None;
|
||||||
|
assert!(observed.is_none());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn public_pre_003_account_wrappers_are_available_from_crate_root() {
|
fn public_pre_003_account_wrappers_are_available_from_crate_root() {
|
||||||
let _get_account_info = ksp_onchain_transport_lib::HttpTransportPool::get_account_info;
|
let _get_account_info = ksp_onchain_transport_lib::HttpTransportPool::get_account_info;
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-onchain-transport-lib/unit_tests/rpc_transactions.rs
|
// file: crates/ksp-onchain-transport-lib/unit_tests/rpc_transactions.rs
|
||||||
// version: 10
|
// version: 11
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn transaction_encoding_strings_match_current_and_legacy_wire_labels() {
|
fn transaction_encoding_strings_match_current_and_legacy_wire_labels() {
|
||||||
@@ -966,6 +966,80 @@ async fn typed_get_transaction_preserves_raw_json_meta_version_and_transaction_i
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn typed_get_transaction_observed_reports_actual_winner_after_retry_reroute() {
|
||||||
|
let (first_url, first_handle) = serve_transaction_status_and_count("429 Too Many Requests");
|
||||||
|
let (winner_url, winner_handle) = serve_transaction_once(include_str!("../fixtures/http/get_transaction.base64.json"));
|
||||||
|
let urls = [(first_url.as_str(), "first-endpoint", "first-provider"), (winner_url.as_str(), "winner-endpoint", "winner-provider")];
|
||||||
|
let mut endpoints = std::vec::Vec::with_capacity(urls.len());
|
||||||
|
for (url, endpoint_name, provider) in urls {
|
||||||
|
let role = crate::HttpEndpointRoleSettings::new(
|
||||||
|
crate::HttpRoleName::new("default"),
|
||||||
|
true,
|
||||||
|
std::vec![crate::HttpRequestKind::wildcard()],
|
||||||
|
10,
|
||||||
|
crate::HttpRoleLimits::new(std::option::Option::None, std::option::Option::None, std::option::Option::None, std::option::Option::None),
|
||||||
|
);
|
||||||
|
endpoints.push(crate::HttpEndpointSettings::new(
|
||||||
|
endpoint_name,
|
||||||
|
true,
|
||||||
|
crate::HttpProviderName::new(provider),
|
||||||
|
crate::HttpClusterName::new("local"),
|
||||||
|
crate::HttpEndpointUrl::parse(url).expect("fixture URL must parse"),
|
||||||
|
std::time::Duration::from_millis(100),
|
||||||
|
std::time::Duration::from_secs(1),
|
||||||
|
std::option::Option::Some(1),
|
||||||
|
std::vec![role],
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let pool = crate::HttpTransportPool::new(crate::HttpTransportSettings::new(
|
||||||
|
endpoints,
|
||||||
|
crate::HttpRetrySettings::new(1, std::time::Duration::from_millis(1), std::time::Duration::from_millis(2)),
|
||||||
|
))
|
||||||
|
.expect("observed fixture pool must build");
|
||||||
|
let config = crate::SolanaGetTransactionConfig::new(
|
||||||
|
std::option::Option::Some(crate::SolanaCommitment::Confirmed),
|
||||||
|
std::option::Option::Some(crate::SolanaTransactionEncoding::Base64),
|
||||||
|
std::option::Option::Some(0),
|
||||||
|
);
|
||||||
|
let observed = pool
|
||||||
|
.get_transaction_observed(&crate::HttpRoleName::new("default"), "fixture-signature", std::option::Option::Some(&config))
|
||||||
|
.await
|
||||||
|
.expect("retry-safe observed getTransaction must succeed on the second endpoint");
|
||||||
|
assert_eq!(observed.endpoint_name(), "winner-endpoint");
|
||||||
|
assert_eq!(observed.provider().as_str(), "winner-provider");
|
||||||
|
let transaction = observed.value().as_ref().expect("winning response must contain a transaction");
|
||||||
|
assert_eq!(transaction.slot(), 431_000_062);
|
||||||
|
assert!(matches!(
|
||||||
|
transaction.transaction(),
|
||||||
|
crate::SolanaEncodedTransaction::Binary { encoding: crate::SolanaTransactionBinaryEncoding::Base64, .. }
|
||||||
|
));
|
||||||
|
let (first_count, first_request) = first_handle.join().expect("first fixture server must join");
|
||||||
|
assert_eq!(first_count, 1);
|
||||||
|
assert_eq!(transaction_request_body(first_request.as_str())["method"], serde_json::json!("getTransaction"));
|
||||||
|
let winner_request = winner_handle.join().expect("winner fixture server must join");
|
||||||
|
assert_eq!(transaction_request_body(winner_request.as_str())["method"], serde_json::json!("getTransaction"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test(flavor = "current_thread")]
|
||||||
|
async fn typed_get_transaction_observed_preserves_null_and_redacts_typed_value_debug() {
|
||||||
|
let (url, handle) = serve_transaction_once(include_str!("../fixtures/http/get_transaction.null.json"));
|
||||||
|
let pool = transaction_pool_for_url(url.as_str());
|
||||||
|
let observed = pool
|
||||||
|
.get_transaction_observed(&crate::HttpRoleName::new("default"), "fixture-signature", std::option::Option::None)
|
||||||
|
.await
|
||||||
|
.expect("observed null getTransaction must succeed");
|
||||||
|
assert!(observed.value().is_none());
|
||||||
|
assert_eq!(observed.endpoint_name(), "fixture-0");
|
||||||
|
assert_eq!(observed.provider().as_str(), "fixture");
|
||||||
|
let rendered = format!("{observed:?}");
|
||||||
|
assert!(rendered.contains("fixture-0"));
|
||||||
|
assert!(rendered.contains("fixture"));
|
||||||
|
assert!(rendered.contains("<available>"));
|
||||||
|
assert!(!rendered.contains("fixture-signature"));
|
||||||
|
handle.join().expect("fixture server must join");
|
||||||
|
}
|
||||||
|
|
||||||
#[tokio::test(flavor = "current_thread")]
|
#[tokio::test(flavor = "current_thread")]
|
||||||
async fn typed_get_transaction_preserves_unsupported_version_rpc_error() {
|
async fn typed_get_transaction_preserves_unsupported_version_rpc_error() {
|
||||||
let (url, handle) = serve_transaction_once(include_str!("../fixtures/http/get_transaction.error_unsupported_version.json"));
|
let (url, handle) = serve_transaction_once(include_str!("../fixtures/http/get_transaction.error_unsupported_version.json"));
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
// file: crates/ksp-program-api/src/program_instruction_decode.rs
|
// file: crates/ksp-program-api/src/program_instruction_decode.rs
|
||||||
// version: 1
|
// version: 2
|
||||||
|
|
||||||
/// Recognition strength reported by one Program instruction implementation.
|
/// Recognition strength reported by one Program instruction implementation.
|
||||||
///
|
///
|
||||||
@@ -38,10 +38,10 @@ pub enum ProgramInstructionDecodeOutcome<Decoded> {
|
|||||||
|
|
||||||
impl<Decoded> std::fmt::Debug for ProgramInstructionDecodeOutcome<Decoded> {
|
impl<Decoded> std::fmt::Debug for ProgramInstructionDecodeOutcome<Decoded> {
|
||||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||||
match self {
|
return match self {
|
||||||
Self::Decoded(_) => return formatter.write_str("Decoded"),
|
Self::Decoded(_) => formatter.write_str("Decoded"),
|
||||||
Self::Unsupported => return formatter.write_str("Unsupported"),
|
Self::Unsupported => formatter.write_str("Unsupported"),
|
||||||
}
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.004.md -->
|
<!-- file: deltas/0.1.4/pre.004.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# Delta 0.1.4-pre.004 — squelette Rust/Tauri de `ksp-app-config-desk`
|
# Delta 0.1.4-pre.004 — squelette Rust/Tauri de `ksp-app-config-desk`
|
||||||
|
|
||||||
@@ -74,11 +74,11 @@ La bibliothèque utilise les crate types Tauri usuels `staticlib`, `cdylib` et `
|
|||||||
|
|
||||||
Les versions ont été revérifiées au moment de cette tranche :
|
Les versions ont été revérifiées au moment de cette tranche :
|
||||||
|
|
||||||
| Dépendance | Version auditée | Contrainte workspace | Usage immédiat |
|
| Dépendance | Version auditée | Contrainte workspace | Usage immédiat |
|
||||||
| ------------- | --------------: | -------------------: | -------------------------- |
|
|---------------|----------------:|---------------------:|------------------------|
|
||||||
| `tauri` | `2.11.5` | `^2.11` | runtime/application |
|
| `tauri` | `2.11.5` | `^2.11` | runtime/application |
|
||||||
| `tauri-build` | `2.6.3` | `^2.6` | build script Tauri |
|
| `tauri-build` | `2.6.3` | `^2.6` | build script Tauri |
|
||||||
| `fs2` | `0.4.3` | `^0.4` | verrou single-instance |
|
| `fs2` | `0.4.3` | `^0.4` | verrou single-instance |
|
||||||
|
|
||||||
Les contraintes sont déclarées sous `[workspace.dependencies]`; la crate applicative les consomme avec `.workspace = true`.
|
Les contraintes sont déclarées sous `[workspace.dependencies]`; la crate applicative les consomme avec `.workspace = true`.
|
||||||
|
|
||||||
@@ -142,7 +142,7 @@ La destination de build est fixée dès maintenant à :
|
|||||||
Le premier couple de ports desk est réservé :
|
Le premier couple de ports desk est réservé :
|
||||||
|
|
||||||
| Usage | Port |
|
| Usage | Port |
|
||||||
| --------- | ---: |
|
|-----------|-----:|
|
||||||
| Vite HTTP | 1430 |
|
| Vite HTTP | 1430 |
|
||||||
| Vite HMR | 1431 |
|
| Vite HMR | 1431 |
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.005-fix.001.md -->
|
<!-- file: deltas/0.1.4/pre.005-fix.001.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# `0.1.4-pre.005-fix.001` — Classification npm et commande Tauri workspace
|
# `0.1.4-pre.005-fix.001` — Classification npm et commande Tauri workspace
|
||||||
|
|
||||||
@@ -98,7 +98,7 @@ Capability: tracing:default
|
|||||||
## 7. Fichiers modifiés
|
## 7. Fichiers modifiés
|
||||||
|
|
||||||
| Fichier | Action |
|
| Fichier | Action |
|
||||||
| ---------------------------------------------- | ------: |
|
|------------------------------------------------|--------:|
|
||||||
| `Cargo.toml` | modifié |
|
| `Cargo.toml` | modifié |
|
||||||
| `crates/ksp-app-config-desk/package.json` | modifié |
|
| `crates/ksp-app-config-desk/package.json` | modifié |
|
||||||
| `crates/ksp-app-config-desk/README.md` | modifié |
|
| `crates/ksp-app-config-desk/README.md` | modifié |
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.005.md -->
|
<!-- file: deltas/0.1.4/pre.005.md -->
|
||||||
<!-- version: 2 -->
|
<!-- version: 3 -->
|
||||||
|
|
||||||
# Delta 0.1.4-pre.005 — gabarit frontend Vite/TypeScript/SCSS et tracing Tauri
|
# Delta 0.1.4-pre.005 — gabarit frontend Vite/TypeScript/SCSS et tracing Tauri
|
||||||
|
|
||||||
@@ -97,18 +97,18 @@ La permission tracing entre dans le même delta que le plugin, conformément à
|
|||||||
|
|
||||||
Les générations actuelles ont été revérifiées avant rédaction du manifest :
|
Les générations actuelles ont été revérifiées avant rédaction du manifest :
|
||||||
|
|
||||||
| Dépendance | Version actuelle auditée | Contrainte déclarée |
|
| Dépendance | Version actuelle auditée | Contrainte déclarée |
|
||||||
| ---------------------------------- | -----------------------: | ------------------: |
|
|---------------------------------|-------------------------:|--------------------:|
|
||||||
| `@fltsci/tauri-plugin-tracing` | `0.3.4` | `^0.3` |
|
| `@fltsci/tauri-plugin-tracing` | `0.3.4` | `^0.3` |
|
||||||
| `@fortawesome/fontawesome-free` | `7.3.1` | `^7.3` |
|
| `@fortawesome/fontawesome-free` | `7.3.1` | `^7.3` |
|
||||||
| `@tauri-apps/api` | `2.11.1` | `^2.11` |
|
| `@tauri-apps/api` | `2.11.1` | `^2.11` |
|
||||||
| `@types/node` | `26.1.2` | `^26.1` |
|
| `@types/node` | `26.1.2` | `^26.1` |
|
||||||
| `bootstrap` | `5.3.8` | `^5.3` |
|
| `bootstrap` | `5.3.8` | `^5.3` |
|
||||||
| `resize-observer-polyfill` | `1.5.1` | `^1.5` |
|
| `resize-observer-polyfill` | `1.5.1` | `^1.5` |
|
||||||
| `sass-embedded` | `1.102.0` | `^1.102` |
|
| `sass-embedded` | `1.102.0` | `^1.102` |
|
||||||
| `simplebar` | `6.3.3` | `^6.3` |
|
| `simplebar` | `6.3.3` | `^6.3` |
|
||||||
| `typescript` | `7.0.2` | `^7.0` |
|
| `typescript` | `7.0.2` | `^7.0` |
|
||||||
| `vite` | `8.2.0` | `^8.2` |
|
| `vite` | `8.2.0` | `^8.2` |
|
||||||
|
|
||||||
Ces dépendances sont placées sous `devDependencies`, conformément à l'usage `npm i -D` retenu pour les applications desk : elles participent au build du bundle desktop et ne constituent pas un runtime Node déployé séparément.
|
Ces dépendances sont placées sous `devDependencies`, conformément à l'usage `npm i -D` retenu pour les applications desk : elles participent au build du bundle desktop et ne constituent pas un runtime Node déployé séparément.
|
||||||
|
|
||||||
@@ -155,10 +155,10 @@ afin de ne pas introduire dès la première application KSP des options désorma
|
|||||||
|
|
||||||
Le serveur est configuré ainsi :
|
Le serveur est configuré ainsi :
|
||||||
|
|
||||||
| Usage | Port | Politique |
|
| Usage | Port | Politique |
|
||||||
| ------------------------ | ---: | --------- |
|
|-------------------|-----:|-----------|
|
||||||
| Vite HTTP | 1430 | strict |
|
| Vite HTTP | 1430 | strict |
|
||||||
| WebSocket dev/HMR | 1431 | dédié |
|
| WebSocket dev/HMR | 1431 | dédié |
|
||||||
|
|
||||||
`strictPort = true` interdit le basculement silencieux du serveur HTTP vers un autre port.
|
`strictPort = true` interdit le basculement silencieux du serveur HTTP vers un autre port.
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.006.md -->
|
<!-- file: deltas/0.1.4/pre.006.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# `0.1.4-pre.006` — Bootstrap backend, `AppState` et runtime Logging initial
|
# `0.1.4-pre.006` — Bootstrap backend, `AppState` et runtime Logging initial
|
||||||
|
|
||||||
@@ -193,25 +193,25 @@ ts-rs
|
|||||||
|
|
||||||
## 11. Fichiers
|
## 11. Fichiers
|
||||||
|
|
||||||
| Fichier | Action |
|
| Fichier | Action |
|
||||||
| ----------------------------------------------------------- | ------: |
|
|-------------------------------------------------------|--------:|
|
||||||
| `Cargo.toml` | modifié |
|
| `Cargo.toml` | modifié |
|
||||||
| `crates/ksp-app-config-desk/Cargo.toml` | modifié |
|
| `crates/ksp-app-config-desk/Cargo.toml` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/app_state.rs` | ajouté |
|
| `crates/ksp-app-config-desk/src/app_state.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/src/bootstrap.rs` | ajouté |
|
| `crates/ksp-app-config-desk/src/bootstrap.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/src/constants.rs` | ajouté |
|
| `crates/ksp-app-config-desk/src/constants.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/src/dto_common.rs` | ajouté |
|
| `crates/ksp-app-config-desk/src/dto_common.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/src/errors.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/errors.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/lib.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/lib.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/tauri.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/tauri.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/unit_tests/bootstrap.rs` | ajouté |
|
| `crates/ksp-app-config-desk/unit_tests/bootstrap.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/unit_tests/dto_common.rs` | ajouté |
|
| `crates/ksp-app-config-desk/unit_tests/dto_common.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/frontend/fonts/README.md` | ajouté |
|
| `crates/ksp-app-config-desk/frontend/fonts/README.md` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/README.md` | modifié |
|
| `crates/ksp-app-config-desk/README.md` | modifié |
|
||||||
| `crates/ksp-app-config-desk/USAGE.md` | modifié |
|
| `crates/ksp-app-config-desk/USAGE.md` | modifié |
|
||||||
| `crates/ksp-app-config-desk/TODO.md` | modifié |
|
| `crates/ksp-app-config-desk/TODO.md` | modifié |
|
||||||
| `docs/plans/006-V0_1_4_CONFIG_DESKTOP_PLAN.md` | modifié |
|
| `docs/plans/006-V0_1_4_CONFIG_DESKTOP_PLAN.md` | modifié |
|
||||||
| `deltas/0.1.4/pre.006.md` | ajouté |
|
| `deltas/0.1.4/pre.006.md` | ajouté |
|
||||||
|
|
||||||
## 12. Validations à exécuter
|
## 12. Validations à exécuter
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.007.md -->
|
<!-- file: deltas/0.1.4/pre.007.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# `0.1.4-pre.007` — Bridge frontend logging KSP
|
# `0.1.4-pre.007` — Bridge frontend logging KSP
|
||||||
|
|
||||||
@@ -165,22 +165,22 @@ Le derive TS-RS ajoute le test d'export de `FrontendLogPayloadDto`.
|
|||||||
|
|
||||||
## 11. Fichiers
|
## 11. Fichiers
|
||||||
|
|
||||||
| Fichier | Action |
|
| Fichier | Action |
|
||||||
| ------------------------------------------------------------- | ------: |
|
|-------------------------------------------------------------|--------:|
|
||||||
| `Cargo.toml` | modifié |
|
| `Cargo.toml` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/constants.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/constants.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/errors.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/errors.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/frontend_logging.rs` | ajouté |
|
| `crates/ksp-app-config-desk/src/frontend_logging.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/src/lib.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/lib.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/tauri.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/tauri.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/unit_tests/frontend_logging.rs` | ajouté |
|
| `crates/ksp-app-config-desk/unit_tests/frontend_logging.rs` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/frontend/ts/frontend_log.ts` | ajouté |
|
| `crates/ksp-app-config-desk/frontend/ts/frontend_log.ts` | ajouté |
|
||||||
| `crates/ksp-app-config-desk/frontend/ts/main.ts` | modifié |
|
| `crates/ksp-app-config-desk/frontend/ts/main.ts` | modifié |
|
||||||
| `crates/ksp-app-config-desk/frontend/ts/splash.ts` | modifié |
|
| `crates/ksp-app-config-desk/frontend/ts/splash.ts` | modifié |
|
||||||
| `crates/ksp-app-config-desk/README.md` | modifié |
|
| `crates/ksp-app-config-desk/README.md` | modifié |
|
||||||
| `crates/ksp-app-config-desk/USAGE.md` | modifié |
|
| `crates/ksp-app-config-desk/USAGE.md` | modifié |
|
||||||
| `crates/ksp-app-config-desk/TODO.md` | modifié |
|
| `crates/ksp-app-config-desk/TODO.md` | modifié |
|
||||||
| `deltas/0.1.4/pre.007.md` | ajouté |
|
| `deltas/0.1.4/pre.007.md` | ajouté |
|
||||||
|
|
||||||
## 12. Validations à exécuter
|
## 12. Validations à exécuter
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.008-fix.001.md -->
|
<!-- file: deltas/0.1.4/pre.008-fix.001.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# `0.1.4-pre.008-fix.001` — Contextes splash numériques et réexports de labels inutiles
|
# `0.1.4-pre.008-fix.001` — Contextes splash numériques et réexports de labels inutiles
|
||||||
|
|
||||||
@@ -86,13 +86,13 @@ Il vérifie que les erreurs bornées conservent bien ces valeurs dans `Error::co
|
|||||||
|
|
||||||
## 6. Fichiers modifiés
|
## 6. Fichiers modifiés
|
||||||
|
|
||||||
| Fichier | Action |
|
| Fichier | Action |
|
||||||
| ------------------------------------------------------ | ------: |
|
|---------------------------------------------------|--------:|
|
||||||
| `Cargo.toml` | modifié |
|
| `Cargo.toml` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/lib.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/lib.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/src/splash.rs` | modifié |
|
| `crates/ksp-app-config-desk/src/splash.rs` | modifié |
|
||||||
| `crates/ksp-app-config-desk/unit_tests/splash.rs` | modifié |
|
| `crates/ksp-app-config-desk/unit_tests/splash.rs` | modifié |
|
||||||
| `deltas/0.1.4/pre.008-fix.001.md` | ajouté |
|
| `deltas/0.1.4/pre.008-fix.001.md` | ajouté |
|
||||||
|
|
||||||
## 7. Validations demandées
|
## 7. Validations demandées
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
<!-- file: deltas/0.1.4/pre.008-fix.003.md -->
|
<!-- file: deltas/0.1.4/pre.008-fix.003.md -->
|
||||||
<!-- version: 1 -->
|
<!-- version: 2 -->
|
||||||
|
|
||||||
# `0.1.4-pre.008-fix.003` — visibilité des logs `debug` de Config Desk
|
# `0.1.4-pre.008-fix.003` — visibilité des logs `debug` de Config Desk
|
||||||
|
|
||||||
@@ -57,12 +57,12 @@ ksp-app-config-desk -> Debug
|
|||||||
|
|
||||||
## 5. Fichiers modifiés
|
## 5. Fichiers modifiés
|
||||||
|
|
||||||
| Fichier | Action |
|
| Fichier | Action |
|
||||||
| ------------------------------------------------- | ------: |
|
|-----------------------------------------------|--------:|
|
||||||
| `Cargo.toml` | modifié |
|
| `Cargo.toml` | modifié |
|
||||||
| `config/std.logging.json` | modifié |
|
| `config/std.logging.json` | modifié |
|
||||||
| `crates/ksp-config-lib/unit_tests/logging.rs` | modifié |
|
| `crates/ksp-config-lib/unit_tests/logging.rs` | modifié |
|
||||||
| `deltas/0.1.4/pre.008-fix.003.md` | ajouté |
|
| `deltas/0.1.4/pre.008-fix.003.md` | ajouté |
|
||||||
|
|
||||||
Aucune dépendance n'est ajoutée.
|
Aucune dépendance n'est ajoutée.
|
||||||
|
|
||||||
|
|||||||
191
deltas/0.3.5/pre.001-fix.001.md
Normal file
191
deltas/0.3.5/pre.001-fix.001.md
Normal file
@@ -0,0 +1,191 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.001-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.001-fix.001` — correction du gate Interface acquisition events
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.001
|
||||||
|
workspace.package.version = 0.3.5-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce fix est strictement documentaire. Conformément au workflow KSP, il ne modifie pas `workspace.package.version` et ne touche aucun fichier Cargo/Rust/runtime/config.
|
||||||
|
|
||||||
|
## Motif du fix
|
||||||
|
|
||||||
|
Le brainstorming post-gate a identifié deux décisions trop conservatrices dans `pre.001` :
|
||||||
|
|
||||||
|
1. `OptimisticConfirmation <-> Yellowstone Confirmed` et `Root <-> Yellowstone Finalized` représentent bien les mêmes faits de lifecycle lorsqu'ils sont normalisés avec des noms KSP qui conservent la sémantique source ;
|
||||||
|
2. la catégorie générique « transaction status » mélangeait snapshot, commitment et exécution, masquant un candidat event-only transversal possible.
|
||||||
|
|
||||||
|
## Corrections de conception
|
||||||
|
|
||||||
|
### Slot lifecycle
|
||||||
|
|
||||||
|
L'intersection admise devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleStage
|
||||||
|
+ Processed
|
||||||
|
+ FirstShredReceived
|
||||||
|
+ Completed
|
||||||
|
+ CreatedBank
|
||||||
|
+ Dead
|
||||||
|
+ OptimisticallyConfirmed
|
||||||
|
+ Rooted
|
||||||
|
```
|
||||||
|
|
||||||
|
Mappings corrigés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana slotsUpdates::OptimisticConfirmation
|
||||||
|
-> SlotLifecycleStage::OptimisticallyConfirmed
|
||||||
|
<- Yellowstone SlotStatus::Confirmed
|
||||||
|
|
||||||
|
Solana rootSubscribe / slotsUpdates::Root
|
||||||
|
-> SlotLifecycleStage::Rooted
|
||||||
|
<- Yellowstone SlotStatus::Finalized
|
||||||
|
```
|
||||||
|
|
||||||
|
Les noms `OptimisticallyConfirmed` et `Rooted` évitent de promouvoir les labels Yellowstone `Confirmed`/`Finalized` en terminologie KSP universelle. `Frozen` reste hors intersection.
|
||||||
|
|
||||||
|
### Transaction execution
|
||||||
|
|
||||||
|
`TransactionExecutionEvent` est rouvert comme **candidat actif de `0.3.5`**, conceptuellement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
TransactionExecutionEvent {
|
||||||
|
slot,
|
||||||
|
signature,
|
||||||
|
outcome = Succeeded | Failed,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Producers candidats :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana logsSubscribe
|
||||||
|
Yellowstone TransactionStatus
|
||||||
|
Helius transactionSubscribe
|
||||||
|
```
|
||||||
|
|
||||||
|
Le type n'est pas encore admis. `pre.003` doit d'abord auditer la représentation KSP de signature, les mappings exacts, l'anti-duplication Store et le consumer concret. Aucun type partiel ne doit être créé si ce gate échoue.
|
||||||
|
|
||||||
|
### Logs et commitment
|
||||||
|
|
||||||
|
Les familles sont désormais explicitement séparées :
|
||||||
|
|
||||||
|
```text
|
||||||
|
TransactionLogEvent
|
||||||
|
-> idée différée ; bornes/consumer/multi-producer à démontrer
|
||||||
|
|
||||||
|
getSignatureStatuses
|
||||||
|
-> snapshot Transport
|
||||||
|
|
||||||
|
signatureSubscribe
|
||||||
|
-> transition one-shot de commitment Transport
|
||||||
|
|
||||||
|
TransactionCommitmentEvent
|
||||||
|
-> non admis ; ne jamais fusionner avec TransactionExecutionEvent par Option soup
|
||||||
|
```
|
||||||
|
|
||||||
|
### Extensions provider/protocol
|
||||||
|
|
||||||
|
La règle d'ownership est clarifiée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
DTO provider/protocol spécifique
|
||||||
|
-> Transport
|
||||||
|
|
||||||
|
fait passif provider-neutral réellement commun
|
||||||
|
-> Interface après gate sémantique
|
||||||
|
```
|
||||||
|
|
||||||
|
Ainsi `Helius transactionSubscribe` et Yellowstone `Entry` restent Transport-owned comme DTOs. Leur caractère non standard n'interdit pas qu'un futur fait commun qu'ils permettent d'observer soit Interface-owned.
|
||||||
|
|
||||||
|
## Traçabilité TODO/IDEAS
|
||||||
|
|
||||||
|
Le fix ne crée pas artificiellement une entrée durable séparée pendant que le gate `0.3.5` est encore actif :
|
||||||
|
|
||||||
|
```text
|
||||||
|
TransactionExecutionEvent -> suivi actif dans plan/validation 0.3.5
|
||||||
|
TransactionLogEvent -> idée différée explicitement tracée dans plan/validation
|
||||||
|
```
|
||||||
|
|
||||||
|
Si `TransactionLogEvent` reste hors scope à la réconciliation documentaire finale, il devra être transféré dans le TODO/IDEAS durable approprié.
|
||||||
|
|
||||||
|
## Plan de prereleases corrigé
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 implémentation SlotLifecycleEvent/Stage à 7 stages
|
||||||
|
pre.003 gate TransactionExecutionEvent + implémentation seulement si gate complet
|
||||||
|
pre.004 canaris externes + complétude API
|
||||||
|
pre.005 hardening final
|
||||||
|
pre.006 gate technique final
|
||||||
|
pre.007 réconciliation documentaire finale
|
||||||
|
pre.008 préparation de publication
|
||||||
|
rel.001 stable 0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur reçu pour `pre.001`
|
||||||
|
|
||||||
|
Avant ce fix, l'opérateur a rejoué :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo clean
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat fourni :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (251 table(s), 137 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
cargo clippy --workspace --all-targets: PASS
|
||||||
|
cargo test -p ksp-interface-lib: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les tests Interface détaillés fournis sont tous propres : 10 unitaires, 4 dependency-boundary, 1 external-consumer, 5 public-api, 3 release-completeness, aucun échec de doc-test.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.5/pre.001-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers explicitement inchangés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/**
|
||||||
|
README.md
|
||||||
|
ROADMAP.md
|
||||||
|
CHANGELOG.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate demandé après application du fix
|
||||||
|
|
||||||
|
Le fix ne change que Markdown. Rejouer :
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun rerun Cargo n'est requis par le contenu du fix lui-même.
|
||||||
119
deltas/0.3.5/pre.001.md
Normal file
119
deltas/0.3.5/pre.001.md
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.001` — audit et planification Interface acquisition events
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.4
|
||||||
|
workspace.package.version = 0.3.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le log opérateur fourni avec l'archive d'entrée montre le gate stable `v0.3.4` propre. Cette tranche part exclusivement de l'archive `khadhroony-solana-project-v0.3.4.zip` fournie par l'opérateur.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Exécuter le gate `pre.001` prescrit par `prompts/024-V0_3_5_START_PROMPT.md` : lecture, inventaire producer/consumer, audit sémantique multi-transport actuel, ownership Interface/Transport/Store, threat model, sizing et planification.
|
||||||
|
|
||||||
|
Aucun modèle Rust n'est implémenté dans cette tranche.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
Une seule famille passe le gate d'admission Interface :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleEvent
|
||||||
|
+ Processed
|
||||||
|
+ FirstShredReceived
|
||||||
|
+ Completed
|
||||||
|
+ CreatedBank
|
||||||
|
+ Dead
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette intersection correspond uniquement aux faits communs démontrés entre Solana WebSocket et Yellowstone. Les metadata riches restent Transport.
|
||||||
|
|
||||||
|
Décisions négatives :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Root -X-> Finalized
|
||||||
|
OptimisticConfirmation -X-> Confirmed
|
||||||
|
logsSubscribe -> reporté
|
||||||
|
TransactionStatusObservation -> reporté
|
||||||
|
vote -> reporté
|
||||||
|
account/program -> Store/Transport
|
||||||
|
transaction/block -> Store/Transport
|
||||||
|
Yellowstone Entry -> Transport
|
||||||
|
Helius Enhanced transaction -> Transport
|
||||||
|
notification after commit -> Store API selon KSP-NOTIFY-001..006
|
||||||
|
```
|
||||||
|
|
||||||
|
La release est redimensionnée autour d'une seule famille et réserve explicitement les couloirs hardening, gate technique final, réconciliation documentaire puis publication.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
deltas/0.3.5/pre.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
Modifications Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
header version: 382 -> 383
|
||||||
|
workspace.package.version: 0.3.4 -> 0.3.5-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
-> Markdown table audit: clean (251 table(s), 137 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo` n'est pas installé dans l'environnement d'assemblage. Les commandes suivantes ne sont donc pas déclarées réussies :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Elles doivent être rejouées par l'opérateur après application de l'overlay.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question bloquante pour `pre.002`.
|
||||||
|
|
||||||
|
Les familles logs/status/vote/root restent volontairement reportées et ne doivent être rouvertes que si un nouveau producer/consumer concret apporte une convergence qui n'existe pas dans ce gate.
|
||||||
|
|
||||||
|
## Suite prévue
|
||||||
|
|
||||||
|
```text
|
||||||
|
pre.002 implémentation SlotLifecycleEvent/Stage
|
||||||
|
pre.003 canaris externes + complétude API
|
||||||
|
pre.004 hardening final
|
||||||
|
pre.005 gate technique final
|
||||||
|
pre.006 réconciliation documentaire finale
|
||||||
|
pre.007 préparation de publication
|
||||||
|
rel.001 stable 0.3.5
|
||||||
|
```
|
||||||
67
deltas/0.3.5/pre.002-fix.001.md
Normal file
67
deltas/0.3.5/pre.002-fix.001.md
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
# Delta `0.3.5-pre.002-fix.001`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Corriger les défauts de conformité Rust révélés par le gate opérateur de `0.3.5-pre.002`, sans modifier le contrat fonctionnel `SlotLifecycleEvent` / `SlotLifecycleStage`.
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.002
|
||||||
|
workspace.package.version = 0.3.5-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
## Diagnostic opérateur
|
||||||
|
|
||||||
|
Le code fonctionnel et les tests ciblés passent, mais le gate global n'est pas propre :
|
||||||
|
|
||||||
|
- 15 violations `RUST-FMT-101` pour des lignes vides à l'intérieur de fonctions de tests ;
|
||||||
|
- 2 usages de l'opérateur `?` rejetés par `clippy::question_mark_used` ;
|
||||||
|
- closures à retour implicite rejetées par `clippy::implicit_return` ;
|
||||||
|
- documentation crate-level manquante dans les deux nouveaux tests d'intégration.
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
- supprimer les lignes vides internes aux fonctions concernées ;
|
||||||
|
- remplacer `?` par une propagation explicite via `match` ;
|
||||||
|
- rendre explicites les `return` des closures concernées ;
|
||||||
|
- ajouter la documentation crate-level aux tests d'intégration ;
|
||||||
|
- conserver strictement les sept variantes de `SlotLifecycleStage` et la forme de `SlotLifecycleEvent` ;
|
||||||
|
- ne pas ajouter de `allow`, de dépendance, de codec, de runtime ou de conversion Transport.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le fix modifie des fichiers Rust ; la version workspace est donc incrémentée suivant la convention de fix prerelease :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.5-pre.2.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le numéro documentaire reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.002-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-interface-lib/tests/slot_lifecycle_public_api.rs
|
||||||
|
crates/ksp-interface-lib/unit_tests/slot_lifecycle.rs
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
deltas/0.3.5/pre.002-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
180
deltas/0.3.5/pre.002.md
Normal file
180
deltas/0.3.5/pre.002.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.002` — `SlotLifecycleEvent`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.001-fix.001
|
||||||
|
workspace.package.version = 0.3.5-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Gate d'entrée opérateur reçu après le fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (240 table(s), 138 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser uniquement la famille passive provider-neutral `SlotLifecycleEvent` admise par le gate corrigé de `pre.001`, sans ouvrir la famille candidate `TransactionExecutionEvent` et sans déplacer les DTOs protocol/provider hors de Transport.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Cette tranche est une nouvelle prerelease non-fix. La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifest `crates/ksp-interface-lib/Cargo.toml` reste inchangé et continue d'hériter de la version workspace.
|
||||||
|
|
||||||
|
## Implémentation
|
||||||
|
|
||||||
|
Nouvelle surface crate-root :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleStage
|
||||||
|
SlotLifecycleEvent
|
||||||
|
```
|
||||||
|
|
||||||
|
`SlotLifecycleStage` est `#[non_exhaustive]` et contient exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Processed
|
||||||
|
FirstShredReceived
|
||||||
|
Completed
|
||||||
|
CreatedBank
|
||||||
|
Dead
|
||||||
|
OptimisticallyConfirmed
|
||||||
|
Rooted
|
||||||
|
```
|
||||||
|
|
||||||
|
`SlotLifecycleEvent` reste volontairement minimal :
|
||||||
|
|
||||||
|
```text
|
||||||
|
slot: u64
|
||||||
|
stage: SlotLifecycleStage
|
||||||
|
```
|
||||||
|
|
||||||
|
avec uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleEvent::new(slot, stage)
|
||||||
|
SlotLifecycleEvent::slot()
|
||||||
|
SlotLifecycleEvent::stage()
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune validation faillible n'est ajoutée : tout `u64` est un slot admissible au niveau de ce contrat passif.
|
||||||
|
|
||||||
|
## Ownership et dépendances
|
||||||
|
|
||||||
|
Préservé :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-interface-lib -> ksp-core-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Absent :
|
||||||
|
|
||||||
|
```text
|
||||||
|
serde
|
||||||
|
codec wire
|
||||||
|
tracing/logging runtime
|
||||||
|
tokio/channel/event bus
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-store-api
|
||||||
|
ksp-store-lib
|
||||||
|
ksp-store-postgres-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun converter depuis les DTOs Solana WS, Yellowstone ou Helius n'est ajouté dans Interface. Les conversions appartiennent à la future composition/consumer.
|
||||||
|
|
||||||
|
## Canaris
|
||||||
|
|
||||||
|
Ajoutés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/unit_tests/slot_lifecycle.rs
|
||||||
|
crates/ksp-interface-lib/tests/slot_lifecycle_public_api.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
Le premier verrouille les 7 stages, le domaine `u64`, les traits passifs retenus et un `Debug` borné/provider-neutral. Le second vérifie la consommation depuis crate root et la capacité d'évolution downstream de l'enum non-exhaustive.
|
||||||
|
|
||||||
|
Le canari historique `tests/release_completeness.rs` est ajusté afin que son inventaire exact accepte la seule extension prévue de cette tranche : `slot_lifecycle` et ses deux reexports publics.
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/src/lib.rs
|
||||||
|
crates/ksp-interface-lib/tests/release_completeness.rs
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/src/slot_lifecycle.rs
|
||||||
|
crates/ksp-interface-lib/unit_tests/slot_lifecycle.rs
|
||||||
|
crates/ksp-interface-lib/tests/slot_lifecycle_public_api.rs
|
||||||
|
deltas/0.3.5/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers explicitement hors scope
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/**
|
||||||
|
crates/ksp-store-api/**
|
||||||
|
crates/ksp-store-lib/**
|
||||||
|
crates/ksp-store-postgres-lib/**
|
||||||
|
README.md
|
||||||
|
ROADMAP.md
|
||||||
|
CHANGELOG.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
Validations statiques de l'overlay :
|
||||||
|
|
||||||
|
```text
|
||||||
|
version workspace = 0.3.5-pre.2
|
||||||
|
nouvelle source Interface sans serde/tracing/tokio/Transport/Store
|
||||||
|
crate-root = 4 modules privés / 8 reexports publics attendus
|
||||||
|
SlotLifecycleStage = 7 variantes prévues
|
||||||
|
aucun TransactionExecutionEvent introduit
|
||||||
|
archive = fichiers modifiés/ajoutés uniquement
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit pas le workspace Cargo complet exécutable ; aucun `cargo check`, Clippy ou test Rust de cette tranche n'est donc déclaré réussi ici.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
|
|
||||||
|
`ksp-program-api` est inclus car il est le consumer actuel direct de `ksp-interface-lib`, même si `pre.002` ne modifie aucun contrat Program existant.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `SlotLifecycleEvent`.
|
||||||
|
|
||||||
|
Le seul gate fonctionnel suivant reste `TransactionExecutionEvent` : représentation de signature, convergence exacte Solana/Yellowstone/Helius, anti-duplication Store et consumer concret doivent être prouvés en `pre.003` avant toute implémentation.
|
||||||
|
|
||||||
|
## Suite
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.003 — gate TransactionExecutionEvent ; implémentation uniquement si le gate est complet
|
||||||
|
```
|
||||||
193
deltas/0.3.5/pre.003.md
Normal file
193
deltas/0.3.5/pre.003.md
Normal file
@@ -0,0 +1,193 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.003` — `TransactionExecutionEvent`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.2.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de cette base confirme les audits Rust, `cargo check --workspace`, Clippy, `cargo test -p ksp-interface-lib` et `cargo test -p ksp-program-api`. Le seul défaut restant est une double ligne vide dans `docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md`; conformément à la décision opérateur, cette correction documentaire est absorbée par le présent delta au lieu de créer un fix dédié.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le delta modifie la surface Rust Interface. La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
## Mission
|
||||||
|
|
||||||
|
Fermer le gate dédié à `TransactionExecutionEvent` prévu par le plan corrigé de `0.3.5` : représentation de signature, convergence Solana/Yellowstone/Helius, anti-duplication Store et consumer concret doivent tous être démontrés avant création du type.
|
||||||
|
|
||||||
|
## Décision de gate
|
||||||
|
|
||||||
|
Le gate passe avec une projection strictement minimale :
|
||||||
|
|
||||||
|
```text
|
||||||
|
TransactionSignature([u8; 64])
|
||||||
|
|
||||||
|
TransactionExecutionOutcome {
|
||||||
|
Succeeded,
|
||||||
|
Failed,
|
||||||
|
}
|
||||||
|
|
||||||
|
TransactionExecutionEvent {
|
||||||
|
slot,
|
||||||
|
signature,
|
||||||
|
outcome,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
La convergence retenue est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana logsSubscribe
|
||||||
|
context.slot + signature + err
|
||||||
|
|
||||||
|
Yellowstone TransactionStatus
|
||||||
|
slot + signature + err
|
||||||
|
|
||||||
|
Helius transactionSubscribe
|
||||||
|
slot + signature + résultat d'exécution seulement lorsque la forme/detail mode le rend déterminable
|
||||||
|
```
|
||||||
|
|
||||||
|
`err` n'est jamais copié dans Interface. Solana `err: null`, Yellowstone `err: None` et un Helius `err: null` explicitement disponible donnent `Succeeded`; une erreur explicite donne `Failed`.
|
||||||
|
|
||||||
|
Pour Helius, la projection est conservative : un `err` omis, une forme `none`, inconnue ou sans information d'erreur exploitable ne produit pas artificiellement un `TransactionExecutionEvent`.
|
||||||
|
|
||||||
|
## Signature
|
||||||
|
|
||||||
|
`TransactionSignature` possède exactement 64 octets déjà décodés.
|
||||||
|
|
||||||
|
Interface n'ajoute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
aucun base58
|
||||||
|
aucun serde
|
||||||
|
aucun codec
|
||||||
|
aucune crate Solana/Anza supplémentaire
|
||||||
|
aucune capability Wallet de signature
|
||||||
|
```
|
||||||
|
|
||||||
|
Le `Debug` est redacted et ne rend jamais les bytes.
|
||||||
|
|
||||||
|
`ksp-store-api::RawTransactionSignature` reste Store-owned. Aucun alias/reexport/converter cross-domain n'est ajouté ; la future composition convertit explicitement les 64 octets si nécessaire.
|
||||||
|
|
||||||
|
## Ownership
|
||||||
|
|
||||||
|
Restent Transport-owned :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Solana logs notification DTO
|
||||||
|
Yellowstone TransactionStatus DTO
|
||||||
|
Helius transactionSubscribe DTOs
|
||||||
|
commitment
|
||||||
|
subscription/filter metadata
|
||||||
|
transaction index
|
||||||
|
logs
|
||||||
|
memo
|
||||||
|
block time
|
||||||
|
confirmation status
|
||||||
|
provider timestamps
|
||||||
|
opaque transaction errors
|
||||||
|
```
|
||||||
|
|
||||||
|
Interface possède uniquement le fait passif commun.
|
||||||
|
|
||||||
|
## Consumer
|
||||||
|
|
||||||
|
Le futur worker RAW live peut utiliser `TransactionExecutionEvent` comme signal provider-neutral d'exécution observée et décider ensuite d'hydrater/persister un `RawTransaction` par sa propre composition/policy.
|
||||||
|
|
||||||
|
Le premier backfill historique `0.3.6` reste indépendant de cette API.
|
||||||
|
|
||||||
|
## Surface Rust
|
||||||
|
|
||||||
|
Ajouts crate-root :
|
||||||
|
|
||||||
|
```text
|
||||||
|
TransactionSignature
|
||||||
|
TransactionExecutionOutcome
|
||||||
|
TransactionExecutionEvent
|
||||||
|
```
|
||||||
|
|
||||||
|
`TransactionExecutionOutcome` est `#[non_exhaustive]`.
|
||||||
|
|
||||||
|
Les champs des structs restent privés et la surface ne possède ni runtime ni comportement réseau.
|
||||||
|
|
||||||
|
## Tests/canaris
|
||||||
|
|
||||||
|
Ajoutés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
unit_tests/transaction_execution.rs
|
||||||
|
tests/transaction_execution_public_api.rs
|
||||||
|
```
|
||||||
|
|
||||||
|
Le canari `tests/release_completeness.rs` évolue pour figer :
|
||||||
|
|
||||||
|
```text
|
||||||
|
5 modules privés de production
|
||||||
|
11 reexports crate-root
|
||||||
|
transaction_execution comme seule nouvelle famille pre.003
|
||||||
|
absence de serde/tracing/Transport/Store dans la racine Interface
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
Le plan et la validation passent `TransactionExecutionEvent` de candidat actif à `ADMIS pre.003`.
|
||||||
|
|
||||||
|
`TransactionLogEvent` reste une idée différée distincte. `TransactionCommitmentEvent` reste non admis. Yellowstone `Entry` et les DTOs Helius restent Transport-owned même lorsqu'ils pourront contribuer à de futurs faits Interface réellement transversaux.
|
||||||
|
|
||||||
|
La double ligne vide signalée par le gate opérateur précédent est supprimée dans la validation.
|
||||||
|
|
||||||
|
## Fichiers
|
||||||
|
|
||||||
|
Modifiés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/src/lib.rs
|
||||||
|
crates/ksp-interface-lib/tests/release_completeness.rs
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Ajoutés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/src/transaction_execution.rs
|
||||||
|
crates/ksp-interface-lib/unit_tests/transaction_execution.rs
|
||||||
|
crates/ksp-interface-lib/tests/transaction_execution_public_api.rs
|
||||||
|
deltas/0.3.5/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/**
|
||||||
|
crates/ksp-store-api/**
|
||||||
|
crates/ksp-store-lib/**
|
||||||
|
crates/ksp-store-postgres-lib/**
|
||||||
|
README.md
|
||||||
|
ROADMAP.md
|
||||||
|
CHANGELOG.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
|
|
||||||
|
Les gates Cargo ne sont pas déclarés réussis par l'environnement d'assemblage.
|
||||||
153
deltas/0.3.5/pre.004.md
Normal file
153
deltas/0.3.5/pre.004.md
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.004` — canaris externes et complétude API
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.003` est entièrement propre : audits Rust/Markdown, `cargo check --workspace`, Clippy, tests `ksp-interface-lib` et `ksp-program-api` passent.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
La tranche modifie des sources Rust de test/canari. La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
## Mission
|
||||||
|
|
||||||
|
Verrouiller la surface des deux familles passives admises sans ajouter de comportement ni de type de production :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleEvent / SlotLifecycleStage
|
||||||
|
TransactionExecutionEvent / TransactionExecutionOutcome / TransactionSignature
|
||||||
|
```
|
||||||
|
|
||||||
|
## Consumer externe
|
||||||
|
|
||||||
|
`tests/external_consumer.rs` consomme maintenant les deux familles acquisition uniquement depuis le crate-root public.
|
||||||
|
|
||||||
|
Le canari prouve notamment qu'un consumer downstream peut récupérer :
|
||||||
|
|
||||||
|
```text
|
||||||
|
slot + stage
|
||||||
|
slot + signature + outcome
|
||||||
|
```
|
||||||
|
|
||||||
|
sans accès aux modules privés, à Transport ou à Store.
|
||||||
|
|
||||||
|
Le canari Program historique reste conservé.
|
||||||
|
|
||||||
|
## Inventaire exact
|
||||||
|
|
||||||
|
`tests/release_completeness.rs` fige :
|
||||||
|
|
||||||
|
```text
|
||||||
|
5 modules privés de production
|
||||||
|
11 reexports crate-root
|
||||||
|
2 familles passives d'acquisition exactement
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun nouveau module ni export de production n'est ajouté par `pre.004`.
|
||||||
|
|
||||||
|
## Evolutivité
|
||||||
|
|
||||||
|
Les deux enums publics évolutifs restent explicitement `#[non_exhaustive]` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleStage
|
||||||
|
TransactionExecutionOutcome
|
||||||
|
```
|
||||||
|
|
||||||
|
Les canaris publics downstream de `pre.002` et `pre.003` restent inchangés et continuent d'utiliser un fallback `_`.
|
||||||
|
|
||||||
|
## Firewalls négatifs
|
||||||
|
|
||||||
|
Le scan de production inclut désormais les deux modules acquisition.
|
||||||
|
|
||||||
|
Un canari spécifique interdit dans `slot_lifecycle.rs` et `transaction_execution.rs` les paths de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Config
|
||||||
|
Logging
|
||||||
|
Transport
|
||||||
|
Store
|
||||||
|
reqwest
|
||||||
|
serde
|
||||||
|
Solana instruction
|
||||||
|
Tauri
|
||||||
|
Tokio
|
||||||
|
Tonic
|
||||||
|
tracing
|
||||||
|
borsh/bincode/wincode
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifeste `ksp-interface-lib` reste Core-only et inchangé.
|
||||||
|
|
||||||
|
## Hors scope
|
||||||
|
|
||||||
|
Toujours interdits/reportés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
converters Transport -> Interface
|
||||||
|
provider DTOs
|
||||||
|
Store RAW
|
||||||
|
serde/codecs
|
||||||
|
runtime/channels
|
||||||
|
logging
|
||||||
|
TransactionLogEvent
|
||||||
|
TransactionCommitmentEvent
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers
|
||||||
|
|
||||||
|
Modifiés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-interface-lib/tests/external_consumer.rs
|
||||||
|
crates/ksp-interface-lib/tests/release_completeness.rs
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Ajouté :
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.5/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/Cargo.toml
|
||||||
|
crates/ksp-interface-lib/src/**
|
||||||
|
crates/ksp-interface-lib/unit_tests/**
|
||||||
|
crates/ksp-onchain-transport-lib/**
|
||||||
|
crates/ksp-store-api/**
|
||||||
|
crates/ksp-store-lib/**
|
||||||
|
crates/ksp-store-postgres-lib/**
|
||||||
|
crates/ksp-program-api/**
|
||||||
|
README.md
|
||||||
|
ROADMAP.md
|
||||||
|
CHANGELOG.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
|
|
||||||
|
Les gates Cargo restent à exécuter par l'opérateur.
|
||||||
78
deltas/0.3.5/pre.005-fix.001.md
Normal file
78
deltas/0.3.5/pre.005-fix.001.md
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
# Delta `0.3.5-pre.005-fix.001`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Corriger le faux négatif du canari de manifeste ajouté par `0.3.5-pre.005`, sans modifier le contrat fonctionnel ni le graphe de dépendances de `ksp-interface-lib`.
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.005
|
||||||
|
workspace.package.version = 0.3.5-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Diagnostic opérateur
|
||||||
|
|
||||||
|
Le gate du `2026-08-31` est propre pour les audits Rust/Markdown et `cargo check --workspace`, mais le nouveau test :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0_3_5_pre_005_manifest_keeps_exact_core_only_graph_without_features_or_hidden_dependency_sections
|
||||||
|
```
|
||||||
|
|
||||||
|
échoue avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
left: 2
|
||||||
|
right: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
La cause est le comptage textuel `manifest.matches("ksp-core-lib")` appliqué à la déclaration :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib = { path = "../ksp-core-lib" }
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette ligne contient légitimement deux occurrences de la chaîne tout en déclarant une seule dépendance. Le manifeste lui-même est correct.
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
- conserver les interdictions de sections `[features]`, `[dev-dependencies]` et `[build-dependencies]` ;
|
||||||
|
- conserver l'exigence d'une seule section `[dependencies]` ;
|
||||||
|
- extraire cette section puis réutiliser `manifest_dependency_names(...)` ;
|
||||||
|
- vérifier l'inventaire exact des clés de dépendance : `ksp-core-lib` uniquement ;
|
||||||
|
- ne modifier aucun manifeste de crate, source de production, type public ou dependency graph.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le fix modifie une source Rust de test ; la version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.5-pre.5.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le numéro documentaire reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.005-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/tests/dependency_boundary.rs
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
deltas/0.3.5/pre.005-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
138
deltas/0.3.5/pre.005.md
Normal file
138
deltas/0.3.5/pre.005.md
Normal file
@@ -0,0 +1,138 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.005` — hardening et complétude Interface
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.004` est entièrement propre : audits Rust/Markdown, `cargo check --workspace`, Clippy, tests `ksp-interface-lib` et `ksp-program-api` passent.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
La tranche ajoute/modifie des sources Rust de test/canari. La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Mission
|
||||||
|
|
||||||
|
Fermer le hardening fonctionnel de `0.3.5` sans ajouter de troisième famille Interface ni modifier les contrats de production admis :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleEvent / SlotLifecycleStage
|
||||||
|
TransactionExecutionEvent / TransactionExecutionOutcome / TransactionSignature
|
||||||
|
```
|
||||||
|
|
||||||
|
## Hardening adversarial
|
||||||
|
|
||||||
|
Le nouveau `tests/security_hardening.rs` vérifie :
|
||||||
|
|
||||||
|
- conservation byte-exact d'une signature hostile de 64 octets ;
|
||||||
|
- redaction de la signature dans son `Debug` et dans celui de `TransactionExecutionEvent` ;
|
||||||
|
- conservation de `u64::MAX` sans narrowing ;
|
||||||
|
- structure minimale exacte des deux events ;
|
||||||
|
- absence de collection/payload/timestamp/provider/endpoint/filter/network dans les contrats acquisition ;
|
||||||
|
- absence de second RAW ou de famille reportée exportée silencieusement ;
|
||||||
|
- non-régression des bornes `ProgramInstruction` historiques.
|
||||||
|
|
||||||
|
## Graphe et dépendances
|
||||||
|
|
||||||
|
`tests/dependency_boundary.rs` ajoute un canari final imposant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
une seule section [dependencies]
|
||||||
|
exactement une référence ksp-core-lib
|
||||||
|
aucune [features]
|
||||||
|
aucune [dev-dependencies]
|
||||||
|
aucune [build-dependencies]
|
||||||
|
```
|
||||||
|
|
||||||
|
Le manifeste `ksp-interface-lib` reste inchangé et Core-only.
|
||||||
|
|
||||||
|
## Candidats reportés
|
||||||
|
|
||||||
|
Aucun candidat reporté n'est promu par hardening :
|
||||||
|
|
||||||
|
```text
|
||||||
|
TransactionLogEvent -> idée différée
|
||||||
|
TransactionCommitmentEvent -> reporté
|
||||||
|
VoteEvent -> reporté
|
||||||
|
LedgerEntryEvent -> reporté
|
||||||
|
Account/Program RAW event -> rejet Interface
|
||||||
|
Transaction complète -> rejet Interface
|
||||||
|
Block event -> rejet Interface
|
||||||
|
Persisted-data available -> Store API owner
|
||||||
|
```
|
||||||
|
|
||||||
|
`TransactionLogEvent` devra être transféré vers TODO/IDEAS durable pendant `pre.007` s'il reste hors scope.
|
||||||
|
|
||||||
|
## Fichiers
|
||||||
|
|
||||||
|
Modifiés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/tests/dependency_boundary.rs
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Ajoutés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/tests/security_hardening.rs
|
||||||
|
deltas/0.3.5/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-interface-lib/Cargo.toml
|
||||||
|
crates/ksp-interface-lib/src/**
|
||||||
|
crates/ksp-interface-lib/unit_tests/**
|
||||||
|
crates/ksp-onchain-transport-lib/**
|
||||||
|
crates/ksp-store-api/**
|
||||||
|
crates/ksp-store-lib/**
|
||||||
|
crates/ksp-store-postgres-lib/**
|
||||||
|
crates/ksp-program-api/**
|
||||||
|
README.md
|
||||||
|
ROADMAP.md
|
||||||
|
CHANGELOG.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
`cargo` n'est pas installé dans le sandbox d'assemblage. Restent donc à exécuter côté opérateur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
|
|
||||||
|
## Décisions
|
||||||
|
|
||||||
|
- deux familles acquisition exactement restent admises ;
|
||||||
|
- aucun contrat production n'est modifié ;
|
||||||
|
- aucun nouveau bound métier n'est inventé ;
|
||||||
|
- la signature reste une primitive passive fixe de 64 octets sans codec ;
|
||||||
|
- aucune metadata source/provider n'entre dans Interface ;
|
||||||
|
- aucun second RAW n'est créé ;
|
||||||
|
- Program instruction foundation reste inchangée.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question fonctionnelle bloquante pour `pre.006`. La seule idée différée explicitement à conserver durablement est `TransactionLogEvent`.
|
||||||
80
deltas/0.3.5/pre.006.md
Normal file
80
deltas/0.3.5/pre.006.md
Normal file
@@ -0,0 +1,80 @@
|
|||||||
|
# Delta `0.3.5-pre.006`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Matérialiser le gate technique final de `0.3.5` après le hardening Interface et son fix de canari, sans rouvrir la surface fonctionnelle.
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.005-fix.001
|
||||||
|
workspace.package.version = 0.3.5-pre.5.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate d'entrée opérateur
|
||||||
|
|
||||||
|
Le gate du `2026-08-31` est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (240 table(s), 144 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
cargo clippy --workspace --all-targets: PASS
|
||||||
|
cargo test -p ksp-interface-lib: PASS
|
||||||
|
cargo test -p ksp-program-api: PASS
|
||||||
|
cargo test --workspace: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les tests réseau/live et probes diagnostiques restent `ignored` conformément à leur contrat. Aucun test non ignoré n'échoue.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
`pre.006` ne modifie aucun code Rust, aucun manifeste de crate et aucun contrat Interface. Elle prépare uniquement le gate technique final prévu par `prompts/024-V0_3_5_START_PROMPT.md`.
|
||||||
|
|
||||||
|
La clôture doit établir :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-interface-lib -> ksp-core-lib uniquement
|
||||||
|
aucune feature Interface
|
||||||
|
aucune dev/build dependency Interface
|
||||||
|
2 familles acquisition exactement
|
||||||
|
aucun runtime/serde/codec/logging/Transport/Store dans Interface
|
||||||
|
aucun duplicate dependency nouveau imputable à 0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.5-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
deltas/0.3.5/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun autre fichier n'est modifié.
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test --workspace
|
||||||
|
cargo tree -p ksp-interface-lib --edges normal
|
||||||
|
cargo tree -p ksp-interface-lib -e features
|
||||||
|
cargo tree --duplicates
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun smoke réseau n'est requis par défaut.
|
||||||
154
deltas/0.3.5/pre.007.md
Normal file
154
deltas/0.3.5/pre.007.md
Normal file
@@ -0,0 +1,154 @@
|
|||||||
|
# Delta `0.3.5-pre.007`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Réconcilier la documentation durable avec la surface Interface réellement acquise après le gate technique final propre de `pre.006`, sans rouvrir le code ni les contrats publics.
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.006
|
||||||
|
workspace.package.version = 0.3.5-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
La base opérateur inclut les corrections locales de numéros d'en-tête signalées après l'overlay `pre.006` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml header 389
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md header 8
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces corrections d'en-tête ne changent pas le contenu fonctionnel du gate validé.
|
||||||
|
|
||||||
|
## Gate d'entrée
|
||||||
|
|
||||||
|
Le gate `pre.006` est intégralement propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (240 table(s), 145 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
cargo clippy --workspace --all-targets: PASS
|
||||||
|
cargo test -p ksp-interface-lib: PASS
|
||||||
|
cargo test --workspace: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Les graphes confirment :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-interface-lib -> ksp-core-lib uniquement
|
||||||
|
aucune feature Interface
|
||||||
|
aucune dev/build dependency Interface
|
||||||
|
aucun duplicate dependency nouveau imputable à 0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Les tests live/réseau et probes diagnostiques restent `ignored` conformément à leur contrat.
|
||||||
|
|
||||||
|
## Réconciliation
|
||||||
|
|
||||||
|
La documentation durable était encore partiellement figée sur la foundation Program-only :
|
||||||
|
|
||||||
|
```text
|
||||||
|
README Interface -> rôle Program-only et affirmation « aucun événement runtime »
|
||||||
|
USAGE Interface -> page explicitement matérialisée par 0.2.13
|
||||||
|
architecture -> rôle Interface décrit surtout comme façade wire Program
|
||||||
|
TransactionLogEvent -> idée différée encore enfermée dans le plan/validation de release
|
||||||
|
```
|
||||||
|
|
||||||
|
`pre.007` corrige ces incohérences sans transformer Interface en runtime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Program wire contracts -> Interface
|
||||||
|
provider-neutral passive acquisition -> Interface lorsque convergence exacte démontrée
|
||||||
|
rich HTTP/WS/gRPC/provider DTOs -> Transport
|
||||||
|
persistent/replayable RAW -> Store API
|
||||||
|
post-commit durable-data wake-up -> Store API
|
||||||
|
converters Transport -> Interface -> composition/consumer
|
||||||
|
backlog/recovery -> Store
|
||||||
|
```
|
||||||
|
|
||||||
|
La surface durable documentée contient exactement les deux familles acquisition acquises :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleEvent
|
||||||
|
TransactionExecutionEvent
|
||||||
|
```
|
||||||
|
|
||||||
|
`TransactionLogEvent` est transféré dans `docs/IDEAS.md` et reste soumis à un futur gate consumer + bornes + convergence multi-source.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.5-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers
|
||||||
|
|
||||||
|
Modifiés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-interface-lib/README.md
|
||||||
|
crates/ksp-interface-lib/USAGE.md
|
||||||
|
docs/IDEAS.md
|
||||||
|
docs/architecture/002-LAYERS_AND_DEPENDENCIES.md
|
||||||
|
docs/architecture/003-COMPONENT_CONTRACTS.md
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/006-WIRE_AND_PROGRAM.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Ajouté :
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.5/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
prompts/**
|
||||||
|
crates/ksp-interface-lib/src/**
|
||||||
|
crates/ksp-interface-lib/tests/**
|
||||||
|
crates/ksp-interface-lib/unit_tests/**
|
||||||
|
crates/ksp-interface-lib/Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/**
|
||||||
|
crates/ksp-store-api/**
|
||||||
|
crates/ksp-store-lib/**
|
||||||
|
crates/ksp-store-postgres-lib/**
|
||||||
|
crates/ksp-program-api/**
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate d'assemblage
|
||||||
|
|
||||||
|
Les contrôles statiques disponibles dans l'environnement d'assemblage sont propres :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (240 table(s), 146 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun code Rust n'est modifié par cette tranche. Les commandes Cargo restent à confirmer dans le repository opérateur après application de l'overlay.
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun smoke réseau n'est requis. Après gate propre, la tranche suivante est la préparation de publication minimale `0.3.5-pre.008`.
|
||||||
177
deltas/0.3.5/pre.008-fix.001.md
Normal file
177
deltas/0.3.5/pre.008-fix.001.md
Normal file
@@ -0,0 +1,177 @@
|
|||||||
|
<!-- file: deltas/0.3.5/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-pre.008-fix.001` — enrichissement du prompt `0.3.6` et trajectoire Job/Worker
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.008
|
||||||
|
workspace.package.version = 0.3.5-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce fix est **strictement documentaire**. Il ne modifie aucun fichier Rust/build/runtime/config et ne change donc pas `workspace.package.version`.
|
||||||
|
|
||||||
|
`Cargo.toml` reste exactement celui de `pre.008` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.5-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
## Motif du fix
|
||||||
|
|
||||||
|
Le prompt `025` de `pre.008` était insuffisamment détaillé par rapport :
|
||||||
|
|
||||||
|
- aux règles de construction des prompts KSP ;
|
||||||
|
- à la portée réelle attendue de `0.3.6` ;
|
||||||
|
- aux fonctionnalités de backfill déjà présentes historiquement dans kbot3 ;
|
||||||
|
- au besoin d'observabilité par une couche supérieure ;
|
||||||
|
- à la trajectoire Worker qui doit reprendre le même pattern de notifications.
|
||||||
|
|
||||||
|
Il laissait également le nom de la crate concrète à décider alors que la direction est désormais fixée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
## Corrections du prompt `0.3.6`
|
||||||
|
|
||||||
|
Le prompt version 2 ouvre désormais explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-job-api
|
||||||
|
+
|
||||||
|
ksp-job-backfill-lib
|
||||||
|
+
|
||||||
|
backfill RawTransaction historique fonctionnel
|
||||||
|
+
|
||||||
|
lifecycle/progression/outcome
|
||||||
|
+
|
||||||
|
notifications/listeners sûrs pour UI/composition
|
||||||
|
```
|
||||||
|
|
||||||
|
### Archive kbot3
|
||||||
|
|
||||||
|
L'archive :
|
||||||
|
|
||||||
|
```text
|
||||||
|
khadhroony-bot3_v0.5.3-pre.005-fix010.zip
|
||||||
|
```
|
||||||
|
|
||||||
|
redevient **obligatoire en `0.3.6-pre.001`**, mais uniquement comme inventaire fonctionnel historique.
|
||||||
|
|
||||||
|
Le plan doit construire une matrice :
|
||||||
|
|
||||||
|
```text
|
||||||
|
REPRENDRE FONCTIONNELLEMENT
|
||||||
|
REDESSINER POUR KSP
|
||||||
|
REPORTER
|
||||||
|
REJETER
|
||||||
|
```
|
||||||
|
|
||||||
|
Le code, les anciennes crates, DTOs, SQL, retries et choix runtime kbot3 ne sont pas à copier.
|
||||||
|
|
||||||
|
### Parité fonctionnelle minimale à réauditer
|
||||||
|
|
||||||
|
Le prompt rend explicites les invariants historiquement démontrés :
|
||||||
|
|
||||||
|
- scans latest/before/after/explicit selon audit ;
|
||||||
|
- anchors et directions ;
|
||||||
|
- candidats/pages dédupliqués ;
|
||||||
|
- nearest-newer/gap-fill si conservé ;
|
||||||
|
- frontier de complétion contiguë ;
|
||||||
|
- reprise après cancellation sans saut ;
|
||||||
|
- cancellation coopérative pendant RPC long et wait/backoff ;
|
||||||
|
- ownership séparé du retry Transport et du retry Job ;
|
||||||
|
- compteurs de campagne distinguant sélection, complétion, insert/idempotence/existing, missing et observations.
|
||||||
|
|
||||||
|
### Notifications Job
|
||||||
|
|
||||||
|
Le prompt exige désormais un vrai contrat d'observabilité `ksp-job-api` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
identity
|
||||||
|
state/phase
|
||||||
|
progress/snapshot
|
||||||
|
checkpoint/frontier
|
||||||
|
sequence ou ordre observable
|
||||||
|
terminal outcome
|
||||||
|
safe error code
|
||||||
|
listener isolation/backpressure
|
||||||
|
resynchronisation après perte/coalescing
|
||||||
|
```
|
||||||
|
|
||||||
|
Ces notifications sont explicitement distinctes :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Interface acquisition events
|
||||||
|
Store KSP-NOTIFY-* post-commit wake-ups
|
||||||
|
logging/tracing
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune dépendance Tokio/channel précise n'est imposée dans `ksp-job-api` avant l'audit `pre.001`.
|
||||||
|
|
||||||
|
### Alignement Worker futur
|
||||||
|
|
||||||
|
Le même pattern conceptuel devra être repris par `ksp-worker-api`, sans créer dès `0.3.6` une crate générique spéculative supplémentaire.
|
||||||
|
|
||||||
|
## Corrections du ROADMAP
|
||||||
|
|
||||||
|
La trajectoire RAW opérationnelle devient explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6 ksp-job-api + ksp-job-backfill-lib + notifications
|
||||||
|
0.3.7 application backfill/inspection RAW
|
||||||
|
0.3.8 ksp-worker-api
|
||||||
|
0.3.9 ksp-worker-live-transactions-retriever-lib
|
||||||
|
0.3.10 application monitoring/visualisation Jobs + Workers
|
||||||
|
```
|
||||||
|
|
||||||
|
L'ancien placeholder générique « worker/service live ensuite » est supprimé au profit de releases nommées et ordonnées.
|
||||||
|
|
||||||
|
## Correction du CHANGELOG
|
||||||
|
|
||||||
|
Le paragraphe ouvrant `0.3.6` est aligné sur la portée réelle du prompt version 2 et sur la nouvelle trajectoire ROADMAP.
|
||||||
|
|
||||||
|
Aucune information sur le contenu fonctionnel déjà stabilisé de `0.3.5` n'est changée.
|
||||||
|
|
||||||
|
## Versions d'en-tête
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md 24 -> 25
|
||||||
|
ROADMAP.md 99 -> 100
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md 1 -> 2
|
||||||
|
deltas/0.3.5/pre.008-fix.001.md nouveau -> 1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
Exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md
|
||||||
|
deltas/0.3.5/pre.008-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/**
|
||||||
|
docs/architecture/**
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
Le fix ne modifie aucun Rust. Le gate minimal est :
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun rerun Cargo n'est requis par ce fix documentaire lui-même.
|
||||||
112
deltas/0.3.5/pre.008.md
Normal file
112
deltas/0.3.5/pre.008.md
Normal file
@@ -0,0 +1,112 @@
|
|||||||
|
# Delta `0.3.5-pre.008`
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Préparer la publication de `0.3.5` après le gate technique final et la réconciliation documentaire propres, sans rouvrir le code ni les documents de conception de la release.
|
||||||
|
|
||||||
|
## Base
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.007
|
||||||
|
workspace.package.version = 0.3.5-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur de `pre.007` est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (240 table(s), 146 file(s))
|
||||||
|
cargo check --workspace: PASS
|
||||||
|
cargo clippy --workspace --all-targets: PASS
|
||||||
|
cargo test -p ksp-interface-lib: PASS
|
||||||
|
cargo test -p ksp-program-api: PASS
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate technique final `pre.006` avait déjà confirmé `cargo test --workspace` et les graphes Cargo Interface Core-only.
|
||||||
|
|
||||||
|
## Modifications
|
||||||
|
|
||||||
|
### Version workspace
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
Le numéro d'en-tête de `Cargo.toml` est incrémenté avec cette modification réelle.
|
||||||
|
|
||||||
|
### `CHANGELOG.md`
|
||||||
|
|
||||||
|
Ajout de l'entrée de release `0.3.5` décrivant :
|
||||||
|
|
||||||
|
- `SlotLifecycleEvent` et ses sept stages admis ;
|
||||||
|
- `TransactionSignature`, `TransactionExecutionOutcome` et `TransactionExecutionEvent` ;
|
||||||
|
- les frontières Transport / Interface / Store ;
|
||||||
|
- le graphe Interface Core-only ;
|
||||||
|
- les gates finaux propres ;
|
||||||
|
- l'ouverture de `0.3.6` sur Job API + premier backfill RAW.
|
||||||
|
|
||||||
|
### `ROADMAP.md`
|
||||||
|
|
||||||
|
- `0.3.5` passe à terminé ;
|
||||||
|
- `0.3.6` est précisé comme premier backfill RAW avec ownership Job de la policy/progression/checkpoint ;
|
||||||
|
- les TODO historiques sont réconciliés avec les deux familles Interface réellement acquises ;
|
||||||
|
- `TransactionLogEvent` reste une idée différée et `vote` reste hors Interface.
|
||||||
|
|
||||||
|
### Prompt suivant
|
||||||
|
|
||||||
|
Ajout de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Le prompt ouvre `0.3.6` sur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-job-api
|
||||||
|
+ premier backfill historique RAW concret
|
||||||
|
+ audit prioritaire RawTransaction par adresse
|
||||||
|
+ getSignaturesForAddress -> getTransaction comme candidat
|
||||||
|
+ policy/batch/progression/checkpoint/cancellation Job-owned
|
||||||
|
```
|
||||||
|
|
||||||
|
L'archive historique kbot3 n'est pas requise pour cette release.
|
||||||
|
|
||||||
|
## Fichiers de la tranche
|
||||||
|
|
||||||
|
Exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md
|
||||||
|
deltas/0.3.5/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/**
|
||||||
|
docs/architecture/**
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
crates/ksp-interface-lib/README.md
|
||||||
|
crates/ksp-interface-lib/USAGE.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test -p ksp-program-api
|
||||||
|
```
|
||||||
|
|
||||||
|
Si ce gate reste propre, la tranche suivante est `0.3.5-rel.001` et doit être strictement mécanique : version stable `0.3.5`, delta de release et aucune correction fonctionnelle/documentaire.
|
||||||
123
deltas/0.3.5/rel.001.md
Normal file
123
deltas/0.3.5/rel.001.md
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
<!-- file: deltas/0.3.5/rel.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.5-rel.001` — publication stable
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.008
|
||||||
|
workspace.package.version = 0.3.5-pre.8
|
||||||
|
```
|
||||||
|
|
||||||
|
La base inclut le fix documentaire :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.008-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
dont le gate minimal est propre :
|
||||||
|
|
||||||
|
```text
|
||||||
|
General Rust rule audit: clean
|
||||||
|
Rust export completeness audit: 0 candidate(s)
|
||||||
|
KSP workspace Rust rule audit: clean
|
||||||
|
Markdown table audit: clean (240 table(s), 149 file(s))
|
||||||
|
```
|
||||||
|
|
||||||
|
Les gates techniques et documentaires précédents ont déjà fermé la release :
|
||||||
|
|
||||||
|
- `pre.006` : gate technique final complet, tests workspace et graphes Cargo Interface ;
|
||||||
|
- `pre.007` : réconciliation documentaire finale ;
|
||||||
|
- `pre.008` : lane minimale de publication ;
|
||||||
|
- `pre.008-fix.001` : enrichissement documentaire du prompt `0.3.6` et de la trajectoire Job/Worker.
|
||||||
|
|
||||||
|
## Objet
|
||||||
|
|
||||||
|
Publier mécaniquement la version stable :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune correction fonctionnelle, architecturale ou documentaire n'est autorisée dans cette tranche.
|
||||||
|
|
||||||
|
## Modification
|
||||||
|
|
||||||
|
### `Cargo.toml`
|
||||||
|
|
||||||
|
Le numéro d'en-tête est incrémenté avec la modification réelle :
|
||||||
|
|
||||||
|
```text
|
||||||
|
391 -> 392
|
||||||
|
```
|
||||||
|
|
||||||
|
La version workspace devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.5-pre.8 -> 0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune autre ligne de `Cargo.toml` n'est modifiée.
|
||||||
|
|
||||||
|
## Surface publiée
|
||||||
|
|
||||||
|
`0.3.5` stabilise dans `ksp-interface-lib` les deux familles passives d'acquisition retenues par les gates de la release :
|
||||||
|
|
||||||
|
```text
|
||||||
|
SlotLifecycleEvent
|
||||||
|
TransactionExecutionEvent
|
||||||
|
```
|
||||||
|
|
||||||
|
La surface reste provider-neutral, Core-only, sans runtime/event bus/serde/codec, sans duplication de la couche RAW Store et sans élargissement de la fondation Program existante.
|
||||||
|
|
||||||
|
Le prompt suivant déjà préparé reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md
|
||||||
|
```
|
||||||
|
|
||||||
|
et ouvre `0.3.6` sur le développement parallèle de `ksp-job-api` et `ksp-job-backfill-lib`, avec reprise fonctionnelle du backfill historique kbot3 et notifications/listeners observables.
|
||||||
|
|
||||||
|
## Fichiers de la tranche
|
||||||
|
|
||||||
|
Exactement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
deltas/0.3.5/rel.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Explicitement inchangés :
|
||||||
|
|
||||||
|
```text
|
||||||
|
CHANGELOG.md
|
||||||
|
ROADMAP.md
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md
|
||||||
|
crates/**
|
||||||
|
docs/**
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.5
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-interface-lib
|
||||||
|
cargo test --workspace
|
||||||
|
```
|
||||||
|
|
||||||
|
Après ce gate, la release stable peut être taguée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
La session suivante démarre depuis cette base stable en exécutant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
prompts/025-V0_3_6_START_PROMPT.md
|
||||||
|
```
|
||||||
125
deltas/0.3.6/pre.001-fix.001.md
Normal file
125
deltas/0.3.6/pre.001-fix.001.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.001-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.001-fix.001` — sizing souple et audit Markdown effectif
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.001
|
||||||
|
workspace.package.version = 0.3.6-pre.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce fix corrige la tranche de cadrage `pre.001`. Il modifie un script d'audit consommé par le gate ; conformément à `VER-ID-007` et `VER-ID-010`, la version Cargo devient `0.3.6-pre.1.fix.1`.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
- remplacer le forecast tabulaire par une hiérarchie éditable `### pre.NNN` puis `#### pre.NNN-fix.MMM` ;
|
||||||
|
- corriger le faux négatif de `scripts/audit_markdown_tables.py` ;
|
||||||
|
- imposer l'alignement source gauche, droit ou centré indiqué par la ligne séparatrice ;
|
||||||
|
- réaligner les documents actifs révélés par le validateur corrigé sans réécrire les anciens deltas publiés.
|
||||||
|
|
||||||
|
## Cause racine
|
||||||
|
|
||||||
|
`_is_separator_row` appliquait la regex aux cellules brutes. Une ligne telle que `| --- | ---: |` contient des espaces dans chaque cellule et n'était donc pas reconnue comme séparateur. Le tableau entier était ignoré, son compteur n'augmentait pas et aucune largeur n'était contrôlée.
|
||||||
|
|
||||||
|
Le validateur contrôlait par ailleurs uniquement le padding droit des cellules de contenu. Il n'appliquait pas la sémantique de `---:` pour une colonne alignée à droite ni celle de `:---:` pour une colonne centrée.
|
||||||
|
|
||||||
|
## Corrections
|
||||||
|
|
||||||
|
### Détection et validation
|
||||||
|
|
||||||
|
- détection tolérante par `strip` afin qu'un séparateur plausible mais mal formaté entre dans l'audit ;
|
||||||
|
- validation canonique stricte du séparateur brut, qui doit remplir la colonne sans espaces ;
|
||||||
|
- `---` et `:---` : un espace gauche puis padding droit ;
|
||||||
|
- `---:` : padding gauche puis un espace droit ;
|
||||||
|
- `:---:` : paddings équilibrés à un espace près ;
|
||||||
|
- largeur minimale compatible avec trois tirets et les marqueurs d'alignement ;
|
||||||
|
- erreurs dédiées par orientation.
|
||||||
|
|
||||||
|
### Régressions automatisées
|
||||||
|
|
||||||
|
Le nouveau test `scripts/tests/test_audit_markdown_tables.py` couvre cinq cas :
|
||||||
|
|
||||||
|
- tableau gauche et droit canonique accepté ;
|
||||||
|
- tableau gauche explicite et centré accepté ;
|
||||||
|
- séparateur espacé auparavant ignoré désormais rejeté ;
|
||||||
|
- cellule alignée à gauche sous `---:` rejetée ;
|
||||||
|
- cellule alignée à droite sous `:---` rejetée.
|
||||||
|
|
||||||
|
### Sizing de la v0.3.6
|
||||||
|
|
||||||
|
Le tableau du découpage prévisionnel est remplacé par des sous-sections. Chaque prerelease porte son statut, son budget, son entrée, son contenu et sa sortie. `pre.001-fix.001` est inséré directement sous `pre.001`, comme dans le sizing durable du plan 025.
|
||||||
|
|
||||||
|
### Réalignement documentaire
|
||||||
|
|
||||||
|
Le gate corrigé a révélé 13 documents durables contenant soit un séparateur espacé, soit un padding incompatible avec ses marqueurs. Ils sont réalignés mécaniquement, sans changement sémantique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/formats/KSPWALLET_V1.md
|
||||||
|
docs/formats/KSPWALLET_V2.md
|
||||||
|
docs/plans/005-V0_1_3_CONFIG_FOUNDATION_PLAN.md
|
||||||
|
docs/plans/006-V0_1_4_CONFIG_DESKTOP_PLAN.md
|
||||||
|
docs/plans/011-V0_2_4_HTTP_BLOCKS_ECONOMICS_PLAN.md
|
||||||
|
docs/plans/012-V0_2_5_WALLET_FOUNDATION_PLAN.md
|
||||||
|
docs/plans/014-V0_2_7_ONCHAIN_WEBSOCKET_PLAN.md
|
||||||
|
docs/plans/026-V0_3_5_INTERFACE_ACQUISITION_EVENTS_PLAN.md
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/010-V0_2_7_ONCHAIN_WEBSOCKET.md
|
||||||
|
docs/validation/011-V0_2_8_HELIUS_LASERSTREAM_WEBSOCKET.md
|
||||||
|
docs/validation/022-V0_3_5_INTERFACE_ACQUISITION_EVENTS.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deltas antérieurs restent inchangés conformément à `DOC-TABLE-006`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
scripts/tests/test_audit_markdown_tables.py
|
||||||
|
deltas/0.3.6/pre.001-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
scripts/audit_markdown_tables.py
|
||||||
|
docs/rules/RULES_DOCUMENTATION.md
|
||||||
|
les 13 documents durables listés dans la section de réalignement
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
- le script d'audit est un artefact de code d'outillage et impose donc le suffixe Cargo `.fix.1` ;
|
||||||
|
- le séparateur reste détecté de façon tolérante mais validé selon la forme canonique stricte ;
|
||||||
|
- l'alignement source suit ses marqueurs Markdown, pas seulement le rendu visuel ;
|
||||||
|
- le réalignement historique est limité aux documents durables encore audités, jamais aux deltas publiés ;
|
||||||
|
- aucun contenu fonctionnel Job ou Backfill n'est ouvert par ce fix.
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 -m unittest scripts/tests/test_audit_markdown_tables.py
|
||||||
|
-> 5 tests PASS
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> clean, 264 tables / 141 fichiers
|
||||||
|
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
```
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
`cargo` n'est pas installé dans l'environnement d'assemblage. `cargo fmt`, `cargo check`, `cargo clippy` et `cargo test` ne sont pas annoncés comme réussis. Aucun fichier Rust ou manifeste de crate n'est modifié, mais l'opérateur doit rejouer le gate demandé après application du fix.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.002`.
|
||||||
80
deltas/0.3.6/pre.001.md
Normal file
80
deltas/0.3.6/pre.001.md
Normal file
@@ -0,0 +1,80 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta v0.3.6-pre.1 — cadrage Job API et backfill RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
v0.3.5
|
||||||
|
workspace.package.version = 0.3.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette tranche part exclusivement de l'archive KSP v0.3.5 fournie. L'archive kbot3 ne sert que de référence fonctionnelle isolée.
|
||||||
|
|
||||||
|
## Résumé
|
||||||
|
|
||||||
|
Cette préversion ouvre la v0.3.6 par un audit documentaire et fonctionnel complet. Elle fixe les contrats, limites, ownerships, scénarios de validation et tranches de livraison avant toute implémentation lourde.
|
||||||
|
|
||||||
|
kbot3 a été consulté dans un arbre séparé uniquement pour inventorier les comportements et scénarios. Aucun code, extrait ou fichier de cette archive n'est intégré au projet.
|
||||||
|
|
||||||
|
## Vérifications préalables
|
||||||
|
|
||||||
|
- archive KSP v0.3.5 : test ZIP intégral propre, 1 515 entrées, SHA-256 `c395efb6a09c786069aa5063a7a414b55f8ae24e2b9d6eaa414b2f312cd62588` ;
|
||||||
|
- archive kbot3 : test ZIP intégral propre, 2 501 entrées, SHA-256 `ee47643b9f8b582ee8db97b2381ec107e45aef8c009fee44757531514615d318` ;
|
||||||
|
- aucune entrée absolue, traversée, avec séparateur inversé ou lien symbolique ZIP ;
|
||||||
|
- règles, architecture, roadmap, changelog et historique de publication relus ;
|
||||||
|
- surfaces réelles de Core, Transport et Store auditées ;
|
||||||
|
- contrats officiels Solana `getSignaturesForAddress` et `getTransaction` recoupés ;
|
||||||
|
- journal opérateur v0.3.5 conservé comme preuve externe, sans le confondre avec une exécution locale.
|
||||||
|
|
||||||
|
## Décisions figées
|
||||||
|
|
||||||
|
- deux crates finales obligatoires : `ksp-job-api` et `ksp-job-backfill-lib` ;
|
||||||
|
- Job API passive, runtime-neutral, latest-value, multi-listeners et sans callback producteur ;
|
||||||
|
- portées latest, before, after et signatures explicites ;
|
||||||
|
- Transport propriétaire exclusif du routage, pacing, timeout et retry ;
|
||||||
|
- extension additive observée de `getTransaction` pour obtenir la provenance réelle de l'endpoint victorieux ;
|
||||||
|
- format RAW `ksp.solana.raw_transaction` version `1`, base64, JSON canonique déterministe ;
|
||||||
|
- accès Store uniquement via `ksp-store-lib`, écriture atomique en mode `Normal` ;
|
||||||
|
- checkpoint par frontière contiguë, sans saut de missing, conflit, erreur ou annulation ;
|
||||||
|
- `Missing` observable comme résultat partiel, conflit et erreurs d'infrastructure comme échec ;
|
||||||
|
- pas de Worker API, application, Config, backend direct, journal de tentatives ni pipeline partagé en v0.3.6.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
- `docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md` ;
|
||||||
|
- `docs/validation/023-V0_3_6_JOB_API_BACKFILL.md` ;
|
||||||
|
- `deltas/0.3.6/pre.001.md`.
|
||||||
|
|
||||||
|
## Fichier modifié
|
||||||
|
|
||||||
|
- `Cargo.toml` : header `392` vers `393`, version workspace `0.3.5` vers `0.3.6-pre.1`.
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
- `python3 scripts/audit_rust_workspace_rules.py` : propre sur la baseline v0.3.5 ;
|
||||||
|
- `python3 scripts/audit_markdown_tables.py` : 240 tables et 150 fichiers propres sur la baseline v0.3.5 ;
|
||||||
|
- audit Rust workspace post-modification : propre et zéro candidat d'export ;
|
||||||
|
- audit Markdown post-modification sur racine, docs, prompts, crates et deltas v0.3.5 ou v0.3.6 : 240 tables et 153 fichiers propres ;
|
||||||
|
- contrats de fichiers, tailles et absence de code kbot3 dans le payload vérifiés.
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
`cargo` est indisponible dans l'environnement courant. `cargo fmt`, `cargo check`, `cargo clippy` et `cargo test` ne sont donc pas annoncés comme réussis et devront être rejoués par l'opérateur.
|
||||||
|
|
||||||
|
## Non inclus
|
||||||
|
|
||||||
|
- aucune nouvelle crate ;
|
||||||
|
- aucun code Rust ;
|
||||||
|
- aucun manifeste de crate ;
|
||||||
|
- aucune reprise de code kbot3 ;
|
||||||
|
- aucune publication stable.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.002`. La version exacte et les features du décodeur Base58 seront décidées par audit de dépendance au moment de la tranche de conversion, sans SDK Solana complet.
|
||||||
|
|
||||||
|
## Suite prévue
|
||||||
|
|
||||||
|
`pre.002` implémente uniquement identité, lifecycle et annulation de `ksp-job-api`, conformément au découpage du plan 027.
|
||||||
99
deltas/0.3.6/pre.002-fix.001.md
Normal file
99
deltas/0.3.6/pre.002-fix.001.md
Normal file
@@ -0,0 +1,99 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.002-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.002-fix.001` — compilation des tests lifecycle
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.002
|
||||||
|
workspace.package.version = 0.3.6-pre.2
|
||||||
|
SHA-256 archive = d49f551ce2c04aeb3b545bfc11f0d66852ca10fd75b65c8614a8b99a3eb6cb0f
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger strictement l'erreur de compilation découverte par le gate opérateur dans `unit_tests/lifecycle.rs`, sans modifier l'API publique, le lifecycle, le token d'annulation ni le périmètre fonctionnel de `pre.002`.
|
||||||
|
|
||||||
|
## Défaut observé
|
||||||
|
|
||||||
|
Le premier test créait une variable locale nommée `lifecycle`, puis tentait plus bas de rappeler le helper homonyme `lifecycle()`. Le binding local masquait alors la fonction et provoquait `E0618`; l'inférence qui suivait produisait également `E0282`.
|
||||||
|
|
||||||
|
`cargo check --workspace` était vert parce qu'il ne compilait pas les modules `#[cfg(test)]`. Clippy `--all-targets` et `cargo test -p ksp-job-api` ont correctement exposé le défaut.
|
||||||
|
|
||||||
|
## Correction
|
||||||
|
|
||||||
|
Le helper privé de fixture devient `new_lifecycle()` et ses six appels sont synchronisés. Le nom distinct ne peut plus être masqué par les variables `lifecycle` représentant les instances testées. Les dix tests unitaires et onze canaries d'intégration restent inchangés en nombre et en intention.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.6/pre.002-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-api/unit_tests/lifecycle.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Mécanique Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
header version: 395 -> 396
|
||||||
|
workspace.package.version: 0.3.6-pre.2 -> 0.3.6-pre.2.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le bump est obligatoire car le correctif modifie un fichier Rust compilé par les tests.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
- conserver la fixture commune et lui donner un nom non ambigu ;
|
||||||
|
- ne pas réécrire les tests ni élargir leur périmètre dans ce fix ;
|
||||||
|
- conserver exactement la surface publique et la matrice de transitions de `pre.002` ;
|
||||||
|
- ne pas ouvrir les notifications de `pre.003` avant un gate opérateur vert.
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean (264 tables / 143 fichiers)
|
||||||
|
|
||||||
|
python3 -m unittest scripts/tests/test_audit_markdown_tables.py
|
||||||
|
-> 5 tests / OK
|
||||||
|
```
|
||||||
|
|
||||||
|
La reconstruction de la base et la comparaison différentielle confirment que seuls les cinq fichiers listés appartiennent au correctif.
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo` ni `rustfmt`. Les commandes Rust restent à rejouer par l'opérateur ; elles ne sont pas annoncées comme réussies dans ce delta.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-api
|
||||||
|
cargo tree -p ksp-job-api --edges normal
|
||||||
|
cargo tree -p ksp-job-api -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.003` après un gate opérateur vert.
|
||||||
164
deltas/0.3.6/pre.002.md
Normal file
164
deltas/0.3.6/pre.002.md
Normal file
@@ -0,0 +1,164 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.002` — identité, lifecycle et annulation Job API
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.001-fix.001
|
||||||
|
workspace.package.version = 0.3.6-pre.1.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur communiqué pour la base passe les cinq tests Python du validateur Markdown, l'audit de 264 tables dans 141 fichiers, l'audit Rust workspace et `cargo check --workspace` sur les 14 crates existantes.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Créer la fondation runtime-neutral de `ksp-job-api` avec identité bornée, lifecycle explicite et annulation coopérative, sans ouvrir les notifications de `pre.003` ni aucun contrat Backfill ou Worker.
|
||||||
|
|
||||||
|
## Surface ajoutée
|
||||||
|
|
||||||
|
- `JobId` : code opaque de 1 à 128 octets, alphabet ASCII alphanumérique plus `_`, `-`, `.` et `:` ;
|
||||||
|
- `JobKindCode` : code stable soumis aux mêmes bornes ;
|
||||||
|
- `JobCompletion` : `Complete` ou `Partial` ;
|
||||||
|
- `JobState` : `Created`, `Running`, `Cancelling`, `Completed`, `Cancelled` ou `Failed` ;
|
||||||
|
- `JobLifecycle` : propriétaire non cloneable des transitions ;
|
||||||
|
- `JobCancellationToken` : demande atomique cloneable, idempotente, `Send + Sync` et indépendante de Tokio ;
|
||||||
|
- trois `ErrorCode` stables sous le domaine `job_api` ;
|
||||||
|
- constantes publiques de bornes d'identité.
|
||||||
|
|
||||||
|
## Invariants de lifecycle
|
||||||
|
|
||||||
|
```text
|
||||||
|
Created -> Running
|
||||||
|
Created -> Cancelled
|
||||||
|
Running -> Cancelling
|
||||||
|
Running -> Completed
|
||||||
|
Running -> Failed
|
||||||
|
Cancelling -> Completed
|
||||||
|
Cancelling -> Cancelled
|
||||||
|
Cancelling -> Failed
|
||||||
|
```
|
||||||
|
|
||||||
|
Toute autre transition échoue avec les codes source/cible en contexte sûr et laisse l'état inchangé. Un état terminal n'accepte plus aucune mutation. Le lifecycle n'est pas cloneable afin qu'une copie divergente ne puisse pas créer un second état de vérité.
|
||||||
|
|
||||||
|
Le token d'annulation est volontairement distinct du lifecycle : un consumer demande l'annulation de façon idempotente ; le producteur observe le token puis applique la transition appropriée. Aucun thread, task, callback ou primitive runtime n'est créé par l'API.
|
||||||
|
|
||||||
|
## Firewalls
|
||||||
|
|
||||||
|
La crate dépend exactement de `ksp-core-lib`. Elle n'a aucune feature, dev-dependency ou build-dependency et n'expose aucun type Config, Interface, Logging, Transport, Store, Tokio, serde, Worker ou Backfill.
|
||||||
|
|
||||||
|
Les notifications, séquences, snapshots et listeners restent réservés à `pre.003`.
|
||||||
|
|
||||||
|
## Tests ajoutés
|
||||||
|
|
||||||
|
Dix unitaires :
|
||||||
|
|
||||||
|
- admission exacte et rejets des deux identités ;
|
||||||
|
- alphabet sûr ;
|
||||||
|
- partage, idempotence et état initial du token ;
|
||||||
|
- chemins terminaux, transitions depuis `Cancelling`, erreur sûre, immuabilité terminale et matrice exacte 6 par 6.
|
||||||
|
|
||||||
|
Onze canaries d'intégration :
|
||||||
|
|
||||||
|
- 2 dépendances et runtime-neutralité ;
|
||||||
|
- 3 API publique externe ;
|
||||||
|
- 3 complétude et non-ouverture des tranches futures ;
|
||||||
|
- 3 sécurité, redaction et visibilité cross-thread.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-api/Cargo.toml
|
||||||
|
crates/ksp-job-api/src/cancellation.rs
|
||||||
|
crates/ksp-job-api/src/error.rs
|
||||||
|
crates/ksp-job-api/src/identity.rs
|
||||||
|
crates/ksp-job-api/src/lib.rs
|
||||||
|
crates/ksp-job-api/src/lifecycle.rs
|
||||||
|
crates/ksp-job-api/unit_tests/cancellation.rs
|
||||||
|
crates/ksp-job-api/unit_tests/identity.rs
|
||||||
|
crates/ksp-job-api/unit_tests/lifecycle.rs
|
||||||
|
crates/ksp-job-api/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-api/tests/public_api.rs
|
||||||
|
crates/ksp-job-api/tests/release_completeness.rs
|
||||||
|
crates/ksp-job-api/tests/security_hardening.rs
|
||||||
|
deltas/0.3.6/pre.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Mécanique Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
header version: 394 -> 395
|
||||||
|
workspace.package.version: 0.3.6-pre.1.fix.1 -> 0.3.6-pre.2
|
||||||
|
workspace members: ajout de crates/ksp-job-api
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
- l'hôte fournit le `JobId` pour pouvoir le conserver lors d'une reprise contrôlée ;
|
||||||
|
- aucune génération UUID ou dépendance aléatoire n'entre dans Job API ;
|
||||||
|
- les identités ne sont pas sérialisables par défaut et leurs champs restent privés ;
|
||||||
|
- `JobId` est masqué en `Debug`, alors que le kind stable reste visible ;
|
||||||
|
- le lifecycle est possédé, passif et non cloneable ;
|
||||||
|
- l'annulation partage uniquement un `AtomicBool` standard ;
|
||||||
|
- `Failed` n'embarque pas encore de cause optionnelle ; les snapshots structurés de `pre.003` porteront l'observabilité sans Option-soup ;
|
||||||
|
- aucune README/USAGE intermédiaire n'est créée avant la surface consommable réconciliée prévue par le couloir documentaire.
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean (264 tables / 142 fichiers)
|
||||||
|
|
||||||
|
python3 -m unittest scripts/tests/test_audit_markdown_tables.py
|
||||||
|
-> 5 tests / OK
|
||||||
|
```
|
||||||
|
|
||||||
|
La reconstruction indépendante de la base par extraction successive de v0.3.5, `pre.001` puis `pre.001-fix.001` confirme que le delta ne touche que les 17 fichiers ajoutés ou modifiés listés ci-dessus. Les inventaires statiques confirment dix tests unitaires, onze canaries d'intégration et la dépendance normale unique vers Core.
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo` ni `rustfmt`. Les commandes suivantes ne sont pas annoncées comme réussies :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-api
|
||||||
|
cargo tree -p ksp-job-api --edges normal
|
||||||
|
cargo tree -p ksp-job-api -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-api
|
||||||
|
cargo tree -p ksp-job-api --edges normal
|
||||||
|
cargo tree -p ksp-job-api -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.003` après un gate opérateur vert.
|
||||||
127
deltas/0.3.6/pre.003-fix.001.md
Normal file
127
deltas/0.3.6/pre.003-fix.001.md
Normal file
@@ -0,0 +1,127 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.003-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.003-fix.001` — visibilité du constructeur privé de séquence
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.003 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.3
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour `pre.003` confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 144 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets FAIL / E0423
|
||||||
|
cargo test -p ksp-job-api FAIL / E0423
|
||||||
|
cargo tree -p ksp-job-api --edges normal Core-only confirmé
|
||||||
|
cargo tree -p ksp-job-api -e features aucune feature Job API
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger exclusivement la compilation du helper de test qui construit la position terminale artificielle nécessaire à la canarie d'overflow de `JobNotificationSequence`, sans modifier le contrat public latest-value de `pre.003`.
|
||||||
|
|
||||||
|
## Cause exacte
|
||||||
|
|
||||||
|
Le helper privé est défini dans `crates/ksp-job-api/src/notification.rs`, le module propriétaire du tuple struct :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[cfg(test)]
|
||||||
|
fn exhausted_notification_sequence() -> crate::JobNotificationSequence {
|
||||||
|
return crate::JobNotificationSequence(u64::MAX);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Le champ de `JobNotificationSequence` est volontairement privé afin d'empêcher la construction publique depuis un entier arbitraire. Le type est réexporté à la racine de crate, mais ce re-export ne rend pas son constructeur tuple visible. Le chemin `crate::JobNotificationSequence(u64::MAX)` déclenche donc `E0423` sous les targets de test.
|
||||||
|
|
||||||
|
Le module `notification` possède déjà l'accès légitime au champ privé. Le correctif utilise directement le type local :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
return JobNotificationSequence(u64::MAX);
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette construction reste confinée au helper `#[cfg(test)]`; aucun constructeur arbitraire n'est ajouté à l'API publique.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le fix modifie un fichier Rust. Conformément à `VER-ID-003`, `VER-ID-007` et `VER-ID-010` :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.3.fix.1
|
||||||
|
delivery = 0.3.6-pre.003-fix.001
|
||||||
|
commit = v0.3.6-pre.003-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.6/pre.003-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-api/src/notification.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
- conserver le champ tuple de `JobNotificationSequence` privé ;
|
||||||
|
- ne pas ajouter de `from_u64`, constructeur caché public, feature de test ou autre contournement de l'invariant ;
|
||||||
|
- limiter la construction de `u64::MAX` au helper interne `#[cfg(test)]` déjà prévu par `pre.003` ;
|
||||||
|
- ne modifier ni les treize tests unitaires attendus, ni les quatorze canaries d'intégration, ni leurs assertions ;
|
||||||
|
- ne rouvrir aucune dépendance, feature, surface Backfill/Worker/Transport/Store/Logging, README, USAGE, CHANGELOG, ROADMAP ou prompt.
|
||||||
|
|
||||||
|
## Validations exécutées pour l'assemblage du fix
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean (264 tables / 145 fichiers)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le diff par rapport à `pre.003` est limité aux quatre fichiers modifiés ci-dessus et au présent delta.
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo` ni `rustfmt`. Le gate Rust du fix n'est donc pas annoncé comme vert localement.
|
||||||
|
|
||||||
|
Aucun smoke réseau ou PostgreSQL n'est pertinent pour ce correctif de compilation Job API.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-api
|
||||||
|
cargo tree -p ksp-job-api --edges normal
|
||||||
|
cargo tree -p ksp-job-api -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu : treize tests unitaires et quatorze canaries d'intégration `ksp-job-api` verts, avec le graphe normal toujours limité à `ksp-core-lib` et aucune feature propre à Job API.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune. `pre.004` reste interdit tant que ce gate de fix n'est pas intégralement vert.
|
||||||
180
deltas/0.3.6/pre.003.md
Normal file
180
deltas/0.3.6/pre.003.md
Normal file
@@ -0,0 +1,180 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.003.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.003` — notifications Job latest-value génériques
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.002-fix.001
|
||||||
|
workspace.package.version = 0.3.6-pre.2.fix.1
|
||||||
|
SHA-256 archive = 0b6640155786b92e27508637af2bcc7a0a34e5e8e4185d76ac4dd6d71aa7a869
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour cette base confirme `cargo fmt`, audits Rust/Markdown, `cargo check --workspace`, Clippy, les dix tests unitaires et onze canaries `ksp-job-api`, les suites individuelles, `cargo test --workspace`, les arbres Cargo Job API et l'examen des doublons.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser la tranche `pre.003` du plan 027 : observation Job latest-value runtime-neutral avec séquence monotone, snapshot complet courant, attente abstraite d'une valeur plus récente et resynchronisation multi-listeners, sans ouvrir encore `ksp-job-backfill-lib` ni imposer Tokio à l'API.
|
||||||
|
|
||||||
|
## Audit des archives et des règles
|
||||||
|
|
||||||
|
Les deux archives fournies ont été testées avant modification :
|
||||||
|
|
||||||
|
```text
|
||||||
|
KSP 0.3.6-pre.002-fix.001
|
||||||
|
SHA-256 = 0b6640155786b92e27508637af2bcc7a0a34e5e8e4185d76ac4dd6d71aa7a869
|
||||||
|
1541 entrées / aucune duplication / aucune traversée / aucun chemin absolu / aucun lien symbolique ZIP
|
||||||
|
unzip -t = propre
|
||||||
|
|
||||||
|
kbot3 v0.5.3-pre.005-fix010
|
||||||
|
SHA-256 = ee47643b9f8b582ee8db97b2381ec107e45aef8c009fee44757531514615d318
|
||||||
|
2501 entrées / aucune duplication / aucune traversée / aucun chemin absolu / aucun lien symbolique ZIP
|
||||||
|
unzip -t = propre
|
||||||
|
```
|
||||||
|
|
||||||
|
La base KSP ne contient ni `Cargo.lock`, ni `.env`, ni `target`, ni `node_modules`, ni secret ou artefact de build détecté par l'inventaire statique. L'archive kbot3 reste une référence historique et contient notamment un `scripts/__pycache__`; ce contenu n'est ni copié ni livré.
|
||||||
|
|
||||||
|
Les autorités relues incluent `RULES.md`, `docs/000-README.md`, les règles sous `docs/rules/`, les architectures Layers/Dependencies, Component Contracts, Dependency Graph et Acquisition/Workers/Jobs, le prompt 025, le plan 027, la validation 023 et les deltas `pre.001` à `pre.002-fix.001`.
|
||||||
|
|
||||||
|
Pour cette tranche, kbot3 confirme uniquement le besoin fonctionnel historique : `BackfillObserver::on_progress` appelait directement le consumer applicatif et exposait timestamp, niveau, message libre et compteurs optionnels. KSP conserve l'observabilité mais rejette cette forme callback/producteur ; aucun code, DTO, ID, message ou runtime kbot3 n'est repris.
|
||||||
|
|
||||||
|
## Surface ajoutée
|
||||||
|
|
||||||
|
`JobNotificationSequence` :
|
||||||
|
|
||||||
|
- position initiale contrôlée par l'API ;
|
||||||
|
- progression unitaire par `checked_add` ;
|
||||||
|
- ordre total et test `is_after` ;
|
||||||
|
- erreur stable `job_api.job_notification_sequence_exhausted` avant tout wrap.
|
||||||
|
|
||||||
|
`JobNotification<S>` :
|
||||||
|
|
||||||
|
- `JobId`, `JobKindCode`, séquence, `JobState` et snapshot complet ;
|
||||||
|
- champs privés et valeur immutable ;
|
||||||
|
- `Debug` structurel qui masque toujours le snapshot générique.
|
||||||
|
|
||||||
|
`JobSnapshotFuture<'a, S>` et `JobSnapshotSource` :
|
||||||
|
|
||||||
|
- futur abstrait fondé uniquement sur `std::future::Future`, `Pin` et `Box` ;
|
||||||
|
- lecture du snapshot courant ;
|
||||||
|
- attente d'une valeur plus récente après une séquence observée ;
|
||||||
|
- coalescence autorisée des progressions intermédiaires, avec resynchronisation sur le snapshot complet courant ;
|
||||||
|
- aucun callback producteur, channel Tokio, event log, Tauri ou historique public.
|
||||||
|
|
||||||
|
La source runtime concrète n'est pas créée dans Job API. Elle restera possédée par `ksp-job-backfill-lib` lorsque cette crate ouvrira ; les canaries de `pre.003` prouvent le contrat externe, sans annoncer comme acquises les propriétés runtime O(1), wake-up ou isolation du producteur avant ce consumer réel.
|
||||||
|
|
||||||
|
## Tests ajoutés
|
||||||
|
|
||||||
|
Trois unitaires :
|
||||||
|
|
||||||
|
- progression stricte et ordre de séquence ;
|
||||||
|
- épuisement `u64::MAX` explicite sans wrap ;
|
||||||
|
- conservation de la valeur et redaction du snapshot en `Debug`.
|
||||||
|
|
||||||
|
Trois canaries externes :
|
||||||
|
|
||||||
|
- consommation de l'enveloppe depuis le crate-root ;
|
||||||
|
- deux listeners indépendants resynchronisés vers la dernière valeur après séquences anciennes différentes ;
|
||||||
|
- snapshot terminal encore lisible depuis un handle partagé.
|
||||||
|
|
||||||
|
Les canaries de dépendances et de complétude existantes sont étendues pour couvrir `notification.rs`, les nouveaux exports et l'absence de fuite Backfill/Worker/runtime.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-api/src/notification.rs
|
||||||
|
crates/ksp-job-api/tests/notifications.rs
|
||||||
|
crates/ksp-job-api/unit_tests/notification.rs
|
||||||
|
deltas/0.3.6/pre.003.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-api/src/error.rs
|
||||||
|
crates/ksp-job-api/src/lib.rs
|
||||||
|
crates/ksp-job-api/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-api/tests/release_completeness.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Mécanique Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
header version: 396 -> 397
|
||||||
|
workspace.package.version: 0.3.6-pre.2.fix.1 -> 0.3.6-pre.3
|
||||||
|
workspace members: inchangés
|
||||||
|
workspace dependencies: inchangées
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Décisions prises
|
||||||
|
|
||||||
|
- une séquence publique ne se construit pas depuis un entier arbitraire ; elle part de `initial()` puis progresse via `next()` ;
|
||||||
|
- `JobNotification<S>` reste générique afin que Job API ne possède aucun snapshot Backfill prématuré ;
|
||||||
|
- le `Debug` de l'enveloppe ne demande pas `S: Debug` et ne rend jamais le snapshot ;
|
||||||
|
- le contrat d'attente expose un futur `std` boxed plutôt qu'un type Tokio ou un callback ;
|
||||||
|
- la sémantique publique est latest-value, pas une queue et pas un event log durable ;
|
||||||
|
- une séquence manquée n'exige aucun replay : le listener reprend le snapshot complet le plus récent ;
|
||||||
|
- la source concrète et son wake-up ne sont pas simulés comme fonctionnalité de production dans Job API ; ils seront prouvés avec le premier consumer réel ;
|
||||||
|
- `ksp-job-api` conserve exactement sa dépendance normale Core-only et aucune feature/dev-dependency/build-dependency n'est ajoutée ;
|
||||||
|
- aucun README/USAGE intermédiaire n'est créé avant la surface consommable réconciliée prévue par le plan.
|
||||||
|
|
||||||
|
## Validations exécutées
|
||||||
|
|
||||||
|
```text
|
||||||
|
unzip -tq des deux archives
|
||||||
|
-> propre
|
||||||
|
|
||||||
|
inventaire ZIP
|
||||||
|
-> KSP: 1541 entrées, 0 duplicate, 0 traversal, 0 absolute, 0 symlink
|
||||||
|
-> kbot3: 2501 entrées, 0 duplicate, 0 traversal, 0 absolute, 0 symlink
|
||||||
|
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean (264 tables / 144 fichiers)
|
||||||
|
|
||||||
|
python3 -m unittest scripts/tests/test_audit_markdown_tables.py
|
||||||
|
-> 5 tests / OK
|
||||||
|
|
||||||
|
inventaire statique ksp-job-api
|
||||||
|
-> 13 tests unitaires
|
||||||
|
-> 14 canaries d'intégration
|
||||||
|
```
|
||||||
|
|
||||||
|
L'inventaire différentiel base/travail confirme que seuls les fichiers listés par ce delta sont ajoutés ou modifiés, hors caches Python temporaires supprimés avant assemblage.
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo` ni `rustfmt`. Les commandes Rust de `pre.003` ne sont donc pas annoncées comme réussies localement.
|
||||||
|
|
||||||
|
Le smoke réseau et PostgreSQL ne concernent pas cette tranche API-only.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-api
|
||||||
|
cargo tree -p ksp-job-api --edges normal
|
||||||
|
cargo tree -p ksp-job-api -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Le résultat attendu pour `ksp-job-api` après cette tranche est treize tests unitaires et quatorze canaries d'intégration, sans nouvelle dépendance normale ou feature.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.004` après un gate opérateur vert de `pre.003`.
|
||||||
146
deltas/0.3.6/pre.004.md
Normal file
146
deltas/0.3.6/pre.004.md
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.004.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.004` — provenance Transport observée pour `getTransaction`
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.003-fix.001 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.3.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour cette base confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 145 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS
|
||||||
|
cargo test -p ksp-job-api PASS / 13 unitaires + 14 canaries
|
||||||
|
cargo tree -p ksp-job-api --edges normal Core-only confirmé
|
||||||
|
cargo tree -p ksp-job-api -e features aucune feature Job API
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser exclusivement la tranche `pre.004` du plan 027 : permettre au futur backfill de connaître le provider et l'endpoint HTTP qui ont réellement produit le succès de `getTransaction`, y compris après retry/reroutage, sans dupliquer le client RPC ni déplacer la politique Transport vers Job.
|
||||||
|
|
||||||
|
## Conception
|
||||||
|
|
||||||
|
La nouvelle enveloppe générique `HttpObservedValue<T>` contient uniquement :
|
||||||
|
|
||||||
|
- la valeur typée `T` ;
|
||||||
|
- le nom configuré et validé de l'endpoint victorieux ;
|
||||||
|
- le descripteur provider de cet endpoint.
|
||||||
|
|
||||||
|
Elle ne contient jamais :
|
||||||
|
|
||||||
|
- URL d'endpoint ;
|
||||||
|
- headers HTTP ;
|
||||||
|
- body HTTP brut ;
|
||||||
|
- credentials ou autre détail de connexion.
|
||||||
|
|
||||||
|
Son `Debug` conserve l'identité sûre de routage mais remplace systématiquement la valeur typée par `<available>` afin qu'un diagnostic générique ne rende pas accidentellement un payload transactionnel volumineux.
|
||||||
|
|
||||||
|
Le moteur HTTP commun n'est pas dupliqué. `execute_standard_rpc_with` possède toujours l'unique boucle de support, request-id, admission, deadline, retry, cooldown, HTTP, parsing JSON-RPC et accounting. Une projection de succès interne décide seulement de la forme de retour :
|
||||||
|
|
||||||
|
- `execute_standard_rpc` retourne la valeur historique sans allouer de provenance ;
|
||||||
|
- la voie interne observée capture endpoint/provider uniquement après le succès final ;
|
||||||
|
- `get_transaction_observed` réutilise cette voie et retourne `HttpObservedValue<Option<SolanaConfirmedTransaction>>`.
|
||||||
|
|
||||||
|
Les validations de paramètres `getTransaction` sont factorisées dans un helper commun. `get_transaction` et `get_transaction_observed` rejettent donc exactement les mêmes entrées et utilisent le même décodage typé. L'API historique reste source-compatible.
|
||||||
|
|
||||||
|
## Tests ajoutés
|
||||||
|
|
||||||
|
Deux tests unitaires Transport :
|
||||||
|
|
||||||
|
- `typed_get_transaction_observed_reports_actual_winner_after_retry_reroute` : deux endpoints de même priorité, premier résultat HTTP `429`, retry sur le second ; la valeur observée doit rapporter `winner-endpoint` / `winner-provider`, pas le candidat initial ;
|
||||||
|
- `typed_get_transaction_observed_preserves_null_and_redacts_typed_value_debug` : `result: null` reste `None`, endpoint/provider restent disponibles et le `Debug` ne rend pas la valeur typée.
|
||||||
|
|
||||||
|
Une canarie publique :
|
||||||
|
|
||||||
|
- `public_v0_3_6_pre_004_observed_get_transaction_surface_is_available_from_crate_root` : méthode et enveloppe observée sont consommables depuis le crate-root.
|
||||||
|
|
||||||
|
Aucun fixture wire nouveau n'est nécessaire : les fixtures `get_transaction.base64.json` et `get_transaction.null.json` existantes sont réutilisées.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.6/pre.004.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-onchain-transport-lib/src/http_executor.rs
|
||||||
|
crates/ksp-onchain-transport-lib/src/lib.rs
|
||||||
|
crates/ksp-onchain-transport-lib/src/rpc_transactions.rs
|
||||||
|
crates/ksp-onchain-transport-lib/tests/public_api.rs
|
||||||
|
crates/ksp-onchain-transport-lib/unit_tests/rpc_transactions.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Mécanique Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
header version: 398 -> 399
|
||||||
|
workspace.package.version: 0.3.6-pre.3.fix.1 -> 0.3.6-pre.4
|
||||||
|
workspace members: inchangés
|
||||||
|
workspace dependencies: inchangées
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Frontières conservées
|
||||||
|
|
||||||
|
- aucune crate Job ou Store n'entre dans Transport ;
|
||||||
|
- aucune dépendance ou feature n'est ajoutée ;
|
||||||
|
- aucune sélection d'endpoint, pause ou boucle de retry n'est ajoutée dans Job ;
|
||||||
|
- aucune nouvelle lecture Config/env n'est introduite ;
|
||||||
|
- aucun code, DTO, client, retry loop ou provenance kbot3 n'est copié ; la matrice fonctionnelle `pre.001` reste seulement une référence de besoin ;
|
||||||
|
- le legacy `get_transaction_legacy` reste inchangé et non observé ; le backfill v0.3.6 utilisera le formulaire moderne objet ;
|
||||||
|
- README/USAGE restent fermés jusqu'à la tranche de réconciliation documentaire prévue par le plan.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean
|
||||||
|
|
||||||
|
python3 -m unittest scripts/tests/test_audit_markdown_tables.py
|
||||||
|
-> 5 tests / OK
|
||||||
|
```
|
||||||
|
|
||||||
|
Le premier passage de l'auditeur Rust a détecté deux rustdocs manquantes sur les helpers `pub(crate)` de `HttpObservedValue`; elles ont été ajoutées avant assemblage et le second passage est intégralement propre.
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustfmt`, ni `rustc`. Aucune compilation ou suite Rust de `pre.004` n'est donc annoncée comme exécutée localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-onchain-transport-lib
|
||||||
|
cargo tree -p ksp-onchain-transport-lib --edges normal
|
||||||
|
cargo tree -p ksp-onchain-transport-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate doit notamment confirmer le retry/reroutage déterministe du nouveau test observé et l'absence de nouvelle dépendance/feature Transport.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.005` après un gate opérateur vert de `pre.004`.
|
||||||
129
deltas/0.3.6/pre.005-fix.001.md
Normal file
129
deltas/0.3.6/pre.005-fix.001.md
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.005-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.005-fix.001` — return explicite du curseur Before
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.005 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.5
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour `pre.005` confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 147 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets FAIL / clippy::implicit_return
|
||||||
|
cargo test -p ksp-job-backfill-lib PASS
|
||||||
|
unitaires 11 PASS
|
||||||
|
dependency_boundary 2 PASS
|
||||||
|
public_api 2 PASS
|
||||||
|
release_completeness 2 PASS
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Corriger exclusivement l'écart Clippy `implicit_return` dans la closure privée qui construit le curseur initial `before` du scope `BeforeAddress`, sans modifier la sémantique de découverte de `pre.005`.
|
||||||
|
|
||||||
|
## Cause exacte
|
||||||
|
|
||||||
|
Dans `crates/ksp-job-backfill-lib/src/discovery.rs`, `discover_older` initialise le curseur optionnel à partir de l'ancre du scope :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
crate::BackfillScopeKind::BeforeAddress => request.scope().anchor().map(|value| value.as_str().to_owned()),
|
||||||
|
```
|
||||||
|
|
||||||
|
Le workspace interdit les retours implicites via `-D clippy::implicit-return`. La closure de `Option::map` doit donc rendre son résultat avec un `return` explicite.
|
||||||
|
|
||||||
|
Le correctif est strictement syntaxique :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
crate::BackfillScopeKind::BeforeAddress => request.scope().anchor().map(|value| return value.as_str().to_owned()),
|
||||||
|
```
|
||||||
|
|
||||||
|
La valeur obtenue, l'exclusivité de l'ancre, l'ordre de pagination, la déduplication, les bornes et tous les contrats publics restent identiques.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le fix modifie un fichier Rust. Conformément au workflow de version :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.5.fix.1
|
||||||
|
delivery = 0.3.6-pre.005-fix.001
|
||||||
|
commit = v0.3.6-pre.005-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.6/pre.005-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/discovery.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
- identité transactionnelle/candidate toujours `(RawNetworkId, signature)` ;
|
||||||
|
- provider, endpoint, protocole et rôle Transport restent hors identité ;
|
||||||
|
- aucune modification des quatre scopes, de leurs bornes ou du fingerprint ;
|
||||||
|
- aucune modification du chemin `HttpTransportPool::get_signatures_for_address` ;
|
||||||
|
- aucune nouvelle dépendance, feature, crate, API publique ou test ;
|
||||||
|
- aucune ouverture de RAW, persistance, checkpoint, runtime, README, USAGE, CHANGELOG, ROADMAP ou prompt.
|
||||||
|
|
||||||
|
## Validations exécutées pour l'assemblage du fix
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean (264 tables / 148 fichiers)
|
||||||
|
```
|
||||||
|
|
||||||
|
Le diff par rapport à `pre.005` est limité aux quatre fichiers modifiés ci-dessus et au présent delta.
|
||||||
|
|
||||||
|
## Validations non exécutées
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo` ni `rustfmt`. Le gate Rust du fix n'est donc pas annoncé comme vert localement.
|
||||||
|
|
||||||
|
Aucun smoke réseau ou PostgreSQL n'est pertinent pour ce correctif Clippy.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu : les onze tests unitaires et six canaries d'intégration restent verts, Clippy devient vert et le graphe de dépendances reste inchangé.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune. `pre.006` reste interdit tant que ce gate de fix n'est pas intégralement vert.
|
||||||
286
deltas/0.3.6/pre.005.md
Normal file
286
deltas/0.3.6/pre.005.md
Normal file
@@ -0,0 +1,286 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.005.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.005` — fondation Backfill et découverte bornée
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.004 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.4
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour cette base confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 146 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS
|
||||||
|
cargo test -p ksp-onchain-transport-lib PASS
|
||||||
|
unitaires 385 PASS
|
||||||
|
public_api 50 PASS
|
||||||
|
release_completeness 43 PASS
|
||||||
|
doctests 4 PASS
|
||||||
|
cargo tree -p ksp-onchain-transport-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-onchain-transport-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
Les smokes réseau restent opt-in et ignorés dans le gate normal, conformément à leur contrat.
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser exclusivement la tranche `pre.005` du plan 027 : créer `ksp-job-backfill-lib` et figer la requête, les scopes, l'identité des candidats, les bornes ainsi que la découverte paginée déterministe avant toute hydratation `getTransaction`, conversion RAW, persistance Store, checkpoint ou runtime de Job.
|
||||||
|
|
||||||
|
Cette tranche résout explicitement un invariant d'identité qui ne doit jamais dépendre de la topologie d'acquisition : une transaction/signature est identifiée logiquement par son réseau et sa signature, pas par le provider, l'endpoint ou le protocole qui l'a retrouvée.
|
||||||
|
|
||||||
|
## Identité réseau contre provenance Transport
|
||||||
|
|
||||||
|
Le contrat de découverte introduit :
|
||||||
|
|
||||||
|
```text
|
||||||
|
BackfillCandidateIdentity = (RawNetworkId, BackfillSignature)
|
||||||
|
```
|
||||||
|
|
||||||
|
`RawNetworkId` est un identifiant logique ouvert réexporté par `ksp-store-lib`. Des namespaces comme `mainnet`, `devnet`, `testnet`, `localnet`, `synthetic` ou un futur réseau nommé restent donc distincts sans imposer un enum fermé dans Job.
|
||||||
|
|
||||||
|
En conséquence :
|
||||||
|
|
||||||
|
- la même signature observée sur deux réseaux différents représente deux identités distinctes ;
|
||||||
|
- la même signature acquise plusieurs fois sur le même réseau via HTTP, WebSocket ou gRPC converge vers la même identité transactionnelle ;
|
||||||
|
- rôle Transport, provider, endpoint et protocole décrivent la sélection ou la provenance d'acquisition ; ils ne participent jamais à l'identité transactionnelle ;
|
||||||
|
- `JobId` et la future concurrence d'hydratation ne participent pas non plus au fingerprint sémantique d'un scope.
|
||||||
|
|
||||||
|
Le Store API possède déjà `RawTransactionReference { network, signature }`. Le backend PostgreSQL actuel lie en outre un Store physique à un seul `RawNetworkId` via `ksp_store_identity`; une clé physique locale basée sur la signature y reste donc dans un namespace réseau unique. Tout futur backend hébergeant plusieurs réseaux dans le même namespace physique devra préserver l'identité logique `(network, signature)` par sa clé, sa partition ou un mécanisme équivalent.
|
||||||
|
|
||||||
|
## Crate `ksp-job-backfill-lib`
|
||||||
|
|
||||||
|
La nouvelle crate est une bibliothèque comportementale indépendante, sans exécutable ni Worker. Ses dépendances normales sont limitées à :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp-core-lib
|
||||||
|
ksp-job-api
|
||||||
|
ksp-logging-lib
|
||||||
|
ksp-onchain-transport-lib
|
||||||
|
ksp-store-lib (default-features = false)
|
||||||
|
sha2 (workspace)
|
||||||
|
```
|
||||||
|
|
||||||
|
Tokio est uniquement une `dev-dependency` avec les features de test nécessaires aux canaries async. Aucun backend Store n'est activé par défaut par Backfill.
|
||||||
|
|
||||||
|
La crate ne dépend pas directement de Config, Interface, Program, Wallet, Store API, Store PostgreSQL, reqwest, tonic ou d'une crate protocolaire Solana.
|
||||||
|
|
||||||
|
## Requête bornée
|
||||||
|
|
||||||
|
`BackfillRequest` impose explicitement :
|
||||||
|
|
||||||
|
- `JobId` ;
|
||||||
|
- `RawNetworkId` ;
|
||||||
|
- rôle HTTP Transport ;
|
||||||
|
- engagement `Confirmed` ou `Finalized` ;
|
||||||
|
- un des quatre scopes ;
|
||||||
|
- page size `1..=1000` ;
|
||||||
|
- max pages `1..=10000` ;
|
||||||
|
- max candidates `1..=10000` ;
|
||||||
|
- future concurrence d'hydratation `1..=64` ;
|
||||||
|
- `min_context_slot` optionnel uniquement pour les scopes adresse.
|
||||||
|
|
||||||
|
Le rôle HTTP sert uniquement à la sélection Transport. Il est volontairement exclu du fingerprint du scope avec provider, endpoint et protocole.
|
||||||
|
|
||||||
|
## Scopes
|
||||||
|
|
||||||
|
Quatre scopes sont matérialisés :
|
||||||
|
|
||||||
|
- `LatestAddress(address)` : fenêtre courante depuis le plus récent ;
|
||||||
|
- `BeforeAddress(address, anchor)` : historique plus ancien que l'ancre exclusive ;
|
||||||
|
- `AfterAddress(address, anchor)` : fenêtre plus récente la plus proche d'une ancre exclusive ;
|
||||||
|
- `ExplicitSignatures(signatures)` : liste explicite bornée sans découverte adresse.
|
||||||
|
|
||||||
|
Les signatures explicites sont dédupliquées selon la première occurrence et conservent leur ordre d'entrée.
|
||||||
|
|
||||||
|
`BackfillSignature` valide dans cette tranche uniquement la forme texte Base58 bornée compatible avec une signature Solana de 64 octets. Le décodage Base58 exact vers la signature RAW canonique de 64 octets reste volontairement réservé à `pre.006`; `pre.005` ne prétend donc pas encore valider le contenu binaire canonique.
|
||||||
|
|
||||||
|
## Fingerprint de scope
|
||||||
|
|
||||||
|
`BackfillScopeFingerprint` utilise SHA-256 avec séparation de domaine et couvre les paramètres qui changent réellement la sémantique de découverte :
|
||||||
|
|
||||||
|
- réseau ;
|
||||||
|
- commitment ;
|
||||||
|
- type de scope ;
|
||||||
|
- adresse ou liste explicite ;
|
||||||
|
- direction/ancre ;
|
||||||
|
- page size ;
|
||||||
|
- max pages ;
|
||||||
|
- max candidates ;
|
||||||
|
- `min_context_slot`.
|
||||||
|
|
||||||
|
Il exclut intentionnellement :
|
||||||
|
|
||||||
|
- `JobId` ;
|
||||||
|
- rôle HTTP ;
|
||||||
|
- provider ;
|
||||||
|
- endpoint ;
|
||||||
|
- protocole ;
|
||||||
|
- concurrence d'hydratation.
|
||||||
|
|
||||||
|
Changer de source d'acquisition ne change donc pas l'identité du scope ni celle des transactions candidates.
|
||||||
|
|
||||||
|
## Découverte paginée
|
||||||
|
|
||||||
|
La voie de production appelle uniquement le wrapper typé existant :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HttpTransportPool::get_signatures_for_address
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun client HTTP, retry loop, pacing, cooldown, sélection d'endpoint ou politique fournisseur n'entre dans Job.
|
||||||
|
|
||||||
|
Les règles figées sont :
|
||||||
|
|
||||||
|
- Latest et Before utilisent le curseur `before` et conservent l'ordre RPC du plus récent au plus ancien ;
|
||||||
|
- Before démarre avec l'ancre exclusive fournie ;
|
||||||
|
- After utilise `until = anchor`, avance par `before` et ne conserve que les `max_candidates` les plus proches de l'ancre parmi les signatures plus récentes ;
|
||||||
|
- la déduplication est stable entre pages ;
|
||||||
|
- une page fournisseur plus grande que la limite demandée est rejetée ;
|
||||||
|
- un curseur qui n'avance pas est rejeté ;
|
||||||
|
- une page courte ou vide matérialise `RpcBoundary` ;
|
||||||
|
- atteindre la limite de candidats matérialise `CandidateLimit` ;
|
||||||
|
- épuiser `max_pages` en Latest/Before matérialise `PageLimit` et un résultat partiel ;
|
||||||
|
- épuiser `max_pages` en After avant preuve de la frontière matérialise `AfterAnchorNotReached` et un résultat partiel.
|
||||||
|
|
||||||
|
La borne After est volontairement conservative : un Job ne prétend jamais avoir rejoint l'ancre si la limite de pages empêche de le prouver.
|
||||||
|
|
||||||
|
## Tests matérialisés
|
||||||
|
|
||||||
|
La tranche ajoute **11 tests unitaires** :
|
||||||
|
|
||||||
|
- validation et redaction de signature ;
|
||||||
|
- déduplication explicite stable ;
|
||||||
|
- bornes exactes de requête ;
|
||||||
|
- fingerprint indépendant de JobId/rôle/concurrence mais distinct par réseau ;
|
||||||
|
- distinction sémantique Before/After ;
|
||||||
|
- Latest paginé avec déduplication inter-pages ;
|
||||||
|
- Before avec ancre exclusive et progression du curseur ;
|
||||||
|
- After avec fenêtre la plus proche de l'ancre ;
|
||||||
|
- After borné partiel ;
|
||||||
|
- Latest borné partiel ;
|
||||||
|
- explicite sans appel Transport et avec identité réseau.
|
||||||
|
|
||||||
|
Elle ajoute **6 canaries d'intégration** :
|
||||||
|
|
||||||
|
- deux canaries de dépendances/firewalls ;
|
||||||
|
- deux canaries d'API publique, dont l'identité `(network, signature)` ;
|
||||||
|
- deux canaries de complétude qui empêchent l'ouverture anticipée de RAW/persistance/checkpoint/runtime.
|
||||||
|
|
||||||
|
Les doubles de page sont privés aux unit tests ; aucune abstraction fake n'entre dans l'API publique.
|
||||||
|
|
||||||
|
## Documentation courante réconciliée
|
||||||
|
|
||||||
|
Les documents d'architecture actifs sont alignés avec le nom désormais matérialisé `ksp-job-backfill-lib` et son graphe réel sans Config :
|
||||||
|
|
||||||
|
```text
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Les anciens plans historiques ne sont pas réécrits. README, USAGE, CHANGELOG et ROADMAP restent fermés jusqu'aux tranches prévues de réconciliation/fermeture.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/constants.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/discovery.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/request.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/discovery.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/request.rs
|
||||||
|
deltas/0.3.6/pre.005.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
docs/architecture/004-COMPONENT_INVENTORY.md
|
||||||
|
docs/architecture/005-DEPENDENCY_GRAPH.md
|
||||||
|
docs/architecture/009-ACQUISITION_WORKERS_AND_JOBS.md
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
Mécanique Cargo :
|
||||||
|
|
||||||
|
```text
|
||||||
|
header version: 399 -> 400
|
||||||
|
workspace.package.version: 0.3.6-pre.4 -> 0.3.6-pre.5
|
||||||
|
workspace members: + crates/ksp-job-backfill-lib
|
||||||
|
workspace dependencies versionnées: inchangées
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Frontières conservées
|
||||||
|
|
||||||
|
- aucun code kbot3 n'est copié ; kbot3 reste uniquement une référence fonctionnelle ;
|
||||||
|
- aucune hydratation `getTransaction` n'est encore implémentée dans Job ;
|
||||||
|
- aucune conversion RAW v1 n'est ouverte ;
|
||||||
|
- aucune écriture Store, observation d'acquisition ou politique `ForceRehydrate` n'est ouverte ;
|
||||||
|
- aucun checkpoint, frontier contiguë, runtime Job ou notification concrète n'est ouvert ;
|
||||||
|
- aucune lecture Config/env n'est introduite ;
|
||||||
|
- aucun endpoint/provider/protocole ne devient une clé transactionnelle ;
|
||||||
|
- aucun backend Store n'est imposé ;
|
||||||
|
- aucun README/USAGE/CHANGELOG/ROADMAP n'est rouvert.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
-> General Rust rule audit: clean
|
||||||
|
-> Rust export completeness audit: 0 candidate(s)
|
||||||
|
-> KSP workspace Rust rule audit: clean
|
||||||
|
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
-> Markdown table audit: clean (264 tables / 147 fichiers, présent delta inclus)
|
||||||
|
|
||||||
|
python3 -m unittest scripts/tests/test_audit_markdown_tables.py
|
||||||
|
-> 5 tests / OK
|
||||||
|
```
|
||||||
|
|
||||||
|
Le premier audit Markdown après réconciliation de l'inventaire a détecté uniquement l'élargissement mécanique requis par `ksp-job-backfill-lib`; la table a été réalignée sur le contenu le plus large avant le second passage propre.
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`. Les 11 unitaires et 6 canaries Rust sont matérialisés mais ne sont pas annoncés comme exécutés localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate doit notamment confirmer :
|
||||||
|
|
||||||
|
- 11 unitaires + 6 canaries ;
|
||||||
|
- `ksp-store-lib` sans activation implicite du backend PostgreSQL ;
|
||||||
|
- aucun accès direct à Store API/backend, Config ou dépendance protocolaire ;
|
||||||
|
- pagination Latest/Before/After et explicite déterministe ;
|
||||||
|
- résultat partiel conservateur aux bornes ;
|
||||||
|
- identité candidate scellée par réseau + signature, indépendamment du rôle/provider/endpoint/protocole.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune question ne bloque `pre.006` après un gate opérateur vert de `pre.005`.
|
||||||
128
deltas/0.3.6/pre.006-fix.001.md
Normal file
128
deltas/0.3.6/pre.006-fix.001.md
Normal file
@@ -0,0 +1,128 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.006-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.006-fix.001` — indirection privée de l’acquisition RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.006 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.6
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour `pre.006` confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 149 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS avec warning clippy::large_enum_variant
|
||||||
|
cargo test -p ksp-job-backfill-lib PASS
|
||||||
|
unitaires 19 PASS
|
||||||
|
dependency_boundary 2 PASS
|
||||||
|
public_api 3 PASS
|
||||||
|
release_completeness 2 PASS
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Supprimer le warning `clippy::large_enum_variant` sans masquer le lint et sans changer la forme publique de `BackfillHydrationOutcome`.
|
||||||
|
|
||||||
|
## Cause exacte
|
||||||
|
|
||||||
|
`BackfillHydrationOutcome` expose deux variantes :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
Available(BackfillRawAcquisition),
|
||||||
|
Missing(ksp_store_lib::RawTransactionReference),
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate mesure au moins environ 560 octets pour `Available` contre environ 88 octets pour `Missing`. La différence provient de l’embarquement direct de la transaction RAW et de son observation dans `BackfillRawAcquisition`.
|
||||||
|
|
||||||
|
## Correctif
|
||||||
|
|
||||||
|
La structure publique `BackfillRawAcquisition` conserve ses méthodes et reste la charge utile directe de `Available`, mais ses deux champs privés sont regroupés dans un état privé détenu par une unique `Box`.
|
||||||
|
|
||||||
|
Ainsi :
|
||||||
|
|
||||||
|
- l’API publique `Available(BackfillRawAcquisition)` ne change pas ;
|
||||||
|
- aucun `Box` n’apparaît dans la signature publique de l’enum ;
|
||||||
|
- `transaction()`, `observation()` et `into_parts()` gardent leurs signatures ;
|
||||||
|
- une seule allocation est introduite par acquisition disponible ;
|
||||||
|
- `Missing(reference)` reste sans allocation et sans provenance fabriquée ;
|
||||||
|
- aucun `#[allow(clippy::large_enum_variant)]` n’est ajouté.
|
||||||
|
|
||||||
|
Une canarie unitaire vérifie que `BackfillRawAcquisition` conserve la taille d’un pointeur, ce qui verrouille l’indirection privée destinée à empêcher la régression de taille.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le fix modifie du Rust :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.6.fix.1
|
||||||
|
delivery = 0.3.6-pre.006-fix.001
|
||||||
|
commit = v0.3.6-pre.006-fix.001
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.6/pre.006-fix.001.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/conversion.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/conversion.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions d’en-tête
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml 402 -> 403
|
||||||
|
crates/ksp-job-backfill-lib/src/conversion.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/conversion.rs 1 -> 2
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md 10 -> 11
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md 10 -> 11
|
||||||
|
```
|
||||||
|
|
||||||
|
Le présent delta commence à `version: 1`.
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
- identité logique transactionnelle toujours `(RawNetworkId, signature)` ;
|
||||||
|
- provider/endpoint/protocole restent uniquement de la provenance d’acquisition ;
|
||||||
|
- format RAW v1, canonicalisation JSON, SHA-256 et block time inchangés ;
|
||||||
|
- chemin observé `getTransaction` et paramètres Transport inchangés ;
|
||||||
|
- `Missing` reste structurellement sans provenance ;
|
||||||
|
- aucune persistance Store, checkpoint, concurrence ou snapshot ajoutés ;
|
||||||
|
- aucune nouvelle dépendance, feature, crate ou API publique ;
|
||||||
|
- README, USAGE, CHANGELOG, ROADMAP et prompt restent fermés.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu : **20 tests unitaires + 7 canaries d’intégration**, aucun warning `large_enum_variant`, et graphe de dépendances inchangé.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune. `pre.007` reste interdit tant que le gate de ce fix n’est pas warning-free.
|
||||||
129
deltas/0.3.6/pre.006-fix.002.md
Normal file
129
deltas/0.3.6/pre.006-fix.002.md
Normal file
@@ -0,0 +1,129 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.006-fix.002.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.006-fix.002` — auto-deref idiomatique des accesseurs RAW
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.006-fix.001 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.6.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour `pre.006-fix.001` confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 150 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS avec 2 warnings clippy::explicit_auto_deref
|
||||||
|
cargo test -p ksp-job-backfill-lib PASS
|
||||||
|
unitaires 20 PASS
|
||||||
|
dependency_boundary 2 PASS
|
||||||
|
public_api 3 PASS
|
||||||
|
release_completeness 2 PASS
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Supprimer les deux warnings `clippy::explicit_auto_deref` laissés par `pre.006-fix.001`, sans modifier la représentation, l'API publique ou la sémantique du vertical RAW.
|
||||||
|
|
||||||
|
## Cause exacte
|
||||||
|
|
||||||
|
Après l'introduction de l'indirection privée, les deux getters utilisaient encore un déréférencement explicite :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
&(*self.inner).transaction
|
||||||
|
&(*self.inner).observation
|
||||||
|
```
|
||||||
|
|
||||||
|
Rust effectue déjà cet auto-deref lors de l'accès aux champs d'un `Box<T>`, et Clippy demande donc la forme idiomatique.
|
||||||
|
|
||||||
|
## Correctif
|
||||||
|
|
||||||
|
Les deux expressions deviennent :
|
||||||
|
|
||||||
|
```rust
|
||||||
|
&self.inner.transaction
|
||||||
|
&self.inner.observation
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun autre comportement ne change :
|
||||||
|
|
||||||
|
- `BackfillRawAcquisition` conserve exactement une `Box` privée ;
|
||||||
|
- `BackfillHydrationOutcome::Available(BackfillRawAcquisition)` reste inchangé ;
|
||||||
|
- les signatures de `transaction()`, `observation()` et `into_parts()` restent inchangées ;
|
||||||
|
- la canarie de taille ajoutée en `fix.001` reste inchangée ;
|
||||||
|
- aucune allocation supplémentaire n'est introduite ;
|
||||||
|
- aucun `#[allow]` n'est ajouté.
|
||||||
|
|
||||||
|
## Version
|
||||||
|
|
||||||
|
Le fix modifie du Rust :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.6.fix.2
|
||||||
|
delivery = 0.3.6-pre.006-fix.002
|
||||||
|
commit = v0.3.6-pre.006-fix.002
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
deltas/0.3.6/pre.006-fix.002.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/conversion.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions d'en-tête
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml 403 -> 404
|
||||||
|
crates/ksp-job-backfill-lib/src/conversion.rs 2 -> 3
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md 11 -> 12
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md 11 -> 12
|
||||||
|
```
|
||||||
|
|
||||||
|
Le présent delta commence à `version: 1`.
|
||||||
|
|
||||||
|
## Invariants préservés
|
||||||
|
|
||||||
|
- identité logique transactionnelle toujours `(RawNetworkId, signature)` ;
|
||||||
|
- provider/endpoint/protocole restent uniquement de la provenance d'acquisition ;
|
||||||
|
- format RAW v1, canonicalisation JSON, SHA-256 et block time inchangés ;
|
||||||
|
- chemin observé `getTransaction` et paramètres Transport inchangés ;
|
||||||
|
- `Missing` reste structurellement sans provenance ;
|
||||||
|
- aucune persistance Store, checkpoint, concurrence ou snapshot ajoutés ;
|
||||||
|
- aucune nouvelle dépendance, feature, crate, test ou API publique ;
|
||||||
|
- README, USAGE, CHANGELOG, ROADMAP et prompt restent fermés.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu : **20 tests unitaires + 7 canaries d'intégration**, aucun warning Clippy, et graphe de dépendances inchangé.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune. `pre.007` reste interdit tant que ce gate n'est pas warning-free.
|
||||||
272
deltas/0.3.6/pre.006.md
Normal file
272
deltas/0.3.6/pre.006.md
Normal file
@@ -0,0 +1,272 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.006.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.006` — conversion RAW transaction v1 et provenance observée
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.005-fix.001 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.5.fix.1
|
||||||
|
```
|
||||||
|
|
||||||
|
La base opérateur inclut également la synchronisation manuelle des versions d'en-tête omise dans l'archive initiale du fix :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml header 400 -> 401
|
||||||
|
crates/ksp-job-backfill-lib/src/discovery.rs header 1 -> 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour cette base confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 148 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS
|
||||||
|
cargo test -p ksp-job-backfill-lib PASS
|
||||||
|
unitaires 11 PASS
|
||||||
|
dependency_boundary 2 PASS
|
||||||
|
public_api 2 PASS
|
||||||
|
release_completeness 2 PASS
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser exclusivement la tranche `pre.006` du plan 027 : convertir un candidat Backfill déjà découvert en transaction RAW canonique v1 et en observation d'acquisition sûre à partir du chemin Transport observé `getTransaction`, sans encore persister dans Store ni ouvrir le runtime de Job.
|
||||||
|
|
||||||
|
L'identité logique reste strictement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionReference = (RawNetworkId, RawTransactionSignature)
|
||||||
|
```
|
||||||
|
|
||||||
|
Provider, endpoint, protocole, méthode Transport et `JobId` ne deviennent jamais une identité de transaction. Ils servent uniquement à caractériser l'acquisition et, lorsqu'une transaction existe, l'identité déterministe de son observation.
|
||||||
|
|
||||||
|
## Signature canonique
|
||||||
|
|
||||||
|
`BackfillSignature::to_raw_transaction_signature` ajoute le passage de la forme Base58 bornée de `pre.005` vers `RawTransactionSignature` exactement 64 octets.
|
||||||
|
|
||||||
|
Le décodeur Base58 est privé, borné et spécialisé pour cette frontière :
|
||||||
|
|
||||||
|
- alphabet Base58 Solana exact ;
|
||||||
|
- exactement 64 octets décodés requis ;
|
||||||
|
- zéros initiaux `1` conservés ;
|
||||||
|
- dépassement arithmétique rejeté ;
|
||||||
|
- aucune dépendance SDK/protocolaire Solana ;
|
||||||
|
- aucune nouvelle dépendance Base58 externe.
|
||||||
|
|
||||||
|
La conversion ne modifie pas l'identité réseau du candidat et rejette avant Transport tout candidat dont le réseau diffère de la requête.
|
||||||
|
|
||||||
|
## Hydratation Transport observée
|
||||||
|
|
||||||
|
La voie publique :
|
||||||
|
|
||||||
|
```text
|
||||||
|
hydrate_backfill_candidate
|
||||||
|
```
|
||||||
|
|
||||||
|
appelle exclusivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
HttpTransportPool::get_transaction_observed
|
||||||
|
```
|
||||||
|
|
||||||
|
avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
encoding = base64
|
||||||
|
commitment = engagement explicite du Backfill
|
||||||
|
maxSupportedTransactionVersion = 0
|
||||||
|
```
|
||||||
|
|
||||||
|
Transport reste propriétaire du routage, retry, pacing, cooldown et endpoint victorieux. Job n'introduit aucun second client ni boucle de retry.
|
||||||
|
|
||||||
|
Deux outcomes sont distingués :
|
||||||
|
|
||||||
|
- `Available(BackfillRawAcquisition)` : transaction présente, RAW v1 et observation construits en mémoire ;
|
||||||
|
- `Missing(RawTransactionReference)` : JSON RPC `result: null`, aucune provenance ni observation fabriquée.
|
||||||
|
|
||||||
|
## RAW transaction v1
|
||||||
|
|
||||||
|
Le format KSP figé par la tranche est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
format_id = ksp.solana.raw_transaction
|
||||||
|
format_version = 1
|
||||||
|
```
|
||||||
|
|
||||||
|
Le payload canonique est du JSON UTF-8 compact déterministe. La forme top-level est construite dans cet ordre :
|
||||||
|
|
||||||
|
1. `transaction` ;
|
||||||
|
2. `meta` si le champ wire n'est pas omitted ;
|
||||||
|
3. `version` si le champ wire n'est pas omitted ;
|
||||||
|
4. `transactionIndex` si le champ wire n'est pas omitted.
|
||||||
|
|
||||||
|
La transaction est conservée sans décodage métier sous la forme :
|
||||||
|
|
||||||
|
```json
|
||||||
|
["<base64>","base64"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Les objets JSON imbriqués sont canonisés récursivement par tri lexical des clés ; l'ordre des tableaux reste intact. Les états wire `Omitted`, `Null` et `Value` de `meta`, `version` et `transactionIndex` restent distincts.
|
||||||
|
|
||||||
|
`slot` et `blockTime` appartiennent aux champs structurés de `RawTransaction` et ne sont pas dupliqués dans le payload. Un `blockTime` négatif ou non représentable dans `RawTimestamp` est une erreur de conversion terminale.
|
||||||
|
|
||||||
|
SHA-256 et `byte_len` sont calculés sur les octets canoniques exacts avant construction de `RawPayload`.
|
||||||
|
|
||||||
|
Les réponses transactionnelles non Base64 sont rejetées ; Job ne bascule pas vers une interprétation JSON/jsonParsed ou fournisseur.
|
||||||
|
|
||||||
|
## Provenance d'acquisition
|
||||||
|
|
||||||
|
Une transaction disponible produit `RawAcquisitionProvenance` avec :
|
||||||
|
|
||||||
|
- provider réellement victorieux ;
|
||||||
|
- protocole sûr `solana.http.json_rpc` ;
|
||||||
|
- méthode `getTransaction` ;
|
||||||
|
- endpoint sûr réellement victorieux ;
|
||||||
|
- engagement demandé ;
|
||||||
|
- `JobId` comme capture session id ;
|
||||||
|
- origine `Backfill` ;
|
||||||
|
- timestamp de réception fourni par l'hôte.
|
||||||
|
|
||||||
|
Aucun URL, header, secret, body HTTP ou payload fournisseur brut n'est copié dans la provenance.
|
||||||
|
|
||||||
|
La clé `RawTransactionObservationKey` utilise SHA-256 avec séparation de domaine et couvre :
|
||||||
|
|
||||||
|
- `JobId` ;
|
||||||
|
- fingerprint du scope ;
|
||||||
|
- signature RAW 64 octets ;
|
||||||
|
- provider ;
|
||||||
|
- endpoint ;
|
||||||
|
- engagement ;
|
||||||
|
- version du contrat d'observation.
|
||||||
|
|
||||||
|
Même réseau + même signature + endpoint différent représente donc toujours la même transaction logique mais une observation d'acquisition distincte.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
La crate conserve ses dépendances KSP de `pre.005` et ajoute uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
serde_json.workspace = true
|
||||||
|
```
|
||||||
|
|
||||||
|
`serde_json` est déjà centralisé dans `[workspace.dependencies]`. `sha2` reste la primitive SHA-256 déjà présente. Aucune dépendance `bs58`, SDK Solana, Store API/backend, Config, reqwest ou tonic n'est ajoutée.
|
||||||
|
|
||||||
|
## Tests matérialisés
|
||||||
|
|
||||||
|
La tranche ajoute **8 tests unitaires**, portant le total de `ksp-job-backfill-lib` à **19 unitaires** :
|
||||||
|
|
||||||
|
- décodage Base58 exact 64 octets, longueur excessive et overflow ;
|
||||||
|
- golden payload canonique, longueur, SHA-256 et provenance ;
|
||||||
|
- distinction omitted/null des champs wire ;
|
||||||
|
- rejet des transactions non Base64 ;
|
||||||
|
- block times négatifs ou non représentables ;
|
||||||
|
- clé d'observation déterministe et sensible à provider/endpoint ;
|
||||||
|
- mismatch réseau rejeté avant Transport ;
|
||||||
|
- outcome Missing limité à la référence réseau + signature.
|
||||||
|
|
||||||
|
Une canarie publique supplémentaire porte les canaries d'intégration à **7** :
|
||||||
|
|
||||||
|
- `dependency_boundary` : 2 ;
|
||||||
|
- `public_api` : 3 ;
|
||||||
|
- `release_completeness` : 2.
|
||||||
|
|
||||||
|
Les canaries verrouillent en outre l'usage du chemin observé, `base64`, `maxSupportedTransactionVersion = 0`, l'absence de persistance et les firewalls de dépendances.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/src/conversion.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/conversion.rs
|
||||||
|
deltas/0.3.6/pre.006.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/request.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions d'en-tête
|
||||||
|
|
||||||
|
Tous les fichiers Rust/TOML modifiés par cette tranche incrémentent leur version d'en-tête par rapport à la base opérateur corrigée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml 401 -> 402
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/src/request.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs 1 -> 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Les nouveaux fichiers Rust commencent à `version: 1`. Les documents Markdown modifiés incrémentent eux aussi leur en-tête documentaire : plan 027 et validation 023 passent de `9` à `10`.
|
||||||
|
|
||||||
|
## Version workspace
|
||||||
|
|
||||||
|
La tranche modifie du code Rust et le manifeste runtime de la crate :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.6
|
||||||
|
delivery = 0.3.6-pre.006
|
||||||
|
commit = v0.3.6-pre.006
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Frontières conservées
|
||||||
|
|
||||||
|
- aucune persistance Store ;
|
||||||
|
- aucune prélecture Store avant hydratation ;
|
||||||
|
- aucun checkpoint ou frontier contiguë ;
|
||||||
|
- aucune concurrence d'hydratation concrète ;
|
||||||
|
- aucun snapshot Job concret ;
|
||||||
|
- aucune Config/env ;
|
||||||
|
- aucun Worker ou exécutable ;
|
||||||
|
- aucune modification Transport ou Store ;
|
||||||
|
- aucun endpoint/provider/protocole dans l'identité transactionnelle ;
|
||||||
|
- aucun README, USAGE, CHANGELOG ou ROADMAP rouvert ;
|
||||||
|
- aucun code kbot3 copié.
|
||||||
|
|
||||||
|
## Validations dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage exécute les audits statiques du dépôt et les contrôles d'archive. Il ne possède ni `cargo`, ni `rustc`, ni `rustfmt`; le gate Rust de `pre.006` doit donc être rejoué par l'opérateur avant clôture.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu : **19 tests unitaires + 7 canaries d'intégration**, sans persistance Store et avec un graphe normal qui conserve `ksp-store-lib` sans backend imposé.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune pour `pre.006`. La persistance atomique Store et ses outcomes restent réservés à `pre.007` après gate vert de cette tranche.
|
||||||
286
deltas/0.3.6/pre.007.md
Normal file
286
deltas/0.3.6/pre.007.md
Normal file
@@ -0,0 +1,286 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.007.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.007` — persistance Store atomique et idempotence Backfill
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.006-fix.002 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.6.fix.2
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour cette base confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 151 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS / aucun warning
|
||||||
|
cargo test -p ksp-job-backfill-lib PASS
|
||||||
|
unitaires 20 PASS
|
||||||
|
dependency_boundary 2 PASS
|
||||||
|
public_api 3 PASS
|
||||||
|
release_completeness 2 PASS
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser exclusivement `pre.007` du plan 027 : composer la persistance normale d'un résultat d'hydratation Backfill avec les contrats Store déjà existants, sans prélecture, sans second chemin d'écriture et sans ouvrir la concurrence, la frontier ou le checkpoint de `pre.008`.
|
||||||
|
|
||||||
|
L'identité canonique reste :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionReference = (RawNetworkId, RawTransactionSignature)
|
||||||
|
```
|
||||||
|
|
||||||
|
Provider, endpoint et protocole ne participent toujours qu'à la provenance de l'observation.
|
||||||
|
|
||||||
|
## Contrat public Backfill
|
||||||
|
|
||||||
|
La tranche ajoute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
persist_backfill_hydration
|
||||||
|
BackfillPersistenceOutcome
|
||||||
|
BackfillEntityPersistence
|
||||||
|
BackfillObservationPersistence
|
||||||
|
ERROR_CODE_BACKFILL_PERSISTENCE_INVALID
|
||||||
|
```
|
||||||
|
|
||||||
|
`BackfillPersistenceOutcome` conserve la référence réseau + signature et projette séparément l'état de l'entité canonique et celui de l'observation. Cette projection évite de réduire un conflit, une purge ou une nouvelle observation à un compteur `skip` ambigu.
|
||||||
|
|
||||||
|
Dispositions entité :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted
|
||||||
|
AlreadyPresent
|
||||||
|
SkippedPurged
|
||||||
|
Missing
|
||||||
|
Conflict
|
||||||
|
```
|
||||||
|
|
||||||
|
Dispositions observation :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted
|
||||||
|
AlreadyPresent
|
||||||
|
NotRecorded
|
||||||
|
NotApplicable
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition Store atomique
|
||||||
|
|
||||||
|
Pour `BackfillHydrationOutcome::Available`, la bibliothèque consomme l'acquisition en mémoire puis appelle exclusivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ksp_store_lib::RawTransactionWrite::persist_raw_transaction_acquisition
|
||||||
|
```
|
||||||
|
|
||||||
|
avec :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawTransactionAcquisitionMode::Normal
|
||||||
|
```
|
||||||
|
|
||||||
|
La transaction et son observation restent donc une seule opération logique atomique possédée par Store. La tranche n'appelle jamais `record_raw_transaction_observation` séparément et n'introduit aucune prélecture `get_raw_transaction` avant écriture.
|
||||||
|
|
||||||
|
`ForceRehydrate` n'est pas exposé par le chemin Backfill normal.
|
||||||
|
|
||||||
|
## Réseau
|
||||||
|
|
||||||
|
Avant toute écriture, le réseau logique exposé par le snapshot du Store doit être identique au `RawNetworkId` de la référence hydratée. Un mismatch est rejeté par :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_BACKFILL_PERSISTENCE_INVALID
|
||||||
|
field = store.network
|
||||||
|
```
|
||||||
|
|
||||||
|
Ce contrôle n'utilise ni endpoint, ni provider, ni protocole comme identité.
|
||||||
|
|
||||||
|
Une incohérence interne entre la référence annoncée, la transaction et l'observation est également rejetée avant soumission Store.
|
||||||
|
|
||||||
|
## Outcomes Store admis
|
||||||
|
|
||||||
|
En mode normal, les combinaisons Store suivantes sont acceptées et projetées explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
RawEntityWriteOutcome::Inserted
|
||||||
|
+ RawObservationWriteOutcome::Inserted
|
||||||
|
-> Inserted / Inserted
|
||||||
|
|
||||||
|
RawEntityWriteOutcome::AlreadyPresent
|
||||||
|
+ RawObservationWriteOutcome::Inserted
|
||||||
|
-> AlreadyPresent / Inserted
|
||||||
|
|
||||||
|
RawEntityWriteOutcome::AlreadyPresent
|
||||||
|
+ RawObservationWriteOutcome::AlreadyPresent
|
||||||
|
-> AlreadyPresent / AlreadyPresent
|
||||||
|
|
||||||
|
RawEntityWriteOutcome::SkippedPurged
|
||||||
|
+ RawObservationWriteOutcome::NotRecorded
|
||||||
|
-> SkippedPurged / NotRecorded
|
||||||
|
```
|
||||||
|
|
||||||
|
`RawEntityWriteOutcome::Rehydrated` est impossible dans le chemin normal et est rejeté, comme toute autre combinaison incohérente.
|
||||||
|
|
||||||
|
## Missing
|
||||||
|
|
||||||
|
`BackfillHydrationOutcome::Missing(reference)` devient :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Missing / NotApplicable
|
||||||
|
```
|
||||||
|
|
||||||
|
sans aucune écriture Store. La tranche ne persiste toujours ni observation `missing`, ni observation d'échec fabriquée ; ces contrats n'existent pas dans le Store RAW transaction actuel.
|
||||||
|
|
||||||
|
## Conflit et erreurs
|
||||||
|
|
||||||
|
Une erreur Store dont le code stable est :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_RAW_CONFLICT
|
||||||
|
```
|
||||||
|
|
||||||
|
est projetée en :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Conflict / NotRecorded
|
||||||
|
```
|
||||||
|
|
||||||
|
Elle reste donc visible et distincte de l'idempotence. Les tranches de frontier/snapshot pourront ensuite la traiter comme trou/échec sans parsing de message d'erreur.
|
||||||
|
|
||||||
|
Toutes les autres erreurs Store sont propagées telles quelles.
|
||||||
|
|
||||||
|
## Idempotence
|
||||||
|
|
||||||
|
La même acquisition déterministe peut produire successivement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted / Inserted
|
||||||
|
AlreadyPresent / AlreadyPresent
|
||||||
|
```
|
||||||
|
|
||||||
|
sans doublon logique. Une entité déjà présente peut aussi recevoir une nouvelle observation légitime :
|
||||||
|
|
||||||
|
```text
|
||||||
|
AlreadyPresent / Inserted
|
||||||
|
```
|
||||||
|
|
||||||
|
La distinction repose entièrement sur les outcomes Store officiels ; Job ne reconstruit aucune logique d'idempotence physique.
|
||||||
|
|
||||||
|
## Tests matérialisés
|
||||||
|
|
||||||
|
La tranche ajoute **9 tests unitaires**, portant le total de `ksp-job-backfill-lib` à **29 unitaires** :
|
||||||
|
|
||||||
|
- `Missing` ne soumet aucune écriture ;
|
||||||
|
- mismatch réseau rejeté avant Store ;
|
||||||
|
- insert atomique `Inserted/Inserted` ;
|
||||||
|
- entité existante + nouvelle observation ;
|
||||||
|
- relance idempotente `AlreadyPresent/AlreadyPresent` ;
|
||||||
|
- purge normale `SkippedPurged/NotRecorded` ;
|
||||||
|
- conflit Store explicite et non confondu avec idempotence ;
|
||||||
|
- erreur Store non conflit propagée ;
|
||||||
|
- combinaisons impossibles et mismatch transaction/observation rejetés.
|
||||||
|
|
||||||
|
Une canarie publique supplémentaire porte les canaries d'intégration à **8** :
|
||||||
|
|
||||||
|
```text
|
||||||
|
dependency_boundary 2
|
||||||
|
public_api 4
|
||||||
|
release_completeness 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Les canaries statiques verrouillent l'usage de l'écriture atomique, du mode `Normal`, l'absence d'écriture d'observation séparée, l'absence de `ForceRehydrate`, l'absence de backend Store direct et la fermeture du checkpoint/runtime de `pre.008`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/src/persistence.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/persistence.rs
|
||||||
|
deltas/0.3.6/pre.007.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions d'en-tête
|
||||||
|
|
||||||
|
Les fichiers Rust/TOML modifiés incrémentent tous leur version d'en-tête exactement d'une unité :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml 404 -> 405
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs 2 -> 3
|
||||||
|
```
|
||||||
|
|
||||||
|
Les deux nouveaux fichiers Rust commencent à `version: 1`. Le plan 027 et la validation 023 passent de `12` à `13`. Le présent delta commence à `version: 1`.
|
||||||
|
|
||||||
|
## Version workspace
|
||||||
|
|
||||||
|
La tranche modifie du Rust :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.7
|
||||||
|
delivery = 0.3.6-pre.007
|
||||||
|
commit = v0.3.6-pre.007
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Frontières conservées
|
||||||
|
|
||||||
|
- aucune prélecture Store avant hydratation/persistance ;
|
||||||
|
- aucun backend Store direct ;
|
||||||
|
- aucun `ForceRehydrate` ;
|
||||||
|
- aucune persistence de missing/failed artificielle ;
|
||||||
|
- aucune concurrence d'hydratation concrète ;
|
||||||
|
- aucune frontier contiguë ;
|
||||||
|
- aucun checkpoint/reprise runtime ;
|
||||||
|
- aucun snapshot Backfill concret ;
|
||||||
|
- aucune nouvelle dépendance ou feature ;
|
||||||
|
- aucune modification Transport, Store, Job API, Config, README, USAGE, CHANGELOG ou ROADMAP ;
|
||||||
|
- aucun code kbot3 copié.
|
||||||
|
|
||||||
|
## Validations dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
L'environnement d'assemblage exécute les audits statiques et contrôles d'archive mais ne possède ni `cargo`, ni `rustc`, ni `rustfmt`. Les tests Rust matérialisés ne sont donc pas annoncés comme exécutés localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu : **29 tests unitaires + 8 canaries d'intégration**, aucun warning Clippy, et dépendance Store toujours limitée à `ksp-store-lib` avec `default-features = false`.
|
||||||
|
|
||||||
|
## Questions ouvertes
|
||||||
|
|
||||||
|
Aucune pour `pre.007`. `pre.008` reste fermé tant que ce gate n'est pas vert.
|
||||||
56
deltas/0.3.6/pre.008-fix.001.md
Normal file
56
deltas/0.3.6/pre.008-fix.001.md
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.008-fix.001.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta v0.3.6-pre.008-fix.001
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
- `0.3.6-pre.8` / livraison `pre.008`.
|
||||||
|
- Le gate opérateur de `pre.008` confirme les audits Rust/Markdown et `cargo check --workspace`.
|
||||||
|
- La compilation des targets de test échoue avant exécution des 39 unitaires et 9 canaries à cause d'un masquage lexical dans `unit_tests/checkpoint.rs`.
|
||||||
|
|
||||||
|
## Correctif
|
||||||
|
|
||||||
|
Dans `pre_008_checkpoint_is_bound_to_job_and_scope_fingerprint`, la liaison locale `request` masquait le helper `request(scope)` avant la création de `other_request`, produisant `E0618` puis l'erreur d'inférence secondaire `E0282`. Le correctif renomme uniquement cette liaison en `primary_request` et synchronise ses références.
|
||||||
|
|
||||||
|
Aucun contrat public, comportement checkpoint/frontier, nombre de tests, dépendance, feature ou logique de production n'est modifié.
|
||||||
|
|
||||||
|
## Cargo et versions de fichiers
|
||||||
|
|
||||||
|
- `workspace.package.version` : `0.3.6-pre.8` -> `0.3.6-pre.8.fix.1`.
|
||||||
|
- `Cargo.toml` : version d'en-tête `406` -> `407`.
|
||||||
|
- `crates/ksp-job-backfill-lib/unit_tests/checkpoint.rs` : version d'en-tête `1` -> `2`.
|
||||||
|
- plan 027 : version d'en-tête `14` -> `15`.
|
||||||
|
- validation 023 : version d'en-tête `14` -> `15`.
|
||||||
|
|
||||||
|
## Graphe de dépendances
|
||||||
|
|
||||||
|
`pre.008` ajoutait `futures-util` comme dépendance normale ; les deux arbres Cargo ont donc été inspectés pendant son gate malgré l'échec ultérieur des tests. Ce fix ne modifie ni dépendance ni feature : aucun `cargo clean` et aucun nouveau `cargo tree` ne sont nécessaires pour le valider. `cargo tree` dépend des manifests/résolution Cargo, pas des artefacts du répertoire `target`.
|
||||||
|
|
||||||
|
## Payload
|
||||||
|
|
||||||
|
Ajout :
|
||||||
|
|
||||||
|
- `deltas/0.3.6/pre.008-fix.001.md`
|
||||||
|
|
||||||
|
Modifications :
|
||||||
|
|
||||||
|
- `Cargo.toml`
|
||||||
|
- `crates/ksp-job-backfill-lib/unit_tests/checkpoint.rs`
|
||||||
|
- `docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md`
|
||||||
|
- `docs/validation/023-V0_3_6_JOB_API_BACKFILL.md`
|
||||||
|
|
||||||
|
Aucune suppression.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
```
|
||||||
|
|
||||||
|
Attendu : 39 tests unitaires et 9 canaries d'intégration, sans warning Clippy. Les arbres Cargo ne sont pas à rejouer pour ce fix sans changement de graphe.
|
||||||
306
deltas/0.3.6/pre.008.md
Normal file
306
deltas/0.3.6/pre.008.md
Normal file
@@ -0,0 +1,306 @@
|
|||||||
|
<!-- file: deltas/0.3.6/pre.008.md -->
|
||||||
|
<!-- version: 1 -->
|
||||||
|
|
||||||
|
# Delta `0.3.6-pre.008` — concurrence bornée, frontier contiguë et checkpoint Backfill
|
||||||
|
|
||||||
|
## Base requise
|
||||||
|
|
||||||
|
```text
|
||||||
|
0.3.6-pre.007 appliquée
|
||||||
|
workspace.package.version = 0.3.6-pre.7
|
||||||
|
```
|
||||||
|
|
||||||
|
Le gate opérateur fourni pour cette base confirme :
|
||||||
|
|
||||||
|
```text
|
||||||
|
cargo fmt --all PASS
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py PASS / clean
|
||||||
|
python3 scripts/audit_markdown_tables.py ... PASS / clean (264 tables / 152 fichiers)
|
||||||
|
cargo check --workspace PASS
|
||||||
|
cargo clippy --workspace --all-targets PASS / aucun warning
|
||||||
|
cargo test -p ksp-job-backfill-lib PASS
|
||||||
|
unitaires 29 PASS
|
||||||
|
dependency_boundary 2 PASS
|
||||||
|
public_api 4 PASS
|
||||||
|
release_completeness 2 PASS
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal exécuté
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features exécuté
|
||||||
|
```
|
||||||
|
|
||||||
|
## Objectif
|
||||||
|
|
||||||
|
Matérialiser exclusivement `pre.008` du plan 027 : exécuter les candidats découverts avec une concurrence métier bornée, réconcilier les fins hors ordre par une frontier contiguë et produire un checkpoint sûr pour reprise contrôlée.
|
||||||
|
|
||||||
|
La tranche n'ouvre pas encore l'annulation coopérative, le snapshot latest-value concret, les états terminaux complets ou les courses d'annulation de `pre.009`.
|
||||||
|
|
||||||
|
## Checkpoint caller-owned
|
||||||
|
|
||||||
|
La tranche ajoute le type public opaque :
|
||||||
|
|
||||||
|
```text
|
||||||
|
BackfillCheckpoint
|
||||||
|
```
|
||||||
|
|
||||||
|
Il contient uniquement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
JobId
|
||||||
|
BackfillScopeFingerprint
|
||||||
|
completed_prefix
|
||||||
|
resume_before privé si scope BeforeAddress
|
||||||
|
```
|
||||||
|
|
||||||
|
Il ne contient ni payload RAW, ni URL, ni provider, ni endpoint, ni credential, ni liste de signatures.
|
||||||
|
|
||||||
|
`BackfillRequest::with_checkpoint` valide obligatoirement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
checkpoint.job_id == request.job_id
|
||||||
|
checkpoint.scope_fingerprint == request.scope_fingerprint
|
||||||
|
```
|
||||||
|
|
||||||
|
Un checkpoint incompatible est rejeté par :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_BACKFILL_CHECKPOINT_INVALID
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucune table Store, migration ou persistance durable de Job n'est créée. Le checkpoint est caller-owned et qualifie uniquement une reprise contrôlée ; la v0.3.6 ne revendique pas une reprise crash-safe multi-processus.
|
||||||
|
|
||||||
|
## Sémantique de reprise par scope
|
||||||
|
|
||||||
|
### LatestAddress
|
||||||
|
|
||||||
|
La reprise repart volontairement sans curseur RPC depuis la vue latest courante. L'idempotence Store absorbe les candidats déjà durables. Le préfixe historique du checkpoint n'est jamais utilisé comme offset de la nouvelle vue latest.
|
||||||
|
|
||||||
|
### BeforeAddress
|
||||||
|
|
||||||
|
Le checkpoint conserve comme curseur exclusif le dernier candidat du préfixe contigu durable. Si aucun candidat nouveau n'est durable, le curseur précédent — ou l'ancre initiale — est conservé. Le compteur `completed_prefix` est cumulatif entre reprises contrôlées.
|
||||||
|
|
||||||
|
### AfterAddress
|
||||||
|
|
||||||
|
Le scope borné est rejoué et seul le préfixe contigu validé est ignoré. Si la découverte s'arrête sur :
|
||||||
|
|
||||||
|
```text
|
||||||
|
AfterAnchorNotReached
|
||||||
|
```
|
||||||
|
|
||||||
|
aucun progrès nouveau de checkpoint n'est publié, même si certaines hydrations observées ont fini durablement ; le checkpoint précédent est conservé lorsqu'il existe.
|
||||||
|
|
||||||
|
### ExplicitSignatures
|
||||||
|
|
||||||
|
La liste dédupliquée et fingerprintée est rejouée et seul son préfixe contigu déjà validé est ignoré.
|
||||||
|
|
||||||
|
## Frontier contiguë
|
||||||
|
|
||||||
|
Chaque candidat conserve son index stable de découverte. Une bitmap privée bornée à `max_candidates <= 10_000` suit les résultats durables.
|
||||||
|
|
||||||
|
La frontier avance uniquement pour :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Inserted
|
||||||
|
AlreadyPresent
|
||||||
|
SkippedPurged
|
||||||
|
```
|
||||||
|
|
||||||
|
Les résultats suivants créent un trou et ne sont jamais marqués durables :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Missing
|
||||||
|
Conflict
|
||||||
|
erreur Transport
|
||||||
|
conversion invalide
|
||||||
|
erreur Store
|
||||||
|
```
|
||||||
|
|
||||||
|
Une completion hors ordre ne saute jamais un index incomplet. Les tests reproduisent explicitement :
|
||||||
|
|
||||||
|
```text
|
||||||
|
index 1 durable -> frontier 0
|
||||||
|
index 0 durable -> frontier 2
|
||||||
|
index 3 durable -> frontier 2
|
||||||
|
index 2 durable -> frontier 4
|
||||||
|
```
|
||||||
|
|
||||||
|
## Exécution concurrente bornée
|
||||||
|
|
||||||
|
La tranche ajoute :
|
||||||
|
|
||||||
|
```text
|
||||||
|
execute_backfill_discovery
|
||||||
|
BackfillExecutionBatch
|
||||||
|
```
|
||||||
|
|
||||||
|
L'implémentation utilise un `FuturesUnordered` privé et ne garde jamais plus de :
|
||||||
|
|
||||||
|
```text
|
||||||
|
request.hydration_concurrency()
|
||||||
|
```
|
||||||
|
|
||||||
|
futures candidates en vol.
|
||||||
|
|
||||||
|
Aucun `tokio::spawn`, timer Job, retry Job, pacing Job ou choix d'endpoint n'est introduit. Transport reste propriétaire de sa politique réseau et Store de sa persistance.
|
||||||
|
|
||||||
|
Le batch expose uniquement des faits bornés utiles à la preuve de frontier : candidats, admis, terminés, durables, trous, maximum en vol, préfixe contigu local, checkpoint et premier code d'échec fatal.
|
||||||
|
|
||||||
|
## Missing, conflit et erreurs
|
||||||
|
|
||||||
|
`Missing` est un trou non fatal : les admissions suivantes continuent, mais la frontier reste bloquée tant que ce trou existe.
|
||||||
|
|
||||||
|
Un `Conflict` ou toute erreur Transport/conversion/Store finale :
|
||||||
|
|
||||||
|
1. devient le premier `failure_code` stable du batch ;
|
||||||
|
2. arrête immédiatement les nouvelles admissions ;
|
||||||
|
3. laisse finir toutes les futures déjà admises ;
|
||||||
|
4. produit ensuite le checkpoint correspondant au dernier préfixe réellement durable.
|
||||||
|
|
||||||
|
Cette phase ne publie pas encore l'état Job terminal `Failed`/`Partial`; cette projection appartient au snapshot/runtime de `pre.009`.
|
||||||
|
|
||||||
|
## Horodatage d'acquisition
|
||||||
|
|
||||||
|
Le chemin public d'exécution produit le `RawTimestamp` de réception à partir de `SystemTime` immédiatement avant l'hydratation. Une horloge avant epoch ou un timestamp non représentable devient une erreur structurée :
|
||||||
|
|
||||||
|
```text
|
||||||
|
ERROR_CODE_BACKFILL_EXECUTION_INVALID
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun clock/provider type n'entre dans l'API publique.
|
||||||
|
|
||||||
|
## Dépendances
|
||||||
|
|
||||||
|
La crate ajoute comme dépendance normale :
|
||||||
|
|
||||||
|
```text
|
||||||
|
futures-util = { workspace = true, features = ["std"] }
|
||||||
|
```
|
||||||
|
|
||||||
|
Cette dépendance existe déjà dans `[workspace.dependencies]`; aucune nouvelle version externe n'est introduite.
|
||||||
|
|
||||||
|
Tokio reste exclusivement en `dev-dependencies` pour les tests async et aucun type Tokio/Futures n'est exposé par la crate root.
|
||||||
|
|
||||||
|
## Tests matérialisés
|
||||||
|
|
||||||
|
La tranche ajoute **10 tests unitaires**, portant le total à **39 unitaires** :
|
||||||
|
|
||||||
|
- frontier hors ordre strictement contiguë ;
|
||||||
|
- checkpoint rejeté sur mauvais `JobId` ou fingerprint ;
|
||||||
|
- checkpoint Before cumulatif avec curseur du dernier candidat contigu ;
|
||||||
|
- After sans ancre atteinte n'avance jamais le checkpoint ;
|
||||||
|
- Before reprend au curseur checkpoint plutôt qu'à l'ancre initiale ;
|
||||||
|
- Latest reprend sans curseur depuis la vue courante ;
|
||||||
|
- concurrence observée strictement bornée malgré fins hors ordre ;
|
||||||
|
- Missing non fatal mais bloquant pour la frontier ;
|
||||||
|
- conflit arrêtant les admissions et drainant le travail déjà en vol ;
|
||||||
|
- reprise Explicit sautant uniquement le préfixe checkpointé.
|
||||||
|
|
||||||
|
Une canarie publique supplémentaire porte les canaries d'intégration à **9** :
|
||||||
|
|
||||||
|
```text
|
||||||
|
dependency_boundary 2
|
||||||
|
public_api 5
|
||||||
|
release_completeness 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Les canaries verrouillent également l'inventaire exact des modules, `futures-util` comme détail privé normal, l'absence de Tokio normal, l'absence de retry/pacing/endpoint selection Job et la fermeture des surfaces `BackfillSnapshot`/`BackfillJobHandle` de `pre.009`.
|
||||||
|
|
||||||
|
## Fichiers ajoutés
|
||||||
|
|
||||||
|
```text
|
||||||
|
crates/ksp-job-backfill-lib/src/checkpoint.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/execution.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/checkpoint.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/execution.rs
|
||||||
|
deltas/0.3.6/pre.008.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Fichiers modifiés
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml
|
||||||
|
crates/ksp-job-backfill-lib/src/discovery.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/persistence.rs
|
||||||
|
crates/ksp-job-backfill-lib/src/request.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/discovery.rs
|
||||||
|
docs/plans/027-V0_3_6_JOB_API_BACKFILL_PLAN.md
|
||||||
|
docs/validation/023-V0_3_6_JOB_API_BACKFILL.md
|
||||||
|
```
|
||||||
|
|
||||||
|
## Versions d'en-tête
|
||||||
|
|
||||||
|
Les fichiers Rust/TOML modifiés incrémentent tous leur version d'en-tête exactement d'une unité :
|
||||||
|
|
||||||
|
```text
|
||||||
|
Cargo.toml 405 -> 406
|
||||||
|
crates/ksp-job-backfill-lib/Cargo.toml 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/src/discovery.rs 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/src/error.rs 3 -> 4
|
||||||
|
crates/ksp-job-backfill-lib/src/lib.rs 3 -> 4
|
||||||
|
crates/ksp-job-backfill-lib/src/persistence.rs 1 -> 2
|
||||||
|
crates/ksp-job-backfill-lib/src/request.rs 2 -> 3
|
||||||
|
crates/ksp-job-backfill-lib/tests/dependency_boundary.rs 3 -> 4
|
||||||
|
crates/ksp-job-backfill-lib/tests/public_api.rs 3 -> 4
|
||||||
|
crates/ksp-job-backfill-lib/tests/release_completeness.rs 3 -> 4
|
||||||
|
crates/ksp-job-backfill-lib/unit_tests/discovery.rs 1 -> 2
|
||||||
|
```
|
||||||
|
|
||||||
|
Les quatre nouveaux fichiers Rust commencent à `version: 1`. Le plan 027 et la validation 023 passent de `13` à `14`. Le présent delta commence à `version: 1`.
|
||||||
|
|
||||||
|
## Version workspace
|
||||||
|
|
||||||
|
La tranche modifie du Rust :
|
||||||
|
|
||||||
|
```text
|
||||||
|
workspace.package.version = 0.3.6-pre.8
|
||||||
|
delivery = 0.3.6-pre.008
|
||||||
|
commit = v0.3.6-pre.008
|
||||||
|
```
|
||||||
|
|
||||||
|
Aucun tag prerelease.
|
||||||
|
|
||||||
|
## Fichiers supprimés
|
||||||
|
|
||||||
|
Aucun.
|
||||||
|
|
||||||
|
## Frontières conservées
|
||||||
|
|
||||||
|
- aucune persistence durable de checkpoint dans Store ;
|
||||||
|
- aucune table ou migration Job ;
|
||||||
|
- aucun retry/pacing/timer Job ;
|
||||||
|
- aucune sélection provider/endpoint Job ;
|
||||||
|
- aucun `ForceRehydrate` ;
|
||||||
|
- aucun `tokio::spawn` ;
|
||||||
|
- aucun type Tokio/Futures public ;
|
||||||
|
- aucune annulation coopérative de `pre.009` ;
|
||||||
|
- aucun `BackfillSnapshot` latest-value concret ;
|
||||||
|
- aucune application ou Worker ;
|
||||||
|
- aucun changement README/USAGE/CHANGELOG/ROADMAP/prompt.
|
||||||
|
|
||||||
|
## Validations exécutées dans l'environnement d'assemblage
|
||||||
|
|
||||||
|
```text
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
```
|
||||||
|
|
||||||
|
L'environnement d'assemblage ne fournit ni `cargo`, ni `rustc`, ni `rustfmt`; aucun résultat Rust de `pre.008` n'est donc revendiqué localement.
|
||||||
|
|
||||||
|
## Gate opérateur demandé
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cargo fmt --all
|
||||||
|
python3 scripts/audit_rust_workspace_rules.py
|
||||||
|
python3 scripts/audit_markdown_tables.py README.md RULES.md ROADMAP.md CHANGELOG.md docs prompts crates deltas/0.3.6
|
||||||
|
cargo check --workspace
|
||||||
|
cargo clippy --workspace --all-targets
|
||||||
|
cargo test -p ksp-job-backfill-lib
|
||||||
|
cargo tree -p ksp-job-backfill-lib --edges normal
|
||||||
|
cargo tree -p ksp-job-backfill-lib -e features
|
||||||
|
```
|
||||||
|
|
||||||
|
Résultat attendu après gate : **39 unitaires + 9 canaries**, sans warning. Un échec reste dans la lane `pre.008-fix.001`; un gate vert ouvre `pre.009`.
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user