v0.1.0-pre.029

CHANGELOG.md

@@ -3,6 +3,13 @@
 
 # CHANGELOG
 
+## 0.1.0-pre.029
+
+- Migration initiale de `kb-pipeline` : primitives de planification et extraction canonique Solana vers les tables core.
+- Adaptation aux façades `kb-lib`, `kb-store` et `kb-onchain-transport`.
+- Normalisation du target de tracing `kb-pipeline` et ajout d’un test de configuration.
+
+
 ## 0.1.0-pre.024
 
 - Première tranche fonctionnelle de `kb-onchain-transport`.

README.md

@@ -15,7 +15,7 @@ Khadhroony Bot3 is the consolidated successor workspace to `khadhroony-bot2`.
 - `kb-pipeline`: ingestion, decoding, materialization and execution orchestration.
 - `kb-onchain-transport`: HTTP and WebSocket Solana transports.
 - `kb-store`: store-neutral contracts and PostgreSQL implementation modules.
-- `kb-wallet`: stockage local protégé de keypairs temporaires, résumés non secrets et signature de messages.
+- `kb-wallet`: wallet and signer support.
 - `kb-app-demo`: the only binary retained during the initial migration.
 
 ## Migration principle
@@ -89,3 +89,8 @@ Les modules de matérialisation sont privés. Tous les contrats, types concrets
 `0.1.0-pre.015` rend `ExSolanaCoreExecutor` fonctionnel. Il construit des plans déterministes pour 109 opérations natives réparties sur 18 surfaces, conserve les refus explicites des programmes historiques non exécutables et expose ses intents sous les préfixes bot3 `ExSolanaCore*` et `EX_SOLANA_CORE_*`. Les 103 autres exécuteurs restent réservés jusqu’à leur port fonctionnel.
 
 `0.1.0-pre.016` intègre le moteur transversal `ExSafetyChecker` et rend `ExSplMemoExecutor` fonctionnel. Memo v4 produit une instruction officielle bornée et soumise à simulation ; les générations historiques v1/v3 restent decode-only. Les 102 autres exécuteurs restent réservés jusqu’à leur port fonctionnel.
+
+
+### Migration pipeline
+
+La tranche `0.1.0-pre.029` migre la planification et l’extraction core de `kb-pipeline`.

ROADMAP.md

@@ -276,3 +276,11 @@
 - [x] Routes canoniques `kb-wallet`.
 - [x] Configuration d’exemple sans clé fournisseur en clair.
 - [ ] Migration fonctionnelle de `kb-pipeline` à poursuivre à partir de `pre.029`.
+
+
+### 0.1.0-pre.029 — Pipeline, fondations
+
+- [x] Planification et scopes de replay.
+- [x] Extraction canonique vers les tables core.
+- [ ] Backfill et replay de décodage.
+- [ ] Orchestrations Solana stateful et exécution.

config/example.config.json

@@ -46,7 +46,7 @@
               "kb-store",
               "kb-onchain-transport",
               "kb-wallet",
-              "kb_pipeline",
+              "kb-pipeline",
               "kb_app_demo",
               "kb_executor_metadata_metaplex_token_metadata",
               "kb_executor_metadata_spl_name_service",
@@ -927,42 +927,42 @@
             "ansi": false
           },
           {
-            "name": "file_kb_pipeline_debug",
+            "name": "file_kb-pipeline_debug",
             "enabled": true,
             "sink": "file",
             "level": "debug",
-            "path": "logs/devnet/kb_pipeline/debug.log",
+            "path": "logs/devnet/kb-pipeline/debug.log",
             "rotation": "daily",
             "format": "human",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
-            "name": "file_kb_pipeline_info",
+            "name": "file_kb-pipeline_info",
             "enabled": true,
             "sink": "file",
             "level": "info",
-            "path": "logs/devnet/kb_pipeline/info.log",
+            "path": "logs/devnet/kb-pipeline/info.log",
             "rotation": "daily",
             "format": "human",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
-            "name": "file_kb_pipeline_error",
+            "name": "file_kb-pipeline_error",
             "enabled": true,
             "sink": "file",
             "level": "error",
-            "path": "logs/devnet/kb_pipeline/error.jsonl",
+            "path": "logs/devnet/kb-pipeline/error.jsonl",
             "rotation": "daily",
             "format": "json",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
@@ -1690,7 +1690,7 @@
               "kb-lib.materializer.staking",
               "kb-lib.materializer.token",
               "kb-lib.materializer.transaction",
-              "kb_pipeline",
+              "kb-pipeline",
               "kb-onchain-transport",
               "kb-store",
               "kb-wallet"
@@ -2568,42 +2568,42 @@
             "ansi": false
           },
           {
-            "name": "file_kb_pipeline_debug",
+            "name": "file_kb-pipeline_debug",
             "enabled": true,
             "sink": "file",
             "level": "debug",
-            "path": "logs/mainnet_research/kb_pipeline/debug.log",
+            "path": "logs/mainnet_research/kb-pipeline/debug.log",
             "rotation": "daily",
             "format": "human",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
-            "name": "file_kb_pipeline_info",
+            "name": "file_kb-pipeline_info",
             "enabled": true,
             "sink": "file",
             "level": "info",
-            "path": "logs/mainnet_research/kb_pipeline/info.log",
+            "path": "logs/mainnet_research/kb-pipeline/info.log",
             "rotation": "daily",
             "format": "human",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
-            "name": "file_kb_pipeline_error",
+            "name": "file_kb-pipeline_error",
             "enabled": true,
             "sink": "file",
             "level": "error",
-            "path": "logs/mainnet_research/kb_pipeline/error.jsonl",
+            "path": "logs/mainnet_research/kb-pipeline/error.jsonl",
             "rotation": "daily",
             "format": "json",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
@@ -3531,7 +3531,7 @@
               "kb-lib.materializer.staking",
               "kb-lib.materializer.token",
               "kb-lib.materializer.transaction",
-              "kb_pipeline",
+              "kb-pipeline",
               "kb-onchain-transport",
               "kb-store",
               "kb-wallet"
@@ -4409,42 +4409,42 @@
             "ansi": false
           },
           {
-            "name": "file_kb_pipeline_debug",
+            "name": "file_kb-pipeline_debug",
             "enabled": true,
             "sink": "file",
             "level": "debug",
-            "path": "logs/mainnet/kb_pipeline/debug.log",
+            "path": "logs/mainnet/kb-pipeline/debug.log",
             "rotation": "daily",
             "format": "human",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
-            "name": "file_kb_pipeline_info",
+            "name": "file_kb-pipeline_info",
             "enabled": true,
             "sink": "file",
             "level": "info",
-            "path": "logs/mainnet/kb_pipeline/info.log",
+            "path": "logs/mainnet/kb-pipeline/info.log",
             "rotation": "daily",
             "format": "human",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {
-            "name": "file_kb_pipeline_error",
+            "name": "file_kb-pipeline_error",
             "enabled": true,
             "sink": "file",
             "level": "error",
-            "path": "logs/mainnet/kb_pipeline/error.jsonl",
+            "path": "logs/mainnet/kb-pipeline/error.jsonl",
             "rotation": "daily",
             "format": "json",
             "ansi": false,
             "targets": [
-              "kb_pipeline"
+              "kb-pipeline"
             ]
           },
           {

kb-app-demo/src/main.rs

@@ -1,5 +1,5 @@
 // file: kb-app-demo/src/main.rs
-// version: 1
+// version: 2
 
 #![forbid(unsafe_code)]
 #![deny(unreachable_pub)]
@@ -10,5 +10,9 @@
 fn main() {
     println!("khadhroony-bot3 workspace scaffold");
     println!("logging target: {}", kb_logging::tracing_target());
-    println!("pipeline: {}", kb_pipeline::migration_status());
+    println!(
+        "pipeline: {} v{}",
+        kb_pipeline::CORE_EXTRACTION_PROCESSOR_NAME,
+        kb_pipeline::CORE_EXTRACTION_PROCESSOR_VERSION
+    );
 }

kb-config/src/settings.rs

@@ -1361,6 +1361,16 @@ mod tests {
         }
     }
 
+    #[test]
+    fn example_config_uses_canonical_pipeline_tracing_routes() {
+        let source = include_str!("../../config/example.config.json");
+        assert!(source.contains("\"kb-pipeline\""));
+        assert!(source.contains("logs/devnet/kb-pipeline/debug.log"));
+        assert!(source.contains("logs/mainnet_research/kb-pipeline/info.log"));
+        assert!(source.contains("logs/mainnet/kb-pipeline/error.jsonl"));
+        assert!(!source.contains("kb_pipeline"));
+    }
+
     #[test]
     fn example_config_parses_and_resolves_active_profile() {
         let config_result = super::parse_config_json(EXAMPLE_CONFIG);

kb-pipeline/Cargo.toml

@@ -1,5 +1,5 @@
 # file: kb-pipeline/Cargo.toml
-# version: 2
+# version: 4
 
 [package]
 name = "kb-pipeline"
@@ -9,11 +9,19 @@ license.workspace = true
 publish.workspace = true
 
 [dependencies]
+async-trait.workspace = true
+chrono.workspace = true
+futures-util.workspace = true
 kb-core = { path = "../kb-core" }
 kb-lib = { path = "../kb-lib" }
-kb-store = { path = "../kb-store" }
 kb-onchain-transport = { path = "../kb-onchain-transport" }
+kb-store = { path = "../kb-store" }
 kb-wallet = { path = "../kb-wallet" }
+serde_json.workspace = true
+sha2.workspace = true
+tokio.workspace = true
+tracing.workspace = true
+uuid.workspace = true
 
 [lints]
 workspace = true

kb-pipeline/src/constants.rs

@@ -0,0 +1,7 @@
+// file: kb-pipeline/src/constants.rs
+// version: 2
+
+//! Pipeline constants.
+
+/// Canonical tracing target for pipeline orchestration.
+pub(crate) const TRACING_TARGET: &str = "kb-pipeline";

kb-pipeline/src/lib.rs

@@ -1,5 +1,5 @@
 // file: kb-pipeline/src/lib.rs
-// version: 1
+// version: 3
 
 #![forbid(unsafe_code)]
 #![deny(unreachable_pub)]
@@ -7,7 +7,35 @@
 
 //! Pipeline orchestration boundary.
 
-/// Returns the pipeline migration status.
+mod constants;
-pub fn migration_status() -> &'static str {
+mod core_extraction;
-    return "scaffolded";
+mod plan;
-}
+
+/// Canonical tracing target for pipeline orchestration.
+pub(crate) use self::constants::TRACING_TARGET;
+/// Stable core extraction processor name.
+pub use self::core_extraction::CORE_EXTRACTION_PROCESSOR_NAME;
+/// Stable core extraction processor version.
+pub use self::core_extraction::CORE_EXTRACTION_PROCESSOR_VERSION;
+/// Stable core extraction stage code.
+pub use self::core_extraction::CORE_EXTRACTION_STAGE;
+/// Observer notified during core extraction campaigns.
+pub use self::core_extraction::CoreExtractionObserver;
+/// Progress event emitted by core extraction campaigns.
+pub use self::core_extraction::CoreExtractionProgressEvent;
+/// Severity of one core extraction progress event.
+pub use self::core_extraction::CoreExtractionProgressLevel;
+/// Bounded core extraction request.
+pub use self::core_extraction::CoreExtractionRequest;
+/// Source selection for a core extraction campaign.
+pub use self::core_extraction::CoreExtractionSource;
+/// Summary returned by a core extraction campaign.
+pub use self::core_extraction::CoreExtractionSummary;
+/// Executes one bounded core extraction campaign.
+pub use self::core_extraction::execute_core_extraction;
+/// Extracts one canonical raw transaction into normalized core rows.
+pub use self::core_extraction::extract_raw_transaction_to_core;
+/// Pipeline stage identifier.
+pub use self::plan::PipelineStage;
+/// Replay selection scope.
+pub use self::plan::ReplayScope;

kb-pipeline/src/plan.rs

@@ -0,0 +1,44 @@
+// file: kb-pipeline/src/plan.rs
+// version: 2
+
+//! Pipeline planning primitives.
+
+/// Pipeline stage identifier.
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub enum PipelineStage {
+    /// Ingest raw transactions.
+    Ingest,
+    /// Extract generic Solana structures.
+    Extract,
+    /// Build program observations.
+    Observe,
+    /// Decode protocol events.
+    Decode,
+    /// Materialize business events.
+    Materialize,
+    /// Aggregate materialized events.
+    Aggregate,
+    /// Validate outputs.
+    Validate,
+}
+
+/// Replay selection scope.
+#[derive(Clone, Debug, Eq, PartialEq)]
+pub struct ReplayScope {
+    /// Optional module name.
+    pub module_name: std::option::Option<std::string::String>,
+    /// Optional module version.
+    pub module_version: std::option::Option<std::string::String>,
+    /// Optional program id.
+    pub program_id: std::option::Option<std::string::String>,
+    /// Optional surface code.
+    pub surface_code: std::option::Option<std::string::String>,
+    /// Optional 8-byte discriminator in hexadecimal.
+    pub discriminator_8: std::option::Option<std::string::String>,
+    /// Optional inclusive start slot.
+    pub slot_start: std::option::Option<u64>,
+    /// Optional inclusive end slot.
+    pub slot_end: std::option::Option<u64>,
+    /// Whether to ignore existing ledger rows.
+    pub force: bool,
+}