Sasedev/khadhroony-bobobot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

0.7.49

Browse Source
This commit is contained in:
SinuS von SifriduS
2026-06-05 14:53:16 +02:00
parent 27e25d5bf4
commit f81e0f3bea
66 changed files with 7655 additions and 214 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
CHANGELOG.md
View File

@@ -66,28 +66,17 @@
0.7.33 - Ajout de la classification diagnostique `pairTradingReadiness` pour les paires, avec `quoteAssetClass`, `tradingRouteRequired`, résumé `pairTradingReadinessSummaries`, profil de validation `0.7.33_pair_trading_readiness` et mise à jour de la sélection UI Demo Pipeline 2 sans modifier la matérialisation trade/candle.
0.7.34 - Ajout du profil `0.7.34_non_trade_liquidity_lifecycle`, matérialisation des tables non-trade liquidité/lifecycle, warning non bloquant pour DEX attendus absents du corpus local, première tranche DLMM : `add_liquidity`, `remove_liquidity`, `initialize_position`, `initialize_bin_array`, intégration de la matérialisation non-trade dans les backfills token/pool ciblés, et distinction `PositionOpen`/`PositionClose` dans `LiquidityEventKind`.
0.7.35 - Ajout du profil `0.7.35_non_trade_fee_reward_admin`, matérialisation des événements non-trade fees/rewards/admin, raccordement aux diagnostics locaux et maintien strict de l’invariant : aucun fee/reward/admin ne peut produire de trade, metric ou candle.
0.7.36 - Consolidation de la famille Meteora : corpus mixte `meteora_damm_v1`, `meteora_damm_v2`, `meteora_dbc` et `meteora_dlmm`, correction des discriminants DAMM v2 / DBC, validation du profil `0.7.36_meteora_family_consolidation`, et reclassement explicite des swaps DAMM v2 / DBC sans payload montant/prix en `non_actionable_trade` afin d’éviter tout trade/candle artificiel.
0.7.36 - Consolidation de la famille Meteora : corpus mixte `meteora_damm_v1`, `meteora_damm_v2`, `meteora_dbc` et `meteora_dlmm`, correction des discriminants DAMM v2 / DBC, validation du profil `0.7.36_meteora_family_consolidation`, et reclassement explicite des swaps DAMM v2 / DBC sans payload montant/prix en `non_actionable_trade` afin d’éviter tout trade/candle artificiel.
0.7.37 - Première tranche metadata/catalog : ajout du profil `0.7.37_token_metadata_catalog_enrichment`, exposition des compteurs metadata dans diagnostics/validation et raccordement UI Demo Pipeline 2 sans rendre les metadata manquantes bloquantes.
0.7.38 - Priorisation des metadata manquantes : ajout du profil `0.7.38_token_metadata_gap_prioritization`, samples `tokenMetadataGapSamples`, priorités tradable/quote/catalog, raccordement UI Demo Pipeline 2 et maintien du caractère non bloquant des metadata incomplètes.
0.7.39 - Réorientation DEX-first : distinction explicite des rôles `dex_effective`, `aggregator_router`, `launch_surface` et `to_verify` dans la matrice DEX, suppression de l’alias ambigu `raydium`, ajout de `metaDAO` et `Printr` comme surfaces à vérifier sans `program_id`, profil `0.7.39_dex_first_effective_swap_surfaces`, validation locale avec invariants DEX-first maintenus et report des launch surfaces après les DEX effectifs.
0.7.40 - Ajout de Demo3 pour la constitution de corpus on-chain par `dex_code` / `program_id` via `getSignaturesForAddress` + `getTransaction`, extraction des mints, deltas SPL Token, comptes pool/state/vault/program candidats, ajout du backfill par signature dans Demo Pipeline 2, et validation pratique sur Raydium AMM v4 sans promotion automatique des comptes candidats.
0.7.41 - Raydium AMM v4 swap decoder v1 : décodage des inner instructions `675kPX...`, extraction pool/state, authority, vaults, mints, routeSource et montants exploitables, matérialisation trades/candles sur transactions OK, matrice AMM v4 passée en `supported`, et validation locale avec invariants trade/candle propres.
0.7.42 - Consolidation famille Raydium : audit conservatoire des instructions Raydium non décodées, décodage CLMM legacy `swap`, cleanup des audits remplacés, classification HTTP `getTransaction` comme requête lourde avec retry/backoff de backfill, mapping des événements non-swap prouvés `raydium_clmm` (`increase_liquidity_v2`, `decrease_liquidity_v2`, `open_position_with_token22_nft`, `close_position`) et `raydium_cpmm` (`initialize`, `withdraw`, `collect_creator_fee`), matérialisation de 25 liquidity events, 1 lifecycle event et 2 fee events sur corpus élargi, conservation des non-swaps AMM v4 legacy en audit.
0.7.43-E5C - Reprise documentaire et normalisation DEX-first : `0.7.43` est conservé comme point de reprise non clos pour le lot Meteora, la suite est redécoupée par DEX/version séparés, le besoin d’un ledger de décodage/replay est acté, les statuts `known` / `observed` / `decoded` / `materialized` / `verified_by_corpus` deviennent obligatoires, et aucun `program_id` ne doit être marqué vérifié sans preuve/corpus reproductible.
0.7.43 - Reprise documentaire et normalisation DEX-first : `0.7.43` est conservé comme point de reprise non clos pour le lot Meteora, la suite est redécoupée par DEX/version séparés, le besoin d’un ledger de décodage/replay est acté, les statuts `known` / `observed` / `decoded` / `materialized` / `verified_by_corpus` deviennent obligatoires, et aucun `program_id` ne doit être marqué vérifié sans preuve/corpus reproductible.
0.7.44 - Ledger de décodage/replay DEX : ajout de `k_sol_dex_decode_replay_ledger`, des DTO/entities/queries associés, des re-exports DB/lib, et intégration dans le replay local pour skipper uniquement l’étape de décodage DEX lorsqu’un passage certifié existe pour la même version logique de decoder. Les transactions multi-event ou multi-token restent marquées `unsafe` et sont redécodées sauf option future plus explicite ; le replay continue de reconstruire détection, matérialisation, trades, candles et classifications à partir des events persistés.
0.7.45 - Meteora DLMM normalisation finale : consolidation séparée de `meteora_dlmm` sur corpus dédié, maintien du wrapper Anchor `anchor_self_cpi_log` `e445a52e51cb9a1d`, enrichissement des swaps via `Swap` / `Swap2Evt`, cleanup des audits Anchor CPI swap déjà couverts, ajout des events upstream Git/IDL observés et vérifiés par corpus (`lb_pair_create_event`, `add_liquidity_event`, `remove_liquidity_event`, `claim_fee_event`, `position_create_event`, `position_close_event`, `close_position_if_empty`, `remove_liquidity_by_range2`, `add_liquidity_by_strategy2`, `add_liquidity_by_weight`), conservation des deux audits résiduels `e8abf2613a4d232d` en `instruction_audit` faute de mapping upstream Git/IDL confirmé, matérialisation locale validée avec `15` liquidity events et `6` lifecycle events sur le corpus DLMM élargi, et version logique replay `dex_decode.v0.7.45.dlmm_add_liquidity_strategies1`. Aucun nouveau `program_id` n’est déclaré vérifié sans preuve/corpus reproductible.
0.7.46 - Meteora DAMM v1 events : extension conservatoire du decoder `meteora_damm_v1` à partir du mapping upstream Git decoder source `meteora-pools-decoder` et du corpus local fourni pour les discriminants observés `07a68aabceabecf4`, `3095dc823d0b09b2`, `856d2cb338ee7221`, `a9204f8988e84689`, `3657a51345e3dae0` et `1513d02bed3eff57` ; ajout des events create_pool, add/remove liquidity, claim_fee, create_lock_escrow et lock_liquidity ; ajout des exports publics associés ; raccordement de la persistance DEX et de la classification non-trade ; passage du replay local à `dex_decode.v0.7.46.damm_v1_events1`. Le programme Meteora Vault reste seulement référencé comme compte/programme associé quand il apparaît dans les comptes DAMM v1 ; aucun `program_id` vault n’est marqué vérifié sans corpus direct dédié.
0.7.46-demo3 - Correction ciblée de Demo3 pour la découverte/backfill : ajout d’un décodage léger des instructions `meteora_damm_v1` connues par discriminant upstream Git dans `onchain_dex_pair_discovery`, classification instruction-scoped prioritaire pour éviter qu’un `Swap` soit classé `add_liquidity` à cause de logs mixtes de transaction, filtrage `target_event` strict sur les surfaces explicites, conservation des transactions mixtes quand un target explicite est demandé malgré `excludeSwaps`, et ajout des cibles UI `create_lock_escrow` / `lock_liquidity`.
0.7.46-demo3-paged - Amélioration Demo3 discovery : pagination `getSignaturesForAddress` via `before_signature` / `until_signature`, scan de plusieurs adresses source dans une seule requête, déduplication des signatures multi-pool, compteur de pages, curseurs `next_before` par adresse et ordre de traitement `newest_first` / `oldest_first` pour constituer un corpus depuis les premières signatures d’un pool sans promouvoir de nouveau `program_id`.
0.7.46-final - Renommage documentaire et payload des anciens statuts/fonctions liés au dépôt source vers une terminologie générique `upstream_git_*` : `proofStatus` utilise désormais `upstream_git_local_corpus_observed`, `upstream_git_mapped_unverified` et `upstream_git_layout_unverified`; les payloads DAMM v1 utilisent `upstreamInstructionName`; la documentation prépare `0.7.47` comme Upstream Git Registry / DEX discovery preparation au lieu d’une tranche DAMM v2 immédiate.
0.7.47 - Upstream Git Registry / DEX discovery preparation : ajout d’un registre générique `upstream_git` pour indexer `program_id`, discriminants d’instructions/events, noms d’entrées et familles de programmes depuis Carbon et sources Git/IDL externes ; extension Demo3 aux targets multi-surfaces, orderbook, burn/mint/transfer/wrap/unwrap/stake ; ajout de groupes de signatures réussies/échouées pour alimenter Demo2 ; maintien strict de l’invariant : aucune entrée upstream Git ne produit trade/candle sans decoder spécialisé et corpus local.
0.7.47-openbook-v2-audit - Ajout d’un decoder local `openbook_v2` audit-only : instructions `place_order`, `cancel_order_by_client_order_id`, `consume_events`, `settle_funds`, `close_open_orders_account`; cleanup du fallback `upstream_git.instruction_match`; extraction audit des `Program return` et `Program data`; mapping des logs `FillLog`, `OpenOrdersPositionLog`, `TotalOrderFillEvent`, `SettleFundsLog`; aucune matérialisation trade/candle.
0.7.47-phoenix-v1-audit - Ajout d’un decoder local `phoenix_v1` audit-only : `order_place`, `order_cancel`, `funds_withdraw`, `log`; parsing strict des instructions log `0x0f`; décodage audit du header Phoenix log et des events `Reduce`, `Place`, `TimeInForce`; correction du mapping `PlaceMultiplePostOnlyOrders` tag `0x10`; aucune matérialisation trade/candle.
0.7.47-doc-matrix - Révision documentaire : ajout d’une matrice DEX dédiée, ajout explicite des sources Git/IDL à consulter, et redécoupage du plan `0.7.48+` en un DEX/version par tranche afin d’éviter les lots “tous events/tous decoders” trop larges.
0.7.47-doc-event-coverage - Ajout d'une matrice événementielle complémentaire `DEX_EVENT_COVERAGE_MATRIX.md` pour suivre, par DEX/version, les familles `swap`, `pool_create`, `liquidity`, `position`, `fee`, `reward`, `admin/config`, `mint`, `burn`, `transfer`, `account_create/close`, `wrap/unwrap`, `orderbook`, `vault`, `lock/unlock`, `launch` et `migration`; ajout de `DB_EVENT_MODEL_REVIEW.md` pour clarifier que `k_sol_dex_decoded_events` suffit à l'audit-only mais que des tables transversales sont nécessaires pour exploiter transfers, orderbook, vault, launch/migration et coverage upstream en requêtes métier.
0.7.48-pre-event-coverage-sync - Raccordement de `k_sol_dex_event_coverage_entries` au registre upstream Git : ajout de `DexEventCoverageService`, sync des entrées registry vers SQLite, inférence conservatoire `event_family` / `expected_db_target`, mapping local limité aux events Raydium déjà connus, refresh des compteurs observés/matérialisés depuis `k_sol_dex_decoded_events` et tables non-trade existantes, sans modification des decoders ni de la matérialisation trade/candle.
0.7.48-pre-event-coverage-fix-docs - Correction du refresh SQL `k_sol_dex_event_coverage_entries` pour éviter les requêtes dynamiques non compatibles avec `sqlx::query` 0.9 ; mise à jour documentaire README/ROADMAP pour acter `0.7.48-pre` comme checkpoint DB/reporting et réaligner la suite sur l’ordre Raydium avant Meteora (`0.7.48 raydium_cpmm`, `0.7.49 raydium_clmm`, puis Pump/Meteora).
0.7.48-pre-event-coverage-report - Clôture du checkpoint `0.7.48-pre` : raccordement des summaries `k_sol_dex_event_coverage_entries` aux diagnostics locaux, ajout des compteurs agrégés de couverture au `LocalPipelineDiagnosticSummaryDto` et au `LocalPipelineValidationReportDto`, ajout du profil `0.7.48-pre_event_coverage_db_checkpoint`, exposition du profil dans Demo Pipeline 2, et maintien des invariants : aucun decoder DEX modifié, aucun trade/candle créé, aucun `program_id` promu sans corpus.
0.7.48-pre-event-coverage-validation-scope - Correction du profil `0.7.48-pre_event_coverage_db_checkpoint` : le contrôle bloquant des trade candidates non matérialisés est maintenant borné aux DEX attendus de la tranche Raydium (`raydium_cpmm`, `raydium_clmm`, `raydium_amm_v4`) afin qu’un DEX partiel hors scope, comme `fluxbeam`, reste diagnostiqué sans bloquer le checkpoint DB/event coverage.
0.7.48-raydium-cpmm-program-data - Complément Raydium CPMM : auto-sync conservatoire de `k_sol_dex_event_coverage_entries` lors des diagnostics/backfills si la table de coverage est vide, décodage des events CPMM `Program data` `swap_event` et `lp_change_event` sans sélecteur Anchor self-CPI, conservation explicite de `swap_event` en audit-only pour éviter tout doublon de trade/candle, matérialisation liquidity conservatoire de `lp_change_event` via `changeType`, et maintien des invariants failed transaction / non-trade / upstream Git.
0.7.48 - Raydium CPMM event coverage clôturé : décodage spécialisé des instructions/events CPMM Carbon/Raydium/fnzero, auto-sync coverage, index technique `k_sol_instruction_observations`, recherche Demo3 par instruction/discriminant, matérialisation validée des swaps, lifecycle, fees, admin/config, deposit/withdraw et `lp_change_event`, conservation de `swap_event` et de l’instruction inconnue `40f4bc78a7e9690a` en audit-only, et maintien des entrées `close_permission_pda` / `update_pool_status` en `upstream_git_mapped_unverified` faute de corpus local.
0.7.46 - Meteora DAMM v1 events finalisés : extension conservatoire du decoder `meteora_damm_v1` depuis upstream Git/corpus local, events create_pool/add/remove liquidity/claim_fee/create_lock_escrow/lock_liquidity, corrections Demo3 ciblées et pagination multi-source, renommage documentaire/payload vers `upstream_git_*`, sans promotion de programme vault ou trade/candle sans preuve locale.
0.7.47 - Upstream Git Registry / DEX discovery preparation : registre générique `upstream_git`, extension Demo3 aux targets multi-surfaces, premiers decoders audit-only OpenBook v2 et Phoenix v1, matrices DEX/event coverage, revue DB et invariant maintenu : aucune entrée upstream ne produit trade/candle sans decoder spécialisé et corpus local.
0.7.48 - Raydium CPMM event coverage clôturé : couverture instructions/events CPMM Carbon/Raydium/fnzero, table coverage synchronisée, `k_sol_instruction_observations`, recherche Demo3 par instruction/discriminant, matérialisation validée des swaps, lifecycle, fees, admin/config, deposit/withdraw et `lp_change_event`, `swap_event` audit-only, fallback upstream remplacé quand le decoder local couvre l’entrée.
0.7.49 - Raydium CLMM event coverage clôturé : 45 entrées listées, 33 instructions locales observées/décodées, 25 entrées matérialisées, ajout `k_sol_orderbook_events`, matérialisation des limit orders, liquidity, fees, rewards, admin/config et lifecycle prouvés par corpus, préparation audit-only des 11 Anchor Program-data events non observés, nettoyage des `raydium_clmm.instruction_audit` et `upstream_git.instruction_match` redondants, validation des invariants failed transaction / non-swap / trade-candle.

4
Cargo.toml
View File

@@ -8,7 +8,7 @@ members = [
]
[workspace.package]
version = "0.7.48"
version = "0.7.49"
edition = "2024"
license = "MIT"
repository = "https://git.sasedev.com/Sasedev/khadhroony-bobobot"
@@ -89,4 +89,4 @@ manual_unwrap_or_default = "allow"
manual_find = "allow"
explicit_counter_loop = "allow"
get_first = "allow"
implicit_saturating_sub = "allow"
implicit_saturating_sub = "allow"

108
README.md
View File

@@ -7,18 +7,53 @@
Ce document reflète le point de reprise `0.7.43-E5C` et l’état de consolidation atteint après `0.7.45` pour `meteora_dlmm`. La version Cargo a évolué ensuite à `0.7.46` côté workspace. Le lot Meteora initialement ouvert en bloc a été redécoupé : `meteora_dlmm` est traité séparément, puis la suite reprend `meteora_damm_v1`, `meteora_damm_v2` et `meteora_dbc` un par un.
## État de reprise actuel `0.7.47-1FE5`
## État courant finalisé `0.7.49`
Le point de reprise courant n’est plus `0.7.43-E5C`. La branche de travail actuelle est `0.7.47-1FE5` : le registre upstream Git est en place, OpenBook v2 et Phoenix v1 disposent de decoders locaux **audit-only**, et la suite doit être conduite par DEX/version au lieu de tenter tous les events en une seule session.
La branche de travail `0.7.49` clôture la tranche `raydium_clmm` après la clôture fonctionnelle de `0.7.48 raydium_cpmm`. Le code Rust de `kb_lib` est considéré finalisé pour cette tranche, sous réserve des validations locales habituelles (`cargo fmt`, `cargo test -p kb_lib`, `cargo clippy -p kb_lib --all-targets -- -D warnings`).
Les nouveaux chemins audit-only doivent rester non matérialisants : aucun event OpenBook v2 ou Phoenix v1 ne doit produire de trade, metric ou candle tant que le sens économique complet n’est pas validé.
État CLMM validé sur corpus local après replay forcé :
```text
listed_entry_count = 45
decoded_entry_count = 33
observed_entry_count = 33
materialized_entry_count = 25
total_observed_count = 2560
total_materialized_count = 1367
trade_count = 1186
raydium_clmm.instruction_audit résiduel = 0
upstream_git.instruction_match localement couvert = 0
failed tx matérialisées = 0
non-swap CLMM avec trade_count > 0 = 0
```
Les 11 Anchor / `Program data` events CLMM restent listés en `upstream_git_unverified` car aucun corpus local ne les observe encore. Le code est préparé pour les accueillir en audit-only lorsqu’ils apparaîtront dans un corpus local, sans créer de trade/candle par défaut.
La prochaine tranche fonctionnelle est `0.7.50 raydium_launchpad`, avant `0.7.51 raydium_amm_v4` et `0.7.52 raydium_stable`.
## Organisation documentaire
La racine conserve uniquement les documents de pilotage principaux :
- `README.md` ;
- `ROADMAP.md` ;
- `CHANGELOG.md`.
Les documents spécialisés sont rangés dans :
- `docs/` pour les matrices et revues de modèle ;
- `docs/reports/` pour les rapports de couverture DEX/version ;
- `docs/prompts/` pour les prompts de reprise ;
- `validation_sql/` pour les scripts SQL de validation.
Voir aussi :
- `DEX_DECODER_MATRIX.md` pour la matrice DEX détaillée ;
- `ROADMAP.md` pour le plan révisé `0.7.48` à `0.7.61` ;
- `CHANGELOG.md` pour les tranches `0.7.47-*`.
- `docs/DEX_DECODER_MATRIX.md` pour la matrice DEX détaillée ;
- `docs/DEX_EVENT_COVERAGE_MATRIX.md` pour la matrice de familles d’events ;
- `docs/DB_EVENT_MODEL_REVIEW.md` pour la revue du modèle DB ;
- `docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md` et `validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` pour la clôture CPMM ;
- `docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md` et `validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql` pour la clôture CLMM ;
- `docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md` pour reprendre en `0.7.50`.
## Sources upstream Git / IDL à utiliser en `0.7.47+`
@@ -345,17 +380,14 @@ Si une requête DB est ajoutée ou modifiée, mettre à jour les re-exports dans
La priorité immédiate après le point de reprise `0.7.43-E5C` est :
1. `0.7.43` : resynchronisation documentaire, normalisation DEX-first et gel du point de reprise ;
2. `0.7.44` : ajout du ledger de décodage/replay et options de replay `force` / skip sûr ;
3. `0.7.45` : reprise séparée de `meteora_dlmm` — clôturée côté corpus DLMM actuel ;
4. `0.7.46` : reprise séparée de `meteora_damm_v1` — clôturée côté corpus DAMM v1 local ;
5. `0.7.47` : Upstream Git Registry / DEX discovery preparation — registre générique des `program_id`, discriminants, instructions et events issus de dépôts Git externes, tous non vérifiés par défaut ;
6. `0.7.48-pre` : event coverage + DB model checkpoint — table coverage, sync upstream, refresh counts, diagnostics et profil validation ;
7. `0.7.48` : reprise séparée de `raydium_cpmm` ;
8. `0.7.49` : reprise séparée de `raydium_clmm` ;
9. `0.7.50` : reprise séparée de `pump_swap` ;
10. `0.7.51` : reprise séparée de `pump_fun` ;
11. `0.7.52+` : Meteora puis validation progressive des autres DEX/surfaces issus du registre upstream Git : Orca, FluxBeam, DexLab, Lifinity, Phoenix, OpenBook, Stabble, BonkSwap, Boop, Moonshot, Heaven, Wavebreak, Vertigo, Virtuals, Pancake Swap, OKX DEX, Raydium LaunchLab/Stable/Locking, puis launch surfaces.
1. `0.7.48` : `raydium_cpmm` — clôturé côté event coverage ;
2. `0.7.49` : `raydium_clmm` — clôturé côté instructions observées, matérialisation non-trade prouvée et nettoyage fallback ;
3. `0.7.50` : `raydium_launchpad` ;
4. `0.7.51` : `raydium_amm_v4` ;
5. `0.7.52` : `raydium_stable` ;
6. `0.7.53` : `pump_swap` ;
7. `0.7.54` : `pump_fun` ;
8. `0.7.55+` : Meteora, Phoenix/OpenBook, Orca puis validation progressive des autres DEX/surfaces issus du registre upstream Git.
Garde-fous constants :
@@ -436,9 +468,9 @@ La matrice DEX/version doit être complétée par une matrice événementielle e
Voir :
- `DEX_DECODER_MATRIX.md` pour le statut par DEX/version ;
- `DEX_EVENT_COVERAGE_MATRIX.md` pour les familles d'events à couvrir ;
- `DB_EVENT_MODEL_REVIEW.md` pour les ajouts DB à envisager avant `0.7.48+`.
- `docs/DEX_DECODER_MATRIX.md` pour le statut par DEX/version ;
- `docs/DEX_EVENT_COVERAGE_MATRIX.md` pour les familles d'events à couvrir ;
- `docs/DB_EVENT_MODEL_REVIEW.md` pour les ajouts DB à envisager avant `0.7.48+`.
## Note 0.7.48-pre — Event coverage DB checkpoint
@@ -462,9 +494,10 @@ La suite fonctionnelle reprend par Raydium avant Meteora :
1. `0.7.48` — `raydium_cpmm` ;
2. `0.7.49` — `raydium_clmm` ;
3. `0.7.50` — `pump_swap` ;
4. `0.7.51` — `pump_fun` ;
5. `0.7.52+` — Meteora puis les autres DEX/surfaces.
3. `0.7.50` — `raydium_launchpad` ;
4. `0.7.51` — `raydium_amm_v4` ;
5. `0.7.52` — `raydium_stable` ;
6. `0.7.53+` — Pump, Meteora, Phoenix/OpenBook, Orca puis les autres DEX/surfaces.
## Note 0.7.48 — Raydium CPMM event coverage
@@ -478,9 +511,9 @@ Aucune nouvelle table DB n'est ajoutée en `0.7.48`. Les transfers, token accoun
Voir aussi :
- `DEX_EVENT_COVERAGE_MATRIX.md` pour la couverture par familles ;
- `RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md` pour le rapport de tranche ;
- `SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` pour les requêtes de validation.
- `docs/DEX_EVENT_COVERAGE_MATRIX.md` pour la couverture par familles ;
- `docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md` pour le rapport de tranche ;
- `validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` pour les requêtes de validation.
Complément `0.7.48-raydium-cpmm-program-data` : la coverage DB est maintenant synchronisée automatiquement quand un refresh de coverage est demandé sur une base neuve et qu'aucune ligne n'existe encore. Les backfills token/pool/signature déclenchent aussi un refresh best-effort de la coverage afin d'éviter un état incohérent où `k_sol_dex_decoded_events` est rempli mais `k_sol_dex_event_coverage_entries` reste vide.
@@ -499,3 +532,24 @@ La tranche `0.7.48` clôture la couverture Raydium CPMM sur corpus local. Les en
La table technique `k_sol_instruction_observations` est ajoutée pour indexer localement les instructions observées par `decoder_code`, `instruction_name` et `discriminator_hex`. Demo3 peut rechercher par instruction/discriminant, ce qui reproduit localement l’usage pratique du filtre Solscan `instruction=<discriminator>`.
État final CPMM observé : `561` trades, `50` liquidity events, `9` lifecycle events, `25/25` `lp_change_event` matérialisés, `swap_event` audit-only à `0` trade, et deux entrées connues mais non observées (`close_permission_pda`, `update_pool_status`) conservées en `upstream_git_mapped_unverified`.
## Note 0.7.49 — Raydium CLMM event coverage final
La tranche `0.7.49` clôture `raydium_clmm` comme deuxième tranche Raydium après CPMM. Elle ajoute la couverture complète des instructions CLMM observées depuis Carbon, IDL Raydium, Pinax, fnzero et corpus Solscan/backfill, ainsi que la table transversale `k_sol_orderbook_events` pour les instructions limit-order.
Points finalisés :
- `45` entrées listées dans `k_sol_dex_event_coverage_entries` ;
- `33` instructions CLMM avec `local_event_kind` spécialisé ;
- `33` instructions observées dans le corpus local ;
- `25` entrées matérialisées ;
- swaps matérialisés uniquement via `swap` / `swap_v2` ;
- limit orders `open`, `increase`, `decrease`, `close`, `settle` décodés et matérialisés en `k_sol_orderbook_events` quand la transaction réussit ;
- non-trades CLMM vers `liquidity`, `fee`, `reward`, `admin`, `lifecycle` et `orderbook` sans trade/candle ;
- transactions échouées conservées audit-only ;
- `raydium_clmm.instruction_audit` résiduel à zéro ;
- `upstream_git.instruction_match` localement couvert à zéro après replay ;
- 11 Anchor / `Program data` events CLMM préparés mais conservés `upstream_git_unverified` faute d’observation locale.
La validation finale est dans `validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql`.

131
ROADMAP.md
View File

@@ -4,7 +4,7 @@
## 0.7.47-1FE5 — Décision de planification : ne plus viser “tous les events en une session”
La phase `0.7.47` a montré que l’objectif “réimplémenter tous les décodeurs Carbon et toutes les sources en un seul bloc” est trop large. Le plan est donc redécoupé en **un DEX/version par tranche**, avec une matrice documentaire dédiée : `DEX_DECODER_MATRIX.md`.
La phase `0.7.47` a montré que l’objectif “réimplémenter tous les décodeurs Carbon et toutes les sources en un seul bloc” est trop large. Le plan est donc redécoupé en **un DEX/version par tranche**, avec une matrice documentaire dédiée : `docs/DEX_DECODER_MATRIX.md`.
Règles de planification :
@@ -28,24 +28,25 @@ Règles de planification :
| `https://github.com/all-in-one-blockchain/phoenix-onchain-mm` | Source Phoenix/MM complémentaire. |
| `https://docs.vybenetwork.com/docs/available-dexs-amms` | Source externe de découverte DEX/AMM, non vérifiante. |
### Plan révisé `0.7.48` à `0.7.61`
### Plan révisé `0.7.48` à `0.7.62+`
| Version cible | Scope | Objectif de clôture |
|---|---|---|
| `0.7.48` | `raydium_cpmm` | Reprendre tous les discriminants/events depuis Carbon/Solana Streamer ; vérifier swaps, liquidity, fees/admin ; confirmer matérialisation trade/candle et non-trade. |
| `0.7.49` | `raydium_clmm` | Couvrir toutes les instructions CLMM : swaps, positions, liquidity, fees/rewards, Token-2022 ; valider matérialisation non-trade. |
| `0.7.50` | `pump_swap` | Couvrir `buy/sell` et tous les events auxiliaires disponibles : fees, cashback, volume accumulator, admin/config. |
| `0.7.51` | `pump_fun` | Traiter launch/bonding/migration ; séparer création token, buy/sell bonding, migration vers DEX effectif. |
| `0.7.52` | `meteora_dbc` | Couverture DBC : bonding curve, swap, migration, launch attribution, fees/admin, non-trade. |
| `0.7.53` | `meteora_dlmm` | Audit final de parité avec sources Git/IDL ; fermer ou documenter les audits résiduels. |
| `0.7.54` | `meteora_damm_v1` | Parité upstream complète ; résoudre les cas non matérialisés faute de pool/pair quand possible. |
| `0.7.55` | `meteora_damm_v2` | Couverture DAMM v2 complète : create, swap, liquidity, fees/admin/config ; décider trade actionability. |
| `0.7.56` | `phoenix_v1` | Finir tous les events Git disponibles en audit ; préparer mais ne pas activer trade materialization. |
| `0.7.57` | `openbook_v2` | Finir layouts logs/events ; définir conditions futures de trade/candle sans les activer par défaut. |
| `0.7.58` | `orca_whirlpools` | Reprendre Whirlpools depuis IDL/source : swaps, pools, positions, liquidity, fees/rewards. |
| `0.7.59` | Launch surfaces | Raydium LaunchLab/Launchpad, PumpFun migration, Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk. |
| `0.7.60` | DEX historiques / candidats | FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi, etc. par corpus. |
| `0.7.61` | Validation consolidée | Rejouer une base neuve multi-DEX, vérifier matrice, zéro faux trade/candle, rapport de couverture par DEX/event. |
| `0.7.48` | `raydium_cpmm` | Clôturé : instructions/events CPMM, lifecycle, fees, admin/config, deposit/withdraw, `lp_change_event`, invariants trade/candle. |
| `0.7.49` | `raydium_clmm` | Clôturé : 33 instructions observées/décodées, orderbook CLMM, liquidity/fee/reward/admin/lifecycle, fallbacks upstream nettoyés, 11 Program-data events préparés mais non observés. |
| `0.7.50` | `raydium_launchpad` | Reprendre Launchpad comme surface Raydium prioritaire : identifier les program ids/IDL, launch, pool creation, migration, bonding éventuel, fees/admin, et rattachement au DEX effectif. |
| `0.7.51` | `raydium_amm_v4` | Reprendre AMM v4 legacy au même niveau de couverture que CPMM/CLMM : swaps, pool lifecycle, liquidity, fees/admin, side effects documentés. |
| `0.7.52` | `raydium_stable` | Reprendre Raydium Stable : program ids/IDL, swaps stables, pool lifecycle, liquidity, fees/admin, invariants pricing/candles. |
| `0.7.53` | `pump_swap` | Couvrir `buy/sell` et tous les events auxiliaires disponibles : fees, cashback, volume accumulator, admin/config. |
| `0.7.54` | `pump_fun` | Traiter launch/bonding/migration ; séparer création token, buy/sell bonding, migration vers DEX effectif. |
| `0.7.55` | `meteora_dbc` | Couverture DBC : bonding curve, swap, migration, launch attribution, fees/admin, non-trade. |
| `0.7.56` | `meteora_dlmm` | Audit final de parité avec sources Git/IDL ; fermer ou documenter les audits résiduels. |
| `0.7.57` | `meteora_damm_v1` | Parité upstream complète ; résoudre les cas non matérialisés faute de pool/pair quand possible. |
| `0.7.58` | `meteora_damm_v2` | Couverture DAMM v2 complète : create, swap, liquidity, fees/admin/config ; décider trade actionability. |
| `0.7.59` | `phoenix_v1` | Finir tous les events Git disponibles en audit ; préparer mais ne pas activer trade materialization. |
| `0.7.60` | `openbook_v2` | Finir layouts logs/events ; définir conditions futures de trade/candle sans les activer par défaut. |
| `0.7.61` | `orca_whirlpools` | Reprendre Whirlpools depuis IDL/source : swaps, pools, positions, liquidity, fees/rewards. |
| `0.7.62+` | Launch surfaces / DEX candidats / validation consolidée | Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk, FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi puis base neuve multi-DEX. |
Ce plan remplace les anciens regroupements larges `0.7.50+` qui mélangeaient plusieurs DEX dans une même version.
@@ -1245,7 +1246,7 @@ Statut : implémenté en micro-tranche DB/reporting, sans modifier les decoders
Fait :
- maintien de `DEX_EVENT_COVERAGE_MATRIX.md` en plus de `DEX_DECODER_MATRIX.md` ;
- maintien de `docs/DEX_EVENT_COVERAGE_MATRIX.md` en plus de `docs/DEX_DECODER_MATRIX.md` ;
- ajout de `k_sol_dex_event_coverage_entries` dans `kb_lib/src/db/schema.rs` ;
- ajout des entity/DTO/queries/re-exports associés ;
- ajout de `DexEventCoverageService` pour synchroniser les entrées du registre upstream Git vers la table de coverage ;
@@ -1278,75 +1279,82 @@ Objectif : reprendre `raydium_cpmm` en premier, avant Meteora, avec une couvertu
- vérifier par SQL que les non-trades ne produisent aucun trade/candle.
### 6.081. Version `0.7.49` — `raydium_clmm` event coverage
Objectif : reprendre `raydium_clmm` après CPMM, avec couverture des swaps, positions, liquidité, rewards, fees, protocol fees et cas Token-2022.
Objectif : clôturer `raydium_clmm` après CPMM.
À faire :
Réalisé :
- lister tous les events/instructions CLMM depuis Carbon/fnzero/IDL ;
- consolider `swap`, `swap_v2`, open/close position, increase/decrease liquidity, reward/fee/admin ;
- classer les events non observés en `upstream_git_mapped_unverified` ;
- matérialiser uniquement les events prouvés par corpus ;
- vérifier absence de faux trades/candles.
- couverture locale de `45` entrées CLMM ;
- `33` instructions spécialisées, observées et décodées ;
- matérialisation contrôlée de `25` entrées vers trade, liquidity, fee, reward, admin, lifecycle et orderbook ;
- ajout de `k_sol_orderbook_events` pour les limit orders CLMM ;
- suppression automatique des fallbacks `upstream_git.instruction_match` localement couverts ;
- préparation audit-only des 11 Anchor / `Program data` events non encore observés ;
- invariants validés : aucun faux trade/candle, aucune matérialisation sur transaction échouée, `raydium_clmm.instruction_audit` résiduel à zéro.
### 6.082. Version `0.7.50` — `pump_swap` event coverage
### 6.082. Version `0.7.50` — `raydium_launchpad` event coverage
Objectif : reprendre Raydium Launchpad comme prochaine surface Raydium, avant AMM v4 et Stable.
À faire : identifier les program ids/IDL depuis sources Git/IDL/Solscan, couvrir launch/pool creation, migration/rattachement au DEX effectif, fees/admin/config, éventuels side effects SPL/Token-2022, et matérialiser seulement les events prouvés par corpus local.
### 6.083. Version `0.7.51` — `raydium_amm_v4` event coverage
Objectif : hisser AMM v4 legacy au niveau de couverture CPMM/CLMM.
À faire : revisiter swaps, initialize/pool lifecycle, add/remove liquidity, fees/admin/config, side effects SPL, failed transaction safety, fallback upstream et validation SQL dédiée.
### 6.084. Version `0.7.52` — `raydium_stable` event coverage
Objectif : reprendre Raydium Stable comme troisième tranche Raydium post-CLMM.
À faire : vérifier program ids/IDL, swaps stables, liquidity, pool lifecycle, fees/admin/config, cohérence des montants/prix et absence de faux trades/candles.
### 6.085. Version `0.7.53` — `pump_swap` event coverage
Objectif : compléter `pump_swap` au-delà de `buy/sell`.
À faire : couvrir fees, cashback, volume accumulator, admin/config et autres events upstream disponibles, tout en maintenant l’invariant non-trade = zéro trade/candle.
### 6.083. Version `0.7.51` — `pump_fun` launch/bonding/migration
### 6.086. Version `0.7.54` — `pump_fun` launch/bonding/migration
Objectif : séparer launch/bonding de DEX effectif et valider migration vers PumpSwap ou autre surface tradable.
À faire : traiter create, buy/sell bonding, update/config, mint/burn éventuels, migration/graduate et rattachement au pool tradable.
### 6.084. Version `0.7.52` — `meteora_dbc` séparé
### 6.087. Version `0.7.55` — `meteora_dbc` séparé
Objectif : reprendre Meteora après les tranches Raydium et Pump, en séparant bonding/launch, swap effectif, migration et attribution d’origine.
À faire : vérifier swaps exploitables, migration, lifecycle, mint/burn éventuels, launch attribution, fees/admin, sans candle artificielle sur events non pricés.
### 6.085. Version `0.7.53` — `meteora_dlmm` parité upstream finale
### 6.088. Version `0.7.56` — `meteora_dlmm` parité upstream finale
Objectif : comparer la couverture locale DLMM déjà avancée avec toutes les sources Git/IDL et documenter ou fermer les audits résiduels.
À faire : revalider swaps, liquidity, positions, lifecycle, fees/rewards/admin, et garder les discriminants non mappés en audit documenté.
### 6.086. Version `0.7.54` — `meteora_damm_v1` parité upstream finale
### 6.089. Version `0.7.57` — `meteora_damm_v1` parité upstream finale
Objectif : compléter la tranche DAMM v1 déjà engagée, résoudre les surfaces non observées et améliorer le rattachement pool/pair quand possible.
À faire : vérifier toutes les instructions upstream restantes, matérialiser uniquement les events prouvés et documenter les cas sans pool/pair local.
### 6.087. Version `0.7.55` — `meteora_damm_v2` séparé
### 6.090. Version `0.7.58` — `meteora_damm_v2` séparé
Objectif : reprendre DAMM v2 comme DEX effectif séparé après disponibilité du ledger de coverage.
À faire : consolider create_pool, swaps exploitables, configs dynamiques, liquidity, fees/admin, lifecycle ; conserver les swaps sans payload montant/prix fiable comme `non_actionable_trade`.
### 6.088. Version `0.7.56` — `phoenix_v1` audit-only complet
### 6.091. Version `0.7.59` — `phoenix_v1` audit-only complet
Objectif : finir tous les events Git disponibles en audit, sans activer de trade/candle.
À faire : couvrir `Fill`, `FillSummary`, `Fee`, `Evict`, `ExpiredOrder` et autres logs/events disponibles ; préparer le futur modèle orderbook sans matérialisation marché par défaut.
### 6.089. Version `0.7.57` — `openbook_v2` audit-only complet
### 6.092. Version `0.7.60` — `openbook_v2` audit-only complet
Objectif : finir les layouts logs/events OpenBook v2 et définir les conditions futures de matérialisation orderbook/trade.
À faire : vérifier fills, settle, consume events, open orders create/close, maker/taker, lots/decimals et sens économique avant toute promotion.
### 6.090. Version `0.7.58` — `orca_whirlpools` event coverage
### 6.093. Version `0.7.61` — `orca_whirlpools` event coverage
Objectif : reprendre Whirlpools depuis IDL/source avec corpus dédié.
À faire : swaps, pools, positions, liquidity, fees/rewards, tick arrays, mint/burn/Token-2022 si applicable.
### 6.091. Version `0.7.59` — Launch surfaces
Objectif : traiter les surfaces de lancement après les DEX effectifs prioritaires.
### 6.094. Version `0.7.62+` — Launch surfaces, DEX historiques/candidats et validation consolidée
Objectif : traiter les surfaces restantes puis rejouer une base neuve multi-DEX.
À faire : Raydium LaunchLab/Launchpad, PumpFun migration, Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk, avec séparation stricte launch origin / pool origin / DEX effectif.
### 6.092. Version `0.7.60` — DEX historiques / candidats
Objectif : valider les DEX ou surfaces candidates par corpus, sans promotion automatique depuis les sources externes.
À faire : FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi et autres entrées Vybe/registry.
### 6.093. Version `0.7.61` — Validation consolidée
Objectif : rejouer une base neuve multi-DEX et valider les invariants du pipeline complet.
À faire : rapport coverage par DEX/event, zéro faux trade/candle, corpus documentés, matrices cohérentes, diagnostics bloquants à zéro.
À faire : Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk, FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi et autres entrées Vybe/registry ; rapport coverage par DEX/event, zéro faux trade/candle, corpus documentés, matrices cohérentes, diagnostics bloquants à zéro.
### 6.091. Version `0.8.x` — Analyse et filtrage
Objectif : transformer les événements bruts en signaux exploitables.
@@ -1540,18 +1548,21 @@ Ordre de travail recommandé pour la suite :
3. `0.7.46` : `meteora_damm_v1` — clos côté corpus local ;
4. `0.7.47` : Upstream Git Registry / DEX discovery preparation — acquis ;
5. `0.7.48-pre` : event coverage + DB model checkpoint — clos après table, sync upstream, refresh counts, diagnostics et profil validation ;
6. `0.7.48` : `raydium_cpmm` ;
7. `0.7.49` : `raydium_clmm` ;
8. `0.7.50` : `pump_swap` ;
9. `0.7.51` : `pump_fun` ;
10. `0.7.52` : `meteora_dbc` ;
11. `0.7.53` : `meteora_dlmm` parité upstream finale ;
12. `0.7.54` : `meteora_damm_v1` parité upstream finale ;
13. `0.7.55` : `meteora_damm_v2` ;
14. `0.7.56` : `phoenix_v1` audit-only complet ;
15. `0.7.57` : `openbook_v2` audit-only complet ;
16. `0.7.58` : `orca_whirlpools` ;
17. `0.7.59+` : launch surfaces, DEX candidats/historiques et validation consolidée.
6. `0.7.48` : `raydium_cpmm` — clos ;
7. `0.7.49` : `raydium_clmm` — clos ;
8. `0.7.50` : `raydium_launchpad` ;
9. `0.7.51` : `raydium_amm_v4` ;
10. `0.7.52` : `raydium_stable` ;
11. `0.7.53` : `pump_swap` ;
12. `0.7.54` : `pump_fun` ;
13. `0.7.55` : `meteora_dbc` ;
14. `0.7.56` : `meteora_dlmm` parité upstream finale ;
15. `0.7.57` : `meteora_damm_v1` parité upstream finale ;
16. `0.7.58` : `meteora_damm_v2` ;
17. `0.7.59` : `phoenix_v1` audit-only complet ;
18. `0.7.60` : `openbook_v2` audit-only complet ;
19. `0.7.61` : `orca_whirlpools` ;
20. `0.7.62+` : launch surfaces, DEX candidats/historiques et validation consolidée.
Garde-fous constants :
@@ -1602,7 +1613,7 @@ cargo test -p kb_lib
cargo clippy -p kb_lib --all-targets -- -D warnings
```
Puis relancer la validation SQL `SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` sur la base de corpus CPMM.
Puis relancer la validation SQL `validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` sur la base de corpus CPMM.
Complément appliqué après constitution du corpus CPMM :
@@ -1623,4 +1634,4 @@ La tranche CPMM reconnaît désormais tous les discriminants instruction-level l
`0.7.48` est clôturable côté `raydium_cpmm`. Le decoder couvre les instructions/events CPMM listés par Carbon/fnzero/Raydium CP-Swap, avec matérialisation locale validée pour trades, liquidity, lifecycle, fees et admin/config. `swap_event` reste audit-only pour éviter les doublons avec `swap_base_input` / `swap_base_output`. Les side effects SPL Token / Token-2022 observés via Solscan (`burn`, `transfer`, `transferChecked`, `closeAccount`) restent hors decoder CPMM direct et alimenteront une réflexion transversale future.
La suite reprend en `0.7.49` par `raydium_clmm`, en utilisant les sources Git/IDL habituelles et la page Solscan Program IDL `https://solscan.io/account/CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK#programIdl` pour accélérer la découverte de signatures par discriminant.
La suite après `0.7.49 raydium_clmm` reprend en `0.7.50` par `raydium_launchpad`, puis `0.7.51 raydium_amm_v4` et `0.7.52 raydium_stable`, en gardant la même discipline : sources Git/IDL + Solscan pour accélérer la découverte, mais corpus local obligatoire avant toute promotion métier.

22
docs/ARCHIVE_ORGANIZATION.md Normal file
View File

@@ -0,0 +1,22 @@
<!-- file: docs/ARCHIVE_ORGANIZATION.md -->
# Organisation de l’archive documentaire
La racine du workspace garde les documents de pilotage principaux :
```text
README.md
ROADMAP.md
CHANGELOG.md
```
Les documents spécialisés sont rangés par usage :
```text
docs/ matrices et revues transversales
docs/reports/ rapports de couverture par tranche DEX/version
docs/prompts/ prompts de reprise de session
validation_sql/ scripts SQL de validation
```
Cette réorganisation ne modifie pas le code Rust/Tauri. Elle sert uniquement à séparer les fichiers de pilotage, les rapports, les prompts et les validations SQL.

14
DB_EVENT_MODEL_REVIEW.md → docs/DB_EVENT_MODEL_REVIEW.md
View File

@@ -306,3 +306,17 @@ Aucune nouvelle table n'est ajoutée pour CPMM :
La tranche `raydium_cpmm` ajoute `k_sol_instruction_observations` comme table technique d’index local, non comme table métier. Elle sert à chercher les instructions observées par `decoder_code`, `instruction_name` et `discriminator_hex`, puis à relier ces observations au corpus backfillé/rejoué.
La matérialisation métier reste limitée aux tables existantes : `k_sol_trade_events`, `k_sol_liquidity_events`, `k_sol_pool_lifecycle_events`, `k_sol_fee_events` et `k_sol_pool_admin_events`. Les opérations SPL Token / Token-2022 visibles dans Solscan (`burn`, `transfer`, `transferChecked`, `closeAccount`) ne sont pas promues en tables métier dans `0.7.48`; elles justifient seulement une future table transversale si plusieurs DEX le nécessitent.
## Note 0.7.49 — Raydium CLMM sans nouvelle table transversale immédiate
La reprise `raydium_clmm` confirme que la table `k_sol_dex_event_coverage_entries` suffit pour inventorier les instructions/events CLMM avant promotion métier. Les entrées IDL-only ajoutées en `0.7.49` restent des lignes de coverage et de recherche, pas des tables métier.
Aucune nouvelle table transversale n'est ajoutée dans ce delta :
- les swaps spécialisés restent dans `k_sol_trade_events` seulement lorsque les montants et le sens économique sont validés ;
- les liquidity/positions CLMM restent à distinguer : une position NFT/tick n'est pas forcément une liquidity row simple ;
- les fees et rewards peuvent utiliser `k_sol_fee_events` / `k_sol_reward_events` seulement après preuve de corpus et typage fin ;
- les side effects SPL Token / Token-2022 (`mint`, `burn`, `transfer`, `closeAccount`, wrap/unwrap SOL) restent indirects tant qu'une preuve multi-DEX ne justifie pas une table transversale.
La table `k_sol_instruction_observations` reste technique : elle sert à trouver des signatures et discriminants observés localement, sans être une preuve métier.

63
DEX_DECODER_MATRIX.md → docs/DEX_DECODER_MATRIX.md
View File

@@ -28,31 +28,33 @@ Cette matrice complète `kb_lib/src/dex_support_matrix.rs`. Elle documente **ce
| Ordre | DEX/version | État actuel | Fait | Reste à faire |
|---:|---|---|---|---|
| 1 | `raydium_cpmm` | `supported` | Swaps matérialisés ; premiers non-swaps prouvés (`initialize`, `withdraw`, `collect_creator_fee`) ; trades/candles OK. | Comparer tous les discriminants Carbon/IDL ; compléter fees/admin/lifecycle ; rejouer sur base dédiée ; vérifier absence d’audits orphelins. |
| 2 | `raydium_clmm` | `supported` | Swaps `swap`/`swap_v2` ; events positions/liquidité prouvés ; matérialisation non-trade existante. | Repasser tous les events Carbon/IDL : open/close position, rewards, protocol fees, Token-2022 ; compléter audit → materialized si corpus. |
| 3 | `pump_swap` | `supported` | `buy`/`sell` décodés et matérialisés ; trade/candle OK. | Ajouter tous les events Carbon/Solana Streamer : cashback, fee, volume accumulator, admin/config ; conserver les non-trades hors candles. |
| 4 | `pump_fun` | `partial / launch_surface` | Création/token launch partiellement décodée ; intégrée au pipeline de listings. | Traiter tous les events Pump.fun disponibles : buy/sell/migrate/create/update ; séparer bonding/launch de DEX effectif ; valider migration vers PumpSwap. |
| 5 | `meteora_dbc` | `partial` | Swaps/instruction audits observés ; Demo3 donne du corpus. | Couverture complète DBC : launch/bonding curve, swap, migration, config/admin, fees ; matérialiser seulement ce qui est prouvé. |
| 6 | `meteora_dlmm` | `supported` | Couverture avancée validée en `0.7.45` : swaps, liquidity, positions, lifecycle, fees ; non-trade matérialisé. | Résoudre les audits résiduels non mappés ; comparer Carbon/IDL pour events rewards/admin restants ; revalidation base neuve. |
| 7 | `meteora_damm_v1` | `supported / partial events` | Couverture `0.7.46` : swap, create_pool, add/remove liquidity, claim_fee, create_lock_escrow, lock_liquidity. | Vérifier les surfaces upstream non observées ; améliorer rattachement pool/pair pour remove_liquidity non matérialisés ; revalidation stricte. |
| 8 | `meteora_damm_v2` | `partial` | `swap`, `instruction_audit`, registry/discriminants et corpus Demo3 existent. | Couvrir tous les events Carbon/source : create pool, liquidity, fees, dynamic config, admin ; déterminer actionability des swaps ; matérialiser si montants fiables. |
| 9 | `phoenix_v1` | `audit-only` | Decoder local audit-only ; `log_audit`, order place/cancel, withdraw ; parsing strict `0x0f`; events `Reduce`, `Place`, `TimeInForce` observés ; `trade_count=0`. | Terminer tous les events Git : `Fill`, `FillSummary`, `Fee`, `Evict`, `ExpiredOrder`, etc. ; ajouter counts/flags audit ; seulement ensuite étudier trade materialization. |
| 10 | `openbook_v2` | `audit-only` | Decoder local audit-only ; instructions order/cancel/consume/settle ; `Program data` mappé : `FillLog`, `OpenOrdersPositionLog`, `TotalOrderFillEvent`, `SettleFundsLog`; `trade_count=0`. | Vérifier layouts fill/out et sens maker/taker/base/quote ; ajouter table audit éventuelle ; ne matérialiser trades qu’après validation du sens économique. |
| 11 | `orca_whirlpools` | `partial` | Premier decoder historique présent ; swaps/create_pool partiels. | Comparer Carbon/IDL complet ; couvrir liquidity, positions, fees/rewards, tick arrays ; valider swaps exploitables et non-trades. |
| 12 | `raydium_launchlab` | `planned launch` | Entrée canonique locale LaunchLab/Launchpad ; program id connu localement. | Décoder launch/migration ; ne pas confondre avec CPMM/CLMM/AMM v4 ; rattacher aux pools tradables. |
| 13 | `meteora_vault` | `to_verify` | Présent comme indice upstream / compte associé. | Corpus direct obligatoire ; decoder séparé si events vault réels ; aucune promotion via DAMM indirect. |
| 14 | `fluxbeam` | `partial/to_verify` | Decoder initial existant ; Demo3 peut produire des candidats. | Vérifier source/IDL ; compléter swap, pool, liquidity, fees/admin ; matérialisation uniquement après corpus. |
| 15 | `dexlab` | `partial/to_verify` | Decoder initial historique ; ancienne entrée beta supprimée. | Reconfirmer program id/source ; décoder events disponibles ; distinguer DexLab natif et liens OpenBook/market. |
| 16 | `lifinity_v2` | `to_verify` | Program id listé par sources externes/Vybe ; pas de corpus concluant. | Trouver IDL/source ; Demo3 par program/market ; audit-only d’abord. |
| 17 | `stabble_stable_swap` / `stabble_weighted_swap` | `to_verify` | Program ids/indices via sources externes ; candidats Demo3 observables. | Source/IDL + corpus + decoder audit-only ; déterminer surface AMM et montants exploitables. |
| 18 | `bonkswap` | `to_verify` | Program id/Carbon/Vybe selon registre ; swaps candidats possibles. | Vérifier program id, source et corpus ; décoder tous events ; pas de trade sans montants. |
| 19 | `boop` / `boop_fun` | `to_verify / launch` | Entrée de découverte. | Séparer launch surface et swap effectif ; corpus + source obligatoire. |
| 20 | `moonshot` / `moonit` | `to_verify / launch` | Moonshot buy/sell observés via upstream candidates ; Moonit launch attribution historique. | Source/IDL + migration + rattachement pools ; éviter heuristiques seules. |
| 21 | `heaven` | `to_verify` | Program id/candidat ajouté en matrice. | Vérifier s’il est launch, AMM ou les deux ; corpus dédié. |
| 22 | `printr` | `to_verify` | Preset Demo3 ajouté ; candidats observables. | Source/IDL, discriminants, corpus, decoder audit-only. |
| 23 | `metadao_*` | `to_verify` | Presets spécifiques : launchpad, bid wall, futarchy, AMM. | Traiter par programme séparé ; ne pas utiliser mint ids comme program ids ; corpus obligatoire. |
| 24 | `raydium_stable_swap` | `planned/historical` | Entrée conservée. | Reprendre uniquement si corpus réel ; stable AMM séparé de CPMM/CLMM. |
| 25 | `jupiter_*`, `dflow_aggregator_v4`, `okx_dex` | `aggregator_router` | Registry/discovery pour contexte transactionnel. | Ne pas matérialiser en DEX direct ; utiliser pour routeSource/routing/context. |
| 1 | `raydium_cpmm` | `supported / 0.7.48 closed` | Couverture CPMM clôturée : swaps, lifecycle, fees, admin/config, deposit/withdraw, `lp_change_event`, `swap_event` audit-only. | Réouvrir seulement en cas de nouveau corpus ou divergence upstream. |
| 2 | `raydium_clmm` | `supported / 0.7.49 closed` | Couverture CLMM clôturée : 45 entrées listées, 33 instructions observées/décodées, 25 matérialisées, orderbook events, fallback upstream nettoyé. | 11 Anchor Program-data events restent préparés mais `upstream_git_unverified` faute de corpus local. |
| 3 | `raydium_launchpad` | `planned / 0.7.50` | Prochaine tranche Raydium. | Identifier program ids/IDL, launch/pool creation, migration, fees/admin/config, corpus Solscan/Demo3 puis decoder spécialisé. |
| 4 | `raydium_amm_v4` | `supported / 0.7.51 planned` | Swaps AMM v4 legacy matérialisés. | Reprendre AMM v4 au niveau CPMM/CLMM : pool lifecycle, liquidity, fees/admin, side effects, fallback cleanup. |
| 5 | `raydium_stable_swap` | `planned / 0.7.52` | Entrée conservée. | Reprendre Stable séparément : swaps stables, pool lifecycle, liquidity, fees/admin, montants/prix exploitables. |
| 6 | `pump_swap` | `supported / 0.7.53 planned` | `buy`/`sell` décodés et matérialisés ; trade/candle OK. | Ajouter tous les events Carbon/Solana Streamer : cashback, fee, volume accumulator, admin/config ; conserver les non-trades hors candles. |
| 7 | `pump_fun` | `partial / launch_surface` | Création/token launch partiellement décodée ; intégrée au pipeline de listings. | Traiter tous les events Pump.fun disponibles : buy/sell/migrate/create/update ; séparer bonding/launch de DEX effectif ; valider migration vers PumpSwap. |
| 8 | `meteora_dbc` | `partial` | Swaps/instruction audits observés ; Demo3 donne du corpus. | Couverture complète DBC : launch/bonding curve, swap, migration, config/admin, fees ; matérialiser seulement ce qui est prouvé. |
| 9 | `meteora_dlmm` | `supported` | Couverture avancée validée en `0.7.45` : swaps, liquidity, positions, lifecycle, fees ; non-trade matérialisé. | Résoudre les audits résiduels non mappés ; comparer Carbon/IDL pour events rewards/admin restants ; revalidation base neuve. |
| 10 | `meteora_damm_v1` | `supported / partial events` | Couverture `0.7.46` : swap, create_pool, add/remove liquidity, claim_fee, create_lock_escrow, lock_liquidity. | Vérifier les surfaces upstream non observées ; améliorer rattachement pool/pair pour remove_liquidity non matérialisés ; revalidation stricte. |
| 11 | `meteora_damm_v2` | `partial` | `swap`, `instruction_audit`, registry/discriminants et corpus Demo3 existent. | Couvrir tous les events Carbon/source : create pool, liquidity, fees, dynamic config, admin ; déterminer actionability des swaps ; matérialiser si montants fiables. |
| 12 | `phoenix_v1` | `audit-only` | Decoder local audit-only ; `log_audit`, order place/cancel, withdraw ; parsing strict `0x0f`; events `Reduce`, `Place`, `TimeInForce` observés ; `trade_count=0`. | Terminer tous les events Git : `Fill`, `FillSummary`, `Fee`, `Evict`, `ExpiredOrder`, etc. ; ajouter counts/flags audit ; seulement ensuite étudier trade materialization. |
| 13 | `openbook_v2` | `audit-only` | Decoder local audit-only ; instructions order/cancel/consume/settle ; `Program data` mappé : `FillLog`, `OpenOrdersPositionLog`, `TotalOrderFillEvent`, `SettleFundsLog`; `trade_count=0`. | Vérifier layouts fill/out et sens maker/taker/base/quote ; ajouter table audit éventuelle ; ne matérialiser trades qu’après validation du sens économique. |
| 14 | `orca_whirlpools` | `partial` | Premier decoder historique présent ; swaps/create_pool partiels. | Comparer Carbon/IDL complet ; couvrir liquidity, positions, fees/rewards, tick arrays ; valider swaps exploitables et non-trades. |
| 15 | `legacy_launch_candidates` | `planned launch` | Anciennes entrées launch à réévaluer après `raydium_launchpad`. | Ne pas confondre Launchpad, LaunchLab, CPMM/CLMM/AMM v4 ; rattacher aux pools tradables seulement après corpus. |
| 16 | `meteora_vault` | `to_verify` | Présent comme indice upstream / compte associé. | Corpus direct obligatoire ; decoder séparé si events vault réels ; aucune promotion via DAMM indirect. |
| 17 | `fluxbeam` | `partial/to_verify` | Decoder initial existant ; Demo3 peut produire des candidats. | Vérifier source/IDL ; compléter swap, pool, liquidity, fees/admin ; matérialisation uniquement après corpus. |
| 18 | `dexlab` | `partial/to_verify` | Decoder initial historique ; ancienne entrée beta supprimée. | Reconfirmer program id/source ; décoder events disponibles ; distinguer DexLab natif et liens OpenBook/market. |
| 19 | `lifinity_v2` | `to_verify` | Program id listé par sources externes/Vybe ; pas de corpus concluant. | Trouver IDL/source ; Demo3 par program/market ; audit-only d’abord. |
| 20 | `stabble_stable_swap` / `stabble_weighted_swap` | `to_verify` | Program ids/indices via sources externes ; candidats Demo3 observables. | Source/IDL + corpus + decoder audit-only ; déterminer surface AMM et montants exploitables. |
| 21 | `bonkswap` | `to_verify` | Program id/Carbon/Vybe selon registre ; swaps candidats possibles. | Vérifier program id, source et corpus ; décoder tous events ; pas de trade sans montants. |
| 22 | `boop` / `boop_fun` | `to_verify / launch` | Entrée de découverte. | Séparer launch surface et swap effectif ; corpus + source obligatoire. |
| 23 | `moonshot` / `moonit` | `to_verify / launch` | Moonshot buy/sell observés via upstream candidates ; Moonit launch attribution historique. | Source/IDL + migration + rattachement pools ; éviter heuristiques seules. |
| 24 | `heaven` | `to_verify` | Program id/candidat ajouté en matrice. | Vérifier s’il est launch, AMM ou les deux ; corpus dédié. |
| 25 | `printr` | `to_verify` | Preset Demo3 ajouté ; candidats observables. | Source/IDL, discriminants, corpus, decoder audit-only. |
| 26 | `metadao_*` | `to_verify` | Presets spécifiques : launchpad, bid wall, futarchy, AMM. | Traiter par programme séparé ; ne pas utiliser mint ids comme program ids ; corpus obligatoire. |
| 27 | `jupiter_*`, `dflow_aggregator_v4`, `okx_dex` | `aggregator_router` | Registry/discovery pour contexte transactionnel. | Ne pas matérialiser en DEX direct ; utiliser pour routeSource/routing/context. |
## Checklist obligatoire par DEX/version
@@ -223,3 +225,14 @@ Entrées CPMM couvertes localement depuis Carbon/fnzero/IDL :
`raydium_cpmm` est considéré `supported` et clôturable pour la tranche `0.7.48`. Les entrées matérialisées couvrent swaps (`swap_base_input`, `swap_base_output`), liquidity (`deposit`, `withdraw`, `lp_change_event`), lifecycle (`initialize`, `initialize_with_permission`), fees (`collect_creator_fee`, `collect_fund_fee`, `collect_protocol_fee`) et admin/config (`create_amm_config`, `create_permission_pda`, `update_amm_config`).
`swap_event` reste audit-only pour éviter tout doublon de trade/candle. `close_permission_pda` et `update_pool_status` restent connus upstream mais non observés localement.
## Note `0.7.49` — Raydium CLMM initial coverage sync
`raydium_clmm` reste `supported`, mais la tranche `0.7.49` rouvre sa couverture événementielle complète au lieu de se limiter aux swaps.
État local repris : `swap`, `swap_v2`, `increase_liquidity_v2`, `decrease_liquidity_v2`, `open_position_with_token22_nft` et `close_position` disposent déjà d'un chemin local spécialisé ou mappé. Les autres entrées CLMM restent à confirmer par corpus avant toute promotion.
Le registre est complété avec les entrées issues de l'IDL officiel Raydium non présentes dans le snapshot Carbon courant : `close_limit_order`, `close_protocol_position`, `create_customizable_pool`, `create_dynamic_fee_config`, `create_support_mint_associated` et `settle_limit_order`.
Règle de clôture : les positions CLMM, fees/rewards et surfaces limit-order ne doivent produire aucune ligne trade/candle tant que le sens économique, les montants, les comptes et les mints ne sont pas prouvés par replay local.

35
DEX_EVENT_COVERAGE_MATRIX.md → docs/DEX_EVENT_COVERAGE_MATRIX.md
View File

@@ -1,6 +1,6 @@
# DEX Event Coverage Matrix — `khadhroony-bobobot` `0.7.48`
# DEX Event Coverage Matrix — `khadhroony-bobobot` `0.7.49`
Cette matrice complète `DEX_DECODER_MATRIX.md` avec une lecture par familles d'événements. Elle ne remplace pas la preuve locale : une entrée Git/IDL reste un indice tant qu'elle n'est pas observée dans le corpus local puis validée par replay et SQL.
Cette matrice complète `docs/DEX_DECODER_MATRIX.md` avec une lecture par familles d'événements. Elle ne remplace pas la preuve locale : une entrée Git/IDL reste un indice tant qu'elle n'est pas observée dans le corpus local puis validée par replay et SQL.
## Règles de statut
@@ -69,3 +69,34 @@ La couverture `raydium_cpmm` est alignée avec les instructions exposées par le
Validation finale locale : `deposit` = `11/11` liquidity, `withdraw` = `14/14` liquidity, `lp_change_event` = `25/25` liquidity, fees = `26/26` fee, admin/config observés = `23/23` admin, lifecycle = `9/9`, `swap_event` = audit-only avec `0` trade, et trades matérialisés uniquement depuis `swap_base_input` / `swap_base_output`.
`close_permission_pda` et `update_pool_status` restent `upstream_git_mapped_unverified` faute de corpus local. Les filtres Solscan `instruction=9c5420764587467b` et `instruction=82576c062ee0757b` n'ont pas fourni de signatures exploitables dans l'horizon testé.
## `0.7.49` — `raydium_clmm` final coverage
Sources inventoriées : Carbon `raydium-clmm-decoder`, fnzero `sol-parser-sdk`, Pinax `substreams-solana-idls/src/raydium/clmm`, Raydium/Solscan Program IDL.
État final validé : `45` entrées listées, `33` instructions locales observées/décodées, `25` entrées matérialisées, `1186` trades, `raydium_clmm.instruction_audit = 0`, fallback `upstream_git.instruction_match` localement couvert = `0`.
| Famille | Entrées Raydium CLMM | Statut `0.7.49` | Cible DB | Justification / règle |
|---|---|---|---|---|
| `swap` | `swap`, `swap_v2` | `materialized` | `k_sol_trade_events` | Seuls ces swaps produisent trades/candles lorsque les montants sont exploitables. |
| `swap audit` | `swap_event`, `swap_router_base_in` | `observed/audit` ou `upstream_git_unverified` selon corpus | `k_sol_dex_decoded_events_only` | Pas de double trade/candle. |
| `pool_create` | `create_pool`, `create_customizable_pool` | `decoded`; `create_pool` matérialisé quand prouvé | `k_sol_pool_lifecycle_events` | Création de pool = lifecycle, pas admin. |
| `add_liquidity` | `increase_liquidity`, `increase_liquidity_v2`, open position variants | `decoded`; variantes prouvées matérialisées | `k_sol_liquidity_events` | CLMM implique position/tick/NFT ; matérialisation seulement sur corpus OK. |
| `remove_liquidity` | `decrease_liquidity`, `decrease_liquidity_v2`, close position variants | `decoded`; variantes prouvées matérialisées | `k_sol_liquidity_events` | Aucune promotion trade/candle. |
| `fee` | `collect_fund_fee`, `collect_protocol_fee` | `decoded`; protocol fee matérialisé si tx OK | `k_sol_fee_events` | Mapping fee sans trade/candle. |
| `reward` | `initialize_reward`, `collect_remaining_rewards`, `set_reward_params`, `transfer_reward_owner`, `update_reward_infos` | `decoded`; variantes prouvées matérialisées | `k_sol_reward_events` / `k_sol_pool_admin_events` | Reward/config selon instruction. |
| `admin/config` | `create_amm_config`, `create_dynamic_fee_config`, `create_operation_account`, `update_amm_config`, `update_operation_account`, `update_pool_status` | `decoded`; variantes prouvées matérialisées | `k_sol_pool_admin_events` ou decoded-only | Config/admin sans trade/candle. |
| `account_create` | `create_support_mint_associated` | `observed/decoded` | `k_sol_token_account_events` prévu / decoded-only selon corpus | Side effect technique ; pas de trade/candle. |
| `order_place` | `open_limit_order`, `increase_limit_order` | `decoded/materialized` | `k_sol_orderbook_events` | Orderbook CLMM, jamais trade/candle. |
| `order_cancel` | `decrease_limit_order`, `close_limit_order` | `decoded/materialized` | `k_sol_orderbook_events` | Orderbook CLMM, jamais trade/candle. |
| `settle_funds` | `settle_limit_order` | `decoded/materialized` | `k_sol_orderbook_events` | Settlement orderbook, pas trade direct. |
| `mint/burn/transfer/account_close/wrap/unwrap` | Side effects SPL Token / Token-2022 | `indirect` | decoded-only | Pas de promotion en `raydium_clmm.*` sans instruction directe. |
| `launch/migration/lock/unlock/stake/unstake/vault` | `-` | `not_applicable` | `-` | Surfaces hors CLMM direct. |
| `unknown/unmapped audit` | `raydium_clmm.instruction_audit`, fallback upstream | `closed` | `k_sol_dex_decoded_events_only` | Résidu validé à zéro après replay. |
Les 11 Anchor / `Program data` events restent `upstream_git_unverified` et préparés audit-only faute d’observation locale : `collect_personal_fee_event`, `collect_protocol_fee_event`, `config_change_event`, `create_personal_position_event`, `decrease_liquidity_event`, `increase_liquidity_event`, `liquidity_calculate_event`, `liquidity_change_event`, `pool_created_event`, `swap_event`, `update_reward_infos_event`.
## `0.7.50` — `raydium_launchpad` planned
Prochaine tranche : identifier program ids/IDL, lister instructions/events/discriminants, constituer corpus Demo3/Solscan/Demo2, puis appliquer les mêmes règles de coverage et matérialisation que CPMM/CLMM.

2
NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md → docs/prompts/NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md
View File

@@ -16,7 +16,7 @@ Joindre aussi les docs mises à jour :
README.md
ROADMAP.md
CHANGELOG.md
DEX_DECODER_MATRIX.md
docs/DEX_DECODER_MATRIX.md
```
## Décision de planification

6
NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md → docs/prompts/NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md
View File

@@ -16,9 +16,9 @@ Et les docs :
README.md
ROADMAP.md
CHANGELOG.md
DEX_DECODER_MATRIX.md
DEX_EVENT_COVERAGE_MATRIX.md
DB_EVENT_MODEL_REVIEW.md
docs/DEX_DECODER_MATRIX.md
docs/DEX_EVENT_COVERAGE_MATRIX.md
docs/DB_EVENT_MODEL_REVIEW.md
```
## Décision de reprise

0
NEXT_SESSION_PROMPT_0.7.47_UPSTREAM_REGISTRY.md → docs/prompts/NEXT_SESSION_PROMPT_0.7.47_UPSTREAM_REGISTRY.md
View File

10
NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md → docs/prompts/NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md
View File

@@ -16,11 +16,11 @@ Docs à fournir aussi :
README.md
ROADMAP.md
CHANGELOG.md
DEX_DECODER_MATRIX.md
DEX_EVENT_COVERAGE_MATRIX.md
DB_EVENT_MODEL_REVIEW.md
RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md
SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql
docs/DEX_DECODER_MATRIX.md
docs/DEX_EVENT_COVERAGE_MATRIX.md
docs/DB_EVENT_MODEL_REVIEW.md
docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md
validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql
```
## État validé avant reprise

245
docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.48-raydium-cpmm.md Normal file
View File

@@ -0,0 +1,245 @@
# Prompt de reprise — khadhroony-bobobot `0.7.48` / Raydium CPMM event coverage
Reprise du projet `khadhroony-bobobot` après clôture de `0.7.48-pre`.
## Archive de départ
Utiliser la dernière archive complète du workspace intégrant les deltas validés jusqu'à :
```text
0.7.48-pre-event-coverage-report
```
Docs à fournir aussi :
```text
README.md
ROADMAP.md
CHANGELOG.md
docs/DEX_DECODER_MATRIX.md
docs/DEX_EVENT_COVERAGE_MATRIX.md
docs/DB_EVENT_MODEL_REVIEW.md
```
## État validé avant reprise
`0.7.48-pre` a ajouté et clôturé le checkpoint DB/reporting de couverture événementielle :
```text
k_sol_dex_event_coverage_entries
DexEventCoverageService
sync upstream registry -> coverage table
refresh local counts depuis k_sol_dex_decoded_events + tables métier existantes
summaries coverage dans LocalPipelineDiagnosticSummaryDto
summaries/counters coverage dans LocalPipelineValidationReportDto
profil validation 0.7.48-pre_event_coverage_db_checkpoint
profil exposé dans Demo Pipeline 2
```
Invariants maintenus :
```text
aucun decoder DEX modifié
aucun trade/candle créé par la couverture
aucun program_id promu sans corpus local
upstream Git/IDL = indice, pas preuve métier
failed transaction = audit-only
non-trade event = jamais trade/candle
```
## Décision de reprise
Commencer maintenant par Raydium avant Meteora.
Ordre courant :
```text
0.7.48 raydium_cpmm
0.7.49 raydium_clmm
0.7.50 pump_swap
0.7.51 pump_fun
0.7.52 meteora_dbc
0.7.53 meteora_dlmm upstream parity
0.7.54 meteora_damm_v1 upstream parity
0.7.55 meteora_damm_v2
0.7.56 phoenix_v1 audit-only completion
0.7.57 openbook_v2 audit-only completion
0.7.58 orca_whirlpools
0.7.59+ launch surfaces, candidats/historiques, validation consolidée
```
## Sources Git/IDL à utiliser systématiquement
- https://github.com/sevenlabs-hq/carbon/tree/main/decoders
- https://github.com/0xfnzero/solana-streamer
- https://github.com/0xfnzero/sol-parser-sdk/tree/main/idl
- https://github.com/pinax-network/substreams-solana-idls/tree/main/src
- https://github.com/hodlwarden/solana-tx-parser/tree/main/src
- https://github.com/openbook-dex/openbook-v2
- https://github.com/all-in-one-blockchain/phoenix-onchain-mm
- https://docs.vybenetwork.com/docs/available-dexs-amms
Pour `0.7.48`, commencer par Carbon + fnzero pour Raydium CPMM, puis comparer aux IDL complémentaires si disponibles.
## Objectif `0.7.48` — `raydium_cpmm`
Objectif : reprendre `raydium_cpmm` comme première tranche DEX/version après le checkpoint coverage.
À faire :
1. lister tous les discriminants/instructions/events `raydium_cpmm` depuis Carbon/fnzero/IDL ;
2. synchroniser/remplir `k_sol_dex_event_coverage_entries` pour `raydium-cpmm` ;
3. comparer listed/decoded/observed/materialized/trade_count via le rapport coverage ;
4. compléter le decoder spécialisé `raydium_cpmm` seulement pour les events CPMM confirmables ;
5. remplacer/nettoyer le fallback `upstream_git.instruction_match` quand un decoder local spécialisé couvre l'entrée ;
6. garder les events connus mais non observés en `upstream_git_mapped_unverified` ;
7. garder les events observés mais non matérialisés en audit-only/decoded ;
8. ne matérialiser que les non-trades déjà prouvés par corpus et compatibles avec les tables existantes ;
9. ne pas ajouter encore `k_sol_token_transfer_events` ou `k_sol_orderbook_events`, sauf besoin bloquant démontré ;
10. ne pas modifier les règles trade/candle sauf bug de faux positif prouvé.
## Events/familles à couvrir explicitement
Ne pas se limiter aux swaps.
Inclure dans l'audit coverage :
```text
swap
pool_create
add_liquidity
remove_liquidity
position_open
position_close
fee
reward
admin/config
mint
burn
transfer
account_create
account_close
wrap_sol
unwrap_sol
order_place
order_cancel
order_fill
consume_events
settle_funds
vault_deposit
vault_withdraw
lock
unlock
launch
migration
stake
unstake
unknown/unmapped audit
```
Pour `raydium_cpmm`, certaines familles seront probablement `-` ou `non applicable`, mais elles doivent être explicitement justifiées dans la coverage matrix ou la table.
## Règles fixes
- Un event non-trade ne produit jamais `trade_event`, metric ou candle.
- Une transaction failed reste audit, jamais trade/candle.
- Un discriminator upstream n'est pas une preuve métier.
- Un program id upstream n'est pas vérifié sans corpus local.
- Chaque decoder spécialisé doit remplacer le fallback `upstream_git.instruction_match` pour éviter les doublons.
- Tout event connu mais non observé reste `upstream_git_mapped_unverified`.
- Tout event observé mais non matérialisé reste audit-only ou decoded, pas materialized.
- Ne pas promouvoir de nouvelle table DB métier sans preuve que plusieurs DEX en auront besoin.
## Requêtes SQL utiles
Après sync/refresh coverage :
```sql
SELECT *
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium-cpmm'
ORDER BY entry_kind, entry_name, discriminator_hex;
```
```sql
SELECT
decoder_code,
listed_entry_count,
decoded_entry_count,
observed_entry_count,
materialized_entry_count,
total_observed_count,
total_materialized_count,
trade_count,
audit_only_entry_count,
upstream_git_mapped_unverified_entry_count,
upstream_git_local_corpus_observed_entry_count,
upstream_git_local_corpus_materialized_entry_count
FROM (
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count,
SUM(CASE WHEN expected_db_target = 'k_sol_dex_decoded_events_only' THEN 1 ELSE 0 END) AS audit_only_entry_count,
SUM(CASE WHEN proof_status = 'upstream_git_mapped_unverified' THEN 1 ELSE 0 END) AS upstream_git_mapped_unverified_entry_count,
SUM(CASE WHEN proof_status = 'upstream_git_local_corpus_observed' THEN 1 ELSE 0 END) AS upstream_git_local_corpus_observed_entry_count,
SUM(CASE WHEN proof_status = 'upstream_git_local_corpus_materialized' THEN 1 ELSE 0 END) AS upstream_git_local_corpus_materialized_entry_count
FROM k_sol_dex_event_coverage_entries
GROUP BY decoder_code
)
WHERE decoder_code = 'raydium-cpmm';
```
Audit-only safety :
```sql
SELECT
de.protocol_name,
de.event_kind,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_cpmm'
AND (
de.event_kind LIKE '%audit%'
OR json_extract(de.payload_json, '$.eventActionability') IN ('non_trade_useful', 'informational', 'non_actionable_trade')
)
GROUP BY de.protocol_name, de.event_kind
ORDER BY trade_count DESC, de.event_kind;
```
## Contraintes de code
Conserver les règles du workspace :
```text
Rust 2024
pas de mod.rs
fichiers Rust avec // file: ...
pas de anyhow
pas de thiserror
pas de ? / unwrap / expect dans kb_lib applicatif
match / if let Err / let Err = ... else
rustdoc sur API publique
re-exports db.rs puis lib.rs si DB modifiée
```
## Livrables attendus pour `0.7.48`
1. delta archive avec uniquement les fichiers ajoutés/modifiés ;
2. mise à jour `README.md`, `ROADMAP.md`, `CHANGELOG.md` si la tranche avance ;
3. rapport de couverture `raydium_cpmm` ;
4. SQL de validation ;
5. tests verts :
```bash
cargo fmt
cargo test -p kb_lib
cargo clippy -p kb_lib --all-targets -- -D warnings
```

338
docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md Normal file
View File

@@ -0,0 +1,338 @@
<!-- file: docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md -->
# khadhroony-bobobot `0.7.50` / Raydium Launchpad event coverage
Reprise du projet `khadhroony-bobobot` après clôture fonctionnelle de `0.7.49 raydium_clmm`.
## Archive de départ
Utiliser la dernière archive complète du workspace intégrant les deltas validés jusqu'à :
```text
0.7.49-raydium-clmm-final
```
Inclure les docs et SQL de validation produits en fin de `0.7.49`, notamment :
```text
README.md
ROADMAP.md
CHANGELOG.md
docs/DEX_DECODER_MATRIX.md
docs/DEX_EVENT_COVERAGE_MATRIX.md
docs/DB_EVENT_MODEL_REVIEW.md
docs/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md
docs/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md
validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql
```
## État validé avant reprise
`0.7.48` a clôturé `raydium_cpmm`.
`0.7.49` a clôturé `raydium_clmm` avec les invariants suivants :
```text
Raydium CLMM decoder_code local = raydium_clmm
coverage synchronisée avec Carbon / Raydium IDL / Pinax / fnzero
45 entrées coverage CLMM listées
33 entrées CLMM décodées localement
33 entrées observées localement
25 entrées matérialisées localement
residual raydium_clmm.instruction_audit = 0
fallback upstream_git.instruction_match localement couvert = 0 après cleanup FK-safe
non-trade CLMM = jamais trade/candle
failed transaction = jamais matérialisée dans les tables métier
Program data CLMM préparé mais non promu sans corpus local observé
side effects SPL Token / Token-2022 restent transversaux, pas raydium_clmm.* directs
```
Dernière validation locale observée :
```text
cargo test -p kb_lib: ok
local pipeline replay: 2197 replayed, 0 decode skipped, 2197 ledger upserts, 1461 unsafe ledger rows, 1217 trades, 111 liquidity, 25 lifecycle, 4868 candle upserts, instructionObservations='19798'
```
## Décision de reprise
Commencer `0.7.50` par `raydium_launchpad`, avant Pump/Meteora.
Ordre strict Raydium proposé avant Pump :
```text
0.7.50 raydium_launchpad / Raydium LaunchLab-Launchpad surface
0.7.51 raydium_amm_v4
0.7.52 raydium_stable
0.7.53 raydium_pool_v4 audit / program-id decision, seulement si program id distinct et corpus exploitable
0.7.54 pump_swap
0.7.55 pump_fun
0.7.56 meteora_dbc
0.7.57 meteora_dlmm upstream parity
0.7.58 meteora_damm_v1 upstream parity
0.7.59 meteora_damm_v2
0.7.60 phoenix_v1 audit-only completion
0.7.61 openbook_v2 audit-only completion
0.7.62 orca_whirlpools
0.7.63+ launch surfaces, candidats/historiques, validation consolidée
```
`raydium_pool_v4` ne doit pas être promu automatiquement comme DEX/surface métier tant que son program id et son rôle exact n'ont pas été confirmés. Le fichier `raydium_pool_v4.json` de `sol-parser-sdk` doit être audité comme source IDL annexe, pas comme preuve métier.
## Sources Git/IDL à utiliser systématiquement
- https://github.com/sevenlabs-hq/carbon/tree/main/decoders
- https://github.com/0xfnzero/solana-streamer
- https://github.com/0xfnzero/sol-parser-sdk/tree/main/idl
- copie locale fournie de `0xfnzero/sol-parser-sdk`, si présente dans l'archive de reprise
- https://github.com/pinax-network/substreams-solana-idls/tree/main/src
- https://github.com/hodlwarden/solana-tx-parser/tree/main/src
- https://github.com/openbook-dex/openbook-v2
- https://github.com/all-in-one-blockchain/phoenix-onchain-mm
- https://docs.vybenetwork.com/docs/available-dexs-amms
Pour `0.7.50 raydium_launchpad`, utiliser aussi explicitement :
```text
https://solscan.io/account/LanMV9sAd7wArD4vJFi2qDdfnVhFxYSUg6eADduJ3uj#programIdl
```
et les filtres Solscan de type :
```text
https://solscan.io/account/LanMV9sAd7wArD4vJFi2qDdfnVhFxYSUg6eADduJ3uj?instruction=<DISCRIMINATOR>&hide_spam=true&hide_failed=true&show_related=false&sort=desc
```
Solscan sert à trouver vite des signatures à backfiller. Solscan ne doit jamais être considéré comme preuve métier finale sans corpus local et validation SQL.
## Base de données de travail
Initialiser une nouvelle base SQLite dédiée à `0.7.50`.
Procédure attendue :
1. créer une nouvelle DB via `config.json` ou équivalent ;
2. démarrer `kb_demo_app` pour initialiser le schéma ;
3. backfiller des signatures ciblées par discriminant / instruction ;
4. backfiller des pools/comptes pertinents si la surface en expose ;
5. exécuter un replay local avec `forceDexDecode=yes` ;
6. relancer les SQL de coverage ;
7. ne promouvoir une entrée que si le corpus local confirme son sens métier.
## Objectif `0.7.50` — `raydium_launchpad`
Objectif : couvrir Raydium Launchpad/LaunchLab comme nouvelle surface Raydium après CPMM et CLMM.
À faire :
1. lire le code local existant lié à Raydium Launchpad, s'il existe ;
2. lister toutes les instructions/events depuis Carbon/fnzero/IDL/Pinax/Solscan Program IDL ;
3. synchroniser/remplir `k_sol_dex_event_coverage_entries` pour `raydium_launchpad` ;
4. vérifier `decoder_code` local en snake_case : `raydium_launchpad` ;
5. utiliser `k_sol_instruction_observations` pour inspecter les discriminants observés localement ;
6. utiliser Demo3 / Solscan `instruction=<discriminator>` pour trouver des signatures ciblées ;
7. backfiller les signatures via Demo2, idéalement par batch textarea ;
8. rejouer localement avec `forceDexDecode=yes` et `deferInstructionObservations=yes` ;
9. comparer listed/decoded/observed/materialized/trade_count via SQL coverage ;
10. compléter le decoder spécialisé seulement pour les entrées confirmées ;
11. supprimer/nettoyer `upstream_git.instruction_match` lorsque l'entrée est couverte localement ;
12. garder les entrées connues mais non observées en `upstream_git_unverified` ou `upstream_git_mapped_unverified` ;
13. garder les entrées observées mais non matérialisées en audit-only/decoded ;
14. ne matérialiser que les non-trades prouvés par corpus local et compatibles avec les tables existantes ;
15. ne modifier les règles trade/candle que si un faux positif est prouvé.
## Familles à auditer explicitement
Ne pas se limiter aux swaps.
Inclure dans la coverage :
```text
swap
pool_create
add_liquidity
remove_liquidity
position_open
position_close
fee
reward
admin/config
mint
burn
transfer
account_create
account_close
wrap_sol
unwrap_sol
order_place
order_cancel
order_fill
consume_events
settle_funds
vault_deposit
vault_withdraw
lock
unlock
launch
migration
stake
unstake
unknown/unmapped audit
```
Certaines familles peuvent être non applicables ou uniquement visibles comme side effects SPL Token / Token-2022. Elles doivent être explicitement justifiées dans `DEX_EVENT_COVERAGE_MATRIX.md`.
## Points d'attention hérités de CPMM/CLMM
- `decoder_code` local doit rester en `snake_case`.
- Les slugs upstream peuvent garder les tirets.
- `upstream Git/IDL/Solscan = indice, pas preuve métier`.
- `program_id` upstream non promu sans corpus local.
- Chaque decoder spécialisé doit remplacer le fallback `upstream_git.instruction_match` pour les entrées localement couvertes.
- Les side effects SPL Token (`mintTo`, `burn`, `transfer`, `transferChecked`, `closeAccount`) ne deviennent pas `raydium_launchpad.*` sans preuve qu'il s'agit d'instructions directes du programme Launchpad.
- Failed transaction = decoded/audit possible, jamais matérialisée métier.
- Non-trade event = jamais trade/candle.
- Pas de nouvelle table métier transversale sans preuve multi-DEX.
## Requêtes SQL minimales à produire
Créer un fichier :
```text
validation_sql/SQL_VALIDATION_RAYDIUM_LAUNCHPAD_0_7_50.sql
```
Inclure au minimum :
```sql
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_launchpad'
ORDER BY entry_kind, entry_name, discriminator_hex;
```
Coverage summary :
```sql
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_launchpad'
GROUP BY decoder_code;
```
Instruction observations :
```sql
SELECT
instruction_name,
discriminator_hex,
COUNT(*) AS observed_count,
COUNT(DISTINCT signature) AS tx_count
FROM k_sol_instruction_observations
WHERE decoder_code = 'raydium_launchpad'
GROUP BY instruction_name, discriminator_hex
ORDER BY observed_count DESC, instruction_name;
```
```sql
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_launchpad'
GROUP BY de.event_kind
ORDER BY decoded_count DESC, de.event_kind;
```
```sql
SELECT
json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
COUNT(*) AS fallback_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'upstream_git'
AND event_kind = 'upstream_git.instruction_match'
AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_launchpad'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex
ORDER BY fallback_count DESC, entry_name;
```
```sql
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_launchpad'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
ORDER BY trade_count DESC, decoded_count DESC;
```
## Contraintes de code
Conserver les règles du workspace :
```text
Rust 2024
pas de mod.rs
fichiers Rust avec // file: ...
pas de anyhow
pas de thiserror
pas de ? / unwrap / expect dans kb_lib applicatif
match / if let Err / let Err = ... else
rustdoc sur API publique
re-exports db.rs puis lib.rs si DB modifiée
```
## Livrables attendus pour `0.7.50`
1. delta archive avec uniquement les fichiers ajoutés/modifiés ;
2. mise à jour `README.md`, `ROADMAP.md`, `CHANGELOG.md` ;
3. rapport `docs/RAYDIUM_LAUNCHPAD_EVENT_COVERAGE_REPORT.md` ;
4. SQL `validation_sql/SQL_VALIDATION_RAYDIUM_LAUNCHPAD_0_7_50.sql` ;
5. tests verts :
```bash
cargo fmt
cargo test -p kb_lib
cargo clippy -p kb_lib --all-targets -- -D warnings
```
## Question ouverte à traiter tôt dans `0.7.50`
Auditer `raydium_pool_v4.json` dans `sol-parser-sdk` :
- confirmer s'il correspond à un program id distinct ;
- confirmer s'il s'agit d'une surface Raydium Pool/Lending/Strategy différente de `raydium_amm_v4` ;
- décider si une version dédiée `0.7.53 raydium_pool_v4` est nécessaire ;
- ne pas modifier la roadmap comme surface finale tant que le program id n'est pas confirmé.

134
docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md Normal file
View File

@@ -0,0 +1,134 @@
<!-- file: docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md -->
# Rapport `0.7.49` — Raydium CLMM event coverage
## Résumé
La tranche `0.7.49` clôture la couverture fonctionnelle `raydium_clmm` sur le corpus local validé. Elle reprend CLMM après `0.7.48 raydium_cpmm` et applique la même méthode : inventaire upstream, corpus local, replay forcé, materialization contrôlée, suppression des fallbacks remplacés, validation SQL.
Validation locale observée après le dernier replay :
```text
local replay: 2197 replayed, 0 decode skipped, 2197 ledger upserts, 1461 unsafe ledger rows, 1217 trades, 111 liquidity, 25 lifecycle, 4868 candle upserts, instructionObservations=19798
catalog: 41 tokens, 63 pools, 63 pairs
```
Synthèse coverage :
```text
listed_entry_count = 45
decoded_entry_count = 33
observed_entry_count = 33
materialized_entry_count = 25
total_observed_count = 2560
total_materialized_count = 1367
trade_count = 1186
```
## Entrées couvertes
La tranche couvre `33` instructions locales CLMM observées et décodées, dont :
```text
swap
swap_v2
swap_router_base_in
open_position
open_position_v2
open_position_with_token22_nft
close_position
close_protocol_position
increase_liquidity
increase_liquidity_v2
decrease_liquidity
decrease_liquidity_v2
create_pool
create_customizable_pool
create_amm_config
create_dynamic_fee_config
update_amm_config
update_pool_status
create_operation_account
update_operation_account
create_support_mint_associated
collect_fund_fee
collect_protocol_fee
collect_remaining_rewards
initialize_reward
set_reward_params
transfer_reward_owner
update_reward_infos
open_limit_order
increase_limit_order
decrease_limit_order
close_limit_order
settle_limit_order
```
## Matérialisations validées
Les matérialisations sont limitées aux transactions réussies et aux familles déjà supportées par le modèle DB :
| Famille | Table | Règle |
|---|---|---|
| Swaps | `k_sol_trade_events` + candles | Uniquement `raydium_clmm.swap` / `raydium_clmm.swap_v2` quand les montants sont exploitables. |
| Liquidity / positions | `k_sol_liquidity_events` | Positions/liquidity prouvées par corpus, sans trade/candle. |
| Fees | `k_sol_fee_events` | Fees prouvées par corpus, sans trade/candle. |
| Rewards | `k_sol_reward_events` | Rewards prouvées par corpus, sans trade/candle. |
| Admin/config | `k_sol_pool_admin_events` | Config/admin prouvés par corpus, sans trade/candle. |
| Lifecycle | `k_sol_pool_lifecycle_events` | Pool creation prouvée par corpus, sans trade/candle. |
| Limit orders | `k_sol_orderbook_events` | Open/increase/decrease/close/settle matérialisés comme orderbook, jamais trade/candle. |
## Invariants validés
Les requêtes SQL finales valident :
```text
raydium_clmm.instruction_audit résiduel = 0
upstream_git.instruction_match localement couvert = 0
non-swap CLMM avec trade_count > 0 = 0
failed tx matérialisées = 0
```
Les fallbacks `upstream_git.instruction_match` localement couverts sont supprimés automatiquement, y compris quand `k_sol_instruction_observations.decoded_event_id` pointait encore vers une ligne fallback.
## Anchor / Program-data events non observés
Les `11` Anchor / `Program data` events CLMM ci-dessous restent listés en `upstream_git_unverified`, car aucun corpus local ne les observe encore comme event direct :
```text
collect_personal_fee_event a6ae69c051a15369
collect_protocol_fee_event ce57114f2d29d53d
config_change_event f7bd07776a705f97
create_personal_position_event 641e57f9c4df9ace
decrease_liquidity_event 3ade563a44325538
increase_liquidity_event 314f69d420221e54
liquidity_calculate_event ed7094e63954b4a2
liquidity_change_event 7ef0afce9e58996b
pool_created_event 195e4b2f7063353f
swap_event 40c6cde8260871e2
update_reward_infos_event 6d7fba4e724125ec
```
Le code est préparé pour les accueillir comme audit-only lorsqu’ils seront observés dans un corpus local. Ils ne produisent pas de trade/candle par défaut.
## Sources utilisées
Les entrées ont été comparées aux sources Raydium CLMM suivantes :
```text
Solscan Program IDL CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK
sevenlabs-hq/carbon raydium-clmm-decoder
pinax-network/substreams-solana-idls raydium/clmm
0xfnzero/sol-parser-sdk idl
```
Les sources upstream restent des indices. La promotion locale dépend du corpus local, du replay et des validations SQL.
## SQL final
La validation finale est dans :
```text
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql
```

130
docs/reports/RAYDIUM_CLMM_UPSTREAM_COVERAGE_REVIEW_PRE19.md Normal file
View File

@@ -0,0 +1,130 @@
# file: docs/reports/RAYDIUM_CLMM_UPSTREAM_COVERAGE_REVIEW_PRE19.md
# Raydium CLMM upstream coverage review — `0.7.49-pre.19`
## Scope
Decoder under review:
```text
raydium_clmm
program_id = CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK
```
External sources checked for the local coverage registry:
```text
https://solscan.io/account/CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK#programIdl
https://github.com/sevenlabs-hq/carbon/tree/main/decoders/raydium-clmm-decoder
https://github.com/pinax-network/substreams-solana-idls/tree/main/src/raydium/clmm
https://github.com/0xfnzero/sol-parser-sdk/tree/main/idl
```
Notes:
- Solscan is used as discovery / IDL cross-check, not as final business proof.
- `sol-parser-sdk/idl/raydium_clmm.json` maps to `raydium/amm_v3_with_swapv2.json` and exposes the CLMM program address.
- `pinax-network/substreams-solana-idls` exposes Raydium CLMM under `src/raydium/clmm/v3`.
- `sevenlabs-hq/carbon` exposes a dedicated `raydium-clmm-decoder` crate.
## Local registry status
The local registry now lists:
```text
45 entries total:
- 33 instructions
- 11 Anchor/program-data events
- 1 program row
```
All 33 instruction entries have a `local_event_kind` mapping in `dex_event_coverage.rs`, `instruction_observation_index.rs`, `dex_decode.rs`, and `upstream_registry_generated.rs`.
The 11 event entries are listed as upstream facts and stay `upstream_git_unverified` until local corpus provides program-data/Anchor event proof.
## Instruction coverage matrix
| Entry | Discriminator | Family | Local event kind | Expected target | Status |
|---|---|---|---|---|---|
| close_limit_order | `4c7c800fd55725fa` | order_cancel | `raydium_clmm.close_limit_order` | `k_sol_orderbook_events` | decoded/materializable |
| open_limit_order | `9d20dab7471d1293` | order_place | `raydium_clmm.open_limit_order` | `k_sol_orderbook_events` | decoded/materializable |
| increase_limit_order | `b19059ecfaba7d63` | order_place | `raydium_clmm.increase_limit_order` | `k_sol_orderbook_events` | decoded/materializable |
| decrease_limit_order | `759d3c674231a300` | order_cancel | `raydium_clmm.decrease_limit_order` | `k_sol_orderbook_events` | decoded/materializable |
| close_position | `7b86510031446262` | position_close | `raydium_clmm.close_position` | `k_sol_liquidity_events` | decoded/materializable |
| close_protocol_position | `c975989055556cb2` | position_close | `raydium_clmm.close_protocol_position` | `k_sol_liquidity_events` | decoded/materializable |
| collect_fund_fee | `a78a4e95dfc2067e` | fee | `raydium_clmm.collect_fund_fee` | `k_sol_fee_events` | decoded/materializable |
| collect_protocol_fee | `8888fcddc2427e59` | fee | `raydium_clmm.collect_protocol_fee` | `k_sol_fee_events` | decoded/materializable |
| collect_remaining_rewards | `12eda6c52210d590` | reward | `raydium_clmm.collect_remaining_rewards` | `k_sol_reward_events` | decoded/materializable |
| create_amm_config | `8934edd4d7756c68` | admin_config | `raydium_clmm.create_amm_config` | `k_sol_pool_admin_events` | decoded/materializable |
| create_customizable_pool | `2b44d4a7592fa401` | pool_create | `raydium_clmm.create_customizable_pool` | `k_sol_pool_lifecycle_events` | decoded/materializable |
| create_dynamic_fee_config | `bd0eb5785576e33e` | admin_config | `raydium_clmm.create_dynamic_fee_config` | `k_sol_pool_admin_events` | decoded/materializable |
| create_operation_account | `3f5794216d230868` | unknown | `raydium_clmm.create_operation_account` | `k_sol_dex_decoded_events_only` | decoded/audit-only unless admin evidence is required |
| create_pool | `e992d18ecf6840bc` | pool_create | `raydium_clmm.create_pool` | `k_sol_pool_lifecycle_events` | decoded/materializable |
| create_support_mint_associated | `11fb415c88f20ea9` | account_create | `raydium_clmm.create_support_mint_associated` | `k_sol_token_account_events` | decoded; token-account materialization remains a later cross-DEX topic |
| decrease_liquidity | `a026d06f685b2c01` | liquidity_remove | `raydium_clmm.decrease_liquidity` | `k_sol_liquidity_events` | decoded/materializable |
| decrease_liquidity_v2 | `3a7fbc3e4f52c460` | liquidity_remove | `raydium_clmm.decrease_liquidity_v2` | `k_sol_liquidity_events` | decoded/materializable |
| increase_liquidity | `2e9cf3760dcdfbb2` | liquidity_add | `raydium_clmm.increase_liquidity` | `k_sol_liquidity_events` | decoded/materializable |
| increase_liquidity_v2 | `851d59df45eeb00a` | liquidity_add | `raydium_clmm.increase_liquidity_v2` | `k_sol_liquidity_events` | decoded/materializable |
| initialize_reward | `5f87c0c4f281e644` | reward | `raydium_clmm.initialize_reward` | `k_sol_reward_events` | decoded/materializable |
| open_position | `87802f4d0f98f031` | position_open | `raydium_clmm.open_position` | `k_sol_liquidity_events` | decoded/materializable |
| open_position_v2 | `4db84ad67056f1c7` | position_open | `raydium_clmm.open_position_v2` | `k_sol_liquidity_events` | decoded/materializable |
| open_position_with_token22_nft | `4dffae527d1dc92e` | position_open | `raydium_clmm.open_position_with_token22_nft` | `k_sol_liquidity_events` | decoded/materializable |
| set_reward_params | `7034a74b20c9d389` | reward | `raydium_clmm.set_reward_params` | `k_sol_reward_events` | decoded/materializable |
| settle_limit_order | `cd4e74215c691a60` | settle_funds | `raydium_clmm.settle_limit_order` | `k_sol_orderbook_events` | decoded/materializable |
| swap | `f8c69e91e17587c8` | swap | `raydium_clmm.swap` | `k_sol_trade_events` | decoded/materializable as trade |
| swap_router_base_in | `457d73daf5baf2c4` | swap | `raydium_clmm.swap_router_base_in` | `k_sol_trade_events` | decoded; observed but not promoted to trade without corpus proof |
| swap_v2 | `2b04ed0b1ac91e62` | swap | `raydium_clmm.swap_v2` | `k_sol_trade_events` | decoded/materializable as trade |
| transfer_reward_owner | `07160c53f22b3079` | reward | `raydium_clmm.transfer_reward_owner` | `k_sol_reward_events` | decoded/materializable |
| update_amm_config | `313cae889a1c74c8` | admin_config | `raydium_clmm.update_amm_config` | `k_sol_pool_admin_events` | decoded/materializable |
| update_operation_account | `7f467728bce33d07` | unknown | `raydium_clmm.update_operation_account` | `k_sol_dex_decoded_events_only` | decoded/audit-only unless admin evidence is required |
| update_pool_status | `82576c062ee0757b` | admin_config | `raydium_clmm.update_pool_status` | `k_sol_pool_admin_events` | decoded/materializable |
| update_reward_infos | `a3ace0340b9a6adf` | reward | `raydium_clmm.update_reward_infos` | `k_sol_reward_events` | decoded/materializable |
## Anchor/program-data event entries
These entries remain listed, but no local corpus row has observed them as direct CLMM decoded events in the current replay corpus:
| Event entry | Discriminator | Family | Expected target | Status |
|---|---|---|---|---|
| collect_personal_fee_event | `a6ae69c051a15369` | fee | `k_sol_fee_events` | upstream listed, local corpus unobserved |
| collect_protocol_fee_event | `ce57114f2d29d53d` | fee | `k_sol_fee_events` | upstream listed, local corpus unobserved |
| config_change_event | `f7bd07776a705f97` | admin_config | `k_sol_pool_admin_events` | upstream listed, local corpus unobserved |
| create_personal_position_event | `641e57f9c4df9ace` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved |
| decrease_liquidity_event | `3ade563a44325538` | liquidity_remove | `k_sol_liquidity_events` | upstream listed, local corpus unobserved |
| increase_liquidity_event | `314f69d420221e54` | liquidity_add | `k_sol_liquidity_events` | upstream listed, local corpus unobserved |
| liquidity_calculate_event | `ed7094e63954b4a2` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved |
| liquidity_change_event | `7ef0afce9e58996b` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved |
| pool_created_event | `195e4b2f7063353f` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved |
| swap_event | `40c6cde8260871e2` | swap | `k_sol_trade_events` | upstream listed, local corpus unobserved |
| update_reward_infos_event | `6d7fba4e724125ec` | reward | `k_sol_reward_events` | upstream listed, local corpus unobserved |
## Patch `pre.19`
Patch goal:
```text
Stop replay/backfill from leaving `upstream_git.instruction_match` rows when a local specialized decoder already covers the same upstream decoder + entry + discriminator.
```
New DB query:
```text
query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(database, upstream_decoder_code)
```
Called from:
```text
local_pipeline_replay.rs::refresh_event_coverage_best_effort
token_backfill.rs::refresh_event_coverage_best_effort
```
Expected validation after replay:
```text
raydium_clmm.instruction_audit residual query -> empty
upstream_git.instruction_match where upstreamDecoderCode = raydium_clmm -> empty
non-swap CLMM trade_count -> empty
failed CLMM materialization query -> empty
coverage summary remains populated around 45 listed / 33 decoded / 33 observed
```

0
RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md → docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md
View File

72
docs/reports/RAYDIUM_CPMM_UPSTREAM_COVERAGE_REVIEW_PRE22.md Normal file
View File

@@ -0,0 +1,72 @@
# Raydium CPMM upstream coverage review — 0.7.49-pre.22
## Scope
Compared local `raydium_cpmm` coverage against the currently referenced upstream surfaces:
- Solscan program IDL for `CPMMoo8L3F4NbTegBCKVNunggL7H1ZpdTHKxQB5qKP1C`.
- `sevenlabs-hq/carbon` `raydium-cpmm-decoder`.
- `0xfnzero/sol-parser-sdk` `idl/raydium_cpmm.json`.
- `pinax-network/substreams-solana-idls` `src/raydium/cpmm`.
- Raydium official `raydium-cp-swap` source.
## Local CPMM coverage entries
Local registry currently lists 16 `raydium_cpmm` entries:
### Instructions
- `close_permission_pda` — `9c5420764587467b`
- `collect_creator_fee` — `1416567bc61cdb84`
- `collect_fund_fee` — `a78a4e95dfc2067e`
- `collect_protocol_fee` — `8888fcddc2427e59`
- `create_amm_config` — `8934edd4d7756c68`
- `create_permission_pda` — `878802d889a9b5ca`
- `deposit` — `f223c68952e1f2b6`
- `initialize` — `afaf6d1f0d989bed`
- `initialize_with_permission` — `3f37fe4131b25979`
- `swap_base_input` — `8fbe5adac41e33de`
- `swap_base_output` — `37d96256a34ab4ad`
- `update_amm_config` — `313cae889a1c74c8`
- `update_pool_status` — `82576c062ee0757b`
- `withdraw` — `b712469c946da122`
### Anchor / Program-data events
- `lp_change_event` — `79a3cdc939da753c`
- `swap_event` — `40c6cde8260871e2`
## Upstream comparison
Carbon `raydium-cpmm-decoder` exposes the same 14 instruction modules and the two event-like discriminator entries, `lp_change_event` and `swap_event`.
`sol-parser-sdk` `idl/raydium_cpmm.json` exposes the core CPMM instruction set (`createAmmConfig`, `updateAmmConfig`, `updatePoolStatus`, `collectProtocolFee`, `collectFundFee`, `initialize`, `deposit`, `withdraw`, `swapBaseInput`, `swapBaseOutput`) and IDL events `LpChangeEvent` and `SwapEvent`. The local registry also includes the permission and creator-fee entries present in Carbon / Raydium source.
The official Raydium `raydium-cp-swap` source lists the CPMM program ID and the main program instructions including admin/config, fee collection, permission PDA, initialize, initialize with permission, deposit, withdraw, swap base input, and swap base output.
## Finding
No missing CPMM instruction/event discriminator was identified relative to the reviewed Carbon / Raydium / fnzero / Pinax surfaces available during this check.
## Current local caveat
CPMM remains covered by the earlier 0.7.48 tranche. The useful final validation remains DB-side:
```sql
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_cpmm'
ORDER BY entry_kind, entry_name, discriminator_hex;
```
Any future upstream addition should appear as a new entry in Carbon/Solscan/IDL and should be added to `upstream_registry_generated.rs`, `known_local_event_kind` only after local decoder support exists, and then validated with local corpus evidence.

34
kb_demo_app/frontend/demo_pipeline2.html
View File

@@ -122,6 +122,29 @@
Backfill signature
</button>
</div>
<hr class="my-4" />
<div class="mb-3">
<label for="demoPipeline2SignatureBatchTextarea" class="form-label">Signatures batch</label>
<textarea id="demoPipeline2SignatureBatchTextarea" class="form-control font-monospace" rows="8" spellcheck="false" placeholder="Une signature par ligne"></textarea>
<div class="form-text">
Backfill ciblé de plusieurs signatures. Les lignes vides et les doublons sont ignorés.
</div>
</div>
<div class="form-check mb-3">
<input id="demoPipeline2SignatureBatchContinueOnErrorCheckbox" class="form-check-input" type="checkbox" checked />
<label for="demoPipeline2SignatureBatchContinueOnErrorCheckbox" class="form-check-label">
Continuer après une erreur de signature
</label>
</div>
<div class="d-flex gap-2">
<button id="demoPipeline2BackfillSignatureBatchButton" type="button" class="btn btn-outline-primary">
Backfill signatures batch
</button>
</div>
</div>
</div>
</div>
@@ -162,15 +185,22 @@
</label>
</div>
<div class="form-check mb-3">
<div class="form-check mb-2">
<input class="form-check-input" type="checkbox" id="demoPipeline2ReplayForceDexDecodeCheckbox" />
<label class="form-check-label" for="demoPipeline2ReplayForceDexDecodeCheckbox">
Force DEX decode replay
</label>
</div>
<div class="form-check mb-3">
<input class="form-check-input" type="checkbox" id="demoPipeline2ReplayDeferInstructionObservationCheckbox" checked />
<label class="form-check-label" for="demoPipeline2ReplayDeferInstructionObservationCheckbox">
Refresh instruction observations once after replay
</label>
</div>
<p class="small text-body-secondary mb-3">
Le skip ne concerne que l’étape de décodage DEX certifiée par le ledger. Le reste du replay continue pour reconstruire les tables dérivées.
Le skip ne concerne que l’étape de décodage DEX certifiée par le ledger. Le reste du replay continue pour reconstruire les tables dérivées. Le refresh différé des observations accélère le replay mais les compteurs SQL d’instructions ne sont finalisés qu’à la fin.
</p>
<div class="d-flex gap-2">

18
kb_demo_app/frontend/ts/bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts Normal file
View File

@@ -0,0 +1,18 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Request payload for batch signature backfill.
*/
export type DemoPipeline2BackfillSignaturesBatchRequest = {
/**
* Transaction signatures to resolve and replay.
*/
signatures: Array<string>,
/**
* Optional HTTP role.
*/
httpRole: string | null,
/**
* Whether the batch should continue after a hard per-signature error.
*/
continueOnError: boolean, };

84
kb_demo_app/frontend/ts/demo_pipeline2.ts
View File

@@ -11,6 +11,7 @@ import type { DemoPipeline2CatalogPayload } from "./bindings/DemoPipeline2Catalo
import type { DemoPipeline2BackfillTokenRequest } from "./bindings/DemoPipeline2BackfillTokenRequest.ts";
import type { DemoPipeline2BackfillPoolRequest } from "./bindings/DemoPipeline2BackfillPoolRequest.ts";
import type { DemoPipeline2BackfillSignatureRequest } from "./bindings/DemoPipeline2BackfillSignatureRequest.ts";
import type { DemoPipeline2BackfillSignaturesBatchRequest } from "./bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts";
import type { DemoPipeline2BackfillPayload } from "./bindings/DemoPipeline2BackfillPayload.ts";
import type { DemoPipeline2PairCandlesRequest } from "./bindings/DemoPipeline2PairCandlesRequest.ts";
import type { DemoPipeline2PairCandlesPayload } from "./bindings/DemoPipeline2PairCandlesPayload.ts";
@@ -75,6 +76,8 @@ interface LocalPipelineReplayResult {
tokenMetadataUpdatedCount: number;
pairSymbolUpdatedCount: number;
resetMarketMaterializationDeletedCount: number;
instructionObservationScannedCount: number;
instructionObservationUpsertedCount: number;
globalErrorCount: number;
}
function appendLogLine(textarea: HTMLTextAreaElement, line: string): void {
@@ -152,6 +155,25 @@ function readOptionalPositiveIntegerInput(
return parsed;
}
function parseSignatureBatchText(raw: string): string[] {
const signatures: string[] = [];
const seen = new Set<string>();
for (const line of raw.split(/\r?\n/)) {
const signature = line.trim();
if (signature === "") {
continue;
}
if (seen.has(signature)) {
continue;
}
seen.add(signature);
signatures.push(signature);
}
return signatures;
}
function refreshPairSelect(
catalog: DemoPipeline2CatalogPayload,
select: HTMLSelectElement,
@@ -410,12 +432,16 @@ document.addEventListener("DOMContentLoaded", async () => {
const signatureInput = document.querySelector<HTMLInputElement>("#demoPipeline2SignatureInput");
const backfillSignatureButton = document.querySelector<HTMLButtonElement>("#demoPipeline2BackfillSignatureButton");
const signatureBatchTextarea = document.querySelector<HTMLTextAreaElement>("#demoPipeline2SignatureBatchTextarea");
const signatureBatchContinueOnErrorCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2SignatureBatchContinueOnErrorCheckbox");
const backfillSignatureBatchButton = document.querySelector<HTMLButtonElement>("#demoPipeline2BackfillSignatureBatchButton");
const replayLimitInput = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayLimitInput");
const replayMetadataCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayMetadataCheckbox");
const replayMetadataLimitInput = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayMetadataLimitInput");
const replaySkipCertifiedDexDecodeCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplaySkipCertifiedDexDecodeCheckbox");
const replayForceDexDecodeCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayForceDexDecodeCheckbox");
const replayDeferInstructionObservationCheckbox = document.querySelector<HTMLInputElement>("#demoPipeline2ReplayDeferInstructionObservationCheckbox");
const replayLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2ReplayLocalPipelineButton");
const diagnoseLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2DiagnoseLocalPipelineButton");
const validateLocalPipelineButton = document.querySelector<HTMLButtonElement>("#demoPipeline2ValidateLocalPipelineButton");
@@ -462,11 +488,15 @@ document.addEventListener("DOMContentLoaded", async () => {
!backfillPoolButton ||
!signatureInput ||
!backfillSignatureButton ||
!signatureBatchTextarea ||
!signatureBatchContinueOnErrorCheckbox ||
!backfillSignatureBatchButton ||
!replayLimitInput ||
!replayMetadataCheckbox ||
!replayMetadataLimitInput ||
!replaySkipCertifiedDexDecodeCheckbox ||
!replayForceDexDecodeCheckbox ||
!replayDeferInstructionObservationCheckbox ||
!replayLocalPipelineButton ||
!diagnoseLocalPipelineButton ||
!validateLocalPipelineButton ||
@@ -685,6 +715,55 @@ document.addEventListener("DOMContentLoaded", async () => {
}
});
backfillSignatureBatchButton.addEventListener("click", async () => {
const signatures = parseSignatureBatchText(signatureBatchTextarea.value);
if (signatures.length === 0) {
appendLogLine(logTextarea, "[ui] signature batch is empty");
return;
}
const httpRoleText = httpRoleInput.value.trim();
const httpRole = httpRoleText === "" ? null : httpRoleText;
const continueOnError = signatureBatchContinueOnErrorCheckbox.checked;
appendLogLine(
logTextarea,
`[ui] launching signature batch backfill count='${signatures.length.toString()}' role='${httpRole ?? "history_backfill"}' continueOnError='${continueOnError ? "yes" : "no"}'`,
);
const request: DemoPipeline2BackfillSignaturesBatchRequest = {
signatures,
httpRole,
continueOnError,
};
backfillSignatureBatchButton.disabled = true;
backfillSignatureButton.disabled = true;
backfillMintButton.disabled = true;
backfillPoolButton.disabled = true;
try {
const payload = await invoke<DemoPipeline2BackfillPayload>(
"demo_pipeline2_backfill_signatures_batch",
{ request },
);
backfillSummaryTextarea.value = payload.summaryJson;
currentCatalog = payload.catalog;
renderCatalogTextareas(payload.catalog, tokensTextarea, poolsTextarea, pairsTextarea);
refreshPairSelect(payload.catalog, pairSelect);
appendLogLine(logTextarea, `[ui] signature batch backfill completed for '${payload.objectKey}'`);
} catch (error) {
appendLogLine(logTextarea, `[ui] signature batch backfill error: ${String(error)}`);
} finally {
backfillSignatureBatchButton.disabled = false;
backfillSignatureButton.disabled = false;
backfillMintButton.disabled = false;
backfillPoolButton.disabled = false;
}
});
replayLocalPipelineButton.addEventListener("click", async () => {
const replayLimit = readOptionalPositiveIntegerInput(
replayLimitInput,
@@ -706,7 +785,7 @@ document.addEventListener("DOMContentLoaded", async () => {
appendLogLine(
logTextarea,
`[ui] launching local pipeline replay limit='${replayLimit ?? "none"}' metadata='${replayMetadataCheckbox.checked ? "yes" : "no"}' skipDexDecode='${replaySkipCertifiedDexDecodeCheckbox.checked ? "yes" : "no"}' forceDexDecode='${replayForceDexDecodeCheckbox.checked ? "yes" : "no"}'`,
`[ui] launching local pipeline replay limit='${replayLimit ?? "none"}' metadata='${replayMetadataCheckbox.checked ? "yes" : "no"}' skipDexDecode='${replaySkipCertifiedDexDecodeCheckbox.checked ? "yes" : "no"}' forceDexDecode='${replayForceDexDecodeCheckbox.checked ? "yes" : "no"}' deferInstructionObservations='${replayDeferInstructionObservationCheckbox.checked ? "yes" : "no"}'`,
);
try {
@@ -718,6 +797,7 @@ document.addEventListener("DOMContentLoaded", async () => {
tokenMetadataLimit,
skipCertifiedDexDecode: replaySkipCertifiedDexDecodeCheckbox.checked,
forceDecodeReplay: replayForceDexDecodeCheckbox.checked,
deferInstructionObservationIndexRefresh: replayDeferInstructionObservationCheckbox.checked,
},
);
@@ -725,7 +805,7 @@ document.addEventListener("DOMContentLoaded", async () => {
appendLogLine(
logTextarea,
`[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.decodeSkippedCount.toString()} decode skipped, ${result.decodeLedgerUpsertCount.toString()} ledger upserts, ${result.decodeLedgerUnsafeCount.toString()} unsafe ledger rows, ${result.tradeEventCount.toString()} trades, ${result.liquidityEventCount.toString()} liquidity, ${result.poolLifecycleEventCount.toString()} lifecycle, ${result.pairCandleUpsertCount.toString()} candle upserts, resetDeleted='${result.resetMarketMaterializationDeletedCount.toString()}'`,
`[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.decodeSkippedCount.toString()} decode skipped, ${result.decodeLedgerUpsertCount.toString()} ledger upserts, ${result.decodeLedgerUnsafeCount.toString()} unsafe ledger rows, ${result.tradeEventCount.toString()} trades, ${result.liquidityEventCount.toString()} liquidity, ${result.poolLifecycleEventCount.toString()} lifecycle, ${result.pairCandleUpsertCount.toString()} candle upserts, instructionObservations='${result.instructionObservationUpsertedCount.toString()}', resetDeleted='${result.resetMarketMaterializationDeletedCount.toString()}'`,
);
await refreshCatalog();

2
kb_demo_app/package.json
View File

@@ -1,7 +1,7 @@
{
"name": "kb-demo-app",
"private": true,
"version": "0.7.48",
"version": "0.7.49",
"type": "module",
"scripts": {
"dev": "vite",

75
kb_demo_app/src/demo_pipeline2.rs
View File

@@ -1153,6 +1153,22 @@ pub(crate) struct DemoPipeline2BackfillSignatureRequest {
pub http_role: std::option::Option<std::string::String>,
}
/// Request payload for batch signature backfill.
#[derive(Clone, Debug, serde::Deserialize, TS)]
#[ts(
export,
export_to = "../frontend/ts/bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts"
)]
#[serde(rename_all = "camelCase")]
pub(crate) struct DemoPipeline2BackfillSignaturesBatchRequest {
/// Transaction signatures to resolve and replay.
pub signatures: std::vec::Vec<std::string::String>,
/// Optional HTTP role.
pub http_role: std::option::Option<std::string::String>,
/// Whether the batch should continue after a hard per-signature error.
pub continue_on_error: bool,
}
/// Shared backfill response payload.
#[derive(Clone, Debug, serde::Serialize, TS)]
#[ts(export, export_to = "../frontend/ts/bindings/DemoPipeline2BackfillPayload.ts")]
@@ -1604,6 +1620,63 @@ pub(crate) async fn demo_pipeline2_backfill_signature(
})
}
/// Runs a targeted batch signature backfill then returns the refreshed catalog.
#[tauri::command]
pub(crate) async fn demo_pipeline2_backfill_signatures_batch(
state: tauri::State<'_, crate::AppState>,
request: DemoPipeline2BackfillSignaturesBatchRequest,
) -> Result<DemoPipeline2BackfillPayload, std::string::String> {
let mut signatures = std::vec::Vec::<std::string::String>::new();
for signature in request.signatures {
let trimmed_signature = signature.trim().to_string();
if trimmed_signature.is_empty() {
continue;
}
signatures.push(trimmed_signature);
}
if signatures.is_empty() {
return Err("signature batch must contain at least one signature".to_string());
}
let http_role = demo_pipeline2_normalize_http_role(request.http_role);
let database = state.database.clone();
let http_pool = std::sync::Arc::new(state.http_pool.clone());
let service = kb_lib::TokenBackfillService::new(http_pool, database.clone(), http_role.clone());
let result = service
.backfill_signatures(signatures.as_slice(), request.continue_on_error)
.await;
let backfill = match result {
Ok(backfill) => backfill,
Err(error) => {
return Err(format!(
"cannot backfill signature batch with role '{}': {}",
http_role, error
));
},
};
let summary_json_result = serde_json::to_string_pretty(&backfill);
let summary_json = match summary_json_result {
Ok(summary_json) => summary_json,
Err(error) => {
return Err(format!(
"cannot serialize signature batch backfill result: {}",
error
));
},
};
let catalog_result = demo_pipeline2_build_catalog(database).await;
let catalog = match catalog_result {
Ok(catalog) => catalog,
Err(error) => return Err(error),
};
Ok(DemoPipeline2BackfillPayload {
object_key: format!("{} signatures", backfill.unique_signature_count),
mode: "signatureBatch".to_string(),
http_role,
summary_json,
catalog,
})
}
/// Loads candles for one pair and one timeframe.
#[tauri::command]
pub(crate) async fn demo_pipeline2_get_pair_candles(
@@ -1783,6 +1856,7 @@ pub(crate) async fn demo_pipeline2_replay_local_pipeline(
token_metadata_limit: std::option::Option<i64>,
skip_certified_dex_decode: bool,
force_decode_replay: bool,
defer_instruction_observation_index_refresh: bool,
) -> Result<kb_lib::LocalPipelineReplayResult, std::string::String> {
let config = kb_lib::LocalPipelineReplayConfig {
limit,
@@ -1791,6 +1865,7 @@ pub(crate) async fn demo_pipeline2_replay_local_pipeline(
reset_market_materialization_before_replay: true,
skip_certified_dex_decode,
force_decode_replay,
defer_instruction_observation_index_refresh,
};
let database = state.database.clone();
let service = if refresh_missing_token_metadata {

1
kb_demo_app/src/lib.rs
View File

@@ -158,6 +158,7 @@ pub async fn run() -> Result<(), kb_lib::Error> {
crate::demo_pipeline2::demo_pipeline2_backfill_token_mint,
crate::demo_pipeline2::demo_pipeline2_backfill_pool_address,
crate::demo_pipeline2::demo_pipeline2_backfill_signature,
crate::demo_pipeline2::demo_pipeline2_backfill_signatures_batch,
crate::demo_pipeline2::demo_pipeline2_get_pair_candles,
crate::demo_pipeline2::demo_pipeline2_replay_local_pipeline,
crate::demo_pipeline2::demo_pipeline2_diagnose_local_pipeline,

2
kb_demo_app/tauri.conf.json
View File

@@ -1,7 +1,7 @@
{
"$schema": "https://schema.tauri.app/config/2",
"productName": "kb-demo-app",
"version": "0.7.48",
"version": "0.7.49",
"identifier": "com.sasedev.kb-demo-app",
"build": {
"beforeDevCommand": "npm run dev",

5
kb_lib/src/db.rs
View File

@@ -61,6 +61,7 @@ pub use dtos::LocalRaydiumSurfaceDiagnosticSummaryDto;
pub use dtos::LocalTokenMetadataGapDiagnosticSampleDto;
pub use dtos::ObservedTokenDto;
pub use dtos::OnchainObservationDto;
pub use dtos::OrderbookEventDto;
pub use dtos::PairAnalyticSignalDto;
pub use dtos::PairCandleDto;
pub use dtos::PairDto;
@@ -106,6 +107,7 @@ pub use entities::LaunchSurfaceKeyEntity;
pub use entities::LiquidityEventEntity;
pub use entities::ObservedTokenEntity;
pub use entities::OnchainObservationEntity;
pub use entities::OrderbookEventEntity;
pub use entities::PairAnalyticSignalEntity;
pub use entities::PairCandleEntity;
pub use entities::PairEntity;
@@ -152,8 +154,10 @@ pub use queries::query_dex_decode_replay_ledger_get_by_signature;
pub use queries::query_dex_decode_replay_ledger_get_by_transaction;
pub use queries::query_dex_decode_replay_ledger_upsert;
pub use queries::query_dex_decoded_events_delete_by_key;
pub use queries::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches;
pub use queries::query_dex_decoded_events_delete_meteora_dlmm_anchor_swap_instruction_audits;
pub use queries::query_dex_decoded_events_delete_related_instruction_audit;
pub use queries::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits;
pub use queries::query_dex_decoded_events_get_by_key;
pub use queries::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint;
pub use queries::query_dex_decoded_events_list_by_transaction_id;
@@ -216,6 +220,7 @@ pub use queries::query_observed_tokens_list;
pub use queries::query_observed_tokens_upsert;
pub use queries::query_onchain_observations_insert;
pub use queries::query_onchain_observations_list_recent;
pub use queries::query_orderbook_events_upsert;
pub use queries::query_pair_analytic_signals_get_by_key;
pub use queries::query_pair_analytic_signals_list_by_pair_id;
pub use queries::query_pair_analytic_signals_upsert;

2
kb_lib/src/db/dtos.rs
View File

@@ -24,6 +24,7 @@ mod local_dex_corpus_search;
mod local_pipeline_diagnostics;
mod observed_token;
mod onchain_observation;
mod orderbook_event;
mod pair;
mod pair_analytic_signal;
mod pair_candle;
@@ -117,6 +118,7 @@ pub use local_pipeline_diagnostics::LocalRaydiumSurfaceDiagnosticSummaryDto;
pub use local_pipeline_diagnostics::LocalTokenMetadataGapDiagnosticSampleDto;
pub use observed_token::ObservedTokenDto;
pub use onchain_observation::OnchainObservationDto;
pub use orderbook_event::OrderbookEventDto;
pub use pair::PairDto;
pub use pair_analytic_signal::PairAnalyticSignalDto;
pub use pair_candle::PairCandleDto;

190
kb_lib/src/db/dtos/orderbook_event.rs Normal file
View File

@@ -0,0 +1,190 @@
// file: kb_lib/src/db/dtos/orderbook_event.rs
//! Orderbook event DTO.
/// Application-facing normalized orderbook or limit-order event DTO.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct OrderbookEventDto {
/// Optional numeric primary key.
pub id: std::option::Option<i64>,
/// Related transaction id.
pub transaction_id: i64,
/// Related decoded DEX event id, when available.
pub decoded_event_id: std::option::Option<i64>,
/// Related DEX id, when the DEX row is known.
pub dex_id: std::option::Option<i64>,
/// Related pool id, when the pool row is known.
pub pool_id: std::option::Option<i64>,
/// Related pair id, when the pair row is known.
pub pair_id: std::option::Option<i64>,
/// Transaction signature.
pub signature: std::string::String,
/// Optional slot number.
pub slot: std::option::Option<u64>,
/// Protocol name that emitted the decoded event.
pub protocol_name: std::string::String,
/// Program id that emitted the decoded event.
pub program_id: std::string::String,
/// Stable decoded event kind.
pub event_kind: std::string::String,
/// Normalized orderbook action.
pub order_action: std::string::String,
/// Pool account address, when decoded.
pub pool_account: std::option::Option<std::string::String>,
/// Market/orderbook account, when decoded.
pub market_account: std::option::Option<std::string::String>,
/// Wallet or authority associated with the event, when decoded.
pub actor_wallet: std::option::Option<std::string::String>,
/// Limit/order account, when decoded.
pub order_account: std::option::Option<std::string::String>,
/// Base or token A mint, when decoded.
pub base_token_mint: std::option::Option<std::string::String>,
/// Quote or token B mint, when decoded.
pub quote_token_mint: std::option::Option<std::string::String>,
/// Raw order amount as decimal text, when decoded.
pub amount_raw: std::option::Option<std::string::String>,
/// Raw minimum amount as decimal text, when decoded.
pub amount_min_raw: std::option::Option<std::string::String>,
/// Optional tick index for CLMM limit orders.
pub tick_index: std::option::Option<i64>,
/// Optional zero-for-one side flag.
pub zero_for_one: std::option::Option<bool>,
/// Source decoded payload JSON.
pub payload_json: std::string::String,
/// Execution timestamp.
pub executed_at: chrono::DateTime<chrono::Utc>,
/// Creation timestamp.
pub created_at: chrono::DateTime<chrono::Utc>,
}
impl OrderbookEventDto {
/// Creates a new orderbook event DTO.
#[allow(clippy::too_many_arguments)]
pub fn new(
transaction_id: i64,
decoded_event_id: std::option::Option<i64>,
dex_id: std::option::Option<i64>,
pool_id: std::option::Option<i64>,
pair_id: std::option::Option<i64>,
signature: std::string::String,
slot: std::option::Option<u64>,
protocol_name: std::string::String,
program_id: std::string::String,
event_kind: std::string::String,
order_action: std::string::String,
pool_account: std::option::Option<std::string::String>,
market_account: std::option::Option<std::string::String>,
actor_wallet: std::option::Option<std::string::String>,
order_account: std::option::Option<std::string::String>,
base_token_mint: std::option::Option<std::string::String>,
quote_token_mint: std::option::Option<std::string::String>,
amount_raw: std::option::Option<std::string::String>,
amount_min_raw: std::option::Option<std::string::String>,
tick_index: std::option::Option<i64>,
zero_for_one: std::option::Option<bool>,
payload_json: std::string::String,
) -> Self {
let now = chrono::Utc::now();
return Self {
id: None,
transaction_id,
decoded_event_id,
dex_id,
pool_id,
pair_id,
signature,
slot,
protocol_name,
program_id,
event_kind,
order_action,
pool_account,
market_account,
actor_wallet,
order_account,
base_token_mint,
quote_token_mint,
amount_raw,
amount_min_raw,
tick_index,
zero_for_one,
payload_json,
executed_at: now,
created_at: now,
};
}
}
impl TryFrom<crate::OrderbookEventEntity> for OrderbookEventDto {
type Error = crate::Error;
fn try_from(entity: crate::OrderbookEventEntity) -> Result<Self, Self::Error> {
let executed_at_result = chrono::DateTime::parse_from_rfc3339(&entity.executed_at);
let executed_at = match executed_at_result {
Ok(executed_at) => executed_at.with_timezone(&chrono::Utc),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot parse orderbook event executed_at '{}': {}",
entity.executed_at, error
)));
},
};
let created_at_result = chrono::DateTime::parse_from_rfc3339(&entity.created_at);
let created_at = match created_at_result {
Ok(created_at) => created_at.with_timezone(&chrono::Utc),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot parse orderbook event created_at '{}': {}",
entity.created_at, error
)));
},
};
let slot = match entity.slot {
Some(slot) => {
let slot_result = u64::try_from(slot);
match slot_result {
Ok(slot) => Some(slot),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot convert orderbook event slot '{}' to u64: {}",
slot, error
)));
},
}
},
None => None,
};
let zero_for_one = match entity.zero_for_one {
Some(0) => Some(false),
Some(_) => Some(true),
None => None,
};
return Ok(Self {
id: Some(entity.id),
transaction_id: entity.transaction_id,
decoded_event_id: entity.decoded_event_id,
dex_id: entity.dex_id,
pool_id: entity.pool_id,
pair_id: entity.pair_id,
signature: entity.signature,
slot,
protocol_name: entity.protocol_name,
program_id: entity.program_id,
event_kind: entity.event_kind,
order_action: entity.order_action,
pool_account: entity.pool_account,
market_account: entity.market_account,
actor_wallet: entity.actor_wallet,
order_account: entity.order_account,
base_token_mint: entity.base_token_mint,
quote_token_mint: entity.quote_token_mint,
amount_raw: entity.amount_raw,
amount_min_raw: entity.amount_min_raw,
tick_index: entity.tick_index,
zero_for_one,
payload_json: entity.payload_json,
executed_at,
created_at,
});
}
}

2
kb_lib/src/db/entities.rs
View File

@@ -24,6 +24,7 @@ mod launch_surface_key;
mod liquidity_event;
mod observed_token;
mod onchain_observation;
mod orderbook_event;
mod pair;
mod pair_analytic_signal;
mod pair_candle;
@@ -70,6 +71,7 @@ pub use launch_surface_key::LaunchSurfaceKeyEntity;
pub use liquidity_event::LiquidityEventEntity;
pub use observed_token::ObservedTokenEntity;
pub use onchain_observation::OnchainObservationEntity;
pub use orderbook_event::OrderbookEventEntity;
pub use pair::PairEntity;
pub use pair_analytic_signal::PairAnalyticSignalEntity;
pub use pair_candle::PairCandleEntity;

58
kb_lib/src/db/entities/orderbook_event.rs Normal file
View File

@@ -0,0 +1,58 @@
// file: kb_lib/src/db/entities/orderbook_event.rs
//! Orderbook event entity.
/// Persisted normalized orderbook or limit-order event row.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, sqlx::FromRow)]
pub struct OrderbookEventEntity {
/// Numeric primary key.
pub id: i64,
/// Related transaction id.
pub transaction_id: i64,
/// Related decoded DEX event id, when available.
pub decoded_event_id: std::option::Option<i64>,
/// Related DEX id, when the DEX row is known.
pub dex_id: std::option::Option<i64>,
/// Related pool id, when the pool row is known.
pub pool_id: std::option::Option<i64>,
/// Related pair id, when the pair row is known.
pub pair_id: std::option::Option<i64>,
/// Transaction signature.
pub signature: std::string::String,
/// Optional slot number.
pub slot: std::option::Option<i64>,
/// Protocol name that emitted the decoded event.
pub protocol_name: std::string::String,
/// Program id that emitted the decoded event.
pub program_id: std::string::String,
/// Stable decoded event kind.
pub event_kind: std::string::String,
/// Normalized orderbook action.
pub order_action: std::string::String,
/// Pool account address, when decoded.
pub pool_account: std::option::Option<std::string::String>,
/// Market/orderbook account, when decoded.
pub market_account: std::option::Option<std::string::String>,
/// Wallet or authority associated with the event, when decoded.
pub actor_wallet: std::option::Option<std::string::String>,
/// Limit/order account, when decoded.
pub order_account: std::option::Option<std::string::String>,
/// Base or token A mint, when decoded.
pub base_token_mint: std::option::Option<std::string::String>,
/// Quote or token B mint, when decoded.
pub quote_token_mint: std::option::Option<std::string::String>,
/// Raw order amount as decimal text, when decoded.
pub amount_raw: std::option::Option<std::string::String>,
/// Raw minimum amount as decimal text, when decoded.
pub amount_min_raw: std::option::Option<std::string::String>,
/// Optional tick index for CLMM limit orders.
pub tick_index: std::option::Option<i64>,
/// Optional zero-for-one side flag, stored as 0/1.
pub zero_for_one: std::option::Option<i64>,
/// Source decoded payload JSON.
pub payload_json: std::string::String,
/// Execution timestamp encoded as RFC3339 UTC text.
pub executed_at: std::string::String,
/// Creation timestamp encoded as RFC3339 UTC text.
pub created_at: std::string::String,
}

10
kb_lib/src/db/queries.rs
View File

@@ -13,9 +13,9 @@ mod dex_decode_replay_ledger;
mod dex_decoded_event;
mod dex_event_coverage_entry;
mod fee_event;
mod instruction_observation;
mod known_http_endpoint;
mod known_ws_endpoint;
mod instruction_observation;
mod launch_attribution;
mod launch_surface;
mod launch_surface_key;
@@ -24,6 +24,7 @@ mod local_dex_corpus_search;
mod local_pipeline_diagnostics;
mod observed_token;
mod onchain_observation;
mod orderbook_event;
mod pair;
mod pair_analytic_signal;
mod pair_candle;
@@ -72,8 +73,10 @@ pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_get_by_signatur
pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_get_by_transaction;
pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_upsert;
pub use dex_decoded_event::query_dex_decoded_events_delete_by_key;
pub use dex_decoded_event::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches;
pub use dex_decoded_event::query_dex_decoded_events_delete_meteora_dlmm_anchor_swap_instruction_audits;
pub use dex_decoded_event::query_dex_decoded_events_delete_related_instruction_audit;
pub use dex_decoded_event::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits;
pub use dex_decoded_event::query_dex_decoded_events_get_by_key;
pub use dex_decoded_event::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint;
pub use dex_decoded_event::query_dex_decoded_events_list_by_transaction_id;
@@ -87,14 +90,14 @@ pub use dex_event_coverage_entry::query_dex_event_coverage_entries_upsert;
pub use fee_event::query_fee_events_get_by_decoded_event_id;
pub use fee_event::query_fee_events_list_recent;
pub use fee_event::query_fee_events_upsert;
pub use instruction_observation::query_instruction_observations_list_by_filter;
pub use instruction_observation::query_instruction_observations_upsert;
pub use known_http_endpoint::query_known_http_endpoints_get;
pub use known_http_endpoint::query_known_http_endpoints_list;
pub use known_http_endpoint::query_known_http_endpoints_upsert;
pub use known_ws_endpoint::query_known_ws_endpoints_get;
pub use known_ws_endpoint::query_known_ws_endpoints_list;
pub use known_ws_endpoint::query_known_ws_endpoints_upsert;
pub use instruction_observation::query_instruction_observations_list_by_filter;
pub use instruction_observation::query_instruction_observations_upsert;
pub use launch_attribution::query_launch_attributions_get_by_decoded_event_id;
pub use launch_attribution::query_launch_attributions_list_by_pool_id;
pub use launch_attribution::query_launch_attributions_upsert;
@@ -133,6 +136,7 @@ pub use observed_token::query_observed_tokens_list;
pub use observed_token::query_observed_tokens_upsert;
pub use onchain_observation::query_onchain_observations_insert;
pub use onchain_observation::query_onchain_observations_list_recent;
pub use orderbook_event::query_orderbook_events_upsert;
pub use pair::query_pairs_get_by_pool_id;
pub use pair::query_pairs_list;
pub use pair::query_pairs_update_symbol;

179
kb_lib/src/db/queries/dex_decoded_event.rs
View File

@@ -191,6 +191,185 @@ WHERE transaction_id = ?
}
}
/// Deletes Raydium CLMM instruction-audit rows for locally mapped CLMM instructions.
///
/// The CLMM specialized decoder now emits named `raydium_clmm.*` rows for all
/// locally mapped instruction discriminants. Keeping the former
/// `raydium_clmm.instruction_audit` rows for the same discriminants creates
/// duplicate coverage and makes residual audit diagnostics noisy. Unknown
/// discriminants remain untouched because they are intentionally absent from
/// this allow-list.
pub async fn query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits(
database: &crate::Database,
transaction_id: std::option::Option<i64>,
) -> Result<u64, crate::Error> {
match database.connection() {
crate::DatabaseConnection::Sqlite(pool) => {
let query_result = sqlx::query(
r#"
DELETE FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
AND (? IS NULL OR transaction_id = ?)
AND COALESCE(
json_extract(payload_json, '$.discriminatorHex'),
json_extract(payload_json, '$.discriminator_hex'),
json_extract(payload_json, '$.instructionDiscriminatorHex'),
json_extract(payload_json, '$.instruction_discriminator_hex')
) IN (
'4c7c800fd55725fa',
'9d20dab7471d1293',
'b19059ecfaba7d63',
'759d3c674231a300',
'7b86510031446262',
'c975989055556cb2',
'a78a4e95dfc2067e',
'8888fcddc2427e59',
'12eda6c52210d590',
'8934edd4d7756c68',
'2b44d4a7592fa401',
'bd0eb5785576e33e',
'3f5794216d230868',
'e992d18ecf6840bc',
'11fb415c88f20ea9',
'a026d06f685b2c01',
'3a7fbc3e4f52c460',
'2e9cf3760dcdfbb2',
'851d59df45eeb00a',
'5f87c0c4f281e644',
'87802f4d0f98f031',
'4db84ad67056f1c7',
'4dffae527d1dc92e',
'7034a74b20c9d389',
'cd4e74215c691a60',
'f8c69e91e17587c8',
'457d73daf5baf2c4',
'2b04ed0b1ac91e62',
'07160c53f22b3079',
'313cae889a1c74c8',
'7f467728bce33d07',
'82576c062ee0757b',
'a3ace0340b9a6adf'
)
"#,
)
.bind(transaction_id)
.bind(transaction_id)
.execute(pool)
.await;
match query_result {
Ok(result) => return Ok(result.rows_affected()),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot delete mapped Raydium CLMM instruction audit events on sqlite: {}",
error
)));
},
}
},
}
}
/// Deletes upstream registry instruction-match rows already covered by specialized local decoders.
///
/// The upstream registry fallback is useful only while an instruction is not yet
/// handled by a protocol-specific decoder. Once `k_sol_dex_event_coverage_entries`
/// has a non-empty `local_event_kind` for the same decoder/entry/discriminator,
/// the fallback row is redundant and must be removed so coverage queries do not
/// report both `upstream_git.instruction_match` and the local `protocol.*` event.
pub async fn query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(
database: &crate::Database,
upstream_decoder_code: std::option::Option<&str>,
) -> Result<u64, crate::Error> {
match database.connection() {
crate::DatabaseConnection::Sqlite(pool) => {
let unlink_result = sqlx::query(
r#"
UPDATE k_sol_instruction_observations
SET decoded_event_id = NULL
WHERE decoded_event_id IN (
SELECT de.id
FROM k_sol_dex_decoded_events de
WHERE de.protocol_name = 'upstream_git'
AND de.event_kind = 'upstream_git.instruction_match'
AND (? IS NULL OR json_extract(de.payload_json, '$.upstreamDecoderCode') = ?)
AND (
COALESCE(json_extract(de.payload_json, '$.upstreamDecoderCode'), '')
|| '|'
|| COALESCE(json_extract(de.payload_json, '$.upstreamEntryName'), '')
|| '|'
|| COALESCE(json_extract(de.payload_json, '$.upstreamDiscriminatorHex'), '')
) IN (
SELECT
ce.decoder_code
|| '|'
|| ce.entry_name
|| '|'
|| ce.discriminator_hex
FROM k_sol_dex_event_coverage_entries ce
WHERE ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
AND ce.discriminator_hex IS NOT NULL
AND ce.discriminator_hex <> ''
)
)
"#,
)
.bind(upstream_decoder_code)
.bind(upstream_decoder_code)
.execute(pool)
.await;
if let Err(error) = unlink_result {
return Err(crate::Error::Db(format!(
"cannot unlink locally covered upstream instruction matches from instruction observations on sqlite: {}",
error
)));
}
let query_result = sqlx::query(
r#"
DELETE FROM k_sol_dex_decoded_events
WHERE protocol_name = 'upstream_git'
AND event_kind = 'upstream_git.instruction_match'
AND (? IS NULL OR json_extract(payload_json, '$.upstreamDecoderCode') = ?)
AND (
COALESCE(json_extract(payload_json, '$.upstreamDecoderCode'), '')
|| '|'
|| COALESCE(json_extract(payload_json, '$.upstreamEntryName'), '')
|| '|'
|| COALESCE(json_extract(payload_json, '$.upstreamDiscriminatorHex'), '')
) IN (
SELECT
ce.decoder_code
|| '|'
|| ce.entry_name
|| '|'
|| ce.discriminator_hex
FROM k_sol_dex_event_coverage_entries ce
WHERE ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
AND ce.discriminator_hex IS NOT NULL
AND ce.discriminator_hex <> ''
)
"#,
)
.bind(upstream_decoder_code)
.bind(upstream_decoder_code)
.execute(pool)
.await;
match query_result {
Ok(result) => return Ok(result.rows_affected()),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot delete locally covered upstream instruction matches on sqlite: {}",
error
)));
},
}
},
}
}
/// Deletes Meteora DLMM Anchor self-CPI swap audit rows already covered by decoded swaps.
///
/// This targets only local-corpus-observed Anchor event discriminators that are

35
kb_lib/src/db/queries/dex_event_coverage_entry.rs
View File

@@ -569,6 +569,41 @@ SET
)
)
)
WHEN expected_db_target = 'k_sol_orderbook_events' THEN (
SELECT COUNT(oe.id)
FROM k_sol_dex_decoded_events de
JOIN k_sol_orderbook_events oe ON oe.decoded_event_id = de.id
WHERE (
(k_sol_dex_event_coverage_entries.program_id IS NULL OR de.program_id = k_sol_dex_event_coverage_entries.program_id)
AND (
(
k_sol_dex_event_coverage_entries.local_event_kind IS NOT NULL
AND k_sol_dex_event_coverage_entries.local_event_kind <> ''
AND de.event_kind = k_sol_dex_event_coverage_entries.local_event_kind
)
OR (
k_sol_dex_event_coverage_entries.entry_name IS NOT NULL
AND (
json_extract(de.payload_json, '$.upstreamEntryName') = k_sol_dex_event_coverage_entries.entry_name
OR json_extract(de.payload_json, '$.upstreamInstructionName') = k_sol_dex_event_coverage_entries.entry_name
OR json_extract(de.payload_json, '$.upstreamEventName') = k_sol_dex_event_coverage_entries.entry_name
OR json_extract(de.payload_json, '$.entryName') = k_sol_dex_event_coverage_entries.entry_name
)
)
OR (
k_sol_dex_event_coverage_entries.discriminator_hex IS NOT NULL
AND k_sol_dex_event_coverage_entries.discriminator_hex <> ''
AND (
json_extract(de.payload_json, '$.upstreamDiscriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex
OR json_extract(de.payload_json, '$.instructionDiscriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex
OR json_extract(de.payload_json, '$.anchorEventDiscriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex
OR json_extract(de.payload_json, '$.anchorEventDiscriminator') = k_sol_dex_event_coverage_entries.discriminator_hex
OR json_extract(de.payload_json, '$.discriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex
)
)
)
)
)
ELSE materialized_count
END,
first_signature = (

212
kb_lib/src/db/queries/orderbook_event.rs Normal file
View File

@@ -0,0 +1,212 @@
// file: kb_lib/src/db/queries/orderbook_event.rs
//! Queries for `k_sol_orderbook_events`.
/// Inserts or updates one normalized orderbook event row.
pub async fn query_orderbook_events_upsert(
database: &crate::Database,
dto: &crate::OrderbookEventDto,
) -> Result<i64, crate::Error> {
let slot_i64 = match dto.slot {
Some(slot) => {
let slot_result = i64::try_from(slot);
match slot_result {
Ok(slot) => Some(slot),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot convert orderbook event slot '{}' to i64: {}",
slot, error
)));
},
}
},
None => None,
};
let zero_for_one_i64 = match dto.zero_for_one {
Some(true) => Some(1_i64),
Some(false) => Some(0_i64),
None => None,
};
match database.connection() {
crate::DatabaseConnection::Sqlite(pool) => {
let existing_id = match dto.decoded_event_id {
Some(decoded_event_id) => {
let existing_result = sqlx::query_scalar::<sqlx::Sqlite, i64>(
r#"
SELECT id
FROM k_sol_orderbook_events
WHERE decoded_event_id = ?
LIMIT 1
"#,
)
.bind(decoded_event_id)
.fetch_optional(pool)
.await;
match existing_result {
Ok(existing_id) => existing_id,
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot fetch k_sol_orderbook_events id for decoded_event_id '{}' on sqlite: {}",
decoded_event_id, error
)));
},
}
},
None => None,
};
if let Some(id) = existing_id {
let update_result = sqlx::query(
r#"
UPDATE k_sol_orderbook_events
SET
transaction_id = ?,
dex_id = ?,
pool_id = ?,
pair_id = ?,
signature = ?,
slot = ?,
protocol_name = ?,
program_id = ?,
event_kind = ?,
order_action = ?,
pool_account = ?,
market_account = ?,
actor_wallet = ?,
order_account = ?,
base_token_mint = ?,
quote_token_mint = ?,
amount_raw = ?,
amount_min_raw = ?,
tick_index = ?,
zero_for_one = ?,
payload_json = ?,
executed_at = ?
WHERE id = ?
"#,
)
.bind(dto.transaction_id)
.bind(dto.dex_id)
.bind(dto.pool_id)
.bind(dto.pair_id)
.bind(dto.signature.clone())
.bind(slot_i64)
.bind(dto.protocol_name.clone())
.bind(dto.program_id.clone())
.bind(dto.event_kind.clone())
.bind(dto.order_action.clone())
.bind(dto.pool_account.clone())
.bind(dto.market_account.clone())
.bind(dto.actor_wallet.clone())
.bind(dto.order_account.clone())
.bind(dto.base_token_mint.clone())
.bind(dto.quote_token_mint.clone())
.bind(dto.amount_raw.clone())
.bind(dto.amount_min_raw.clone())
.bind(dto.tick_index)
.bind(zero_for_one_i64)
.bind(dto.payload_json.clone())
.bind(dto.executed_at.to_rfc3339())
.bind(id)
.execute(pool)
.await;
if let Err(error) = update_result {
return Err(crate::Error::Db(format!(
"cannot update k_sol_orderbook_events id '{}' on sqlite: {}",
id, error
)));
}
return Ok(id);
}
let insert_result = sqlx::query(
r#"
INSERT INTO k_sol_orderbook_events (
transaction_id,
decoded_event_id,
dex_id,
pool_id,
pair_id,
signature,
slot,
protocol_name,
program_id,
event_kind,
order_action,
pool_account,
market_account,
actor_wallet,
order_account,
base_token_mint,
quote_token_mint,
amount_raw,
amount_min_raw,
tick_index,
zero_for_one,
payload_json,
executed_at,
created_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
"#,
)
.bind(dto.transaction_id)
.bind(dto.decoded_event_id)
.bind(dto.dex_id)
.bind(dto.pool_id)
.bind(dto.pair_id)
.bind(dto.signature.clone())
.bind(slot_i64)
.bind(dto.protocol_name.clone())
.bind(dto.program_id.clone())
.bind(dto.event_kind.clone())
.bind(dto.order_action.clone())
.bind(dto.pool_account.clone())
.bind(dto.market_account.clone())
.bind(dto.actor_wallet.clone())
.bind(dto.order_account.clone())
.bind(dto.base_token_mint.clone())
.bind(dto.quote_token_mint.clone())
.bind(dto.amount_raw.clone())
.bind(dto.amount_min_raw.clone())
.bind(dto.tick_index)
.bind(zero_for_one_i64)
.bind(dto.payload_json.clone())
.bind(dto.executed_at.to_rfc3339())
.bind(dto.created_at.to_rfc3339())
.execute(pool)
.await;
if let Err(error) = insert_result {
return Err(crate::Error::Db(format!(
"cannot insert k_sol_orderbook_events on sqlite: {}",
error
)));
}
let id_result = sqlx::query_scalar::<sqlx::Sqlite, i64>(
r#"
SELECT id
FROM k_sol_orderbook_events
WHERE transaction_id = ?
AND protocol_name = ?
AND event_kind = ?
AND signature = ?
ORDER BY id DESC
LIMIT 1
"#,
)
.bind(dto.transaction_id)
.bind(dto.protocol_name.clone())
.bind(dto.event_kind.clone())
.bind(dto.signature.clone())
.fetch_one(pool)
.await;
match id_result {
Ok(id) => return Ok(id),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot fetch inserted k_sol_orderbook_events id for signature '{}' on sqlite: {}",
dto.signature, error
)));
},
}
},
}
}

60
kb_lib/src/db/queries/trade_event.rs
View File

@@ -16,7 +16,67 @@ pub async fn query_trade_market_materialization_delete_all(
("k_sol_pair_analytic_signals", "DELETE FROM k_sol_pair_analytic_signals"),
("k_sol_pair_candles", "DELETE FROM k_sol_pair_candles"),
("k_sol_pair_metrics", "DELETE FROM k_sol_pair_metrics"),
("k_sol_orderbook_events", "DELETE FROM k_sol_orderbook_events"),
("k_sol_trade_events", "DELETE FROM k_sol_trade_events"),
(
"locally_covered_upstream_instruction_observation_links",
r#"
UPDATE k_sol_instruction_observations
SET decoded_event_id = NULL
WHERE decoded_event_id IN (
SELECT de.id
FROM k_sol_dex_decoded_events de
WHERE de.protocol_name = 'upstream_git'
AND de.event_kind = 'upstream_git.instruction_match'
AND (
COALESCE(json_extract(de.payload_json, '$.upstreamDecoderCode'), '')
|| '|'
|| COALESCE(json_extract(de.payload_json, '$.upstreamEntryName'), '')
|| '|'
|| COALESCE(json_extract(de.payload_json, '$.upstreamDiscriminatorHex'), '')
) IN (
SELECT
ce.decoder_code
|| '|'
|| ce.entry_name
|| '|'
|| ce.discriminator_hex
FROM k_sol_dex_event_coverage_entries ce
WHERE ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
AND ce.discriminator_hex IS NOT NULL
AND ce.discriminator_hex <> ''
)
)
"#,
),
(
"locally_covered_upstream_instruction_matches",
r#"
DELETE FROM k_sol_dex_decoded_events
WHERE protocol_name = 'upstream_git'
AND event_kind = 'upstream_git.instruction_match'
AND (
COALESCE(json_extract(payload_json, '$.upstreamDecoderCode'), '')
|| '|'
|| COALESCE(json_extract(payload_json, '$.upstreamEntryName'), '')
|| '|'
|| COALESCE(json_extract(payload_json, '$.upstreamDiscriminatorHex'), '')
) IN (
SELECT
ce.decoder_code
|| '|'
|| ce.entry_name
|| '|'
|| ce.discriminator_hex
FROM k_sol_dex_event_coverage_entries ce
WHERE ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
AND ce.discriminator_hex IS NOT NULL
AND ce.discriminator_hex <> ''
)
"#,
),
];
let mut deleted_count = 0_u64;
for (table_name, statement) in statements {

98
kb_lib/src/db/schema.rs
View File

@@ -374,6 +374,22 @@ pub(crate) async fn ensure_schema(database: &crate::Database) -> Result<(), crat
if let Err(error) = result {
return Err(error);
}
let result = create_tbl_orderbook_events(pool).await;
if let Err(error) = result {
return Err(error);
}
let result = create_idx_orderbook_events_transaction_id(pool).await;
if let Err(error) = result {
return Err(error);
}
let result = create_idx_orderbook_events_pool_id(pool).await;
if let Err(error) = result {
return Err(error);
}
let result = create_uix_orderbook_events_decoded_event_id(pool).await;
if let Err(error) = result {
return Err(error);
}
let result = create_tbl_launch_surfaces(pool).await;
if let Err(error) = result {
return Err(error);
@@ -2710,3 +2726,85 @@ WHERE decoded_event_id IS NOT NULL
)
.await;
}
/// Creates `k_sol_orderbook_events`.
async fn create_tbl_orderbook_events(pool: &sqlx::SqlitePool) -> Result<(), crate::Error> {
return execute_sqlite_schema_statement(
pool,
"create_tbl_orderbook_events",
r#"
CREATE TABLE IF NOT EXISTS k_sol_orderbook_events (
id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
transaction_id INTEGER NOT NULL,
decoded_event_id INTEGER NULL,
dex_id INTEGER NULL,
pool_id INTEGER NULL,
pair_id INTEGER NULL,
signature TEXT NOT NULL,
slot INTEGER NULL,
protocol_name TEXT NOT NULL,
program_id TEXT NOT NULL,
event_kind TEXT NOT NULL,
order_action TEXT NOT NULL,
pool_account TEXT NULL,
market_account TEXT NULL,
actor_wallet TEXT NULL,
order_account TEXT NULL,
base_token_mint TEXT NULL,
quote_token_mint TEXT NULL,
amount_raw TEXT NULL,
amount_min_raw TEXT NULL,
tick_index INTEGER NULL,
zero_for_one INTEGER NULL,
payload_json TEXT NOT NULL,
executed_at TEXT NOT NULL,
created_at TEXT NOT NULL
)
"#,
)
.await;
}
/// Creates index on `k_sol_orderbook_events(transaction_id)`.
async fn create_idx_orderbook_events_transaction_id(
pool: &sqlx::SqlitePool,
) -> Result<(), crate::Error> {
return execute_sqlite_schema_statement(
pool,
"create_idx_orderbook_events_transaction_id",
r#"
CREATE INDEX IF NOT EXISTS idx_orderbook_events_transaction_id
ON k_sol_orderbook_events (transaction_id)
"#,
)
.await;
}
/// Creates index on `k_sol_orderbook_events(pool_id)`.
async fn create_idx_orderbook_events_pool_id(pool: &sqlx::SqlitePool) -> Result<(), crate::Error> {
return execute_sqlite_schema_statement(
pool,
"create_idx_orderbook_events_pool_id",
r#"
CREATE INDEX IF NOT EXISTS idx_orderbook_events_pool_id
ON k_sol_orderbook_events (pool_id)
"#,
)
.await;
}
/// Creates partial unique index on `k_sol_orderbook_events(decoded_event_id)`.
async fn create_uix_orderbook_events_decoded_event_id(
pool: &sqlx::SqlitePool,
) -> Result<(), crate::Error> {
return execute_sqlite_schema_statement(
pool,
"create_uix_orderbook_events_decoded_event_id",
r#"
CREATE UNIQUE INDEX IF NOT EXISTS uix_orderbook_events_decoded_event_id
ON k_sol_orderbook_events (decoded_event_id)
WHERE decoded_event_id IS NOT NULL
"#,
)
.await;
}

4
kb_lib/src/dex.rs
View File

@@ -70,12 +70,16 @@ pub use raydium_amm_v4::RaydiumAmmV4DecodedEvent;
pub use raydium_amm_v4::RaydiumAmmV4Decoder;
pub use raydium_amm_v4::RaydiumAmmV4Initialize2PoolDecoded;
pub use raydium_amm_v4::RaydiumAmmV4SwapDecoded;
pub use raydium_clmm::RaydiumClmmCollectProtocolFeeDecoded;
pub use raydium_clmm::RaydiumClmmCreatePoolDecoded;
pub use raydium_clmm::RaydiumClmmDecodedEvent;
pub use raydium_clmm::RaydiumClmmDecodedInstructionEvent;
pub use raydium_clmm::RaydiumClmmProgramDataEventDecoded;
pub use raydium_clmm::RaydiumClmmDecoder;
pub use raydium_clmm::RaydiumClmmSwapLegacyDecoded;
pub use raydium_clmm::RaydiumClmmSwapV2Decoded;
pub use raydium_clmm::decode_raydium_clmm_instruction;
pub use raydium_clmm::decode_raydium_clmm_program_data_event;
pub use raydium_cpmm::RaydiumCpmmDecodedEvent;
pub use raydium_cpmm::RaydiumCpmmLpChangeEventDecoded;
pub use raydium_cpmm::RaydiumCpmmSwapDecoded;

1094
kb_lib/src/dex/raydium_clmm.rs
View File

File diff suppressed because it is too large Load Diff

1365
kb_lib/src/dex_decode.rs
View File

File diff suppressed because it is too large Load Diff

16
kb_lib/src/dex_decoded_event_materialization.rs
View File

@@ -148,7 +148,7 @@ fn prepare_payload_for_transaction_status(
transaction: &crate::ChainTransactionDto,
payload_json: serde_json::Value,
) -> serde_json::Value {
if transaction.err_json.is_none() {
if !transaction_has_effective_error(transaction) {
return payload_json;
}
let mut object = match payload_json {
@@ -177,3 +177,17 @@ fn prepare_payload_for_transaction_status(
);
return serde_json::Value::Object(object);
}
fn transaction_has_effective_error(transaction: &crate::ChainTransactionDto) -> bool {
let err_json = match transaction.err_json.as_ref() {
Some(err_json) => err_json.trim(),
None => return false,
};
if err_json.is_empty() {
return false;
}
if err_json == "null" {
return false;
}
return true;
}

50
kb_lib/src/dex_event_classification.rs
View File

@@ -238,9 +238,15 @@ pub fn classify_dex_event_actionability(
if trade_candidate {
return DexEventActionability::TradeCandidate;
}
if is_dex_informational_event_kind(event_kind) {
return DexEventActionability::Informational;
}
if is_dex_trade_event_kind(event_kind) {
return DexEventActionability::NonActionableTrade;
}
if is_dex_orderbook_event_kind(event_kind) {
return DexEventActionability::NonTradeUseful;
}
let category = classify_dex_event_category(event_kind);
match category {
DexEventCategory::Liquidity => return DexEventActionability::NonTradeUseful,
@@ -323,6 +329,12 @@ pub fn is_dex_liquidity_event_kind(event_kind: &str) -> bool {
if event_kind.contains(".lp_change_event") {
return true;
}
if event_kind.contains(".liquidity_change_event") {
return true;
}
if event_kind.contains(".liquidity_calculate_event") {
return true;
}
if event_kind.contains(".withdraw") {
return true;
}
@@ -341,6 +353,9 @@ pub fn is_dex_liquidity_event_kind(event_kind: &str) -> bool {
if event_kind.contains(".initialize_position") {
return true;
}
if event_kind.contains(".create_personal_position_event") {
return true;
}
if event_kind.contains(".open_position") {
return true;
}
@@ -440,6 +455,38 @@ pub fn is_dex_reward_event_kind(event_kind: &str) -> bool {
return false;
}
/// Returns true for orderbook or limit-order events that must not become candles.
pub fn is_dex_orderbook_event_kind(event_kind: &str) -> bool {
if event_kind.contains(".order_place") {
return true;
}
if event_kind.contains(".order_cancel") {
return true;
}
if event_kind.contains(".order_fill") {
return true;
}
if event_kind.contains(".settle_funds") {
return true;
}
if event_kind.contains(".open_limit_order") {
return true;
}
if event_kind.contains(".increase_limit_order") {
return true;
}
if event_kind.contains(".decrease_limit_order") {
return true;
}
if event_kind.contains(".close_limit_order") {
return true;
}
if event_kind.contains(".settle_limit_order") {
return true;
}
return false;
}
/// Returns true for pool, pair, launch, mint, burn or migration lifecycle events.
pub fn is_dex_pool_lifecycle_event_kind(event_kind: &str) -> bool {
if event_kind.contains(".create_lock_escrow") {
@@ -539,6 +586,9 @@ pub fn is_dex_pool_creation_event_kind(event_kind: &str) -> bool {
if event_kind.contains(".create_pool") {
return true;
}
if event_kind.contains(".pool_created_event") {
return true;
}
if event_kind.contains(".create_amm") {
return true;
}

201
kb_lib/src/dex_event_coverage.rs
View File

@@ -214,6 +214,9 @@ fn infer_expected_db_target_for_entry(
if decoder_code == "raydium_cpmm" && entry_name == "swap_event" {
return Some(crate::DexEventCoverageEntryDto::DB_TARGET_DECODED_EVENTS_ONLY.to_string());
}
if decoder_code == "raydium_clmm" && entry_name == "initialize_reward" {
return Some(crate::DexEventCoverageEntryDto::DB_TARGET_REWARD_EVENTS.to_string());
}
return infer_expected_db_target(event_family, entry_kind);
}
@@ -238,8 +241,8 @@ fn infer_expected_db_target(
"liquidity" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS,
"liquidity_add" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS,
"liquidity_remove" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS,
"position_open" => crate::DexEventCoverageEntryDto::DB_TARGET_POOL_LIFECYCLE_EVENTS,
"position_close" => crate::DexEventCoverageEntryDto::DB_TARGET_POOL_LIFECYCLE_EVENTS,
"position_open" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS,
"position_close" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS,
"fee" => crate::DexEventCoverageEntryDto::DB_TARGET_FEE_EVENTS,
"reward" => crate::DexEventCoverageEntryDto::DB_TARGET_REWARD_EVENTS,
"admin_config" => crate::DexEventCoverageEntryDto::DB_TARGET_POOL_ADMIN_EVENTS,
@@ -283,6 +286,7 @@ fn infer_event_family(
return Some("swap".to_string());
}
if contains_any(normalized.as_str(), &["create_pool", "initialize_pool", "initialize2"])
|| normalized == "create_customizable_pool"
|| normalized == "initialize"
|| normalized.starts_with("initialize_")
{
@@ -306,11 +310,13 @@ fn infer_event_family(
}
if contains_any(normalized.as_str(), &["close_position", "position_close"])
|| normalized.contains("close_position_if_empty")
|| normalized == "close_protocol_position"
{
return Some("position_close".to_string());
}
if contains_any(normalized.as_str(), &["fee", "collect", "claim_fee"])
&& !normalized.contains("reward")
&& !normalized.contains("config")
{
return Some("fee".to_string());
}
@@ -323,6 +329,9 @@ fn infer_event_family(
) {
return Some("admin_config".to_string());
}
if normalized == "create_support_mint_associated" {
return Some("account_create".to_string());
}
if normalized.contains("mint") {
return Some("mint".to_string());
}
@@ -335,21 +344,24 @@ fn infer_event_family(
if contains_any(normalized.as_str(), &["create_ata", "init_account", "open_orders_create"]) {
return Some("account_create".to_string());
}
if contains_any(normalized.as_str(), &["close_account", "close_open_orders"])
|| normalized.starts_with("close_")
{
return Some("account_close".to_string());
}
if normalized.contains("wrap_sol") {
return Some("wrap_sol".to_string());
}
if normalized.contains("unwrap_sol") {
return Some("unwrap_sol".to_string());
}
if normalized.contains("place_order") || normalized.contains("post_order") {
if normalized.contains("place_order")
|| normalized.contains("post_order")
|| normalized == "open_limit_order"
|| normalized == "increase_limit_order"
{
return Some("order_place".to_string());
}
if normalized.contains("cancel_order") || normalized.contains("cancel_all") {
if normalized.contains("cancel_order")
|| normalized.contains("cancel_all")
|| normalized == "close_limit_order"
|| normalized == "decrease_limit_order"
{
return Some("order_cancel".to_string());
}
if normalized.contains("fill") {
@@ -358,9 +370,14 @@ fn infer_event_family(
if normalized.contains("consume_events") {
return Some("consume_events".to_string());
}
if normalized.contains("settle_funds") {
if normalized.contains("settle_funds") || normalized == "settle_limit_order" {
return Some("settle_funds".to_string());
}
if contains_any(normalized.as_str(), &["close_account", "close_open_orders"])
|| normalized.starts_with("close_")
{
return Some("account_close".to_string());
}
if normalized.contains("vault") && normalized.contains("deposit") {
return Some("vault_deposit".to_string());
}
@@ -399,7 +416,7 @@ fn contains_any(value: &str, needles: &[&str]) -> bool {
return false;
}
fn known_local_event_kind(
pub(crate) fn known_local_event_kind(
decoder_code: &str,
entry_name: &str,
) -> std::option::Option<std::string::String> {
@@ -444,19 +461,98 @@ fn known_local_event_kind(
return Some("raydium_cpmm.update_pool_status".to_string());
},
("raydium_cpmm", "withdraw") => return Some("raydium_cpmm.withdraw".to_string()),
("raydium_clmm", "swap") => return Some("raydium_clmm.swap".to_string()),
("raydium_clmm", "swap_v2") => return Some("raydium_clmm.swap_v2".to_string()),
("raydium_clmm", "increase_liquidity_v2") => {
return Some("raydium_clmm.increase_liquidity_v2".to_string());
("raydium_clmm", "close_limit_order") => {
return Some("raydium_clmm.close_limit_order".to_string());
},
("raydium_clmm", "open_limit_order") => {
return Some("raydium_clmm.open_limit_order".to_string());
},
("raydium_clmm", "increase_limit_order") => {
return Some("raydium_clmm.increase_limit_order".to_string());
},
("raydium_clmm", "decrease_limit_order") => {
return Some("raydium_clmm.decrease_limit_order".to_string());
},
("raydium_clmm", "close_position") => {
return Some("raydium_clmm.close_position".to_string());
},
("raydium_clmm", "close_protocol_position") => {
return Some("raydium_clmm.close_protocol_position".to_string());
},
("raydium_clmm", "collect_fund_fee") => {
return Some("raydium_clmm.collect_fund_fee".to_string());
},
("raydium_clmm", "collect_protocol_fee") => {
return Some("raydium_clmm.collect_protocol_fee".to_string());
},
("raydium_clmm", "collect_remaining_rewards") => {
return Some("raydium_clmm.collect_remaining_rewards".to_string());
},
("raydium_clmm", "create_amm_config") => {
return Some("raydium_clmm.create_amm_config".to_string());
},
("raydium_clmm", "create_customizable_pool") => {
return Some("raydium_clmm.create_customizable_pool".to_string());
},
("raydium_clmm", "create_dynamic_fee_config") => {
return Some("raydium_clmm.create_dynamic_fee_config".to_string());
},
("raydium_clmm", "create_operation_account") => {
return Some("raydium_clmm.create_operation_account".to_string());
},
("raydium_clmm", "create_pool") => return Some("raydium_clmm.create_pool".to_string()),
("raydium_clmm", "create_support_mint_associated") => {
return Some("raydium_clmm.create_support_mint_associated".to_string());
},
("raydium_clmm", "decrease_liquidity") => {
return Some("raydium_clmm.decrease_liquidity".to_string());
},
("raydium_clmm", "decrease_liquidity_v2") => {
return Some("raydium_clmm.decrease_liquidity_v2".to_string());
},
("raydium_clmm", "increase_liquidity") => {
return Some("raydium_clmm.increase_liquidity".to_string());
},
("raydium_clmm", "increase_liquidity_v2") => {
return Some("raydium_clmm.increase_liquidity_v2".to_string());
},
("raydium_clmm", "initialize_reward") => {
return Some("raydium_clmm.initialize_reward".to_string());
},
("raydium_clmm", "open_position") => {
return Some("raydium_clmm.open_position".to_string());
},
("raydium_clmm", "open_position_v2") => {
return Some("raydium_clmm.open_position_v2".to_string());
},
("raydium_clmm", "open_position_with_token22_nft") => {
return Some("raydium_clmm.open_position_with_token22_nft".to_string());
},
("raydium_clmm", "close_position") => {
return Some("raydium_clmm.close_position".to_string());
("raydium_clmm", "set_reward_params") => {
return Some("raydium_clmm.set_reward_params".to_string());
},
("raydium_clmm", "settle_limit_order") => {
return Some("raydium_clmm.settle_limit_order".to_string());
},
("raydium_clmm", "swap") => return Some("raydium_clmm.swap".to_string()),
("raydium_clmm", "swap_router_base_in") => {
return Some("raydium_clmm.swap_router_base_in".to_string());
},
("raydium_clmm", "swap_v2") => return Some("raydium_clmm.swap_v2".to_string()),
("raydium_clmm", "transfer_reward_owner") => {
return Some("raydium_clmm.transfer_reward_owner".to_string());
},
("raydium_clmm", "update_amm_config") => {
return Some("raydium_clmm.update_amm_config".to_string());
},
("raydium_clmm", "update_operation_account") => {
return Some("raydium_clmm.update_operation_account".to_string());
},
("raydium_clmm", "update_pool_status") => {
return Some("raydium_clmm.update_pool_status".to_string());
},
("raydium_clmm", "update_reward_infos") => {
return Some("raydium_clmm.update_reward_infos".to_string());
},
_ => return None,
}
@@ -511,6 +607,77 @@ mod tests {
);
}
#[test]
fn event_family_inference_covers_raydium_clmm_idl_entries() {
assert_eq!(
super::infer_event_family("create_customizable_pool", crate::ENTRY_KIND_INSTRUCTION),
Some("pool_create".to_string())
);
assert_eq!(
super::infer_event_family("create_dynamic_fee_config", crate::ENTRY_KIND_INSTRUCTION),
Some("admin_config".to_string())
);
assert_eq!(
super::infer_event_family(
"create_support_mint_associated",
crate::ENTRY_KIND_INSTRUCTION,
),
Some("account_create".to_string())
);
assert_eq!(
super::infer_event_family("close_limit_order", crate::ENTRY_KIND_INSTRUCTION),
Some("order_cancel".to_string())
);
assert_eq!(
super::infer_event_family("open_limit_order", crate::ENTRY_KIND_INSTRUCTION),
Some("order_place".to_string())
);
assert_eq!(
super::infer_event_family("increase_limit_order", crate::ENTRY_KIND_INSTRUCTION),
Some("order_place".to_string())
);
assert_eq!(
super::infer_event_family("decrease_limit_order", crate::ENTRY_KIND_INSTRUCTION),
Some("order_cancel".to_string())
);
assert_eq!(
super::infer_event_family("close_protocol_position", crate::ENTRY_KIND_INSTRUCTION),
Some("position_close".to_string())
);
assert_eq!(
super::infer_event_family("settle_limit_order", crate::ENTRY_KIND_INSTRUCTION),
Some("settle_funds".to_string())
);
assert_eq!(
super::infer_expected_db_target(Some("position_open"), crate::ENTRY_KIND_INSTRUCTION),
Some(crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS.to_string())
);
assert_eq!(
super::infer_expected_db_target(Some("position_close"), crate::ENTRY_KIND_INSTRUCTION),
Some(crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS.to_string())
);
assert_eq!(
super::known_local_event_kind("raydium_clmm", "create_pool"),
Some("raydium_clmm.create_pool".to_string())
);
assert_eq!(
super::known_local_event_kind("raydium_clmm", "collect_protocol_fee"),
Some("raydium_clmm.collect_protocol_fee".to_string())
);
assert_eq!(
super::known_local_event_kind("raydium_clmm", "open_limit_order"),
Some("raydium_clmm.open_limit_order".to_string())
);
assert_eq!(
super::known_local_event_kind("raydium_clmm", "increase_limit_order"),
Some("raydium_clmm.increase_limit_order".to_string())
);
assert_eq!(
super::known_local_event_kind("raydium_clmm", "decrease_limit_order"),
Some("raydium_clmm.decrease_limit_order".to_string())
);
}
#[tokio::test]
async fn sync_upstream_registry_persists_raydium_cpmm_coverage_rows() {
let database = make_database().await;

119
kb_lib/src/instruction_observation_index.rs
View File

@@ -60,6 +60,19 @@ impl InstructionObservationIndexService {
return self.upsert_source_rows(rows).await;
}
/// Refreshes observations for the same transaction window used by local replay.
pub async fn refresh_replay_window(
&self,
limit: std::option::Option<i64>,
) -> Result<crate::InstructionObservationIndexRefreshResult, crate::Error> {
let rows_result = self.list_replay_window_source_rows(limit).await;
let rows = match rows_result {
Ok(rows) => rows,
Err(error) => return Err(error),
};
return self.upsert_source_rows(rows).await;
}
/// Refreshes observations for recently persisted instructions.
pub async fn refresh_recent(
&self,
@@ -145,6 +158,73 @@ ORDER BY ins.instruction_index ASC, ins.inner_instruction_index ASC, ins.id ASC
}
}
async fn list_replay_window_source_rows(
&self,
limit: std::option::Option<i64>,
) -> Result<std::vec::Vec<InstructionObservationSourceRow>, crate::Error> {
let effective_limit = match limit {
Some(limit) => {
if limit <= 0 {
10_000
} else {
limit
}
},
None => 10_000,
};
match self.database.connection() {
crate::DatabaseConnection::Sqlite(pool) => {
let query_result = sqlx::query_as::<sqlx::Sqlite, InstructionObservationSourceRow>(
r#"
WITH replay_transactions AS (
SELECT id
FROM k_sol_chain_transactions
ORDER BY id ASC
LIMIT ?
)
SELECT
tx.id AS transaction_id,
tx.signature AS signature,
tx.slot AS slot,
tx.block_time_unix AS block_time,
tx.err_json AS err_json,
ins.id AS instruction_id,
ins.parent_instruction_id AS parent_instruction_id,
ins.instruction_index AS instruction_index,
ins.inner_instruction_index AS inner_instruction_index,
ins.program_id AS program_id,
ins.accounts_json AS accounts_json,
ins.data_json AS data_json,
de.pool_account AS pool_account,
de.event_kind AS decoded_event_kind,
de.id AS decoded_event_id
FROM k_sol_chain_instructions ins
JOIN replay_transactions replay_tx
ON replay_tx.id = ins.transaction_id
JOIN k_sol_chain_transactions tx
ON tx.id = ins.transaction_id
LEFT JOIN k_sol_dex_decoded_events de
ON de.transaction_id = tx.id
AND de.instruction_id = ins.id
ORDER BY tx.id ASC, ins.instruction_index ASC, ins.inner_instruction_index ASC, ins.id ASC
"#,
)
.bind(effective_limit)
.fetch_all(pool)
.await;
match query_result {
Ok(rows) => return Ok(rows),
Err(error) => {
return Err(crate::Error::Db(format!(
"cannot list instruction observation source rows for replay window: {}",
error
)));
},
}
},
}
}
async fn list_recent_source_rows(
&self,
limit: u32,
@@ -280,6 +360,45 @@ fn resolve_instruction_name(
};
return Some(name.to_string());
}
if program_id == crate::RAYDIUM_CLMM_PROGRAM_ID || decoder_code == Some("raydium_clmm") {
let name = match discriminator_hex {
"4c7c800fd55725fa" => "raydium_clmm.close_limit_order",
"9d20dab7471d1293" => "raydium_clmm.open_limit_order",
"b19059ecfaba7d63" => "raydium_clmm.increase_limit_order",
"759d3c674231a300" => "raydium_clmm.decrease_limit_order",
"7b86510031446262" => "raydium_clmm.close_position",
"c975989055556cb2" => "raydium_clmm.close_protocol_position",
"a78a4e95dfc2067e" => "raydium_clmm.collect_fund_fee",
"8888fcddc2427e59" => "raydium_clmm.collect_protocol_fee",
"12eda6c52210d590" => "raydium_clmm.collect_remaining_rewards",
"8934edd4d7756c68" => "raydium_clmm.create_amm_config",
"2b44d4a7592fa401" => "raydium_clmm.create_customizable_pool",
"bd0eb5785576e33e" => "raydium_clmm.create_dynamic_fee_config",
"3f5794216d230868" => "raydium_clmm.create_operation_account",
"e992d18ecf6840bc" => "raydium_clmm.create_pool",
"11fb415c88f20ea9" => "raydium_clmm.create_support_mint_associated",
"a026d06f685b2c01" => "raydium_clmm.decrease_liquidity",
"3a7fbc3e4f52c460" => "raydium_clmm.decrease_liquidity_v2",
"2e9cf3760dcdfbb2" => "raydium_clmm.increase_liquidity",
"851d59df45eeb00a" => "raydium_clmm.increase_liquidity_v2",
"5f87c0c4f281e644" => "raydium_clmm.initialize_reward",
"87802f4d0f98f031" => "raydium_clmm.open_position",
"4db84ad67056f1c7" => "raydium_clmm.open_position_v2",
"4dffae527d1dc92e" => "raydium_clmm.open_position_with_token22_nft",
"7034a74b20c9d389" => "raydium_clmm.set_reward_params",
"cd4e74215c691a60" => "raydium_clmm.settle_limit_order",
"f8c69e91e17587c8" => "raydium_clmm.swap",
"457d73daf5baf2c4" => "raydium_clmm.swap_router_base_in",
"2b04ed0b1ac91e62" => "raydium_clmm.swap_v2",
"07160c53f22b3079" => "raydium_clmm.transfer_reward_owner",
"313cae889a1c74c8" => "raydium_clmm.update_amm_config",
"7f467728bce33d07" => "raydium_clmm.update_operation_account",
"82576c062ee0757b" => "raydium_clmm.update_pool_status",
"a3ace0340b9a6adf" => "raydium_clmm.update_reward_infos",
_ => return None,
};
return Some(name.to_string());
}
return None;
}

22
kb_lib/src/lib.rs
View File

@@ -585,6 +585,9 @@ pub use db::ObservedTokenStatus;
pub use db::OnchainObservationDto;
/// Persisted on-chain observation row.
pub use db::OnchainObservationEntity;
/// Application-facing normalized orderbook or limit-order event DTO.
pub use db::OrderbookEventDto;
pub use db::OrderbookEventEntity;
/// Application-facing pair-analytic-signal DTO.
pub use db::PairAnalyticSignalDto;
/// Persisted pair-analytic-signal row.
@@ -737,10 +740,13 @@ pub use db::query_dex_decode_replay_ledger_get_by_transaction;
pub use db::query_dex_decode_replay_ledger_upsert;
/// Deletes one decoded DEX event row by its natural key.
pub use db::query_dex_decoded_events_delete_by_key;
/// Deletes upstream registry instruction-match rows already covered by specialized local decoders.
pub use db::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches;
/// Deletes Meteora DLMM Anchor self-CPI swap audit rows already covered by decoded swaps.
pub use db::query_dex_decoded_events_delete_meteora_dlmm_anchor_swap_instruction_audits;
/// Deletes decoded DEX instruction audit rows related to one decoded instruction.
pub use db::query_dex_decoded_events_delete_related_instruction_audit;
pub use db::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits;
/// Reads one decoded DEX event by its natural key.
pub use db::query_dex_decoded_events_get_by_key;
/// Returns the latest Pump.fun create payload associated with a token mint.
@@ -864,6 +870,8 @@ pub use db::query_observed_tokens_upsert;
pub use db::query_onchain_observations_insert;
/// Lists recent on-chain observations ordered from newest to oldest.
pub use db::query_onchain_observations_list_recent;
/// Inserts or updates one normalized orderbook event row.
pub use db::query_orderbook_events_upsert;
/// Returns one pair-analytic-signal row identified by its key, if it exists.
pub use db::query_pair_analytic_signals_get_by_key;
/// Lists all pair-analytic signals for one pair ordered by key.
@@ -1138,12 +1146,18 @@ pub use dex::RaydiumAmmV4Decoder;
pub use dex::RaydiumAmmV4Initialize2PoolDecoded;
/// Decoded Raydium AMM v4 swap event.
pub use dex::RaydiumAmmV4SwapDecoded;
/// Decoded Raydium CLMM collect_protocol_fee instruction.
pub use dex::RaydiumClmmCollectProtocolFeeDecoded;
/// Decoded Raydium CLMM create_pool instruction.
pub use dex::RaydiumClmmCreatePoolDecoded;
/// Decoded Raydium CLMM event.
pub use dex::RaydiumClmmDecodedEvent;
/// Decoded Raydium CLMM instruction event with projected instruction id.
pub use dex::RaydiumClmmDecodedInstructionEvent;
/// Raydium CLMM transaction decoder.
pub use dex::RaydiumClmmDecoder;
/// Decoded Raydium CLMM Anchor Program data event payload.
pub use dex::RaydiumClmmProgramDataEventDecoded;
/// Decoded Raydium CLMM legacy swap event.
pub use dex::RaydiumClmmSwapLegacyDecoded;
/// Decoded Raydium CLMM swap_v2 instruction.
@@ -1162,6 +1176,8 @@ pub use dex::RaydiumCpmmSwapMode;
pub use dex::classify_raydium_cpmm_instruction_data;
/// Decodes a Raydium CLMM instruction.
pub use dex::decode_raydium_clmm_instruction;
/// Decodes one Raydium CLMM Anchor Program data event.
pub use dex::decode_raydium_clmm_program_data_event;
/// Decodes one Raydium CPMM instruction from projected instruction fields.
pub use dex::decode_raydium_cpmm_instruction;
/// Decodes Raydium CPMM Anchor events emitted in `Program data:` logs.
@@ -1217,6 +1233,8 @@ pub use dex_event_classification::is_dex_liquidity_event_kind;
pub use dex_event_classification::is_dex_liquidity_remove_event_kind;
/// Returns true for migration DEX events.
pub use dex_event_classification::is_dex_migration_event_kind;
/// Returns true for orderbook or limit-order events that must not become candles.
pub use dex_event_classification::is_dex_orderbook_event_kind;
/// Returns true for pair creation DEX events.
pub use dex_event_classification::is_dex_pair_creation_event_kind;
/// Returns true for pool creation DEX events.
@@ -1408,6 +1426,10 @@ pub use solana_pubsub_ws::parse_solana_ws_typed_notification_from_event;
pub use token_backfill::PoolBackfillResult;
/// One signature-backfill result summary.
pub use token_backfill::SignatureBackfillResult;
/// One item produced by a batch signature backfill.
pub use token_backfill::SignatureBatchBackfillItemResult;
/// Batch signature-backfill result summary.
pub use token_backfill::SignatureBatchBackfillResult;
/// One token-backfill result summary.
pub use token_backfill::TokenBackfillResult;
/// Historical token backfill service.

143
kb_lib/src/local_pipeline_replay.rs
View File

@@ -13,6 +13,10 @@ fn default_skip_certified_dex_decode() -> bool {
return true;
}
fn default_defer_instruction_observation_index_refresh() -> bool {
return true;
}
/// Configuration for a local pipeline replay pass.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
#[serde(rename_all = "camelCase")]
@@ -31,6 +35,9 @@ pub struct LocalPipelineReplayConfig {
/// Whether DEX decoding must run even when the replay ledger certifies a safe prior pass.
#[serde(default)]
pub force_decode_replay: bool,
/// Whether instruction observation indexing is deferred and refreshed once after replay.
#[serde(default = "default_defer_instruction_observation_index_refresh")]
pub defer_instruction_observation_index_refresh: bool,
}
impl Default for LocalPipelineReplayConfig {
@@ -42,6 +49,7 @@ impl Default for LocalPipelineReplayConfig {
reset_market_materialization_before_replay: true,
skip_certified_dex_decode: true,
force_decode_replay: false,
defer_instruction_observation_index_refresh: true,
};
}
}
@@ -90,6 +98,8 @@ pub struct LocalPipelineReplayResult {
pub reward_event_count: usize,
/// Total pool administration event materialization results returned by replayed non-trade calls.
pub pool_admin_event_count: usize,
/// Total orderbook event materialization results returned by replayed non-trade calls.
pub orderbook_event_count: usize,
/// Total candle upsert results returned by replayed candle calls.
///
/// This is a replay write/result counter, not the number of distinct rows
@@ -111,6 +121,10 @@ pub struct LocalPipelineReplayResult {
pub pair_symbol_updated_count: usize,
/// Number of derived market materialization rows deleted before replay.
pub reset_market_materialization_deleted_count: u64,
/// Total instruction source rows scanned by the observation index refresh.
pub instruction_observation_scanned_count: usize,
/// Total instruction-observation rows upserted by the observation index refresh.
pub instruction_observation_upserted_count: usize,
/// Number of errors outside per-signature replay.
pub global_error_count: usize,
}
@@ -352,6 +366,7 @@ impl LocalPipelineReplayService {
result.fee_event_count += non_trade_result.fee_event_count;
result.reward_event_count += non_trade_result.reward_event_count;
result.pool_admin_event_count += non_trade_result.pool_admin_event_count;
result.orderbook_event_count += non_trade_result.orderbook_event_count;
},
Err(error) => {
result.non_trade_materialization_error_count += 1;
@@ -426,25 +441,55 @@ impl LocalPipelineReplayService {
);
},
}
if !config.defer_instruction_observation_index_refresh {
let instruction_index_result =
instruction_observation_index.refresh_signature(signature.as_str()).await;
match instruction_index_result {
Ok(index_result) => {
result.instruction_observation_scanned_count +=
index_result.scanned_instruction_count;
result.instruction_observation_upserted_count +=
index_result.upserted_observation_count;
tracing::debug!(
signature = %signature,
upserted_observation_count = index_result.upserted_observation_count,
"instruction observation index refreshed during local replay"
);
},
Err(error) => {
tracing::warn!(
signature = %signature,
error = %error,
"instruction observation index refresh failed during local replay"
);
},
}
}
result.replayed_transaction_count += 1;
}
if config.defer_instruction_observation_index_refresh {
let instruction_index_result =
instruction_observation_index.refresh_signature(signature.as_str()).await;
instruction_observation_index.refresh_replay_window(config.limit).await;
match instruction_index_result {
Ok(index_result) => {
result.instruction_observation_scanned_count +=
index_result.scanned_instruction_count;
result.instruction_observation_upserted_count +=
index_result.upserted_observation_count;
tracing::debug!(
signature = %signature,
scanned_instruction_count = index_result.scanned_instruction_count,
upserted_observation_count = index_result.upserted_observation_count,
"instruction observation index refreshed during local replay"
"instruction observation index refreshed after local replay"
);
},
Err(error) => {
result.global_error_count += 1;
tracing::warn!(
signature = %signature,
error = %error,
"instruction observation index refresh failed during local replay"
"instruction observation index refresh failed after local replay"
);
},
}
result.replayed_transaction_count += 1;
}
if config.refresh_missing_token_metadata {
let metadata_service = match &self.http_pool {
@@ -476,6 +521,52 @@ impl LocalPipelineReplayService {
}
async fn refresh_event_coverage_best_effort(&self) {
let cleanup_result =
crate::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits(
self.database.as_ref(),
None,
)
.await;
match cleanup_result {
Ok(deleted_count) => {
if deleted_count > 0 {
tracing::info!(
deleted_count = deleted_count,
"replaced Raydium CLMM instruction audits cleaned before dex event coverage refresh"
);
}
},
Err(error) => {
tracing::warn!(
error = %error,
"Raydium CLMM replaced instruction-audit cleanup failed before dex event coverage refresh"
);
},
}
let upstream_cleanup_result =
crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(
self.database.as_ref(),
None,
)
.await;
match upstream_cleanup_result {
Ok(deleted_count) => {
if deleted_count > 0 {
tracing::info!(
deleted_count = deleted_count,
"locally covered upstream instruction matches cleaned before dex event coverage refresh"
);
}
},
Err(error) => {
tracing::warn!(
error = %error,
"locally covered upstream instruction-match cleanup failed before dex event coverage refresh"
);
},
}
let coverage_service = crate::DexEventCoverageService::new(self.database.clone());
let refresh_result = coverage_service.refresh_local_counts(None).await;
match refresh_result {
@@ -494,6 +585,46 @@ impl LocalPipelineReplayService {
);
},
}
let post_refresh_upstream_cleanup_result =
crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(
self.database.as_ref(),
None,
)
.await;
match post_refresh_upstream_cleanup_result {
Ok(deleted_count) => {
if deleted_count > 0 {
tracing::info!(
deleted_count = deleted_count,
"locally covered upstream instruction matches cleaned after dex event coverage refresh"
);
let second_refresh_result = coverage_service.refresh_local_counts(None).await;
match second_refresh_result {
Ok(second_refresh_result) => {
tracing::debug!(
upserted_entry_count = second_refresh_result.upserted_entry_count,
refreshed_entry_count = second_refresh_result.refreshed_entry_count,
summary_count = second_refresh_result.summaries.len(),
"dex event coverage refreshed after upstream instruction-match cleanup"
);
},
Err(error) => {
tracing::warn!(
error = %error,
"dex event coverage refresh failed after upstream instruction-match cleanup"
);
},
}
}
},
Err(error) => {
tracing::warn!(
error = %error,
"locally covered upstream instruction-match cleanup failed after dex event coverage refresh"
);
},
}
}
async fn get_certified_dex_decode_skip_ledger(

280
kb_lib/src/non_trade_event_materialization.rs
View File

@@ -19,6 +19,8 @@ pub struct NonTradeEventMaterializationResult {
pub reward_event_count: usize,
/// Number of pool administration events inserted or refreshed.
pub pool_admin_event_count: usize,
/// Number of orderbook or limit-order events inserted or refreshed.
pub orderbook_event_count: usize,
}
/// Materializes useful non-trade decoded DEX events.
@@ -61,7 +63,7 @@ impl NonTradeEventMaterializationService {
)));
},
};
if transaction.err_json.is_some() {
if transaction_has_effective_error(&transaction) {
tracing::debug!(
signature = %transaction.signature,
"skipping non-trade materialization for failed transaction"
@@ -189,6 +191,24 @@ impl NonTradeEventMaterializationService {
Err(error) => return Err(error),
}
}
if crate::is_dex_orderbook_event_kind(decoded_event.event_kind.as_str()) {
let materialized = self
.materialize_orderbook_event(
&transaction,
transaction_id,
decoded_event,
&payload,
)
.await;
match materialized {
Ok(was_materialized) => {
if was_materialized {
result.orderbook_event_count += 1;
}
},
Err(error) => return Err(error),
}
}
}
for decoded_event in &decoded_events {
if !decoded_event.event_kind.ends_with(".lp_change_event") {
@@ -673,6 +693,86 @@ WHERE decoded_event_id = ?
}
}
async fn materialize_orderbook_event(
&self,
transaction: &crate::ChainTransactionDto,
transaction_id: i64,
decoded_event: &crate::DexDecodedEventDto,
payload: &serde_json::Value,
) -> Result<bool, crate::Error> {
let decoded_event_id = match decoded_event.id {
Some(decoded_event_id) => decoded_event_id,
None => return Ok(false),
};
let context = self.resolve_decoded_event_context(decoded_event).await;
let context = match context {
Ok(context) => context,
Err(error) => return Err(error),
};
let order_action = normalize_orderbook_action(decoded_event.event_kind.as_str());
let actor_wallet = extract_first_string(
payload,
&["actorWallet", "actor_wallet", "owner", "authority", "payer", "user"],
);
let order_account = match extract_first_string(
payload,
&["orderAccount", "order_account", "limitOrder", "limit_order", "order"],
) {
Some(order_account) => Some(order_account),
None => fallback_order_account(decoded_event.event_kind.as_str(), payload),
};
let amount_raw = extract_first_amount_string(
payload,
&[
"amountRaw",
"amount_raw",
"amount",
"decreasedAmountRaw",
"decreased_amount_raw",
"decreasedAmount",
"increasedAmountRaw",
"increased_amount_raw",
"increasedAmount",
],
);
let amount_min_raw = extract_first_amount_string(
payload,
&["amountMinRaw", "amount_min_raw", "amountMin", "amount_min"],
);
let tick_index = extract_first_i64(payload, &["tickIndex", "tick_index"]);
let zero_for_one = extract_first_bool(payload, &["zeroForOne", "zero_for_one"]);
let dto = crate::OrderbookEventDto::new(
transaction_id,
Some(decoded_event_id),
context.dex_id,
context.pool_id,
context.pair_id,
transaction.signature.clone(),
transaction.slot,
decoded_event.protocol_name.clone(),
decoded_event.program_id.clone(),
decoded_event.event_kind.clone(),
order_action,
decoded_event.pool_account.clone(),
decoded_event.market_account.clone(),
actor_wallet,
order_account,
decoded_event.token_a_mint.clone(),
decoded_event.token_b_mint.clone(),
amount_raw,
amount_min_raw,
tick_index,
zero_for_one,
decoded_event.payload_json.clone(),
);
let upsert_result =
crate::query_orderbook_events_upsert(self.database.as_ref(), &dto).await;
match upsert_result {
Ok(_) => return Ok(true),
Err(error) => return Err(error),
}
}
async fn ensure_liquidity_context_from_decoded_event(
&self,
decoded_event: &crate::DexDecodedEventDto,
@@ -789,6 +889,162 @@ WHERE decoded_event_id = ?
}
}
fn normalize_orderbook_action(event_kind: &str) -> std::string::String {
if event_kind.contains(".open_limit_order") {
return "open_limit_order".to_string();
}
if event_kind.contains(".increase_limit_order") {
return "increase_limit_order".to_string();
}
if event_kind.contains(".decrease_limit_order") {
return "decrease_limit_order".to_string();
}
if event_kind.contains(".close_limit_order") {
return "close_limit_order".to_string();
}
if event_kind.contains(".settle_limit_order") {
return "settle_limit_order".to_string();
}
if event_kind.contains("order_place") {
return "order_place".to_string();
}
if event_kind.contains("order_cancel") {
return "order_cancel".to_string();
}
if event_kind.contains("settle_funds") {
return "settle_funds".to_string();
}
return event_kind.to_string();
}
fn fallback_order_account(
event_kind: &str,
payload: &serde_json::Value,
) -> std::option::Option<std::string::String> {
if event_kind.contains(".close_limit_order") {
return extract_account_at(payload, 2);
}
if event_kind.contains(".open_limit_order")
|| event_kind.contains(".increase_limit_order")
|| event_kind.contains(".decrease_limit_order")
{
return extract_account_at(payload, 3);
}
return None;
}
fn extract_account_at(
value: &serde_json::Value,
index: usize,
) -> std::option::Option<std::string::String> {
if let Some(object) = value.as_object() {
let accounts = object.get("accounts");
if let Some(accounts) = accounts {
if let Some(array) = accounts.as_array() {
let candidate = array.get(index);
if let Some(candidate) = candidate {
if let Some(text) = candidate.as_str() {
let trimmed = text.trim();
if !trimmed.is_empty() {
return Some(trimmed.to_string());
}
}
}
}
}
for nested_value in object.values() {
let nested = extract_account_at(nested_value, index);
if nested.is_some() {
return nested;
}
}
}
return None;
}
fn extract_first_i64(
value: &serde_json::Value,
candidate_keys: &[&str],
) -> std::option::Option<i64> {
if let Some(object) = value.as_object() {
for candidate_key in candidate_keys {
let candidate_value = object.get(*candidate_key);
if let Some(candidate_value) = candidate_value {
if let Some(number) = candidate_value.as_i64() {
return Some(number);
}
if let Some(number) = candidate_value.as_u64() {
let converted = i64::try_from(number);
if let Ok(converted) = converted {
return Some(converted);
}
}
if let Some(text) = candidate_value.as_str() {
let parsed = text.parse::<i64>();
if let Ok(parsed) = parsed {
return Some(parsed);
}
}
}
}
for nested_value in object.values() {
let nested = extract_first_i64(nested_value, candidate_keys);
if nested.is_some() {
return nested;
}
}
}
return None;
}
fn extract_first_bool(
value: &serde_json::Value,
candidate_keys: &[&str],
) -> std::option::Option<bool> {
if let Some(object) = value.as_object() {
for candidate_key in candidate_keys {
let candidate_value = object.get(*candidate_key);
if let Some(candidate_value) = candidate_value {
if let Some(flag) = candidate_value.as_bool() {
return Some(flag);
}
if let Some(number) = candidate_value.as_i64() {
return Some(number != 0);
}
if let Some(text) = candidate_value.as_str() {
if text == "true" || text == "1" {
return Some(true);
}
if text == "false" || text == "0" {
return Some(false);
}
}
}
}
for nested_value in object.values() {
let nested = extract_first_bool(nested_value, candidate_keys);
if nested.is_some() {
return nested;
}
}
}
return None;
}
fn transaction_has_effective_error(transaction: &crate::ChainTransactionDto) -> bool {
let err_json = match transaction.err_json.as_ref() {
Some(err_json) => err_json.trim(),
None => return false,
};
if err_json.is_empty() {
return false;
}
if err_json == "null" {
return false;
}
return true;
}
fn extract_first_u64(
value: &serde_json::Value,
candidate_keys: &[&str],
@@ -902,6 +1158,28 @@ fn extract_first_number_as_string(
#[cfg(test)]
mod tests {
#[test]
fn blank_or_null_err_json_is_not_effective_failure() {
let mut transaction = crate::ChainTransactionDto::new(
"sig-non-trade-effective-error".to_string(),
Some(1),
None,
None,
None,
None,
None,
"{}".to_string(),
);
assert!(!super::transaction_has_effective_error(&transaction));
transaction.err_json = Some("".to_string());
assert!(!super::transaction_has_effective_error(&transaction));
transaction.err_json = Some("null".to_string());
assert!(!super::transaction_has_effective_error(&transaction));
transaction.err_json = Some("{\"InstructionError\":[0,\"Custom\"]}".to_string());
assert!(super::transaction_has_effective_error(&transaction));
}
#[test]
fn extracts_nested_liquidity_amounts() {
let payload = serde_json::json!({

47
kb_lib/src/onchain_dex_pair_discovery.rs
View File

@@ -1279,6 +1279,53 @@ fn decode_raydium_clmm_candidate(
),
});
},
crate::RaydiumClmmDecodedEvent::CreatePool(event) => {
return Some(crate::OnchainDexPairCandidateDto {
signature: signature.to_string(),
slot,
block_time,
failed,
program_id: program_id.to_string(),
dex_code,
candidate_kind: "create_pool".to_string(),
confidence: "high".to_string(),
instruction_index: instruction.instruction_index,
inner_instruction_index: instruction.inner_instruction_index,
instruction_name: Some("raydium_clmm.create_pool".to_string()),
instruction_data_prefix: instruction_data_prefix(instruction.data.as_deref()),
instruction_discriminator_hex: instruction_discriminator_hex(
instruction.data.as_deref(),
),
pool_address: Some(event.pool_state.clone()),
token_a_mint: Some(event.token_mint_0),
token_b_mint: Some(event.token_mint_1),
verified_pool_address: Some(event.pool_state.clone()),
observed_token_mints: std::vec::Vec::new(),
token_balance_deltas: std::vec::Vec::new(),
candidate_pool_accounts: std::vec::Vec::new(),
candidate_token_vault_accounts: std::vec::Vec::new(),
candidate_program_accounts: std::vec::Vec::new(),
account_samples: sample_strings(instruction.accounts.as_slice(), 12),
log_samples: sample_logs(logs, 8),
backfill_hint: build_backfill_hint(
"pool",
Some(event.pool_state.as_str()),
signature,
),
});
},
crate::RaydiumClmmDecodedEvent::CollectProtocolFee(_)
| crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(_)
| crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(_)
| crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(_)
| crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(_)
| crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(_)
| crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(_)
| crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(_)
| crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(_)
| crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(_)
| crate::RaydiumClmmDecodedEvent::SwapEvent(_)
| crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(_) => return None,
}
}
return None;

319
kb_lib/src/token_backfill.rs
View File

@@ -131,6 +131,68 @@ pub struct SignatureBackfillResult {
pub pair_candle_count: usize,
}
/// One item produced by a batch signature backfill.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct SignatureBatchBackfillItemResult {
/// Input transaction signature.
pub signature: std::string::String,
/// Whether the signature was replayed successfully.
pub success: bool,
/// Error text when this signature failed before a replay result could be produced.
pub error: std::option::Option<std::string::String>,
/// Per-signature replay result when available.
pub result: std::option::Option<crate::SignatureBackfillResult>,
}
/// Batch signature-backfill result summary.
#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
pub struct SignatureBatchBackfillResult {
/// Number of raw signatures submitted by the UI.
pub input_signature_count: usize,
/// Number of unique non-empty signatures processed.
pub unique_signature_count: usize,
/// Number of successfully replayed signatures.
pub success_count: usize,
/// Number of signatures that failed before a replay result could be produced.
pub failure_count: usize,
/// Whether processing stopped after the first hard failure.
pub aborted: bool,
/// Number of transactions resolved through HTTP during this run.
pub resolved_transaction_count: usize,
/// Number of signatures whose `getTransaction` lookup returned `null`.
pub missing_transaction_count: usize,
/// Number of signatures whose `getTransaction` lookup failed after retries.
pub transaction_fetch_error_count: usize,
/// Last transaction fetch error observed during this run, if any.
pub last_transaction_fetch_error: std::option::Option<std::string::String>,
/// Total number of decoded DEX events replayed during this run.
pub decoded_event_count: usize,
/// Total number of DEX detection results produced during this run.
pub detection_count: usize,
/// Total number of launch-attribution results produced during this run.
pub launch_attribution_count: usize,
/// Total number of pool-origin results produced during this run.
pub pool_origin_count: usize,
/// Total number of wallet-participation observations produced during this run.
pub wallet_participation_count: usize,
/// Total number of trade-aggregation results produced during this run.
pub trade_event_count: usize,
/// Total number of liquidity event materialization results produced during this run.
pub liquidity_event_count: usize,
/// Total number of pool lifecycle event materialization results produced during this run.
pub pool_lifecycle_event_count: usize,
/// Total number of fee event materialization results produced during this run.
pub fee_event_count: usize,
/// Total number of reward event materialization results produced during this run.
pub reward_event_count: usize,
/// Total number of pool administration event materialization results produced during this run.
pub pool_admin_event_count: usize,
/// Total number of pair-candle aggregation results produced during this run.
pub pair_candle_count: usize,
/// Detailed per-signature results.
pub items: std::vec::Vec<crate::SignatureBatchBackfillItemResult>,
}
/// Historical token backfill service.
///
/// This service reuses the existing transaction projection and downstream
@@ -878,6 +940,178 @@ impl TokenBackfillService {
return Ok(result);
}
/// Replays a batch of known transaction signatures through the existing pipeline.
///
/// Unlike [`Self::backfill_signature`], this method refreshes token metadata and
/// event coverage only once after the whole batch has been processed. This keeps
/// manual discovery backfills responsive when many signatures were collected from
/// an external explorer.
pub async fn backfill_signatures(
&self,
signatures: &[std::string::String],
continue_on_error: bool,
) -> Result<crate::SignatureBatchBackfillResult, crate::Error> {
let mut result = crate::SignatureBatchBackfillResult {
input_signature_count: signatures.len(),
unique_signature_count: 0,
success_count: 0,
failure_count: 0,
aborted: false,
resolved_transaction_count: 0,
missing_transaction_count: 0,
transaction_fetch_error_count: 0,
last_transaction_fetch_error: None,
decoded_event_count: 0,
detection_count: 0,
launch_attribution_count: 0,
pool_origin_count: 0,
wallet_participation_count: 0,
trade_event_count: 0,
liquidity_event_count: 0,
pool_lifecycle_event_count: 0,
fee_event_count: 0,
reward_event_count: 0,
pool_admin_event_count: 0,
pair_candle_count: 0,
items: std::vec::Vec::new(),
};
let mut seen = std::collections::BTreeSet::<std::string::String>::new();
for signature in signatures {
let trimmed_signature = signature.trim().to_string();
if trimmed_signature.is_empty() {
continue;
}
if seen.contains(trimmed_signature.as_str()) {
continue;
}
seen.insert(trimmed_signature.clone());
result.unique_signature_count += 1;
let replay_result = self.replay_signature(trimmed_signature.clone()).await;
let replay = match replay_result {
Ok(replay) => replay,
Err(error) => {
result.failure_count += 1;
result.items.push(crate::SignatureBatchBackfillItemResult {
signature: trimmed_signature.clone(),
success: false,
error: Some(error.to_string()),
result: None,
});
if !continue_on_error {
result.aborted = true;
break;
}
continue;
},
};
let signature_result = crate::SignatureBackfillResult {
signature: trimmed_signature.clone(),
resolved_transaction_count: replay.resolved_transaction_count,
missing_transaction_count: replay.missing_transaction_count,
transaction_fetch_error_count: replay.transaction_fetch_error_count,
last_transaction_fetch_error: replay.last_transaction_fetch_error.clone(),
decoded_event_count: replay.decoded_event_count,
detection_count: replay.detection_count,
launch_attribution_count: replay.launch_attribution_count,
pool_origin_count: replay.pool_origin_count,
wallet_participation_count: replay.wallet_participation_count,
trade_event_count: replay.trade_event_count,
liquidity_event_count: replay.liquidity_event_count,
pool_lifecycle_event_count: replay.pool_lifecycle_event_count,
fee_event_count: replay.fee_event_count,
reward_event_count: replay.reward_event_count,
pool_admin_event_count: replay.pool_admin_event_count,
pair_candle_count: replay.pair_candle_count,
};
result.success_count += 1;
result.resolved_transaction_count += signature_result.resolved_transaction_count;
result.missing_transaction_count += signature_result.missing_transaction_count;
result.transaction_fetch_error_count += signature_result.transaction_fetch_error_count;
if signature_result.last_transaction_fetch_error.is_some() {
result.last_transaction_fetch_error =
signature_result.last_transaction_fetch_error.clone();
}
result.decoded_event_count += signature_result.decoded_event_count;
result.detection_count += signature_result.detection_count;
result.launch_attribution_count += signature_result.launch_attribution_count;
result.pool_origin_count += signature_result.pool_origin_count;
result.wallet_participation_count += signature_result.wallet_participation_count;
result.trade_event_count += signature_result.trade_event_count;
result.liquidity_event_count += signature_result.liquidity_event_count;
result.pool_lifecycle_event_count += signature_result.pool_lifecycle_event_count;
result.fee_event_count += signature_result.fee_event_count;
result.reward_event_count += signature_result.reward_event_count;
result.pool_admin_event_count += signature_result.pool_admin_event_count;
result.pair_candle_count += signature_result.pair_candle_count;
result.items.push(crate::SignatureBatchBackfillItemResult {
signature: trimmed_signature,
success: true,
error: None,
result: Some(signature_result),
});
}
if result.unique_signature_count == 0 {
return Err(crate::Error::Config(
"signature batch must contain at least one non-empty signature".to_string(),
));
}
self.backfill_missing_token_metadata_best_effort(100).await;
self.refresh_event_coverage_best_effort().await;
let summary_payload = serde_json::json!({
"inputSignatureCount": result.input_signature_count,
"uniqueSignatureCount": result.unique_signature_count,
"successCount": result.success_count,
"failureCount": result.failure_count,
"aborted": result.aborted,
"resolvedTransactionCount": result.resolved_transaction_count,
"missingTransactionCount": result.missing_transaction_count,
"transactionFetchErrorCount": result.transaction_fetch_error_count,
"lastTransactionFetchError": result.last_transaction_fetch_error,
"decodedEventCount": result.decoded_event_count,
"detectionCount": result.detection_count,
"launchAttributionCount": result.launch_attribution_count,
"poolOriginCount": result.pool_origin_count,
"walletParticipationCount": result.wallet_participation_count,
"tradeEventCount": result.trade_event_count,
"liquidityEventCount": result.liquidity_event_count,
"poolLifecycleEventCount": result.pool_lifecycle_event_count,
"feeEventCount": result.fee_event_count,
"rewardEventCount": result.reward_event_count,
"poolAdminEventCount": result.pool_admin_event_count,
"pairCandleCount": result.pair_candle_count
});
let observation_result = self
.persistence
.record_observation(&crate::DetectionObservationInput::new(
"signature_batch.backfill.completed".to_string(),
crate::ObservationSourceKind::HttpRpc,
Some(format!("backfill:{}", self.http_role)),
format!("{} signatures", result.unique_signature_count),
None,
summary_payload.clone(),
))
.await;
let observation_id = match observation_result {
Ok(observation_id) => observation_id,
Err(error) => return Err(error),
};
let signal_result = self
.persistence
.record_signal(&crate::DetectionSignalInput::new(
"signal.signature_batch.backfill.completed".to_string(),
crate::AnalysisSignalSeverity::Low,
format!("{} signatures", result.unique_signature_count),
Some(observation_id),
None,
summary_payload,
))
.await;
if let Err(error) = signal_result {
return Err(error);
}
return Ok(result);
}
async fn fetch_transaction_value_with_retry(
&self,
signature: &str,
@@ -943,6 +1177,52 @@ impl TokenBackfillService {
}
async fn refresh_event_coverage_best_effort(&self) {
let cleanup_result =
crate::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits(
self.database.as_ref(),
None,
)
.await;
match cleanup_result {
Ok(deleted_count) => {
if deleted_count > 0 {
tracing::info!(
deleted_count = deleted_count,
"replaced Raydium CLMM instruction audits cleaned before dex event coverage refresh"
);
}
},
Err(error) => {
tracing::warn!(
error = %error,
"Raydium CLMM replaced instruction-audit cleanup failed before dex event coverage refresh"
);
},
}
let upstream_cleanup_result =
crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(
self.database.as_ref(),
None,
)
.await;
match upstream_cleanup_result {
Ok(deleted_count) => {
if deleted_count > 0 {
tracing::info!(
deleted_count = deleted_count,
"locally covered upstream instruction matches cleaned before dex event coverage refresh"
);
}
},
Err(error) => {
tracing::warn!(
error = %error,
"locally covered upstream instruction-match cleanup failed before dex event coverage refresh"
);
},
}
let coverage_service = crate::DexEventCoverageService::new(self.database.clone());
let refresh_result = coverage_service.refresh_local_counts(None).await;
match refresh_result {
@@ -960,6 +1240,45 @@ impl TokenBackfillService {
);
},
}
let post_refresh_upstream_cleanup_result =
crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(
self.database.as_ref(),
None,
)
.await;
match post_refresh_upstream_cleanup_result {
Ok(deleted_count) => {
if deleted_count > 0 {
tracing::info!(
deleted_count = deleted_count,
"locally covered upstream instruction matches cleaned after dex event coverage refresh"
);
let second_refresh_result = coverage_service.refresh_local_counts(None).await;
match second_refresh_result {
Ok(second_refresh_result) => {
tracing::debug!(
upserted_entry_count = second_refresh_result.upserted_entry_count,
summary_count = second_refresh_result.summaries.len(),
"dex event coverage refreshed after upstream instruction-match cleanup"
);
},
Err(error) => {
tracing::warn!(
error = %error,
"dex event coverage refresh failed after upstream instruction-match cleanup"
);
},
}
}
},
Err(error) => {
tracing::warn!(
error = %error,
"locally covered upstream instruction-match cleanup failed after dex event coverage refresh"
);
},
}
}
}

130
kb_lib/src/upstream_registry_generated.rs
View File

@@ -12,6 +12,37 @@ const UPSTREAM_GIT_PROGRAM_NOTES: &str = "program id extracted from upstream Git
const UPSTREAM_GIT_DISCRIMINATOR_NOTES: &str = "entry name and discriminator extracted from upstream Git decoder source or from the discriminator convention used by that upstream decoder; not corpus-verified; no trade/candle/materialization proof";
const UPSTREAM_GIT_ALIAS_PROGRAM_NOTES: &str = "upstream Git decoder name kept as a discovery alias; program id and discriminator rows are represented by the canonical decoder entry to avoid duplicate registry keys";
const RAYDIUM_IDL_SOURCE_REPO: &str = "raydium-io/raydium-idl";
const RAYDIUM_IDL_DISCRIMINATOR_NOTES: &str = "entry name and discriminator extracted from Raydium official IDL snapshot; not corpus-verified; no trade/candle/materialization proof";
const fn raydium_idl_discriminator_entry(
decoder_code: &'static str,
program_id: std::option::Option<&'static str>,
program_family: &'static str,
surface_kind: &'static str,
entry_kind: &'static str,
entry_name: &'static str,
discriminator_hex: &'static str,
discriminator_len: u16,
source_path: &'static str,
) -> crate::UpstreamRegistryEntry {
return crate::UpstreamRegistryEntry {
source_repo: Some(RAYDIUM_IDL_SOURCE_REPO),
source_path: Some(source_path),
decoder_code,
program_id,
program_family,
surface_kind,
entry_kind,
entry_name,
discriminator_hex: Some(discriminator_hex),
discriminator_len: Some(discriminator_len),
proof_status: crate::PROOF_STATUS_UPSTREAM_GIT_UNVERIFIED,
notes: RAYDIUM_IDL_DISCRIMINATOR_NOTES,
};
}
const fn upstream_git_program_entry(
decoder_code: &'static str,
program_id: std::option::Option<&'static str>,
@@ -11775,6 +11806,61 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[
8,
"decoders/raydium-clmm-decoder/src/instructions/close_position.rs",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"close_limit_order",
"4c7c800fd55725fa",
8,
"raydium_clmm/raydium_clmm.json",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"open_limit_order",
"9d20dab7471d1293",
8,
"raydium_clmm/raydium_clmm.json",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"increase_limit_order",
"b19059ecfaba7d63",
8,
"raydium_clmm/raydium_clmm.json",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"decrease_limit_order",
"759d3c674231a300",
8,
"raydium_clmm/raydium_clmm.json",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"close_protocol_position",
"c975989055556cb2",
8,
"raydium_clmm/raydium_clmm.json",
),
upstream_git_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
@@ -11852,6 +11938,28 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[
8,
"decoders/raydium-clmm-decoder/src/instructions/create_amm_config.rs",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"create_customizable_pool",
"2b44d4a7592fa401",
8,
"raydium_clmm/raydium_clmm.json",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"create_dynamic_fee_config",
"bd0eb5785576e33e",
8,
"raydium_clmm/raydium_clmm.json",
),
upstream_git_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
@@ -11885,6 +11993,17 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[
8,
"decoders/raydium-clmm-decoder/src/instructions/create_pool.rs",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"create_support_mint_associated",
"11fb415c88f20ea9",
8,
"raydium_clmm/raydium_clmm.json",
),
upstream_git_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
@@ -12039,6 +12158,17 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[
8,
"decoders/raydium-clmm-decoder/src/instructions/set_reward_params.rs",
),
raydium_idl_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),
"raydium",
"clmm",
crate::ENTRY_KIND_INSTRUCTION,
"settle_limit_order",
"cd4e74215c691a60",
8,
"raydium_clmm/raydium_clmm.json",
),
upstream_git_discriminator_entry(
"raydium_clmm",
Some(crate::RAYDIUM_CLMM_PROGRAM_ID),

132
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql Normal file
View File

@@ -0,0 +1,132 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql
-- Raydium CLMM final validation SQL for 0.7.49.
-- 1. CLMM coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Residual CLMM instruction audits. Expected: zero rows.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 3. Redundant upstream fallback matches for locally covered CLMM entries. Expected: zero rows.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT ug.transaction_id) AS tx_count
FROM k_sol_dex_decoded_events ug
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 4. Instruction-observation links still pointing to redundant upstream fallback rows. Expected: zero rows.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS upstream_discriminator_hex,
COUNT(*) AS linked_observation_count
FROM k_sol_instruction_observations io
JOIN k_sol_dex_decoded_events ug
ON ug.id = io.decoded_event_id
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY
json_extract(ug.payload_json, '$.upstreamDecoderCode'),
json_extract(ug.payload_json, '$.upstreamEntryName'),
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
ORDER BY linked_observation_count DESC, entry_name;
-- 5. Any non-swap CLMM trade. Expected: zero rows.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind NOT IN (
'raydium_clmm.swap',
'raydium_clmm.swap_v2'
)
GROUP BY de.event_kind
HAVING COUNT(te.id) > 0
ORDER BY trade_count DESC, de.event_kind;
-- 6. Failed transaction materialization guard. Expected: zero rows.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
HAVING
COUNT(le.id) > 0
OR COUNT(fe.id) > 0
OR COUNT(re.id) > 0
OR COUNT(pa.id) > 0
OR COUNT(ple.id) > 0
OR COUNT(oe.id) > 0
OR COUNT(te.id) > 0
ORDER BY de.event_kind;

111
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE10.sql Normal file
View File

@@ -0,0 +1,111 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE10.sql
-- Raydium CLMM validation after pre.10 mapped instruction expansion.
-- 1. Coverage rows.
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
ORDER BY entry_kind, entry_name, discriminator_hex;
-- 2. Coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 3. Instruction observations, including unknown discriminants.
SELECT
instruction_name,
discriminator_hex,
COUNT(*) AS observed_count,
COUNT(DISTINCT signature) AS tx_count
FROM k_sol_instruction_observations
WHERE decoder_code = 'raydium_clmm'
GROUP BY instruction_name, discriminator_hex
ORDER BY observed_count DESC, instruction_name;
-- 4. Decoded CLMM distribution. After pre.10, known listed instructions should move
-- from raydium_clmm.instruction_audit to named raydium_clmm.<instruction> events.
SELECT
de.protocol_name,
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(DISTINCT de.transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events de
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.protocol_name, de.event_kind
ORDER BY decoded_count DESC, de.event_kind;
-- 5. Non-trade materialization distribution.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 6. Residual known CLMM instruction audits: should be zero for listed instruction discriminants.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS decoded_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
AND json_extract(payload_json, '$.discriminatorHex') IN (
'4c7c800fd55725fa', '7b86510031446262', 'c975989055556cb2',
'a78a4e95dfc2067e', '8888fcddc2427e59', '12eda6c52210d590',
'8934edd4d7756c68', '2b44d4a7592fa401', 'bd0eb5785576e33e',
'3f5794216d230868', 'e992d18ecf6840bc', '11fb415c88f20ea9',
'a026d06f685b2c01', '3a7fbc3e4f52c460', '2e9cf3760dcdfbb2',
'851d59df45eeb00a', '5f87c0c4f281e644', '87802f4d0f98f031',
'4db84ad67056f1c7', '4dffae527d1dc92e', '7034a74b20c9d389',
'cd4e74215c691a60', 'f8c69e91e17587c8', '457d73daf5baf2c4',
'2b04ed0b1ac91e62', '07160c53f22b3079', '313cae889a1c74c8',
'7f467728bce33d07', '82576c062ee0757b', 'a3ace0340b9a6adf'
)
GROUP BY discriminator_hex
ORDER BY decoded_count DESC;
-- 7. Unknown observed discriminants to keep audit-only until identified.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
json_extract(payload_json, '$.accountCount') AS account_count,
COUNT(*) AS decoded_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, account_count
ORDER BY decoded_count DESC;

100
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE11.sql Normal file
View File

@@ -0,0 +1,100 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE11.sql
-- Raydium CLMM validation after 0.7.49-pre.11.
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
ORDER BY entry_kind, entry_name, discriminator_hex;
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
SELECT
de.protocol_name,
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(DISTINCT de.transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events de
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.protocol_name, de.event_kind
ORDER BY decoded_count DESC, de.event_kind;
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- Residual duplicated audits: should only return unknown/unmapped discriminators
-- or known discriminators for which no same-transaction named event exists.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT audit.transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events audit
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- Duplicated known audits that still have a named CLMM event in the same transaction.
-- Expected: zero rows.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;

71
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE12.sql Normal file
View File

@@ -0,0 +1,71 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE12.sql
-- Raydium CLMM pre.12 validation: residual audit cleanup and materialization counters.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.event_kind
ORDER BY de.event_kind;

77
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE13.sql Normal file
View File

@@ -0,0 +1,77 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE13.sql
-- Raydium CLMM validation after pre.13 audit cleanup.
-- 1. CLMM coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Duplicate audit rows that should have been deleted.
-- Expected result after pre.13: zero rows.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;
-- 3. Residual instruction audits by discriminator.
-- Expected after pre.13: mostly the true unmapped values, currently 759d..., 9d20..., b190...
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 4. CLMM materialization distribution.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.event_kind
ORDER BY de.event_kind;

78
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE14.sql Normal file
View File

@@ -0,0 +1,78 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE14.sql
-- Raydium CLMM validation after pre.14 EXISTS-based audit cleanup.
-- 1. Coverage summary should not be reset to zero.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Duplicate audit rows. Expected: zero rows after pre.14.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;
-- 3. Residual audits should mostly be unknown discriminants.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 4. Manual cleanup equivalent for emergency diagnosis only.
-- Do not run unless the application cleanup still fails; keep it here to compare SQL behavior.
-- DELETE FROM k_sol_dex_decoded_events
-- WHERE protocol_name = 'raydium_clmm'
-- AND event_kind = 'raydium_clmm.instruction_audit'
-- AND EXISTS (
-- SELECT 1
-- FROM k_sol_dex_decoded_events named
-- WHERE named.transaction_id = k_sol_dex_decoded_events.transaction_id
-- AND named.protocol_name = 'raydium_clmm'
-- AND named.event_kind <> 'raydium_clmm.instruction_audit'
-- AND COALESCE(
-- json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
-- json_extract(named.payload_json, '$.instruction_discriminator_hex'),
-- json_extract(named.payload_json, '$.discriminatorHex'),
-- json_extract(named.payload_json, '$.discriminator_hex')
-- ) = COALESCE(
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminatorHex'),
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminator_hex'),
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instructionDiscriminatorHex'),
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instruction_discriminator_hex')
-- )
-- );

77
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE15.sql Normal file
View File

@@ -0,0 +1,77 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE15.sql
-- Raydium CLMM validation after pre.15 final-refresh audit cleanup.
-- 1. Coverage summary should stay populated after cleanup + refresh.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Duplicate audit rows should be zero after pre.15.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;
-- 3. Residual audits should mostly be unknown discriminants.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 4. Manual cleanup equivalent: run only if query 2 still returns rows.
-- DELETE FROM k_sol_dex_decoded_events
-- WHERE protocol_name = 'raydium_clmm'
-- AND event_kind = 'raydium_clmm.instruction_audit'
-- AND EXISTS (
-- SELECT 1
-- FROM k_sol_dex_decoded_events named
-- WHERE named.transaction_id = k_sol_dex_decoded_events.transaction_id
-- AND named.protocol_name = 'raydium_clmm'
-- AND named.event_kind <> 'raydium_clmm.instruction_audit'
-- AND COALESCE(
-- json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
-- json_extract(named.payload_json, '$.instruction_discriminator_hex'),
-- json_extract(named.payload_json, '$.discriminatorHex'),
-- json_extract(named.payload_json, '$.discriminator_hex')
-- ) = COALESCE(
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminatorHex'),
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminator_hex'),
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instructionDiscriminatorHex'),
-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instruction_discriminator_hex')
-- )
-- );

76
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE16.sql Normal file
View File

@@ -0,0 +1,76 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE16.sql
-- Raydium CLMM validation after pre.16 mapped-audit allow-list cleanup.
-- 1. Coverage summary must stay populated.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Duplicate audit rows should be zero.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;
-- 3. Residual audits should be dominated by unknown discriminants.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 4. Manual cleanup equivalent for immediate diagnosis only.
-- DELETE FROM k_sol_dex_decoded_events
-- WHERE protocol_name = 'raydium_clmm'
-- AND event_kind = 'raydium_clmm.instruction_audit'
-- AND COALESCE(
-- json_extract(payload_json, '$.discriminatorHex'),
-- json_extract(payload_json, '$.discriminator_hex'),
-- json_extract(payload_json, '$.instructionDiscriminatorHex'),
-- json_extract(payload_json, '$.instruction_discriminator_hex')
-- ) IN (
-- '4c7c800fd55725fa','7b86510031446262','c975989055556cb2',
-- 'a78a4e95dfc2067e','8888fcddc2427e59','12eda6c52210d590',
-- '8934edd4d7756c68','2b44d4a7592fa401','bd0eb5785576e33e',
-- '3f5794216d230868','e992d18ecf6840bc','11fb415c88f20ea9',
-- 'a026d06f685b2c01','3a7fbc3e4f52c460','2e9cf3760dcdfbb2',
-- '851d59df45eeb00a','5f87c0c4f281e644','87802f4d0f98f031',
-- '4db84ad67056f1c7','4dffae527d1dc92e','7034a74b20c9d389',
-- 'cd4e74215c691a60','f8c69e91e17587c8','457d73daf5baf2c4',
-- '2b04ed0b1ac91e62','07160c53f22b3079','313cae889a1c74c8',
-- '7f467728bce33d07','82576c062ee0757b','a3ace0340b9a6adf'
-- );

120
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE17.sql Normal file
View File

@@ -0,0 +1,120 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE17.sql
-- Raydium CLMM validation after limit-order instruction mapping.
-- Expected coverage listed_entry_count becomes 45 because open/increase/decrease_limit_order are now listed.
-- 1. CLMM coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. New CLMM limit-order coverage rows.
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
AND entry_name IN (
'open_limit_order',
'increase_limit_order',
'decrease_limit_order',
'close_limit_order',
'settle_limit_order'
)
ORDER BY entry_name;
-- 3. The formerly residual discriminants must now decode as named events.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(DISTINCT de.transaction_id) AS transaction_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind IN (
'raydium_clmm.open_limit_order',
'raydium_clmm.increase_limit_order',
'raydium_clmm.decrease_limit_order'
)
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 4. No residual audit rows for the newly mapped discriminants.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
AND json_extract(payload_json, '$.discriminatorHex') IN (
'9d20dab7471d1293',
'b19059ecfaba7d63',
'759d3c674231a300'
)
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC;
-- 5. No trade/candle safety: limit-order maintenance must never produce trade rows.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind IN (
'raydium_clmm.open_limit_order',
'raydium_clmm.increase_limit_order',
'raydium_clmm.decrease_limit_order',
'raydium_clmm.close_limit_order',
'raydium_clmm.settle_limit_order'
)
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 6. Duplicate audit rows should stay empty after mapped-audit cleanup.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;

117
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE18.sql Normal file
View File

@@ -0,0 +1,117 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE18.sql
-- Raydium CLMM validation after pre.18 orderbook-event schema and materialization.
-- 1. Schema must contain the new orderbook materialization table.
SELECT
name
FROM sqlite_master
WHERE type = 'table'
AND name = 'k_sol_orderbook_events';
-- 2. Cleanup any stale mapped CLMM instruction audits left from earlier runs.
-- This is safe: the discriminants below are now decoded as named raydium_clmm.* events.
DELETE FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(payload_json, '$.discriminatorHex'),
json_extract(payload_json, '$.discriminator_hex'),
json_extract(payload_json, '$.instructionDiscriminatorHex'),
json_extract(payload_json, '$.instruction_discriminator_hex')
) IN (
'4c7c800fd55725fa','7b86510031446262','c975989055556cb2',
'a78a4e95dfc2067e','8888fcddc2427e59','12eda6c52210d590',
'8934edd4d7756c68','2b44d4a7592fa401','bd0eb5785576e33e',
'3f5794216d230868','e992d18ecf6840bc','11fb415c88f20ea9',
'a026d06f685b2c01','3a7fbc3e4f52c460','2e9cf3760dcdfbb2',
'851d59df45eeb00a','5f87c0c4f281e644','87802f4d0f98f031',
'4db84ad67056f1c7','4dffae527d1dc92e','7034a74b20c9d389',
'cd4e74215c691a60','f8c69e91e17587c8','457d73daf5baf2c4',
'2b04ed0b1ac91e62','07160c53f22b3079','313cae889a1c74c8',
'7f467728bce33d07','82576c062ee0757b','a3ace0340b9a6adf',
'759d3c674231a300','9d20dab7471d1293','b19059ecfaba7d63'
);
-- 3. Duplicate audits should be zero.
SELECT
json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex,
audit.event_kind AS audit_event_kind,
named.event_kind AS named_event_kind,
COUNT(*) AS duplicate_count
FROM k_sol_dex_decoded_events audit
JOIN k_sol_dex_decoded_events named
ON named.transaction_id = audit.transaction_id
AND named.protocol_name = 'raydium_clmm'
AND named.event_kind <> 'raydium_clmm.instruction_audit'
AND COALESCE(
json_extract(named.payload_json, '$.instructionDiscriminatorHex'),
json_extract(named.payload_json, '$.instruction_discriminator_hex'),
json_extract(named.payload_json, '$.discriminatorHex'),
json_extract(named.payload_json, '$.discriminator_hex')
) = COALESCE(
json_extract(audit.payload_json, '$.discriminatorHex'),
json_extract(audit.payload_json, '$.discriminator_hex'),
json_extract(audit.payload_json, '$.instructionDiscriminatorHex'),
json_extract(audit.payload_json, '$.instruction_discriminator_hex')
)
WHERE audit.protocol_name = 'raydium_clmm'
AND audit.event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex, audit_event_kind, named_event_kind
ORDER BY duplicate_count DESC;
-- 4. Limit-order decoded events must materialize into k_sol_orderbook_events and never trades.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(DISTINCT de.transaction_id) AS tx_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind IN (
'raydium_clmm.open_limit_order',
'raydium_clmm.increase_limit_order',
'raydium_clmm.decrease_limit_order',
'raydium_clmm.close_limit_order',
'raydium_clmm.settle_limit_order'
)
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 5. No failed transaction should be orderbook-materialized.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
ORDER BY decoded_count DESC, de.event_kind;
-- 6. CLMM coverage summary after refresh.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;

144
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE19.sql Normal file
View File

@@ -0,0 +1,144 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE19.sql
-- Raydium CLMM validation after pre.19 upstream fallback cleanup.
-- 1. CLMM coverage summary must remain populated.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. No local CLMM instruction audit should remain after specialized decoding.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 3. No upstream_git fallback should remain for CLMM entries already covered locally.
SELECT
json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT transaction_id) AS tx_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'upstream_git'
AND event_kind = 'upstream_git.instruction_match'
AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 4. Generic diagnostic for any locally covered upstream fallback, all protocols.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
COUNT(*) AS fallback_count,
COUNT(DISTINCT ug.transaction_id) AS tx_count
FROM k_sol_dex_decoded_events ug
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex
ORDER BY fallback_count DESC, upstream_decoder_code, entry_name;
-- 5. Non-swap CLMM events must never create trades/candles.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind NOT IN (
'raydium_clmm.swap',
'raydium_clmm.swap_v2'
)
GROUP BY de.event_kind
HAVING COUNT(te.id) > 0
ORDER BY trade_count DESC, de.event_kind;
-- 6. Failed CLMM transactions must not materialize into business tables.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
HAVING
COUNT(le.id) > 0
OR COUNT(fe.id) > 0
OR COUNT(re.id) > 0
OR COUNT(pa.id) > 0
OR COUNT(ple.id) > 0
OR COUNT(oe.id) > 0
OR COUNT(te.id) > 0
ORDER BY de.event_kind;
-- 7. Orderbook limit-order materialization safety.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(DISTINCT de.transaction_id) AS tx_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind IN (
'raydium_clmm.open_limit_order',
'raydium_clmm.increase_limit_order',
'raydium_clmm.decrease_limit_order',
'raydium_clmm.close_limit_order',
'raydium_clmm.settle_limit_order'
)
GROUP BY de.event_kind
ORDER BY de.event_kind;

123
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE20.sql Normal file
View File

@@ -0,0 +1,123 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE20.sql
-- Raydium CLMM validation after pre.20 fallback cleanup + Program data event readiness.
-- 1. CLMM coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Residual CLMM instruction audits. Expected: zero rows.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 3. Redundant upstream fallback matches for locally covered CLMM entries. Expected: zero rows.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT ug.transaction_id) AS tx_count
FROM k_sol_dex_decoded_events ug
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 4. Any remaining upstream CLMM fallback matches. Expected: zero rows unless a genuinely unmapped future entry appears.
SELECT
json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT transaction_id) AS tx_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'upstream_git'
AND event_kind = 'upstream_git.instruction_match'
AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 5. Anchor Program data events. These may remain zero until matching Program data logs exist in corpus.
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
AND entry_kind = 'event'
ORDER BY entry_name;
-- 6. Failed transaction materialization guard. Expected: zero rows.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
HAVING
COUNT(le.id) > 0
OR COUNT(fe.id) > 0
OR COUNT(re.id) > 0
OR COUNT(pa.id) > 0
OR COUNT(ple.id) > 0
OR COUNT(oe.id) > 0
OR COUNT(te.id) > 0
ORDER BY de.event_kind;

139
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE22.sql Normal file
View File

@@ -0,0 +1,139 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE22.sql
-- Raydium CLMM validation after pre.22 upstream fallback reset cleanup hardening.
-- 1. CLMM coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Residual CLMM instruction audits. Expected: zero rows.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 3. Redundant upstream fallback matches for locally covered entries. Expected: zero rows.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT ug.transaction_id) AS tx_count
FROM k_sol_dex_decoded_events ug
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 4. Any remaining upstream CLMM fallback matches. Expected: zero rows unless a genuinely unmapped future entry appears.
SELECT
json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT transaction_id) AS tx_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'upstream_git'
AND event_kind = 'upstream_git.instruction_match'
AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 5. Safety: non-swap CLMM rows must not materialize as trades.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind NOT IN (
'raydium_clmm.swap',
'raydium_clmm.swap_v2'
)
GROUP BY de.event_kind
HAVING COUNT(te.id) > 0
ORDER BY trade_count DESC, de.event_kind;
-- 6. Safety: failed CLMM transactions must not materialize into business tables.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
HAVING
COUNT(le.id) > 0
OR COUNT(fe.id) > 0
OR COUNT(re.id) > 0
OR COUNT(pa.id) > 0
OR COUNT(ple.id) > 0
OR COUNT(oe.id) > 0
OR COUNT(te.id) > 0
ORDER BY de.event_kind;
-- 7. Raydium CPMM listed coverage entries.
SELECT
entry_name,
entry_kind,
event_family,
expected_db_target,
proof_status,
local_event_kind,
discriminator_hex,
observed_count,
materialized_count,
trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_cpmm'
ORDER BY entry_kind, entry_name, discriminator_hex;

132
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql Normal file
View File

@@ -0,0 +1,132 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql
-- Raydium CLMM validation after pre.23 FK-safe upstream fallback cleanup.
-- 1. CLMM coverage summary.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;
-- 2. Residual CLMM instruction audits. Expected: zero rows.
SELECT
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
COUNT(*) AS residual_audit_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
GROUP BY discriminator_hex
ORDER BY residual_audit_count DESC, discriminator_hex;
-- 3. Redundant upstream fallback matches for locally covered CLMM entries. Expected: zero rows.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex,
json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo,
COUNT(*) AS fallback_count,
COUNT(DISTINCT ug.transaction_id) AS tx_count
FROM k_sol_dex_decoded_events ug
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo
ORDER BY fallback_count DESC, entry_name;
-- 4. Instruction-observation links still pointing to redundant upstream fallback rows. Expected: zero rows.
SELECT
json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code,
json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name,
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS upstream_discriminator_hex,
COUNT(*) AS linked_observation_count
FROM k_sol_instruction_observations io
JOIN k_sol_dex_decoded_events ug
ON ug.id = io.decoded_event_id
JOIN k_sol_dex_event_coverage_entries ce
ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode')
AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName')
AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
AND ce.local_event_kind IS NOT NULL
AND ce.local_event_kind <> ''
WHERE ug.protocol_name = 'upstream_git'
AND ug.event_kind = 'upstream_git.instruction_match'
AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm'
GROUP BY
json_extract(ug.payload_json, '$.upstreamDecoderCode'),
json_extract(ug.payload_json, '$.upstreamEntryName'),
json_extract(ug.payload_json, '$.upstreamDiscriminatorHex')
ORDER BY linked_observation_count DESC, entry_name;
-- 5. Any non-swap CLMM trade. Expected: zero rows.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind NOT IN (
'raydium_clmm.swap',
'raydium_clmm.swap_v2'
)
GROUP BY de.event_kind
HAVING COUNT(te.id) > 0
ORDER BY trade_count DESC, de.event_kind;
-- 6. Failed transaction materialization guard. Expected: zero rows.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(oe.id) AS orderbook_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
JOIN k_sol_chain_transactions tx
ON tx.id = de.transaction_id
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_orderbook_events oe
ON oe.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND tx.err_json IS NOT NULL
AND tx.err_json <> ''
AND tx.err_json <> 'null'
GROUP BY de.event_kind
HAVING
COUNT(le.id) > 0
OR COUNT(fe.id) > 0
OR COUNT(re.id) > 0
OR COUNT(pa.id) > 0
OR COUNT(ple.id) > 0
OR COUNT(oe.id) > 0
OR COUNT(te.id) > 0
ORDER BY de.event_kind;

88
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE6.sql Normal file
View File

@@ -0,0 +1,88 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE6.sql
-- Raydium CLMM pre.6 validation: specialized non-trade reconciliation.
-- 1. Confirm the two corpus-observed CLMM non-trades are materialized.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(fe.id) AS fee_count,
COUNT(ple.id) AS lifecycle_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind IN (
'raydium_clmm.create_pool',
'raydium_clmm.collect_protocol_fee'
)
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 2. Confirm residual audits for the same discriminants are gone.
SELECT
json_extract(payload_json, '$.instructionDiscriminatorHex') AS instruction_discriminator_hex,
json_extract(payload_json, '$.instruction_discriminator_hex') AS instruction_discriminator_hex_snake,
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
json_extract(payload_json, '$.discriminator_hex') AS discriminator_hex_snake,
json_extract(payload_json, '$.accountCount') AS account_count,
COUNT(*) AS decoded_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
AND (
json_extract(payload_json, '$.instructionDiscriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
OR json_extract(payload_json, '$.instruction_discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
OR json_extract(payload_json, '$.discriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
OR json_extract(payload_json, '$.discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
)
GROUP BY
instruction_discriminator_hex,
instruction_discriminator_hex_snake,
discriminator_hex,
discriminator_hex_snake,
account_count
ORDER BY decoded_count DESC;
-- 3. Full CLMM distribution with materialization counters.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 4. Coverage summary after sync/replay.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;

88
validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE7.sql Normal file
View File

@@ -0,0 +1,88 @@
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE7.sql
-- Raydium CLMM pre.7 validation: direct materialization + audit cleanup.
-- 1. Confirm corpus-observed CLMM non-trades are materialized.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(fe.id) AS fee_count,
COUNT(ple.id) AS lifecycle_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
AND de.event_kind IN (
'raydium_clmm.create_pool',
'raydium_clmm.collect_protocol_fee'
)
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 2. Confirm residual audits for the same discriminants are gone.
SELECT
json_extract(payload_json, '$.instructionDiscriminatorHex') AS instruction_discriminator_hex,
json_extract(payload_json, '$.instruction_discriminator_hex') AS instruction_discriminator_hex_snake,
json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex,
json_extract(payload_json, '$.discriminator_hex') AS discriminator_hex_snake,
json_extract(payload_json, '$.accountCount') AS account_count,
COUNT(*) AS decoded_count,
COUNT(DISTINCT transaction_id) AS transaction_count
FROM k_sol_dex_decoded_events
WHERE protocol_name = 'raydium_clmm'
AND event_kind = 'raydium_clmm.instruction_audit'
AND (
json_extract(payload_json, '$.instructionDiscriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
OR json_extract(payload_json, '$.instruction_discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
OR json_extract(payload_json, '$.discriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
OR json_extract(payload_json, '$.discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59')
)
GROUP BY
instruction_discriminator_hex,
instruction_discriminator_hex_snake,
discriminator_hex,
discriminator_hex_snake,
account_count
ORDER BY decoded_count DESC;
-- 3. Full CLMM distribution with materialization counters.
SELECT
de.event_kind,
COUNT(*) AS decoded_count,
COUNT(le.id) AS liquidity_count,
COUNT(fe.id) AS fee_count,
COUNT(re.id) AS reward_count,
COUNT(pa.id) AS admin_count,
COUNT(ple.id) AS lifecycle_count,
COUNT(te.id) AS trade_count
FROM k_sol_dex_decoded_events de
LEFT JOIN k_sol_liquidity_events le
ON le.decoded_event_id = de.id
LEFT JOIN k_sol_fee_events fe
ON fe.decoded_event_id = de.id
LEFT JOIN k_sol_reward_events re
ON re.decoded_event_id = de.id
LEFT JOIN k_sol_pool_admin_events pa
ON pa.decoded_event_id = de.id
LEFT JOIN k_sol_pool_lifecycle_events ple
ON ple.decoded_event_id = de.id
LEFT JOIN k_sol_trade_events te
ON te.decoded_event_id = de.id
WHERE de.protocol_name = 'raydium_clmm'
GROUP BY de.event_kind
ORDER BY de.event_kind;
-- 4. Coverage summary after sync/replay.
SELECT
decoder_code,
COUNT(*) AS listed_entry_count,
SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count,
SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count,
SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count,
COALESCE(SUM(observed_count), 0) AS total_observed_count,
COALESCE(SUM(materialized_count), 0) AS total_materialized_count,
COALESCE(SUM(trade_count), 0) AS trade_count
FROM k_sol_dex_event_coverage_entries
WHERE decoder_code = 'raydium_clmm'
GROUP BY decoder_code;

2
SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql → validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql
View File

@@ -1,4 +1,4 @@
-- file: SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql
-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql
-- 1. Coverage rows must exist after diagnostics/validation/backfill refresh.
SELECT *