diff --git a/CHANGELOG.md b/CHANGELOG.md index 540b0e4..3fc1ab1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -66,28 +66,17 @@ 0.7.33 - Ajout de la classification diagnostique `pairTradingReadiness` pour les paires, avec `quoteAssetClass`, `tradingRouteRequired`, résumé `pairTradingReadinessSummaries`, profil de validation `0.7.33_pair_trading_readiness` et mise à jour de la sélection UI Demo Pipeline 2 sans modifier la matérialisation trade/candle. 0.7.34 - Ajout du profil `0.7.34_non_trade_liquidity_lifecycle`, matérialisation des tables non-trade liquidité/lifecycle, warning non bloquant pour DEX attendus absents du corpus local, première tranche DLMM : `add_liquidity`, `remove_liquidity`, `initialize_position`, `initialize_bin_array`, intégration de la matérialisation non-trade dans les backfills token/pool ciblés, et distinction `PositionOpen`/`PositionClose` dans `LiquidityEventKind`. 0.7.35 - Ajout du profil `0.7.35_non_trade_fee_reward_admin`, matérialisation des événements non-trade fees/rewards/admin, raccordement aux diagnostics locaux et maintien strict de l’invariant : aucun fee/reward/admin ne peut produire de trade, metric ou candle. -0.7.36 - Consolidation de la famille Meteora : corpus mixte `meteora_damm_v1`, `meteora_damm_v2`, `meteora_dbc` et `meteora_dlmm`, correction des discriminants DAMM v2 / DBC, validation du profil `0.7.36_meteora_family_consolidation`, et reclassement explicite des swaps DAMM v2 / DBC sans payload montant/prix en `non_actionable_trade` afin d’éviter tout trade/candle artificiel. +0.7.36 - Consolidation de la famille Meteora : corpus mixte `meteora_damm_v1`, `meteora_damm_v2`, `meteora_dbc` et `meteora_dlmm`, correction des discriminants DAMM v2 / DBC, validation du profil `0.7.36_meteora_family_consolidation`, et reclassement explicite des swaps DAMM v2 / DBC sans payload montant/prix en `non_actionable_trade` afin d’éviter tout trade/candle artificiel. 0.7.37 - Première tranche metadata/catalog : ajout du profil `0.7.37_token_metadata_catalog_enrichment`, exposition des compteurs metadata dans diagnostics/validation et raccordement UI Demo Pipeline 2 sans rendre les metadata manquantes bloquantes. 0.7.38 - Priorisation des metadata manquantes : ajout du profil `0.7.38_token_metadata_gap_prioritization`, samples `tokenMetadataGapSamples`, priorités tradable/quote/catalog, raccordement UI Demo Pipeline 2 et maintien du caractère non bloquant des metadata incomplètes. 0.7.39 - Réorientation DEX-first : distinction explicite des rôles `dex_effective`, `aggregator_router`, `launch_surface` et `to_verify` dans la matrice DEX, suppression de l’alias ambigu `raydium`, ajout de `metaDAO` et `Printr` comme surfaces à vérifier sans `program_id`, profil `0.7.39_dex_first_effective_swap_surfaces`, validation locale avec invariants DEX-first maintenus et report des launch surfaces après les DEX effectifs. 0.7.40 - Ajout de Demo3 pour la constitution de corpus on-chain par `dex_code` / `program_id` via `getSignaturesForAddress` + `getTransaction`, extraction des mints, deltas SPL Token, comptes pool/state/vault/program candidats, ajout du backfill par signature dans Demo Pipeline 2, et validation pratique sur Raydium AMM v4 sans promotion automatique des comptes candidats. 0.7.41 - Raydium AMM v4 swap decoder v1 : décodage des inner instructions `675kPX...`, extraction pool/state, authority, vaults, mints, routeSource et montants exploitables, matérialisation trades/candles sur transactions OK, matrice AMM v4 passée en `supported`, et validation locale avec invariants trade/candle propres. 0.7.42 - Consolidation famille Raydium : audit conservatoire des instructions Raydium non décodées, décodage CLMM legacy `swap`, cleanup des audits remplacés, classification HTTP `getTransaction` comme requête lourde avec retry/backoff de backfill, mapping des événements non-swap prouvés `raydium_clmm` (`increase_liquidity_v2`, `decrease_liquidity_v2`, `open_position_with_token22_nft`, `close_position`) et `raydium_cpmm` (`initialize`, `withdraw`, `collect_creator_fee`), matérialisation de 25 liquidity events, 1 lifecycle event et 2 fee events sur corpus élargi, conservation des non-swaps AMM v4 legacy en audit. -0.7.43-E5C - Reprise documentaire et normalisation DEX-first : `0.7.43` est conservé comme point de reprise non clos pour le lot Meteora, la suite est redécoupée par DEX/version séparés, le besoin d’un ledger de décodage/replay est acté, les statuts `known` / `observed` / `decoded` / `materialized` / `verified_by_corpus` deviennent obligatoires, et aucun `program_id` ne doit être marqué vérifié sans preuve/corpus reproductible. +0.7.43 - Reprise documentaire et normalisation DEX-first : `0.7.43` est conservé comme point de reprise non clos pour le lot Meteora, la suite est redécoupée par DEX/version séparés, le besoin d’un ledger de décodage/replay est acté, les statuts `known` / `observed` / `decoded` / `materialized` / `verified_by_corpus` deviennent obligatoires, et aucun `program_id` ne doit être marqué vérifié sans preuve/corpus reproductible. 0.7.44 - Ledger de décodage/replay DEX : ajout de `k_sol_dex_decode_replay_ledger`, des DTO/entities/queries associés, des re-exports DB/lib, et intégration dans le replay local pour skipper uniquement l’étape de décodage DEX lorsqu’un passage certifié existe pour la même version logique de decoder. Les transactions multi-event ou multi-token restent marquées `unsafe` et sont redécodées sauf option future plus explicite ; le replay continue de reconstruire détection, matérialisation, trades, candles et classifications à partir des events persistés. 0.7.45 - Meteora DLMM normalisation finale : consolidation séparée de `meteora_dlmm` sur corpus dédié, maintien du wrapper Anchor `anchor_self_cpi_log` `e445a52e51cb9a1d`, enrichissement des swaps via `Swap` / `Swap2Evt`, cleanup des audits Anchor CPI swap déjà couverts, ajout des events upstream Git/IDL observés et vérifiés par corpus (`lb_pair_create_event`, `add_liquidity_event`, `remove_liquidity_event`, `claim_fee_event`, `position_create_event`, `position_close_event`, `close_position_if_empty`, `remove_liquidity_by_range2`, `add_liquidity_by_strategy2`, `add_liquidity_by_weight`), conservation des deux audits résiduels `e8abf2613a4d232d` en `instruction_audit` faute de mapping upstream Git/IDL confirmé, matérialisation locale validée avec `15` liquidity events et `6` lifecycle events sur le corpus DLMM élargi, et version logique replay `dex_decode.v0.7.45.dlmm_add_liquidity_strategies1`. Aucun nouveau `program_id` n’est déclaré vérifié sans preuve/corpus reproductible. -0.7.46 - Meteora DAMM v1 events : extension conservatoire du decoder `meteora_damm_v1` à partir du mapping upstream Git decoder source `meteora-pools-decoder` et du corpus local fourni pour les discriminants observés `07a68aabceabecf4`, `3095dc823d0b09b2`, `856d2cb338ee7221`, `a9204f8988e84689`, `3657a51345e3dae0` et `1513d02bed3eff57` ; ajout des events create_pool, add/remove liquidity, claim_fee, create_lock_escrow et lock_liquidity ; ajout des exports publics associés ; raccordement de la persistance DEX et de la classification non-trade ; passage du replay local à `dex_decode.v0.7.46.damm_v1_events1`. Le programme Meteora Vault reste seulement référencé comme compte/programme associé quand il apparaît dans les comptes DAMM v1 ; aucun `program_id` vault n’est marqué vérifié sans corpus direct dédié. -0.7.46-demo3 - Correction ciblée de Demo3 pour la découverte/backfill : ajout d’un décodage léger des instructions `meteora_damm_v1` connues par discriminant upstream Git dans `onchain_dex_pair_discovery`, classification instruction-scoped prioritaire pour éviter qu’un `Swap` soit classé `add_liquidity` à cause de logs mixtes de transaction, filtrage `target_event` strict sur les surfaces explicites, conservation des transactions mixtes quand un target explicite est demandé malgré `excludeSwaps`, et ajout des cibles UI `create_lock_escrow` / `lock_liquidity`. -0.7.46-demo3-paged - Amélioration Demo3 discovery : pagination `getSignaturesForAddress` via `before_signature` / `until_signature`, scan de plusieurs adresses source dans une seule requête, déduplication des signatures multi-pool, compteur de pages, curseurs `next_before` par adresse et ordre de traitement `newest_first` / `oldest_first` pour constituer un corpus depuis les premières signatures d’un pool sans promouvoir de nouveau `program_id`. -0.7.46-final - Renommage documentaire et payload des anciens statuts/fonctions liés au dépôt source vers une terminologie générique `upstream_git_*` : `proofStatus` utilise désormais `upstream_git_local_corpus_observed`, `upstream_git_mapped_unverified` et `upstream_git_layout_unverified`; les payloads DAMM v1 utilisent `upstreamInstructionName`; la documentation prépare `0.7.47` comme Upstream Git Registry / DEX discovery preparation au lieu d’une tranche DAMM v2 immédiate. -0.7.47 - Upstream Git Registry / DEX discovery preparation : ajout d’un registre générique `upstream_git` pour indexer `program_id`, discriminants d’instructions/events, noms d’entrées et familles de programmes depuis Carbon et sources Git/IDL externes ; extension Demo3 aux targets multi-surfaces, orderbook, burn/mint/transfer/wrap/unwrap/stake ; ajout de groupes de signatures réussies/échouées pour alimenter Demo2 ; maintien strict de l’invariant : aucune entrée upstream Git ne produit trade/candle sans decoder spécialisé et corpus local. -0.7.47-openbook-v2-audit - Ajout d’un decoder local `openbook_v2` audit-only : instructions `place_order`, `cancel_order_by_client_order_id`, `consume_events`, `settle_funds`, `close_open_orders_account`; cleanup du fallback `upstream_git.instruction_match`; extraction audit des `Program return` et `Program data`; mapping des logs `FillLog`, `OpenOrdersPositionLog`, `TotalOrderFillEvent`, `SettleFundsLog`; aucune matérialisation trade/candle. -0.7.47-phoenix-v1-audit - Ajout d’un decoder local `phoenix_v1` audit-only : `order_place`, `order_cancel`, `funds_withdraw`, `log`; parsing strict des instructions log `0x0f`; décodage audit du header Phoenix log et des events `Reduce`, `Place`, `TimeInForce`; correction du mapping `PlaceMultiplePostOnlyOrders` tag `0x10`; aucune matérialisation trade/candle. -0.7.47-doc-matrix - Révision documentaire : ajout d’une matrice DEX dédiée, ajout explicite des sources Git/IDL à consulter, et redécoupage du plan `0.7.48+` en un DEX/version par tranche afin d’éviter les lots “tous events/tous decoders” trop larges. -0.7.47-doc-event-coverage - Ajout d'une matrice événementielle complémentaire `DEX_EVENT_COVERAGE_MATRIX.md` pour suivre, par DEX/version, les familles `swap`, `pool_create`, `liquidity`, `position`, `fee`, `reward`, `admin/config`, `mint`, `burn`, `transfer`, `account_create/close`, `wrap/unwrap`, `orderbook`, `vault`, `lock/unlock`, `launch` et `migration`; ajout de `DB_EVENT_MODEL_REVIEW.md` pour clarifier que `k_sol_dex_decoded_events` suffit à l'audit-only mais que des tables transversales sont nécessaires pour exploiter transfers, orderbook, vault, launch/migration et coverage upstream en requêtes métier. -0.7.48-pre-event-coverage-sync - Raccordement de `k_sol_dex_event_coverage_entries` au registre upstream Git : ajout de `DexEventCoverageService`, sync des entrées registry vers SQLite, inférence conservatoire `event_family` / `expected_db_target`, mapping local limité aux events Raydium déjà connus, refresh des compteurs observés/matérialisés depuis `k_sol_dex_decoded_events` et tables non-trade existantes, sans modification des decoders ni de la matérialisation trade/candle. -0.7.48-pre-event-coverage-fix-docs - Correction du refresh SQL `k_sol_dex_event_coverage_entries` pour éviter les requêtes dynamiques non compatibles avec `sqlx::query` 0.9 ; mise à jour documentaire README/ROADMAP pour acter `0.7.48-pre` comme checkpoint DB/reporting et réaligner la suite sur l’ordre Raydium avant Meteora (`0.7.48 raydium_cpmm`, `0.7.49 raydium_clmm`, puis Pump/Meteora). -0.7.48-pre-event-coverage-report - Clôture du checkpoint `0.7.48-pre` : raccordement des summaries `k_sol_dex_event_coverage_entries` aux diagnostics locaux, ajout des compteurs agrégés de couverture au `LocalPipelineDiagnosticSummaryDto` et au `LocalPipelineValidationReportDto`, ajout du profil `0.7.48-pre_event_coverage_db_checkpoint`, exposition du profil dans Demo Pipeline 2, et maintien des invariants : aucun decoder DEX modifié, aucun trade/candle créé, aucun `program_id` promu sans corpus. -0.7.48-pre-event-coverage-validation-scope - Correction du profil `0.7.48-pre_event_coverage_db_checkpoint` : le contrôle bloquant des trade candidates non matérialisés est maintenant borné aux DEX attendus de la tranche Raydium (`raydium_cpmm`, `raydium_clmm`, `raydium_amm_v4`) afin qu’un DEX partiel hors scope, comme `fluxbeam`, reste diagnostiqué sans bloquer le checkpoint DB/event coverage. -0.7.48-raydium-cpmm-program-data - Complément Raydium CPMM : auto-sync conservatoire de `k_sol_dex_event_coverage_entries` lors des diagnostics/backfills si la table de coverage est vide, décodage des events CPMM `Program data` `swap_event` et `lp_change_event` sans sélecteur Anchor self-CPI, conservation explicite de `swap_event` en audit-only pour éviter tout doublon de trade/candle, matérialisation liquidity conservatoire de `lp_change_event` via `changeType`, et maintien des invariants failed transaction / non-trade / upstream Git. -0.7.48 - Raydium CPMM event coverage clôturé : décodage spécialisé des instructions/events CPMM Carbon/Raydium/fnzero, auto-sync coverage, index technique `k_sol_instruction_observations`, recherche Demo3 par instruction/discriminant, matérialisation validée des swaps, lifecycle, fees, admin/config, deposit/withdraw et `lp_change_event`, conservation de `swap_event` et de l’instruction inconnue `40f4bc78a7e9690a` en audit-only, et maintien des entrées `close_permission_pda` / `update_pool_status` en `upstream_git_mapped_unverified` faute de corpus local. +0.7.46 - Meteora DAMM v1 events finalisés : extension conservatoire du decoder `meteora_damm_v1` depuis upstream Git/corpus local, events create_pool/add/remove liquidity/claim_fee/create_lock_escrow/lock_liquidity, corrections Demo3 ciblées et pagination multi-source, renommage documentaire/payload vers `upstream_git_*`, sans promotion de programme vault ou trade/candle sans preuve locale. +0.7.47 - Upstream Git Registry / DEX discovery preparation : registre générique `upstream_git`, extension Demo3 aux targets multi-surfaces, premiers decoders audit-only OpenBook v2 et Phoenix v1, matrices DEX/event coverage, revue DB et invariant maintenu : aucune entrée upstream ne produit trade/candle sans decoder spécialisé et corpus local. +0.7.48 - Raydium CPMM event coverage clôturé : couverture instructions/events CPMM Carbon/Raydium/fnzero, table coverage synchronisée, `k_sol_instruction_observations`, recherche Demo3 par instruction/discriminant, matérialisation validée des swaps, lifecycle, fees, admin/config, deposit/withdraw et `lp_change_event`, `swap_event` audit-only, fallback upstream remplacé quand le decoder local couvre l’entrée. +0.7.49 - Raydium CLMM event coverage clôturé : 45 entrées listées, 33 instructions locales observées/décodées, 25 entrées matérialisées, ajout `k_sol_orderbook_events`, matérialisation des limit orders, liquidity, fees, rewards, admin/config et lifecycle prouvés par corpus, préparation audit-only des 11 Anchor Program-data events non observés, nettoyage des `raydium_clmm.instruction_audit` et `upstream_git.instruction_match` redondants, validation des invariants failed transaction / non-swap / trade-candle. diff --git a/Cargo.toml b/Cargo.toml index e5c7622..b039e13 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -8,7 +8,7 @@ members = [ ] [workspace.package] -version = "0.7.48" +version = "0.7.49" edition = "2024" license = "MIT" repository = "https://git.sasedev.com/Sasedev/khadhroony-bobobot" @@ -89,4 +89,4 @@ manual_unwrap_or_default = "allow" manual_find = "allow" explicit_counter_loop = "allow" get_first = "allow" -implicit_saturating_sub = "allow" \ No newline at end of file +implicit_saturating_sub = "allow" diff --git a/README.md b/README.md index eda1cd4..738c10e 100644 --- a/README.md +++ b/README.md @@ -7,18 +7,53 @@ Ce document reflète le point de reprise `0.7.43-E5C` et l’état de consolidation atteint après `0.7.45` pour `meteora_dlmm`. La version Cargo a évolué ensuite à `0.7.46` côté workspace. Le lot Meteora initialement ouvert en bloc a été redécoupé : `meteora_dlmm` est traité séparément, puis la suite reprend `meteora_damm_v1`, `meteora_damm_v2` et `meteora_dbc` un par un. -## État de reprise actuel `0.7.47-1FE5` +## État courant finalisé `0.7.49` -Le point de reprise courant n’est plus `0.7.43-E5C`. La branche de travail actuelle est `0.7.47-1FE5` : le registre upstream Git est en place, OpenBook v2 et Phoenix v1 disposent de decoders locaux **audit-only**, et la suite doit être conduite par DEX/version au lieu de tenter tous les events en une seule session. +La branche de travail `0.7.49` clôture la tranche `raydium_clmm` après la clôture fonctionnelle de `0.7.48 raydium_cpmm`. Le code Rust de `kb_lib` est considéré finalisé pour cette tranche, sous réserve des validations locales habituelles (`cargo fmt`, `cargo test -p kb_lib`, `cargo clippy -p kb_lib --all-targets -- -D warnings`). -Les nouveaux chemins audit-only doivent rester non matérialisants : aucun event OpenBook v2 ou Phoenix v1 ne doit produire de trade, metric ou candle tant que le sens économique complet n’est pas validé. +État CLMM validé sur corpus local après replay forcé : + +```text +listed_entry_count = 45 +decoded_entry_count = 33 +observed_entry_count = 33 +materialized_entry_count = 25 +total_observed_count = 2560 +total_materialized_count = 1367 +trade_count = 1186 +raydium_clmm.instruction_audit résiduel = 0 +upstream_git.instruction_match localement couvert = 0 +failed tx matérialisées = 0 +non-swap CLMM avec trade_count > 0 = 0 +``` + +Les 11 Anchor / `Program data` events CLMM restent listés en `upstream_git_unverified` car aucun corpus local ne les observe encore. Le code est préparé pour les accueillir en audit-only lorsqu’ils apparaîtront dans un corpus local, sans créer de trade/candle par défaut. + +La prochaine tranche fonctionnelle est `0.7.50 raydium_launchpad`, avant `0.7.51 raydium_amm_v4` et `0.7.52 raydium_stable`. + +## Organisation documentaire + +La racine conserve uniquement les documents de pilotage principaux : + +- `README.md` ; +- `ROADMAP.md` ; +- `CHANGELOG.md`. + +Les documents spécialisés sont rangés dans : + +- `docs/` pour les matrices et revues de modèle ; +- `docs/reports/` pour les rapports de couverture DEX/version ; +- `docs/prompts/` pour les prompts de reprise ; +- `validation_sql/` pour les scripts SQL de validation. Voir aussi : -- `DEX_DECODER_MATRIX.md` pour la matrice DEX détaillée ; -- `ROADMAP.md` pour le plan révisé `0.7.48` à `0.7.61` ; -- `CHANGELOG.md` pour les tranches `0.7.47-*`. - +- `docs/DEX_DECODER_MATRIX.md` pour la matrice DEX détaillée ; +- `docs/DEX_EVENT_COVERAGE_MATRIX.md` pour la matrice de familles d’events ; +- `docs/DB_EVENT_MODEL_REVIEW.md` pour la revue du modèle DB ; +- `docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md` et `validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` pour la clôture CPMM ; +- `docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md` et `validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql` pour la clôture CLMM ; +- `docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md` pour reprendre en `0.7.50`. ## Sources upstream Git / IDL à utiliser en `0.7.47+` @@ -345,17 +380,14 @@ Si une requête DB est ajoutée ou modifiée, mettre à jour les re-exports dans La priorité immédiate après le point de reprise `0.7.43-E5C` est : -1. `0.7.43` : resynchronisation documentaire, normalisation DEX-first et gel du point de reprise ; -2. `0.7.44` : ajout du ledger de décodage/replay et options de replay `force` / skip sûr ; -3. `0.7.45` : reprise séparée de `meteora_dlmm` — clôturée côté corpus DLMM actuel ; -4. `0.7.46` : reprise séparée de `meteora_damm_v1` — clôturée côté corpus DAMM v1 local ; -5. `0.7.47` : Upstream Git Registry / DEX discovery preparation — registre générique des `program_id`, discriminants, instructions et events issus de dépôts Git externes, tous non vérifiés par défaut ; -6. `0.7.48-pre` : event coverage + DB model checkpoint — table coverage, sync upstream, refresh counts, diagnostics et profil validation ; -7. `0.7.48` : reprise séparée de `raydium_cpmm` ; -8. `0.7.49` : reprise séparée de `raydium_clmm` ; -9. `0.7.50` : reprise séparée de `pump_swap` ; -10. `0.7.51` : reprise séparée de `pump_fun` ; -11. `0.7.52+` : Meteora puis validation progressive des autres DEX/surfaces issus du registre upstream Git : Orca, FluxBeam, DexLab, Lifinity, Phoenix, OpenBook, Stabble, BonkSwap, Boop, Moonshot, Heaven, Wavebreak, Vertigo, Virtuals, Pancake Swap, OKX DEX, Raydium LaunchLab/Stable/Locking, puis launch surfaces. +1. `0.7.48` : `raydium_cpmm` — clôturé côté event coverage ; +2. `0.7.49` : `raydium_clmm` — clôturé côté instructions observées, matérialisation non-trade prouvée et nettoyage fallback ; +3. `0.7.50` : `raydium_launchpad` ; +4. `0.7.51` : `raydium_amm_v4` ; +5. `0.7.52` : `raydium_stable` ; +6. `0.7.53` : `pump_swap` ; +7. `0.7.54` : `pump_fun` ; +8. `0.7.55+` : Meteora, Phoenix/OpenBook, Orca puis validation progressive des autres DEX/surfaces issus du registre upstream Git. Garde-fous constants : @@ -436,9 +468,9 @@ La matrice DEX/version doit être complétée par une matrice événementielle e Voir : -- `DEX_DECODER_MATRIX.md` pour le statut par DEX/version ; -- `DEX_EVENT_COVERAGE_MATRIX.md` pour les familles d'events à couvrir ; -- `DB_EVENT_MODEL_REVIEW.md` pour les ajouts DB à envisager avant `0.7.48+`. +- `docs/DEX_DECODER_MATRIX.md` pour le statut par DEX/version ; +- `docs/DEX_EVENT_COVERAGE_MATRIX.md` pour les familles d'events à couvrir ; +- `docs/DB_EVENT_MODEL_REVIEW.md` pour les ajouts DB à envisager avant `0.7.48+`. ## Note 0.7.48-pre — Event coverage DB checkpoint @@ -462,9 +494,10 @@ La suite fonctionnelle reprend par Raydium avant Meteora : 1. `0.7.48` — `raydium_cpmm` ; 2. `0.7.49` — `raydium_clmm` ; -3. `0.7.50` — `pump_swap` ; -4. `0.7.51` — `pump_fun` ; -5. `0.7.52+` — Meteora puis les autres DEX/surfaces. +3. `0.7.50` — `raydium_launchpad` ; +4. `0.7.51` — `raydium_amm_v4` ; +5. `0.7.52` — `raydium_stable` ; +6. `0.7.53+` — Pump, Meteora, Phoenix/OpenBook, Orca puis les autres DEX/surfaces. ## Note 0.7.48 — Raydium CPMM event coverage @@ -478,9 +511,9 @@ Aucune nouvelle table DB n'est ajoutée en `0.7.48`. Les transfers, token accoun Voir aussi : -- `DEX_EVENT_COVERAGE_MATRIX.md` pour la couverture par familles ; -- `RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md` pour le rapport de tranche ; -- `SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` pour les requêtes de validation. +- `docs/DEX_EVENT_COVERAGE_MATRIX.md` pour la couverture par familles ; +- `docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md` pour le rapport de tranche ; +- `validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` pour les requêtes de validation. Complément `0.7.48-raydium-cpmm-program-data` : la coverage DB est maintenant synchronisée automatiquement quand un refresh de coverage est demandé sur une base neuve et qu'aucune ligne n'existe encore. Les backfills token/pool/signature déclenchent aussi un refresh best-effort de la coverage afin d'éviter un état incohérent où `k_sol_dex_decoded_events` est rempli mais `k_sol_dex_event_coverage_entries` reste vide. @@ -499,3 +532,24 @@ La tranche `0.7.48` clôture la couverture Raydium CPMM sur corpus local. Les en La table technique `k_sol_instruction_observations` est ajoutée pour indexer localement les instructions observées par `decoder_code`, `instruction_name` et `discriminator_hex`. Demo3 peut rechercher par instruction/discriminant, ce qui reproduit localement l’usage pratique du filtre Solscan `instruction=`. État final CPMM observé : `561` trades, `50` liquidity events, `9` lifecycle events, `25/25` `lp_change_event` matérialisés, `swap_event` audit-only à `0` trade, et deux entrées connues mais non observées (`close_permission_pda`, `update_pool_status`) conservées en `upstream_git_mapped_unverified`. + +## Note 0.7.49 — Raydium CLMM event coverage final + +La tranche `0.7.49` clôture `raydium_clmm` comme deuxième tranche Raydium après CPMM. Elle ajoute la couverture complète des instructions CLMM observées depuis Carbon, IDL Raydium, Pinax, fnzero et corpus Solscan/backfill, ainsi que la table transversale `k_sol_orderbook_events` pour les instructions limit-order. + +Points finalisés : + +- `45` entrées listées dans `k_sol_dex_event_coverage_entries` ; +- `33` instructions CLMM avec `local_event_kind` spécialisé ; +- `33` instructions observées dans le corpus local ; +- `25` entrées matérialisées ; +- swaps matérialisés uniquement via `swap` / `swap_v2` ; +- limit orders `open`, `increase`, `decrease`, `close`, `settle` décodés et matérialisés en `k_sol_orderbook_events` quand la transaction réussit ; +- non-trades CLMM vers `liquidity`, `fee`, `reward`, `admin`, `lifecycle` et `orderbook` sans trade/candle ; +- transactions échouées conservées audit-only ; +- `raydium_clmm.instruction_audit` résiduel à zéro ; +- `upstream_git.instruction_match` localement couvert à zéro après replay ; +- 11 Anchor / `Program data` events CLMM préparés mais conservés `upstream_git_unverified` faute d’observation locale. + +La validation finale est dans `validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql`. + diff --git a/ROADMAP.md b/ROADMAP.md index 4caa189..0ab3157 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -4,7 +4,7 @@ ## 0.7.47-1FE5 — Décision de planification : ne plus viser “tous les events en une session” -La phase `0.7.47` a montré que l’objectif “réimplémenter tous les décodeurs Carbon et toutes les sources en un seul bloc” est trop large. Le plan est donc redécoupé en **un DEX/version par tranche**, avec une matrice documentaire dédiée : `DEX_DECODER_MATRIX.md`. +La phase `0.7.47` a montré que l’objectif “réimplémenter tous les décodeurs Carbon et toutes les sources en un seul bloc” est trop large. Le plan est donc redécoupé en **un DEX/version par tranche**, avec une matrice documentaire dédiée : `docs/DEX_DECODER_MATRIX.md`. Règles de planification : @@ -28,24 +28,25 @@ Règles de planification : | `https://github.com/all-in-one-blockchain/phoenix-onchain-mm` | Source Phoenix/MM complémentaire. | | `https://docs.vybenetwork.com/docs/available-dexs-amms` | Source externe de découverte DEX/AMM, non vérifiante. | -### Plan révisé `0.7.48` à `0.7.61` +### Plan révisé `0.7.48` à `0.7.62+` | Version cible | Scope | Objectif de clôture | |---|---|---| -| `0.7.48` | `raydium_cpmm` | Reprendre tous les discriminants/events depuis Carbon/Solana Streamer ; vérifier swaps, liquidity, fees/admin ; confirmer matérialisation trade/candle et non-trade. | -| `0.7.49` | `raydium_clmm` | Couvrir toutes les instructions CLMM : swaps, positions, liquidity, fees/rewards, Token-2022 ; valider matérialisation non-trade. | -| `0.7.50` | `pump_swap` | Couvrir `buy/sell` et tous les events auxiliaires disponibles : fees, cashback, volume accumulator, admin/config. | -| `0.7.51` | `pump_fun` | Traiter launch/bonding/migration ; séparer création token, buy/sell bonding, migration vers DEX effectif. | -| `0.7.52` | `meteora_dbc` | Couverture DBC : bonding curve, swap, migration, launch attribution, fees/admin, non-trade. | -| `0.7.53` | `meteora_dlmm` | Audit final de parité avec sources Git/IDL ; fermer ou documenter les audits résiduels. | -| `0.7.54` | `meteora_damm_v1` | Parité upstream complète ; résoudre les cas non matérialisés faute de pool/pair quand possible. | -| `0.7.55` | `meteora_damm_v2` | Couverture DAMM v2 complète : create, swap, liquidity, fees/admin/config ; décider trade actionability. | -| `0.7.56` | `phoenix_v1` | Finir tous les events Git disponibles en audit ; préparer mais ne pas activer trade materialization. | -| `0.7.57` | `openbook_v2` | Finir layouts logs/events ; définir conditions futures de trade/candle sans les activer par défaut. | -| `0.7.58` | `orca_whirlpools` | Reprendre Whirlpools depuis IDL/source : swaps, pools, positions, liquidity, fees/rewards. | -| `0.7.59` | Launch surfaces | Raydium LaunchLab/Launchpad, PumpFun migration, Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk. | -| `0.7.60` | DEX historiques / candidats | FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi, etc. par corpus. | -| `0.7.61` | Validation consolidée | Rejouer une base neuve multi-DEX, vérifier matrice, zéro faux trade/candle, rapport de couverture par DEX/event. | +| `0.7.48` | `raydium_cpmm` | Clôturé : instructions/events CPMM, lifecycle, fees, admin/config, deposit/withdraw, `lp_change_event`, invariants trade/candle. | +| `0.7.49` | `raydium_clmm` | Clôturé : 33 instructions observées/décodées, orderbook CLMM, liquidity/fee/reward/admin/lifecycle, fallbacks upstream nettoyés, 11 Program-data events préparés mais non observés. | +| `0.7.50` | `raydium_launchpad` | Reprendre Launchpad comme surface Raydium prioritaire : identifier les program ids/IDL, launch, pool creation, migration, bonding éventuel, fees/admin, et rattachement au DEX effectif. | +| `0.7.51` | `raydium_amm_v4` | Reprendre AMM v4 legacy au même niveau de couverture que CPMM/CLMM : swaps, pool lifecycle, liquidity, fees/admin, side effects documentés. | +| `0.7.52` | `raydium_stable` | Reprendre Raydium Stable : program ids/IDL, swaps stables, pool lifecycle, liquidity, fees/admin, invariants pricing/candles. | +| `0.7.53` | `pump_swap` | Couvrir `buy/sell` et tous les events auxiliaires disponibles : fees, cashback, volume accumulator, admin/config. | +| `0.7.54` | `pump_fun` | Traiter launch/bonding/migration ; séparer création token, buy/sell bonding, migration vers DEX effectif. | +| `0.7.55` | `meteora_dbc` | Couverture DBC : bonding curve, swap, migration, launch attribution, fees/admin, non-trade. | +| `0.7.56` | `meteora_dlmm` | Audit final de parité avec sources Git/IDL ; fermer ou documenter les audits résiduels. | +| `0.7.57` | `meteora_damm_v1` | Parité upstream complète ; résoudre les cas non matérialisés faute de pool/pair quand possible. | +| `0.7.58` | `meteora_damm_v2` | Couverture DAMM v2 complète : create, swap, liquidity, fees/admin/config ; décider trade actionability. | +| `0.7.59` | `phoenix_v1` | Finir tous les events Git disponibles en audit ; préparer mais ne pas activer trade materialization. | +| `0.7.60` | `openbook_v2` | Finir layouts logs/events ; définir conditions futures de trade/candle sans les activer par défaut. | +| `0.7.61` | `orca_whirlpools` | Reprendre Whirlpools depuis IDL/source : swaps, pools, positions, liquidity, fees/rewards. | +| `0.7.62+` | Launch surfaces / DEX candidats / validation consolidée | Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk, FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi puis base neuve multi-DEX. | Ce plan remplace les anciens regroupements larges `0.7.50+` qui mélangeaient plusieurs DEX dans une même version. @@ -1245,7 +1246,7 @@ Statut : implémenté en micro-tranche DB/reporting, sans modifier les decoders Fait : -- maintien de `DEX_EVENT_COVERAGE_MATRIX.md` en plus de `DEX_DECODER_MATRIX.md` ; +- maintien de `docs/DEX_EVENT_COVERAGE_MATRIX.md` en plus de `docs/DEX_DECODER_MATRIX.md` ; - ajout de `k_sol_dex_event_coverage_entries` dans `kb_lib/src/db/schema.rs` ; - ajout des entity/DTO/queries/re-exports associés ; - ajout de `DexEventCoverageService` pour synchroniser les entrées du registre upstream Git vers la table de coverage ; @@ -1278,75 +1279,82 @@ Objectif : reprendre `raydium_cpmm` en premier, avant Meteora, avec une couvertu - vérifier par SQL que les non-trades ne produisent aucun trade/candle. ### 6.081. Version `0.7.49` — `raydium_clmm` event coverage -Objectif : reprendre `raydium_clmm` après CPMM, avec couverture des swaps, positions, liquidité, rewards, fees, protocol fees et cas Token-2022. +Objectif : clôturer `raydium_clmm` après CPMM. -À faire : +Réalisé : -- lister tous les events/instructions CLMM depuis Carbon/fnzero/IDL ; -- consolider `swap`, `swap_v2`, open/close position, increase/decrease liquidity, reward/fee/admin ; -- classer les events non observés en `upstream_git_mapped_unverified` ; -- matérialiser uniquement les events prouvés par corpus ; -- vérifier absence de faux trades/candles. +- couverture locale de `45` entrées CLMM ; +- `33` instructions spécialisées, observées et décodées ; +- matérialisation contrôlée de `25` entrées vers trade, liquidity, fee, reward, admin, lifecycle et orderbook ; +- ajout de `k_sol_orderbook_events` pour les limit orders CLMM ; +- suppression automatique des fallbacks `upstream_git.instruction_match` localement couverts ; +- préparation audit-only des 11 Anchor / `Program data` events non encore observés ; +- invariants validés : aucun faux trade/candle, aucune matérialisation sur transaction échouée, `raydium_clmm.instruction_audit` résiduel à zéro. -### 6.082. Version `0.7.50` — `pump_swap` event coverage +### 6.082. Version `0.7.50` — `raydium_launchpad` event coverage +Objectif : reprendre Raydium Launchpad comme prochaine surface Raydium, avant AMM v4 et Stable. + +À faire : identifier les program ids/IDL depuis sources Git/IDL/Solscan, couvrir launch/pool creation, migration/rattachement au DEX effectif, fees/admin/config, éventuels side effects SPL/Token-2022, et matérialiser seulement les events prouvés par corpus local. + +### 6.083. Version `0.7.51` — `raydium_amm_v4` event coverage +Objectif : hisser AMM v4 legacy au niveau de couverture CPMM/CLMM. + +À faire : revisiter swaps, initialize/pool lifecycle, add/remove liquidity, fees/admin/config, side effects SPL, failed transaction safety, fallback upstream et validation SQL dédiée. + +### 6.084. Version `0.7.52` — `raydium_stable` event coverage +Objectif : reprendre Raydium Stable comme troisième tranche Raydium post-CLMM. + +À faire : vérifier program ids/IDL, swaps stables, liquidity, pool lifecycle, fees/admin/config, cohérence des montants/prix et absence de faux trades/candles. + +### 6.085. Version `0.7.53` — `pump_swap` event coverage Objectif : compléter `pump_swap` au-delà de `buy/sell`. À faire : couvrir fees, cashback, volume accumulator, admin/config et autres events upstream disponibles, tout en maintenant l’invariant non-trade = zéro trade/candle. -### 6.083. Version `0.7.51` — `pump_fun` launch/bonding/migration +### 6.086. Version `0.7.54` — `pump_fun` launch/bonding/migration Objectif : séparer launch/bonding de DEX effectif et valider migration vers PumpSwap ou autre surface tradable. À faire : traiter create, buy/sell bonding, update/config, mint/burn éventuels, migration/graduate et rattachement au pool tradable. -### 6.084. Version `0.7.52` — `meteora_dbc` séparé +### 6.087. Version `0.7.55` — `meteora_dbc` séparé Objectif : reprendre Meteora après les tranches Raydium et Pump, en séparant bonding/launch, swap effectif, migration et attribution d’origine. À faire : vérifier swaps exploitables, migration, lifecycle, mint/burn éventuels, launch attribution, fees/admin, sans candle artificielle sur events non pricés. -### 6.085. Version `0.7.53` — `meteora_dlmm` parité upstream finale +### 6.088. Version `0.7.56` — `meteora_dlmm` parité upstream finale Objectif : comparer la couverture locale DLMM déjà avancée avec toutes les sources Git/IDL et documenter ou fermer les audits résiduels. À faire : revalider swaps, liquidity, positions, lifecycle, fees/rewards/admin, et garder les discriminants non mappés en audit documenté. -### 6.086. Version `0.7.54` — `meteora_damm_v1` parité upstream finale +### 6.089. Version `0.7.57` — `meteora_damm_v1` parité upstream finale Objectif : compléter la tranche DAMM v1 déjà engagée, résoudre les surfaces non observées et améliorer le rattachement pool/pair quand possible. À faire : vérifier toutes les instructions upstream restantes, matérialiser uniquement les events prouvés et documenter les cas sans pool/pair local. -### 6.087. Version `0.7.55` — `meteora_damm_v2` séparé +### 6.090. Version `0.7.58` — `meteora_damm_v2` séparé Objectif : reprendre DAMM v2 comme DEX effectif séparé après disponibilité du ledger de coverage. À faire : consolider create_pool, swaps exploitables, configs dynamiques, liquidity, fees/admin, lifecycle ; conserver les swaps sans payload montant/prix fiable comme `non_actionable_trade`. -### 6.088. Version `0.7.56` — `phoenix_v1` audit-only complet +### 6.091. Version `0.7.59` — `phoenix_v1` audit-only complet Objectif : finir tous les events Git disponibles en audit, sans activer de trade/candle. À faire : couvrir `Fill`, `FillSummary`, `Fee`, `Evict`, `ExpiredOrder` et autres logs/events disponibles ; préparer le futur modèle orderbook sans matérialisation marché par défaut. -### 6.089. Version `0.7.57` — `openbook_v2` audit-only complet +### 6.092. Version `0.7.60` — `openbook_v2` audit-only complet Objectif : finir les layouts logs/events OpenBook v2 et définir les conditions futures de matérialisation orderbook/trade. À faire : vérifier fills, settle, consume events, open orders create/close, maker/taker, lots/decimals et sens économique avant toute promotion. -### 6.090. Version `0.7.58` — `orca_whirlpools` event coverage +### 6.093. Version `0.7.61` — `orca_whirlpools` event coverage Objectif : reprendre Whirlpools depuis IDL/source avec corpus dédié. À faire : swaps, pools, positions, liquidity, fees/rewards, tick arrays, mint/burn/Token-2022 si applicable. -### 6.091. Version `0.7.59` — Launch surfaces -Objectif : traiter les surfaces de lancement après les DEX effectifs prioritaires. +### 6.094. Version `0.7.62+` — Launch surfaces, DEX historiques/candidats et validation consolidée +Objectif : traiter les surfaces restantes puis rejouer une base neuve multi-DEX. -À faire : Raydium LaunchLab/Launchpad, PumpFun migration, Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk, avec séparation stricte launch origin / pool origin / DEX effectif. - -### 6.092. Version `0.7.60` — DEX historiques / candidats -Objectif : valider les DEX ou surfaces candidates par corpus, sans promotion automatique depuis les sources externes. - -À faire : FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi et autres entrées Vybe/registry. - -### 6.093. Version `0.7.61` — Validation consolidée -Objectif : rejouer une base neuve multi-DEX et valider les invariants du pipeline complet. - -À faire : rapport coverage par DEX/event, zéro faux trade/candle, corpus documentés, matrices cohérentes, diagnostics bloquants à zéro. +À faire : Moonshot/Moonit, Boop, Heaven, Bags, LetsBonk, FluxBeam, DexLab, Lifinity, Stabble, BonkSwap, GooseFX, Obric, SolFi et autres entrées Vybe/registry ; rapport coverage par DEX/event, zéro faux trade/candle, corpus documentés, matrices cohérentes, diagnostics bloquants à zéro. ### 6.091. Version `0.8.x` — Analyse et filtrage Objectif : transformer les événements bruts en signaux exploitables. @@ -1540,18 +1548,21 @@ Ordre de travail recommandé pour la suite : 3. `0.7.46` : `meteora_damm_v1` — clos côté corpus local ; 4. `0.7.47` : Upstream Git Registry / DEX discovery preparation — acquis ; 5. `0.7.48-pre` : event coverage + DB model checkpoint — clos après table, sync upstream, refresh counts, diagnostics et profil validation ; -6. `0.7.48` : `raydium_cpmm` ; -7. `0.7.49` : `raydium_clmm` ; -8. `0.7.50` : `pump_swap` ; -9. `0.7.51` : `pump_fun` ; -10. `0.7.52` : `meteora_dbc` ; -11. `0.7.53` : `meteora_dlmm` parité upstream finale ; -12. `0.7.54` : `meteora_damm_v1` parité upstream finale ; -13. `0.7.55` : `meteora_damm_v2` ; -14. `0.7.56` : `phoenix_v1` audit-only complet ; -15. `0.7.57` : `openbook_v2` audit-only complet ; -16. `0.7.58` : `orca_whirlpools` ; -17. `0.7.59+` : launch surfaces, DEX candidats/historiques et validation consolidée. +6. `0.7.48` : `raydium_cpmm` — clos ; +7. `0.7.49` : `raydium_clmm` — clos ; +8. `0.7.50` : `raydium_launchpad` ; +9. `0.7.51` : `raydium_amm_v4` ; +10. `0.7.52` : `raydium_stable` ; +11. `0.7.53` : `pump_swap` ; +12. `0.7.54` : `pump_fun` ; +13. `0.7.55` : `meteora_dbc` ; +14. `0.7.56` : `meteora_dlmm` parité upstream finale ; +15. `0.7.57` : `meteora_damm_v1` parité upstream finale ; +16. `0.7.58` : `meteora_damm_v2` ; +17. `0.7.59` : `phoenix_v1` audit-only complet ; +18. `0.7.60` : `openbook_v2` audit-only complet ; +19. `0.7.61` : `orca_whirlpools` ; +20. `0.7.62+` : launch surfaces, DEX candidats/historiques et validation consolidée. Garde-fous constants : @@ -1602,7 +1613,7 @@ cargo test -p kb_lib cargo clippy -p kb_lib --all-targets -- -D warnings ``` -Puis relancer la validation SQL `SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` sur la base de corpus CPMM. +Puis relancer la validation SQL `validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql` sur la base de corpus CPMM. Complément appliqué après constitution du corpus CPMM : @@ -1623,4 +1634,4 @@ La tranche CPMM reconnaît désormais tous les discriminants instruction-level l `0.7.48` est clôturable côté `raydium_cpmm`. Le decoder couvre les instructions/events CPMM listés par Carbon/fnzero/Raydium CP-Swap, avec matérialisation locale validée pour trades, liquidity, lifecycle, fees et admin/config. `swap_event` reste audit-only pour éviter les doublons avec `swap_base_input` / `swap_base_output`. Les side effects SPL Token / Token-2022 observés via Solscan (`burn`, `transfer`, `transferChecked`, `closeAccount`) restent hors decoder CPMM direct et alimenteront une réflexion transversale future. -La suite reprend en `0.7.49` par `raydium_clmm`, en utilisant les sources Git/IDL habituelles et la page Solscan Program IDL `https://solscan.io/account/CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK#programIdl` pour accélérer la découverte de signatures par discriminant. +La suite après `0.7.49 raydium_clmm` reprend en `0.7.50` par `raydium_launchpad`, puis `0.7.51 raydium_amm_v4` et `0.7.52 raydium_stable`, en gardant la même discipline : sources Git/IDL + Solscan pour accélérer la découverte, mais corpus local obligatoire avant toute promotion métier. diff --git a/docs/ARCHIVE_ORGANIZATION.md b/docs/ARCHIVE_ORGANIZATION.md new file mode 100644 index 0000000..43e4610 --- /dev/null +++ b/docs/ARCHIVE_ORGANIZATION.md @@ -0,0 +1,22 @@ + + +# Organisation de l’archive documentaire + +La racine du workspace garde les documents de pilotage principaux : + +```text +README.md +ROADMAP.md +CHANGELOG.md +``` + +Les documents spécialisés sont rangés par usage : + +```text +docs/ matrices et revues transversales +docs/reports/ rapports de couverture par tranche DEX/version +docs/prompts/ prompts de reprise de session +validation_sql/ scripts SQL de validation +``` + +Cette réorganisation ne modifie pas le code Rust/Tauri. Elle sert uniquement à séparer les fichiers de pilotage, les rapports, les prompts et les validations SQL. diff --git a/DB_EVENT_MODEL_REVIEW.md b/docs/DB_EVENT_MODEL_REVIEW.md similarity index 88% rename from DB_EVENT_MODEL_REVIEW.md rename to docs/DB_EVENT_MODEL_REVIEW.md index bd2ad94..86b6556 100644 --- a/DB_EVENT_MODEL_REVIEW.md +++ b/docs/DB_EVENT_MODEL_REVIEW.md @@ -306,3 +306,17 @@ Aucune nouvelle table n'est ajoutée pour CPMM : La tranche `raydium_cpmm` ajoute `k_sol_instruction_observations` comme table technique d’index local, non comme table métier. Elle sert à chercher les instructions observées par `decoder_code`, `instruction_name` et `discriminator_hex`, puis à relier ces observations au corpus backfillé/rejoué. La matérialisation métier reste limitée aux tables existantes : `k_sol_trade_events`, `k_sol_liquidity_events`, `k_sol_pool_lifecycle_events`, `k_sol_fee_events` et `k_sol_pool_admin_events`. Les opérations SPL Token / Token-2022 visibles dans Solscan (`burn`, `transfer`, `transferChecked`, `closeAccount`) ne sont pas promues en tables métier dans `0.7.48`; elles justifient seulement une future table transversale si plusieurs DEX le nécessitent. + +## Note 0.7.49 — Raydium CLMM sans nouvelle table transversale immédiate + +La reprise `raydium_clmm` confirme que la table `k_sol_dex_event_coverage_entries` suffit pour inventorier les instructions/events CLMM avant promotion métier. Les entrées IDL-only ajoutées en `0.7.49` restent des lignes de coverage et de recherche, pas des tables métier. + +Aucune nouvelle table transversale n'est ajoutée dans ce delta : + +- les swaps spécialisés restent dans `k_sol_trade_events` seulement lorsque les montants et le sens économique sont validés ; +- les liquidity/positions CLMM restent à distinguer : une position NFT/tick n'est pas forcément une liquidity row simple ; +- les fees et rewards peuvent utiliser `k_sol_fee_events` / `k_sol_reward_events` seulement après preuve de corpus et typage fin ; +- les side effects SPL Token / Token-2022 (`mint`, `burn`, `transfer`, `closeAccount`, wrap/unwrap SOL) restent indirects tant qu'une preuve multi-DEX ne justifie pas une table transversale. + +La table `k_sol_instruction_observations` reste technique : elle sert à trouver des signatures et discriminants observés localement, sans être une preuve métier. + diff --git a/DEX_DECODER_MATRIX.md b/docs/DEX_DECODER_MATRIX.md similarity index 83% rename from DEX_DECODER_MATRIX.md rename to docs/DEX_DECODER_MATRIX.md index 6600e98..69bb1e4 100644 --- a/DEX_DECODER_MATRIX.md +++ b/docs/DEX_DECODER_MATRIX.md @@ -28,31 +28,33 @@ Cette matrice complète `kb_lib/src/dex_support_matrix.rs`. Elle documente **ce | Ordre | DEX/version | État actuel | Fait | Reste à faire | |---:|---|---|---|---| -| 1 | `raydium_cpmm` | `supported` | Swaps matérialisés ; premiers non-swaps prouvés (`initialize`, `withdraw`, `collect_creator_fee`) ; trades/candles OK. | Comparer tous les discriminants Carbon/IDL ; compléter fees/admin/lifecycle ; rejouer sur base dédiée ; vérifier absence d’audits orphelins. | -| 2 | `raydium_clmm` | `supported` | Swaps `swap`/`swap_v2` ; events positions/liquidité prouvés ; matérialisation non-trade existante. | Repasser tous les events Carbon/IDL : open/close position, rewards, protocol fees, Token-2022 ; compléter audit → materialized si corpus. | -| 3 | `pump_swap` | `supported` | `buy`/`sell` décodés et matérialisés ; trade/candle OK. | Ajouter tous les events Carbon/Solana Streamer : cashback, fee, volume accumulator, admin/config ; conserver les non-trades hors candles. | -| 4 | `pump_fun` | `partial / launch_surface` | Création/token launch partiellement décodée ; intégrée au pipeline de listings. | Traiter tous les events Pump.fun disponibles : buy/sell/migrate/create/update ; séparer bonding/launch de DEX effectif ; valider migration vers PumpSwap. | -| 5 | `meteora_dbc` | `partial` | Swaps/instruction audits observés ; Demo3 donne du corpus. | Couverture complète DBC : launch/bonding curve, swap, migration, config/admin, fees ; matérialiser seulement ce qui est prouvé. | -| 6 | `meteora_dlmm` | `supported` | Couverture avancée validée en `0.7.45` : swaps, liquidity, positions, lifecycle, fees ; non-trade matérialisé. | Résoudre les audits résiduels non mappés ; comparer Carbon/IDL pour events rewards/admin restants ; revalidation base neuve. | -| 7 | `meteora_damm_v1` | `supported / partial events` | Couverture `0.7.46` : swap, create_pool, add/remove liquidity, claim_fee, create_lock_escrow, lock_liquidity. | Vérifier les surfaces upstream non observées ; améliorer rattachement pool/pair pour remove_liquidity non matérialisés ; revalidation stricte. | -| 8 | `meteora_damm_v2` | `partial` | `swap`, `instruction_audit`, registry/discriminants et corpus Demo3 existent. | Couvrir tous les events Carbon/source : create pool, liquidity, fees, dynamic config, admin ; déterminer actionability des swaps ; matérialiser si montants fiables. | -| 9 | `phoenix_v1` | `audit-only` | Decoder local audit-only ; `log_audit`, order place/cancel, withdraw ; parsing strict `0x0f`; events `Reduce`, `Place`, `TimeInForce` observés ; `trade_count=0`. | Terminer tous les events Git : `Fill`, `FillSummary`, `Fee`, `Evict`, `ExpiredOrder`, etc. ; ajouter counts/flags audit ; seulement ensuite étudier trade materialization. | -| 10 | `openbook_v2` | `audit-only` | Decoder local audit-only ; instructions order/cancel/consume/settle ; `Program data` mappé : `FillLog`, `OpenOrdersPositionLog`, `TotalOrderFillEvent`, `SettleFundsLog`; `trade_count=0`. | Vérifier layouts fill/out et sens maker/taker/base/quote ; ajouter table audit éventuelle ; ne matérialiser trades qu’après validation du sens économique. | -| 11 | `orca_whirlpools` | `partial` | Premier decoder historique présent ; swaps/create_pool partiels. | Comparer Carbon/IDL complet ; couvrir liquidity, positions, fees/rewards, tick arrays ; valider swaps exploitables et non-trades. | -| 12 | `raydium_launchlab` | `planned launch` | Entrée canonique locale LaunchLab/Launchpad ; program id connu localement. | Décoder launch/migration ; ne pas confondre avec CPMM/CLMM/AMM v4 ; rattacher aux pools tradables. | -| 13 | `meteora_vault` | `to_verify` | Présent comme indice upstream / compte associé. | Corpus direct obligatoire ; decoder séparé si events vault réels ; aucune promotion via DAMM indirect. | -| 14 | `fluxbeam` | `partial/to_verify` | Decoder initial existant ; Demo3 peut produire des candidats. | Vérifier source/IDL ; compléter swap, pool, liquidity, fees/admin ; matérialisation uniquement après corpus. | -| 15 | `dexlab` | `partial/to_verify` | Decoder initial historique ; ancienne entrée beta supprimée. | Reconfirmer program id/source ; décoder events disponibles ; distinguer DexLab natif et liens OpenBook/market. | -| 16 | `lifinity_v2` | `to_verify` | Program id listé par sources externes/Vybe ; pas de corpus concluant. | Trouver IDL/source ; Demo3 par program/market ; audit-only d’abord. | -| 17 | `stabble_stable_swap` / `stabble_weighted_swap` | `to_verify` | Program ids/indices via sources externes ; candidats Demo3 observables. | Source/IDL + corpus + decoder audit-only ; déterminer surface AMM et montants exploitables. | -| 18 | `bonkswap` | `to_verify` | Program id/Carbon/Vybe selon registre ; swaps candidats possibles. | Vérifier program id, source et corpus ; décoder tous events ; pas de trade sans montants. | -| 19 | `boop` / `boop_fun` | `to_verify / launch` | Entrée de découverte. | Séparer launch surface et swap effectif ; corpus + source obligatoire. | -| 20 | `moonshot` / `moonit` | `to_verify / launch` | Moonshot buy/sell observés via upstream candidates ; Moonit launch attribution historique. | Source/IDL + migration + rattachement pools ; éviter heuristiques seules. | -| 21 | `heaven` | `to_verify` | Program id/candidat ajouté en matrice. | Vérifier s’il est launch, AMM ou les deux ; corpus dédié. | -| 22 | `printr` | `to_verify` | Preset Demo3 ajouté ; candidats observables. | Source/IDL, discriminants, corpus, decoder audit-only. | -| 23 | `metadao_*` | `to_verify` | Presets spécifiques : launchpad, bid wall, futarchy, AMM. | Traiter par programme séparé ; ne pas utiliser mint ids comme program ids ; corpus obligatoire. | -| 24 | `raydium_stable_swap` | `planned/historical` | Entrée conservée. | Reprendre uniquement si corpus réel ; stable AMM séparé de CPMM/CLMM. | -| 25 | `jupiter_*`, `dflow_aggregator_v4`, `okx_dex` | `aggregator_router` | Registry/discovery pour contexte transactionnel. | Ne pas matérialiser en DEX direct ; utiliser pour routeSource/routing/context. | +| 1 | `raydium_cpmm` | `supported / 0.7.48 closed` | Couverture CPMM clôturée : swaps, lifecycle, fees, admin/config, deposit/withdraw, `lp_change_event`, `swap_event` audit-only. | Réouvrir seulement en cas de nouveau corpus ou divergence upstream. | +| 2 | `raydium_clmm` | `supported / 0.7.49 closed` | Couverture CLMM clôturée : 45 entrées listées, 33 instructions observées/décodées, 25 matérialisées, orderbook events, fallback upstream nettoyé. | 11 Anchor Program-data events restent préparés mais `upstream_git_unverified` faute de corpus local. | +| 3 | `raydium_launchpad` | `planned / 0.7.50` | Prochaine tranche Raydium. | Identifier program ids/IDL, launch/pool creation, migration, fees/admin/config, corpus Solscan/Demo3 puis decoder spécialisé. | +| 4 | `raydium_amm_v4` | `supported / 0.7.51 planned` | Swaps AMM v4 legacy matérialisés. | Reprendre AMM v4 au niveau CPMM/CLMM : pool lifecycle, liquidity, fees/admin, side effects, fallback cleanup. | +| 5 | `raydium_stable_swap` | `planned / 0.7.52` | Entrée conservée. | Reprendre Stable séparément : swaps stables, pool lifecycle, liquidity, fees/admin, montants/prix exploitables. | +| 6 | `pump_swap` | `supported / 0.7.53 planned` | `buy`/`sell` décodés et matérialisés ; trade/candle OK. | Ajouter tous les events Carbon/Solana Streamer : cashback, fee, volume accumulator, admin/config ; conserver les non-trades hors candles. | +| 7 | `pump_fun` | `partial / launch_surface` | Création/token launch partiellement décodée ; intégrée au pipeline de listings. | Traiter tous les events Pump.fun disponibles : buy/sell/migrate/create/update ; séparer bonding/launch de DEX effectif ; valider migration vers PumpSwap. | +| 8 | `meteora_dbc` | `partial` | Swaps/instruction audits observés ; Demo3 donne du corpus. | Couverture complète DBC : launch/bonding curve, swap, migration, config/admin, fees ; matérialiser seulement ce qui est prouvé. | +| 9 | `meteora_dlmm` | `supported` | Couverture avancée validée en `0.7.45` : swaps, liquidity, positions, lifecycle, fees ; non-trade matérialisé. | Résoudre les audits résiduels non mappés ; comparer Carbon/IDL pour events rewards/admin restants ; revalidation base neuve. | +| 10 | `meteora_damm_v1` | `supported / partial events` | Couverture `0.7.46` : swap, create_pool, add/remove liquidity, claim_fee, create_lock_escrow, lock_liquidity. | Vérifier les surfaces upstream non observées ; améliorer rattachement pool/pair pour remove_liquidity non matérialisés ; revalidation stricte. | +| 11 | `meteora_damm_v2` | `partial` | `swap`, `instruction_audit`, registry/discriminants et corpus Demo3 existent. | Couvrir tous les events Carbon/source : create pool, liquidity, fees, dynamic config, admin ; déterminer actionability des swaps ; matérialiser si montants fiables. | +| 12 | `phoenix_v1` | `audit-only` | Decoder local audit-only ; `log_audit`, order place/cancel, withdraw ; parsing strict `0x0f`; events `Reduce`, `Place`, `TimeInForce` observés ; `trade_count=0`. | Terminer tous les events Git : `Fill`, `FillSummary`, `Fee`, `Evict`, `ExpiredOrder`, etc. ; ajouter counts/flags audit ; seulement ensuite étudier trade materialization. | +| 13 | `openbook_v2` | `audit-only` | Decoder local audit-only ; instructions order/cancel/consume/settle ; `Program data` mappé : `FillLog`, `OpenOrdersPositionLog`, `TotalOrderFillEvent`, `SettleFundsLog`; `trade_count=0`. | Vérifier layouts fill/out et sens maker/taker/base/quote ; ajouter table audit éventuelle ; ne matérialiser trades qu’après validation du sens économique. | +| 14 | `orca_whirlpools` | `partial` | Premier decoder historique présent ; swaps/create_pool partiels. | Comparer Carbon/IDL complet ; couvrir liquidity, positions, fees/rewards, tick arrays ; valider swaps exploitables et non-trades. | +| 15 | `legacy_launch_candidates` | `planned launch` | Anciennes entrées launch à réévaluer après `raydium_launchpad`. | Ne pas confondre Launchpad, LaunchLab, CPMM/CLMM/AMM v4 ; rattacher aux pools tradables seulement après corpus. | +| 16 | `meteora_vault` | `to_verify` | Présent comme indice upstream / compte associé. | Corpus direct obligatoire ; decoder séparé si events vault réels ; aucune promotion via DAMM indirect. | +| 17 | `fluxbeam` | `partial/to_verify` | Decoder initial existant ; Demo3 peut produire des candidats. | Vérifier source/IDL ; compléter swap, pool, liquidity, fees/admin ; matérialisation uniquement après corpus. | +| 18 | `dexlab` | `partial/to_verify` | Decoder initial historique ; ancienne entrée beta supprimée. | Reconfirmer program id/source ; décoder events disponibles ; distinguer DexLab natif et liens OpenBook/market. | +| 19 | `lifinity_v2` | `to_verify` | Program id listé par sources externes/Vybe ; pas de corpus concluant. | Trouver IDL/source ; Demo3 par program/market ; audit-only d’abord. | +| 20 | `stabble_stable_swap` / `stabble_weighted_swap` | `to_verify` | Program ids/indices via sources externes ; candidats Demo3 observables. | Source/IDL + corpus + decoder audit-only ; déterminer surface AMM et montants exploitables. | +| 21 | `bonkswap` | `to_verify` | Program id/Carbon/Vybe selon registre ; swaps candidats possibles. | Vérifier program id, source et corpus ; décoder tous events ; pas de trade sans montants. | +| 22 | `boop` / `boop_fun` | `to_verify / launch` | Entrée de découverte. | Séparer launch surface et swap effectif ; corpus + source obligatoire. | +| 23 | `moonshot` / `moonit` | `to_verify / launch` | Moonshot buy/sell observés via upstream candidates ; Moonit launch attribution historique. | Source/IDL + migration + rattachement pools ; éviter heuristiques seules. | +| 24 | `heaven` | `to_verify` | Program id/candidat ajouté en matrice. | Vérifier s’il est launch, AMM ou les deux ; corpus dédié. | +| 25 | `printr` | `to_verify` | Preset Demo3 ajouté ; candidats observables. | Source/IDL, discriminants, corpus, decoder audit-only. | +| 26 | `metadao_*` | `to_verify` | Presets spécifiques : launchpad, bid wall, futarchy, AMM. | Traiter par programme séparé ; ne pas utiliser mint ids comme program ids ; corpus obligatoire. | +| 27 | `jupiter_*`, `dflow_aggregator_v4`, `okx_dex` | `aggregator_router` | Registry/discovery pour contexte transactionnel. | Ne pas matérialiser en DEX direct ; utiliser pour routeSource/routing/context. | ## Checklist obligatoire par DEX/version @@ -223,3 +225,14 @@ Entrées CPMM couvertes localement depuis Carbon/fnzero/IDL : `raydium_cpmm` est considéré `supported` et clôturable pour la tranche `0.7.48`. Les entrées matérialisées couvrent swaps (`swap_base_input`, `swap_base_output`), liquidity (`deposit`, `withdraw`, `lp_change_event`), lifecycle (`initialize`, `initialize_with_permission`), fees (`collect_creator_fee`, `collect_fund_fee`, `collect_protocol_fee`) et admin/config (`create_amm_config`, `create_permission_pda`, `update_amm_config`). `swap_event` reste audit-only pour éviter tout doublon de trade/candle. `close_permission_pda` et `update_pool_status` restent connus upstream mais non observés localement. + +## Note `0.7.49` — Raydium CLMM initial coverage sync + +`raydium_clmm` reste `supported`, mais la tranche `0.7.49` rouvre sa couverture événementielle complète au lieu de se limiter aux swaps. + +État local repris : `swap`, `swap_v2`, `increase_liquidity_v2`, `decrease_liquidity_v2`, `open_position_with_token22_nft` et `close_position` disposent déjà d'un chemin local spécialisé ou mappé. Les autres entrées CLMM restent à confirmer par corpus avant toute promotion. + +Le registre est complété avec les entrées issues de l'IDL officiel Raydium non présentes dans le snapshot Carbon courant : `close_limit_order`, `close_protocol_position`, `create_customizable_pool`, `create_dynamic_fee_config`, `create_support_mint_associated` et `settle_limit_order`. + +Règle de clôture : les positions CLMM, fees/rewards et surfaces limit-order ne doivent produire aucune ligne trade/candle tant que le sens économique, les montants, les comptes et les mints ne sont pas prouvés par replay local. + diff --git a/DEX_EVENT_COVERAGE_MATRIX.md b/docs/DEX_EVENT_COVERAGE_MATRIX.md similarity index 61% rename from DEX_EVENT_COVERAGE_MATRIX.md rename to docs/DEX_EVENT_COVERAGE_MATRIX.md index dc80c2c..a714d07 100644 --- a/DEX_EVENT_COVERAGE_MATRIX.md +++ b/docs/DEX_EVENT_COVERAGE_MATRIX.md @@ -1,6 +1,6 @@ -# DEX Event Coverage Matrix — `khadhroony-bobobot` `0.7.48` +# DEX Event Coverage Matrix — `khadhroony-bobobot` `0.7.49` -Cette matrice complète `DEX_DECODER_MATRIX.md` avec une lecture par familles d'événements. Elle ne remplace pas la preuve locale : une entrée Git/IDL reste un indice tant qu'elle n'est pas observée dans le corpus local puis validée par replay et SQL. +Cette matrice complète `docs/DEX_DECODER_MATRIX.md` avec une lecture par familles d'événements. Elle ne remplace pas la preuve locale : une entrée Git/IDL reste un indice tant qu'elle n'est pas observée dans le corpus local puis validée par replay et SQL. ## Règles de statut @@ -69,3 +69,34 @@ La couverture `raydium_cpmm` est alignée avec les instructions exposées par le Validation finale locale : `deposit` = `11/11` liquidity, `withdraw` = `14/14` liquidity, `lp_change_event` = `25/25` liquidity, fees = `26/26` fee, admin/config observés = `23/23` admin, lifecycle = `9/9`, `swap_event` = audit-only avec `0` trade, et trades matérialisés uniquement depuis `swap_base_input` / `swap_base_output`. `close_permission_pda` et `update_pool_status` restent `upstream_git_mapped_unverified` faute de corpus local. Les filtres Solscan `instruction=9c5420764587467b` et `instruction=82576c062ee0757b` n'ont pas fourni de signatures exploitables dans l'horizon testé. + +## `0.7.49` — `raydium_clmm` final coverage + +Sources inventoriées : Carbon `raydium-clmm-decoder`, fnzero `sol-parser-sdk`, Pinax `substreams-solana-idls/src/raydium/clmm`, Raydium/Solscan Program IDL. + +État final validé : `45` entrées listées, `33` instructions locales observées/décodées, `25` entrées matérialisées, `1186` trades, `raydium_clmm.instruction_audit = 0`, fallback `upstream_git.instruction_match` localement couvert = `0`. + +| Famille | Entrées Raydium CLMM | Statut `0.7.49` | Cible DB | Justification / règle | +|---|---|---|---|---| +| `swap` | `swap`, `swap_v2` | `materialized` | `k_sol_trade_events` | Seuls ces swaps produisent trades/candles lorsque les montants sont exploitables. | +| `swap audit` | `swap_event`, `swap_router_base_in` | `observed/audit` ou `upstream_git_unverified` selon corpus | `k_sol_dex_decoded_events_only` | Pas de double trade/candle. | +| `pool_create` | `create_pool`, `create_customizable_pool` | `decoded`; `create_pool` matérialisé quand prouvé | `k_sol_pool_lifecycle_events` | Création de pool = lifecycle, pas admin. | +| `add_liquidity` | `increase_liquidity`, `increase_liquidity_v2`, open position variants | `decoded`; variantes prouvées matérialisées | `k_sol_liquidity_events` | CLMM implique position/tick/NFT ; matérialisation seulement sur corpus OK. | +| `remove_liquidity` | `decrease_liquidity`, `decrease_liquidity_v2`, close position variants | `decoded`; variantes prouvées matérialisées | `k_sol_liquidity_events` | Aucune promotion trade/candle. | +| `fee` | `collect_fund_fee`, `collect_protocol_fee` | `decoded`; protocol fee matérialisé si tx OK | `k_sol_fee_events` | Mapping fee sans trade/candle. | +| `reward` | `initialize_reward`, `collect_remaining_rewards`, `set_reward_params`, `transfer_reward_owner`, `update_reward_infos` | `decoded`; variantes prouvées matérialisées | `k_sol_reward_events` / `k_sol_pool_admin_events` | Reward/config selon instruction. | +| `admin/config` | `create_amm_config`, `create_dynamic_fee_config`, `create_operation_account`, `update_amm_config`, `update_operation_account`, `update_pool_status` | `decoded`; variantes prouvées matérialisées | `k_sol_pool_admin_events` ou decoded-only | Config/admin sans trade/candle. | +| `account_create` | `create_support_mint_associated` | `observed/decoded` | `k_sol_token_account_events` prévu / decoded-only selon corpus | Side effect technique ; pas de trade/candle. | +| `order_place` | `open_limit_order`, `increase_limit_order` | `decoded/materialized` | `k_sol_orderbook_events` | Orderbook CLMM, jamais trade/candle. | +| `order_cancel` | `decrease_limit_order`, `close_limit_order` | `decoded/materialized` | `k_sol_orderbook_events` | Orderbook CLMM, jamais trade/candle. | +| `settle_funds` | `settle_limit_order` | `decoded/materialized` | `k_sol_orderbook_events` | Settlement orderbook, pas trade direct. | +| `mint/burn/transfer/account_close/wrap/unwrap` | Side effects SPL Token / Token-2022 | `indirect` | decoded-only | Pas de promotion en `raydium_clmm.*` sans instruction directe. | +| `launch/migration/lock/unlock/stake/unstake/vault` | `-` | `not_applicable` | `-` | Surfaces hors CLMM direct. | +| `unknown/unmapped audit` | `raydium_clmm.instruction_audit`, fallback upstream | `closed` | `k_sol_dex_decoded_events_only` | Résidu validé à zéro après replay. | + +Les 11 Anchor / `Program data` events restent `upstream_git_unverified` et préparés audit-only faute d’observation locale : `collect_personal_fee_event`, `collect_protocol_fee_event`, `config_change_event`, `create_personal_position_event`, `decrease_liquidity_event`, `increase_liquidity_event`, `liquidity_calculate_event`, `liquidity_change_event`, `pool_created_event`, `swap_event`, `update_reward_infos_event`. + +## `0.7.50` — `raydium_launchpad` planned + +Prochaine tranche : identifier program ids/IDL, lister instructions/events/discriminants, constituer corpus Demo3/Solscan/Demo2, puis appliquer les mêmes règles de coverage et matérialisation que CPMM/CLMM. + diff --git a/NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md b/docs/prompts/NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md similarity index 99% rename from NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md rename to docs/prompts/NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md index 3d8fb6c..6181af5 100644 --- a/NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md +++ b/docs/prompts/NEXT_SESSION_PROMPT_0.7.47_1FE5_CONTINUATION_V2.md @@ -16,7 +16,7 @@ Joindre aussi les docs mises à jour : README.md ROADMAP.md CHANGELOG.md -DEX_DECODER_MATRIX.md +docs/DEX_DECODER_MATRIX.md ``` ## Décision de planification diff --git a/NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md b/docs/prompts/NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md similarity index 97% rename from NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md rename to docs/prompts/NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md index 8b552e7..e85cd45 100644 --- a/NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md +++ b/docs/prompts/NEXT_SESSION_PROMPT_0.7.47_EVENT_COVERAGE_V3.md @@ -16,9 +16,9 @@ Et les docs : README.md ROADMAP.md CHANGELOG.md -DEX_DECODER_MATRIX.md -DEX_EVENT_COVERAGE_MATRIX.md -DB_EVENT_MODEL_REVIEW.md +docs/DEX_DECODER_MATRIX.md +docs/DEX_EVENT_COVERAGE_MATRIX.md +docs/DB_EVENT_MODEL_REVIEW.md ``` ## Décision de reprise diff --git a/NEXT_SESSION_PROMPT_0.7.47_UPSTREAM_REGISTRY.md b/docs/prompts/NEXT_SESSION_PROMPT_0.7.47_UPSTREAM_REGISTRY.md similarity index 100% rename from NEXT_SESSION_PROMPT_0.7.47_UPSTREAM_REGISTRY.md rename to docs/prompts/NEXT_SESSION_PROMPT_0.7.47_UPSTREAM_REGISTRY.md diff --git a/NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md b/docs/prompts/NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md similarity index 98% rename from NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md rename to docs/prompts/NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md index 6a7bd1e..a7d8111 100644 --- a/NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md +++ b/docs/prompts/NEXT_SESSION_PROMPT_0.7.49_RAYDIUM_CLMM.md @@ -16,11 +16,11 @@ Docs à fournir aussi : README.md ROADMAP.md CHANGELOG.md -DEX_DECODER_MATRIX.md -DEX_EVENT_COVERAGE_MATRIX.md -DB_EVENT_MODEL_REVIEW.md -RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md -SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql +docs/DEX_DECODER_MATRIX.md +docs/DEX_EVENT_COVERAGE_MATRIX.md +docs/DB_EVENT_MODEL_REVIEW.md +docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md +validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql ``` ## État validé avant reprise diff --git a/docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.48-raydium-cpmm.md b/docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.48-raydium-cpmm.md new file mode 100644 index 0000000..d1e81b4 --- /dev/null +++ b/docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.48-raydium-cpmm.md @@ -0,0 +1,245 @@ +# Prompt de reprise — khadhroony-bobobot `0.7.48` / Raydium CPMM event coverage + +Reprise du projet `khadhroony-bobobot` après clôture de `0.7.48-pre`. + +## Archive de départ + +Utiliser la dernière archive complète du workspace intégrant les deltas validés jusqu'à : + +```text +0.7.48-pre-event-coverage-report +``` + +Docs à fournir aussi : + +```text +README.md +ROADMAP.md +CHANGELOG.md +docs/DEX_DECODER_MATRIX.md +docs/DEX_EVENT_COVERAGE_MATRIX.md +docs/DB_EVENT_MODEL_REVIEW.md +``` + +## État validé avant reprise + +`0.7.48-pre` a ajouté et clôturé le checkpoint DB/reporting de couverture événementielle : + +```text +k_sol_dex_event_coverage_entries +DexEventCoverageService +sync upstream registry -> coverage table +refresh local counts depuis k_sol_dex_decoded_events + tables métier existantes +summaries coverage dans LocalPipelineDiagnosticSummaryDto +summaries/counters coverage dans LocalPipelineValidationReportDto +profil validation 0.7.48-pre_event_coverage_db_checkpoint +profil exposé dans Demo Pipeline 2 +``` + +Invariants maintenus : + +```text +aucun decoder DEX modifié +aucun trade/candle créé par la couverture +aucun program_id promu sans corpus local +upstream Git/IDL = indice, pas preuve métier +failed transaction = audit-only +non-trade event = jamais trade/candle +``` + +## Décision de reprise + +Commencer maintenant par Raydium avant Meteora. + +Ordre courant : + +```text +0.7.48 raydium_cpmm +0.7.49 raydium_clmm +0.7.50 pump_swap +0.7.51 pump_fun +0.7.52 meteora_dbc +0.7.53 meteora_dlmm upstream parity +0.7.54 meteora_damm_v1 upstream parity +0.7.55 meteora_damm_v2 +0.7.56 phoenix_v1 audit-only completion +0.7.57 openbook_v2 audit-only completion +0.7.58 orca_whirlpools +0.7.59+ launch surfaces, candidats/historiques, validation consolidée +``` + +## Sources Git/IDL à utiliser systématiquement + +- https://github.com/sevenlabs-hq/carbon/tree/main/decoders +- https://github.com/0xfnzero/solana-streamer +- https://github.com/0xfnzero/sol-parser-sdk/tree/main/idl +- https://github.com/pinax-network/substreams-solana-idls/tree/main/src +- https://github.com/hodlwarden/solana-tx-parser/tree/main/src +- https://github.com/openbook-dex/openbook-v2 +- https://github.com/all-in-one-blockchain/phoenix-onchain-mm +- https://docs.vybenetwork.com/docs/available-dexs-amms + +Pour `0.7.48`, commencer par Carbon + fnzero pour Raydium CPMM, puis comparer aux IDL complémentaires si disponibles. + +## Objectif `0.7.48` — `raydium_cpmm` + +Objectif : reprendre `raydium_cpmm` comme première tranche DEX/version après le checkpoint coverage. + +À faire : + +1. lister tous les discriminants/instructions/events `raydium_cpmm` depuis Carbon/fnzero/IDL ; +2. synchroniser/remplir `k_sol_dex_event_coverage_entries` pour `raydium-cpmm` ; +3. comparer listed/decoded/observed/materialized/trade_count via le rapport coverage ; +4. compléter le decoder spécialisé `raydium_cpmm` seulement pour les events CPMM confirmables ; +5. remplacer/nettoyer le fallback `upstream_git.instruction_match` quand un decoder local spécialisé couvre l'entrée ; +6. garder les events connus mais non observés en `upstream_git_mapped_unverified` ; +7. garder les events observés mais non matérialisés en audit-only/decoded ; +8. ne matérialiser que les non-trades déjà prouvés par corpus et compatibles avec les tables existantes ; +9. ne pas ajouter encore `k_sol_token_transfer_events` ou `k_sol_orderbook_events`, sauf besoin bloquant démontré ; +10. ne pas modifier les règles trade/candle sauf bug de faux positif prouvé. + +## Events/familles à couvrir explicitement + +Ne pas se limiter aux swaps. + +Inclure dans l'audit coverage : + +```text +swap +pool_create +add_liquidity +remove_liquidity +position_open +position_close +fee +reward +admin/config +mint +burn +transfer +account_create +account_close +wrap_sol +unwrap_sol +order_place +order_cancel +order_fill +consume_events +settle_funds +vault_deposit +vault_withdraw +lock +unlock +launch +migration +stake +unstake +unknown/unmapped audit +``` + +Pour `raydium_cpmm`, certaines familles seront probablement `-` ou `non applicable`, mais elles doivent être explicitement justifiées dans la coverage matrix ou la table. + +## Règles fixes + +- Un event non-trade ne produit jamais `trade_event`, metric ou candle. +- Une transaction failed reste audit, jamais trade/candle. +- Un discriminator upstream n'est pas une preuve métier. +- Un program id upstream n'est pas vérifié sans corpus local. +- Chaque decoder spécialisé doit remplacer le fallback `upstream_git.instruction_match` pour éviter les doublons. +- Tout event connu mais non observé reste `upstream_git_mapped_unverified`. +- Tout event observé mais non matérialisé reste audit-only ou decoded, pas materialized. +- Ne pas promouvoir de nouvelle table DB métier sans preuve que plusieurs DEX en auront besoin. + +## Requêtes SQL utiles + +Après sync/refresh coverage : + +```sql +SELECT * +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium-cpmm' +ORDER BY entry_kind, entry_name, discriminator_hex; +``` + +```sql +SELECT + decoder_code, + listed_entry_count, + decoded_entry_count, + observed_entry_count, + materialized_entry_count, + total_observed_count, + total_materialized_count, + trade_count, + audit_only_entry_count, + upstream_git_mapped_unverified_entry_count, + upstream_git_local_corpus_observed_entry_count, + upstream_git_local_corpus_materialized_entry_count +FROM ( + SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count, + SUM(CASE WHEN expected_db_target = 'k_sol_dex_decoded_events_only' THEN 1 ELSE 0 END) AS audit_only_entry_count, + SUM(CASE WHEN proof_status = 'upstream_git_mapped_unverified' THEN 1 ELSE 0 END) AS upstream_git_mapped_unverified_entry_count, + SUM(CASE WHEN proof_status = 'upstream_git_local_corpus_observed' THEN 1 ELSE 0 END) AS upstream_git_local_corpus_observed_entry_count, + SUM(CASE WHEN proof_status = 'upstream_git_local_corpus_materialized' THEN 1 ELSE 0 END) AS upstream_git_local_corpus_materialized_entry_count + FROM k_sol_dex_event_coverage_entries + GROUP BY decoder_code +) +WHERE decoder_code = 'raydium-cpmm'; +``` + +Audit-only safety : + +```sql +SELECT + de.protocol_name, + de.event_kind, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_cpmm' + AND ( + de.event_kind LIKE '%audit%' + OR json_extract(de.payload_json, '$.eventActionability') IN ('non_trade_useful', 'informational', 'non_actionable_trade') + ) +GROUP BY de.protocol_name, de.event_kind +ORDER BY trade_count DESC, de.event_kind; +``` + +## Contraintes de code + +Conserver les règles du workspace : + +```text +Rust 2024 +pas de mod.rs +fichiers Rust avec // file: ... +pas de anyhow +pas de thiserror +pas de ? / unwrap / expect dans kb_lib applicatif +match / if let Err / let Err = ... else +rustdoc sur API publique +re-exports db.rs puis lib.rs si DB modifiée +``` + +## Livrables attendus pour `0.7.48` + +1. delta archive avec uniquement les fichiers ajoutés/modifiés ; +2. mise à jour `README.md`, `ROADMAP.md`, `CHANGELOG.md` si la tranche avance ; +3. rapport de couverture `raydium_cpmm` ; +4. SQL de validation ; +5. tests verts : + +```bash +cargo fmt +cargo test -p kb_lib +cargo clippy -p kb_lib --all-targets -- -D warnings +``` diff --git a/docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md b/docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md new file mode 100644 index 0000000..7dabd13 --- /dev/null +++ b/docs/prompts/PROMPT_REPRISE_khadhroony-bobobot_0.7.50-raydium-launchpad.md @@ -0,0 +1,338 @@ + + +# khadhroony-bobobot `0.7.50` / Raydium Launchpad event coverage + +Reprise du projet `khadhroony-bobobot` après clôture fonctionnelle de `0.7.49 raydium_clmm`. + +## Archive de départ + +Utiliser la dernière archive complète du workspace intégrant les deltas validés jusqu'à : + +```text +0.7.49-raydium-clmm-final +``` + +Inclure les docs et SQL de validation produits en fin de `0.7.49`, notamment : + +```text +README.md +ROADMAP.md +CHANGELOG.md +docs/DEX_DECODER_MATRIX.md +docs/DEX_EVENT_COVERAGE_MATRIX.md +docs/DB_EVENT_MODEL_REVIEW.md +docs/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md +docs/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md +validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql +validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql +``` + +## État validé avant reprise + +`0.7.48` a clôturé `raydium_cpmm`. + +`0.7.49` a clôturé `raydium_clmm` avec les invariants suivants : + +```text +Raydium CLMM decoder_code local = raydium_clmm +coverage synchronisée avec Carbon / Raydium IDL / Pinax / fnzero +45 entrées coverage CLMM listées +33 entrées CLMM décodées localement +33 entrées observées localement +25 entrées matérialisées localement +residual raydium_clmm.instruction_audit = 0 +fallback upstream_git.instruction_match localement couvert = 0 après cleanup FK-safe +non-trade CLMM = jamais trade/candle +failed transaction = jamais matérialisée dans les tables métier +Program data CLMM préparé mais non promu sans corpus local observé +side effects SPL Token / Token-2022 restent transversaux, pas raydium_clmm.* directs +``` + +Dernière validation locale observée : + +```text +cargo test -p kb_lib: ok +local pipeline replay: 2197 replayed, 0 decode skipped, 2197 ledger upserts, 1461 unsafe ledger rows, 1217 trades, 111 liquidity, 25 lifecycle, 4868 candle upserts, instructionObservations='19798' +``` + +## Décision de reprise + +Commencer `0.7.50` par `raydium_launchpad`, avant Pump/Meteora. + +Ordre strict Raydium proposé avant Pump : + +```text +0.7.50 raydium_launchpad / Raydium LaunchLab-Launchpad surface +0.7.51 raydium_amm_v4 +0.7.52 raydium_stable +0.7.53 raydium_pool_v4 audit / program-id decision, seulement si program id distinct et corpus exploitable +0.7.54 pump_swap +0.7.55 pump_fun +0.7.56 meteora_dbc +0.7.57 meteora_dlmm upstream parity +0.7.58 meteora_damm_v1 upstream parity +0.7.59 meteora_damm_v2 +0.7.60 phoenix_v1 audit-only completion +0.7.61 openbook_v2 audit-only completion +0.7.62 orca_whirlpools +0.7.63+ launch surfaces, candidats/historiques, validation consolidée +``` + +`raydium_pool_v4` ne doit pas être promu automatiquement comme DEX/surface métier tant que son program id et son rôle exact n'ont pas été confirmés. Le fichier `raydium_pool_v4.json` de `sol-parser-sdk` doit être audité comme source IDL annexe, pas comme preuve métier. + +## Sources Git/IDL à utiliser systématiquement + +- https://github.com/sevenlabs-hq/carbon/tree/main/decoders +- https://github.com/0xfnzero/solana-streamer +- https://github.com/0xfnzero/sol-parser-sdk/tree/main/idl +- copie locale fournie de `0xfnzero/sol-parser-sdk`, si présente dans l'archive de reprise +- https://github.com/pinax-network/substreams-solana-idls/tree/main/src +- https://github.com/hodlwarden/solana-tx-parser/tree/main/src +- https://github.com/openbook-dex/openbook-v2 +- https://github.com/all-in-one-blockchain/phoenix-onchain-mm +- https://docs.vybenetwork.com/docs/available-dexs-amms + +Pour `0.7.50 raydium_launchpad`, utiliser aussi explicitement : + +```text +https://solscan.io/account/LanMV9sAd7wArD4vJFi2qDdfnVhFxYSUg6eADduJ3uj#programIdl +``` + +et les filtres Solscan de type : + +```text +https://solscan.io/account/LanMV9sAd7wArD4vJFi2qDdfnVhFxYSUg6eADduJ3uj?instruction=&hide_spam=true&hide_failed=true&show_related=false&sort=desc +``` + +Solscan sert à trouver vite des signatures à backfiller. Solscan ne doit jamais être considéré comme preuve métier finale sans corpus local et validation SQL. + +## Base de données de travail + +Initialiser une nouvelle base SQLite dédiée à `0.7.50`. + +Procédure attendue : + +1. créer une nouvelle DB via `config.json` ou équivalent ; +2. démarrer `kb_demo_app` pour initialiser le schéma ; +3. backfiller des signatures ciblées par discriminant / instruction ; +4. backfiller des pools/comptes pertinents si la surface en expose ; +5. exécuter un replay local avec `forceDexDecode=yes` ; +6. relancer les SQL de coverage ; +7. ne promouvoir une entrée que si le corpus local confirme son sens métier. + +## Objectif `0.7.50` — `raydium_launchpad` + +Objectif : couvrir Raydium Launchpad/LaunchLab comme nouvelle surface Raydium après CPMM et CLMM. + +À faire : + +1. lire le code local existant lié à Raydium Launchpad, s'il existe ; +2. lister toutes les instructions/events depuis Carbon/fnzero/IDL/Pinax/Solscan Program IDL ; +3. synchroniser/remplir `k_sol_dex_event_coverage_entries` pour `raydium_launchpad` ; +4. vérifier `decoder_code` local en snake_case : `raydium_launchpad` ; +5. utiliser `k_sol_instruction_observations` pour inspecter les discriminants observés localement ; +6. utiliser Demo3 / Solscan `instruction=` pour trouver des signatures ciblées ; +7. backfiller les signatures via Demo2, idéalement par batch textarea ; +8. rejouer localement avec `forceDexDecode=yes` et `deferInstructionObservations=yes` ; +9. comparer listed/decoded/observed/materialized/trade_count via SQL coverage ; +10. compléter le decoder spécialisé seulement pour les entrées confirmées ; +11. supprimer/nettoyer `upstream_git.instruction_match` lorsque l'entrée est couverte localement ; +12. garder les entrées connues mais non observées en `upstream_git_unverified` ou `upstream_git_mapped_unverified` ; +13. garder les entrées observées mais non matérialisées en audit-only/decoded ; +14. ne matérialiser que les non-trades prouvés par corpus local et compatibles avec les tables existantes ; +15. ne modifier les règles trade/candle que si un faux positif est prouvé. + +## Familles à auditer explicitement + +Ne pas se limiter aux swaps. + +Inclure dans la coverage : + +```text +swap +pool_create +add_liquidity +remove_liquidity +position_open +position_close +fee +reward +admin/config +mint +burn +transfer +account_create +account_close +wrap_sol +unwrap_sol +order_place +order_cancel +order_fill +consume_events +settle_funds +vault_deposit +vault_withdraw +lock +unlock +launch +migration +stake +unstake +unknown/unmapped audit +``` + +Certaines familles peuvent être non applicables ou uniquement visibles comme side effects SPL Token / Token-2022. Elles doivent être explicitement justifiées dans `DEX_EVENT_COVERAGE_MATRIX.md`. + +## Points d'attention hérités de CPMM/CLMM + +- `decoder_code` local doit rester en `snake_case`. +- Les slugs upstream peuvent garder les tirets. +- `upstream Git/IDL/Solscan = indice, pas preuve métier`. +- `program_id` upstream non promu sans corpus local. +- Chaque decoder spécialisé doit remplacer le fallback `upstream_git.instruction_match` pour les entrées localement couvertes. +- Les side effects SPL Token (`mintTo`, `burn`, `transfer`, `transferChecked`, `closeAccount`) ne deviennent pas `raydium_launchpad.*` sans preuve qu'il s'agit d'instructions directes du programme Launchpad. +- Failed transaction = decoded/audit possible, jamais matérialisée métier. +- Non-trade event = jamais trade/candle. +- Pas de nouvelle table métier transversale sans preuve multi-DEX. + +## Requêtes SQL minimales à produire + +Créer un fichier : + +```text +validation_sql/SQL_VALIDATION_RAYDIUM_LAUNCHPAD_0_7_50.sql +``` + +Inclure au minimum : + +```sql +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_launchpad' +ORDER BY entry_kind, entry_name, discriminator_hex; +``` + +Coverage summary : + +```sql +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_launchpad' +GROUP BY decoder_code; +``` + +Instruction observations : + +```sql +SELECT + instruction_name, + discriminator_hex, + COUNT(*) AS observed_count, + COUNT(DISTINCT signature) AS tx_count +FROM k_sol_instruction_observations +WHERE decoder_code = 'raydium_launchpad' +GROUP BY instruction_name, discriminator_hex +ORDER BY observed_count DESC, instruction_name; +``` + +```sql +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_launchpad' +GROUP BY de.event_kind +ORDER BY decoded_count DESC, de.event_kind; +``` + +```sql +SELECT + json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + COUNT(*) AS fallback_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'upstream_git' + AND event_kind = 'upstream_git.instruction_match' + AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_launchpad' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex +ORDER BY fallback_count DESC, entry_name; +``` + +```sql +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_launchpad' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +ORDER BY trade_count DESC, decoded_count DESC; +``` + +## Contraintes de code + +Conserver les règles du workspace : + +```text +Rust 2024 +pas de mod.rs +fichiers Rust avec // file: ... +pas de anyhow +pas de thiserror +pas de ? / unwrap / expect dans kb_lib applicatif +match / if let Err / let Err = ... else +rustdoc sur API publique +re-exports db.rs puis lib.rs si DB modifiée +``` + +## Livrables attendus pour `0.7.50` + +1. delta archive avec uniquement les fichiers ajoutés/modifiés ; +2. mise à jour `README.md`, `ROADMAP.md`, `CHANGELOG.md` ; +3. rapport `docs/RAYDIUM_LAUNCHPAD_EVENT_COVERAGE_REPORT.md` ; +4. SQL `validation_sql/SQL_VALIDATION_RAYDIUM_LAUNCHPAD_0_7_50.sql` ; +5. tests verts : + +```bash +cargo fmt +cargo test -p kb_lib +cargo clippy -p kb_lib --all-targets -- -D warnings +``` + +## Question ouverte à traiter tôt dans `0.7.50` + +Auditer `raydium_pool_v4.json` dans `sol-parser-sdk` : + +- confirmer s'il correspond à un program id distinct ; +- confirmer s'il s'agit d'une surface Raydium Pool/Lending/Strategy différente de `raydium_amm_v4` ; +- décider si une version dédiée `0.7.53 raydium_pool_v4` est nécessaire ; +- ne pas modifier la roadmap comme surface finale tant que le program id n'est pas confirmé. diff --git a/docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md b/docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md new file mode 100644 index 0000000..6694e5a --- /dev/null +++ b/docs/reports/RAYDIUM_CLMM_EVENT_COVERAGE_REPORT.md @@ -0,0 +1,134 @@ + + +# Rapport `0.7.49` — Raydium CLMM event coverage + +## Résumé + +La tranche `0.7.49` clôture la couverture fonctionnelle `raydium_clmm` sur le corpus local validé. Elle reprend CLMM après `0.7.48 raydium_cpmm` et applique la même méthode : inventaire upstream, corpus local, replay forcé, materialization contrôlée, suppression des fallbacks remplacés, validation SQL. + +Validation locale observée après le dernier replay : + +```text +local replay: 2197 replayed, 0 decode skipped, 2197 ledger upserts, 1461 unsafe ledger rows, 1217 trades, 111 liquidity, 25 lifecycle, 4868 candle upserts, instructionObservations=19798 +catalog: 41 tokens, 63 pools, 63 pairs +``` + +Synthèse coverage : + +```text +listed_entry_count = 45 +decoded_entry_count = 33 +observed_entry_count = 33 +materialized_entry_count = 25 +total_observed_count = 2560 +total_materialized_count = 1367 +trade_count = 1186 +``` + +## Entrées couvertes + +La tranche couvre `33` instructions locales CLMM observées et décodées, dont : + +```text +swap +swap_v2 +swap_router_base_in +open_position +open_position_v2 +open_position_with_token22_nft +close_position +close_protocol_position +increase_liquidity +increase_liquidity_v2 +decrease_liquidity +decrease_liquidity_v2 +create_pool +create_customizable_pool +create_amm_config +create_dynamic_fee_config +update_amm_config +update_pool_status +create_operation_account +update_operation_account +create_support_mint_associated +collect_fund_fee +collect_protocol_fee +collect_remaining_rewards +initialize_reward +set_reward_params +transfer_reward_owner +update_reward_infos +open_limit_order +increase_limit_order +decrease_limit_order +close_limit_order +settle_limit_order +``` + +## Matérialisations validées + +Les matérialisations sont limitées aux transactions réussies et aux familles déjà supportées par le modèle DB : + +| Famille | Table | Règle | +|---|---|---| +| Swaps | `k_sol_trade_events` + candles | Uniquement `raydium_clmm.swap` / `raydium_clmm.swap_v2` quand les montants sont exploitables. | +| Liquidity / positions | `k_sol_liquidity_events` | Positions/liquidity prouvées par corpus, sans trade/candle. | +| Fees | `k_sol_fee_events` | Fees prouvées par corpus, sans trade/candle. | +| Rewards | `k_sol_reward_events` | Rewards prouvées par corpus, sans trade/candle. | +| Admin/config | `k_sol_pool_admin_events` | Config/admin prouvés par corpus, sans trade/candle. | +| Lifecycle | `k_sol_pool_lifecycle_events` | Pool creation prouvée par corpus, sans trade/candle. | +| Limit orders | `k_sol_orderbook_events` | Open/increase/decrease/close/settle matérialisés comme orderbook, jamais trade/candle. | + +## Invariants validés + +Les requêtes SQL finales valident : + +```text +raydium_clmm.instruction_audit résiduel = 0 +upstream_git.instruction_match localement couvert = 0 +non-swap CLMM avec trade_count > 0 = 0 +failed tx matérialisées = 0 +``` + +Les fallbacks `upstream_git.instruction_match` localement couverts sont supprimés automatiquement, y compris quand `k_sol_instruction_observations.decoded_event_id` pointait encore vers une ligne fallback. + +## Anchor / Program-data events non observés + +Les `11` Anchor / `Program data` events CLMM ci-dessous restent listés en `upstream_git_unverified`, car aucun corpus local ne les observe encore comme event direct : + +```text +collect_personal_fee_event a6ae69c051a15369 +collect_protocol_fee_event ce57114f2d29d53d +config_change_event f7bd07776a705f97 +create_personal_position_event 641e57f9c4df9ace +decrease_liquidity_event 3ade563a44325538 +increase_liquidity_event 314f69d420221e54 +liquidity_calculate_event ed7094e63954b4a2 +liquidity_change_event 7ef0afce9e58996b +pool_created_event 195e4b2f7063353f +swap_event 40c6cde8260871e2 +update_reward_infos_event 6d7fba4e724125ec +``` + +Le code est préparé pour les accueillir comme audit-only lorsqu’ils seront observés dans un corpus local. Ils ne produisent pas de trade/candle par défaut. + +## Sources utilisées + +Les entrées ont été comparées aux sources Raydium CLMM suivantes : + +```text +Solscan Program IDL CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK +sevenlabs-hq/carbon raydium-clmm-decoder +pinax-network/substreams-solana-idls raydium/clmm +0xfnzero/sol-parser-sdk idl +``` + +Les sources upstream restent des indices. La promotion locale dépend du corpus local, du replay et des validations SQL. + +## SQL final + +La validation finale est dans : + +```text +validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql +``` diff --git a/docs/reports/RAYDIUM_CLMM_UPSTREAM_COVERAGE_REVIEW_PRE19.md b/docs/reports/RAYDIUM_CLMM_UPSTREAM_COVERAGE_REVIEW_PRE19.md new file mode 100644 index 0000000..84d3ae4 --- /dev/null +++ b/docs/reports/RAYDIUM_CLMM_UPSTREAM_COVERAGE_REVIEW_PRE19.md @@ -0,0 +1,130 @@ +# file: docs/reports/RAYDIUM_CLMM_UPSTREAM_COVERAGE_REVIEW_PRE19.md + +# Raydium CLMM upstream coverage review — `0.7.49-pre.19` + +## Scope + +Decoder under review: + +```text +raydium_clmm +program_id = CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK +``` + +External sources checked for the local coverage registry: + +```text +https://solscan.io/account/CAMMCzo5YL8w4VFF8KVHrK22GGUsp5VTaW7grrKgrWqK#programIdl +https://github.com/sevenlabs-hq/carbon/tree/main/decoders/raydium-clmm-decoder +https://github.com/pinax-network/substreams-solana-idls/tree/main/src/raydium/clmm +https://github.com/0xfnzero/sol-parser-sdk/tree/main/idl +``` + +Notes: + +- Solscan is used as discovery / IDL cross-check, not as final business proof. +- `sol-parser-sdk/idl/raydium_clmm.json` maps to `raydium/amm_v3_with_swapv2.json` and exposes the CLMM program address. +- `pinax-network/substreams-solana-idls` exposes Raydium CLMM under `src/raydium/clmm/v3`. +- `sevenlabs-hq/carbon` exposes a dedicated `raydium-clmm-decoder` crate. + +## Local registry status + +The local registry now lists: + +```text +45 entries total: +- 33 instructions +- 11 Anchor/program-data events +- 1 program row +``` + +All 33 instruction entries have a `local_event_kind` mapping in `dex_event_coverage.rs`, `instruction_observation_index.rs`, `dex_decode.rs`, and `upstream_registry_generated.rs`. + +The 11 event entries are listed as upstream facts and stay `upstream_git_unverified` until local corpus provides program-data/Anchor event proof. + +## Instruction coverage matrix + +| Entry | Discriminator | Family | Local event kind | Expected target | Status | +|---|---|---|---|---|---| +| close_limit_order | `4c7c800fd55725fa` | order_cancel | `raydium_clmm.close_limit_order` | `k_sol_orderbook_events` | decoded/materializable | +| open_limit_order | `9d20dab7471d1293` | order_place | `raydium_clmm.open_limit_order` | `k_sol_orderbook_events` | decoded/materializable | +| increase_limit_order | `b19059ecfaba7d63` | order_place | `raydium_clmm.increase_limit_order` | `k_sol_orderbook_events` | decoded/materializable | +| decrease_limit_order | `759d3c674231a300` | order_cancel | `raydium_clmm.decrease_limit_order` | `k_sol_orderbook_events` | decoded/materializable | +| close_position | `7b86510031446262` | position_close | `raydium_clmm.close_position` | `k_sol_liquidity_events` | decoded/materializable | +| close_protocol_position | `c975989055556cb2` | position_close | `raydium_clmm.close_protocol_position` | `k_sol_liquidity_events` | decoded/materializable | +| collect_fund_fee | `a78a4e95dfc2067e` | fee | `raydium_clmm.collect_fund_fee` | `k_sol_fee_events` | decoded/materializable | +| collect_protocol_fee | `8888fcddc2427e59` | fee | `raydium_clmm.collect_protocol_fee` | `k_sol_fee_events` | decoded/materializable | +| collect_remaining_rewards | `12eda6c52210d590` | reward | `raydium_clmm.collect_remaining_rewards` | `k_sol_reward_events` | decoded/materializable | +| create_amm_config | `8934edd4d7756c68` | admin_config | `raydium_clmm.create_amm_config` | `k_sol_pool_admin_events` | decoded/materializable | +| create_customizable_pool | `2b44d4a7592fa401` | pool_create | `raydium_clmm.create_customizable_pool` | `k_sol_pool_lifecycle_events` | decoded/materializable | +| create_dynamic_fee_config | `bd0eb5785576e33e` | admin_config | `raydium_clmm.create_dynamic_fee_config` | `k_sol_pool_admin_events` | decoded/materializable | +| create_operation_account | `3f5794216d230868` | unknown | `raydium_clmm.create_operation_account` | `k_sol_dex_decoded_events_only` | decoded/audit-only unless admin evidence is required | +| create_pool | `e992d18ecf6840bc` | pool_create | `raydium_clmm.create_pool` | `k_sol_pool_lifecycle_events` | decoded/materializable | +| create_support_mint_associated | `11fb415c88f20ea9` | account_create | `raydium_clmm.create_support_mint_associated` | `k_sol_token_account_events` | decoded; token-account materialization remains a later cross-DEX topic | +| decrease_liquidity | `a026d06f685b2c01` | liquidity_remove | `raydium_clmm.decrease_liquidity` | `k_sol_liquidity_events` | decoded/materializable | +| decrease_liquidity_v2 | `3a7fbc3e4f52c460` | liquidity_remove | `raydium_clmm.decrease_liquidity_v2` | `k_sol_liquidity_events` | decoded/materializable | +| increase_liquidity | `2e9cf3760dcdfbb2` | liquidity_add | `raydium_clmm.increase_liquidity` | `k_sol_liquidity_events` | decoded/materializable | +| increase_liquidity_v2 | `851d59df45eeb00a` | liquidity_add | `raydium_clmm.increase_liquidity_v2` | `k_sol_liquidity_events` | decoded/materializable | +| initialize_reward | `5f87c0c4f281e644` | reward | `raydium_clmm.initialize_reward` | `k_sol_reward_events` | decoded/materializable | +| open_position | `87802f4d0f98f031` | position_open | `raydium_clmm.open_position` | `k_sol_liquidity_events` | decoded/materializable | +| open_position_v2 | `4db84ad67056f1c7` | position_open | `raydium_clmm.open_position_v2` | `k_sol_liquidity_events` | decoded/materializable | +| open_position_with_token22_nft | `4dffae527d1dc92e` | position_open | `raydium_clmm.open_position_with_token22_nft` | `k_sol_liquidity_events` | decoded/materializable | +| set_reward_params | `7034a74b20c9d389` | reward | `raydium_clmm.set_reward_params` | `k_sol_reward_events` | decoded/materializable | +| settle_limit_order | `cd4e74215c691a60` | settle_funds | `raydium_clmm.settle_limit_order` | `k_sol_orderbook_events` | decoded/materializable | +| swap | `f8c69e91e17587c8` | swap | `raydium_clmm.swap` | `k_sol_trade_events` | decoded/materializable as trade | +| swap_router_base_in | `457d73daf5baf2c4` | swap | `raydium_clmm.swap_router_base_in` | `k_sol_trade_events` | decoded; observed but not promoted to trade without corpus proof | +| swap_v2 | `2b04ed0b1ac91e62` | swap | `raydium_clmm.swap_v2` | `k_sol_trade_events` | decoded/materializable as trade | +| transfer_reward_owner | `07160c53f22b3079` | reward | `raydium_clmm.transfer_reward_owner` | `k_sol_reward_events` | decoded/materializable | +| update_amm_config | `313cae889a1c74c8` | admin_config | `raydium_clmm.update_amm_config` | `k_sol_pool_admin_events` | decoded/materializable | +| update_operation_account | `7f467728bce33d07` | unknown | `raydium_clmm.update_operation_account` | `k_sol_dex_decoded_events_only` | decoded/audit-only unless admin evidence is required | +| update_pool_status | `82576c062ee0757b` | admin_config | `raydium_clmm.update_pool_status` | `k_sol_pool_admin_events` | decoded/materializable | +| update_reward_infos | `a3ace0340b9a6adf` | reward | `raydium_clmm.update_reward_infos` | `k_sol_reward_events` | decoded/materializable | + +## Anchor/program-data event entries + +These entries remain listed, but no local corpus row has observed them as direct CLMM decoded events in the current replay corpus: + +| Event entry | Discriminator | Family | Expected target | Status | +|---|---|---|---|---| +| collect_personal_fee_event | `a6ae69c051a15369` | fee | `k_sol_fee_events` | upstream listed, local corpus unobserved | +| collect_protocol_fee_event | `ce57114f2d29d53d` | fee | `k_sol_fee_events` | upstream listed, local corpus unobserved | +| config_change_event | `f7bd07776a705f97` | admin_config | `k_sol_pool_admin_events` | upstream listed, local corpus unobserved | +| create_personal_position_event | `641e57f9c4df9ace` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved | +| decrease_liquidity_event | `3ade563a44325538` | liquidity_remove | `k_sol_liquidity_events` | upstream listed, local corpus unobserved | +| increase_liquidity_event | `314f69d420221e54` | liquidity_add | `k_sol_liquidity_events` | upstream listed, local corpus unobserved | +| liquidity_calculate_event | `ed7094e63954b4a2` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved | +| liquidity_change_event | `7ef0afce9e58996b` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved | +| pool_created_event | `195e4b2f7063353f` | unknown | `k_sol_dex_decoded_events_only` | upstream listed, local corpus unobserved | +| swap_event | `40c6cde8260871e2` | swap | `k_sol_trade_events` | upstream listed, local corpus unobserved | +| update_reward_infos_event | `6d7fba4e724125ec` | reward | `k_sol_reward_events` | upstream listed, local corpus unobserved | + +## Patch `pre.19` + +Patch goal: + +```text +Stop replay/backfill from leaving `upstream_git.instruction_match` rows when a local specialized decoder already covers the same upstream decoder + entry + discriminator. +``` + +New DB query: + +```text +query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches(database, upstream_decoder_code) +``` + +Called from: + +```text +local_pipeline_replay.rs::refresh_event_coverage_best_effort +token_backfill.rs::refresh_event_coverage_best_effort +``` + +Expected validation after replay: + +```text +raydium_clmm.instruction_audit residual query -> empty +upstream_git.instruction_match where upstreamDecoderCode = raydium_clmm -> empty +non-swap CLMM trade_count -> empty +failed CLMM materialization query -> empty +coverage summary remains populated around 45 listed / 33 decoded / 33 observed +``` diff --git a/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md b/docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md similarity index 100% rename from RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md rename to docs/reports/RAYDIUM_CPMM_EVENT_COVERAGE_REPORT.md diff --git a/docs/reports/RAYDIUM_CPMM_UPSTREAM_COVERAGE_REVIEW_PRE22.md b/docs/reports/RAYDIUM_CPMM_UPSTREAM_COVERAGE_REVIEW_PRE22.md new file mode 100644 index 0000000..b02469d --- /dev/null +++ b/docs/reports/RAYDIUM_CPMM_UPSTREAM_COVERAGE_REVIEW_PRE22.md @@ -0,0 +1,72 @@ +# Raydium CPMM upstream coverage review — 0.7.49-pre.22 + +## Scope + +Compared local `raydium_cpmm` coverage against the currently referenced upstream surfaces: + +- Solscan program IDL for `CPMMoo8L3F4NbTegBCKVNunggL7H1ZpdTHKxQB5qKP1C`. +- `sevenlabs-hq/carbon` `raydium-cpmm-decoder`. +- `0xfnzero/sol-parser-sdk` `idl/raydium_cpmm.json`. +- `pinax-network/substreams-solana-idls` `src/raydium/cpmm`. +- Raydium official `raydium-cp-swap` source. + +## Local CPMM coverage entries + +Local registry currently lists 16 `raydium_cpmm` entries: + +### Instructions + +- `close_permission_pda` — `9c5420764587467b` +- `collect_creator_fee` — `1416567bc61cdb84` +- `collect_fund_fee` — `a78a4e95dfc2067e` +- `collect_protocol_fee` — `8888fcddc2427e59` +- `create_amm_config` — `8934edd4d7756c68` +- `create_permission_pda` — `878802d889a9b5ca` +- `deposit` — `f223c68952e1f2b6` +- `initialize` — `afaf6d1f0d989bed` +- `initialize_with_permission` — `3f37fe4131b25979` +- `swap_base_input` — `8fbe5adac41e33de` +- `swap_base_output` — `37d96256a34ab4ad` +- `update_amm_config` — `313cae889a1c74c8` +- `update_pool_status` — `82576c062ee0757b` +- `withdraw` — `b712469c946da122` + +### Anchor / Program-data events + +- `lp_change_event` — `79a3cdc939da753c` +- `swap_event` — `40c6cde8260871e2` + +## Upstream comparison + +Carbon `raydium-cpmm-decoder` exposes the same 14 instruction modules and the two event-like discriminator entries, `lp_change_event` and `swap_event`. + +`sol-parser-sdk` `idl/raydium_cpmm.json` exposes the core CPMM instruction set (`createAmmConfig`, `updateAmmConfig`, `updatePoolStatus`, `collectProtocolFee`, `collectFundFee`, `initialize`, `deposit`, `withdraw`, `swapBaseInput`, `swapBaseOutput`) and IDL events `LpChangeEvent` and `SwapEvent`. The local registry also includes the permission and creator-fee entries present in Carbon / Raydium source. + +The official Raydium `raydium-cp-swap` source lists the CPMM program ID and the main program instructions including admin/config, fee collection, permission PDA, initialize, initialize with permission, deposit, withdraw, swap base input, and swap base output. + +## Finding + +No missing CPMM instruction/event discriminator was identified relative to the reviewed Carbon / Raydium / fnzero / Pinax surfaces available during this check. + +## Current local caveat + +CPMM remains covered by the earlier 0.7.48 tranche. The useful final validation remains DB-side: + +```sql +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_cpmm' +ORDER BY entry_kind, entry_name, discriminator_hex; +``` + +Any future upstream addition should appear as a new entry in Carbon/Solscan/IDL and should be added to `upstream_registry_generated.rs`, `known_local_event_kind` only after local decoder support exists, and then validated with local corpus evidence. diff --git a/kb_demo_app/frontend/demo_pipeline2.html b/kb_demo_app/frontend/demo_pipeline2.html index 9e1b47b..ae069df 100644 --- a/kb_demo_app/frontend/demo_pipeline2.html +++ b/kb_demo_app/frontend/demo_pipeline2.html @@ -122,6 +122,29 @@ Backfill signature + +
+ +
+ + +
+ Backfill ciblé de plusieurs signatures. Les lignes vides et les doublons sont ignorés. +
+
+ +
+ + +
+ +
+ +
@@ -162,15 +185,22 @@ -
+
+
+ + +
+

- Le skip ne concerne que l’étape de décodage DEX certifiée par le ledger. Le reste du replay continue pour reconstruire les tables dérivées. + Le skip ne concerne que l’étape de décodage DEX certifiée par le ledger. Le reste du replay continue pour reconstruire les tables dérivées. Le refresh différé des observations accélère le replay mais les compteurs SQL d’instructions ne sont finalisés qu’à la fin.

diff --git a/kb_demo_app/frontend/ts/bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts b/kb_demo_app/frontend/ts/bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts new file mode 100644 index 0000000..4d9b74f --- /dev/null +++ b/kb_demo_app/frontend/ts/bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts @@ -0,0 +1,18 @@ +// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. + +/** + * Request payload for batch signature backfill. + */ +export type DemoPipeline2BackfillSignaturesBatchRequest = { +/** + * Transaction signatures to resolve and replay. + */ +signatures: Array, +/** + * Optional HTTP role. + */ +httpRole: string | null, +/** + * Whether the batch should continue after a hard per-signature error. + */ +continueOnError: boolean, }; diff --git a/kb_demo_app/frontend/ts/demo_pipeline2.ts b/kb_demo_app/frontend/ts/demo_pipeline2.ts index fbf3d27..d45cd48 100644 --- a/kb_demo_app/frontend/ts/demo_pipeline2.ts +++ b/kb_demo_app/frontend/ts/demo_pipeline2.ts @@ -11,6 +11,7 @@ import type { DemoPipeline2CatalogPayload } from "./bindings/DemoPipeline2Catalo import type { DemoPipeline2BackfillTokenRequest } from "./bindings/DemoPipeline2BackfillTokenRequest.ts"; import type { DemoPipeline2BackfillPoolRequest } from "./bindings/DemoPipeline2BackfillPoolRequest.ts"; import type { DemoPipeline2BackfillSignatureRequest } from "./bindings/DemoPipeline2BackfillSignatureRequest.ts"; +import type { DemoPipeline2BackfillSignaturesBatchRequest } from "./bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts"; import type { DemoPipeline2BackfillPayload } from "./bindings/DemoPipeline2BackfillPayload.ts"; import type { DemoPipeline2PairCandlesRequest } from "./bindings/DemoPipeline2PairCandlesRequest.ts"; import type { DemoPipeline2PairCandlesPayload } from "./bindings/DemoPipeline2PairCandlesPayload.ts"; @@ -75,6 +76,8 @@ interface LocalPipelineReplayResult { tokenMetadataUpdatedCount: number; pairSymbolUpdatedCount: number; resetMarketMaterializationDeletedCount: number; + instructionObservationScannedCount: number; + instructionObservationUpsertedCount: number; globalErrorCount: number; } function appendLogLine(textarea: HTMLTextAreaElement, line: string): void { @@ -152,6 +155,25 @@ function readOptionalPositiveIntegerInput( return parsed; } +function parseSignatureBatchText(raw: string): string[] { + const signatures: string[] = []; + const seen = new Set(); + + for (const line of raw.split(/\r?\n/)) { + const signature = line.trim(); + if (signature === "") { + continue; + } + if (seen.has(signature)) { + continue; + } + seen.add(signature); + signatures.push(signature); + } + + return signatures; +} + function refreshPairSelect( catalog: DemoPipeline2CatalogPayload, select: HTMLSelectElement, @@ -410,12 +432,16 @@ document.addEventListener("DOMContentLoaded", async () => { const signatureInput = document.querySelector("#demoPipeline2SignatureInput"); const backfillSignatureButton = document.querySelector("#demoPipeline2BackfillSignatureButton"); + const signatureBatchTextarea = document.querySelector("#demoPipeline2SignatureBatchTextarea"); + const signatureBatchContinueOnErrorCheckbox = document.querySelector("#demoPipeline2SignatureBatchContinueOnErrorCheckbox"); + const backfillSignatureBatchButton = document.querySelector("#demoPipeline2BackfillSignatureBatchButton"); const replayLimitInput = document.querySelector("#demoPipeline2ReplayLimitInput"); const replayMetadataCheckbox = document.querySelector("#demoPipeline2ReplayMetadataCheckbox"); const replayMetadataLimitInput = document.querySelector("#demoPipeline2ReplayMetadataLimitInput"); const replaySkipCertifiedDexDecodeCheckbox = document.querySelector("#demoPipeline2ReplaySkipCertifiedDexDecodeCheckbox"); const replayForceDexDecodeCheckbox = document.querySelector("#demoPipeline2ReplayForceDexDecodeCheckbox"); + const replayDeferInstructionObservationCheckbox = document.querySelector("#demoPipeline2ReplayDeferInstructionObservationCheckbox"); const replayLocalPipelineButton = document.querySelector("#demoPipeline2ReplayLocalPipelineButton"); const diagnoseLocalPipelineButton = document.querySelector("#demoPipeline2DiagnoseLocalPipelineButton"); const validateLocalPipelineButton = document.querySelector("#demoPipeline2ValidateLocalPipelineButton"); @@ -462,11 +488,15 @@ document.addEventListener("DOMContentLoaded", async () => { !backfillPoolButton || !signatureInput || !backfillSignatureButton || + !signatureBatchTextarea || + !signatureBatchContinueOnErrorCheckbox || + !backfillSignatureBatchButton || !replayLimitInput || !replayMetadataCheckbox || !replayMetadataLimitInput || !replaySkipCertifiedDexDecodeCheckbox || !replayForceDexDecodeCheckbox || + !replayDeferInstructionObservationCheckbox || !replayLocalPipelineButton || !diagnoseLocalPipelineButton || !validateLocalPipelineButton || @@ -685,6 +715,55 @@ document.addEventListener("DOMContentLoaded", async () => { } }); + backfillSignatureBatchButton.addEventListener("click", async () => { + const signatures = parseSignatureBatchText(signatureBatchTextarea.value); + if (signatures.length === 0) { + appendLogLine(logTextarea, "[ui] signature batch is empty"); + return; + } + + const httpRoleText = httpRoleInput.value.trim(); + const httpRole = httpRoleText === "" ? null : httpRoleText; + const continueOnError = signatureBatchContinueOnErrorCheckbox.checked; + + appendLogLine( + logTextarea, + `[ui] launching signature batch backfill count='${signatures.length.toString()}' role='${httpRole ?? "history_backfill"}' continueOnError='${continueOnError ? "yes" : "no"}'`, + ); + + const request: DemoPipeline2BackfillSignaturesBatchRequest = { + signatures, + httpRole, + continueOnError, + }; + + backfillSignatureBatchButton.disabled = true; + backfillSignatureButton.disabled = true; + backfillMintButton.disabled = true; + backfillPoolButton.disabled = true; + + try { + const payload = await invoke( + "demo_pipeline2_backfill_signatures_batch", + { request }, + ); + + backfillSummaryTextarea.value = payload.summaryJson; + currentCatalog = payload.catalog; + renderCatalogTextareas(payload.catalog, tokensTextarea, poolsTextarea, pairsTextarea); + refreshPairSelect(payload.catalog, pairSelect); + + appendLogLine(logTextarea, `[ui] signature batch backfill completed for '${payload.objectKey}'`); + } catch (error) { + appendLogLine(logTextarea, `[ui] signature batch backfill error: ${String(error)}`); + } finally { + backfillSignatureBatchButton.disabled = false; + backfillSignatureButton.disabled = false; + backfillMintButton.disabled = false; + backfillPoolButton.disabled = false; + } + }); + replayLocalPipelineButton.addEventListener("click", async () => { const replayLimit = readOptionalPositiveIntegerInput( replayLimitInput, @@ -706,7 +785,7 @@ document.addEventListener("DOMContentLoaded", async () => { appendLogLine( logTextarea, - `[ui] launching local pipeline replay limit='${replayLimit ?? "none"}' metadata='${replayMetadataCheckbox.checked ? "yes" : "no"}' skipDexDecode='${replaySkipCertifiedDexDecodeCheckbox.checked ? "yes" : "no"}' forceDexDecode='${replayForceDexDecodeCheckbox.checked ? "yes" : "no"}'`, + `[ui] launching local pipeline replay limit='${replayLimit ?? "none"}' metadata='${replayMetadataCheckbox.checked ? "yes" : "no"}' skipDexDecode='${replaySkipCertifiedDexDecodeCheckbox.checked ? "yes" : "no"}' forceDexDecode='${replayForceDexDecodeCheckbox.checked ? "yes" : "no"}' deferInstructionObservations='${replayDeferInstructionObservationCheckbox.checked ? "yes" : "no"}'`, ); try { @@ -718,6 +797,7 @@ document.addEventListener("DOMContentLoaded", async () => { tokenMetadataLimit, skipCertifiedDexDecode: replaySkipCertifiedDexDecodeCheckbox.checked, forceDecodeReplay: replayForceDexDecodeCheckbox.checked, + deferInstructionObservationIndexRefresh: replayDeferInstructionObservationCheckbox.checked, }, ); @@ -725,7 +805,7 @@ document.addEventListener("DOMContentLoaded", async () => { appendLogLine( logTextarea, - `[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.decodeSkippedCount.toString()} decode skipped, ${result.decodeLedgerUpsertCount.toString()} ledger upserts, ${result.decodeLedgerUnsafeCount.toString()} unsafe ledger rows, ${result.tradeEventCount.toString()} trades, ${result.liquidityEventCount.toString()} liquidity, ${result.poolLifecycleEventCount.toString()} lifecycle, ${result.pairCandleUpsertCount.toString()} candle upserts, resetDeleted='${result.resetMarketMaterializationDeletedCount.toString()}'`, + `[ui] local pipeline replay completed: ${result.replayedTransactionCount.toString()} replayed, ${result.decodeSkippedCount.toString()} decode skipped, ${result.decodeLedgerUpsertCount.toString()} ledger upserts, ${result.decodeLedgerUnsafeCount.toString()} unsafe ledger rows, ${result.tradeEventCount.toString()} trades, ${result.liquidityEventCount.toString()} liquidity, ${result.poolLifecycleEventCount.toString()} lifecycle, ${result.pairCandleUpsertCount.toString()} candle upserts, instructionObservations='${result.instructionObservationUpsertedCount.toString()}', resetDeleted='${result.resetMarketMaterializationDeletedCount.toString()}'`, ); await refreshCatalog(); diff --git a/kb_demo_app/package.json b/kb_demo_app/package.json index ebbb15a..3125ccf 100644 --- a/kb_demo_app/package.json +++ b/kb_demo_app/package.json @@ -1,7 +1,7 @@ { "name": "kb-demo-app", "private": true, - "version": "0.7.48", + "version": "0.7.49", "type": "module", "scripts": { "dev": "vite", diff --git a/kb_demo_app/src/demo_pipeline2.rs b/kb_demo_app/src/demo_pipeline2.rs index 3a66b94..2553a8b 100644 --- a/kb_demo_app/src/demo_pipeline2.rs +++ b/kb_demo_app/src/demo_pipeline2.rs @@ -1153,6 +1153,22 @@ pub(crate) struct DemoPipeline2BackfillSignatureRequest { pub http_role: std::option::Option, } +/// Request payload for batch signature backfill. +#[derive(Clone, Debug, serde::Deserialize, TS)] +#[ts( + export, + export_to = "../frontend/ts/bindings/DemoPipeline2BackfillSignaturesBatchRequest.ts" +)] +#[serde(rename_all = "camelCase")] +pub(crate) struct DemoPipeline2BackfillSignaturesBatchRequest { + /// Transaction signatures to resolve and replay. + pub signatures: std::vec::Vec, + /// Optional HTTP role. + pub http_role: std::option::Option, + /// Whether the batch should continue after a hard per-signature error. + pub continue_on_error: bool, +} + /// Shared backfill response payload. #[derive(Clone, Debug, serde::Serialize, TS)] #[ts(export, export_to = "../frontend/ts/bindings/DemoPipeline2BackfillPayload.ts")] @@ -1604,6 +1620,63 @@ pub(crate) async fn demo_pipeline2_backfill_signature( }) } +/// Runs a targeted batch signature backfill then returns the refreshed catalog. +#[tauri::command] +pub(crate) async fn demo_pipeline2_backfill_signatures_batch( + state: tauri::State<'_, crate::AppState>, + request: DemoPipeline2BackfillSignaturesBatchRequest, +) -> Result { + let mut signatures = std::vec::Vec::::new(); + for signature in request.signatures { + let trimmed_signature = signature.trim().to_string(); + if trimmed_signature.is_empty() { + continue; + } + signatures.push(trimmed_signature); + } + if signatures.is_empty() { + return Err("signature batch must contain at least one signature".to_string()); + } + let http_role = demo_pipeline2_normalize_http_role(request.http_role); + let database = state.database.clone(); + let http_pool = std::sync::Arc::new(state.http_pool.clone()); + let service = kb_lib::TokenBackfillService::new(http_pool, database.clone(), http_role.clone()); + let result = service + .backfill_signatures(signatures.as_slice(), request.continue_on_error) + .await; + let backfill = match result { + Ok(backfill) => backfill, + Err(error) => { + return Err(format!( + "cannot backfill signature batch with role '{}': {}", + http_role, error + )); + }, + }; + let summary_json_result = serde_json::to_string_pretty(&backfill); + let summary_json = match summary_json_result { + Ok(summary_json) => summary_json, + Err(error) => { + return Err(format!( + "cannot serialize signature batch backfill result: {}", + error + )); + }, + }; + let catalog_result = demo_pipeline2_build_catalog(database).await; + let catalog = match catalog_result { + Ok(catalog) => catalog, + Err(error) => return Err(error), + }; + Ok(DemoPipeline2BackfillPayload { + object_key: format!("{} signatures", backfill.unique_signature_count), + mode: "signatureBatch".to_string(), + http_role, + summary_json, + catalog, + }) +} + /// Loads candles for one pair and one timeframe. #[tauri::command] pub(crate) async fn demo_pipeline2_get_pair_candles( @@ -1783,6 +1856,7 @@ pub(crate) async fn demo_pipeline2_replay_local_pipeline( token_metadata_limit: std::option::Option, skip_certified_dex_decode: bool, force_decode_replay: bool, + defer_instruction_observation_index_refresh: bool, ) -> Result { let config = kb_lib::LocalPipelineReplayConfig { limit, @@ -1791,6 +1865,7 @@ pub(crate) async fn demo_pipeline2_replay_local_pipeline( reset_market_materialization_before_replay: true, skip_certified_dex_decode, force_decode_replay, + defer_instruction_observation_index_refresh, }; let database = state.database.clone(); let service = if refresh_missing_token_metadata { diff --git a/kb_demo_app/src/lib.rs b/kb_demo_app/src/lib.rs index 3a1253f..40868d6 100644 --- a/kb_demo_app/src/lib.rs +++ b/kb_demo_app/src/lib.rs @@ -158,6 +158,7 @@ pub async fn run() -> Result<(), kb_lib::Error> { crate::demo_pipeline2::demo_pipeline2_backfill_token_mint, crate::demo_pipeline2::demo_pipeline2_backfill_pool_address, crate::demo_pipeline2::demo_pipeline2_backfill_signature, + crate::demo_pipeline2::demo_pipeline2_backfill_signatures_batch, crate::demo_pipeline2::demo_pipeline2_get_pair_candles, crate::demo_pipeline2::demo_pipeline2_replay_local_pipeline, crate::demo_pipeline2::demo_pipeline2_diagnose_local_pipeline, diff --git a/kb_demo_app/tauri.conf.json b/kb_demo_app/tauri.conf.json index a0ad342..2d58366 100644 --- a/kb_demo_app/tauri.conf.json +++ b/kb_demo_app/tauri.conf.json @@ -1,7 +1,7 @@ { "$schema": "https://schema.tauri.app/config/2", "productName": "kb-demo-app", - "version": "0.7.48", + "version": "0.7.49", "identifier": "com.sasedev.kb-demo-app", "build": { "beforeDevCommand": "npm run dev", diff --git a/kb_lib/src/db.rs b/kb_lib/src/db.rs index 6399007..18d6ffd 100644 --- a/kb_lib/src/db.rs +++ b/kb_lib/src/db.rs @@ -61,6 +61,7 @@ pub use dtos::LocalRaydiumSurfaceDiagnosticSummaryDto; pub use dtos::LocalTokenMetadataGapDiagnosticSampleDto; pub use dtos::ObservedTokenDto; pub use dtos::OnchainObservationDto; +pub use dtos::OrderbookEventDto; pub use dtos::PairAnalyticSignalDto; pub use dtos::PairCandleDto; pub use dtos::PairDto; @@ -106,6 +107,7 @@ pub use entities::LaunchSurfaceKeyEntity; pub use entities::LiquidityEventEntity; pub use entities::ObservedTokenEntity; pub use entities::OnchainObservationEntity; +pub use entities::OrderbookEventEntity; pub use entities::PairAnalyticSignalEntity; pub use entities::PairCandleEntity; pub use entities::PairEntity; @@ -152,8 +154,10 @@ pub use queries::query_dex_decode_replay_ledger_get_by_signature; pub use queries::query_dex_decode_replay_ledger_get_by_transaction; pub use queries::query_dex_decode_replay_ledger_upsert; pub use queries::query_dex_decoded_events_delete_by_key; +pub use queries::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches; pub use queries::query_dex_decoded_events_delete_meteora_dlmm_anchor_swap_instruction_audits; pub use queries::query_dex_decoded_events_delete_related_instruction_audit; +pub use queries::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits; pub use queries::query_dex_decoded_events_get_by_key; pub use queries::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint; pub use queries::query_dex_decoded_events_list_by_transaction_id; @@ -216,6 +220,7 @@ pub use queries::query_observed_tokens_list; pub use queries::query_observed_tokens_upsert; pub use queries::query_onchain_observations_insert; pub use queries::query_onchain_observations_list_recent; +pub use queries::query_orderbook_events_upsert; pub use queries::query_pair_analytic_signals_get_by_key; pub use queries::query_pair_analytic_signals_list_by_pair_id; pub use queries::query_pair_analytic_signals_upsert; diff --git a/kb_lib/src/db/dtos.rs b/kb_lib/src/db/dtos.rs index eb4a8c7..7cafdb0 100644 --- a/kb_lib/src/db/dtos.rs +++ b/kb_lib/src/db/dtos.rs @@ -24,6 +24,7 @@ mod local_dex_corpus_search; mod local_pipeline_diagnostics; mod observed_token; mod onchain_observation; +mod orderbook_event; mod pair; mod pair_analytic_signal; mod pair_candle; @@ -117,6 +118,7 @@ pub use local_pipeline_diagnostics::LocalRaydiumSurfaceDiagnosticSummaryDto; pub use local_pipeline_diagnostics::LocalTokenMetadataGapDiagnosticSampleDto; pub use observed_token::ObservedTokenDto; pub use onchain_observation::OnchainObservationDto; +pub use orderbook_event::OrderbookEventDto; pub use pair::PairDto; pub use pair_analytic_signal::PairAnalyticSignalDto; pub use pair_candle::PairCandleDto; diff --git a/kb_lib/src/db/dtos/orderbook_event.rs b/kb_lib/src/db/dtos/orderbook_event.rs new file mode 100644 index 0000000..f6637a1 --- /dev/null +++ b/kb_lib/src/db/dtos/orderbook_event.rs @@ -0,0 +1,190 @@ +// file: kb_lib/src/db/dtos/orderbook_event.rs + +//! Orderbook event DTO. + +/// Application-facing normalized orderbook or limit-order event DTO. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct OrderbookEventDto { + /// Optional numeric primary key. + pub id: std::option::Option, + /// Related transaction id. + pub transaction_id: i64, + /// Related decoded DEX event id, when available. + pub decoded_event_id: std::option::Option, + /// Related DEX id, when the DEX row is known. + pub dex_id: std::option::Option, + /// Related pool id, when the pool row is known. + pub pool_id: std::option::Option, + /// Related pair id, when the pair row is known. + pub pair_id: std::option::Option, + /// Transaction signature. + pub signature: std::string::String, + /// Optional slot number. + pub slot: std::option::Option, + /// Protocol name that emitted the decoded event. + pub protocol_name: std::string::String, + /// Program id that emitted the decoded event. + pub program_id: std::string::String, + /// Stable decoded event kind. + pub event_kind: std::string::String, + /// Normalized orderbook action. + pub order_action: std::string::String, + /// Pool account address, when decoded. + pub pool_account: std::option::Option, + /// Market/orderbook account, when decoded. + pub market_account: std::option::Option, + /// Wallet or authority associated with the event, when decoded. + pub actor_wallet: std::option::Option, + /// Limit/order account, when decoded. + pub order_account: std::option::Option, + /// Base or token A mint, when decoded. + pub base_token_mint: std::option::Option, + /// Quote or token B mint, when decoded. + pub quote_token_mint: std::option::Option, + /// Raw order amount as decimal text, when decoded. + pub amount_raw: std::option::Option, + /// Raw minimum amount as decimal text, when decoded. + pub amount_min_raw: std::option::Option, + /// Optional tick index for CLMM limit orders. + pub tick_index: std::option::Option, + /// Optional zero-for-one side flag. + pub zero_for_one: std::option::Option, + /// Source decoded payload JSON. + pub payload_json: std::string::String, + /// Execution timestamp. + pub executed_at: chrono::DateTime, + /// Creation timestamp. + pub created_at: chrono::DateTime, +} + +impl OrderbookEventDto { + /// Creates a new orderbook event DTO. + #[allow(clippy::too_many_arguments)] + pub fn new( + transaction_id: i64, + decoded_event_id: std::option::Option, + dex_id: std::option::Option, + pool_id: std::option::Option, + pair_id: std::option::Option, + signature: std::string::String, + slot: std::option::Option, + protocol_name: std::string::String, + program_id: std::string::String, + event_kind: std::string::String, + order_action: std::string::String, + pool_account: std::option::Option, + market_account: std::option::Option, + actor_wallet: std::option::Option, + order_account: std::option::Option, + base_token_mint: std::option::Option, + quote_token_mint: std::option::Option, + amount_raw: std::option::Option, + amount_min_raw: std::option::Option, + tick_index: std::option::Option, + zero_for_one: std::option::Option, + payload_json: std::string::String, + ) -> Self { + let now = chrono::Utc::now(); + return Self { + id: None, + transaction_id, + decoded_event_id, + dex_id, + pool_id, + pair_id, + signature, + slot, + protocol_name, + program_id, + event_kind, + order_action, + pool_account, + market_account, + actor_wallet, + order_account, + base_token_mint, + quote_token_mint, + amount_raw, + amount_min_raw, + tick_index, + zero_for_one, + payload_json, + executed_at: now, + created_at: now, + }; + } +} + +impl TryFrom for OrderbookEventDto { + type Error = crate::Error; + + fn try_from(entity: crate::OrderbookEventEntity) -> Result { + let executed_at_result = chrono::DateTime::parse_from_rfc3339(&entity.executed_at); + let executed_at = match executed_at_result { + Ok(executed_at) => executed_at.with_timezone(&chrono::Utc), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot parse orderbook event executed_at '{}': {}", + entity.executed_at, error + ))); + }, + }; + let created_at_result = chrono::DateTime::parse_from_rfc3339(&entity.created_at); + let created_at = match created_at_result { + Ok(created_at) => created_at.with_timezone(&chrono::Utc), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot parse orderbook event created_at '{}': {}", + entity.created_at, error + ))); + }, + }; + let slot = match entity.slot { + Some(slot) => { + let slot_result = u64::try_from(slot); + match slot_result { + Ok(slot) => Some(slot), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot convert orderbook event slot '{}' to u64: {}", + slot, error + ))); + }, + } + }, + None => None, + }; + let zero_for_one = match entity.zero_for_one { + Some(0) => Some(false), + Some(_) => Some(true), + None => None, + }; + return Ok(Self { + id: Some(entity.id), + transaction_id: entity.transaction_id, + decoded_event_id: entity.decoded_event_id, + dex_id: entity.dex_id, + pool_id: entity.pool_id, + pair_id: entity.pair_id, + signature: entity.signature, + slot, + protocol_name: entity.protocol_name, + program_id: entity.program_id, + event_kind: entity.event_kind, + order_action: entity.order_action, + pool_account: entity.pool_account, + market_account: entity.market_account, + actor_wallet: entity.actor_wallet, + order_account: entity.order_account, + base_token_mint: entity.base_token_mint, + quote_token_mint: entity.quote_token_mint, + amount_raw: entity.amount_raw, + amount_min_raw: entity.amount_min_raw, + tick_index: entity.tick_index, + zero_for_one, + payload_json: entity.payload_json, + executed_at, + created_at, + }); + } +} diff --git a/kb_lib/src/db/entities.rs b/kb_lib/src/db/entities.rs index 779a14f..d705364 100644 --- a/kb_lib/src/db/entities.rs +++ b/kb_lib/src/db/entities.rs @@ -24,6 +24,7 @@ mod launch_surface_key; mod liquidity_event; mod observed_token; mod onchain_observation; +mod orderbook_event; mod pair; mod pair_analytic_signal; mod pair_candle; @@ -70,6 +71,7 @@ pub use launch_surface_key::LaunchSurfaceKeyEntity; pub use liquidity_event::LiquidityEventEntity; pub use observed_token::ObservedTokenEntity; pub use onchain_observation::OnchainObservationEntity; +pub use orderbook_event::OrderbookEventEntity; pub use pair::PairEntity; pub use pair_analytic_signal::PairAnalyticSignalEntity; pub use pair_candle::PairCandleEntity; diff --git a/kb_lib/src/db/entities/orderbook_event.rs b/kb_lib/src/db/entities/orderbook_event.rs new file mode 100644 index 0000000..29fbc9d --- /dev/null +++ b/kb_lib/src/db/entities/orderbook_event.rs @@ -0,0 +1,58 @@ +// file: kb_lib/src/db/entities/orderbook_event.rs + +//! Orderbook event entity. + +/// Persisted normalized orderbook or limit-order event row. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize, sqlx::FromRow)] +pub struct OrderbookEventEntity { + /// Numeric primary key. + pub id: i64, + /// Related transaction id. + pub transaction_id: i64, + /// Related decoded DEX event id, when available. + pub decoded_event_id: std::option::Option, + /// Related DEX id, when the DEX row is known. + pub dex_id: std::option::Option, + /// Related pool id, when the pool row is known. + pub pool_id: std::option::Option, + /// Related pair id, when the pair row is known. + pub pair_id: std::option::Option, + /// Transaction signature. + pub signature: std::string::String, + /// Optional slot number. + pub slot: std::option::Option, + /// Protocol name that emitted the decoded event. + pub protocol_name: std::string::String, + /// Program id that emitted the decoded event. + pub program_id: std::string::String, + /// Stable decoded event kind. + pub event_kind: std::string::String, + /// Normalized orderbook action. + pub order_action: std::string::String, + /// Pool account address, when decoded. + pub pool_account: std::option::Option, + /// Market/orderbook account, when decoded. + pub market_account: std::option::Option, + /// Wallet or authority associated with the event, when decoded. + pub actor_wallet: std::option::Option, + /// Limit/order account, when decoded. + pub order_account: std::option::Option, + /// Base or token A mint, when decoded. + pub base_token_mint: std::option::Option, + /// Quote or token B mint, when decoded. + pub quote_token_mint: std::option::Option, + /// Raw order amount as decimal text, when decoded. + pub amount_raw: std::option::Option, + /// Raw minimum amount as decimal text, when decoded. + pub amount_min_raw: std::option::Option, + /// Optional tick index for CLMM limit orders. + pub tick_index: std::option::Option, + /// Optional zero-for-one side flag, stored as 0/1. + pub zero_for_one: std::option::Option, + /// Source decoded payload JSON. + pub payload_json: std::string::String, + /// Execution timestamp encoded as RFC3339 UTC text. + pub executed_at: std::string::String, + /// Creation timestamp encoded as RFC3339 UTC text. + pub created_at: std::string::String, +} diff --git a/kb_lib/src/db/queries.rs b/kb_lib/src/db/queries.rs index 85bd70f..1cdb913 100644 --- a/kb_lib/src/db/queries.rs +++ b/kb_lib/src/db/queries.rs @@ -13,9 +13,9 @@ mod dex_decode_replay_ledger; mod dex_decoded_event; mod dex_event_coverage_entry; mod fee_event; +mod instruction_observation; mod known_http_endpoint; mod known_ws_endpoint; -mod instruction_observation; mod launch_attribution; mod launch_surface; mod launch_surface_key; @@ -24,6 +24,7 @@ mod local_dex_corpus_search; mod local_pipeline_diagnostics; mod observed_token; mod onchain_observation; +mod orderbook_event; mod pair; mod pair_analytic_signal; mod pair_candle; @@ -72,8 +73,10 @@ pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_get_by_signatur pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_get_by_transaction; pub use dex_decode_replay_ledger::query_dex_decode_replay_ledger_upsert; pub use dex_decoded_event::query_dex_decoded_events_delete_by_key; +pub use dex_decoded_event::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches; pub use dex_decoded_event::query_dex_decoded_events_delete_meteora_dlmm_anchor_swap_instruction_audits; pub use dex_decoded_event::query_dex_decoded_events_delete_related_instruction_audit; +pub use dex_decoded_event::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits; pub use dex_decoded_event::query_dex_decoded_events_get_by_key; pub use dex_decoded_event::query_dex_decoded_events_get_latest_pump_fun_create_payload_by_mint; pub use dex_decoded_event::query_dex_decoded_events_list_by_transaction_id; @@ -87,14 +90,14 @@ pub use dex_event_coverage_entry::query_dex_event_coverage_entries_upsert; pub use fee_event::query_fee_events_get_by_decoded_event_id; pub use fee_event::query_fee_events_list_recent; pub use fee_event::query_fee_events_upsert; +pub use instruction_observation::query_instruction_observations_list_by_filter; +pub use instruction_observation::query_instruction_observations_upsert; pub use known_http_endpoint::query_known_http_endpoints_get; pub use known_http_endpoint::query_known_http_endpoints_list; pub use known_http_endpoint::query_known_http_endpoints_upsert; pub use known_ws_endpoint::query_known_ws_endpoints_get; pub use known_ws_endpoint::query_known_ws_endpoints_list; pub use known_ws_endpoint::query_known_ws_endpoints_upsert; -pub use instruction_observation::query_instruction_observations_list_by_filter; -pub use instruction_observation::query_instruction_observations_upsert; pub use launch_attribution::query_launch_attributions_get_by_decoded_event_id; pub use launch_attribution::query_launch_attributions_list_by_pool_id; pub use launch_attribution::query_launch_attributions_upsert; @@ -133,6 +136,7 @@ pub use observed_token::query_observed_tokens_list; pub use observed_token::query_observed_tokens_upsert; pub use onchain_observation::query_onchain_observations_insert; pub use onchain_observation::query_onchain_observations_list_recent; +pub use orderbook_event::query_orderbook_events_upsert; pub use pair::query_pairs_get_by_pool_id; pub use pair::query_pairs_list; pub use pair::query_pairs_update_symbol; diff --git a/kb_lib/src/db/queries/dex_decoded_event.rs b/kb_lib/src/db/queries/dex_decoded_event.rs index e190b91..b6be598 100644 --- a/kb_lib/src/db/queries/dex_decoded_event.rs +++ b/kb_lib/src/db/queries/dex_decoded_event.rs @@ -191,6 +191,185 @@ WHERE transaction_id = ? } } +/// Deletes Raydium CLMM instruction-audit rows for locally mapped CLMM instructions. +/// +/// The CLMM specialized decoder now emits named `raydium_clmm.*` rows for all +/// locally mapped instruction discriminants. Keeping the former +/// `raydium_clmm.instruction_audit` rows for the same discriminants creates +/// duplicate coverage and makes residual audit diagnostics noisy. Unknown +/// discriminants remain untouched because they are intentionally absent from +/// this allow-list. +pub async fn query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits( + database: &crate::Database, + transaction_id: std::option::Option, +) -> Result { + match database.connection() { + crate::DatabaseConnection::Sqlite(pool) => { + let query_result = sqlx::query( + r#" +DELETE FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND (? IS NULL OR transaction_id = ?) + AND COALESCE( + json_extract(payload_json, '$.discriminatorHex'), + json_extract(payload_json, '$.discriminator_hex'), + json_extract(payload_json, '$.instructionDiscriminatorHex'), + json_extract(payload_json, '$.instruction_discriminator_hex') + ) IN ( + '4c7c800fd55725fa', + '9d20dab7471d1293', + 'b19059ecfaba7d63', + '759d3c674231a300', + '7b86510031446262', + 'c975989055556cb2', + 'a78a4e95dfc2067e', + '8888fcddc2427e59', + '12eda6c52210d590', + '8934edd4d7756c68', + '2b44d4a7592fa401', + 'bd0eb5785576e33e', + '3f5794216d230868', + 'e992d18ecf6840bc', + '11fb415c88f20ea9', + 'a026d06f685b2c01', + '3a7fbc3e4f52c460', + '2e9cf3760dcdfbb2', + '851d59df45eeb00a', + '5f87c0c4f281e644', + '87802f4d0f98f031', + '4db84ad67056f1c7', + '4dffae527d1dc92e', + '7034a74b20c9d389', + 'cd4e74215c691a60', + 'f8c69e91e17587c8', + '457d73daf5baf2c4', + '2b04ed0b1ac91e62', + '07160c53f22b3079', + '313cae889a1c74c8', + '7f467728bce33d07', + '82576c062ee0757b', + 'a3ace0340b9a6adf' + ) + "#, + ) + .bind(transaction_id) + .bind(transaction_id) + .execute(pool) + .await; + match query_result { + Ok(result) => return Ok(result.rows_affected()), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot delete mapped Raydium CLMM instruction audit events on sqlite: {}", + error + ))); + }, + } + }, + } +} + +/// Deletes upstream registry instruction-match rows already covered by specialized local decoders. +/// +/// The upstream registry fallback is useful only while an instruction is not yet +/// handled by a protocol-specific decoder. Once `k_sol_dex_event_coverage_entries` +/// has a non-empty `local_event_kind` for the same decoder/entry/discriminator, +/// the fallback row is redundant and must be removed so coverage queries do not +/// report both `upstream_git.instruction_match` and the local `protocol.*` event. +pub async fn query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches( + database: &crate::Database, + upstream_decoder_code: std::option::Option<&str>, +) -> Result { + match database.connection() { + crate::DatabaseConnection::Sqlite(pool) => { + let unlink_result = sqlx::query( + r#" +UPDATE k_sol_instruction_observations +SET decoded_event_id = NULL +WHERE decoded_event_id IN ( + SELECT de.id + FROM k_sol_dex_decoded_events de + WHERE de.protocol_name = 'upstream_git' + AND de.event_kind = 'upstream_git.instruction_match' + AND (? IS NULL OR json_extract(de.payload_json, '$.upstreamDecoderCode') = ?) + AND ( + COALESCE(json_extract(de.payload_json, '$.upstreamDecoderCode'), '') + || '|' + || COALESCE(json_extract(de.payload_json, '$.upstreamEntryName'), '') + || '|' + || COALESCE(json_extract(de.payload_json, '$.upstreamDiscriminatorHex'), '') + ) IN ( + SELECT + ce.decoder_code + || '|' + || ce.entry_name + || '|' + || ce.discriminator_hex + FROM k_sol_dex_event_coverage_entries ce + WHERE ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' + AND ce.discriminator_hex IS NOT NULL + AND ce.discriminator_hex <> '' + ) +) + "#, + ) + .bind(upstream_decoder_code) + .bind(upstream_decoder_code) + .execute(pool) + .await; + if let Err(error) = unlink_result { + return Err(crate::Error::Db(format!( + "cannot unlink locally covered upstream instruction matches from instruction observations on sqlite: {}", + error + ))); + } + + let query_result = sqlx::query( + r#" +DELETE FROM k_sol_dex_decoded_events +WHERE protocol_name = 'upstream_git' + AND event_kind = 'upstream_git.instruction_match' + AND (? IS NULL OR json_extract(payload_json, '$.upstreamDecoderCode') = ?) + AND ( + COALESCE(json_extract(payload_json, '$.upstreamDecoderCode'), '') + || '|' + || COALESCE(json_extract(payload_json, '$.upstreamEntryName'), '') + || '|' + || COALESCE(json_extract(payload_json, '$.upstreamDiscriminatorHex'), '') + ) IN ( + SELECT + ce.decoder_code + || '|' + || ce.entry_name + || '|' + || ce.discriminator_hex + FROM k_sol_dex_event_coverage_entries ce + WHERE ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' + AND ce.discriminator_hex IS NOT NULL + AND ce.discriminator_hex <> '' + ) + "#, + ) + .bind(upstream_decoder_code) + .bind(upstream_decoder_code) + .execute(pool) + .await; + match query_result { + Ok(result) => return Ok(result.rows_affected()), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot delete locally covered upstream instruction matches on sqlite: {}", + error + ))); + }, + } + }, + } +} + /// Deletes Meteora DLMM Anchor self-CPI swap audit rows already covered by decoded swaps. /// /// This targets only local-corpus-observed Anchor event discriminators that are diff --git a/kb_lib/src/db/queries/dex_event_coverage_entry.rs b/kb_lib/src/db/queries/dex_event_coverage_entry.rs index a774a4b..1077f52 100644 --- a/kb_lib/src/db/queries/dex_event_coverage_entry.rs +++ b/kb_lib/src/db/queries/dex_event_coverage_entry.rs @@ -569,6 +569,41 @@ SET ) ) ) + WHEN expected_db_target = 'k_sol_orderbook_events' THEN ( + SELECT COUNT(oe.id) + FROM k_sol_dex_decoded_events de + JOIN k_sol_orderbook_events oe ON oe.decoded_event_id = de.id + WHERE ( + (k_sol_dex_event_coverage_entries.program_id IS NULL OR de.program_id = k_sol_dex_event_coverage_entries.program_id) + AND ( + ( + k_sol_dex_event_coverage_entries.local_event_kind IS NOT NULL + AND k_sol_dex_event_coverage_entries.local_event_kind <> '' + AND de.event_kind = k_sol_dex_event_coverage_entries.local_event_kind + ) + OR ( + k_sol_dex_event_coverage_entries.entry_name IS NOT NULL + AND ( + json_extract(de.payload_json, '$.upstreamEntryName') = k_sol_dex_event_coverage_entries.entry_name + OR json_extract(de.payload_json, '$.upstreamInstructionName') = k_sol_dex_event_coverage_entries.entry_name + OR json_extract(de.payload_json, '$.upstreamEventName') = k_sol_dex_event_coverage_entries.entry_name + OR json_extract(de.payload_json, '$.entryName') = k_sol_dex_event_coverage_entries.entry_name + ) + ) + OR ( + k_sol_dex_event_coverage_entries.discriminator_hex IS NOT NULL + AND k_sol_dex_event_coverage_entries.discriminator_hex <> '' + AND ( + json_extract(de.payload_json, '$.upstreamDiscriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex + OR json_extract(de.payload_json, '$.instructionDiscriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex + OR json_extract(de.payload_json, '$.anchorEventDiscriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex + OR json_extract(de.payload_json, '$.anchorEventDiscriminator') = k_sol_dex_event_coverage_entries.discriminator_hex + OR json_extract(de.payload_json, '$.discriminatorHex') = k_sol_dex_event_coverage_entries.discriminator_hex + ) + ) + ) + ) + ) ELSE materialized_count END, first_signature = ( diff --git a/kb_lib/src/db/queries/orderbook_event.rs b/kb_lib/src/db/queries/orderbook_event.rs new file mode 100644 index 0000000..2a9d36e --- /dev/null +++ b/kb_lib/src/db/queries/orderbook_event.rs @@ -0,0 +1,212 @@ +// file: kb_lib/src/db/queries/orderbook_event.rs + +//! Queries for `k_sol_orderbook_events`. + +/// Inserts or updates one normalized orderbook event row. +pub async fn query_orderbook_events_upsert( + database: &crate::Database, + dto: &crate::OrderbookEventDto, +) -> Result { + let slot_i64 = match dto.slot { + Some(slot) => { + let slot_result = i64::try_from(slot); + match slot_result { + Ok(slot) => Some(slot), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot convert orderbook event slot '{}' to i64: {}", + slot, error + ))); + }, + } + }, + None => None, + }; + let zero_for_one_i64 = match dto.zero_for_one { + Some(true) => Some(1_i64), + Some(false) => Some(0_i64), + None => None, + }; + match database.connection() { + crate::DatabaseConnection::Sqlite(pool) => { + let existing_id = match dto.decoded_event_id { + Some(decoded_event_id) => { + let existing_result = sqlx::query_scalar::( + r#" +SELECT id +FROM k_sol_orderbook_events +WHERE decoded_event_id = ? +LIMIT 1 + "#, + ) + .bind(decoded_event_id) + .fetch_optional(pool) + .await; + match existing_result { + Ok(existing_id) => existing_id, + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot fetch k_sol_orderbook_events id for decoded_event_id '{}' on sqlite: {}", + decoded_event_id, error + ))); + }, + } + }, + None => None, + }; + if let Some(id) = existing_id { + let update_result = sqlx::query( + r#" +UPDATE k_sol_orderbook_events +SET + transaction_id = ?, + dex_id = ?, + pool_id = ?, + pair_id = ?, + signature = ?, + slot = ?, + protocol_name = ?, + program_id = ?, + event_kind = ?, + order_action = ?, + pool_account = ?, + market_account = ?, + actor_wallet = ?, + order_account = ?, + base_token_mint = ?, + quote_token_mint = ?, + amount_raw = ?, + amount_min_raw = ?, + tick_index = ?, + zero_for_one = ?, + payload_json = ?, + executed_at = ? +WHERE id = ? + "#, + ) + .bind(dto.transaction_id) + .bind(dto.dex_id) + .bind(dto.pool_id) + .bind(dto.pair_id) + .bind(dto.signature.clone()) + .bind(slot_i64) + .bind(dto.protocol_name.clone()) + .bind(dto.program_id.clone()) + .bind(dto.event_kind.clone()) + .bind(dto.order_action.clone()) + .bind(dto.pool_account.clone()) + .bind(dto.market_account.clone()) + .bind(dto.actor_wallet.clone()) + .bind(dto.order_account.clone()) + .bind(dto.base_token_mint.clone()) + .bind(dto.quote_token_mint.clone()) + .bind(dto.amount_raw.clone()) + .bind(dto.amount_min_raw.clone()) + .bind(dto.tick_index) + .bind(zero_for_one_i64) + .bind(dto.payload_json.clone()) + .bind(dto.executed_at.to_rfc3339()) + .bind(id) + .execute(pool) + .await; + if let Err(error) = update_result { + return Err(crate::Error::Db(format!( + "cannot update k_sol_orderbook_events id '{}' on sqlite: {}", + id, error + ))); + } + return Ok(id); + } + let insert_result = sqlx::query( + r#" +INSERT INTO k_sol_orderbook_events ( + transaction_id, + decoded_event_id, + dex_id, + pool_id, + pair_id, + signature, + slot, + protocol_name, + program_id, + event_kind, + order_action, + pool_account, + market_account, + actor_wallet, + order_account, + base_token_mint, + quote_token_mint, + amount_raw, + amount_min_raw, + tick_index, + zero_for_one, + payload_json, + executed_at, + created_at +) +VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + "#, + ) + .bind(dto.transaction_id) + .bind(dto.decoded_event_id) + .bind(dto.dex_id) + .bind(dto.pool_id) + .bind(dto.pair_id) + .bind(dto.signature.clone()) + .bind(slot_i64) + .bind(dto.protocol_name.clone()) + .bind(dto.program_id.clone()) + .bind(dto.event_kind.clone()) + .bind(dto.order_action.clone()) + .bind(dto.pool_account.clone()) + .bind(dto.market_account.clone()) + .bind(dto.actor_wallet.clone()) + .bind(dto.order_account.clone()) + .bind(dto.base_token_mint.clone()) + .bind(dto.quote_token_mint.clone()) + .bind(dto.amount_raw.clone()) + .bind(dto.amount_min_raw.clone()) + .bind(dto.tick_index) + .bind(zero_for_one_i64) + .bind(dto.payload_json.clone()) + .bind(dto.executed_at.to_rfc3339()) + .bind(dto.created_at.to_rfc3339()) + .execute(pool) + .await; + if let Err(error) = insert_result { + return Err(crate::Error::Db(format!( + "cannot insert k_sol_orderbook_events on sqlite: {}", + error + ))); + } + let id_result = sqlx::query_scalar::( + r#" +SELECT id +FROM k_sol_orderbook_events +WHERE transaction_id = ? + AND protocol_name = ? + AND event_kind = ? + AND signature = ? +ORDER BY id DESC +LIMIT 1 + "#, + ) + .bind(dto.transaction_id) + .bind(dto.protocol_name.clone()) + .bind(dto.event_kind.clone()) + .bind(dto.signature.clone()) + .fetch_one(pool) + .await; + match id_result { + Ok(id) => return Ok(id), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot fetch inserted k_sol_orderbook_events id for signature '{}' on sqlite: {}", + dto.signature, error + ))); + }, + } + }, + } +} diff --git a/kb_lib/src/db/queries/trade_event.rs b/kb_lib/src/db/queries/trade_event.rs index 0050c95..7841276 100644 --- a/kb_lib/src/db/queries/trade_event.rs +++ b/kb_lib/src/db/queries/trade_event.rs @@ -16,7 +16,67 @@ pub async fn query_trade_market_materialization_delete_all( ("k_sol_pair_analytic_signals", "DELETE FROM k_sol_pair_analytic_signals"), ("k_sol_pair_candles", "DELETE FROM k_sol_pair_candles"), ("k_sol_pair_metrics", "DELETE FROM k_sol_pair_metrics"), + ("k_sol_orderbook_events", "DELETE FROM k_sol_orderbook_events"), ("k_sol_trade_events", "DELETE FROM k_sol_trade_events"), + ( + "locally_covered_upstream_instruction_observation_links", + r#" +UPDATE k_sol_instruction_observations +SET decoded_event_id = NULL +WHERE decoded_event_id IN ( + SELECT de.id + FROM k_sol_dex_decoded_events de + WHERE de.protocol_name = 'upstream_git' + AND de.event_kind = 'upstream_git.instruction_match' + AND ( + COALESCE(json_extract(de.payload_json, '$.upstreamDecoderCode'), '') + || '|' + || COALESCE(json_extract(de.payload_json, '$.upstreamEntryName'), '') + || '|' + || COALESCE(json_extract(de.payload_json, '$.upstreamDiscriminatorHex'), '') + ) IN ( + SELECT + ce.decoder_code + || '|' + || ce.entry_name + || '|' + || ce.discriminator_hex + FROM k_sol_dex_event_coverage_entries ce + WHERE ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' + AND ce.discriminator_hex IS NOT NULL + AND ce.discriminator_hex <> '' + ) +) + "#, + ), + ( + "locally_covered_upstream_instruction_matches", + r#" +DELETE FROM k_sol_dex_decoded_events +WHERE protocol_name = 'upstream_git' + AND event_kind = 'upstream_git.instruction_match' + AND ( + COALESCE(json_extract(payload_json, '$.upstreamDecoderCode'), '') + || '|' + || COALESCE(json_extract(payload_json, '$.upstreamEntryName'), '') + || '|' + || COALESCE(json_extract(payload_json, '$.upstreamDiscriminatorHex'), '') + ) IN ( + SELECT + ce.decoder_code + || '|' + || ce.entry_name + || '|' + || ce.discriminator_hex + FROM k_sol_dex_event_coverage_entries ce + WHERE ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' + AND ce.discriminator_hex IS NOT NULL + AND ce.discriminator_hex <> '' + ) + "#, + ), ]; let mut deleted_count = 0_u64; for (table_name, statement) in statements { diff --git a/kb_lib/src/db/schema.rs b/kb_lib/src/db/schema.rs index 8ed1a7f..e01ba08 100644 --- a/kb_lib/src/db/schema.rs +++ b/kb_lib/src/db/schema.rs @@ -374,6 +374,22 @@ pub(crate) async fn ensure_schema(database: &crate::Database) -> Result<(), crat if let Err(error) = result { return Err(error); } + let result = create_tbl_orderbook_events(pool).await; + if let Err(error) = result { + return Err(error); + } + let result = create_idx_orderbook_events_transaction_id(pool).await; + if let Err(error) = result { + return Err(error); + } + let result = create_idx_orderbook_events_pool_id(pool).await; + if let Err(error) = result { + return Err(error); + } + let result = create_uix_orderbook_events_decoded_event_id(pool).await; + if let Err(error) = result { + return Err(error); + } let result = create_tbl_launch_surfaces(pool).await; if let Err(error) = result { return Err(error); @@ -2710,3 +2726,85 @@ WHERE decoded_event_id IS NOT NULL ) .await; } + +/// Creates `k_sol_orderbook_events`. +async fn create_tbl_orderbook_events(pool: &sqlx::SqlitePool) -> Result<(), crate::Error> { + return execute_sqlite_schema_statement( + pool, + "create_tbl_orderbook_events", + r#" +CREATE TABLE IF NOT EXISTS k_sol_orderbook_events ( + id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, + transaction_id INTEGER NOT NULL, + decoded_event_id INTEGER NULL, + dex_id INTEGER NULL, + pool_id INTEGER NULL, + pair_id INTEGER NULL, + signature TEXT NOT NULL, + slot INTEGER NULL, + protocol_name TEXT NOT NULL, + program_id TEXT NOT NULL, + event_kind TEXT NOT NULL, + order_action TEXT NOT NULL, + pool_account TEXT NULL, + market_account TEXT NULL, + actor_wallet TEXT NULL, + order_account TEXT NULL, + base_token_mint TEXT NULL, + quote_token_mint TEXT NULL, + amount_raw TEXT NULL, + amount_min_raw TEXT NULL, + tick_index INTEGER NULL, + zero_for_one INTEGER NULL, + payload_json TEXT NOT NULL, + executed_at TEXT NOT NULL, + created_at TEXT NOT NULL +) + "#, + ) + .await; +} + +/// Creates index on `k_sol_orderbook_events(transaction_id)`. +async fn create_idx_orderbook_events_transaction_id( + pool: &sqlx::SqlitePool, +) -> Result<(), crate::Error> { + return execute_sqlite_schema_statement( + pool, + "create_idx_orderbook_events_transaction_id", + r#" +CREATE INDEX IF NOT EXISTS idx_orderbook_events_transaction_id +ON k_sol_orderbook_events (transaction_id) + "#, + ) + .await; +} + +/// Creates index on `k_sol_orderbook_events(pool_id)`. +async fn create_idx_orderbook_events_pool_id(pool: &sqlx::SqlitePool) -> Result<(), crate::Error> { + return execute_sqlite_schema_statement( + pool, + "create_idx_orderbook_events_pool_id", + r#" +CREATE INDEX IF NOT EXISTS idx_orderbook_events_pool_id +ON k_sol_orderbook_events (pool_id) + "#, + ) + .await; +} + +/// Creates partial unique index on `k_sol_orderbook_events(decoded_event_id)`. +async fn create_uix_orderbook_events_decoded_event_id( + pool: &sqlx::SqlitePool, +) -> Result<(), crate::Error> { + return execute_sqlite_schema_statement( + pool, + "create_uix_orderbook_events_decoded_event_id", + r#" +CREATE UNIQUE INDEX IF NOT EXISTS uix_orderbook_events_decoded_event_id +ON k_sol_orderbook_events (decoded_event_id) +WHERE decoded_event_id IS NOT NULL + "#, + ) + .await; +} diff --git a/kb_lib/src/dex.rs b/kb_lib/src/dex.rs index 2ab0831..346e53a 100644 --- a/kb_lib/src/dex.rs +++ b/kb_lib/src/dex.rs @@ -70,12 +70,16 @@ pub use raydium_amm_v4::RaydiumAmmV4DecodedEvent; pub use raydium_amm_v4::RaydiumAmmV4Decoder; pub use raydium_amm_v4::RaydiumAmmV4Initialize2PoolDecoded; pub use raydium_amm_v4::RaydiumAmmV4SwapDecoded; +pub use raydium_clmm::RaydiumClmmCollectProtocolFeeDecoded; +pub use raydium_clmm::RaydiumClmmCreatePoolDecoded; pub use raydium_clmm::RaydiumClmmDecodedEvent; pub use raydium_clmm::RaydiumClmmDecodedInstructionEvent; +pub use raydium_clmm::RaydiumClmmProgramDataEventDecoded; pub use raydium_clmm::RaydiumClmmDecoder; pub use raydium_clmm::RaydiumClmmSwapLegacyDecoded; pub use raydium_clmm::RaydiumClmmSwapV2Decoded; pub use raydium_clmm::decode_raydium_clmm_instruction; +pub use raydium_clmm::decode_raydium_clmm_program_data_event; pub use raydium_cpmm::RaydiumCpmmDecodedEvent; pub use raydium_cpmm::RaydiumCpmmLpChangeEventDecoded; pub use raydium_cpmm::RaydiumCpmmSwapDecoded; diff --git a/kb_lib/src/dex/raydium_clmm.rs b/kb_lib/src/dex/raydium_clmm.rs index 16538fb..9905967 100644 --- a/kb_lib/src/dex/raydium_clmm.rs +++ b/kb_lib/src/dex/raydium_clmm.rs @@ -6,15 +6,69 @@ const RAYDIUM_CLMM_SWAP_V2_DISCRIMINATOR: [u8; 8] = [43, 4, 237, 11, 26, 201, 30 const RAYDIUM_CLMM_SWAP_LEGACY_DISCRIMINATOR: [u8; 8] = [248, 198, 158, 145, 225, 117, 135, 200]; +const RAYDIUM_CLMM_CREATE_POOL_DISCRIMINATOR: [u8; 8] = [233, 146, 209, 142, 207, 104, 64, 188]; + +const RAYDIUM_CLMM_COLLECT_PROTOCOL_FEE_DISCRIMINATOR: [u8; 8] = + [136, 136, 252, 221, 194, 66, 126, 89]; + +const RAYDIUM_CLMM_ANCHOR_SELF_CPI_LOG_SELECTOR: [u8; 8] = [228, 69, 165, 46, 81, 203, 154, 29]; + +const RAYDIUM_CLMM_COLLECT_PERSONAL_FEE_EVENT_DISCRIMINATOR: [u8; 8] = + [166, 174, 105, 192, 81, 161, 83, 105]; +const RAYDIUM_CLMM_COLLECT_PROTOCOL_FEE_EVENT_DISCRIMINATOR: [u8; 8] = + [206, 87, 17, 79, 45, 41, 213, 61]; +const RAYDIUM_CLMM_CONFIG_CHANGE_EVENT_DISCRIMINATOR: [u8; 8] = + [247, 189, 7, 119, 106, 112, 95, 151]; +const RAYDIUM_CLMM_CREATE_PERSONAL_POSITION_EVENT_DISCRIMINATOR: [u8; 8] = + [100, 30, 87, 249, 196, 223, 154, 206]; +const RAYDIUM_CLMM_DECREASE_LIQUIDITY_EVENT_DISCRIMINATOR: [u8; 8] = + [58, 222, 86, 58, 68, 50, 85, 56]; +const RAYDIUM_CLMM_INCREASE_LIQUIDITY_EVENT_DISCRIMINATOR: [u8; 8] = + [49, 79, 105, 212, 32, 34, 30, 84]; +const RAYDIUM_CLMM_LIQUIDITY_CALCULATE_EVENT_DISCRIMINATOR: [u8; 8] = + [237, 112, 148, 230, 57, 84, 180, 162]; +const RAYDIUM_CLMM_LIQUIDITY_CHANGE_EVENT_DISCRIMINATOR: [u8; 8] = + [126, 240, 175, 206, 158, 88, 153, 107]; +const RAYDIUM_CLMM_POOL_CREATED_EVENT_DISCRIMINATOR: [u8; 8] = [25, 94, 75, 47, 112, 99, 53, 63]; +const RAYDIUM_CLMM_SWAP_EVENT_DISCRIMINATOR: [u8; 8] = [64, 198, 205, 232, 38, 8, 113, 226]; +const RAYDIUM_CLMM_UPDATE_REWARD_INFOS_EVENT_DISCRIMINATOR: [u8; 8] = + [109, 127, 186, 78, 114, 65, 37, 236]; + const OBSERVED_JUPITER_V6_PROGRAM_ID: &str = "JUP6LkbZbjS1jKKwapdHNy74zcZ3tLUZoi5QNyVTaV4"; /// Decoded Raydium CLMM event. #[derive(Clone, Debug, serde::Serialize, serde::Deserialize, PartialEq)] pub enum RaydiumClmmDecodedEvent { - /// Raydium CLMM legacy swap event. + /// Raydium CLMM legacy swap instruction. Swap(crate::RaydiumClmmSwapLegacyDecoded), - /// Raydium CLMM swap_v2 event. + /// Raydium CLMM swap_v2 instruction. SwapV2(crate::RaydiumClmmSwapV2Decoded), + /// Raydium CLMM create_pool instruction. + CreatePool(crate::RaydiumClmmCreatePoolDecoded), + /// Raydium CLMM collect_protocol_fee instruction. + CollectProtocolFee(crate::RaydiumClmmCollectProtocolFeeDecoded), + /// Raydium CLMM CollectPersonalFeeEvent `Program data` log. + CollectPersonalFeeEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM CollectProtocolFeeEvent `Program data` log. + CollectProtocolFeeEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM ConfigChangeEvent `Program data` log. + ConfigChangeEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM CreatePersonalPositionEvent `Program data` log. + CreatePersonalPositionEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM DecreaseLiquidityEvent `Program data` log. + DecreaseLiquidityEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM IncreaseLiquidityEvent `Program data` log. + IncreaseLiquidityEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM LiquidityCalculateEvent `Program data` log. + LiquidityCalculateEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM LiquidityChangeEvent `Program data` log. + LiquidityChangeEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM PoolCreatedEvent `Program data` log. + PoolCreatedEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM SwapEvent `Program data` log retained as audit evidence. + SwapEvent(crate::RaydiumClmmProgramDataEventDecoded), + /// Raydium CLMM UpdateRewardInfosEvent `Program data` log. + UpdateRewardInfosEvent(crate::RaydiumClmmProgramDataEventDecoded), } impl RaydiumClmmDecodedEvent { @@ -23,49 +77,201 @@ impl RaydiumClmmDecodedEvent { match self { crate::RaydiumClmmDecodedEvent::Swap(_) => return "raydium_clmm.swap", crate::RaydiumClmmDecodedEvent::SwapV2(_) => return "raydium_clmm.swap_v2", + crate::RaydiumClmmDecodedEvent::CreatePool(_) => return "raydium_clmm.create_pool", + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(_) => { + return "raydium_clmm.collect_protocol_fee"; + }, + crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(_) => { + return "raydium_clmm.collect_personal_fee_event"; + }, + crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(_) => { + return "raydium_clmm.collect_protocol_fee_event"; + }, + crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(_) => { + return "raydium_clmm.config_change_event"; + }, + crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(_) => { + return "raydium_clmm.create_personal_position_event"; + }, + crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(_) => { + return "raydium_clmm.decrease_liquidity_event"; + }, + crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(_) => { + return "raydium_clmm.increase_liquidity_event"; + }, + crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(_) => { + return "raydium_clmm.liquidity_calculate_event"; + }, + crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(_) => { + return "raydium_clmm.liquidity_change_event"; + }, + crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(_) => { + return "raydium_clmm.pool_created_event"; + }, + crate::RaydiumClmmDecodedEvent::SwapEvent(_) => return "raydium_clmm.swap_event", + crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(_) => { + return "raydium_clmm.update_reward_infos_event"; + }, } } - /// Returns the pool account. + /// Returns the pool account when the decoded payload exposes one. + pub fn pool_account_option(&self) -> std::option::Option<&str> { + match self { + crate::RaydiumClmmDecodedEvent::Swap(event) => return Some(event.pool_state.as_str()), + crate::RaydiumClmmDecodedEvent::SwapV2(event) => { + return Some(event.pool_state.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CreatePool(event) => { + return Some(event.pool_state.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event) => { + return Some(event.pool_state.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(event) + | crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(event) + | crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(event) + | crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(event) + | crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(event) + | crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(event) + | crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(event) + | crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(event) + | crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(event) + | crate::RaydiumClmmDecodedEvent::SwapEvent(event) + | crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(event) => { + match event.pool_state.as_ref() { + Some(pool_state) => return Some(pool_state.as_str()), + None => return None, + } + }, + } + } + + /// Returns the pool account, or an empty string for audit events without pool scope. pub fn pool_account(&self) -> &str { - match self { - crate::RaydiumClmmDecodedEvent::Swap(event) => return event.pool_state.as_str(), - crate::RaydiumClmmDecodedEvent::SwapV2(event) => return event.pool_state.as_str(), + match self.pool_account_option() { + Some(pool_account) => return pool_account, + None => return "", } } - /// Returns the normalized base mint. + /// Returns the normalized base mint when known. + pub fn base_mint_option(&self) -> std::option::Option<&str> { + match self { + crate::RaydiumClmmDecodedEvent::Swap(event) => return Some(event.base_mint.as_str()), + crate::RaydiumClmmDecodedEvent::SwapV2(event) => return Some(event.base_mint.as_str()), + crate::RaydiumClmmDecodedEvent::CreatePool(event) => { + return Some(event.token_mint_0.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event) => { + return Some(event.vault_0_mint.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(event) + | crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(event) + | crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(event) + | crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(event) + | crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(event) + | crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(event) + | crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(event) + | crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(event) + | crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(event) + | crate::RaydiumClmmDecodedEvent::SwapEvent(event) + | crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(event) => { + match event.token_mint0.as_ref() { + Some(token_mint0) => return Some(token_mint0.as_str()), + None => return None, + } + }, + } + } + + /// Returns the normalized base mint, or an empty string for audit events without mint scope. pub fn base_mint(&self) -> &str { - match self { - crate::RaydiumClmmDecodedEvent::Swap(event) => return event.base_mint.as_str(), - crate::RaydiumClmmDecodedEvent::SwapV2(event) => return event.base_mint.as_str(), + match self.base_mint_option() { + Some(base_mint) => return base_mint, + None => return "", } } - /// Returns the normalized quote mint. - pub fn quote_mint(&self) -> &str { + /// Returns the normalized quote mint when known. + pub fn quote_mint_option(&self) -> std::option::Option<&str> { match self { - crate::RaydiumClmmDecodedEvent::Swap(event) => return event.quote_mint.as_str(), - crate::RaydiumClmmDecodedEvent::SwapV2(event) => return event.quote_mint.as_str(), + crate::RaydiumClmmDecodedEvent::Swap(event) => return Some(event.quote_mint.as_str()), + crate::RaydiumClmmDecodedEvent::SwapV2(event) => { + return Some(event.quote_mint.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CreatePool(event) => { + return Some(event.token_mint_1.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event) => { + return Some(event.vault_1_mint.as_str()); + }, + crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(event) + | crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(event) + | crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(event) + | crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(event) + | crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(event) + | crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(event) + | crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(event) + | crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(event) + | crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(event) + | crate::RaydiumClmmDecodedEvent::SwapEvent(event) + | crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(event) => { + match event.token_mint1.as_ref() { + Some(token_mint1) => return Some(token_mint1.as_str()), + None => return None, + } + }, + } + } + + /// Returns the normalized quote mint, or an empty string for audit events without mint scope. + pub fn quote_mint(&self) -> &str { + match self.quote_mint_option() { + Some(quote_mint) => return quote_mint, + None => return "", } } /// Converts the decoded event to JSON payload. pub fn to_payload_json(&self) -> std::option::Option { match self { - crate::RaydiumClmmDecodedEvent::Swap(event) => { - let result = serde_json::to_string(event); - match result { - Ok(payload_json) => return Some(payload_json), - Err(_) => return None, - } + crate::RaydiumClmmDecodedEvent::Swap(event) => return serde_json_string(event), + crate::RaydiumClmmDecodedEvent::SwapV2(event) => return serde_json_string(event), + crate::RaydiumClmmDecodedEvent::CreatePool(event) => return serde_json_string(event), + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event) => { + return serde_json_string(event); }, - crate::RaydiumClmmDecodedEvent::SwapV2(event) => { - let result = serde_json::to_string(event); - match result { - Ok(payload_json) => return Some(payload_json), - Err(_) => return None, - } + crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(event) => { + return serde_json_string(event); + }, + crate::RaydiumClmmDecodedEvent::SwapEvent(event) => return serde_json_string(event), + crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(event) => { + return serde_json_string(event); }, } } @@ -80,6 +286,112 @@ pub struct RaydiumClmmDecodedInstructionEvent { pub decoded_event: crate::RaydiumClmmDecodedEvent, } +/// Decoded Raydium CLMM Anchor `Program data` event. +#[derive(Clone, Debug, serde::Serialize, serde::Deserialize, PartialEq)] +#[serde(rename_all = "camelCase")] +pub struct RaydiumClmmProgramDataEventDecoded { + /// Upstream Anchor event name. + pub event_name: std::string::String, + /// Event discriminator in lowercase hexadecimal. + pub event_discriminator_hex: std::string::String, + /// Whether this was decoded from a `Program data:` log. + pub program_data_log: bool, + /// CLMM pool account when present in the event layout. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub pool_state: std::option::Option, + /// Position NFT mint when present in the event layout. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub position_nft_mint: std::option::Option, + /// Event sender/minter/owner authority when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub actor_wallet: std::option::Option, + /// First token mint when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub token_mint0: std::option::Option, + /// Second token mint when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub token_mint1: std::option::Option, + /// First token account when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub token_account0: std::option::Option, + /// Second token account when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub token_account1: std::option::Option, + /// First token vault when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub token_vault0: std::option::Option, + /// Second token vault when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub token_vault1: std::option::Option, + /// Lower tick index when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub tick_lower_index: std::option::Option, + /// Upper tick index when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub tick_upper_index: std::option::Option, + /// Current or initial tick when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub tick: std::option::Option, + /// CLMM tick spacing when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub tick_spacing: std::option::Option, + /// Liquidity value as decimal string when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub liquidity_raw: std::option::Option, + /// Pool liquidity before an event, as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub liquidity_before_raw: std::option::Option, + /// Pool liquidity after an event, as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub liquidity_after_raw: std::option::Option, + /// Sqrt price Q64.64 as decimal string when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub sqrt_price_x64: std::option::Option, + /// First token amount as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub amount0_raw: std::option::Option, + /// Second token amount as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub amount1_raw: std::option::Option, + /// First token transfer fee as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub transfer_fee0_raw: std::option::Option, + /// Second token transfer fee as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub transfer_fee1_raw: std::option::Option, + /// First fee amount as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub fee_amount0_raw: std::option::Option, + /// Second fee amount as decimal string. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub fee_amount1_raw: std::option::Option, + /// Reward growth values as decimal strings. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub reward_growth_global_x64: std::option::Option>, + /// Reward amounts as decimal strings. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub reward_amounts_raw: std::option::Option>, + /// Zero-for-one swap flag when present. + #[serde(skip_serializing_if = "std::option::Option::is_none")] + pub zero_for_one: std::option::Option, + /// Explicitly prevents Anchor audit events from double-counting instruction swaps. + pub trade_candidate: bool, + /// Explicitly prevents Anchor audit events from double-counting instruction candles. + pub candle_candidate: bool, + /// Trade skip reason for audit-only Program data events. + pub skip_trade_reason: std::string::String, + /// Candle skip reason for audit-only Program data events. + pub skip_candle_reason: std::string::String, +} + +fn serde_json_string(value: &T) -> std::option::Option { + let result = serde_json::to_string(value); + match result { + Ok(payload_json) => return Some(payload_json), + Err(_) => return None, + } +} + /// Decoded Raydium CLMM legacy swap instruction. #[derive(Clone, Debug, serde::Serialize, serde::Deserialize, PartialEq)] pub struct RaydiumClmmSwapLegacyDecoded { @@ -180,6 +492,72 @@ pub struct RaydiumClmmSwapV2Decoded { pub is_base_input: bool, } +/// Decoded Raydium CLMM create_pool instruction. +#[derive(Clone, Debug, serde::Serialize, serde::Deserialize, PartialEq)] +pub struct RaydiumClmmCreatePoolDecoded { + /// Pool creator or payer account. + pub payer: std::string::String, + /// AMM config account. + pub amm_config: std::string::String, + /// CLMM pool state account. + pub pool_state: std::string::String, + /// First pool token mint in the instruction account order. + pub token_mint_0: std::string::String, + /// Second pool token mint in the instruction account order. + pub token_mint_1: std::string::String, + /// First pool vault. + pub token_vault_0: std::string::String, + /// Second pool vault. + pub token_vault_1: std::string::String, + /// Observation state account. + pub observation_state: std::string::String, + /// Initial sqrt price as decimal string. + #[serde(rename = "sqrtPriceX64")] + pub sqrt_price_x64: std::string::String, + /// Pool open time argument. + #[serde(rename = "openTime")] + pub open_time: u64, + /// Instruction discriminator used for audit cleanup and coverage reconciliation. + #[serde(rename = "instructionDiscriminatorHex")] + pub instruction_discriminator_hex: std::string::String, +} + +/// Decoded Raydium CLMM collect_protocol_fee instruction. +#[derive(Clone, Debug, serde::Serialize, serde::Deserialize, PartialEq)] +pub struct RaydiumClmmCollectProtocolFeeDecoded { + /// Fee collection authority. + pub authority: std::string::String, + /// CLMM pool state account. + pub pool_state: std::string::String, + /// First recipient token account. + pub recipient_token_account_0: std::string::String, + /// Second recipient token account. + pub recipient_token_account_1: std::string::String, + /// First pool vault. + pub token_vault_0: std::string::String, + /// Second pool vault. + pub token_vault_1: std::string::String, + /// First pool vault mint. + pub vault_0_mint: std::string::String, + /// Second pool vault mint. + pub vault_1_mint: std::string::String, + /// Requested first token amount. + #[serde(rename = "amount0RequestedRaw")] + pub amount_0_requested_raw: std::string::String, + /// Requested second token amount. + #[serde(rename = "amount1RequestedRaw")] + pub amount_1_requested_raw: std::string::String, + /// Alias used by generic fee materialization. + #[serde(rename = "tokenAAmount")] + pub token_a_amount: std::string::String, + /// Alias used by generic fee materialization. + #[serde(rename = "tokenBAmount")] + pub token_b_amount: std::string::String, + /// Instruction discriminator used for audit cleanup and coverage reconciliation. + #[serde(rename = "instructionDiscriminatorHex")] + pub instruction_discriminator_hex: std::string::String, +} + /// Raydium CLMM transaction decoder. #[derive(Clone, Debug, Default)] pub struct RaydiumClmmDecoder; @@ -275,7 +653,7 @@ pub fn decode_raydium_clmm_instruction( Some(data) => data, None => return decoded, }; - if data.len() < 41 { + if data.len() < 8 { return decoded; } let discriminator_option = read_discriminator(data.as_slice()); @@ -283,6 +661,24 @@ pub fn decode_raydium_clmm_instruction( Some(discriminator) => discriminator, None => return decoded, }; + if discriminator == RAYDIUM_CLMM_CREATE_POOL_DISCRIMINATOR { + let event_option = decode_create_pool(accounts.as_slice(), data.as_slice()); + let event = match event_option { + Some(event) => event, + None => return decoded, + }; + decoded.push(crate::RaydiumClmmDecodedEvent::CreatePool(event)); + return decoded; + } + if discriminator == RAYDIUM_CLMM_COLLECT_PROTOCOL_FEE_DISCRIMINATOR { + let event_option = decode_collect_protocol_fee(accounts.as_slice(), data.as_slice()); + let event = match event_option { + Some(event) => event, + None => return decoded, + }; + decoded.push(crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event)); + return decoded; + } if discriminator == RAYDIUM_CLMM_SWAP_LEGACY_DISCRIMINATOR { return decoded; } @@ -298,6 +694,364 @@ pub fn decode_raydium_clmm_instruction( return decoded; } +/// Decodes one Raydium CLMM Anchor event emitted in a `Program data:` log. +pub fn decode_raydium_clmm_program_data_event( + data_base64: &str, +) -> std::option::Option { + let decoded = decode_base64_standard(data_base64); + let data = match decoded { + Some(data) => data, + None => return None, + }; + let event = decode_raydium_clmm_program_data_event_bytes(data.as_slice()); + if event.is_some() { + return event; + } + if data.len() < 16 { + return None; + } + let selector = read_discriminator(data.as_slice()); + let selector = match selector { + Some(selector) => selector, + None => return None, + }; + if selector != RAYDIUM_CLMM_ANCHOR_SELF_CPI_LOG_SELECTOR { + return None; + } + return decode_raydium_clmm_program_data_event_bytes(&data[8..]); +} + +fn decode_raydium_clmm_program_data_event_bytes( + data: &[u8], +) -> std::option::Option { + let discriminator = match read_discriminator(data) { + Some(discriminator) => discriminator, + None => return None, + }; + if discriminator == RAYDIUM_CLMM_COLLECT_PERSONAL_FEE_EVENT_DISCRIMINATOR { + let event = decode_clmm_collect_personal_fee_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_COLLECT_PROTOCOL_FEE_EVENT_DISCRIMINATOR { + let event = decode_clmm_collect_protocol_fee_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_CONFIG_CHANGE_EVENT_DISCRIMINATOR { + let event = decode_clmm_config_change_event(data); + match event { + Some(event) => return Some(crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(event)), + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_CREATE_PERSONAL_POSITION_EVENT_DISCRIMINATOR { + let event = decode_clmm_create_personal_position_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_DECREASE_LIQUIDITY_EVENT_DISCRIMINATOR { + let event = decode_clmm_decrease_liquidity_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_INCREASE_LIQUIDITY_EVENT_DISCRIMINATOR { + let event = decode_clmm_increase_liquidity_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_LIQUIDITY_CALCULATE_EVENT_DISCRIMINATOR { + let event = decode_clmm_liquidity_calculate_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_LIQUIDITY_CHANGE_EVENT_DISCRIMINATOR { + let event = decode_clmm_liquidity_change_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(event)); + }, + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_POOL_CREATED_EVENT_DISCRIMINATOR { + let event = decode_clmm_pool_created_event(data); + match event { + Some(event) => return Some(crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(event)), + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_SWAP_EVENT_DISCRIMINATOR { + let event = decode_clmm_swap_event(data); + match event { + Some(event) => return Some(crate::RaydiumClmmDecodedEvent::SwapEvent(event)), + None => return None, + } + } + if discriminator == RAYDIUM_CLMM_UPDATE_REWARD_INFOS_EVENT_DISCRIMINATOR { + let event = decode_clmm_update_reward_infos_event(data); + match event { + Some(event) => { + return Some(crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(event)); + }, + None => return None, + } + } + return None; +} + +fn new_clmm_program_data_event( + event_name: &str, + event_discriminator_hex: &str, +) -> crate::RaydiumClmmProgramDataEventDecoded { + return crate::RaydiumClmmProgramDataEventDecoded { + event_name: event_name.to_string(), + event_discriminator_hex: event_discriminator_hex.to_string(), + program_data_log: true, + pool_state: None, + position_nft_mint: None, + actor_wallet: None, + token_mint0: None, + token_mint1: None, + token_account0: None, + token_account1: None, + token_vault0: None, + token_vault1: None, + tick_lower_index: None, + tick_upper_index: None, + tick: None, + tick_spacing: None, + liquidity_raw: None, + liquidity_before_raw: None, + liquidity_after_raw: None, + sqrt_price_x64: None, + amount0_raw: None, + amount1_raw: None, + transfer_fee0_raw: None, + transfer_fee1_raw: None, + fee_amount0_raw: None, + fee_amount1_raw: None, + reward_growth_global_x64: None, + reward_amounts_raw: None, + zero_for_one: None, + trade_candidate: false, + candle_candidate: false, + skip_trade_reason: "raydium_clmm_program_data_event_audit_only".to_string(), + skip_candle_reason: "raydium_clmm_program_data_event_audit_only".to_string(), + }; +} + +fn decode_clmm_collect_personal_fee_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 120 { + return None; + } + let mut event = new_clmm_program_data_event("collect_personal_fee_event", "a6ae69c051a15369"); + event.position_nft_mint = read_pubkey_base58(data, 8); + event.token_account0 = read_pubkey_base58(data, 40); + event.token_account1 = read_pubkey_base58(data, 72); + event.amount0_raw = read_u64_le(data, 104).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 112).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_collect_protocol_fee_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 120 { + return None; + } + let mut event = new_clmm_program_data_event("collect_protocol_fee_event", "ce57114f2d29d53d"); + event.pool_state = read_pubkey_base58(data, 8); + event.token_account0 = read_pubkey_base58(data, 40); + event.token_account1 = read_pubkey_base58(data, 72); + event.amount0_raw = read_u64_le(data, 104).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 112).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_config_change_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 88 { + return None; + } + let mut event = new_clmm_program_data_event("config_change_event", "f7bd07776a705f97"); + event.tick_spacing = read_u16_le(data, 50); + event.actor_wallet = read_pubkey_base58(data, 10); + event.fee_amount0_raw = read_u32_le(data, 42).map(|value| return value.to_string()); + event.fee_amount1_raw = read_u32_le(data, 46).map(|value| return value.to_string()); + event.token_account0 = read_pubkey_base58(data, 56); + return Some(event); +} + +fn decode_clmm_create_personal_position_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 160 { + return None; + } + let mut event = + new_clmm_program_data_event("create_personal_position_event", "641e57f9c4df9ace"); + event.pool_state = read_pubkey_base58(data, 8); + event.actor_wallet = read_pubkey_base58(data, 40); + event.token_account0 = read_pubkey_base58(data, 72); + event.tick_lower_index = read_i32_le(data, 104); + event.tick_upper_index = read_i32_le(data, 108); + event.liquidity_raw = read_u128_le(data, 112).map(|value| return value.to_string()); + event.amount0_raw = read_u64_le(data, 128).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 136).map(|value| return value.to_string()); + event.transfer_fee0_raw = read_u64_le(data, 144).map(|value| return value.to_string()); + event.transfer_fee1_raw = read_u64_le(data, 152).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_decrease_liquidity_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 128 { + return None; + } + let mut event = new_clmm_program_data_event("decrease_liquidity_event", "3ade563a44325538"); + event.position_nft_mint = read_pubkey_base58(data, 8); + event.liquidity_raw = read_u128_le(data, 40).map(|value| return value.to_string()); + event.amount0_raw = read_u64_le(data, 56).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 64).map(|value| return value.to_string()); + event.fee_amount0_raw = read_u64_le(data, 72).map(|value| return value.to_string()); + event.fee_amount1_raw = read_u64_le(data, 80).map(|value| return value.to_string()); + event.reward_amounts_raw = read_u64_vec(data, 88, 3); + event.transfer_fee0_raw = read_u64_le(data, 112).map(|value| return value.to_string()); + event.transfer_fee1_raw = read_u64_le(data, 120).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_increase_liquidity_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 88 { + return None; + } + let mut event = new_clmm_program_data_event("increase_liquidity_event", "314f69d420221e54"); + event.position_nft_mint = read_pubkey_base58(data, 8); + event.liquidity_raw = read_u128_le(data, 40).map(|value| return value.to_string()); + event.amount0_raw = read_u64_le(data, 56).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 64).map(|value| return value.to_string()); + event.transfer_fee0_raw = read_u64_le(data, 72).map(|value| return value.to_string()); + event.transfer_fee1_raw = read_u64_le(data, 80).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_liquidity_calculate_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 92 { + return None; + } + let mut event = new_clmm_program_data_event("liquidity_calculate_event", "ed7094e63954b4a2"); + event.liquidity_before_raw = read_u128_le(data, 8).map(|value| return value.to_string()); + event.sqrt_price_x64 = read_u128_le(data, 24).map(|value| return value.to_string()); + event.tick = read_i32_le(data, 40); + event.amount0_raw = read_u64_le(data, 44).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 52).map(|value| return value.to_string()); + event.fee_amount0_raw = read_u64_le(data, 60).map(|value| return value.to_string()); + event.fee_amount1_raw = read_u64_le(data, 68).map(|value| return value.to_string()); + event.transfer_fee0_raw = read_u64_le(data, 76).map(|value| return value.to_string()); + event.transfer_fee1_raw = read_u64_le(data, 84).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_liquidity_change_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 84 { + return None; + } + let mut event = new_clmm_program_data_event("liquidity_change_event", "7ef0afce9e58996b"); + event.pool_state = read_pubkey_base58(data, 8); + event.tick = read_i32_le(data, 40); + event.tick_lower_index = read_i32_le(data, 44); + event.tick_upper_index = read_i32_le(data, 48); + event.liquidity_before_raw = read_u128_le(data, 52).map(|value| return value.to_string()); + event.liquidity_after_raw = read_u128_le(data, 68).map(|value| return value.to_string()); + return Some(event); +} + +fn decode_clmm_pool_created_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 190 { + return None; + } + let mut event = new_clmm_program_data_event("pool_created_event", "195e4b2f7063353f"); + event.token_mint0 = read_pubkey_base58(data, 8); + event.token_mint1 = read_pubkey_base58(data, 40); + event.tick_spacing = read_u16_le(data, 72); + event.pool_state = read_pubkey_base58(data, 74); + event.sqrt_price_x64 = read_u128_le(data, 106).map(|value| return value.to_string()); + event.tick = read_i32_le(data, 122); + event.token_vault0 = read_pubkey_base58(data, 126); + event.token_vault1 = read_pubkey_base58(data, 158); + return Some(event); +} + +fn decode_clmm_swap_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 205 { + return None; + } + let mut event = new_clmm_program_data_event("swap_event", "40c6cde8260871e2"); + event.pool_state = read_pubkey_base58(data, 8); + event.actor_wallet = read_pubkey_base58(data, 40); + event.token_account0 = read_pubkey_base58(data, 72); + event.token_account1 = read_pubkey_base58(data, 104); + event.amount0_raw = read_u64_le(data, 136).map(|value| return value.to_string()); + event.transfer_fee0_raw = read_u64_le(data, 144).map(|value| return value.to_string()); + event.amount1_raw = read_u64_le(data, 152).map(|value| return value.to_string()); + event.transfer_fee1_raw = read_u64_le(data, 160).map(|value| return value.to_string()); + event.zero_for_one = read_bool(data, 168); + event.sqrt_price_x64 = read_u128_le(data, 169).map(|value| return value.to_string()); + event.liquidity_raw = read_u128_le(data, 185).map(|value| return value.to_string()); + event.tick = read_i32_le(data, 201); + return Some(event); +} + +fn decode_clmm_update_reward_infos_event( + data: &[u8], +) -> std::option::Option { + if data.len() < 56 { + return None; + } + let mut event = new_clmm_program_data_event("update_reward_infos_event", "6d7fba4e724125ec"); + event.reward_growth_global_x64 = read_u128_vec(data, 8, 3); + return Some(event); +} + fn decode_raydium_clmm_instruction_with_token_balances( accounts_json: &str, data_json: &str, @@ -326,7 +1080,7 @@ fn decode_raydium_clmm_instruction_with_token_balances( Some(data) => data, None => return Ok(decoded), }; - if data.len() < 41 { + if data.len() < 8 { return Ok(decoded); } let discriminator_option = read_discriminator(data.as_slice()); @@ -334,6 +1088,20 @@ fn decode_raydium_clmm_instruction_with_token_balances( Some(discriminator) => discriminator, None => return Ok(decoded), }; + if discriminator == RAYDIUM_CLMM_CREATE_POOL_DISCRIMINATOR { + let event_option = decode_create_pool(accounts.as_slice(), data.as_slice()); + if let Some(event) = event_option { + decoded.push(crate::RaydiumClmmDecodedEvent::CreatePool(event)); + } + return Ok(decoded); + } + if discriminator == RAYDIUM_CLMM_COLLECT_PROTOCOL_FEE_DISCRIMINATOR { + let event_option = decode_collect_protocol_fee(accounts.as_slice(), data.as_slice()); + if let Some(event) = event_option { + decoded.push(crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event)); + } + return Ok(decoded); + } if discriminator == RAYDIUM_CLMM_SWAP_V2_DISCRIMINATOR { let event_option = decode_swap_v2(accounts.as_slice(), data.as_slice()); if let Some(event) = event_option { @@ -357,6 +1125,126 @@ fn decode_raydium_clmm_instruction_with_token_balances( return Ok(decoded); } +fn decode_create_pool( + accounts: &[std::string::String], + data: &[u8], +) -> std::option::Option { + let payer = match clone_account(accounts, 0) { + Some(value) => value, + None => return None, + }; + let amm_config = match clone_account(accounts, 1) { + Some(value) => value, + None => return None, + }; + let pool_state = match clone_account(accounts, 2) { + Some(value) => value, + None => return None, + }; + let token_mint_0 = match clone_account(accounts, 3) { + Some(value) => value, + None => return None, + }; + let token_mint_1 = match clone_account(accounts, 4) { + Some(value) => value, + None => return None, + }; + let token_vault_0 = match clone_account(accounts, 5) { + Some(value) => value, + None => return None, + }; + let token_vault_1 = match clone_account(accounts, 6) { + Some(value) => value, + None => return None, + }; + let observation_state = match clone_account(accounts, 7) { + Some(value) => value, + None => return None, + }; + let sqrt_price_x64 = match read_u128_le(data, 8) { + Some(value) => value, + None => return None, + }; + let open_time = match read_u64_le(data, 24) { + Some(value) => value, + None => return None, + }; + return Some(crate::RaydiumClmmCreatePoolDecoded { + payer, + amm_config, + pool_state, + token_mint_0, + token_mint_1, + token_vault_0, + token_vault_1, + observation_state, + sqrt_price_x64: sqrt_price_x64.to_string(), + open_time, + instruction_discriminator_hex: "e992d18ecf6840bc".to_string(), + }); +} + +fn decode_collect_protocol_fee( + accounts: &[std::string::String], + data: &[u8], +) -> std::option::Option { + let authority = match clone_account(accounts, 0) { + Some(value) => value, + None => return None, + }; + let pool_state = match clone_account(accounts, 1) { + Some(value) => value, + None => return None, + }; + let recipient_token_account_0 = match clone_account(accounts, 2) { + Some(value) => value, + None => return None, + }; + let recipient_token_account_1 = match clone_account(accounts, 3) { + Some(value) => value, + None => return None, + }; + let token_vault_0 = match clone_account(accounts, 4) { + Some(value) => value, + None => return None, + }; + let token_vault_1 = match clone_account(accounts, 7) { + Some(value) => value, + None => return None, + }; + let vault_0_mint = match clone_account(accounts, 5) { + Some(value) => value, + None => return None, + }; + let vault_1_mint = match clone_account(accounts, 6) { + Some(value) => value, + None => return None, + }; + let amount_0_requested = match read_u64_le(data, 8) { + Some(value) => value.to_string(), + None => return None, + }; + let amount_1_requested = match read_u64_le(data, 16) { + Some(value) => value.to_string(), + None => return None, + }; + return Some(crate::RaydiumClmmCollectProtocolFeeDecoded { + authority, + pool_state, + recipient_token_account_0, + recipient_token_account_1, + token_vault_0, + token_vault_1, + vault_0_mint, + vault_1_mint, + amount_0_requested_raw: amount_0_requested.clone(), + amount_1_requested_raw: amount_1_requested.clone(), + token_a_amount: amount_0_requested, + token_b_amount: amount_1_requested, + instruction_discriminator_hex: "8888fcddc2427e59".to_string(), + }); +} + fn decode_swap_legacy( accounts: &[std::string::String], data: &[u8], @@ -648,6 +1536,82 @@ fn read_bool(data: &[u8], offset: usize) -> std::option::Option { } } +fn decode_base64_standard(encoded: &str) -> std::option::Option> { + use base64::Engine as _; + let decoded = base64::engine::general_purpose::STANDARD.decode(encoded.as_bytes()); + match decoded { + Ok(decoded) => return Some(decoded), + Err(_) => return None, + } +} + +fn read_pubkey_base58(data: &[u8], offset: usize) -> std::option::Option { + if data.len() < offset + 32 { + return None; + } + return Some(bs58::encode(&data[offset..offset + 32]).into_string()); +} + +fn read_u16_le(data: &[u8], offset: usize) -> std::option::Option { + if data.len() < offset + 2 { + return None; + } + let bytes = [data[offset], data[offset + 1]]; + return Some(u16::from_le_bytes(bytes)); +} + +fn read_u32_le(data: &[u8], offset: usize) -> std::option::Option { + if data.len() < offset + 4 { + return None; + } + let bytes = [data[offset], data[offset + 1], data[offset + 2], data[offset + 3]]; + return Some(u32::from_le_bytes(bytes)); +} + +fn read_i32_le(data: &[u8], offset: usize) -> std::option::Option { + if data.len() < offset + 4 { + return None; + } + let bytes = [data[offset], data[offset + 1], data[offset + 2], data[offset + 3]]; + return Some(i32::from_le_bytes(bytes)); +} + +fn read_u64_vec( + data: &[u8], + offset: usize, + count: usize, +) -> std::option::Option> { + let mut values = std::vec::Vec::new(); + let mut index = 0_usize; + while index < count { + let value = match read_u64_le(data, offset + index * 8) { + Some(value) => value, + None => return None, + }; + values.push(value.to_string()); + index += 1; + } + return Some(values); +} + +fn read_u128_vec( + data: &[u8], + offset: usize, + count: usize, +) -> std::option::Option> { + let mut values = std::vec::Vec::new(); + let mut index = 0_usize; + while index < count { + let value = match read_u128_le(data, offset + index * 16) { + Some(value) => value, + None => return None, + }; + values.push(value.to_string()); + index += 1; + } + return Some(values); +} + fn decode_base58(input: &str) -> std::option::Option> { match bs58::decode(input).into_vec() { Ok(decoded) => return Some(decoded), @@ -1048,7 +2012,7 @@ mod tests { assert_eq!(event.sqrt_price_limit_x64, "0"); assert!(event.is_base_input); }, - crate::RaydiumClmmDecodedEvent::Swap(_) => panic!("expected swap_v2 event"), + _ => panic!("expected swap_v2 event"), } } @@ -1070,6 +2034,78 @@ mod tests { assert!(payload.contains("tradeSide")); } + #[test] + fn decodes_create_pool() { + let accounts_json = r#"[ + "Creator111111111111111111111111111111111111", + "AmmConfig1111111111111111111111111111111111", + "PoolState111111111111111111111111111111111", + "TokenMint0111111111111111111111111111111111", + "TokenMint1111111111111111111111111111111111", + "TokenVault011111111111111111111111111111111", + "TokenVault111111111111111111111111111111111", + "Observation11111111111111111111111111111111", + "Bitmap111111111111111111111111111111111111", + "TokenProgram0111111111111111111111111111111", + "TokenProgram1111111111111111111111111111111", + "System111111111111111111111111111111111111", + "Rent11111111111111111111111111111111111111" + ]"#; + let mut data = std::vec::Vec::from(super::RAYDIUM_CLMM_CREATE_POOL_DISCRIMINATOR); + data.extend_from_slice(&123_u128.to_le_bytes()); + data.extend_from_slice(&456_u64.to_le_bytes()); + let encoded = bs58::encode(data).into_string(); + let data_json = format!("\"{}\"", encoded); + let events = crate::decode_raydium_clmm_instruction(accounts_json, data_json.as_str()); + assert_eq!(events.len(), 1); + match &events[0] { + crate::RaydiumClmmDecodedEvent::CreatePool(event) => { + assert_eq!(events[0].event_kind(), "raydium_clmm.create_pool"); + assert_eq!(event.pool_state, "PoolState111111111111111111111111111111111"); + assert_eq!(event.token_mint_0, "TokenMint0111111111111111111111111111111111"); + assert_eq!(event.token_mint_1, "TokenMint1111111111111111111111111111111111"); + assert_eq!(event.sqrt_price_x64, "123"); + assert_eq!(event.open_time, 456); + }, + _ => panic!("expected create_pool event"), + } + } + + #[test] + fn decodes_collect_protocol_fee() { + let accounts_json = r#"[ + "Authority1111111111111111111111111111111111", + "PoolState111111111111111111111111111111111", + "Recipient0111111111111111111111111111111111", + "Recipient1111111111111111111111111111111111", + "TokenVault011111111111111111111111111111111", + "VaultMint0111111111111111111111111111111111", + "VaultMint1111111111111111111111111111111111", + "TokenVault111111111111111111111111111111111", + "TokenProgram0111111111111111111111111111111", + "TokenProgram1111111111111111111111111111111", + "Memo11111111111111111111111111111111111111" + ]"#; + let mut data = std::vec::Vec::from(super::RAYDIUM_CLMM_COLLECT_PROTOCOL_FEE_DISCRIMINATOR); + data.extend_from_slice(&10_u64.to_le_bytes()); + data.extend_from_slice(&20_u64.to_le_bytes()); + let encoded = bs58::encode(data).into_string(); + let data_json = format!("\"{}\"", encoded); + let events = crate::decode_raydium_clmm_instruction(accounts_json, data_json.as_str()); + assert_eq!(events.len(), 1); + match &events[0] { + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(event) => { + assert_eq!(events[0].event_kind(), "raydium_clmm.collect_protocol_fee"); + assert_eq!(event.pool_state, "PoolState111111111111111111111111111111111"); + assert_eq!(event.vault_0_mint, "VaultMint0111111111111111111111111111111111"); + assert_eq!(event.vault_1_mint, "VaultMint1111111111111111111111111111111111"); + assert_eq!(event.amount_0_requested_raw, "10"); + assert_eq!(event.amount_1_requested_raw, "20"); + }, + _ => panic!("expected collect_protocol_fee event"), + } + } + #[test] fn ignores_invalid_data() { let events = crate::decode_raydium_clmm_instruction( diff --git a/kb_lib/src/dex_decode.rs b/kb_lib/src/dex_decode.rs index 14ccc92..a7647cd 100644 --- a/kb_lib/src/dex_decode.rs +++ b/kb_lib/src/dex_decode.rs @@ -93,6 +93,11 @@ impl DexDecodeService { if let Err(error) = append_result { return Err(error); } + let cleanup_result = + self.cleanup_replaced_raydium_clmm_instruction_audits(&transaction).await; + if let Err(error) = cleanup_result { + return Err(error); + } let append_result = append_persisted_events_result( &mut persisted, self.decode_and_persist_pump_fun_events(&transaction, &instructions).await, @@ -197,9 +202,325 @@ impl DexDecodeService { if let Err(error) = append_result { return Err(error); } + let cleanup_result = + self.cleanup_replaced_raydium_clmm_instruction_audits(&transaction).await; + if let Err(error) = cleanup_result { + return Err(error); + } + let reconcile_result = + self.reconcile_raydium_clmm_confirmed_non_trade_events(&transaction).await; + if let Err(error) = reconcile_result { + return Err(error); + } return Ok(persisted); } + async fn cleanup_replaced_raydium_clmm_instruction_audits( + &self, + transaction: &crate::ChainTransactionDto, + ) -> Result<(), crate::Error> { + let transaction_id = match transaction.id { + Some(transaction_id) => transaction_id, + None => return Ok(()), + }; + let cleanup_result = + crate::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits( + self.database.as_ref(), + Some(transaction_id), + ) + .await; + match cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::debug!( + signature = %transaction.signature, + deleted_count, + "cleaned replaced Raydium CLMM instruction audits" + ); + } + return Ok(()); + }, + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot cleanup replaced Raydium CLMM instruction audits for signature '{}': {}", + transaction.signature, error + ))); + }, + } + } + + async fn reconcile_raydium_clmm_confirmed_non_trade_events( + &self, + transaction: &crate::ChainTransactionDto, + ) -> Result<(), crate::Error> { + if dex_decode_transaction_has_effective_error(transaction) { + return Ok(()); + } + let transaction_id = match transaction.id { + Some(transaction_id) => transaction_id, + None => { + return Err(crate::Error::InvalidState(format!( + "transaction '{}' has no internal id", + transaction.signature + ))); + }, + }; + let decoded_events_result = crate::query_dex_decoded_events_list_by_transaction_id( + self.database.as_ref(), + transaction_id, + ) + .await; + let decoded_events = match decoded_events_result { + Ok(decoded_events) => decoded_events, + Err(error) => return Err(error), + }; + let mut delete_create_pool_audit = false; + let mut delete_collect_protocol_fee_audit = false; + for decoded_event in &decoded_events { + if decoded_event.protocol_name != "raydium_clmm" { + continue; + } + if decoded_event.event_kind == "raydium_clmm.create_pool" { + let materialize_result = self + .materialize_raydium_clmm_create_pool_lifecycle(transaction, decoded_event) + .await; + if let Err(error) = materialize_result { + return Err(error); + } + delete_create_pool_audit = true; + continue; + } + if decoded_event.event_kind == "raydium_clmm.collect_protocol_fee" { + let materialize_result = self + .materialize_raydium_clmm_collect_protocol_fee(transaction, decoded_event) + .await; + if let Err(error) = materialize_result { + return Err(error); + } + delete_collect_protocol_fee_audit = true; + } + } + if delete_create_pool_audit { + let delete_result = self + .delete_raydium_clmm_instruction_audit_by_discriminator( + transaction_id, + "e992d18ecf6840bc", + ) + .await; + if let Err(error) = delete_result { + return Err(error); + } + } + if delete_collect_protocol_fee_audit { + let delete_result = self + .delete_raydium_clmm_instruction_audit_by_discriminator( + transaction_id, + "8888fcddc2427e59", + ) + .await; + if let Err(error) = delete_result { + return Err(error); + } + } + return Ok(()); + } + + async fn materialize_raydium_clmm_create_pool_lifecycle( + &self, + transaction: &crate::ChainTransactionDto, + decoded_event: &crate::DexDecodedEventDto, + ) -> Result<(), crate::Error> { + let decoded_event_id = match decoded_event.id { + Some(decoded_event_id) => decoded_event_id, + None => return Ok(()), + }; + let context_result = self.resolve_decoded_event_db_context(decoded_event).await; + let context = match context_result { + Ok(context) => context, + Err(error) => return Err(error), + }; + let dto = crate::PoolLifecycleEventDto::new( + decoded_event.transaction_id, + Some(decoded_event_id), + context.0, + context.1, + context.2, + transaction.signature.clone(), + transaction.slot, + decoded_event.protocol_name.clone(), + decoded_event.program_id.clone(), + decoded_event.event_kind.clone(), + decoded_event.pool_account.clone(), + decoded_event.token_a_mint.clone(), + decoded_event.token_b_mint.clone(), + decoded_event.payload_json.clone(), + ); + let upsert_result = + crate::query_pool_lifecycle_events_upsert(self.database.as_ref(), &dto).await; + match upsert_result { + Ok(_) => return Ok(()), + Err(error) => return Err(error), + } + } + + async fn materialize_raydium_clmm_collect_protocol_fee( + &self, + transaction: &crate::ChainTransactionDto, + decoded_event: &crate::DexDecodedEventDto, + ) -> Result<(), crate::Error> { + let decoded_event_id = match decoded_event.id { + Some(decoded_event_id) => decoded_event_id, + None => return Ok(()), + }; + let payload = dex_decode_payload_value(decoded_event.payload_json.as_str()); + let context_result = self.resolve_decoded_event_db_context(decoded_event).await; + let context = match context_result { + Ok(context) => context, + Err(error) => return Err(error), + }; + let actor_wallet = dex_decode_extract_first_string( + &payload, + &["authority", "actorWallet", "actor_wallet", "owner", "payer", "user"], + ); + let fee_token_mint = dex_decode_extract_first_string( + &payload, + &[ + "vault_0_mint", + "vault0Mint", + "feeTokenMint", + "fee_token_mint", + "tokenMint", + "token_mint", + "mint", + ], + ); + let fee_amount_raw = dex_decode_extract_first_amount_string( + &payload, + &[ + "amount0RequestedRaw", + "amount_0_requested_raw", + "tokenAAmount", + "token_a_amount", + "feeAmountRaw", + "fee_amount_raw", + "protocolFeeAmount", + "protocol_fee_amount", + "amount", + ], + ); + let dto = crate::FeeEventDto::new( + decoded_event.transaction_id, + Some(decoded_event_id), + context.0, + context.1, + context.2, + transaction.signature.clone(), + transaction.slot, + decoded_event.protocol_name.clone(), + decoded_event.program_id.clone(), + decoded_event.event_kind.clone(), + decoded_event.pool_account.clone(), + actor_wallet, + fee_token_mint, + fee_amount_raw, + decoded_event.payload_json.clone(), + ); + let upsert_result = crate::query_fee_events_upsert(self.database.as_ref(), &dto).await; + match upsert_result { + Ok(_) => return Ok(()), + Err(error) => return Err(error), + } + } + + async fn resolve_decoded_event_db_context( + &self, + decoded_event: &crate::DexDecodedEventDto, + ) -> Result< + (std::option::Option, std::option::Option, std::option::Option), + crate::Error, + > { + let dex_result = crate::query_dexs_get_by_code( + self.database.as_ref(), + decoded_event.protocol_name.as_str(), + ) + .await; + let dex_id = match dex_result { + Ok(Some(dex)) => dex.id, + Ok(None) => None, + Err(error) => return Err(error), + }; + let pool_account = match decoded_event.pool_account.as_ref() { + Some(pool_account) => pool_account, + None => return Ok((dex_id, None, None)), + }; + let pool_result = + crate::query_pools_get_by_address(self.database.as_ref(), pool_account.as_str()).await; + let pool = match pool_result { + Ok(Some(pool)) => pool, + Ok(None) => return Ok((dex_id, None, None)), + Err(error) => return Err(error), + }; + let pool_id = match pool.id { + Some(pool_id) => pool_id, + None => return Ok((dex_id, None, None)), + }; + let pair_result = crate::query_pairs_get_by_pool_id(self.database.as_ref(), pool_id).await; + let pair = match pair_result { + Ok(pair) => pair, + Err(error) => return Err(error), + }; + let pair_id = match pair { + Some(pair) => pair.id, + None => None, + }; + return Ok((dex_id, Some(pool_id), pair_id)); + } + + async fn delete_raydium_clmm_instruction_audit_by_discriminator( + &self, + transaction_id: i64, + discriminator_hex: &str, + ) -> Result<(), crate::Error> { + match self.database.connection() { + crate::DatabaseConnection::Sqlite(pool) => { + let delete_result = sqlx::query( + r#" +DELETE FROM k_sol_dex_decoded_events +WHERE transaction_id = ? + AND protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND ( + json_extract(payload_json, '$.discriminatorHex') = ? + OR json_extract(payload_json, '$.discriminator_hex') = ? + OR json_extract(payload_json, '$.instructionDiscriminatorHex') = ? + OR json_extract(payload_json, '$.instruction_discriminator_hex') = ? + OR json_extract(payload_json, '$.anchorEventDiscriminatorHex') = ? + OR json_extract(payload_json, '$.anchor_event_discriminator_hex') = ? + ) + "#, + ) + .bind(transaction_id) + .bind(discriminator_hex.to_string()) + .bind(discriminator_hex.to_string()) + .bind(discriminator_hex.to_string()) + .bind(discriminator_hex.to_string()) + .bind(discriminator_hex.to_string()) + .bind(discriminator_hex.to_string()) + .execute(pool) + .await; + match delete_result { + Ok(_) => return Ok(()), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot delete Raydium CLMM residual instruction audit '{}': {}", + discriminator_hex, error + ))); + }, + } + }, + } + } + async fn materialize_named_dex_event( &self, transaction: &crate::ChainTransactionDto, @@ -215,6 +536,7 @@ impl DexDecodeService { lp_mint: std::option::Option, payload_json: serde_json::Value, ) -> Result { + let payload_json_for_cleanup = payload_json.clone(); let input = crate::dex_decoded_event_materialization::DexDecodedEventMaterializationInput { database: self.database.as_ref(), persistence: &self.persistence, @@ -252,6 +574,17 @@ impl DexDecodeService { if let Err(error) = cleanup_result { return Err(error); } + let cleanup_result = self + .delete_replaced_instruction_audit_by_discriminator( + transaction_id, + protocol_name, + event_kind, + &payload_json_for_cleanup, + ) + .await; + if let Err(error) = cleanup_result { + return Err(error); + } let cleanup_result = self .delete_replaced_upstream_registry_match( transaction_id, @@ -263,9 +596,101 @@ impl DexDecodeService { if let Err(error) = cleanup_result { return Err(error); } + let non_trade_result = self + .materialize_direct_decoded_non_trade_if_needed(transaction, &materialized) + .await; + if let Err(error) = non_trade_result { + return Err(error); + } return Ok(materialized); } + async fn materialize_direct_decoded_non_trade_if_needed( + &self, + transaction: &crate::ChainTransactionDto, + decoded_event: &crate::DexDecodedEventDto, + ) -> Result<(), crate::Error> { + if dex_decode_transaction_has_effective_error(transaction) { + return Ok(()); + } + if !should_immediately_materialize_decoded_non_trade_event( + decoded_event.event_kind.as_str(), + ) { + return Ok(()); + } + if decoded_event.event_kind == "raydium_clmm.create_pool" { + return self + .materialize_raydium_clmm_create_pool_lifecycle(transaction, decoded_event) + .await; + } + if decoded_event.event_kind == "raydium_clmm.collect_protocol_fee" { + return self + .materialize_raydium_clmm_collect_protocol_fee(transaction, decoded_event) + .await; + } + return Ok(()); + } + + async fn delete_replaced_instruction_audit_by_discriminator( + &self, + transaction_id: i64, + protocol_name: &str, + event_kind: &str, + payload_json: &serde_json::Value, + ) -> Result<(), crate::Error> { + if event_kind.ends_with(".instruction_audit") { + return Ok(()); + } + let audit_event_kind = match instruction_audit_event_kind_by_protocol(protocol_name) { + Some(audit_event_kind) => audit_event_kind, + None => return Ok(()), + }; + let discriminator_hex = match instruction_discriminator_hex_from_payload(payload_json) { + Some(discriminator_hex) => discriminator_hex, + None => return Ok(()), + }; + match self.database.connection() { + crate::DatabaseConnection::Sqlite(pool) => { + let delete_result = sqlx::query( + r#" +DELETE FROM k_sol_dex_decoded_events +WHERE transaction_id = ? + AND protocol_name = ? + AND event_kind = ? + AND ( + json_extract(payload_json, '$.discriminatorHex') = ? + OR json_extract(payload_json, '$.discriminator_hex') = ? + OR json_extract(payload_json, '$.instructionDiscriminatorHex') = ? + OR json_extract(payload_json, '$.instruction_discriminator_hex') = ? + OR json_extract(payload_json, '$.anchorEventDiscriminatorHex') = ? + OR json_extract(payload_json, '$.anchor_event_discriminator_hex') = ? + ) + "#, + ) + .bind(transaction_id) + .bind(protocol_name.to_string()) + .bind(audit_event_kind.to_string()) + .bind(discriminator_hex.clone()) + .bind(discriminator_hex.clone()) + .bind(discriminator_hex.clone()) + .bind(discriminator_hex.clone()) + .bind(discriminator_hex.clone()) + .bind(discriminator_hex) + .execute(pool) + .await; + match delete_result { + Ok(_) => return Ok(()), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot delete replaced instruction audit by discriminator on sqlite: {}", + error + ))); + }, + } + }, + } + } + async fn delete_replaced_upstream_registry_match( &self, transaction_id: i64, @@ -374,6 +799,9 @@ impl DexDecodeService { Some(registry_match) => registry_match, None => continue, }; + if upstream_registry_instruction_match_is_locally_covered(®istry_match) { + continue; + } let payload = build_upstream_registry_instruction_match_payload( transaction, instruction, @@ -952,10 +1380,10 @@ impl DexDecodeService { "raydium_clmm", crate::RAYDIUM_CLMM_PROGRAM_ID.to_string(), event_kind.as_str(), - Some(decoded_event.pool_account().to_string()), + decoded_event.pool_account_option().map(|value| return value.to_string()), None, - Some(decoded_event.base_mint().to_string()), - Some(decoded_event.quote_mint().to_string()), + decoded_event.base_mint_option().map(|value| return value.to_string()), + decoded_event.quote_mint_option().map(|value| return value.to_string()), None, payload_value, ) @@ -1244,6 +1672,31 @@ impl DexDecodeService { }; persisted.push(persisted_event); } + + let mut program_data_events = collect_raydium_clmm_program_data_events(transaction); + for instruction in instructions { + let program_id = match instruction.program_id.as_ref() { + Some(program_id) => program_id, + None => continue, + }; + if program_id.as_str() != crate::RAYDIUM_CLMM_PROGRAM_ID { + continue; + } + let data_base58 = parse_instruction_data_base58(instruction.data_json.as_deref()); + let discriminator_hex = discriminator_hex_from_base58(data_base58.as_deref()); + let persist_result = persist_matching_raydium_clmm_program_data_events( + self, + transaction, + instruction, + discriminator_hex.as_deref(), + &mut program_data_events, + &mut persisted, + ) + .await; + if let Err(error) = persist_result { + return Err(error); + } + } return Ok(persisted); } @@ -1295,6 +1748,8 @@ impl DexDecodeService { Err(error) => return Err(error), }; let mut decoded_instruction_ids = std::collections::HashSet::::new(); + let mut decoded_discriminator_keys = + std::collections::HashSet::::new(); for decoded_event in &decoded_events { if !decoded_event.protocol_name.starts_with("raydium_") { continue; @@ -1302,11 +1757,17 @@ impl DexDecodeService { if decoded_event.event_kind.ends_with(".instruction_audit") { continue; } - let instruction_id = match decoded_event.instruction_id { - Some(instruction_id) => instruction_id, - None => continue, - }; - decoded_instruction_ids.insert(instruction_id); + if let Some(instruction_id) = decoded_event.instruction_id { + decoded_instruction_ids.insert(instruction_id); + } + let discriminator = + instruction_discriminator_hex_from_payload_str(decoded_event.payload_json.as_str()); + if let Some(discriminator) = discriminator { + decoded_discriminator_keys.insert(raydium_decoded_discriminator_key( + decoded_event.protocol_name.as_str(), + discriminator.as_str(), + )); + } } let mut persisted = std::vec::Vec::new(); for instruction in instructions { @@ -1328,11 +1789,27 @@ impl DexDecodeService { let accounts = parse_instruction_accounts_vec(instruction.accounts_json.as_str()); let data_base58 = parse_instruction_data_base58(instruction.data_json.as_deref()); let discriminator_hex = discriminator_hex_from_base58(data_base58.as_deref()); + if raydium_instruction_already_decoded_by_discriminator( + &decoded_discriminator_keys, + audit_spec.protocol_name, + discriminator_hex.as_deref(), + ) { + continue; + } let mapped_spec = raydium_mapped_non_trade_instruction_spec( audit_spec.protocol_name, discriminator_hex.as_deref(), accounts.len(), ); + if let Some(mapped_spec) = mapped_spec { + if raydium_mapped_event_kind_already_decoded( + decoded_events.as_slice(), + audit_spec.protocol_name, + mapped_spec.event_kind, + ) { + continue; + } + } let event_kind = match mapped_spec { Some(mapped_spec) => mapped_spec.event_kind, None => audit_spec.event_kind, @@ -1822,7 +2299,12 @@ struct RaydiumMappedNonTradeInstructionSpec { #[derive(Clone, Copy)] enum RaydiumMappedNonTradeAmountLayout { None, + ClmmCreatePool, + ClmmFeePair, ClmmLiquidityV2, + ClmmOpenLimitOrder, + ClmmIncreaseLimitOrder, + ClmmDecreaseLimitOrder, CpmmAmmConfig, CpmmDeposit, CpmmFeePair, @@ -1868,6 +2350,303 @@ fn raydium_mapped_non_trade_instruction_spec( None => return None, }; if protocol_name == "raydium_clmm" { + if discriminator_hex == "4c7c800fd55725fa" && account_count >= 3 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "close_limit_order", + event_kind: "raydium_clmm.close_limit_order", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "9d20dab7471d1293" && account_count >= 11 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "open_limit_order", + event_kind: "raydium_clmm.open_limit_order", + pool_account_index: Some(1), + token_a_mint_index: Some(7), + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::ClmmOpenLimitOrder, + }); + } + if discriminator_hex == "b19059ecfaba7d63" && account_count >= 8 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "increase_limit_order", + event_kind: "raydium_clmm.increase_limit_order", + pool_account_index: Some(1), + token_a_mint_index: Some(6), + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::ClmmIncreaseLimitOrder, + }); + } + if discriminator_hex == "759d3c674231a300" && account_count >= 12 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "decrease_limit_order", + event_kind: "raydium_clmm.decrease_limit_order", + pool_account_index: Some(1), + token_a_mint_index: Some(8), + token_b_mint_index: Some(9), + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::ClmmDecreaseLimitOrder, + }); + } + if discriminator_hex == "c975989055556cb2" && account_count >= 2 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "close_protocol_position", + event_kind: "raydium_clmm.close_protocol_position", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "a78a4e95dfc2067e" && account_count >= 7 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "collect_fund_fee", + event_kind: "raydium_clmm.collect_fund_fee", + pool_account_index: Some(1), + token_a_mint_index: Some(5), + token_b_mint_index: Some(6), + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::ClmmFeePair, + }); + } + if discriminator_hex == "8888fcddc2427e59" && account_count >= 7 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "collect_protocol_fee", + event_kind: "raydium_clmm.collect_protocol_fee", + pool_account_index: Some(1), + token_a_mint_index: Some(5), + token_b_mint_index: Some(6), + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::ClmmFeePair, + }); + } + if discriminator_hex == "e992d18ecf6840bc" && account_count >= 13 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "create_pool", + event_kind: "raydium_clmm.create_pool", + pool_account_index: Some(2), + token_a_mint_index: Some(3), + token_b_mint_index: Some(4), + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::ClmmCreatePool, + }); + } + if discriminator_hex == "12eda6c52210d590" && account_count >= 5 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "collect_remaining_rewards", + event_kind: "raydium_clmm.collect_remaining_rewards", + pool_account_index: Some(2), + token_a_mint_index: Some(4), + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "8934edd4d7756c68" && account_count >= 3 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "create_amm_config", + event_kind: "raydium_clmm.create_amm_config", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "2b44d4a7592fa401" && account_count >= 13 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "create_customizable_pool", + event_kind: "raydium_clmm.create_customizable_pool", + pool_account_index: Some(2), + token_a_mint_index: Some(3), + token_b_mint_index: Some(4), + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "bd0eb5785576e33e" && account_count >= 3 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "create_dynamic_fee_config", + event_kind: "raydium_clmm.create_dynamic_fee_config", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "3f5794216d230868" && account_count >= 2 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "create_operation_account", + event_kind: "raydium_clmm.create_operation_account", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "11fb415c88f20ea9" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "create_support_mint_associated", + event_kind: "raydium_clmm.create_support_mint_associated", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "a026d06f685b2c01" && account_count >= 4 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "decrease_liquidity", + event_kind: "raydium_clmm.decrease_liquidity", + pool_account_index: Some(3), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: Some(1), + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "2e9cf3760dcdfbb2" && account_count >= 3 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "increase_liquidity", + event_kind: "raydium_clmm.increase_liquidity", + pool_account_index: Some(2), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: Some(1), + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "5f87c0c4f281e644" && account_count >= 2 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "initialize_reward", + event_kind: "raydium_clmm.initialize_reward", + pool_account_index: Some(1), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "87802f4d0f98f031" && account_count >= 6 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "open_position", + event_kind: "raydium_clmm.open_position", + pool_account_index: Some(5), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: Some(2), + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "4db84ad67056f1c7" && account_count >= 6 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "open_position_v2", + event_kind: "raydium_clmm.open_position_v2", + pool_account_index: Some(5), + token_a_mint_index: if account_count >= 22 { Some(20) } else { None }, + token_b_mint_index: if account_count >= 22 { Some(21) } else { None }, + lp_mint_index: Some(2), + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "7034a74b20c9d389" && account_count >= 2 { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "set_reward_params", + event_kind: "raydium_clmm.set_reward_params", + pool_account_index: Some(1), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "cd4e74215c691a60" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "settle_limit_order", + event_kind: "raydium_clmm.settle_limit_order", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "457d73daf5baf2c4" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "swap_router_base_in", + event_kind: "raydium_clmm.swap_router_base_in", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "07160c53f22b3079" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "transfer_reward_owner", + event_kind: "raydium_clmm.transfer_reward_owner", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "313cae889a1c74c8" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "update_amm_config", + event_kind: "raydium_clmm.update_amm_config", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "7f467728bce33d07" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "update_operation_account", + event_kind: "raydium_clmm.update_operation_account", + pool_account_index: None, + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "82576c062ee0757b" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "update_pool_status", + event_kind: "raydium_clmm.update_pool_status", + pool_account_index: Some(0), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } + if discriminator_hex == "a3ace0340b9a6adf" { + return Some(RaydiumMappedNonTradeInstructionSpec { + instruction_name: "update_reward_infos", + event_kind: "raydium_clmm.update_reward_infos", + pool_account_index: Some(0), + token_a_mint_index: None, + token_b_mint_index: None, + lp_mint_index: None, + amount_layout: RaydiumMappedNonTradeAmountLayout::None, + }); + } if discriminator_hex == "3a7fbc3e4f52c460" && account_count >= 16 { return Some(RaydiumMappedNonTradeInstructionSpec { instruction_name: "decrease_liquidity_v2", @@ -2136,6 +2915,42 @@ fn insert_raydium_mapped_amounts( ) { match amount_layout { RaydiumMappedNonTradeAmountLayout::None => return, + RaydiumMappedNonTradeAmountLayout::ClmmCreatePool => { + if let Some(sqrt_price_x64) = read_u128_le_from_bytes(data, 8) { + object.insert( + "sqrtPriceX64".to_string(), + serde_json::Value::String(sqrt_price_x64.to_string()), + ); + } + if let Some(open_time) = read_u64_le_from_bytes(data, 24) { + object.insert( + "openTime".to_string(), + serde_json::Value::String(open_time.to_string()), + ); + } + }, + RaydiumMappedNonTradeAmountLayout::ClmmFeePair => { + if let Some(amount_0) = read_u64_le_from_bytes(data, 8) { + object.insert( + "tokenAAmount".to_string(), + serde_json::Value::String(amount_0.to_string()), + ); + object.insert( + "amount0RequestedRaw".to_string(), + serde_json::Value::String(amount_0.to_string()), + ); + } + if let Some(amount_1) = read_u64_le_from_bytes(data, 16) { + object.insert( + "tokenBAmount".to_string(), + serde_json::Value::String(amount_1.to_string()), + ); + object.insert( + "amount1RequestedRaw".to_string(), + serde_json::Value::String(amount_1.to_string()), + ); + } + }, RaydiumMappedNonTradeAmountLayout::ClmmLiquidityV2 => { if let Some(liquidity) = read_u128_le_from_bytes(data, 8) { object.insert( @@ -2160,6 +2975,57 @@ fn insert_raydium_mapped_amounts( ); } }, + RaydiumMappedNonTradeAmountLayout::ClmmOpenLimitOrder => { + if let Some(nonce_index) = read_u8_from_bytes(data, 8) { + object.insert( + "nonceIndex".to_string(), + serde_json::Value::Number(serde_json::Number::from(nonce_index as u64)), + ); + } + if let Some(zero_for_one) = read_u8_from_bytes(data, 9) { + object.insert("zeroForOne".to_string(), serde_json::Value::Bool(zero_for_one != 0)); + } + if let Some(tick_index) = read_i32_le_from_bytes(data, 10) { + object.insert( + "tickIndex".to_string(), + serde_json::Value::Number(serde_json::Number::from(tick_index as i64)), + ); + } + if let Some(amount) = read_u64_le_from_bytes(data, 14) { + object + .insert("amountRaw".to_string(), serde_json::Value::String(amount.to_string())); + object.insert( + "orderAmountRaw".to_string(), + serde_json::Value::String(amount.to_string()), + ); + } + }, + RaydiumMappedNonTradeAmountLayout::ClmmIncreaseLimitOrder => { + if let Some(amount) = read_u64_le_from_bytes(data, 8) { + object + .insert("amountRaw".to_string(), serde_json::Value::String(amount.to_string())); + object.insert( + "increasedAmountRaw".to_string(), + serde_json::Value::String(amount.to_string()), + ); + } + }, + RaydiumMappedNonTradeAmountLayout::ClmmDecreaseLimitOrder => { + if let Some(amount) = read_u64_le_from_bytes(data, 8) { + object + .insert("amountRaw".to_string(), serde_json::Value::String(amount.to_string())); + object.insert( + "decreasedAmountRaw".to_string(), + serde_json::Value::String(amount.to_string()), + ); + } + if let Some(amount_min) = read_u64_le_from_bytes(data, 16) { + object.insert( + "amountMinRaw".to_string(), + serde_json::Value::String(amount_min.to_string()), + ); + } + }, RaydiumMappedNonTradeAmountLayout::CpmmAmmConfig => { if let Some(param) = read_u8_from_bytes(data, 8) { object.insert( @@ -2296,6 +3162,19 @@ fn read_u8_from_bytes(data: &[u8], offset: usize) -> std::option::Option { return Some(data[offset]); } +fn read_i32_le_from_bytes(data: &[u8], offset: usize) -> std::option::Option { + if data.len() < offset + 4 { + return None; + } + let mut bytes = [0_u8; 4]; + let mut index = 0_usize; + while index < 4 { + bytes[index] = data[offset + index]; + index += 1; + } + return Some(i32::from_le_bytes(bytes)); +} + fn read_u64_le_from_bytes(data: &[u8], offset: usize) -> std::option::Option { if data.len() < offset + 8 { return None; @@ -2469,6 +3348,197 @@ fn build_meteora_instruction_audit_payload( }); } +fn instruction_discriminator_hex_from_payload( + payload_json: &serde_json::Value, +) -> std::option::Option { + let candidates = [ + "instructionDiscriminatorHex", + "instruction_discriminator_hex", + "discriminatorHex", + "discriminator_hex", + "anchorEventDiscriminatorHex", + "anchor_event_discriminator_hex", + ]; + for candidate in candidates { + let value = payload_json.get(candidate).and_then(serde_json::Value::as_str); + let value = match value { + Some(value) => value.trim(), + None => continue, + }; + if !value.is_empty() { + return Some(value.to_string()); + } + } + return None; +} + +fn instruction_discriminator_hex_from_payload_str( + payload_json: &str, +) -> std::option::Option { + let parsed = serde_json::from_str::(payload_json); + let parsed = match parsed { + Ok(parsed) => parsed, + Err(_) => return None, + }; + return instruction_discriminator_hex_from_payload(&parsed); +} + +fn raydium_decoded_discriminator_key( + protocol_name: &str, + discriminator_hex: &str, +) -> std::string::String { + return format!("{}:{}", protocol_name, discriminator_hex); +} + +fn raydium_instruction_already_decoded_by_discriminator( + decoded_discriminator_keys: &std::collections::HashSet, + protocol_name: &str, + discriminator_hex: std::option::Option<&str>, +) -> bool { + let discriminator_hex = match discriminator_hex { + Some(discriminator_hex) => discriminator_hex, + None => return false, + }; + let key = raydium_decoded_discriminator_key(protocol_name, discriminator_hex); + return decoded_discriminator_keys.contains(&key); +} + +fn raydium_mapped_event_kind_already_decoded( + decoded_events: &[crate::DexDecodedEventDto], + protocol_name: &str, + event_kind: &str, +) -> bool { + for decoded_event in decoded_events { + if decoded_event.protocol_name != protocol_name { + continue; + } + if decoded_event.event_kind == event_kind { + return true; + } + } + return false; +} + +fn should_immediately_materialize_decoded_non_trade_event(event_kind: &str) -> bool { + if event_kind == "raydium_clmm.create_pool" { + return true; + } + if event_kind == "raydium_clmm.collect_protocol_fee" { + return true; + } + return false; +} + +fn dex_decode_transaction_has_effective_error(transaction: &crate::ChainTransactionDto) -> bool { + let err_json = match transaction.err_json.as_ref() { + Some(err_json) => err_json.trim(), + None => return false, + }; + if err_json.is_empty() { + return false; + } + if err_json == "null" { + return false; + } + return true; +} + +fn dex_decode_payload_value(payload_json: &str) -> serde_json::Value { + let parsed = serde_json::from_str::(payload_json); + match parsed { + Ok(parsed) => return parsed, + Err(_) => return serde_json::Value::Object(serde_json::Map::new()), + } +} + +fn dex_decode_extract_first_amount_string( + value: &serde_json::Value, + candidate_keys: &[&str], +) -> std::option::Option { + let text = dex_decode_extract_first_string(value, candidate_keys); + if text.is_some() { + return text; + } + return dex_decode_extract_first_number_as_string(value, candidate_keys); +} + +fn dex_decode_extract_first_string( + value: &serde_json::Value, + candidate_keys: &[&str], +) -> std::option::Option { + if let Some(object) = value.as_object() { + for candidate_key in candidate_keys { + let candidate_value = object.get(*candidate_key); + let candidate_value = match candidate_value { + Some(candidate_value) => candidate_value, + None => continue, + }; + if let Some(text) = candidate_value.as_str() { + let trimmed = text.trim(); + if !trimmed.is_empty() { + return Some(trimmed.to_string()); + } + } + } + for nested_value in object.values() { + let nested = dex_decode_extract_first_string(nested_value, candidate_keys); + if nested.is_some() { + return nested; + } + } + return None; + } + if let Some(array) = value.as_array() { + for nested_value in array { + let nested = dex_decode_extract_first_string(nested_value, candidate_keys); + if nested.is_some() { + return nested; + } + } + } + return None; +} + +fn dex_decode_extract_first_number_as_string( + value: &serde_json::Value, + candidate_keys: &[&str], +) -> std::option::Option { + if let Some(object) = value.as_object() { + for candidate_key in candidate_keys { + let candidate_value = object.get(*candidate_key); + let candidate_value = match candidate_value { + Some(candidate_value) => candidate_value, + None => continue, + }; + if let Some(number) = candidate_value.as_i64() { + return Some(number.to_string()); + } + if let Some(number) = candidate_value.as_u64() { + return Some(number.to_string()); + } + if let Some(number) = candidate_value.as_f64() { + return Some(number.to_string()); + } + } + for nested_value in object.values() { + let nested = dex_decode_extract_first_number_as_string(nested_value, candidate_keys); + if nested.is_some() { + return nested; + } + } + return None; + } + if let Some(array) = value.as_array() { + for nested_value in array { + let nested = dex_decode_extract_first_number_as_string(nested_value, candidate_keys); + if nested.is_some() { + return nested; + } + } + } + return None; +} + fn instruction_audit_event_kind_by_protocol( protocol_name: &str, ) -> std::option::Option<&'static str> { @@ -2551,6 +3621,22 @@ fn candidate_raydium_audit_pool_account( return accounts.get(spec.candidate_pool_account_index).cloned(); } +fn upstream_registry_instruction_match_is_locally_covered( + registry_match: &crate::UpstreamRegistryEntryDto, +) -> bool { + if registry_match.entry_kind != crate::ENTRY_KIND_INSTRUCTION { + return false; + } + let local_event_kind = crate::dex_event_coverage::known_local_event_kind( + registry_match.decoder_code.as_str(), + registry_match.entry_name.as_str(), + ); + match local_event_kind { + Some(_) => return true, + None => return false, + } +} + fn build_upstream_registry_instruction_match_payload( transaction: &crate::ChainTransactionDto, instruction: &crate::ChainInstructionDto, @@ -2676,6 +3762,156 @@ fn append_persisted_events( } } +#[derive(Clone, Debug)] +struct RaydiumClmmProgramDataEventCandidate { + decoded_event: crate::RaydiumClmmDecodedEvent, + consumed: bool, +} + +fn collect_raydium_clmm_program_data_events( + transaction: &crate::ChainTransactionDto, +) -> std::vec::Vec { + let logs = extract_transaction_log_messages(transaction.transaction_json.as_str()); + let mut events = std::vec::Vec::new(); + let mut clmm_stack_depth = 0_u32; + for log_message in logs { + if is_program_invoke_log(log_message.as_str(), crate::RAYDIUM_CLMM_PROGRAM_ID) { + clmm_stack_depth += 1; + continue; + } + if is_program_success_or_failed_log(log_message.as_str(), crate::RAYDIUM_CLMM_PROGRAM_ID) { + clmm_stack_depth = clmm_stack_depth.saturating_sub(1); + continue; + } + if clmm_stack_depth == 0 { + continue; + } + let data_base64 = match log_message.strip_prefix("Program data: ") { + Some(data_base64) => data_base64.trim(), + None => continue, + }; + if data_base64.is_empty() { + continue; + } + let decoded_event = crate::decode_raydium_clmm_program_data_event(data_base64); + if let Some(decoded_event) = decoded_event { + events.push(RaydiumClmmProgramDataEventCandidate { decoded_event, consumed: false }); + } + } + return events; +} + +async fn persist_matching_raydium_clmm_program_data_events( + service: &DexDecodeService, + transaction: &crate::ChainTransactionDto, + instruction: &crate::ChainInstructionDto, + instruction_discriminator_hex: std::option::Option<&str>, + program_data_events: &mut [RaydiumClmmProgramDataEventCandidate], + persisted: &mut std::vec::Vec, +) -> Result<(), crate::Error> { + let instruction_id = match instruction.id { + Some(instruction_id) => instruction_id, + None => return Ok(()), + }; + let expected_event_kinds = + raydium_clmm_program_data_event_kinds_for_instruction(instruction_discriminator_hex); + if expected_event_kinds.is_empty() { + return Ok(()); + } + let mut index = 0_usize; + while index < program_data_events.len() { + if program_data_events[index].consumed { + index += 1; + continue; + } + let event_kind = program_data_events[index].decoded_event.event_kind(); + if !string_slice_contains(expected_event_kinds.as_slice(), event_kind) { + index += 1; + continue; + } + program_data_events[index].consumed = true; + let persist_result = service + .persist_raydium_clmm_event( + transaction, + instruction_id, + &program_data_events[index].decoded_event, + ) + .await; + let persisted_event = match persist_result { + Ok(persisted_event) => persisted_event, + Err(error) => return Err(error), + }; + persisted.push(persisted_event); + index += 1; + } + return Ok(()); +} + +fn raydium_clmm_program_data_event_kinds_for_instruction( + instruction_discriminator_hex: std::option::Option<&str>, +) -> std::vec::Vec<&'static str> { + let discriminator = match instruction_discriminator_hex { + Some(discriminator) => discriminator, + None => return std::vec::Vec::new(), + }; + match discriminator { + "e992d18ecf6840bc" | "2b44d4a7592fa401" => { + return vec!["raydium_clmm.pool_created_event"]; + }, + "8888fcddc2427e59" => { + return vec!["raydium_clmm.collect_protocol_fee_event"]; + }, + "f8c69e91e17587c8" | "2b04ed0b1ac91e62" | "457d73daf5baf2c4" => { + return vec!["raydium_clmm.swap_event"]; + }, + "87802f4d0f98f031" | "4db84ad67056f1c7" | "4dffae527d1dc92e" => { + return vec![ + "raydium_clmm.liquidity_calculate_event", + "raydium_clmm.create_personal_position_event", + "raydium_clmm.increase_liquidity_event", + "raydium_clmm.liquidity_change_event", + ]; + }, + "2e9cf3760dcdfbb2" | "851d59df45eeb00a" => { + return vec![ + "raydium_clmm.liquidity_calculate_event", + "raydium_clmm.increase_liquidity_event", + "raydium_clmm.liquidity_change_event", + ]; + }, + "a026d06f685b2c01" | "3a7fbc3e4f52c460" => { + return vec![ + "raydium_clmm.liquidity_calculate_event", + "raydium_clmm.decrease_liquidity_event", + "raydium_clmm.liquidity_change_event", + ]; + }, + "7b86510031446262" | "c975989055556cb2" => { + return vec![ + "raydium_clmm.decrease_liquidity_event", + "raydium_clmm.collect_personal_fee_event", + "raydium_clmm.liquidity_change_event", + ]; + }, + "8934edd4d7756c68" | "313cae889a1c74c8" | "bd0eb5785576e33e" => { + return vec!["raydium_clmm.config_change_event"]; + }, + "5f87c0c4f281e644" | "7034a74b20c9d389" | "a3ace0340b9a6adf" => { + return vec!["raydium_clmm.update_reward_infos_event"]; + }, + _ => return std::vec::Vec::new(), + } +} + +fn string_slice_contains(values: &[&'static str], expected: &str) -> bool { + for value in values { + if *value == expected { + return true; + } + } + return false; +} + #[derive(Clone, Debug)] struct RaydiumCpmmProgramDataEventCandidate { decoded_event: crate::RaydiumCpmmDecodedEvent, @@ -3827,6 +5063,32 @@ mod tests { #[test] fn maps_observed_raydium_clmm_non_swap_discriminators() { + let create_pool = super::raydium_mapped_non_trade_instruction_spec( + "raydium_clmm", + Some("e992d18ecf6840bc"), + 13, + ); + let create_pool = match create_pool { + Some(create_pool) => create_pool, + None => panic!("create_pool discriminator must be mapped"), + }; + assert_eq!(create_pool.event_kind, "raydium_clmm.create_pool"); + assert_eq!(create_pool.pool_account_index, Some(2)); + assert_eq!(create_pool.token_a_mint_index, Some(3)); + assert_eq!(create_pool.token_b_mint_index, Some(4)); + let collect_protocol_fee = super::raydium_mapped_non_trade_instruction_spec( + "raydium_clmm", + Some("8888fcddc2427e59"), + 11, + ); + let collect_protocol_fee = match collect_protocol_fee { + Some(collect_protocol_fee) => collect_protocol_fee, + None => panic!("collect_protocol_fee discriminator must be mapped"), + }; + assert_eq!(collect_protocol_fee.event_kind, "raydium_clmm.collect_protocol_fee"); + assert_eq!(collect_protocol_fee.pool_account_index, Some(1)); + assert_eq!(collect_protocol_fee.token_a_mint_index, Some(5)); + assert_eq!(collect_protocol_fee.token_b_mint_index, Some(6)); let decrease = super::raydium_mapped_non_trade_instruction_spec( "raydium_clmm", Some("3a7fbc3e4f52c460"), @@ -3851,6 +5113,43 @@ mod tests { }; assert_eq!(increase.event_kind, "raydium_clmm.increase_liquidity_v2"); assert_eq!(increase.pool_account_index, Some(2)); + let open_limit_order = super::raydium_mapped_non_trade_instruction_spec( + "raydium_clmm", + Some("9d20dab7471d1293"), + 11, + ); + let open_limit_order = match open_limit_order { + Some(open_limit_order) => open_limit_order, + None => panic!("open_limit_order discriminator must be mapped"), + }; + assert_eq!(open_limit_order.event_kind, "raydium_clmm.open_limit_order"); + assert_eq!(open_limit_order.pool_account_index, Some(1)); + assert_eq!(open_limit_order.token_a_mint_index, Some(7)); + let increase_limit_order = super::raydium_mapped_non_trade_instruction_spec( + "raydium_clmm", + Some("b19059ecfaba7d63"), + 8, + ); + let increase_limit_order = match increase_limit_order { + Some(increase_limit_order) => increase_limit_order, + None => panic!("increase_limit_order discriminator must be mapped"), + }; + assert_eq!(increase_limit_order.event_kind, "raydium_clmm.increase_limit_order"); + assert_eq!(increase_limit_order.pool_account_index, Some(1)); + assert_eq!(increase_limit_order.token_a_mint_index, Some(6)); + let decrease_limit_order = super::raydium_mapped_non_trade_instruction_spec( + "raydium_clmm", + Some("759d3c674231a300"), + 13, + ); + let decrease_limit_order = match decrease_limit_order { + Some(decrease_limit_order) => decrease_limit_order, + None => panic!("decrease_limit_order discriminator must be mapped"), + }; + assert_eq!(decrease_limit_order.event_kind, "raydium_clmm.decrease_limit_order"); + assert_eq!(decrease_limit_order.pool_account_index, Some(1)); + assert_eq!(decrease_limit_order.token_a_mint_index, Some(8)); + assert_eq!(decrease_limit_order.token_b_mint_index, Some(9)); } #[test] @@ -3883,6 +5182,56 @@ mod tests { } } + #[test] + fn extracts_instruction_discriminator_from_camel_and_snake_payload_keys() { + let camel_payload = serde_json::json!({ + "instructionDiscriminatorHex": "e992d18ecf6840bc" + }); + assert_eq!( + super::instruction_discriminator_hex_from_payload(&camel_payload), + Some("e992d18ecf6840bc".to_string()) + ); + let snake_payload = serde_json::json!({ + "instruction_discriminator_hex": "8888fcddc2427e59" + }); + assert_eq!( + super::instruction_discriminator_hex_from_payload(&snake_payload), + Some("8888fcddc2427e59".to_string()) + ); + } + + #[test] + fn skips_raydium_audit_when_discriminator_was_already_decoded() { + let mut keys = std::collections::HashSet::::new(); + keys.insert(super::raydium_decoded_discriminator_key("raydium_clmm", "e992d18ecf6840bc")); + assert!(super::raydium_instruction_already_decoded_by_discriminator( + &keys, + "raydium_clmm", + Some("e992d18ecf6840bc"), + )); + assert!(!super::raydium_instruction_already_decoded_by_discriminator( + &keys, + "raydium_clmm", + Some("8888fcddc2427e59"), + )); + } + + #[test] + fn immediately_materializes_only_targeted_clmm_non_trade_events() { + assert!(super::should_immediately_materialize_decoded_non_trade_event( + "raydium_clmm.create_pool", + )); + assert!(super::should_immediately_materialize_decoded_non_trade_event( + "raydium_clmm.collect_protocol_fee", + )); + assert!(!super::should_immediately_materialize_decoded_non_trade_event( + "raydium_clmm.swap", + )); + assert!(!super::should_immediately_materialize_decoded_non_trade_event( + "raydium_cpmm.collect_protocol_fee", + )); + } + #[test] fn maps_instruction_audit_event_kind_for_raydium_and_meteora_dlmm_protocols() { assert_eq!( diff --git a/kb_lib/src/dex_decoded_event_materialization.rs b/kb_lib/src/dex_decoded_event_materialization.rs index 2715f1f..248a388 100644 --- a/kb_lib/src/dex_decoded_event_materialization.rs +++ b/kb_lib/src/dex_decoded_event_materialization.rs @@ -148,7 +148,7 @@ fn prepare_payload_for_transaction_status( transaction: &crate::ChainTransactionDto, payload_json: serde_json::Value, ) -> serde_json::Value { - if transaction.err_json.is_none() { + if !transaction_has_effective_error(transaction) { return payload_json; } let mut object = match payload_json { @@ -177,3 +177,17 @@ fn prepare_payload_for_transaction_status( ); return serde_json::Value::Object(object); } + +fn transaction_has_effective_error(transaction: &crate::ChainTransactionDto) -> bool { + let err_json = match transaction.err_json.as_ref() { + Some(err_json) => err_json.trim(), + None => return false, + }; + if err_json.is_empty() { + return false; + } + if err_json == "null" { + return false; + } + return true; +} diff --git a/kb_lib/src/dex_event_classification.rs b/kb_lib/src/dex_event_classification.rs index b35132f..2dc0a28 100644 --- a/kb_lib/src/dex_event_classification.rs +++ b/kb_lib/src/dex_event_classification.rs @@ -238,9 +238,15 @@ pub fn classify_dex_event_actionability( if trade_candidate { return DexEventActionability::TradeCandidate; } + if is_dex_informational_event_kind(event_kind) { + return DexEventActionability::Informational; + } if is_dex_trade_event_kind(event_kind) { return DexEventActionability::NonActionableTrade; } + if is_dex_orderbook_event_kind(event_kind) { + return DexEventActionability::NonTradeUseful; + } let category = classify_dex_event_category(event_kind); match category { DexEventCategory::Liquidity => return DexEventActionability::NonTradeUseful, @@ -323,6 +329,12 @@ pub fn is_dex_liquidity_event_kind(event_kind: &str) -> bool { if event_kind.contains(".lp_change_event") { return true; } + if event_kind.contains(".liquidity_change_event") { + return true; + } + if event_kind.contains(".liquidity_calculate_event") { + return true; + } if event_kind.contains(".withdraw") { return true; } @@ -341,6 +353,9 @@ pub fn is_dex_liquidity_event_kind(event_kind: &str) -> bool { if event_kind.contains(".initialize_position") { return true; } + if event_kind.contains(".create_personal_position_event") { + return true; + } if event_kind.contains(".open_position") { return true; } @@ -440,6 +455,38 @@ pub fn is_dex_reward_event_kind(event_kind: &str) -> bool { return false; } +/// Returns true for orderbook or limit-order events that must not become candles. +pub fn is_dex_orderbook_event_kind(event_kind: &str) -> bool { + if event_kind.contains(".order_place") { + return true; + } + if event_kind.contains(".order_cancel") { + return true; + } + if event_kind.contains(".order_fill") { + return true; + } + if event_kind.contains(".settle_funds") { + return true; + } + if event_kind.contains(".open_limit_order") { + return true; + } + if event_kind.contains(".increase_limit_order") { + return true; + } + if event_kind.contains(".decrease_limit_order") { + return true; + } + if event_kind.contains(".close_limit_order") { + return true; + } + if event_kind.contains(".settle_limit_order") { + return true; + } + return false; +} + /// Returns true for pool, pair, launch, mint, burn or migration lifecycle events. pub fn is_dex_pool_lifecycle_event_kind(event_kind: &str) -> bool { if event_kind.contains(".create_lock_escrow") { @@ -539,6 +586,9 @@ pub fn is_dex_pool_creation_event_kind(event_kind: &str) -> bool { if event_kind.contains(".create_pool") { return true; } + if event_kind.contains(".pool_created_event") { + return true; + } if event_kind.contains(".create_amm") { return true; } diff --git a/kb_lib/src/dex_event_coverage.rs b/kb_lib/src/dex_event_coverage.rs index bb02dcb..8561b24 100644 --- a/kb_lib/src/dex_event_coverage.rs +++ b/kb_lib/src/dex_event_coverage.rs @@ -214,6 +214,9 @@ fn infer_expected_db_target_for_entry( if decoder_code == "raydium_cpmm" && entry_name == "swap_event" { return Some(crate::DexEventCoverageEntryDto::DB_TARGET_DECODED_EVENTS_ONLY.to_string()); } + if decoder_code == "raydium_clmm" && entry_name == "initialize_reward" { + return Some(crate::DexEventCoverageEntryDto::DB_TARGET_REWARD_EVENTS.to_string()); + } return infer_expected_db_target(event_family, entry_kind); } @@ -238,8 +241,8 @@ fn infer_expected_db_target( "liquidity" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS, "liquidity_add" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS, "liquidity_remove" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS, - "position_open" => crate::DexEventCoverageEntryDto::DB_TARGET_POOL_LIFECYCLE_EVENTS, - "position_close" => crate::DexEventCoverageEntryDto::DB_TARGET_POOL_LIFECYCLE_EVENTS, + "position_open" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS, + "position_close" => crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS, "fee" => crate::DexEventCoverageEntryDto::DB_TARGET_FEE_EVENTS, "reward" => crate::DexEventCoverageEntryDto::DB_TARGET_REWARD_EVENTS, "admin_config" => crate::DexEventCoverageEntryDto::DB_TARGET_POOL_ADMIN_EVENTS, @@ -283,6 +286,7 @@ fn infer_event_family( return Some("swap".to_string()); } if contains_any(normalized.as_str(), &["create_pool", "initialize_pool", "initialize2"]) + || normalized == "create_customizable_pool" || normalized == "initialize" || normalized.starts_with("initialize_") { @@ -306,11 +310,13 @@ fn infer_event_family( } if contains_any(normalized.as_str(), &["close_position", "position_close"]) || normalized.contains("close_position_if_empty") + || normalized == "close_protocol_position" { return Some("position_close".to_string()); } if contains_any(normalized.as_str(), &["fee", "collect", "claim_fee"]) && !normalized.contains("reward") + && !normalized.contains("config") { return Some("fee".to_string()); } @@ -323,6 +329,9 @@ fn infer_event_family( ) { return Some("admin_config".to_string()); } + if normalized == "create_support_mint_associated" { + return Some("account_create".to_string()); + } if normalized.contains("mint") { return Some("mint".to_string()); } @@ -335,21 +344,24 @@ fn infer_event_family( if contains_any(normalized.as_str(), &["create_ata", "init_account", "open_orders_create"]) { return Some("account_create".to_string()); } - if contains_any(normalized.as_str(), &["close_account", "close_open_orders"]) - || normalized.starts_with("close_") - { - return Some("account_close".to_string()); - } if normalized.contains("wrap_sol") { return Some("wrap_sol".to_string()); } if normalized.contains("unwrap_sol") { return Some("unwrap_sol".to_string()); } - if normalized.contains("place_order") || normalized.contains("post_order") { + if normalized.contains("place_order") + || normalized.contains("post_order") + || normalized == "open_limit_order" + || normalized == "increase_limit_order" + { return Some("order_place".to_string()); } - if normalized.contains("cancel_order") || normalized.contains("cancel_all") { + if normalized.contains("cancel_order") + || normalized.contains("cancel_all") + || normalized == "close_limit_order" + || normalized == "decrease_limit_order" + { return Some("order_cancel".to_string()); } if normalized.contains("fill") { @@ -358,9 +370,14 @@ fn infer_event_family( if normalized.contains("consume_events") { return Some("consume_events".to_string()); } - if normalized.contains("settle_funds") { + if normalized.contains("settle_funds") || normalized == "settle_limit_order" { return Some("settle_funds".to_string()); } + if contains_any(normalized.as_str(), &["close_account", "close_open_orders"]) + || normalized.starts_with("close_") + { + return Some("account_close".to_string()); + } if normalized.contains("vault") && normalized.contains("deposit") { return Some("vault_deposit".to_string()); } @@ -399,7 +416,7 @@ fn contains_any(value: &str, needles: &[&str]) -> bool { return false; } -fn known_local_event_kind( +pub(crate) fn known_local_event_kind( decoder_code: &str, entry_name: &str, ) -> std::option::Option { @@ -444,19 +461,98 @@ fn known_local_event_kind( return Some("raydium_cpmm.update_pool_status".to_string()); }, ("raydium_cpmm", "withdraw") => return Some("raydium_cpmm.withdraw".to_string()), - ("raydium_clmm", "swap") => return Some("raydium_clmm.swap".to_string()), - ("raydium_clmm", "swap_v2") => return Some("raydium_clmm.swap_v2".to_string()), - ("raydium_clmm", "increase_liquidity_v2") => { - return Some("raydium_clmm.increase_liquidity_v2".to_string()); + ("raydium_clmm", "close_limit_order") => { + return Some("raydium_clmm.close_limit_order".to_string()); + }, + ("raydium_clmm", "open_limit_order") => { + return Some("raydium_clmm.open_limit_order".to_string()); + }, + ("raydium_clmm", "increase_limit_order") => { + return Some("raydium_clmm.increase_limit_order".to_string()); + }, + ("raydium_clmm", "decrease_limit_order") => { + return Some("raydium_clmm.decrease_limit_order".to_string()); + }, + ("raydium_clmm", "close_position") => { + return Some("raydium_clmm.close_position".to_string()); + }, + ("raydium_clmm", "close_protocol_position") => { + return Some("raydium_clmm.close_protocol_position".to_string()); + }, + ("raydium_clmm", "collect_fund_fee") => { + return Some("raydium_clmm.collect_fund_fee".to_string()); + }, + ("raydium_clmm", "collect_protocol_fee") => { + return Some("raydium_clmm.collect_protocol_fee".to_string()); + }, + ("raydium_clmm", "collect_remaining_rewards") => { + return Some("raydium_clmm.collect_remaining_rewards".to_string()); + }, + ("raydium_clmm", "create_amm_config") => { + return Some("raydium_clmm.create_amm_config".to_string()); + }, + ("raydium_clmm", "create_customizable_pool") => { + return Some("raydium_clmm.create_customizable_pool".to_string()); + }, + ("raydium_clmm", "create_dynamic_fee_config") => { + return Some("raydium_clmm.create_dynamic_fee_config".to_string()); + }, + ("raydium_clmm", "create_operation_account") => { + return Some("raydium_clmm.create_operation_account".to_string()); + }, + ("raydium_clmm", "create_pool") => return Some("raydium_clmm.create_pool".to_string()), + ("raydium_clmm", "create_support_mint_associated") => { + return Some("raydium_clmm.create_support_mint_associated".to_string()); + }, + ("raydium_clmm", "decrease_liquidity") => { + return Some("raydium_clmm.decrease_liquidity".to_string()); }, ("raydium_clmm", "decrease_liquidity_v2") => { return Some("raydium_clmm.decrease_liquidity_v2".to_string()); }, + ("raydium_clmm", "increase_liquidity") => { + return Some("raydium_clmm.increase_liquidity".to_string()); + }, + ("raydium_clmm", "increase_liquidity_v2") => { + return Some("raydium_clmm.increase_liquidity_v2".to_string()); + }, + ("raydium_clmm", "initialize_reward") => { + return Some("raydium_clmm.initialize_reward".to_string()); + }, + ("raydium_clmm", "open_position") => { + return Some("raydium_clmm.open_position".to_string()); + }, + ("raydium_clmm", "open_position_v2") => { + return Some("raydium_clmm.open_position_v2".to_string()); + }, ("raydium_clmm", "open_position_with_token22_nft") => { return Some("raydium_clmm.open_position_with_token22_nft".to_string()); }, - ("raydium_clmm", "close_position") => { - return Some("raydium_clmm.close_position".to_string()); + ("raydium_clmm", "set_reward_params") => { + return Some("raydium_clmm.set_reward_params".to_string()); + }, + ("raydium_clmm", "settle_limit_order") => { + return Some("raydium_clmm.settle_limit_order".to_string()); + }, + ("raydium_clmm", "swap") => return Some("raydium_clmm.swap".to_string()), + ("raydium_clmm", "swap_router_base_in") => { + return Some("raydium_clmm.swap_router_base_in".to_string()); + }, + ("raydium_clmm", "swap_v2") => return Some("raydium_clmm.swap_v2".to_string()), + ("raydium_clmm", "transfer_reward_owner") => { + return Some("raydium_clmm.transfer_reward_owner".to_string()); + }, + ("raydium_clmm", "update_amm_config") => { + return Some("raydium_clmm.update_amm_config".to_string()); + }, + ("raydium_clmm", "update_operation_account") => { + return Some("raydium_clmm.update_operation_account".to_string()); + }, + ("raydium_clmm", "update_pool_status") => { + return Some("raydium_clmm.update_pool_status".to_string()); + }, + ("raydium_clmm", "update_reward_infos") => { + return Some("raydium_clmm.update_reward_infos".to_string()); }, _ => return None, } @@ -511,6 +607,77 @@ mod tests { ); } + #[test] + fn event_family_inference_covers_raydium_clmm_idl_entries() { + assert_eq!( + super::infer_event_family("create_customizable_pool", crate::ENTRY_KIND_INSTRUCTION), + Some("pool_create".to_string()) + ); + assert_eq!( + super::infer_event_family("create_dynamic_fee_config", crate::ENTRY_KIND_INSTRUCTION), + Some("admin_config".to_string()) + ); + assert_eq!( + super::infer_event_family( + "create_support_mint_associated", + crate::ENTRY_KIND_INSTRUCTION, + ), + Some("account_create".to_string()) + ); + assert_eq!( + super::infer_event_family("close_limit_order", crate::ENTRY_KIND_INSTRUCTION), + Some("order_cancel".to_string()) + ); + assert_eq!( + super::infer_event_family("open_limit_order", crate::ENTRY_KIND_INSTRUCTION), + Some("order_place".to_string()) + ); + assert_eq!( + super::infer_event_family("increase_limit_order", crate::ENTRY_KIND_INSTRUCTION), + Some("order_place".to_string()) + ); + assert_eq!( + super::infer_event_family("decrease_limit_order", crate::ENTRY_KIND_INSTRUCTION), + Some("order_cancel".to_string()) + ); + assert_eq!( + super::infer_event_family("close_protocol_position", crate::ENTRY_KIND_INSTRUCTION), + Some("position_close".to_string()) + ); + assert_eq!( + super::infer_event_family("settle_limit_order", crate::ENTRY_KIND_INSTRUCTION), + Some("settle_funds".to_string()) + ); + assert_eq!( + super::infer_expected_db_target(Some("position_open"), crate::ENTRY_KIND_INSTRUCTION), + Some(crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS.to_string()) + ); + assert_eq!( + super::infer_expected_db_target(Some("position_close"), crate::ENTRY_KIND_INSTRUCTION), + Some(crate::DexEventCoverageEntryDto::DB_TARGET_LIQUIDITY_EVENTS.to_string()) + ); + assert_eq!( + super::known_local_event_kind("raydium_clmm", "create_pool"), + Some("raydium_clmm.create_pool".to_string()) + ); + assert_eq!( + super::known_local_event_kind("raydium_clmm", "collect_protocol_fee"), + Some("raydium_clmm.collect_protocol_fee".to_string()) + ); + assert_eq!( + super::known_local_event_kind("raydium_clmm", "open_limit_order"), + Some("raydium_clmm.open_limit_order".to_string()) + ); + assert_eq!( + super::known_local_event_kind("raydium_clmm", "increase_limit_order"), + Some("raydium_clmm.increase_limit_order".to_string()) + ); + assert_eq!( + super::known_local_event_kind("raydium_clmm", "decrease_limit_order"), + Some("raydium_clmm.decrease_limit_order".to_string()) + ); + } + #[tokio::test] async fn sync_upstream_registry_persists_raydium_cpmm_coverage_rows() { let database = make_database().await; diff --git a/kb_lib/src/instruction_observation_index.rs b/kb_lib/src/instruction_observation_index.rs index 2d28c06..4af28d8 100644 --- a/kb_lib/src/instruction_observation_index.rs +++ b/kb_lib/src/instruction_observation_index.rs @@ -60,6 +60,19 @@ impl InstructionObservationIndexService { return self.upsert_source_rows(rows).await; } + /// Refreshes observations for the same transaction window used by local replay. + pub async fn refresh_replay_window( + &self, + limit: std::option::Option, + ) -> Result { + let rows_result = self.list_replay_window_source_rows(limit).await; + let rows = match rows_result { + Ok(rows) => rows, + Err(error) => return Err(error), + }; + return self.upsert_source_rows(rows).await; + } + /// Refreshes observations for recently persisted instructions. pub async fn refresh_recent( &self, @@ -145,6 +158,73 @@ ORDER BY ins.instruction_index ASC, ins.inner_instruction_index ASC, ins.id ASC } } + async fn list_replay_window_source_rows( + &self, + limit: std::option::Option, + ) -> Result, crate::Error> { + let effective_limit = match limit { + Some(limit) => { + if limit <= 0 { + 10_000 + } else { + limit + } + }, + None => 10_000, + }; + match self.database.connection() { + crate::DatabaseConnection::Sqlite(pool) => { + let query_result = sqlx::query_as::( + r#" +WITH replay_transactions AS ( + SELECT id + FROM k_sol_chain_transactions + ORDER BY id ASC + LIMIT ? +) +SELECT + tx.id AS transaction_id, + tx.signature AS signature, + tx.slot AS slot, + tx.block_time_unix AS block_time, + tx.err_json AS err_json, + ins.id AS instruction_id, + ins.parent_instruction_id AS parent_instruction_id, + ins.instruction_index AS instruction_index, + ins.inner_instruction_index AS inner_instruction_index, + ins.program_id AS program_id, + ins.accounts_json AS accounts_json, + ins.data_json AS data_json, + de.pool_account AS pool_account, + de.event_kind AS decoded_event_kind, + de.id AS decoded_event_id +FROM k_sol_chain_instructions ins +JOIN replay_transactions replay_tx + ON replay_tx.id = ins.transaction_id +JOIN k_sol_chain_transactions tx + ON tx.id = ins.transaction_id +LEFT JOIN k_sol_dex_decoded_events de + ON de.transaction_id = tx.id + AND de.instruction_id = ins.id +ORDER BY tx.id ASC, ins.instruction_index ASC, ins.inner_instruction_index ASC, ins.id ASC + "#, + ) + .bind(effective_limit) + .fetch_all(pool) + .await; + match query_result { + Ok(rows) => return Ok(rows), + Err(error) => { + return Err(crate::Error::Db(format!( + "cannot list instruction observation source rows for replay window: {}", + error + ))); + }, + } + }, + } + } + async fn list_recent_source_rows( &self, limit: u32, @@ -280,6 +360,45 @@ fn resolve_instruction_name( }; return Some(name.to_string()); } + if program_id == crate::RAYDIUM_CLMM_PROGRAM_ID || decoder_code == Some("raydium_clmm") { + let name = match discriminator_hex { + "4c7c800fd55725fa" => "raydium_clmm.close_limit_order", + "9d20dab7471d1293" => "raydium_clmm.open_limit_order", + "b19059ecfaba7d63" => "raydium_clmm.increase_limit_order", + "759d3c674231a300" => "raydium_clmm.decrease_limit_order", + "7b86510031446262" => "raydium_clmm.close_position", + "c975989055556cb2" => "raydium_clmm.close_protocol_position", + "a78a4e95dfc2067e" => "raydium_clmm.collect_fund_fee", + "8888fcddc2427e59" => "raydium_clmm.collect_protocol_fee", + "12eda6c52210d590" => "raydium_clmm.collect_remaining_rewards", + "8934edd4d7756c68" => "raydium_clmm.create_amm_config", + "2b44d4a7592fa401" => "raydium_clmm.create_customizable_pool", + "bd0eb5785576e33e" => "raydium_clmm.create_dynamic_fee_config", + "3f5794216d230868" => "raydium_clmm.create_operation_account", + "e992d18ecf6840bc" => "raydium_clmm.create_pool", + "11fb415c88f20ea9" => "raydium_clmm.create_support_mint_associated", + "a026d06f685b2c01" => "raydium_clmm.decrease_liquidity", + "3a7fbc3e4f52c460" => "raydium_clmm.decrease_liquidity_v2", + "2e9cf3760dcdfbb2" => "raydium_clmm.increase_liquidity", + "851d59df45eeb00a" => "raydium_clmm.increase_liquidity_v2", + "5f87c0c4f281e644" => "raydium_clmm.initialize_reward", + "87802f4d0f98f031" => "raydium_clmm.open_position", + "4db84ad67056f1c7" => "raydium_clmm.open_position_v2", + "4dffae527d1dc92e" => "raydium_clmm.open_position_with_token22_nft", + "7034a74b20c9d389" => "raydium_clmm.set_reward_params", + "cd4e74215c691a60" => "raydium_clmm.settle_limit_order", + "f8c69e91e17587c8" => "raydium_clmm.swap", + "457d73daf5baf2c4" => "raydium_clmm.swap_router_base_in", + "2b04ed0b1ac91e62" => "raydium_clmm.swap_v2", + "07160c53f22b3079" => "raydium_clmm.transfer_reward_owner", + "313cae889a1c74c8" => "raydium_clmm.update_amm_config", + "7f467728bce33d07" => "raydium_clmm.update_operation_account", + "82576c062ee0757b" => "raydium_clmm.update_pool_status", + "a3ace0340b9a6adf" => "raydium_clmm.update_reward_infos", + _ => return None, + }; + return Some(name.to_string()); + } return None; } diff --git a/kb_lib/src/lib.rs b/kb_lib/src/lib.rs index 255cbe1..c4f129f 100644 --- a/kb_lib/src/lib.rs +++ b/kb_lib/src/lib.rs @@ -585,6 +585,9 @@ pub use db::ObservedTokenStatus; pub use db::OnchainObservationDto; /// Persisted on-chain observation row. pub use db::OnchainObservationEntity; +/// Application-facing normalized orderbook or limit-order event DTO. +pub use db::OrderbookEventDto; +pub use db::OrderbookEventEntity; /// Application-facing pair-analytic-signal DTO. pub use db::PairAnalyticSignalDto; /// Persisted pair-analytic-signal row. @@ -737,10 +740,13 @@ pub use db::query_dex_decode_replay_ledger_get_by_transaction; pub use db::query_dex_decode_replay_ledger_upsert; /// Deletes one decoded DEX event row by its natural key. pub use db::query_dex_decoded_events_delete_by_key; +/// Deletes upstream registry instruction-match rows already covered by specialized local decoders. +pub use db::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches; /// Deletes Meteora DLMM Anchor self-CPI swap audit rows already covered by decoded swaps. pub use db::query_dex_decoded_events_delete_meteora_dlmm_anchor_swap_instruction_audits; /// Deletes decoded DEX instruction audit rows related to one decoded instruction. pub use db::query_dex_decoded_events_delete_related_instruction_audit; +pub use db::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits; /// Reads one decoded DEX event by its natural key. pub use db::query_dex_decoded_events_get_by_key; /// Returns the latest Pump.fun create payload associated with a token mint. @@ -864,6 +870,8 @@ pub use db::query_observed_tokens_upsert; pub use db::query_onchain_observations_insert; /// Lists recent on-chain observations ordered from newest to oldest. pub use db::query_onchain_observations_list_recent; +/// Inserts or updates one normalized orderbook event row. +pub use db::query_orderbook_events_upsert; /// Returns one pair-analytic-signal row identified by its key, if it exists. pub use db::query_pair_analytic_signals_get_by_key; /// Lists all pair-analytic signals for one pair ordered by key. @@ -1138,12 +1146,18 @@ pub use dex::RaydiumAmmV4Decoder; pub use dex::RaydiumAmmV4Initialize2PoolDecoded; /// Decoded Raydium AMM v4 swap event. pub use dex::RaydiumAmmV4SwapDecoded; +/// Decoded Raydium CLMM collect_protocol_fee instruction. +pub use dex::RaydiumClmmCollectProtocolFeeDecoded; +/// Decoded Raydium CLMM create_pool instruction. +pub use dex::RaydiumClmmCreatePoolDecoded; /// Decoded Raydium CLMM event. pub use dex::RaydiumClmmDecodedEvent; /// Decoded Raydium CLMM instruction event with projected instruction id. pub use dex::RaydiumClmmDecodedInstructionEvent; /// Raydium CLMM transaction decoder. pub use dex::RaydiumClmmDecoder; +/// Decoded Raydium CLMM Anchor Program data event payload. +pub use dex::RaydiumClmmProgramDataEventDecoded; /// Decoded Raydium CLMM legacy swap event. pub use dex::RaydiumClmmSwapLegacyDecoded; /// Decoded Raydium CLMM swap_v2 instruction. @@ -1162,6 +1176,8 @@ pub use dex::RaydiumCpmmSwapMode; pub use dex::classify_raydium_cpmm_instruction_data; /// Decodes a Raydium CLMM instruction. pub use dex::decode_raydium_clmm_instruction; +/// Decodes one Raydium CLMM Anchor Program data event. +pub use dex::decode_raydium_clmm_program_data_event; /// Decodes one Raydium CPMM instruction from projected instruction fields. pub use dex::decode_raydium_cpmm_instruction; /// Decodes Raydium CPMM Anchor events emitted in `Program data:` logs. @@ -1217,6 +1233,8 @@ pub use dex_event_classification::is_dex_liquidity_event_kind; pub use dex_event_classification::is_dex_liquidity_remove_event_kind; /// Returns true for migration DEX events. pub use dex_event_classification::is_dex_migration_event_kind; +/// Returns true for orderbook or limit-order events that must not become candles. +pub use dex_event_classification::is_dex_orderbook_event_kind; /// Returns true for pair creation DEX events. pub use dex_event_classification::is_dex_pair_creation_event_kind; /// Returns true for pool creation DEX events. @@ -1408,6 +1426,10 @@ pub use solana_pubsub_ws::parse_solana_ws_typed_notification_from_event; pub use token_backfill::PoolBackfillResult; /// One signature-backfill result summary. pub use token_backfill::SignatureBackfillResult; +/// One item produced by a batch signature backfill. +pub use token_backfill::SignatureBatchBackfillItemResult; +/// Batch signature-backfill result summary. +pub use token_backfill::SignatureBatchBackfillResult; /// One token-backfill result summary. pub use token_backfill::TokenBackfillResult; /// Historical token backfill service. diff --git a/kb_lib/src/local_pipeline_replay.rs b/kb_lib/src/local_pipeline_replay.rs index 9f1613d..9835faf 100644 --- a/kb_lib/src/local_pipeline_replay.rs +++ b/kb_lib/src/local_pipeline_replay.rs @@ -13,6 +13,10 @@ fn default_skip_certified_dex_decode() -> bool { return true; } +fn default_defer_instruction_observation_index_refresh() -> bool { + return true; +} + /// Configuration for a local pipeline replay pass. #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] #[serde(rename_all = "camelCase")] @@ -31,6 +35,9 @@ pub struct LocalPipelineReplayConfig { /// Whether DEX decoding must run even when the replay ledger certifies a safe prior pass. #[serde(default)] pub force_decode_replay: bool, + /// Whether instruction observation indexing is deferred and refreshed once after replay. + #[serde(default = "default_defer_instruction_observation_index_refresh")] + pub defer_instruction_observation_index_refresh: bool, } impl Default for LocalPipelineReplayConfig { @@ -42,6 +49,7 @@ impl Default for LocalPipelineReplayConfig { reset_market_materialization_before_replay: true, skip_certified_dex_decode: true, force_decode_replay: false, + defer_instruction_observation_index_refresh: true, }; } } @@ -90,6 +98,8 @@ pub struct LocalPipelineReplayResult { pub reward_event_count: usize, /// Total pool administration event materialization results returned by replayed non-trade calls. pub pool_admin_event_count: usize, + /// Total orderbook event materialization results returned by replayed non-trade calls. + pub orderbook_event_count: usize, /// Total candle upsert results returned by replayed candle calls. /// /// This is a replay write/result counter, not the number of distinct rows @@ -111,6 +121,10 @@ pub struct LocalPipelineReplayResult { pub pair_symbol_updated_count: usize, /// Number of derived market materialization rows deleted before replay. pub reset_market_materialization_deleted_count: u64, + /// Total instruction source rows scanned by the observation index refresh. + pub instruction_observation_scanned_count: usize, + /// Total instruction-observation rows upserted by the observation index refresh. + pub instruction_observation_upserted_count: usize, /// Number of errors outside per-signature replay. pub global_error_count: usize, } @@ -352,6 +366,7 @@ impl LocalPipelineReplayService { result.fee_event_count += non_trade_result.fee_event_count; result.reward_event_count += non_trade_result.reward_event_count; result.pool_admin_event_count += non_trade_result.pool_admin_event_count; + result.orderbook_event_count += non_trade_result.orderbook_event_count; }, Err(error) => { result.non_trade_materialization_error_count += 1; @@ -426,25 +441,55 @@ impl LocalPipelineReplayService { ); }, } + if !config.defer_instruction_observation_index_refresh { + let instruction_index_result = + instruction_observation_index.refresh_signature(signature.as_str()).await; + match instruction_index_result { + Ok(index_result) => { + result.instruction_observation_scanned_count += + index_result.scanned_instruction_count; + result.instruction_observation_upserted_count += + index_result.upserted_observation_count; + tracing::debug!( + signature = %signature, + upserted_observation_count = index_result.upserted_observation_count, + "instruction observation index refreshed during local replay" + ); + }, + Err(error) => { + tracing::warn!( + signature = %signature, + error = %error, + "instruction observation index refresh failed during local replay" + ); + }, + } + } + result.replayed_transaction_count += 1; + } + if config.defer_instruction_observation_index_refresh { let instruction_index_result = - instruction_observation_index.refresh_signature(signature.as_str()).await; + instruction_observation_index.refresh_replay_window(config.limit).await; match instruction_index_result { Ok(index_result) => { + result.instruction_observation_scanned_count += + index_result.scanned_instruction_count; + result.instruction_observation_upserted_count += + index_result.upserted_observation_count; tracing::debug!( - signature = %signature, + scanned_instruction_count = index_result.scanned_instruction_count, upserted_observation_count = index_result.upserted_observation_count, - "instruction observation index refreshed during local replay" + "instruction observation index refreshed after local replay" ); }, Err(error) => { + result.global_error_count += 1; tracing::warn!( - signature = %signature, error = %error, - "instruction observation index refresh failed during local replay" + "instruction observation index refresh failed after local replay" ); }, } - result.replayed_transaction_count += 1; } if config.refresh_missing_token_metadata { let metadata_service = match &self.http_pool { @@ -476,6 +521,52 @@ impl LocalPipelineReplayService { } async fn refresh_event_coverage_best_effort(&self) { + let cleanup_result = + crate::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits( + self.database.as_ref(), + None, + ) + .await; + match cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::info!( + deleted_count = deleted_count, + "replaced Raydium CLMM instruction audits cleaned before dex event coverage refresh" + ); + } + }, + Err(error) => { + tracing::warn!( + error = %error, + "Raydium CLMM replaced instruction-audit cleanup failed before dex event coverage refresh" + ); + }, + } + + let upstream_cleanup_result = + crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches( + self.database.as_ref(), + None, + ) + .await; + match upstream_cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::info!( + deleted_count = deleted_count, + "locally covered upstream instruction matches cleaned before dex event coverage refresh" + ); + } + }, + Err(error) => { + tracing::warn!( + error = %error, + "locally covered upstream instruction-match cleanup failed before dex event coverage refresh" + ); + }, + } + let coverage_service = crate::DexEventCoverageService::new(self.database.clone()); let refresh_result = coverage_service.refresh_local_counts(None).await; match refresh_result { @@ -494,6 +585,46 @@ impl LocalPipelineReplayService { ); }, } + + let post_refresh_upstream_cleanup_result = + crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches( + self.database.as_ref(), + None, + ) + .await; + match post_refresh_upstream_cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::info!( + deleted_count = deleted_count, + "locally covered upstream instruction matches cleaned after dex event coverage refresh" + ); + let second_refresh_result = coverage_service.refresh_local_counts(None).await; + match second_refresh_result { + Ok(second_refresh_result) => { + tracing::debug!( + upserted_entry_count = second_refresh_result.upserted_entry_count, + refreshed_entry_count = second_refresh_result.refreshed_entry_count, + summary_count = second_refresh_result.summaries.len(), + "dex event coverage refreshed after upstream instruction-match cleanup" + ); + }, + Err(error) => { + tracing::warn!( + error = %error, + "dex event coverage refresh failed after upstream instruction-match cleanup" + ); + }, + } + } + }, + Err(error) => { + tracing::warn!( + error = %error, + "locally covered upstream instruction-match cleanup failed after dex event coverage refresh" + ); + }, + } } async fn get_certified_dex_decode_skip_ledger( diff --git a/kb_lib/src/non_trade_event_materialization.rs b/kb_lib/src/non_trade_event_materialization.rs index 87d1f05..03e9f06 100644 --- a/kb_lib/src/non_trade_event_materialization.rs +++ b/kb_lib/src/non_trade_event_materialization.rs @@ -19,6 +19,8 @@ pub struct NonTradeEventMaterializationResult { pub reward_event_count: usize, /// Number of pool administration events inserted or refreshed. pub pool_admin_event_count: usize, + /// Number of orderbook or limit-order events inserted or refreshed. + pub orderbook_event_count: usize, } /// Materializes useful non-trade decoded DEX events. @@ -61,7 +63,7 @@ impl NonTradeEventMaterializationService { ))); }, }; - if transaction.err_json.is_some() { + if transaction_has_effective_error(&transaction) { tracing::debug!( signature = %transaction.signature, "skipping non-trade materialization for failed transaction" @@ -189,6 +191,24 @@ impl NonTradeEventMaterializationService { Err(error) => return Err(error), } } + if crate::is_dex_orderbook_event_kind(decoded_event.event_kind.as_str()) { + let materialized = self + .materialize_orderbook_event( + &transaction, + transaction_id, + decoded_event, + &payload, + ) + .await; + match materialized { + Ok(was_materialized) => { + if was_materialized { + result.orderbook_event_count += 1; + } + }, + Err(error) => return Err(error), + } + } } for decoded_event in &decoded_events { if !decoded_event.event_kind.ends_with(".lp_change_event") { @@ -673,6 +693,86 @@ WHERE decoded_event_id = ? } } + async fn materialize_orderbook_event( + &self, + transaction: &crate::ChainTransactionDto, + transaction_id: i64, + decoded_event: &crate::DexDecodedEventDto, + payload: &serde_json::Value, + ) -> Result { + let decoded_event_id = match decoded_event.id { + Some(decoded_event_id) => decoded_event_id, + None => return Ok(false), + }; + let context = self.resolve_decoded_event_context(decoded_event).await; + let context = match context { + Ok(context) => context, + Err(error) => return Err(error), + }; + let order_action = normalize_orderbook_action(decoded_event.event_kind.as_str()); + let actor_wallet = extract_first_string( + payload, + &["actorWallet", "actor_wallet", "owner", "authority", "payer", "user"], + ); + let order_account = match extract_first_string( + payload, + &["orderAccount", "order_account", "limitOrder", "limit_order", "order"], + ) { + Some(order_account) => Some(order_account), + None => fallback_order_account(decoded_event.event_kind.as_str(), payload), + }; + let amount_raw = extract_first_amount_string( + payload, + &[ + "amountRaw", + "amount_raw", + "amount", + "decreasedAmountRaw", + "decreased_amount_raw", + "decreasedAmount", + "increasedAmountRaw", + "increased_amount_raw", + "increasedAmount", + ], + ); + let amount_min_raw = extract_first_amount_string( + payload, + &["amountMinRaw", "amount_min_raw", "amountMin", "amount_min"], + ); + let tick_index = extract_first_i64(payload, &["tickIndex", "tick_index"]); + let zero_for_one = extract_first_bool(payload, &["zeroForOne", "zero_for_one"]); + let dto = crate::OrderbookEventDto::new( + transaction_id, + Some(decoded_event_id), + context.dex_id, + context.pool_id, + context.pair_id, + transaction.signature.clone(), + transaction.slot, + decoded_event.protocol_name.clone(), + decoded_event.program_id.clone(), + decoded_event.event_kind.clone(), + order_action, + decoded_event.pool_account.clone(), + decoded_event.market_account.clone(), + actor_wallet, + order_account, + decoded_event.token_a_mint.clone(), + decoded_event.token_b_mint.clone(), + amount_raw, + amount_min_raw, + tick_index, + zero_for_one, + decoded_event.payload_json.clone(), + ); + let upsert_result = + crate::query_orderbook_events_upsert(self.database.as_ref(), &dto).await; + match upsert_result { + Ok(_) => return Ok(true), + Err(error) => return Err(error), + } + } + async fn ensure_liquidity_context_from_decoded_event( &self, decoded_event: &crate::DexDecodedEventDto, @@ -789,6 +889,162 @@ WHERE decoded_event_id = ? } } +fn normalize_orderbook_action(event_kind: &str) -> std::string::String { + if event_kind.contains(".open_limit_order") { + return "open_limit_order".to_string(); + } + if event_kind.contains(".increase_limit_order") { + return "increase_limit_order".to_string(); + } + if event_kind.contains(".decrease_limit_order") { + return "decrease_limit_order".to_string(); + } + if event_kind.contains(".close_limit_order") { + return "close_limit_order".to_string(); + } + if event_kind.contains(".settle_limit_order") { + return "settle_limit_order".to_string(); + } + if event_kind.contains("order_place") { + return "order_place".to_string(); + } + if event_kind.contains("order_cancel") { + return "order_cancel".to_string(); + } + if event_kind.contains("settle_funds") { + return "settle_funds".to_string(); + } + return event_kind.to_string(); +} + +fn fallback_order_account( + event_kind: &str, + payload: &serde_json::Value, +) -> std::option::Option { + if event_kind.contains(".close_limit_order") { + return extract_account_at(payload, 2); + } + if event_kind.contains(".open_limit_order") + || event_kind.contains(".increase_limit_order") + || event_kind.contains(".decrease_limit_order") + { + return extract_account_at(payload, 3); + } + return None; +} + +fn extract_account_at( + value: &serde_json::Value, + index: usize, +) -> std::option::Option { + if let Some(object) = value.as_object() { + let accounts = object.get("accounts"); + if let Some(accounts) = accounts { + if let Some(array) = accounts.as_array() { + let candidate = array.get(index); + if let Some(candidate) = candidate { + if let Some(text) = candidate.as_str() { + let trimmed = text.trim(); + if !trimmed.is_empty() { + return Some(trimmed.to_string()); + } + } + } + } + } + for nested_value in object.values() { + let nested = extract_account_at(nested_value, index); + if nested.is_some() { + return nested; + } + } + } + return None; +} + +fn extract_first_i64( + value: &serde_json::Value, + candidate_keys: &[&str], +) -> std::option::Option { + if let Some(object) = value.as_object() { + for candidate_key in candidate_keys { + let candidate_value = object.get(*candidate_key); + if let Some(candidate_value) = candidate_value { + if let Some(number) = candidate_value.as_i64() { + return Some(number); + } + if let Some(number) = candidate_value.as_u64() { + let converted = i64::try_from(number); + if let Ok(converted) = converted { + return Some(converted); + } + } + if let Some(text) = candidate_value.as_str() { + let parsed = text.parse::(); + if let Ok(parsed) = parsed { + return Some(parsed); + } + } + } + } + for nested_value in object.values() { + let nested = extract_first_i64(nested_value, candidate_keys); + if nested.is_some() { + return nested; + } + } + } + return None; +} + +fn extract_first_bool( + value: &serde_json::Value, + candidate_keys: &[&str], +) -> std::option::Option { + if let Some(object) = value.as_object() { + for candidate_key in candidate_keys { + let candidate_value = object.get(*candidate_key); + if let Some(candidate_value) = candidate_value { + if let Some(flag) = candidate_value.as_bool() { + return Some(flag); + } + if let Some(number) = candidate_value.as_i64() { + return Some(number != 0); + } + if let Some(text) = candidate_value.as_str() { + if text == "true" || text == "1" { + return Some(true); + } + if text == "false" || text == "0" { + return Some(false); + } + } + } + } + for nested_value in object.values() { + let nested = extract_first_bool(nested_value, candidate_keys); + if nested.is_some() { + return nested; + } + } + } + return None; +} + +fn transaction_has_effective_error(transaction: &crate::ChainTransactionDto) -> bool { + let err_json = match transaction.err_json.as_ref() { + Some(err_json) => err_json.trim(), + None => return false, + }; + if err_json.is_empty() { + return false; + } + if err_json == "null" { + return false; + } + return true; +} + fn extract_first_u64( value: &serde_json::Value, candidate_keys: &[&str], @@ -902,6 +1158,28 @@ fn extract_first_number_as_string( #[cfg(test)] mod tests { + + #[test] + fn blank_or_null_err_json_is_not_effective_failure() { + let mut transaction = crate::ChainTransactionDto::new( + "sig-non-trade-effective-error".to_string(), + Some(1), + None, + None, + None, + None, + None, + "{}".to_string(), + ); + assert!(!super::transaction_has_effective_error(&transaction)); + transaction.err_json = Some("".to_string()); + assert!(!super::transaction_has_effective_error(&transaction)); + transaction.err_json = Some("null".to_string()); + assert!(!super::transaction_has_effective_error(&transaction)); + transaction.err_json = Some("{\"InstructionError\":[0,\"Custom\"]}".to_string()); + assert!(super::transaction_has_effective_error(&transaction)); + } + #[test] fn extracts_nested_liquidity_amounts() { let payload = serde_json::json!({ diff --git a/kb_lib/src/onchain_dex_pair_discovery.rs b/kb_lib/src/onchain_dex_pair_discovery.rs index 7fa8806..10336a5 100644 --- a/kb_lib/src/onchain_dex_pair_discovery.rs +++ b/kb_lib/src/onchain_dex_pair_discovery.rs @@ -1279,6 +1279,53 @@ fn decode_raydium_clmm_candidate( ), }); }, + crate::RaydiumClmmDecodedEvent::CreatePool(event) => { + return Some(crate::OnchainDexPairCandidateDto { + signature: signature.to_string(), + slot, + block_time, + failed, + program_id: program_id.to_string(), + dex_code, + candidate_kind: "create_pool".to_string(), + confidence: "high".to_string(), + instruction_index: instruction.instruction_index, + inner_instruction_index: instruction.inner_instruction_index, + instruction_name: Some("raydium_clmm.create_pool".to_string()), + instruction_data_prefix: instruction_data_prefix(instruction.data.as_deref()), + instruction_discriminator_hex: instruction_discriminator_hex( + instruction.data.as_deref(), + ), + pool_address: Some(event.pool_state.clone()), + token_a_mint: Some(event.token_mint_0), + token_b_mint: Some(event.token_mint_1), + verified_pool_address: Some(event.pool_state.clone()), + observed_token_mints: std::vec::Vec::new(), + token_balance_deltas: std::vec::Vec::new(), + candidate_pool_accounts: std::vec::Vec::new(), + candidate_token_vault_accounts: std::vec::Vec::new(), + candidate_program_accounts: std::vec::Vec::new(), + account_samples: sample_strings(instruction.accounts.as_slice(), 12), + log_samples: sample_logs(logs, 8), + backfill_hint: build_backfill_hint( + "pool", + Some(event.pool_state.as_str()), + signature, + ), + }); + }, + crate::RaydiumClmmDecodedEvent::CollectProtocolFee(_) + | crate::RaydiumClmmDecodedEvent::CollectPersonalFeeEvent(_) + | crate::RaydiumClmmDecodedEvent::CollectProtocolFeeEvent(_) + | crate::RaydiumClmmDecodedEvent::ConfigChangeEvent(_) + | crate::RaydiumClmmDecodedEvent::CreatePersonalPositionEvent(_) + | crate::RaydiumClmmDecodedEvent::DecreaseLiquidityEvent(_) + | crate::RaydiumClmmDecodedEvent::IncreaseLiquidityEvent(_) + | crate::RaydiumClmmDecodedEvent::LiquidityCalculateEvent(_) + | crate::RaydiumClmmDecodedEvent::LiquidityChangeEvent(_) + | crate::RaydiumClmmDecodedEvent::PoolCreatedEvent(_) + | crate::RaydiumClmmDecodedEvent::SwapEvent(_) + | crate::RaydiumClmmDecodedEvent::UpdateRewardInfosEvent(_) => return None, } } return None; diff --git a/kb_lib/src/token_backfill.rs b/kb_lib/src/token_backfill.rs index 4983fdc..e85e363 100644 --- a/kb_lib/src/token_backfill.rs +++ b/kb_lib/src/token_backfill.rs @@ -131,6 +131,68 @@ pub struct SignatureBackfillResult { pub pair_candle_count: usize, } +/// One item produced by a batch signature backfill. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct SignatureBatchBackfillItemResult { + /// Input transaction signature. + pub signature: std::string::String, + /// Whether the signature was replayed successfully. + pub success: bool, + /// Error text when this signature failed before a replay result could be produced. + pub error: std::option::Option, + /// Per-signature replay result when available. + pub result: std::option::Option, +} + +/// Batch signature-backfill result summary. +#[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] +pub struct SignatureBatchBackfillResult { + /// Number of raw signatures submitted by the UI. + pub input_signature_count: usize, + /// Number of unique non-empty signatures processed. + pub unique_signature_count: usize, + /// Number of successfully replayed signatures. + pub success_count: usize, + /// Number of signatures that failed before a replay result could be produced. + pub failure_count: usize, + /// Whether processing stopped after the first hard failure. + pub aborted: bool, + /// Number of transactions resolved through HTTP during this run. + pub resolved_transaction_count: usize, + /// Number of signatures whose `getTransaction` lookup returned `null`. + pub missing_transaction_count: usize, + /// Number of signatures whose `getTransaction` lookup failed after retries. + pub transaction_fetch_error_count: usize, + /// Last transaction fetch error observed during this run, if any. + pub last_transaction_fetch_error: std::option::Option, + /// Total number of decoded DEX events replayed during this run. + pub decoded_event_count: usize, + /// Total number of DEX detection results produced during this run. + pub detection_count: usize, + /// Total number of launch-attribution results produced during this run. + pub launch_attribution_count: usize, + /// Total number of pool-origin results produced during this run. + pub pool_origin_count: usize, + /// Total number of wallet-participation observations produced during this run. + pub wallet_participation_count: usize, + /// Total number of trade-aggregation results produced during this run. + pub trade_event_count: usize, + /// Total number of liquidity event materialization results produced during this run. + pub liquidity_event_count: usize, + /// Total number of pool lifecycle event materialization results produced during this run. + pub pool_lifecycle_event_count: usize, + /// Total number of fee event materialization results produced during this run. + pub fee_event_count: usize, + /// Total number of reward event materialization results produced during this run. + pub reward_event_count: usize, + /// Total number of pool administration event materialization results produced during this run. + pub pool_admin_event_count: usize, + /// Total number of pair-candle aggregation results produced during this run. + pub pair_candle_count: usize, + /// Detailed per-signature results. + pub items: std::vec::Vec, +} + /// Historical token backfill service. /// /// This service reuses the existing transaction projection and downstream @@ -878,6 +940,178 @@ impl TokenBackfillService { return Ok(result); } + /// Replays a batch of known transaction signatures through the existing pipeline. + /// + /// Unlike [`Self::backfill_signature`], this method refreshes token metadata and + /// event coverage only once after the whole batch has been processed. This keeps + /// manual discovery backfills responsive when many signatures were collected from + /// an external explorer. + pub async fn backfill_signatures( + &self, + signatures: &[std::string::String], + continue_on_error: bool, + ) -> Result { + let mut result = crate::SignatureBatchBackfillResult { + input_signature_count: signatures.len(), + unique_signature_count: 0, + success_count: 0, + failure_count: 0, + aborted: false, + resolved_transaction_count: 0, + missing_transaction_count: 0, + transaction_fetch_error_count: 0, + last_transaction_fetch_error: None, + decoded_event_count: 0, + detection_count: 0, + launch_attribution_count: 0, + pool_origin_count: 0, + wallet_participation_count: 0, + trade_event_count: 0, + liquidity_event_count: 0, + pool_lifecycle_event_count: 0, + fee_event_count: 0, + reward_event_count: 0, + pool_admin_event_count: 0, + pair_candle_count: 0, + items: std::vec::Vec::new(), + }; + let mut seen = std::collections::BTreeSet::::new(); + for signature in signatures { + let trimmed_signature = signature.trim().to_string(); + if trimmed_signature.is_empty() { + continue; + } + if seen.contains(trimmed_signature.as_str()) { + continue; + } + seen.insert(trimmed_signature.clone()); + result.unique_signature_count += 1; + let replay_result = self.replay_signature(trimmed_signature.clone()).await; + let replay = match replay_result { + Ok(replay) => replay, + Err(error) => { + result.failure_count += 1; + result.items.push(crate::SignatureBatchBackfillItemResult { + signature: trimmed_signature.clone(), + success: false, + error: Some(error.to_string()), + result: None, + }); + if !continue_on_error { + result.aborted = true; + break; + } + continue; + }, + }; + let signature_result = crate::SignatureBackfillResult { + signature: trimmed_signature.clone(), + resolved_transaction_count: replay.resolved_transaction_count, + missing_transaction_count: replay.missing_transaction_count, + transaction_fetch_error_count: replay.transaction_fetch_error_count, + last_transaction_fetch_error: replay.last_transaction_fetch_error.clone(), + decoded_event_count: replay.decoded_event_count, + detection_count: replay.detection_count, + launch_attribution_count: replay.launch_attribution_count, + pool_origin_count: replay.pool_origin_count, + wallet_participation_count: replay.wallet_participation_count, + trade_event_count: replay.trade_event_count, + liquidity_event_count: replay.liquidity_event_count, + pool_lifecycle_event_count: replay.pool_lifecycle_event_count, + fee_event_count: replay.fee_event_count, + reward_event_count: replay.reward_event_count, + pool_admin_event_count: replay.pool_admin_event_count, + pair_candle_count: replay.pair_candle_count, + }; + result.success_count += 1; + result.resolved_transaction_count += signature_result.resolved_transaction_count; + result.missing_transaction_count += signature_result.missing_transaction_count; + result.transaction_fetch_error_count += signature_result.transaction_fetch_error_count; + if signature_result.last_transaction_fetch_error.is_some() { + result.last_transaction_fetch_error = + signature_result.last_transaction_fetch_error.clone(); + } + result.decoded_event_count += signature_result.decoded_event_count; + result.detection_count += signature_result.detection_count; + result.launch_attribution_count += signature_result.launch_attribution_count; + result.pool_origin_count += signature_result.pool_origin_count; + result.wallet_participation_count += signature_result.wallet_participation_count; + result.trade_event_count += signature_result.trade_event_count; + result.liquidity_event_count += signature_result.liquidity_event_count; + result.pool_lifecycle_event_count += signature_result.pool_lifecycle_event_count; + result.fee_event_count += signature_result.fee_event_count; + result.reward_event_count += signature_result.reward_event_count; + result.pool_admin_event_count += signature_result.pool_admin_event_count; + result.pair_candle_count += signature_result.pair_candle_count; + result.items.push(crate::SignatureBatchBackfillItemResult { + signature: trimmed_signature, + success: true, + error: None, + result: Some(signature_result), + }); + } + if result.unique_signature_count == 0 { + return Err(crate::Error::Config( + "signature batch must contain at least one non-empty signature".to_string(), + )); + } + self.backfill_missing_token_metadata_best_effort(100).await; + self.refresh_event_coverage_best_effort().await; + let summary_payload = serde_json::json!({ + "inputSignatureCount": result.input_signature_count, + "uniqueSignatureCount": result.unique_signature_count, + "successCount": result.success_count, + "failureCount": result.failure_count, + "aborted": result.aborted, + "resolvedTransactionCount": result.resolved_transaction_count, + "missingTransactionCount": result.missing_transaction_count, + "transactionFetchErrorCount": result.transaction_fetch_error_count, + "lastTransactionFetchError": result.last_transaction_fetch_error, + "decodedEventCount": result.decoded_event_count, + "detectionCount": result.detection_count, + "launchAttributionCount": result.launch_attribution_count, + "poolOriginCount": result.pool_origin_count, + "walletParticipationCount": result.wallet_participation_count, + "tradeEventCount": result.trade_event_count, + "liquidityEventCount": result.liquidity_event_count, + "poolLifecycleEventCount": result.pool_lifecycle_event_count, + "feeEventCount": result.fee_event_count, + "rewardEventCount": result.reward_event_count, + "poolAdminEventCount": result.pool_admin_event_count, + "pairCandleCount": result.pair_candle_count + }); + let observation_result = self + .persistence + .record_observation(&crate::DetectionObservationInput::new( + "signature_batch.backfill.completed".to_string(), + crate::ObservationSourceKind::HttpRpc, + Some(format!("backfill:{}", self.http_role)), + format!("{} signatures", result.unique_signature_count), + None, + summary_payload.clone(), + )) + .await; + let observation_id = match observation_result { + Ok(observation_id) => observation_id, + Err(error) => return Err(error), + }; + let signal_result = self + .persistence + .record_signal(&crate::DetectionSignalInput::new( + "signal.signature_batch.backfill.completed".to_string(), + crate::AnalysisSignalSeverity::Low, + format!("{} signatures", result.unique_signature_count), + Some(observation_id), + None, + summary_payload, + )) + .await; + if let Err(error) = signal_result { + return Err(error); + } + return Ok(result); + } + async fn fetch_transaction_value_with_retry( &self, signature: &str, @@ -943,6 +1177,52 @@ impl TokenBackfillService { } async fn refresh_event_coverage_best_effort(&self) { + let cleanup_result = + crate::query_dex_decoded_events_delete_replaced_raydium_clmm_instruction_audits( + self.database.as_ref(), + None, + ) + .await; + match cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::info!( + deleted_count = deleted_count, + "replaced Raydium CLMM instruction audits cleaned before dex event coverage refresh" + ); + } + }, + Err(error) => { + tracing::warn!( + error = %error, + "Raydium CLMM replaced instruction-audit cleanup failed before dex event coverage refresh" + ); + }, + } + + let upstream_cleanup_result = + crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches( + self.database.as_ref(), + None, + ) + .await; + match upstream_cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::info!( + deleted_count = deleted_count, + "locally covered upstream instruction matches cleaned before dex event coverage refresh" + ); + } + }, + Err(error) => { + tracing::warn!( + error = %error, + "locally covered upstream instruction-match cleanup failed before dex event coverage refresh" + ); + }, + } + let coverage_service = crate::DexEventCoverageService::new(self.database.clone()); let refresh_result = coverage_service.refresh_local_counts(None).await; match refresh_result { @@ -960,6 +1240,45 @@ impl TokenBackfillService { ); }, } + + let post_refresh_upstream_cleanup_result = + crate::query_dex_decoded_events_delete_locally_covered_upstream_instruction_matches( + self.database.as_ref(), + None, + ) + .await; + match post_refresh_upstream_cleanup_result { + Ok(deleted_count) => { + if deleted_count > 0 { + tracing::info!( + deleted_count = deleted_count, + "locally covered upstream instruction matches cleaned after dex event coverage refresh" + ); + let second_refresh_result = coverage_service.refresh_local_counts(None).await; + match second_refresh_result { + Ok(second_refresh_result) => { + tracing::debug!( + upserted_entry_count = second_refresh_result.upserted_entry_count, + summary_count = second_refresh_result.summaries.len(), + "dex event coverage refreshed after upstream instruction-match cleanup" + ); + }, + Err(error) => { + tracing::warn!( + error = %error, + "dex event coverage refresh failed after upstream instruction-match cleanup" + ); + }, + } + } + }, + Err(error) => { + tracing::warn!( + error = %error, + "locally covered upstream instruction-match cleanup failed after dex event coverage refresh" + ); + }, + } } } diff --git a/kb_lib/src/upstream_registry_generated.rs b/kb_lib/src/upstream_registry_generated.rs index b568fff..eeebd46 100644 --- a/kb_lib/src/upstream_registry_generated.rs +++ b/kb_lib/src/upstream_registry_generated.rs @@ -12,6 +12,37 @@ const UPSTREAM_GIT_PROGRAM_NOTES: &str = "program id extracted from upstream Git const UPSTREAM_GIT_DISCRIMINATOR_NOTES: &str = "entry name and discriminator extracted from upstream Git decoder source or from the discriminator convention used by that upstream decoder; not corpus-verified; no trade/candle/materialization proof"; const UPSTREAM_GIT_ALIAS_PROGRAM_NOTES: &str = "upstream Git decoder name kept as a discovery alias; program id and discriminator rows are represented by the canonical decoder entry to avoid duplicate registry keys"; +const RAYDIUM_IDL_SOURCE_REPO: &str = "raydium-io/raydium-idl"; + +const RAYDIUM_IDL_DISCRIMINATOR_NOTES: &str = "entry name and discriminator extracted from Raydium official IDL snapshot; not corpus-verified; no trade/candle/materialization proof"; + +const fn raydium_idl_discriminator_entry( + decoder_code: &'static str, + program_id: std::option::Option<&'static str>, + program_family: &'static str, + surface_kind: &'static str, + entry_kind: &'static str, + entry_name: &'static str, + discriminator_hex: &'static str, + discriminator_len: u16, + source_path: &'static str, +) -> crate::UpstreamRegistryEntry { + return crate::UpstreamRegistryEntry { + source_repo: Some(RAYDIUM_IDL_SOURCE_REPO), + source_path: Some(source_path), + decoder_code, + program_id, + program_family, + surface_kind, + entry_kind, + entry_name, + discriminator_hex: Some(discriminator_hex), + discriminator_len: Some(discriminator_len), + proof_status: crate::PROOF_STATUS_UPSTREAM_GIT_UNVERIFIED, + notes: RAYDIUM_IDL_DISCRIMINATOR_NOTES, + }; +} + const fn upstream_git_program_entry( decoder_code: &'static str, program_id: std::option::Option<&'static str>, @@ -11775,6 +11806,61 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[ 8, "decoders/raydium-clmm-decoder/src/instructions/close_position.rs", ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "close_limit_order", + "4c7c800fd55725fa", + 8, + "raydium_clmm/raydium_clmm.json", + ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "open_limit_order", + "9d20dab7471d1293", + 8, + "raydium_clmm/raydium_clmm.json", + ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "increase_limit_order", + "b19059ecfaba7d63", + 8, + "raydium_clmm/raydium_clmm.json", + ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "decrease_limit_order", + "759d3c674231a300", + 8, + "raydium_clmm/raydium_clmm.json", + ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "close_protocol_position", + "c975989055556cb2", + 8, + "raydium_clmm/raydium_clmm.json", + ), upstream_git_discriminator_entry( "raydium_clmm", Some(crate::RAYDIUM_CLMM_PROGRAM_ID), @@ -11852,6 +11938,28 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[ 8, "decoders/raydium-clmm-decoder/src/instructions/create_amm_config.rs", ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "create_customizable_pool", + "2b44d4a7592fa401", + 8, + "raydium_clmm/raydium_clmm.json", + ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "create_dynamic_fee_config", + "bd0eb5785576e33e", + 8, + "raydium_clmm/raydium_clmm.json", + ), upstream_git_discriminator_entry( "raydium_clmm", Some(crate::RAYDIUM_CLMM_PROGRAM_ID), @@ -11885,6 +11993,17 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[ 8, "decoders/raydium-clmm-decoder/src/instructions/create_pool.rs", ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "create_support_mint_associated", + "11fb415c88f20ea9", + 8, + "raydium_clmm/raydium_clmm.json", + ), upstream_git_discriminator_entry( "raydium_clmm", Some(crate::RAYDIUM_CLMM_PROGRAM_ID), @@ -12039,6 +12158,17 @@ pub(crate) const UPSTREAM_REGISTRY_ENTRIES: &[crate::UpstreamRegistryEntry] = &[ 8, "decoders/raydium-clmm-decoder/src/instructions/set_reward_params.rs", ), + raydium_idl_discriminator_entry( + "raydium_clmm", + Some(crate::RAYDIUM_CLMM_PROGRAM_ID), + "raydium", + "clmm", + crate::ENTRY_KIND_INSTRUCTION, + "settle_limit_order", + "cd4e74215c691a60", + 8, + "raydium_clmm/raydium_clmm.json", + ), upstream_git_discriminator_entry( "raydium_clmm", Some(crate::RAYDIUM_CLMM_PROGRAM_ID), diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql new file mode 100644 index 0000000..753a30d --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql @@ -0,0 +1,132 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49.sql +-- Raydium CLMM final validation SQL for 0.7.49. + +-- 1. CLMM coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Residual CLMM instruction audits. Expected: zero rows. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 3. Redundant upstream fallback matches for locally covered CLMM entries. Expected: zero rows. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT ug.transaction_id) AS tx_count +FROM k_sol_dex_decoded_events ug +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' + AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 4. Instruction-observation links still pointing to redundant upstream fallback rows. Expected: zero rows. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS upstream_discriminator_hex, + COUNT(*) AS linked_observation_count +FROM k_sol_instruction_observations io +JOIN k_sol_dex_decoded_events ug + ON ug.id = io.decoded_event_id +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' + AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY + json_extract(ug.payload_json, '$.upstreamDecoderCode'), + json_extract(ug.payload_json, '$.upstreamEntryName'), + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') +ORDER BY linked_observation_count DESC, entry_name; + +-- 5. Any non-swap CLMM trade. Expected: zero rows. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind NOT IN ( + 'raydium_clmm.swap', + 'raydium_clmm.swap_v2' + ) +GROUP BY de.event_kind +HAVING COUNT(te.id) > 0 +ORDER BY trade_count DESC, de.event_kind; + +-- 6. Failed transaction materialization guard. Expected: zero rows. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +HAVING + COUNT(le.id) > 0 + OR COUNT(fe.id) > 0 + OR COUNT(re.id) > 0 + OR COUNT(pa.id) > 0 + OR COUNT(ple.id) > 0 + OR COUNT(oe.id) > 0 + OR COUNT(te.id) > 0 +ORDER BY de.event_kind; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE10.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE10.sql new file mode 100644 index 0000000..07b54d6 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE10.sql @@ -0,0 +1,111 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE10.sql +-- Raydium CLMM validation after pre.10 mapped instruction expansion. + +-- 1. Coverage rows. +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +ORDER BY entry_kind, entry_name, discriminator_hex; + +-- 2. Coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 3. Instruction observations, including unknown discriminants. +SELECT + instruction_name, + discriminator_hex, + COUNT(*) AS observed_count, + COUNT(DISTINCT signature) AS tx_count +FROM k_sol_instruction_observations +WHERE decoder_code = 'raydium_clmm' +GROUP BY instruction_name, discriminator_hex +ORDER BY observed_count DESC, instruction_name; + +-- 4. Decoded CLMM distribution. After pre.10, known listed instructions should move +-- from raydium_clmm.instruction_audit to named raydium_clmm. events. +SELECT + de.protocol_name, + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(DISTINCT de.transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events de +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.protocol_name, de.event_kind +ORDER BY decoded_count DESC, de.event_kind; + +-- 5. Non-trade materialization distribution. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 6. Residual known CLMM instruction audits: should be zero for listed instruction discriminants. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS decoded_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND json_extract(payload_json, '$.discriminatorHex') IN ( + '4c7c800fd55725fa', '7b86510031446262', 'c975989055556cb2', + 'a78a4e95dfc2067e', '8888fcddc2427e59', '12eda6c52210d590', + '8934edd4d7756c68', '2b44d4a7592fa401', 'bd0eb5785576e33e', + '3f5794216d230868', 'e992d18ecf6840bc', '11fb415c88f20ea9', + 'a026d06f685b2c01', '3a7fbc3e4f52c460', '2e9cf3760dcdfbb2', + '851d59df45eeb00a', '5f87c0c4f281e644', '87802f4d0f98f031', + '4db84ad67056f1c7', '4dffae527d1dc92e', '7034a74b20c9d389', + 'cd4e74215c691a60', 'f8c69e91e17587c8', '457d73daf5baf2c4', + '2b04ed0b1ac91e62', '07160c53f22b3079', '313cae889a1c74c8', + '7f467728bce33d07', '82576c062ee0757b', 'a3ace0340b9a6adf' + ) +GROUP BY discriminator_hex +ORDER BY decoded_count DESC; + +-- 7. Unknown observed discriminants to keep audit-only until identified. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + json_extract(payload_json, '$.accountCount') AS account_count, + COUNT(*) AS decoded_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, account_count +ORDER BY decoded_count DESC; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE11.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE11.sql new file mode 100644 index 0000000..4ffcd24 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE11.sql @@ -0,0 +1,100 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE11.sql +-- Raydium CLMM validation after 0.7.49-pre.11. + +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +ORDER BY entry_kind, entry_name, discriminator_hex; + +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +SELECT + de.protocol_name, + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(DISTINCT de.transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events de +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.protocol_name, de.event_kind +ORDER BY decoded_count DESC, de.event_kind; + +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- Residual duplicated audits: should only return unknown/unmapped discriminators +-- or known discriminators for which no same-transaction named event exists. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT audit.transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events audit +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- Duplicated known audits that still have a named CLMM event in the same transaction. +-- Expected: zero rows. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE12.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE12.sql new file mode 100644 index 0000000..bd6cf25 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE12.sql @@ -0,0 +1,71 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE12.sql +-- Raydium CLMM pre.12 validation: residual audit cleanup and materialization counters. + +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; + +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.event_kind +ORDER BY de.event_kind; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE13.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE13.sql new file mode 100644 index 0000000..b648d3e --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE13.sql @@ -0,0 +1,77 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE13.sql +-- Raydium CLMM validation after pre.13 audit cleanup. + +-- 1. CLMM coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Duplicate audit rows that should have been deleted. +-- Expected result after pre.13: zero rows. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; + +-- 3. Residual instruction audits by discriminator. +-- Expected after pre.13: mostly the true unmapped values, currently 759d..., 9d20..., b190... +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 4. CLMM materialization distribution. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_liquidity_events le ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.event_kind +ORDER BY de.event_kind; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE14.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE14.sql new file mode 100644 index 0000000..d2d0064 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE14.sql @@ -0,0 +1,78 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE14.sql +-- Raydium CLMM validation after pre.14 EXISTS-based audit cleanup. + +-- 1. Coverage summary should not be reset to zero. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Duplicate audit rows. Expected: zero rows after pre.14. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; + +-- 3. Residual audits should mostly be unknown discriminants. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 4. Manual cleanup equivalent for emergency diagnosis only. +-- Do not run unless the application cleanup still fails; keep it here to compare SQL behavior. +-- DELETE FROM k_sol_dex_decoded_events +-- WHERE protocol_name = 'raydium_clmm' +-- AND event_kind = 'raydium_clmm.instruction_audit' +-- AND EXISTS ( +-- SELECT 1 +-- FROM k_sol_dex_decoded_events named +-- WHERE named.transaction_id = k_sol_dex_decoded_events.transaction_id +-- AND named.protocol_name = 'raydium_clmm' +-- AND named.event_kind <> 'raydium_clmm.instruction_audit' +-- AND COALESCE( +-- json_extract(named.payload_json, '$.instructionDiscriminatorHex'), +-- json_extract(named.payload_json, '$.instruction_discriminator_hex'), +-- json_extract(named.payload_json, '$.discriminatorHex'), +-- json_extract(named.payload_json, '$.discriminator_hex') +-- ) = COALESCE( +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminatorHex'), +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminator_hex'), +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instructionDiscriminatorHex'), +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instruction_discriminator_hex') +-- ) +-- ); diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE15.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE15.sql new file mode 100644 index 0000000..3696f90 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE15.sql @@ -0,0 +1,77 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE15.sql +-- Raydium CLMM validation after pre.15 final-refresh audit cleanup. + +-- 1. Coverage summary should stay populated after cleanup + refresh. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Duplicate audit rows should be zero after pre.15. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; + +-- 3. Residual audits should mostly be unknown discriminants. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 4. Manual cleanup equivalent: run only if query 2 still returns rows. +-- DELETE FROM k_sol_dex_decoded_events +-- WHERE protocol_name = 'raydium_clmm' +-- AND event_kind = 'raydium_clmm.instruction_audit' +-- AND EXISTS ( +-- SELECT 1 +-- FROM k_sol_dex_decoded_events named +-- WHERE named.transaction_id = k_sol_dex_decoded_events.transaction_id +-- AND named.protocol_name = 'raydium_clmm' +-- AND named.event_kind <> 'raydium_clmm.instruction_audit' +-- AND COALESCE( +-- json_extract(named.payload_json, '$.instructionDiscriminatorHex'), +-- json_extract(named.payload_json, '$.instruction_discriminator_hex'), +-- json_extract(named.payload_json, '$.discriminatorHex'), +-- json_extract(named.payload_json, '$.discriminator_hex') +-- ) = COALESCE( +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminatorHex'), +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.discriminator_hex'), +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instructionDiscriminatorHex'), +-- json_extract(k_sol_dex_decoded_events.payload_json, '$.instruction_discriminator_hex') +-- ) +-- ); diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE16.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE16.sql new file mode 100644 index 0000000..5de0c68 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE16.sql @@ -0,0 +1,76 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE16.sql +-- Raydium CLMM validation after pre.16 mapped-audit allow-list cleanup. + +-- 1. Coverage summary must stay populated. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Duplicate audit rows should be zero. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; + +-- 3. Residual audits should be dominated by unknown discriminants. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 4. Manual cleanup equivalent for immediate diagnosis only. +-- DELETE FROM k_sol_dex_decoded_events +-- WHERE protocol_name = 'raydium_clmm' +-- AND event_kind = 'raydium_clmm.instruction_audit' +-- AND COALESCE( +-- json_extract(payload_json, '$.discriminatorHex'), +-- json_extract(payload_json, '$.discriminator_hex'), +-- json_extract(payload_json, '$.instructionDiscriminatorHex'), +-- json_extract(payload_json, '$.instruction_discriminator_hex') +-- ) IN ( +-- '4c7c800fd55725fa','7b86510031446262','c975989055556cb2', +-- 'a78a4e95dfc2067e','8888fcddc2427e59','12eda6c52210d590', +-- '8934edd4d7756c68','2b44d4a7592fa401','bd0eb5785576e33e', +-- '3f5794216d230868','e992d18ecf6840bc','11fb415c88f20ea9', +-- 'a026d06f685b2c01','3a7fbc3e4f52c460','2e9cf3760dcdfbb2', +-- '851d59df45eeb00a','5f87c0c4f281e644','87802f4d0f98f031', +-- '4db84ad67056f1c7','4dffae527d1dc92e','7034a74b20c9d389', +-- 'cd4e74215c691a60','f8c69e91e17587c8','457d73daf5baf2c4', +-- '2b04ed0b1ac91e62','07160c53f22b3079','313cae889a1c74c8', +-- '7f467728bce33d07','82576c062ee0757b','a3ace0340b9a6adf' +-- ); diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE17.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE17.sql new file mode 100644 index 0000000..af79f6f --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE17.sql @@ -0,0 +1,120 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE17.sql +-- Raydium CLMM validation after limit-order instruction mapping. +-- Expected coverage listed_entry_count becomes 45 because open/increase/decrease_limit_order are now listed. + +-- 1. CLMM coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. New CLMM limit-order coverage rows. +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' + AND entry_name IN ( + 'open_limit_order', + 'increase_limit_order', + 'decrease_limit_order', + 'close_limit_order', + 'settle_limit_order' + ) +ORDER BY entry_name; + +-- 3. The formerly residual discriminants must now decode as named events. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(DISTINCT de.transaction_id) AS transaction_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind IN ( + 'raydium_clmm.open_limit_order', + 'raydium_clmm.increase_limit_order', + 'raydium_clmm.decrease_limit_order' + ) +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 4. No residual audit rows for the newly mapped discriminants. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND json_extract(payload_json, '$.discriminatorHex') IN ( + '9d20dab7471d1293', + 'b19059ecfaba7d63', + '759d3c674231a300' + ) +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC; + +-- 5. No trade/candle safety: limit-order maintenance must never produce trade rows. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind IN ( + 'raydium_clmm.open_limit_order', + 'raydium_clmm.increase_limit_order', + 'raydium_clmm.decrease_limit_order', + 'raydium_clmm.close_limit_order', + 'raydium_clmm.settle_limit_order' + ) +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 6. Duplicate audit rows should stay empty after mapped-audit cleanup. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE18.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE18.sql new file mode 100644 index 0000000..e736c13 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE18.sql @@ -0,0 +1,117 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE18.sql +-- Raydium CLMM validation after pre.18 orderbook-event schema and materialization. + +-- 1. Schema must contain the new orderbook materialization table. +SELECT + name +FROM sqlite_master +WHERE type = 'table' + AND name = 'k_sol_orderbook_events'; + +-- 2. Cleanup any stale mapped CLMM instruction audits left from earlier runs. +-- This is safe: the discriminants below are now decoded as named raydium_clmm.* events. +DELETE FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(payload_json, '$.discriminatorHex'), + json_extract(payload_json, '$.discriminator_hex'), + json_extract(payload_json, '$.instructionDiscriminatorHex'), + json_extract(payload_json, '$.instruction_discriminator_hex') + ) IN ( + '4c7c800fd55725fa','7b86510031446262','c975989055556cb2', + 'a78a4e95dfc2067e','8888fcddc2427e59','12eda6c52210d590', + '8934edd4d7756c68','2b44d4a7592fa401','bd0eb5785576e33e', + '3f5794216d230868','e992d18ecf6840bc','11fb415c88f20ea9', + 'a026d06f685b2c01','3a7fbc3e4f52c460','2e9cf3760dcdfbb2', + '851d59df45eeb00a','5f87c0c4f281e644','87802f4d0f98f031', + '4db84ad67056f1c7','4dffae527d1dc92e','7034a74b20c9d389', + 'cd4e74215c691a60','f8c69e91e17587c8','457d73daf5baf2c4', + '2b04ed0b1ac91e62','07160c53f22b3079','313cae889a1c74c8', + '7f467728bce33d07','82576c062ee0757b','a3ace0340b9a6adf', + '759d3c674231a300','9d20dab7471d1293','b19059ecfaba7d63' + ); + +-- 3. Duplicate audits should be zero. +SELECT + json_extract(audit.payload_json, '$.discriminatorHex') AS discriminator_hex, + audit.event_kind AS audit_event_kind, + named.event_kind AS named_event_kind, + COUNT(*) AS duplicate_count +FROM k_sol_dex_decoded_events audit +JOIN k_sol_dex_decoded_events named + ON named.transaction_id = audit.transaction_id + AND named.protocol_name = 'raydium_clmm' + AND named.event_kind <> 'raydium_clmm.instruction_audit' + AND COALESCE( + json_extract(named.payload_json, '$.instructionDiscriminatorHex'), + json_extract(named.payload_json, '$.instruction_discriminator_hex'), + json_extract(named.payload_json, '$.discriminatorHex'), + json_extract(named.payload_json, '$.discriminator_hex') + ) = COALESCE( + json_extract(audit.payload_json, '$.discriminatorHex'), + json_extract(audit.payload_json, '$.discriminator_hex'), + json_extract(audit.payload_json, '$.instructionDiscriminatorHex'), + json_extract(audit.payload_json, '$.instruction_discriminator_hex') + ) +WHERE audit.protocol_name = 'raydium_clmm' + AND audit.event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex, audit_event_kind, named_event_kind +ORDER BY duplicate_count DESC; + +-- 4. Limit-order decoded events must materialize into k_sol_orderbook_events and never trades. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(DISTINCT de.transaction_id) AS tx_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind IN ( + 'raydium_clmm.open_limit_order', + 'raydium_clmm.increase_limit_order', + 'raydium_clmm.decrease_limit_order', + 'raydium_clmm.close_limit_order', + 'raydium_clmm.settle_limit_order' + ) +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 5. No failed transaction should be orderbook-materialized. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +ORDER BY decoded_count DESC, de.event_kind; + +-- 6. CLMM coverage summary after refresh. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE19.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE19.sql new file mode 100644 index 0000000..c0e520b --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE19.sql @@ -0,0 +1,144 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE19.sql +-- Raydium CLMM validation after pre.19 upstream fallback cleanup. + +-- 1. CLMM coverage summary must remain populated. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. No local CLMM instruction audit should remain after specialized decoding. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 3. No upstream_git fallback should remain for CLMM entries already covered locally. +SELECT + json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT transaction_id) AS tx_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'upstream_git' + AND event_kind = 'upstream_git.instruction_match' + AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 4. Generic diagnostic for any locally covered upstream fallback, all protocols. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + COUNT(*) AS fallback_count, + COUNT(DISTINCT ug.transaction_id) AS tx_count +FROM k_sol_dex_decoded_events ug +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex +ORDER BY fallback_count DESC, upstream_decoder_code, entry_name; + +-- 5. Non-swap CLMM events must never create trades/candles. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind NOT IN ( + 'raydium_clmm.swap', + 'raydium_clmm.swap_v2' + ) +GROUP BY de.event_kind +HAVING COUNT(te.id) > 0 +ORDER BY trade_count DESC, de.event_kind; + +-- 6. Failed CLMM transactions must not materialize into business tables. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +HAVING + COUNT(le.id) > 0 + OR COUNT(fe.id) > 0 + OR COUNT(re.id) > 0 + OR COUNT(pa.id) > 0 + OR COUNT(ple.id) > 0 + OR COUNT(oe.id) > 0 + OR COUNT(te.id) > 0 +ORDER BY de.event_kind; + +-- 7. Orderbook limit-order materialization safety. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(DISTINCT de.transaction_id) AS tx_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind IN ( + 'raydium_clmm.open_limit_order', + 'raydium_clmm.increase_limit_order', + 'raydium_clmm.decrease_limit_order', + 'raydium_clmm.close_limit_order', + 'raydium_clmm.settle_limit_order' + ) +GROUP BY de.event_kind +ORDER BY de.event_kind; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE20.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE20.sql new file mode 100644 index 0000000..4670b22 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE20.sql @@ -0,0 +1,123 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE20.sql +-- Raydium CLMM validation after pre.20 fallback cleanup + Program data event readiness. + +-- 1. CLMM coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Residual CLMM instruction audits. Expected: zero rows. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 3. Redundant upstream fallback matches for locally covered CLMM entries. Expected: zero rows. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT ug.transaction_id) AS tx_count +FROM k_sol_dex_decoded_events ug +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' + AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 4. Any remaining upstream CLMM fallback matches. Expected: zero rows unless a genuinely unmapped future entry appears. +SELECT + json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT transaction_id) AS tx_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'upstream_git' + AND event_kind = 'upstream_git.instruction_match' + AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 5. Anchor Program data events. These may remain zero until matching Program data logs exist in corpus. +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' + AND entry_kind = 'event' +ORDER BY entry_name; + +-- 6. Failed transaction materialization guard. Expected: zero rows. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +HAVING + COUNT(le.id) > 0 + OR COUNT(fe.id) > 0 + OR COUNT(re.id) > 0 + OR COUNT(pa.id) > 0 + OR COUNT(ple.id) > 0 + OR COUNT(oe.id) > 0 + OR COUNT(te.id) > 0 +ORDER BY de.event_kind; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE22.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE22.sql new file mode 100644 index 0000000..626c4a1 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE22.sql @@ -0,0 +1,139 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE22.sql +-- Raydium CLMM validation after pre.22 upstream fallback reset cleanup hardening. + +-- 1. CLMM coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Residual CLMM instruction audits. Expected: zero rows. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 3. Redundant upstream fallback matches for locally covered entries. Expected: zero rows. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT ug.transaction_id) AS tx_count +FROM k_sol_dex_decoded_events ug +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' + AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 4. Any remaining upstream CLMM fallback matches. Expected: zero rows unless a genuinely unmapped future entry appears. +SELECT + json_extract(payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT transaction_id) AS tx_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'upstream_git' + AND event_kind = 'upstream_git.instruction_match' + AND json_extract(payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 5. Safety: non-swap CLMM rows must not materialize as trades. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind NOT IN ( + 'raydium_clmm.swap', + 'raydium_clmm.swap_v2' + ) +GROUP BY de.event_kind +HAVING COUNT(te.id) > 0 +ORDER BY trade_count DESC, de.event_kind; + +-- 6. Safety: failed CLMM transactions must not materialize into business tables. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +HAVING + COUNT(le.id) > 0 + OR COUNT(fe.id) > 0 + OR COUNT(re.id) > 0 + OR COUNT(pa.id) > 0 + OR COUNT(ple.id) > 0 + OR COUNT(oe.id) > 0 + OR COUNT(te.id) > 0 +ORDER BY de.event_kind; + +-- 7. Raydium CPMM listed coverage entries. +SELECT + entry_name, + entry_kind, + event_family, + expected_db_target, + proof_status, + local_event_kind, + discriminator_hex, + observed_count, + materialized_count, + trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_cpmm' +ORDER BY entry_kind, entry_name, discriminator_hex; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql new file mode 100644 index 0000000..c5f9a1d --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql @@ -0,0 +1,132 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE23.sql +-- Raydium CLMM validation after pre.23 FK-safe upstream fallback cleanup. + +-- 1. CLMM coverage summary. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; + +-- 2. Residual CLMM instruction audits. Expected: zero rows. +SELECT + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + COUNT(*) AS residual_audit_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' +GROUP BY discriminator_hex +ORDER BY residual_audit_count DESC, discriminator_hex; + +-- 3. Redundant upstream fallback matches for locally covered CLMM entries. Expected: zero rows. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS discriminator_hex, + json_extract(ug.payload_json, '$.upstreamSourceRepo') AS source_repo, + COUNT(*) AS fallback_count, + COUNT(DISTINCT ug.transaction_id) AS tx_count +FROM k_sol_dex_decoded_events ug +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' + AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY upstream_decoder_code, entry_name, discriminator_hex, source_repo +ORDER BY fallback_count DESC, entry_name; + +-- 4. Instruction-observation links still pointing to redundant upstream fallback rows. Expected: zero rows. +SELECT + json_extract(ug.payload_json, '$.upstreamDecoderCode') AS upstream_decoder_code, + json_extract(ug.payload_json, '$.upstreamEntryName') AS entry_name, + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') AS upstream_discriminator_hex, + COUNT(*) AS linked_observation_count +FROM k_sol_instruction_observations io +JOIN k_sol_dex_decoded_events ug + ON ug.id = io.decoded_event_id +JOIN k_sol_dex_event_coverage_entries ce + ON ce.decoder_code = json_extract(ug.payload_json, '$.upstreamDecoderCode') + AND ce.entry_name = json_extract(ug.payload_json, '$.upstreamEntryName') + AND ce.discriminator_hex = json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') + AND ce.local_event_kind IS NOT NULL + AND ce.local_event_kind <> '' +WHERE ug.protocol_name = 'upstream_git' + AND ug.event_kind = 'upstream_git.instruction_match' + AND json_extract(ug.payload_json, '$.upstreamDecoderCode') = 'raydium_clmm' +GROUP BY + json_extract(ug.payload_json, '$.upstreamDecoderCode'), + json_extract(ug.payload_json, '$.upstreamEntryName'), + json_extract(ug.payload_json, '$.upstreamDiscriminatorHex') +ORDER BY linked_observation_count DESC, entry_name; + +-- 5. Any non-swap CLMM trade. Expected: zero rows. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind NOT IN ( + 'raydium_clmm.swap', + 'raydium_clmm.swap_v2' + ) +GROUP BY de.event_kind +HAVING COUNT(te.id) > 0 +ORDER BY trade_count DESC, de.event_kind; + +-- 6. Failed transaction materialization guard. Expected: zero rows. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(oe.id) AS orderbook_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +JOIN k_sol_chain_transactions tx + ON tx.id = de.transaction_id +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_orderbook_events oe + ON oe.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND tx.err_json IS NOT NULL + AND tx.err_json <> '' + AND tx.err_json <> 'null' +GROUP BY de.event_kind +HAVING + COUNT(le.id) > 0 + OR COUNT(fe.id) > 0 + OR COUNT(re.id) > 0 + OR COUNT(pa.id) > 0 + OR COUNT(ple.id) > 0 + OR COUNT(oe.id) > 0 + OR COUNT(te.id) > 0 +ORDER BY de.event_kind; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE6.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE6.sql new file mode 100644 index 0000000..8bc4b67 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE6.sql @@ -0,0 +1,88 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE6.sql +-- Raydium CLMM pre.6 validation: specialized non-trade reconciliation. + +-- 1. Confirm the two corpus-observed CLMM non-trades are materialized. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(fe.id) AS fee_count, + COUNT(ple.id) AS lifecycle_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind IN ( + 'raydium_clmm.create_pool', + 'raydium_clmm.collect_protocol_fee' + ) +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 2. Confirm residual audits for the same discriminants are gone. +SELECT + json_extract(payload_json, '$.instructionDiscriminatorHex') AS instruction_discriminator_hex, + json_extract(payload_json, '$.instruction_discriminator_hex') AS instruction_discriminator_hex_snake, + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + json_extract(payload_json, '$.discriminator_hex') AS discriminator_hex_snake, + json_extract(payload_json, '$.accountCount') AS account_count, + COUNT(*) AS decoded_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND ( + json_extract(payload_json, '$.instructionDiscriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + OR json_extract(payload_json, '$.instruction_discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + OR json_extract(payload_json, '$.discriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + OR json_extract(payload_json, '$.discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + ) +GROUP BY + instruction_discriminator_hex, + instruction_discriminator_hex_snake, + discriminator_hex, + discriminator_hex_snake, + account_count +ORDER BY decoded_count DESC; + +-- 3. Full CLMM distribution with materialization counters. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 4. Coverage summary after sync/replay. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; diff --git a/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE7.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE7.sql new file mode 100644 index 0000000..a03f986 --- /dev/null +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE7.sql @@ -0,0 +1,88 @@ +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CLMM_0_7_49_PRE7.sql +-- Raydium CLMM pre.7 validation: direct materialization + audit cleanup. + +-- 1. Confirm corpus-observed CLMM non-trades are materialized. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(fe.id) AS fee_count, + COUNT(ple.id) AS lifecycle_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' + AND de.event_kind IN ( + 'raydium_clmm.create_pool', + 'raydium_clmm.collect_protocol_fee' + ) +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 2. Confirm residual audits for the same discriminants are gone. +SELECT + json_extract(payload_json, '$.instructionDiscriminatorHex') AS instruction_discriminator_hex, + json_extract(payload_json, '$.instruction_discriminator_hex') AS instruction_discriminator_hex_snake, + json_extract(payload_json, '$.discriminatorHex') AS discriminator_hex, + json_extract(payload_json, '$.discriminator_hex') AS discriminator_hex_snake, + json_extract(payload_json, '$.accountCount') AS account_count, + COUNT(*) AS decoded_count, + COUNT(DISTINCT transaction_id) AS transaction_count +FROM k_sol_dex_decoded_events +WHERE protocol_name = 'raydium_clmm' + AND event_kind = 'raydium_clmm.instruction_audit' + AND ( + json_extract(payload_json, '$.instructionDiscriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + OR json_extract(payload_json, '$.instruction_discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + OR json_extract(payload_json, '$.discriminatorHex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + OR json_extract(payload_json, '$.discriminator_hex') IN ('e992d18ecf6840bc', '8888fcddc2427e59') + ) +GROUP BY + instruction_discriminator_hex, + instruction_discriminator_hex_snake, + discriminator_hex, + discriminator_hex_snake, + account_count +ORDER BY decoded_count DESC; + +-- 3. Full CLMM distribution with materialization counters. +SELECT + de.event_kind, + COUNT(*) AS decoded_count, + COUNT(le.id) AS liquidity_count, + COUNT(fe.id) AS fee_count, + COUNT(re.id) AS reward_count, + COUNT(pa.id) AS admin_count, + COUNT(ple.id) AS lifecycle_count, + COUNT(te.id) AS trade_count +FROM k_sol_dex_decoded_events de +LEFT JOIN k_sol_liquidity_events le + ON le.decoded_event_id = de.id +LEFT JOIN k_sol_fee_events fe + ON fe.decoded_event_id = de.id +LEFT JOIN k_sol_reward_events re + ON re.decoded_event_id = de.id +LEFT JOIN k_sol_pool_admin_events pa + ON pa.decoded_event_id = de.id +LEFT JOIN k_sol_pool_lifecycle_events ple + ON ple.decoded_event_id = de.id +LEFT JOIN k_sol_trade_events te + ON te.decoded_event_id = de.id +WHERE de.protocol_name = 'raydium_clmm' +GROUP BY de.event_kind +ORDER BY de.event_kind; + +-- 4. Coverage summary after sync/replay. +SELECT + decoder_code, + COUNT(*) AS listed_entry_count, + SUM(CASE WHEN local_event_kind IS NOT NULL AND local_event_kind <> '' THEN 1 ELSE 0 END) AS decoded_entry_count, + SUM(CASE WHEN observed_count > 0 THEN 1 ELSE 0 END) AS observed_entry_count, + SUM(CASE WHEN materialized_count > 0 THEN 1 ELSE 0 END) AS materialized_entry_count, + COALESCE(SUM(observed_count), 0) AS total_observed_count, + COALESCE(SUM(materialized_count), 0) AS total_materialized_count, + COALESCE(SUM(trade_count), 0) AS trade_count +FROM k_sol_dex_event_coverage_entries +WHERE decoder_code = 'raydium_clmm' +GROUP BY decoder_code; diff --git a/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql b/validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql similarity index 99% rename from SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql rename to validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql index ac02d0e..e9f3b64 100644 --- a/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql +++ b/validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql @@ -1,4 +1,4 @@ --- file: SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql +-- file: validation_sql/SQL_VALIDATION_RAYDIUM_CPMM_0_7_48.sql -- 1. Coverage rows must exist after diagnostics/validation/backfill refresh. SELECT *