Current Status & 48-Hour Trend (September 15–17)

Following the elevated queues observed earlier in the week, the Solana ingestion pipeline has demonstrated significantly improved baseline stability over the past 48 hours:

• September 15 (Afternoon/Evening): Elevated transfer queues gradually cleared from ~850 items, fully normalizing by 03:00 UTC on Wednesday, September 16.

• September 16 – Midday September 17: The pipeline maintained sustained real-time processing throughput, with minor, brief volume spikes (peaking around ~180 items) resolving rapidly.

• September 17 (Current): A slight uptick in queue depth is emerging heading into the daily global peak traffic window. Dynamic worker capacity remains active to absorb incoming transaction volume.

Engineering Roadmap & Long-Term Resolution

While temporary capacity scaling and dynamic buffer allocations continue to absorb standard network surges effectively, our engineering teams are actively developing a comprehensive architectural upgrade to permanently expand continuous ingestion and parsing throughput.

Given the extensive scope of this pipeline migration and the rigorous benchmarking required across production-grade datasets, the full rollout of these structural enhancements is scheduled to deploy across phases over the coming weeks, targeting complete implementation within the next month.

Operational Highlights

• Zero Data Loss: All incoming transactions and screening requests remain completely protected; no evaluations or webhooks are dropped.

• Sequential Execution: Queues are processed in standard FIFO order, with minimal latency expected outside peak network congestion periods.

• Ongoing Oversight: Our infrastructure team continues 24/7 monitoring of queue health and will deploy intermediate performance optimizations as components become ready.

We will continue to share operational progress and milestones as deployment phases advance.

Current Status (September 15)

Ingestion and evaluation queues are currently clearing a recent spike. During the morning peak window on September 15, transfer screening queues experienced a sharp volume surge, peaking at ~2,100 queued items before worker capacity began steadily absorbing the backlog (currently recovered to under 500 and trending downward). Intermittent latency on real-time Solana address/transaction evaluations and webhook deliveries is expected while this backlog normalizes.

7-Day Performance & Queue Trends (September 9–15)

• September 9–10: Network peak traffic windows drove elevated queue depths averaging ~1,000 items, resulting in periodic 60–90 minute evaluation delays.

• September 11–14 (Weekend & Monday): Processing stabilized significantly. Over the weekend and through Monday, queue depths remained low and flat, with the pipeline running in near real-time.

• Tuesday, September 15 (Today): A concentrated burst in network volume drove a sharp intra-day queue spike to a 7-day high of 2.1k items. Scaling workers engaged immediately, and the backlog is actively draining.

Key Customer Takeaways

• Zero Data Loss: All incoming transactions and screening requests remain safely queued; no evaluations or webhooks are dropped.

• Processing Order: Requests are evaluated sequentially in FIFO order as the queue clears.

• Ongoing Mitigation: Dynamic worker allocation remains active during network volume surges while longer-term throughput optimizations continue in development.

Our team continues to monitor queue recovery and pipeline throughput closely.

A processing delay is currently active. Ingestion queues began accumulating latency again around 15:00 UTC, and our engineering team is actively monitoring queue processing rates as parsing workers absorb the current surge in block data.

48-Hour Network & Ingestion Summary

Over the past two days (September 10–11), our Solana ingestion pipeline has experienced intermittent queuing bottlenecks during global network peak-volume windows:

• Thursday, September 10: Sustained elevated throughput across the Solana network created significant ingestion queues, reaching a peak lag of ~16,300 blocks between 06:00 and 08:00 UTC (approximately 60–90 minutes of evaluation latency) before normalizing around 22:00 UTC.

• Friday, September 11 (00:00–15:00 UTC): Queues stabilized completely, with pipeline processing operating in near real-time throughout the early and midday hours.

• Friday, September 11 (15:00 UTC – Present): Entering the network's daily peak volume window, a new processing backlog is actively building. Parsing workers continue to evaluate incoming blocks sequentially.

Key Customer Takeaways

• Data Integrity: No requests, transactions, or screening payloads are dropped. All events remain safely buffered in the ingestion queue.

• Impact: Address and transaction screening evaluations, as well as associated webhook notifications, are currently experiencing intermittent processing delays.

• Mitigation: Dynamic worker capacity and temporary queue buffers remain provisioned to accelerate throughput during peak traffic periods.

Our team is continuously monitoring queue metrics and pipeline throughput, and we will provide a follow-up performance update in 24 hours.

Identified – Our engineering team is actively investigating intermittent processing delays affecting the evaluation and screening of Solana transactions and addresses.

• Impact: During daily global network peak activity windows, clients may experience latency in Solana evaluation results and webhook deliveries. Peak delays can intermittently reach up to 60 to 90 minutes.

• Data Integrity: All incoming transactions and screening requests are safely queued. No data or evaluation requests are dropped, and records are processed sequentially as queues normalize. All other supported blockchains remain unaffected.

Root Cause & Context:

Recent Solana network protocol updates—most notably the activation of SIMD-0286, which expanded block compute unit limits from 60M to 100M CUs—have significantly increased the density of transactions processed per block across the network.

While this upgrade was documented and implemented by the network as a backward-compatible, non-breaking protocol change (requiring no data schema modifications), the downstream effect has been a substantial surge in sustained transaction throughput and block payload volumes during high-traffic periods. Although our infrastructure was aligned with the protocol specification, the resulting bursts in on-chain volume during peak hours have caused temporary bottlenecks in data parsing and risk evaluation queues.

Current Actions:

• We have deployed temporary compute and queue-scaling resources to absorb ingestion spikes and reduce queue recovery times.

• Engineering is rolling out targeted ingestion pipeline enhancements to scale our continuous throughput capacity in line with sustained network load.

• Continuous monitoring is in place, and we are dynamically adjusting queue allocations during peak surges.

We will provide further updates as additional performance optimizations are deployed to the production environment.