Streaming Data Analysis
Exploration of real-time data processing techniques and analysis pipelines for streaming datasets.
The story
This is the least glamorous thing on the entire site. No app, no screenshots, no bots quietly doing my work in the background. Just a maths paper. I am oddly fond of it.
The whole thing is built on a very relatable kind of laziness. Picture numbers arriving one at a time, forever, and you want to know their average and their spread. The obvious approach is to keep every number and recompute from scratch each time a new one lands, which is fine until forever gets expensive. The paper works out how to update the answer using only the previous answer and the newest number, so you can throw the old data away and never look back.
It sounds small, but done naively the textbook variance formula quietly falls apart on a computer, drifting into nonsense as the numbers pile up. So a good chunk of the paper is spent deriving the careful, numerically stable version, and then stretching the same trick all the way to linear regression.
In short, I wrote several pages of formal mathematics to justify not redoing my work, and I stand by it completely. For now it sits under the review of my dear teacher, Dr Andrees Chattha, whose verdict I await with the appropriate blend of confidence and mild dread.
Methodology
- Derivation of recursive formulas for the mean and variance in streaming datasets
- Development of numerically stable variance updates using Welford’s identity
- Extension of recursive statistics to simple linear regression through centered sums (Sxx and Sxy)
- Algorithmic representation and pseudocode for real‑time implementation
- Manual simulations comparing batch and recursive computation
Key Insights
- Recursive statistics allow means and variances to be updated without storing the full dataset
- Welford’s algorithm provides a numerically stable one‑pass variance computation
- Linear regression can be expressed through maintained statistics (Sxx and Sxy) instead of full recomputation
- The recursive formulation preserves the exact results of batch ordinary least squares
- The approach enables constant‑time updates suitable for streaming and real‑time analytics
Tools & Technologies
- Mathematical derivation of recursive statistical identities
- Algorithm design for one‑pass streaming computation
- Pseudocode implementation of Welford’s algorithm and recursive regression updates
- Analytical comparison between batch and recursive computation methods
Final Paper
The full research report is available below.