Today’s episode is the second in a three part series of conversations with Tata Rao, at the time Lead Architect of a Global Financial Services Financial System Platform, covering data volume explosion in a consolidated data supply chain, and the storage and compute trade-offs for movements vs. balances.
Tata begins by discussing how data volumes grow through the Instrument Ledger processes (to understand an Instrument or Arrangement Ledger, watch the system architecture set of videos); input transactions are only part of the transactions needed for financial analysis. There are other types of events, accounting events, temporal events, analytical events that are required. These all can combine, if performed at the detail level, to produce very high data volumes. The vlog series on estimating performance can help you understand this more.
In an early video “The Big Idea: Eliminate Posting”, I noted that one can produce every financial analysis with only the transactions involved, but that doing so for many types of analysis requires very large data volumes.
On the other end of the spectrum, very few analyses are interested in transactions themselves; almost every analytical process begins with needing a balance. And the number of possible permutations of transactions is an equally high number. See “The Daily Financial Cycle” to understand this more clearly.
These two extremes can be plotted on this chart, described in my textbook Balancing Act, Chapters 14 to Chapter 18.
Movements are a concept that allows significant compression of data needed to store “balances.” These ideas are at the heart of what is required to create a Metric Engine as opposed to a Search Engine–an engine the produces the analytical outputs needed in our world today.
This is Episode 154 of Conversations with Kip, the best financial system vlog there is. Literally learn more–about ledgers and financial systems–at Financial Systems Education.com.
Leave a Reply