This week we continue the Apache Spark usefulness and limitations for financial analytics. Last week we discussed input data formats. This week we focus on consolidation of financial data. Indexed access typically helps when searching for single answer to a question, be it a search engine query URL, or an individual financial transaction or balance. This compute pattern utilizes parallelism very effectively. But many types … Continue reading Apache Spark Parallelism and Financial Analytics

In the next three week’s episodes of Conversations with Kip, I focus on Apache Spark, a technology that recognizes the power of scanning large transactional data sets to produce analytical outputs. This week we focus on the Input Processes for analytical processes. The larger the amounts of data we can scan efficiently and quickly, the greater our potential analytics. Yet the scan process can be … Continue reading Apache Spark IO and Index Access