{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,18]],"date-time":"2025-06-18T04:28:16Z","timestamp":1750220896610,"version":"3.41.0"},"reference-count":19,"publisher":"Association for Computing Machinery (ACM)","issue":"3","license":[{"start":{"date-parts":[[2019,12,17]],"date-time":"2019-12-17T00:00:00Z","timestamp":1576540800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["Proc. ACM Meas. Anal. Comput. Syst."],"published-print":{"date-parts":[[2019,12,17]]},"abstract":"<jats:p>We consider the problem of forecasting fine-grained company financials, such as daily revenue, from two input types: noisy proxy signals a la alternative data (e.g. credit card transactions) and sparse ground-truth observations (e.g. quarterly earnings reports). We utilize a classical linear systems model to capture both the evolution of the hidden or latent state (e.g. daily revenue), as well as the proxy signal (e.g. credit cards transactions). The linear system model is particularly well suited here as data is extremely sparse (4 quarterly reports per year). In classical system identification, where the central theme is to learn parameters for such linear systems, unbiased and consistent estimation of parameters is not feasible: the likelihood is non-convex; and worse, the global optimum for maximum likelihood estimation is often non-unique. As the main contribution of this work, we provide a simple, consistent estimator of all parameters for the linear system model of interest; in addition the estimation is unbiased for some of the parameters. In effect, the additional sparse observations of aggregate hidden state (e.g. quarterly reports) enable system identification in our setup that is not feasible in general. For estimating and forecasting hidden state (actual earnings) using the noisy observations (daily credit card transactions), we utilize the learned linear model along with a natural adaptation of classical Kalman filtering (or Belief Propagation). This leads to optimal inference with respect to mean-squared error. Analytically, we argue that even though the underlying linear system may be \"unstable,\" \"uncontrollable,\" or \"undetectable\" in the classical setting, our setup and inference algorithm allow for estimation of hidden state with bounded error. Further, the estimation error of the algorithm monotonically decreases as the frequency of the sparse observations increases. This, seemingly intuitive insight contradicts the word on the Street. Finally, we utilize our framework to estimate quarterly earnings of 34 public companies using credit card transaction data. Our data-driven method convincingly outperforms the Wall Street consensus (analyst) estimates even though our method uses only credit card data as input, while the Wall Street consensus is based on various data sources including experts' input.<\/jats:p>","DOI":"10.1145\/3366694","type":"journal-article","created":{"date-parts":[[2019,12,18]],"date-time":"2019-12-18T13:21:11Z","timestamp":1576675271000},"page":"1-29","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":0,"title":["Forecasting with Alternative Data"],"prefix":"10.1145","volume":"3","author":[{"given":"Michael","family":"Fleder","sequence":"first","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Devavrat","family":"Shah","sequence":"additional","affiliation":[{"name":"Massachusetts Institute of Technology, Cambridge, MA, USA"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"320","published-online":{"date-parts":[[2019,12,17]]},"reference":[{"key":"e_1_2_1_1_1","unstructured":"Eagle Alpha. 2018. Eagle Alpha Alternative Data Use Cases. https:\/\/eaglealpha.com\/eagle-alphas-alternative-data-use-cases. Accessed: 2018-05--10.  Eagle Alpha. 2018. Eagle Alpha Alternative Data Use Cases. https:\/\/eaglealpha.com\/eagle-alphas-alternative-data-use-cases. Accessed: 2018-05--10."},{"key":"e_1_2_1_2_1","unstructured":"AlternativeData.org. 2018a. Alternative Data by the Numbers. https:\/\/alternativedata.org\/resources\/alternative-data-by-the-numbers. Accessed: 2018-05--17.  AlternativeData.org. 2018a. Alternative Data by the Numbers. https:\/\/alternativedata.org\/resources\/alternative-data-by-the-numbers. Accessed: 2018-05--17."},{"key":"e_1_2_1_3_1","unstructured":"AlternativeData.org. 2018b. Alternative Data Database. https:\/\/alternativedata.org\/data-providers\/. Accessed: 2018--11--10.  AlternativeData.org. 2018b. Alternative Data Database. https:\/\/alternativedata.org\/data-providers\/. Accessed: 2018--11--10."},{"key":"e_1_2_1_4_1","unstructured":"Harry Asada. 2018. MIT 2.160. Lecture Notes.  Harry Asada. 2018. MIT 2.160. Lecture Notes."},{"key":"e_1_2_1_5_1","unstructured":"Dimitri P Bertsekas. 1995. Dynamic programming and optimal control . Vol. 1. Athena scientific Belmont MA.  Dimitri P Bertsekas. 1995. Dynamic programming and optimal control . Vol. 1. Athena scientific Belmont MA."},{"key":"e_1_2_1_6_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.jocs.2010.12.007"},{"key":"e_1_2_1_7_1","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-018-24271-9"},{"key":"e_1_2_1_8_1","unstructured":"Ryan Dezember. 2018. Your Smartphone's Location Data Is Worth Big Money to Wall Street. https:\/\/www.wsj.com\/articles\/your-smartphones-location-data-is-worth-big-money-to-wall-street-1541131260. Accessed: 2018--11-04.  Ryan Dezember. 2018. Your Smartphone's Location Data Is Worth Big Money to Wall Street. https:\/\/www.wsj.com\/articles\/your-smartphones-location-data-is-worth-big-money-to-wall-street-1541131260. Accessed: 2018--11-04."},{"key":"e_1_2_1_9_1","unstructured":"Amir Efrati. 2018. U.S. Slowdown at Uber and Lyft . https:\/\/www.theinformation.com\/articles\/u-s-slowdown-at-uber-and-lyft . Accessed: 2018--10--25.  Amir Efrati. 2018. U.S. Slowdown at Uber and Lyft . https:\/\/www.theinformation.com\/articles\/u-s-slowdown-at-uber-and-lyft . Accessed: 2018--10--25."},{"key":"e_1_2_1_10_1","unstructured":"Virtu Financial. 2015. Virtu Financial Announces First Quarter 2015 Results. http:\/\/ir.virtu.com\/financials-and-filings\/quarterly-results\/default.aspx Accessed: 2019-07--10.  Virtu Financial. 2015. Virtu Financial Announces First Quarter 2015 Results. http:\/\/ir.virtu.com\/financials-and-filings\/quarterly-results\/default.aspx Accessed: 2019-07--10."},{"key":"e_1_2_1_11_1","unstructured":"Graham C. Goodwin and Kwai Sang Sin. 1984. Adaptive Filtering Prediction and Control .Dover Publications Mineola NY.  Graham C. Goodwin and Kwai Sang Sin. 1984. Adaptive Filtering Prediction and Control .Dover Publications Mineola NY."},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1049\/ip-f-2.1993.0015"},{"volume-title":"Kalman Filtering in the Presence of State Space Equality Constraints. In 2007 Chinese Control Conference. IEEE, 107--113","year":"2006","author":"Gupta Nachi","key":"e_1_2_1_13_1"},{"key":"e_1_2_1_14_1","unstructured":"James Douglas Hamilton. 1994. Time series analysis .Princeton Univ. Press Princeton NJ.  James Douglas Hamilton. 1994. Time series analysis .Princeton Univ. Press Princeton NJ."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1109\/7.826316"},{"volume-title":"Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium. IEEE, 153--158","year":"2016","author":"Hardt Moritz","key":"e_1_2_1_16_1"},{"key":"e_1_2_1_17_1","unstructured":"Joseph White. 2018. GM to drop monthly U.S. vehicle sale reports. https:\/\/www.reuters.com\/article\/us-usa-autos-gm\/gm-to-drop-monthly-u-s-vehicle-sale-reports-idUSKCN1HA0C9. Accessed: 2018-05-07.  Joseph White. 2018. GM to drop monthly U.S. vehicle sale reports. https:\/\/www.reuters.com\/article\/us-usa-autos-gm\/gm-to-drop-monthly-u-s-vehicle-sale-reports-idUSKCN1HA0C9. Accessed: 2018-05-07."},{"key":"e_1_2_1_18_1","unstructured":"Robin Wigglesworth. 2018a. Asset management's fight for alternative data analysts heats up. https:\/\/www.ft.com\/content\/2f454550-02c8--11e8--9650--9c0ad2d7c5b5. Accessed: 2018-05-07.  Robin Wigglesworth. 2018a. Asset management's fight for alternative data analysts heats up. https:\/\/www.ft.com\/content\/2f454550-02c8--11e8--9650--9c0ad2d7c5b5. Accessed: 2018-05-07."},{"key":"e_1_2_1_19_1","unstructured":"Robin Wigglesworth. 2018b. Asset managers double spending on new data in hunt for edge. https:\/\/www.ft.com\/content\/3c321c14--52d4--11e8-b24e-cad6aa67e23e. Accessed: 2018-05--10.  Robin Wigglesworth. 2018b. Asset managers double spending on new data in hunt for edge. https:\/\/www.ft.com\/content\/3c321c14--52d4--11e8-b24e-cad6aa67e23e. Accessed: 2018-05--10."}],"container-title":["Proceedings of the ACM on Measurement and Analysis of Computing Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3366694","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3366694","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:44:38Z","timestamp":1750203878000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3366694"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,12,17]]},"references-count":19,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2019,12,17]]}},"alternative-id":["10.1145\/3366694"],"URL":"https:\/\/doi.org\/10.1145\/3366694","relation":{},"ISSN":["2476-1249"],"issn-type":[{"type":"electronic","value":"2476-1249"}],"subject":[],"published":{"date-parts":[[2019,12,17]]},"assertion":[{"value":"2019-12-17","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}