Dimension Estimation and Application in Finance

Bahadur, Nitish

Etd

Dimension Estimation and Application in Finance

Public

Dimension Estimation (DE) and Dimension Reduction (DR) are two closely related topics, but with quite different goals. In DR, one attempts to project a random vector, either linearly or non-linearly, to a lower dimensional space that preserves the information contained in the original higher dimensional space. However, in DE, one attempts to estimate the intrinsic dimensionality or number of latent variables in a set of measurements of a random vector. Our research incorporates both theories and applications involving DE and deep learning in the context of machine learning and financial markets. The precise mapping between low-dimensional ambient space and high-dimensional Euclidean space is complex and non-linear. Further, high-dimensional data contains noise, anomalies, and missing observations. These challenges make dimension estimation difficult. To address these challenges, we develop a novel robust deep autoencoder dimension estimation pipeline called RAEDE. The RAEDE leverages the computational power of an autoencoder (AE) with the well-known mathematical foundation of sparsity and consistency seen in recent work on robust deep AEs. This combination results in a unified robust dimension estimation pipeline that can successfully perform dimension estimation on a variety of real-world domains such as Finance and Cybersecurity. Towards this goal, we show: Repurposing Autoencoder innermost layer to create singular value proxies can facilitate dimension estimation of complex non-linear manifolds where traditional methods such as PCA, MDS, and Isomap overestimate the dimension. We validate our approach on synthetic data and apply our DE pipeline to sector Exchange Traded Funds. Using our DE pipeline, we estimate the dimension of Equity markets over last 30 years and find that sudden change in the dimension can be used as a proxy for market crash. Under stressed market conditions, the dimension of financial markets reduces drastically. Further, the dimension of the market reverts to its long-term historical average as the stress diminishes. Using our RAEDE, we estimate the dimension of high-frequency trading data that has anomalies and missing entries.

Creator