Text

theses

With the increasing availability of big data, it is challenging to analyze complex data that is high dimensional; high volume but heterogeneous; or imposed with constraints. My dissertation compiles researches from three different areas to address the solutions to those challenging problems in complex data analysis.

In Part I, we first solve the constrained sampling problem in state space models, which is usually difficult due to potential strong constraints. The proposed Sequential Monte Carlo with constraints (SMCc) algorithm provides a general framework to sample efficiently from a state space model with constraints. An optimal priority score used in the resampling step of sequential Monte Carlo (SMC) is introduced as a compromise between accuracy and computation. Several computationally efficient ways of approximating the optimal priority are presented.

The second half of Part I focuses on utilizing state space models and SMC to solve high dimensional optimization problems, in which traditional optimization algorithms usually have their limitations. We propose to first reformulate the optimization problem into the likelihood function of an artificially designed state space model (the emulation step) and then find the optimal solution through a novel simulated annealing algorithm for state space models (the annealed SMC step). The procedure is demonstrated with several canonical statistical examples.

In Part II, we propose an individualized group learning (iGroup) framework, lying at the intersection fusion learning and individualized inference, to provide a more concrete statistical inference on a particular individual of interest, by aggregating information of similar individuals from a potentially heterogeneous population. The optimality of such a methodology is shown under the asymptotic setting that the population size approaches infinity while each individual has a finite number of observations. The improvement of iGroup over individual level estimate and the population level estimate (as in traditional fusion learning) are demonstrated with simulations and real data examples.

In Part III, we consider the family of KoPA approaches, which approximate a high dimensional matrix with one or more Kronecker products. Using Kronecker product instead of vector outer product introduces much higher flexibility in choosing the configuration (sizes of the two smaller matrices), while it gives rise to the problem of choosing the optimal one. An extended information criterion is proposed to automatically select the optimal configuration. Consistency of the configuration selection is provided with rigorous analysis. In addition, the KoPA approach can be extended to matrix completion problems as well with a superior performance over traditional SVD as demonstrated in Part III with a real image example.

ETD_11143

Ph.D.

Includes bibliographical references

ETD doctoral

doi:10.7282/t3-wwr4-gp69

The author owns the copyright to this work.