Nir Billfeld

Nir Billfeld, Moshe Kim

A new and an enriched JPEG algorithm is provided for identifying redundancies in a sequence of irregular noisy data points which also accommodates a reference-free criterion function. Our main contribution is by formulating analytically (instead of approximating) the inverse of the transpose of JPEGwavelet transform without involving matrices which are computationally cumbersome. The algorithm is suitable for the widely-spread situations where the original data distribution is unobservable such as in cases where there is deficient representation of the entire population in the training data (in machine learning) and thus the covariate shift assumption is violated. The proposed estimator corrects for both biases, the one generated by endogenous truncation and the one generated by endogenous covariates. Results from utilizing 2,000,000 different distribution functions verify the applicability and high accuracy of our procedure to cases in which the disturbances are neither jointly nor marginally normally distributed.

Nir Billfeld, Moshe Kim

We synthesize the knowledge present in various scientific disciplines for the development of semiparametric endogenous truncation-proof algorithm, correcting for truncation bias due to endogenous self-selection. This synthesis enriches the algorithm's accuracy, efficiency and applicability. Improving upon the covariate shift assumption, data are intrinsically affected and largely generated by their own behavior (cognition). Refining the concept of Vox Populi (Wisdom of Crowd) allows data points to sort themselves out depending on their estimated latent reference group opinion space. Monte Carlo simulations, based on 2,000,000 different distribution functions, practically generating 100 million realizations, attest to a very high accuracy of our model.