Research Reports

Cost functional method

The original paper propose is to minimize the Kolmogorov-Smirnov standard statstics, given by: \[ \mathcal{K}\big[F\big] = \max_{x_j\geq x_{\min}}\left\lbrace \mathcal{S}(x_j)-F(x_j;\theta) \right\rbrace, \] but we can give a weight \(w(x)\) to this measure, defining the weighted-KS statistics for a theoretical distribution \(F\): \[ \mathcal{K}_{w}\big[F\big] = \max_{x_j\geq x_{\min}}\left\lbrace \frac{\mathcal{S}(x_j)-F(x_j;\theta)}{w(x_j)} \right\rbrace. \] In fact, one can show that to choose \(w(x)=\sqrt{F(x;\theta)(1-F(x;\theta))}\) is to make the measure variations uniformly over the interval \(x_j\geq x_{\min}\).

With these two stats, the standard one and the uniformly one, the long tail has the same weight as the data in the \(x_{\min}\) neighborhood. So, if we want to maximize the long tail, we can build a weight that is low near \(x_{\min}\) and grows in the tail. One solution is to consider \(w(x)=F(x;\theta)\) as \(F(x_{\min})=0\) and \(F(\infty)\to1\). That is: \[ \mathcal{K}_{w}\big[F\big] \max_{x_j\geq x_{\min}}\left\lbrace \frac{\mathcal{S}(x_j)-F(x_j;\theta)}{F(x_j;\theta)} \right\rbrace. \] This weight is, however, instable and does not give a good statistical measure.

In other way, we can define a cost functional: \[ \mathcal{L}[F,x_{\min}] = \mathcal{K}_{w}\big[F\big] - \lambda\frac{n_{\text{tail}}(x_{\min})}{N}, \] where \(N\) is the original data size. The interpratation is imediate: \(\lambda\geq0\) is the cost of the longtail length and minimize \(\mathcal{K}_{w}\big[F\big]\) while considering the importance of the long tail is to minimize the functional \(\mathcal{L}[F,x_{\min}]\). This revealed to be much more stable and rigorous.

Fitting with cost functional

The quantity \(\mathcal{L}[F,x_{\min}]\) does not represent any formal statistical measure; instead, we need information theory to give a formal sense to it. The cost \(\lambda\) is, somehow, like a lagrangian multiplier so that \[-\lambda\frac{n_{\text{tail}}(x_{\min})}{N}\] represents the lost information if we drop \(N-n_{\text{tail}}+1\) points from the original data. Minimizing \(\mathcal{L}[F,x_{\min}]\), then, is minimize the KS measure minimizing the lost of information in the tail.

\(\lambda\) is a trade off parameters which tells us, for a given KS value, how plausible a given \(x_{\min}\) can be for the cost functional. So we have to apply the p-value bootstrap test to verify what are the plausible values for \(\lambda\); in other words, \(\lambda\) cannot be a free parameters for the model. The values ​​that we can accept for \(\lambda\) are that ones which can give rise to best values for \(x_{\min}\) while keeping the traditional null hypothesis test, \(p\geq 0.1\).