library: libPhysics #include "TRobustEstimator.h"

# TRobustEstimator

class description - source file - inheritance tree (.pdf)

## class TRobustEstimator : public TObject

| Inheritance Chart:
`TObject`
| <- | `TRobustEstimator` |

        protected:

              void AddToSscp(TMatrixD& sscp, TVectorD& vec)
              void Classic()
              void ClearSscp(TMatrixD& sscp)
              void Correl()
              void Covar(TMatrixD& sscp, TVectorD& m, TMatrixDSym& cov, TVectorD& sd, Int_t nvec)
              void CreateOrtSubset(TMatrixD& dat, Int_t* index, Int_t hmerged, Int_t nmerged, TMatrixD& sscp, Double_t* ndist)
              void CreateSubset(Int_t ntotal, Int_t htotal, Int_t p, Int_t* index, TMatrixD& data, TMatrixD& sscp, Double_t* ndist)
          Double_t CStep(Int_t ntotal, Int_t htotal, Int_t* index, TMatrixD& data, TMatrixD& sscp, Double_t* ndist)
             Int_t Exact(Double_t* ndist)
             Int_t Exact2(TMatrixD& mstockbig, TMatrixD& cstockbig, TMatrixD& hyperplane, Double_t* deti, Int_t nbest, Int_t kgroup, TMatrixD& sscp, Double_t* ndist)
          Double_t KOrdStat(Int_t ntotal, Double_t* arr, Int_t k, Int_t* work)
             Int_t Partition(Int_t nmini, Int_t* indsubdat)
             Int_t RDist(TMatrixD& sscp)
              void RDraw(Int_t* subdat, Int_t ngroup, Int_t* indsubdat)

        public:

                             TRobustEstimator()
                             TRobustEstimator(Int_t nvectors, Int_t nvariables, Int_t hh = 0)
                             TRobustEstimator(const TRobustEstimator&)
                     virtual ~TRobustEstimator()
                        void AddColumn(Double_t* col)
                        void AddRow(Double_t* row)
              static TClass* Class()
                        void Evaluate()
                        void EvaluateUni(Int_t nvectors, Double_t* data, Double_t& mean, Double_t& sigma, Int_t hh = 0)
                       Int_t GetBDPoint()
                    Double_t GetChiQuant(Int_t i) const
                        void GetCorrelation(TMatrixDSym& matr)
          const TMatrixDSym* GetCorrelation() const
                        void GetCovariance(TMatrixDSym& matr)
          const TMatrixDSym* GetCovariance() const
                        void GetHyperplane(TVectorD& vec)
             const TVectorD* GetHyperplane() const
                        void GetMean(TVectorD& means)
             const TVectorD* GetMean() const
                       Int_t GetNHyp()
                       Int_t GetNOut()
                       Int_t GetNumberObservations() const
                       Int_t GetNvar() const
              const TArrayI* GetOuliers() const
                        void GetRDistances(TVectorD& rdist)
             const TVectorD* GetRDistances() const
             virtual TClass* IsA() const
           TRobustEstimator& operator=(const TRobustEstimator&)
                virtual void ShowMembers(TMemberInspector& insp, char* parent)
                virtual void Streamer(TBuffer& b)
                        void StreamerNVirtual(TBuffer& b)

### Data Members

        protected:

                Int_t fNvar           number of variables
                Int_t fH              algorithm parameter, determining the subsample size
                Int_t fN              number of observations
                Int_t fVarTemp        number of variables already added to the data matrix
                Int_t fVecTemp        number of observations already added to the data matrix
                Int_t fExact          if there was an exact fit, stores the number of points on a hyperplane
             TVectorD fMean           location estimate (mean values)
          TMatrixDSym fCovariance     covariance matrix estimate
          TMatrixDSym fInvcovariance  inverse of the covariance matrix
          TMatrixDSym fCorrelation    correlation matrix
             TVectorD fRd             array of robust distances, size n
             TVectorD fSd             array of standard deviations
              TArrayI fOut            array of indexes of ouliers, size <0.5*n
             TVectorD fHyperplane     in case more than fH observations lie on a hyperplane
             TMatrixD fData           the original data

## Class Description

      TRobustEstimator

     Minimum Covariance Determinant Estimator - a Fast Algorithm
     invented by Peter J.Rousseeuw and Katrien Van Dreissen
     "A Fast Algorithm for the Minimum covariance Determinant Estimator"
     Technometrics, August 1999, Vol.41, NO.3

     What are robust estimators?
     "An important property of an estimator is its robustness. An estimator
     is called robust if it is insensitive to measurements that deviate
     from the expected behaviour. There are 2 ways to treat such deviating
     measurements: one may either try to recongize them and then remove
     them from the data sample; or one may leave them in the sample, taking
     care that they do not influence the estimate unduly. In both cases robust
     estimators are needed...Robust procedures compensate for systematic errors
     as much as possible, and indicate any situation in which a danger of not being
     able to operate reliably is detected."
     R.Fruhwirth, M.Regler, R.K.Bock, H.Grote, D.Notz
     "Data Analysis Techniques for High-Energy Physics", 2nd edition

     What does this algorithm do?
     It computes a highly robust estimator of multivariate location and scatter.
     Then, it takes those estimates to compute robust distances of all the
     data vectors. Those with large robust distances are considered outliers.
     Robust distances can then be plotted for better visualization of the data.

     How does this algorithm do it?
     The MCD objective is to find h observations(out of n) whose classical
     covariance matrix has the lowest determinant. The MCD estimator of location
     is then the average of those h points and the MCD estimate of scatter
     is their covariance matrix. The minimum(and default) h = (n+nvariables+1)/2
     so the algorithm is effective when less than (n+nvar+1)/2 variables are outliers.
     The algorithm also allows for exact fit situations - that is, when h or more
     observations lie on a hyperplane. Then the algorithm still yields the MCD location T
     and scatter matrix S, the latter being singular as it should be. From (T,S) the
     program then computes the equation of the hyperplane.

     How can this algorithm be used?
     In any case, when contamination of data is suspected, that might influence
     the classical estimates.
     Also, robust estimation of location and scatter is a tool to robustify
     other multivariate techniques such as, for example, principal-component analysis
     and discriminant analysis.

     Technical details of the algorithm:
     0.The default h = (n+nvariables+1)/2, but the user may choose any interger h with
       (n+nvariables+1)/2<=h<=n. The program then reports the MCD's breakdown value
       (n-h+1)/n. If you are sure that the dataset contains less than 25% contamination
       which is usually the case, a good compromise between breakdown value and
      efficiency is obtained by putting h=[.75*n].
     1.If h=n,the MCD location estimate is the average of the whole dataset, and
       the MCD scatter estimate is its covariance matrix. Report this and stop
     2.If nvariables=1 (univariate data), compute the MCD estimate by the exact
       algorithm of Rousseeuw and Leroy (1987, pp.171-172) in O(nlogn)time and stop
     3.From here on, h<n and nvariables>=2.
       3a.If n is small:
        - repeat (say) 500 times:
        -- construct an initial h-subset, starting from a random (nvar+1)-subset
        -- carry out 2 C-steps (described in the comments of CStep funtion)
        - for the 10 results with lowest det(S):
        -- carry out C-steps until convergence
        - report the solution (T, S) with the lowest det(S)
       3b.If n is larger (say, n>600), then
        - construct up to 5 disjoint random subsets of size nsub (say, nsub=300)
        - inside each subset repeat 500/5 times:
        -- construct an initial subset of size hsub=[nsub*h/n]
        -- carry out 2 C-steps
        -- keep the best 10 results (Tsub, Ssub)
        - pool the subsets, yielding the merged set (say, of size nmerged=1500)
        - in the merged set, repeat for each of the 50 solutions (Tsub, Ssub)
        -- carry out 2 C-steps
        -- keep the 10 best results
        - in the full dataset, repeat for those best results:
        -- take several C-steps, using n and h
        -- report the best final result (T, S)
     4.To obtain consistency when the data comes from a multivariate normal
       distribution, covariance matrix is multiplied by a correction factor
     5.Robust distances for all elements, using the final (T, S) are calculated
       Then the very final mean and covariance estimates are calculated only for
       values, whose robust distances are less than a cutoff value (0.975 quantile
       of chi2 distribution with nvariables degrees of freedom)

TRobustEstimator()

    this constructor should be used in a univariate case:
    first call this constructor, then - the EvaluateUni(..) fucntion

TRobustEstimator(Int_t nvectors, Int_t nvariables, Int_t hh) :fMean(nvariables), fCovariance(nvariables), fInvcovariance(nvariables), fCorrelation(nvariables), fRd(nvectors), fSd(nvectors), fOut(1), fHyperplane(nvariables), fData(nvectors, nvariables)

void AddColumn(Double_t *col)

    adds a column to the data matrix
    it is assumed that the column has size fN
    variable fVarTemp keeps the number of columns l
    already added

void AddRow(Double_t *row)

    adds a vector to the data matrix
    it is supposed that the vector is of size fNvar

void Evaluate()

void EvaluateUni(Int_t nvectors, Double_t *data, Double_t &mean, Double_t &sigma, Int_t hh)

    for the univariate case
    estimates of location and scatter are returned in mean and sigma parameters
    the algorithm works on the same principle as in multivariate case -
    it finds a subset of size hh with smallest sigma, and then returns mean and
    sigma of this subset

Int_t GetBDPoint()

    returns the breakdown point of the algorithm

Double_t GetChiQuant(Int_t i) const

void GetCovariance(TMatrixDSym &matr)

void GetCorrelation(TMatrixDSym &matr)

const TVectorD* GetHyperplane() const

void GetHyperplane(TVectorD &vec)

void GetMean(TVectorD &means)

void GetRDistances(TVectorD &rdist)

Int_t GetNOut()

void AddToSscp(TMatrixD &sscp, TVectorD &vec)

    update the sscp matrix with vector vec

void ClearSscp(TMatrixD &sscp)

    clear the sscp matrix, used for covariance and mean calculation

void Classic()

    called when h=n. Returns classic covariance matrix
    and mean

void Covar(TMatrixD &sscp, TVectorD &m, TMatrixDSym &cov, TVectorD &sd, Int_t nvec)

    calculates mean and covariance

void Correl()

    transforms covariance matrix into correlation matrix

void CreateSubset(Int_t ntotal, Int_t htotal, Int_t p, Int_t *index, TMatrixD &data, TMatrixD &sscp, Double_t *ndist)

    creates a subset of htotal elements from ntotal elements
    first, p+1 elements are drawn randomly(without repetitions)
    if their covariance matrix is singular, more elements are
    added one by one, until their covariance matrix becomes regular
    or it becomes clear that htotal observations lie on a hyperplane
    If covariance matrix determinant!=0, distances of all ntotal elements
    are calculated, using formula d_i=Sqrt((x_i-M)*S_inv*(x_i-M)), where
    M is mean and S_inv is the inverse of the covariance matrix
    htotal points with smallest distances are included in the returned subset.

void CreateOrtSubset(TMatrixD &dat,Int_t *index, Int_t hmerged, Int_t nmerged, TMatrixD &sscp, Double_t *ndist)

    creates a subset of hmerged vectors with smallest orthogonal distances to the hyperplane
    hyp[1]*(x1-mean[1])+...+hyp[nvar]*(xnvar-mean[nvar])=0
    This function is called in case when less than fH samples lie on a hyperplane.

Double_t CStep(Int_t ntotal, Int_t htotal, Int_t *index, TMatrixD &data, TMatrixD &sscp, Double_t *ndist)

    from the input htotal-subset constructs another htotal subset with lower determinant

    As proven by Peter J.Rousseeuw and Katrien Van Driessen, if distances for all elements
    are calculated, using the formula:d_i=Sqrt((x_i-M)*S_inv*(x_i-M)), where M is the mean
    of the input htotal-subset, and S_inv - the inverse of its covariance matrix, then
    htotal elements with smallest distances will have covariance matrix with determinant
    less or equal to the determinant of the input subset covariance matrix.

    determinant for this htotal-subset with smallest distances is returned

Int_t Exact(Double_t *ndist)

    for the exact fit situaions
    returns number of observations on the hyperplane

Int_t Exact2(TMatrixD &mstockbig, TMatrixD &cstockbig, TMatrixD &hyperplane, Double_t *deti, Int_t nbest, Int_t kgroup, TMatrixD &sscp, Double_t *ndist)

    This function is called if determinant of the covariance matrix of a subset=0.

    If there are more then fH vectors on a hyperplane,
    returns this hyperplane and stops
    else stores the hyperplane coordinates in hyperplane matrix

Int_t Partition(Int_t nmini, Int_t *indsubdat)

    divides the elements into approximately equal subgroups
    number of elements in each subgroup is stored in indsubdat
    number of subgroups is returned

Int_t RDist(TMatrixD &sscp)

    Calculates robust distances.Then the samples with robust distances
    greater than a cutoff value (0.975 quantile of chi2 distribution with
    fNvar degrees of freedom, multiplied by a correction factor), are given
    weiht=0, and new, reweighted estimates of location and scatter are calculated
    The function returns the number of outliers.

void RDraw(Int_t *subdat, Int_t ngroup, Int_t *indsubdat)

    Draws ngroup nonoverlapping subdatasets out of a dataset of size n
    such that the selected case numbers are uniformly distributed from 1 to n

Double_t KOrdStat(Int_t ntotal, Double_t *a, Int_t k, Int_t *work)

    because I need an Int_t work array

### Inline Functions

                          void ~TRobustEstimator()
            const TMatrixDSym* GetCovariance() const
            const TMatrixDSym* GetCorrelation() const
                         Int_t GetNHyp()
               const TVectorD* GetMean() const
               const TVectorD* GetRDistances() const
                         Int_t GetNumberObservations() const
                         Int_t GetNvar() const
                const TArrayI* GetOuliers() const
                       TClass* Class()
                       TClass* IsA() const
                          void ShowMembers(TMemberInspector& insp, char* parent)
                          void Streamer(TBuffer& b)
                          void StreamerNVirtual(TBuffer& b)
              TRobustEstimator TRobustEstimator(const TRobustEstimator&)
             TRobustEstimator& operator=(const TRobustEstimator&)

Author: Anna Kreshuk 08/10/2004
Last update: root/physics:$Name: $:$Id: TRobustEstimator.cxx,v 1.9 2004/10/21 14:44:02 rdm Exp $
Copyright (C) 1995-2004, Rene Brun and Fons Rademakers. *