CPerformanceMeasures Class Reference


Detailed Description

Class to implement various performance measures.

based on: Fawcett, T: March 2004, ROC Graphs: Notes and Practical Considerations for Researchers and input from Sonnenburg, S: Feburary 2008, various discussions

Definition at line 41 of file PerformanceMeasures.h.

Inheritance diagram for CPerformanceMeasures:
Inheritance graph
[legend]

List of all members.

Public Member Functions

 CPerformanceMeasures ()
 CPerformanceMeasures (CLabels *true_labels, CLabels *output)
virtual ~CPerformanceMeasures ()
void init (CLabels *true_labels, CLabels *output)
bool set_true_labels (CLabels *true_labels)
CLabelsget_true_labels () const
bool set_output (CLabels *output)
CLabelsget_output () const
int32_t get_num_labels () const
void get_ROC (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_auROC ()
float64_t get_aoROC ()
void get_PRC (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_auPRC ()
float64_t get_aoPRC ()
void get_DET (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_auDET ()
float64_t get_aoDET ()
void get_all_accuracy (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_accuracy (float64_t threshold=0)
void get_all_error (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_error (float64_t threshold=0)
void get_all_fmeasure (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_fmeasure (float64_t threshold=0)
void get_all_CC (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_CC (float64_t threshold=0)
void get_all_WRAcc (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_WRAcc (float64_t threshold=0)
void get_all_BAL (float64_t **result, int32_t *num, int32_t *dim)
float64_t get_BAL (float64_t threshold=0)
virtual const char * get_name () const

Protected Member Functions

void init_nolabels ()
float64_t trapezoid_area (float64_t x1, float64_t x2, float64_t y1, float64_t y2)
void create_sortedROC ()
void compute_ROC (float64_t **result)
void compute_accuracy (float64_t **result, int32_t *num, int32_t *dim, bool do_error=false)
void compute_PRC (float64_t **result)
void compute_DET (float64_t **result)
void compute_confusion_matrix (float64_t threshold, int32_t *tp, int32_t *fp, int32_t *fn, int32_t *tn)

Protected Attributes

CLabelsm_true_labels
CLabelsm_output
int32_t m_num_labels
int32_t m_all_true
int32_t m_all_false
int32_t * m_sortedROC
float64_t m_auROC
float64_t m_auPRC
float64_t m_auDET

Constructor & Destructor Documentation

CPerformanceMeasures::CPerformanceMeasures (  ) 

default constructor

Definition at line 16 of file PerformanceMeasures.cpp.

CPerformanceMeasures::CPerformanceMeasures ( CLabels true_labels,
CLabels output 
)

constructor

Parameters:
true_labels true labels as seen in real world
output output labels/hypothesis from a classifier

Definition at line 22 of file PerformanceMeasures.cpp.

CPerformanceMeasures::~CPerformanceMeasures (  )  [virtual]

Definition at line 29 of file PerformanceMeasures.cpp.


Member Function Documentation

void CPerformanceMeasures::compute_accuracy ( float64_t **  result,
int32_t *  num,
int32_t *  dim,
bool  do_error = false 
) [protected]

compute ROC accuracy/error

Parameters:
result where the result will be stored
num number of points
dim dimension == 2 (output, accuracy/error)
do_error if error instead of accuracy shall be computed

Definition at line 378 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::compute_confusion_matrix ( float64_t  threshold,
int32_t *  tp,
int32_t *  fp,
int32_t *  fn,
int32_t *  tn 
) [protected]

compute confusion matrix

caller has to delete[]

Parameters:
threshold threshold to compute against
tp storage of true positives or NULL if unused
fp storage of false positives or NULL if unused
fn storage of false negatives or NULL if unused
tn storage of true negatives or NULL if unused

Definition at line 141 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::compute_DET ( float64_t **  result  )  [protected]

compute DET points and auDET

Parameters:
result where the result will be stored

Definition at line 325 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::compute_PRC ( float64_t **  result  )  [protected]

compute PRC points and auPRC

Parameters:
result where the result will be stored

Definition at line 275 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::compute_ROC ( float64_t **  result  )  [protected]

compute ROC points and auROC

Definition at line 201 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::create_sortedROC (  )  [protected]

create index for ROC sorting

Definition at line 113 of file PerformanceMeasures.cpp.

float64_t CPerformanceMeasures::get_accuracy ( float64_t  threshold = 0  ) 

get classifier's accuracy at given threshold

Parameters:
threshold all values below are considered negative examples (default 0)
Returns:
classifer's accuracy at threshold

Definition at line 366 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_all_accuracy ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get classifier's accuracies (swig compatible)

accuracy = (true positives + true negatives) / all labels

caller has to free

Parameters:
result storage of accuracies in 2 dim array: (output, accuracy), sorted by output
num number of accuracy points
dim dimension == 2 (output, accuracy)

Definition at line 406 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_all_BAL ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get classifier's Balanced Error (swig compatible)

BAL = 0.5 * ( true positives / all true labels + true negatives / all false labels )

caller has to free

Parameters:
result storage of BAL in 2 dim array: (output, BAL), sorted by output
num number of BAL points
dim dimension == 2 (output, BAL)

Definition at line 578 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_all_CC ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get classifier's Cross Correlation coefficients (swig compatible)

CC = ( true positives * true negatives

  • false positives * false negatives ) / sqrt( (true positives + false positives) * (true positives + false negatives) * (true negatives + false positives) * (true negatives + false negatives) )

also check http://en.wikipedia.org/wiki/Correlation

caller has to free

Parameters:
result storage of CCs in 2 dim array: (output, CC), sorted by output
num number of CC points
dim dimension == 2 (output, CC)

Definition at line 487 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_all_error ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get classifier's error rates (swig compatible)

value is 1 - accuracy

caller has to free

Parameters:
result storage of errors in 2 dim array: (output, error), sorted by output
num number of accuracy points
dim dimension == 2 (output, error)

Definition at line 412 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_all_fmeasure ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get classifier's F-measure (swig compatible)

F-measure = 2 / (1 / precision + 1 / recall)

caller has to free

Parameters:
result storage of F-measure in 2 dim array (output, fmeasure), sorted by output
num number of accuracy points
dim dimension == 2 (output, fmeasure)

Definition at line 449 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_all_WRAcc ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get classifier's Weighted Relative Accuracy (swig compatible)

WRAcc = ( true positives / (true positives + false negatives) )

  • ( false positives / (false positives + true negatives) )

caller has to free

Parameters:
result storage of WRAccs in 2 dim array: (output, WRAcc), sorted by output
num number of WRAcc points
dim dimension == 2 (output, WRAcc)

Definition at line 534 of file PerformanceMeasures.cpp.

float64_t CPerformanceMeasures::get_aoDET (  ) 

return area over Detection Error Tradeoff curve

value is 1 - auDET

Returns:
area over DET curve

Definition at line 226 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_aoPRC (  ) 

return area over Precision Recall Curve

value is 1 - auPRC

Returns:
area over PRC

Definition at line 184 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_aoROC (  ) 

return area over Reveiver Operating Curve

value is 1 - auROC

Returns:
area over ROC

Definition at line 142 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_auDET (  ) 

return area under Detection Error Tradeoff curve

calculated by adding trapezoids

Returns:
area under DET curve

Definition at line 209 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_auPRC (  ) 

return area under Precision Recall Curve

calculated by adding trapezoids

Returns:
area under PRC

Definition at line 167 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_auROC (  ) 

return area under Receiver Operating Curve

calculated by adding trapezoids

Returns:
area under ROC

Definition at line 125 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_BAL ( float64_t  threshold = 0  ) 

get classifier's Balanced Error at threshold 0

Returns:
classifer's BAL at threshold 0

Definition at line 562 of file PerformanceMeasures.cpp.

float64_t CPerformanceMeasures::get_CC ( float64_t  threshold = 0  ) 

get classifier's Cross Correlation coefficient at threshold

Returns:
classifer's CC at threshold

Definition at line 473 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_DET ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get Detection Error Tradeoff curve for previously given labels (swig compatible)

DET point = false positives / all false labels, false negatives / all false labels

caller has to free

Parameters:
result where computed DET values will be stored
num number of labels/examples
dim dimension == 2 (false positive rate, false negative rate)

Definition at line 316 of file PerformanceMeasures.cpp.

float64_t CPerformanceMeasures::get_error ( float64_t  threshold = 0  ) 

get classifier's error at threshold

value is 1 - accuracy0

Parameters:
threshold all values below are considered negative examples (default 0)
Returns:
classifer's error at threshold

Definition at line 273 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_fmeasure ( float64_t  threshold = 0  ) 

get classifier's F-measure at threshold 0

Returns:
classifer's F-measure at threshold 0

Definition at line 420 of file PerformanceMeasures.cpp.

virtual const char* CPerformanceMeasures::get_name (  )  const [virtual]

get the name of tghe object

Returns:
name of object

Implements CSGObject.

Definition at line 386 of file PerformanceMeasures.h.

int32_t CPerformanceMeasures::get_num_labels (  )  const

get number of labels in output/true labels

Returns:
number of labels in output/true labels

Definition at line 103 of file PerformanceMeasures.h.

CLabels* CPerformanceMeasures::get_output (  )  const

get classifier's output labels/hypothesis

Returns:
output labels

Definition at line 97 of file PerformanceMeasures.h.

void CPerformanceMeasures::get_PRC ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get Precision Recall Curve for previously given labels (swig compatible)

PRC point = true positives / all true labels, true positives / (true positives + false positives)

caller has to free

Parameters:
result where computed ROC values will be stored
num number of labels/examples
dim dimension == 2 (recall, precision)

Definition at line 266 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::get_ROC ( float64_t **  result,
int32_t *  num,
int32_t *  dim 
)

get Receiver Operating Curve for previously given labels (swig compatible)

ROC point = false positives / all false labels, true positives / all true labels

caller has to free

Parameters:
result where computed ROC values will be stored
num number of labels/examples
dim dimensions == 2 (false positive rate, true positive rate)

Definition at line 193 of file PerformanceMeasures.cpp.

CLabels* CPerformanceMeasures::get_true_labels (  )  const

get true labels as seen in real world

Returns:
true labels as seen in real world

Definition at line 79 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::get_WRAcc ( float64_t  threshold = 0  ) 

get classifier's Weighted Relative Accuracy at threshold 0

Returns:
classifer's WRAcc at threshold 0

Definition at line 515 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::init ( CLabels true_labels,
CLabels output 
)

initialise performance measures

Parameters:
true_labels true labels as seen in real world
output output labels/hypothesis from a classifier

Definition at line 53 of file PerformanceMeasures.cpp.

void CPerformanceMeasures::init_nolabels (  )  [protected]

initialise values independent from true labels/output

Definition at line 43 of file PerformanceMeasures.cpp.

bool CPerformanceMeasures::set_output ( CLabels output  ) 

set output labels/hypothesis from a classifier

Parameters:
output output labels
Returns:
if setting was successful

Definition at line 86 of file PerformanceMeasures.h.

bool CPerformanceMeasures::set_true_labels ( CLabels true_labels  ) 

set true labels as seen in real world

Parameters:
true_labels true labels
Returns:
if setting was successful

Definition at line 68 of file PerformanceMeasures.h.

float64_t CPerformanceMeasures::trapezoid_area ( float64_t  x1,
float64_t  x2,
float64_t  y1,
float64_t  y2 
) [protected]

calculate trapezoid area for auROC

Parameters:
x1 x coordinate of point 1
x2 x coordinate of point 2
y1 y coordinate of point 1
y2 y coordinate of point 2
Returns:
trapezoid area for auROC

Definition at line 133 of file PerformanceMeasures.cpp.


Member Data Documentation

int32_t CPerformanceMeasures::m_all_false [protected]

number of negative examples in true_labels

Definition at line 459 of file PerformanceMeasures.h.

int32_t CPerformanceMeasures::m_all_true [protected]

number of positive examples in true_labels

Definition at line 457 of file PerformanceMeasures.h.

area under DET; 1 - area over DET

Definition at line 469 of file PerformanceMeasures.h.

area under PRC; 1 - area over PRC

Definition at line 467 of file PerformanceMeasures.h.

area under ROC; 1 - area over ROC

Definition at line 465 of file PerformanceMeasures.h.

number of true labels/outputs/accuracies/ROC points

Definition at line 454 of file PerformanceMeasures.h.

output labels/hypothesis from a classifier

Definition at line 452 of file PerformanceMeasures.h.

int32_t* CPerformanceMeasures::m_sortedROC [protected]

array of size num_labels with indices of true_labels/output sorted to fit ROC algorithm

Definition at line 463 of file PerformanceMeasures.h.

true labels/examples as seen in real world

Definition at line 450 of file PerformanceMeasures.h.


The documentation for this class was generated from the following files:

SHOGUN Machine Learning Toolbox - Documentation