Descriptions¶
This is a short overview of the used naming conventions. For a more detailed discussion check out for example the paper [WLG11].
- Ground truth events:
- actual events that were annotated and will be considered as true instances of an activity. (However depending on the annotation quality they are not always 100% correct.)
- Detected events:
- predicted events - typically results of a classifier’s prediction step
Standard scores¶
Can be defined on a frame-by-frame basis. To calculate score values each frame is categorized with one of the following score labels:
- True positives (TP):
- ground truth label for the current frame or segment equals the detected (predicted) label
- False positives (FP):
- current frame or segment has a detected label for a particular class however the ground truth indicates that this class is not active
- False negatives (FN):
- detected label is not equal to the ground truth when ground truth shows presence of the class
- True negatives (TN):
- when neither the detected nor the corresponding ground truth label represents the class
Segments can be defined as a group of sequential frames where the assigned score labels don’t change.
- Precision:
- \[precision = \frac{TP}{TP + FP}\]
- Recall:
- \[recall = \frac{TP}{TP + FN}\]
Calculation of event-based precision and recall¶
We can Assumption for True positive events False positive False negatives
TP_det, FP_det todo: add formula
As mentioned in [WLG11], there are several issues with these methods for event-based evaluation, especially that the nature of the errors remain mostly hidden. The authors propose additional segment categories to better describe the results.
Detailed scores¶
Detailed scores for ground truth events¶
- C - Correct:
- todo
- F - fragmented:
- todo
- M - merged:
- todo
- FM - fragmented and merged:
- todo
- D - deletion:
- todo practically equivalent of a false negative ground truth event
Detailed scores for detection events¶
C - Correct F’ - fragmenting: … M’ - merging: … FM’ - fragmenting and merging: … I’ - insertion: … basically equivalent of a false positive detection event
Detailed scores for segments¶
Detailed segment categories:¶
By using the detailed score categories, we can also evaluate the dataset on a frame-by-frame base receiving additional information on the event based errors.
To each segment (group of sequential frames with the same score label) we can assign one of the following categories:
- True positive - TP
- True negative - TN
- Insertion - I
- Merge - M,
- Deletion - D
- Fragmenting - F
- Start overfill - Os
- End overfill - Oe
- Start underfill - Us
- End underfill - Ue
As shown on the next figure:
Segment based metrics (2SET-metrics):¶
todo: add example plot