2014 Looking At People ECCV Challenge - Track 1: Human Pose Recovery

The Jaccard Index (overlapping) will be used. Thus, for each one of the n=14 limbs labelled for each subject at frame i, the Jaccard Index is defined as follows:
 

where A_{i,n} is the ground truth of limb n, and B_{i,n} is the prediction for the same limb at image i. For the dataset in this challenge, both A_{i,n} and B_{i,n} are binary images where ‘1’ pixels denote the region in which the n-th limb is predicted.  Particularly, since A_{i,n}  (ground truth) is a binary image and 1-value pixels indicate the region of the n-th limb, this positive region does not necessarily need to be square. However, in all cases the positive region is a polyhedron defined by four points. Thus, numerator is the number ‘1’ pixels that intersects in both images A_{i,n} and B_{i,n}, and denominator is the number of union ‘1’ pixels after applying local or operator. 

In the case of false positives (e.g predicting a limb that is not on the ground truth because of being occluded), the prediction will not affect the mean Hit Rate calculation. In other words n is computed as the intersection of the limb categories in the ground truth and the predictions.

Participant methods will be evaluated upon hit rate (HR) detection of limbs. That is, for each limb n at each image i, a hit will be computed if J_{i,n}≥0.5. Then, the mean hit rate among all limbs for all images will be computed (where all limb detections will have the same weight) and the participant with the highest mean hit rate will be the winner.
 

In some images a limb may not labelled in the ground truth because of occlusions. In that case where n<14, participants must not provide any prediction for that particular limb. An example of the mean hit rate calculation for an example of n=3 limbs and i=1 image is show next.
 

This figure shows the Mean hit rate and Jaccard Index calculation for a sample with n=3 limbs and i=1 image. In the top part of the image the Jaccard Index for the head limb is computed. As it is greater than 0.5 then it is computed as a hit for image i and the head limb. Similarly, for the torso limb the Jaccard Index obtained is 0.72 (center part of the image) which also computes as a hit for torso limb. In addition, in the bottom of the image the Jaccard Index obtained for the left thigh limb is shown, which does not compute as a hit since 0.04<0.5. Finally, the mean hit rate is obtained for those three limbs. 

Participants must submit their predictions in the following format: For each RGB image XX_YYYY.jpg in the validation/test sequences participants should submit no more than 2 binary PNG images (one for each actor), where each PNG image has the following name structure XX_YYYY_W.png. Where, W denotes the actor in the sequence (W=1 if the leftmost and W=2 if the rightmost). This image is the concatenation of 14 BW indicator images (one per limb), in increasing order from limb id 1 to limb id 14.