Training Set Acquisition

How are you evaluated?

We evaluate your algorithm across five separate marketplaces and average the score of the five.

For each marketplace, based on the number of data points in your submission, the data samples from each seller (hidden from you) will be drawn uniform random to generate the training set.

The training set would be used to train a logistic regression model (as specified in the DAM document) and evaluate the trained model on the buyer's dataset within each marketplace.

Scoring Metric

The scoring per marketplace would include two components: accuracy and budget.  You will be rewarded for a buying strategy that increases accuracy. You will also be rewarded if your strategy does not use the entire budget.

final_score = alpha * accuracy + (1-alpha) * (budget - cost) / budget

alpha = 0.98

Online evaluation (Dynabench)

Submissions will be evaluated by DynaBench.

The evaluation in DynaBench takes about 1 hour.