As soon as the new formula is defined by the organizing committee, we
will provide the new rankings of
all the qualified teams on the data set A allowing the qualified teams
to appreciate the new formula's ranking.
(2) For the final stage, we will provide soon (mid-June)
the data set B allowing the qualified teams to better
tune their programs.
The data sets B and X will be harder
than the data Set A "pushing the programs to the limits".
The number of instances in the data sets B and X will be increased
in order to represent all the factories of RENAULT.
(3) The weights (coefficents) of the objectives are presently
(10000, 100 and 1), but they can not completely
guarantee the non balance between the objectives. This did not change
the rankings of the qualification stage.
For the final stage, the weights will be respectively
1000000, 1000 and 1.
(4) Several programs had some weird
behaviors (runtime errors, no respect to the maximal paint batch
length)
on a new and unknown data instance we used
to test the programs.
- Bloemen the program does not stop
- Dutot solution invalid : paint color
batch too long
- Gavranovic the program crashes
- Gravel the program crashes
- Jourdan the program does not stop
- Klau solution invalid : paint color
batch too long
- Naddef the program does not generate
a solution (log : "Meilleure solution trouvee : aucune .")
- Nault the program crashes
- Pawlak solution invalid : paint color
batch too long
- Ribeiro solution invalid : paint color
batch too long
This unknown data instance could be found hereDOU_EP_RAF_ENP_chA.zip
(for Windows) and DOU_EP_RAF_ENP_chA.tar.gz
(for Linux).
This unknown data instance has the following characteristics: the cars
of the day D-1 have the same color and they are not subject to any ratio
constraints.
After checking with a candidate the reason why their program has this
behavior, it comes out that their program
uses the solution provided by RENAULT as initial solution. Since
the
unknown data instance is not a solution,
their program generated a unexpected error.
Thus, please to note the data set X will have instances like this unknown
one since the data set X will
contain instances coming from RENAULT's factories and these instances
will not be solutions.
(5) On student travel expense supports and the following sentence
on this challenge home page:
"French OR society ROADEF would support part of the travel expenses
to the conference ROADEF'2005
for the students who reach the final stage".
The junior teams should read:
"French OR society ROADEF would support part of the travel expenses
to the conference ROADEF'2005
for the students who reach the final stage
and finish this challenge as winner or 2nd or 3rd".
These student supports, which are not part
of the prizes, are given on motivated requests (e.g. from a remote country).
This was the case for the previous challenges. Thank you for understanding
that since the French OR society
is a nonlucractive association, so these supports are generally low,
around 450 euros maximum per junior team.
(6) Comparison between the old and the new evaluation functions:
The old evaluation:
(a) For each instance and each candidate,compute the mark with the
weights of the objectives
(the best candidates have the lower marks) :
mark=objective_1_value*10000 + objective_2_value*100 + objective_3_value(b) For each candidate and each data series, compute the average of the candidate's marks on different instances.
(candidate_mark- worst_candidate_mark_on_the_series) / (best_candidate_mark_on_the_series - worst_candidate_mark_on_the_series)
(d) The global mark of the candidate is the average of the weighted
marks obtained over the three data series.
(e) Ranking on the global marks.
Drawbacks of this evaluation: the objectives are not completely without
compensation due to the weights.
The averages reduce the gaps and give favor to the instances with high
objective values. The high 1st objective value reduces
the gap between candidates having the same 1st objective value.
The improved evaluation:
weights over the objectives : 1000000, 1000 and 1 => the objectives
are guaranteed to be without compensation.
(a) For each instance and each candidate,compute the mark with the weights
of the objectives
(the best candidates have the lower marks) :
mark=objective_1_value*1000000 + objective_2_value*1000 + objective_3_value(b) For each candidate and each instance, compute the corrected mark of the candidate :
corrected_mark=mark - best_candidate_objective_1_value_on_the_instance*1000000(c) For each candidate and each instance, compute the candidate's weighted mark according to the following formula:
(corrected_mark - worst_candidate_corrected_mark) / (best_candidate_corrected_mark - worst_candidate_corrected_mark)(d) Compute the candidate's global mark as the average of the candidate's weighted marks of the instances.
We computed as well, for indication only, the rankings for each series of instances.
The main idea of this improved evaluation is to compute a gap for each
instance, and then the average of
the gaps. The advantages of this improved evaluation are the guarantee
of the non compensation between
objectives, same importance for each instance, and the gaps between
candidates better presented.
Though this improved evaluation function is
better and fairer, IT IS NOT THE FINAL ONE, we can still improve it a little
bit.
The final evulation function will be detailed
with its related rankings soon.