===== EvaluatorControl and Evaluation Queue =====

Parallelism is central in the development of NOMAD 4. NOMAD 4 supports OpenMP to use many threads at the same time. Note that it is also possible to compile without OpenMP. It is also possible to limit the number of threads used (parameter ''NB_THREADS_OPENMP''). MPI is not currently supported in NOMAD 4 (it was used in NOMAD 3).

==== Evaluation Queue ====
The ''EvaluatorControl'' manages the evaluations. It has a queue of ''EvalQueuePoints'' which are points that need to be evaluated. The points are generated by ''SearchMethods'', ''PollMethods'', or other ''Steps'' derived from ''IterationUtils''. The ''EvaluatorControl'' also handles the Evaluator(s).

The ''EvalQueuePoints'' know which ''Evaluator'' is needed to evaluated them. The evaluation queue contains all points that need to be evaluated, whether the evaluator is a model, a surrogate, or a blackbox (or other).

The queue is implemented by a vector.

{{ :groupe-dfo-bbo:projets:nomad:architecture:queue1.png?400 |}}


==== Sorting the Queue ====
Points may be ordered before evaluation. This is useful for opportunistic evaluation, more so when there is little CPU available (few threads for OpenMP).

There are multiple sorting criteria: ''DIR_LAST_SUCCESS'', ''LEXICOGRAPHICAL'', ''RANDOM'', ''SURROGATE''; others like ''QUAD_MODEL'' should be added. See parameter ''EVAL_QUEUE_SORT''.
The default is ''DIR_LAST_SUCCESS''.

{{ :groupe-dfo-bbo:projets:nomad:architecture:queue2.png?400 |}}

==== Evaluation of Points ====
The way the ''EvaluatorControl'' is implemented, the points at the end of the vector are evaluated first - they have the greatest priority.

Points are taken by threads for evaluation. As soon as a thread is done evaluating a point, it is free, and it pops the next point in the queue.

{{ :groupe-dfo-bbo:projets:nomad:architecture:queue3.png?400 |}}

When a point is evaluated, updates are done:
  * If opportunism is used, a ''FULL_SUCCESS'' will stop evaluations. Evaluations that are already in progress will continue, but no new evaluation will be started.
  * ''Eval'' of the point, i.e., evaluation information, is updated in the cache.
  * The stats to be displayed are updated.
  * Some members of ''EvaluatorControl'', like the counter for the number of evaluations, are updated.

==== Main thread(s) and secondary threads ====
OpenMP computes the number of available threads and uses them all, unless ''NB_THREADS_OPENMP'' is set, in which case this number of threads is used. There is one main thread, which is thread number 0. The other threads are secondary threads. The main threads is the only one that generates points and adds them to the evaluation queue; it is also used for evaluations. The secondary threads are used solely to run the ''Evaluator'' for evaluating points.

When evaluations are done because all points in the queue are evaluated, or because of opportunism, only the main thread exits from the EvaluatorControl's ''run'' method. The secondary threads keep looping inside the ''run'' methods while waiting for new points to evaluate.

In the case of ''PSD-Mads'', there is more than one main thread. Each subproblem has its own main thread that generate points and manages algorithms. There can still be additional secondary threads for evaluation only. The main thread 0 is used for the pollster and for the management of the PSD-Mads algorithm. Note that any thread may run the evaluation for a point generated from any main thread: it is always the next thread that is available that will evaluated the next point in the queue.


==== Blocks of points ====
The ''EvaluatorControl'' always pops blocks of points from the evaluation queue, but usually these blocks have only one point. If ''BB_MAX_BLOCK_SIZE'' is more than 1, blocks of this size will be popped. The points inside the block may come from different main threads (if applicable), but they all need to be evaluated with the same ''Evaluator''. In this use case, the Evaluator needs to know how to evaluate a block of points. The opportunity condition, if applicable, is verified after all points in the block are evaluated.