Parallelism is central in the development of NOMAD 4. NOMAD 4 supports OpenMP to use many threads at the same time. Note that it is also possible to compile without OpenMP. It is also possible to limit the number of threads used (parameter NB_THREADS_OPENMP). MPI is not currently supported in NOMAD 4 (it was used in NOMAD 3).

The EvaluatorControl manages the evaluations. It has a queue of EvalQueuePoints which are points that need to be evaluated. The points are generated by SearchMethods, PollMethods, or other Steps derived from IterationUtils. The EvaluatorControl also handles the Evaluator(s).

The EvalQueuePoints know which Evaluator is needed to evaluated them. The evaluation queue contains all points that need to be evaluated, whether the evaluator is a model, a surrogate, or a blackbox (or other).

The queue is implemented by a vector.

Points may be ordered before evaluation. This is useful for opportunistic evaluation, more so when there is little CPU available (few threads for OpenMP).

There are multiple sorting criteria: DIR_LAST_SUCCESS, LEXICOGRAPHICAL, RANDOM, SURROGATE; others like QUAD_MODEL should be added. See parameter EVAL_QUEUE_SORT. The default is DIR_LAST_SUCCESS.

The way the EvaluatorControl is implemented, the points at the end of the vector are evaluated first - they have the greatest priority.

Points are taken by threads for evaluation. As soon as a thread is done evaluating a point, it is free, and it pops the next point in the queue.

When a point is evaluated, updates are done:

• If opportunism is used, a FULL_SUCCESS will stop evaluations. Evaluations that are already in progress will continue, but no new evaluation will be started.
• Eval of the point, i.e., evaluation information, is updated in the cache.
• The stats to be displayed are updated.
• Some members of EvaluatorControl, like the counter for the number of evaluations, are updated.

OpenMP computes the number of available threads and uses them all, unless NB_THREADS_OPENMP is set, in which case this number of threads is used. There is one main thread, which is thread number 0. The other threads are secondary threads. The main threads is the only one that generates points and adds them to the evaluation queue; it is also used for evaluations. The secondary threads are used solely to run the Evaluator for evaluating points.

When evaluations are done because all points in the queue are evaluated, or because of opportunism, only the main thread exits from the EvaluatorControl's run method. The secondary threads keep looping inside the run methods while waiting for new points to evaluate.

In the case of PSD-Mads, there is more than one main thread. Each subproblem has its own main thread that generate points and manages algorithms. There can still be additional secondary threads for evaluation only. The main thread 0 is used for the pollster and for the management of the PSD-Mads algorithm. Note that any thread may run the evaluation for a point generated from any main thread: it is always the next thread that is available that will evaluated the next point in the queue.

The EvaluatorControl always pops blocks of points from the evaluation queue, but usually these blocks have only one point. If BB_MAX_BLOCK_SIZE is more than 1, blocks of this size will be popped. The points inside the block may come from different main threads (if applicable), but they all need to be evaluated with the same Evaluator. In this use case, the Evaluator needs to know how to evaluate a block of points. The opportunity condition, if applicable, is verified after all points in the block are evaluated.