cma.optimization_tools.EvaluationsIterator

Examples

>>> import cma.optimization_tools as ot
>>> data = [[1, 44.2, 33.3, 22.2],
...         [2, 32,   22.1, 33],
...         [3, 11,   22,   66]]

The shortest usage example is:

>>> list_of_evals_fval_succs = list(ot.EvaluationsIterator(data))

which creates a list of about n x m length where n = data.shape[0] and m = data.shape[1] - 1 (usually n>>m). Each list element is a three-tuple with (m evaluations, 1 f-value, m bools).

A typical use case computes the ERT-value in the first "column" (as by default argument xvalues=(ei_ert,) of ei_xy_data) and has (by default) f-values in the second (last) "column":

>>> xy = list(ot.EvaluationsIterator(data, return_filter=ot.ei_xy_data()))
>>> print(xy)  # doctest: +ELLIPSIS
[(1.0, 44.2), (1.33333...

Now we can plot the ERT as the x-value (with evaluations as unit):

plt.plot([d[0] for d in xy],
            [d[1] for d in xy], label='ERT')
plt.xlabel('evaluations'), plt.ylabel('f-value');

An example collecting ERT, SP1, the median, the f-value, and the number of successes without creating the full new data array otherwise:

>>> list_of_xs_y_s = [(ot.ei_ert(*d),     # ERT
...                    ot.ei_sp1(*d),     # SP1
...                    ot.ei_median(*d),  # median
...                    d[1],              # f-value
...                    sum(d[2])          # number of runs that reached d[1]
...                   ) for d in ot.EvaluationsIterator(data, return_filter=None)]

Passing return_filter=None avoids an unnecessary internal copy of d[0] and d[2].

We can again plot the ERT as an x-value:

plt.plot([d[0] for d in list_of_xs_y_s],
            [d[3] for d in list_of_xs_y_s], label='ERT')

More conveniently, we can use arrays for plotting, here with annotation of the graph end and of successes too:

from matplotlib import pyplot as plt

xy = np.asarray(list_of_xs_y_s)
p = plt.plot(xy[:, 0], xy[:, -2], '-', label='ERT')
plt.plot(xy[-1, 0], xy[-1, -2], 'o', color=p[-1].get_color())
p = plt.plot(xy[:, 2], xy[:, -2], '-', label='median')
i = np.nonzero(np.isfinite(xy[:, 2]))[0][-1]
plt.plot(xy[i, 2], xy[i, -2], 'o', color=p[-1].get_color())
idx = np.where(np.diff(xy[:, -1], prepend=xy[0, -1]))[0]  # indices where successes changed
for j, i in enumerate(idx):          # annotate:
    if (j == 0 or xy[i, -1] <= 5 or  # first failure and 5..1 successes
        xy[i, -1] <= xy[0, -1] / 2 < xy[idx[j-1], -1]):  # and 50% failures
        plt.text(xy[i, 0], xy[i, -2], int(xy[i, -1]))
# plt.gca().set_xscale('log')
plt.xlabel('evaluations'), plt.ylabel('f-value'), plt.legend();

A similar example as above but with an explicit for-loop instead of a list comprehension and hence longer but not more useful here:

>>> import cma.optimization_tools as ot
>>> fvals, erts, med = [], [], []
>>> xy_succ = {s: None for s in [2, 5, 9]}  # number for successes to annotate
>>> for evals, fval, succ in ot.EvaluationsIterator(data):
...     erts.append(ot.ei_ert(evals, fval, succ))  # compress all runs to one column
...     med.append(ot.ei_median(evals, fval, succ))  # compress all runs to one column
...     fvals.append(fval)  # y-value column
...     if sum(succ) < len(succ):
...         for k in xy_succ:
...             if xy_succ[k] is None and sum(succ) <= k:
...                 xy_succ[k] = erts[-1], fvals[-1]

Plotting the above:

plt.plot(data[:, 0], data[:, 1:])  # plot all data lines
plt.plot(erts, fvals)  # plot an ert line
for k, xy in xy_succ.items():
    plt.text(xy[0], xy[1], k)
plt.xlabel('evaluations'), plt.ylabel('f-value');

Testing:

>>> ot.EvaluationsIterator._no_asserts = False
>>> x = [ot.ei_ert(*d) for d in ot.EvaluationsIterator(data)]
>>> assert x == sorted(x), x
>>> for d in zip(x, [1.0, 1.333333333333333, 1.666666666666667, 2.0, 4.0, 4.5, 9.0]):
...     assert -1e-7 < d[0] - d[1] < 1e-7, (x, d)
>>> y = [d[1] for d in ot.EvaluationsIterator(data, return_filter=None)]
>>> assert y == [44.2, 33.3, 32., 22.2, 22.1, 22., 11.], y
>>> xy = list(ot.EvaluationsIterator(data, return_filter=ot.ei_xy_data()))
>>> for i in range(len(xy)):
...     assert xy[i] == (x[i], y[i]), (xy[i], x[i], y[i])
>>> import numpy as np
>>> n, m = 22, 11
>>> a = np.log(np.abs(np.random.randn(n, m+1) / np.random.randn(n, m+1)))
>>> a.sort(0)
>>> a[:,:] = a[-1::-1, :]
>>> a[:,0] = 2 * np.arange(a.shape[0]) + 1
>>> for agg in [ot.ei_ert, ot.ei_sp1, ot.ei_median]:
...     x = np.asarray([agg(*d) for d in ot.EvaluationsIterator(a)])
...     if agg == ot.ei_median:
...         x = x[np.isfinite(x)]
...         assert np.all(x == np.array(x, dtype=int)), x  # works because all evals are odd
...     assert np.all(np.diff(x) >= 0), x