Implementing a port to scipy optimizers

This notebook shows how to wrap a scipy optimizer as an f3dasm Block. Scipy's minimize routines (CG, L-BFGS-B, Nelder-Mead, ...) run their own inner iteration loop, which doesn't fit the ask/tell update-step shape that LoopBlock expects. Instead, we model a scipy optimizer as a one-shot block: a single call() invokes scipy.optimize.minimize once, scipy runs its full inner loop, and the block appends the scipy callback history to data.

Use this as a template when porting any library with its own internal loop.

See the Optimization Loops tutorial for how one-shot blocks compare to ask/tell update-step blocks.

from typing import Callable, Optional

import numpy as np
import scipy.optimize

from f3dasm import Block, ExperimentData
from f3dasm.datageneration import DataGenerator

Design: a one-shot Block

Because scipy runs the loop internally, the block's call signature is simple:

• arm(data) captures x0 from the last row of data.
• call(data) runs scipy.optimize.minimize once, collects a callback history, and returns data with the history appended via ExperimentData.__add__.

The number of scipy iterations is controlled via maxiter in the options dict at construction time — do not wrap a one-shot block in a LoopBlock, or scipy's full inner loop would run n times.

The callback

scipy.optimize.minimize accepts a callback invoked at each iteration. We use it to record (x, f(x)) pairs so we can reconstruct the scipy trajectory inside the returned ExperimentData:

def callback(intermediate_result: scipy.optimize.OptimizeResult) -> None:
    history_x.append({self.input_name: intermediate_result.x})
    history_y.append({self.output_name: intermediate_result.fun})

Putting it together

class ScipyOptimizer(Block):
    """One-shot scipy optimizer block.

    Parameters
    ----------
    method : str
        Name of the scipy method (e.g. ``'CG'``, ``'L-BFGS-B'``, ``'Nelder-Mead'``).
    data_generator : DataGenerator
        Provides ``f(x)`` as its ``.f`` attribute.
    output_name : str
        Column to minimize.
    input_name : str
        Array-valued input column scipy operates on.
    bounds : scipy.optimize.Bounds, optional
        Variable bounds for methods that support them (e.g. L-BFGS-B).
    grad_f : callable, optional
        Gradient of the objective. If omitted, scipy estimates gradients numerically.
    **hyperparameters
        Forwarded to ``scipy.optimize.minimize`` as ``options={...}`` (e.g. ``maxiter``).
    """

    def __init__(
        self,
        method: str,
        data_generator: DataGenerator,
        output_name: str,
        input_name: str,
        bounds: Optional[scipy.optimize.Bounds] = None,
        grad_f: Optional[Callable] = None,
        **hyperparameters,
    ) -> None:
        self.method = method
        self.data_generator = data_generator
        self.output_name = output_name
        self.input_name = input_name
        self.bounds = bounds
        self.grad_f = grad_f
        self.hyperparameters = hyperparameters

    def arm(self, data: ExperimentData) -> None:
        experiment_sample = data.get_experiment_sample(data.index[-1])
        self._x0 = experiment_sample.input_data[self.input_name]

    def call(self, data: ExperimentData, **kwargs) -> ExperimentData:
        history_x, history_y = [], []

        def callback(
            intermediate_result: scipy.optimize.OptimizeResult,
        ) -> None:
            history_x.append({self.input_name: intermediate_result.x})
            history_y.append({self.output_name: intermediate_result.fun})

        scipy.optimize.minimize(
            fun=self.data_generator.f,
            x0=self._x0,
            method=self.method,
            jac=self.grad_f,
            bounds=self.bounds,
            options={**self.hyperparameters},
            callback=callback,
        )

        history = ExperimentData(
            domain=data.domain,
            input_data=history_x,
            output_data=history_y,
            project_dir=data._project_dir,
        )
        return data + history

Example run

from f3dasm import create_sampler, datagenerator
from f3dasm.design import Domain


@datagenerator(output_names="y")
def f(x: np.ndarray) -> float:
    # y = 0.5 * sum(x^4 - 16 x^2 + 5 x)  (Styblinski-Tang-style)
    return float(0.5 * np.sum(x**4 - 16 * x**2 + 5 * x))


domain = Domain()
domain.add_array(name="x", shape=(10,), low=-10, high=10)
domain.add_output("y")

data = ExperimentData(domain=domain)
data = create_sampler("random").call(data, n_samples=1)

f.arm(data)
data = f.call(data)

optimizer = ScipyOptimizer(
    method="CG",
    data_generator=f,
    output_name="y",
    input_name="x",
    maxiter=50,
)
optimizer.arm(data)
data = optimizer.call(data)

data

Comparison with the built-in scipy factories

The custom ScipyOptimizer above is functionally equivalent to the built-in factories in f3dasm.optimization:

from f3dasm.optimization import cg, lbfgsb, nelder_mead

optimizer = cg(
    data_generator=f, output_name="y", input_name="x", maxiter=50
)

Prefer the built-ins unless you need to customise beyond passing options.

---

Next: Hydra Configuration