NS-Gym Update Functions Module

class ns_gym.update_functions.BudgetBoundedIncrement(scheduler, k, B)

Bases: UpdateDistributionFn

Increment the parameters so that the total amount of change is bounded by some budget.

Overview:

This function contrains the total amount of change in the parameter by some max budget. This formulation is outlined in Cheung et al. 2020.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• k (float) – The amount which the parameter is updated.

• B (Union[int, float]) – The maximum total amount of change allowed in the parameter.

class ns_gym.update_functions.DistributionCyclicUpdate(scheduler, dist_list)

Bases: UpdateDistributionFn

Cycle through a list of distributions, wrapping around when exhausted.

Overview:

Each time the update fires, the distribution is set to the next entry in dist_list. After reaching the end, the index wraps back to 0.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• dist_list (list) – Distributions to cycle through.

class ns_gym.update_functions.DistributionDecrementUpdate(scheduler, k)

Bases: UpdateDistributionFn

Decrement the probability of going in the intended direction by some k.

Overview:

This function is used to decrement the probability distribution by some k. The probability distribution is represented as a list of probabilities. The intended direction is the first element in the probability distribution.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• k (float) – The amount which the parameter is updated.

class ns_gym.update_functions.DistributionIncrementUpdate(scheduler, k)

Bases: UpdateDistributionFn

Increment the the parameter by k.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• k (float) – The amount which the parameter is updated.

Note

This update function is useful for testing the robustness of the agent to changes in the environment. If the parameter is a probability, k would update the probability of going in the intended direction. Otherwise, k would be added to the parameter’s value.

class ns_gym.update_functions.DistributionLinearInterpolation(scheduler, start_dist, end_dist, T)

Bases: UpdateDistributionFn

Linearly interpolate between two distributions over T steps.

Overview:

\[p_t = \text{start} + (\text{end} - \text{start}) \cdot \min\!\left(\frac{t}{T},\; 1\right)\]

The output replaces the current distribution entirely. After t >= T the distribution is clamped at end_dist.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• start_dist (list) – Distribution at t = 0.

• end_dist (list) – Distribution at t = T.

• T (int) – Number of steps over which to interpolate.

class ns_gym.update_functions.DistributionNoUpdate(scheduler)

Bases: UpdateDistributionFn

Does not update the parameter but return correct ns_bench interface.

Overview:

This function does not update the parameter.

class ns_gym.update_functions.DistributionStepWiseUpdate(scheduler, update_values)

Bases: UpdateDistributionFn

Update the parameter to values to a set of predefined values at specific time steps.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• update_values (list) – A list of values that the parameter is updated to at specific time steps.

class ns_gym.update_functions.LCBoundedDistrubutionUpdate(scheduler, L, update_fn=None)

Bases: UpdateDistributionFn

Decrement the parameters so that the change is Lipshitz continuous.

Overview:

This function would call the decrement update function and check if the change is Lipshitz continuous. If not it would recall the decrement update function until the change is Lipshitz continuous.

The Lipshitz continuous constraint between to probability distributions is defined as:

\[W_1(p_t(.|s,a),p_{t'}(.|s,a)) <= L * |t - t'|\]

Where \(W_1\) is the Wasserstein distance between two probability distributions.

Parameters:

• update_fn (Type[base.UpdateDistributionFn]) – The update function that updates the parameter.

• L (float) – The Lipshitz constant.

Note

This update function is an implementation of transition fucntion in Lecarpentier and Rechelson et al. 2019

class ns_gym.update_functions.RandomCategorical(scheduler, seed=None)

Bases: UpdateDistributionFn

Update the distirbution as a random categorical distribution.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• seed (Optional[int]) – Seed for the random number generator. Defaults to None.

Note

This update function would return a new random categorical distribution. The new categorical distribution is sampled from a Dirichlet distribution with all parameters equal to 1.

class ns_gym.update_functions.TargetReversion(scheduler, target, theta)

Bases: UpdateDistributionFn

Mean-revert toward a target distribution (OU analog for distributions).

Overview:

\[p_t = p_{t-1} + \theta \, (\text{target} - p_{t-1})\]

which simplifies to \((1 - \theta) p_{t-1} + \theta \, \text{target}\), a convex combination that stays on the probability simplex for \(\theta \in [0, 1]\).

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• target (list) – The target distribution to revert toward.

• theta (float) – Reversion speed. 0 = no change, 1 = jump to target.

class ns_gym.update_functions.UniformDrift(scheduler, rate)

Bases: UpdateDistributionFn

Drift the distribution toward uniform at a fixed rate.

Overview:

\[p_t = (1 - \alpha) \, p_{t-1} + \alpha \, \mathbf{u}\]

where \(\mathbf{u}\) is the uniform distribution and \(\alpha \in [0, 1]\) is the drift rate. Since this is a convex combination of two valid distributions, the result is always a valid distribution.

Parameters:

• scheduler (Scheduler) – scheduler that determines when the update function fires.

• rate (float) – Mixing rate toward uniform. 0 = no change, 1 = fully uniform.

class ns_gym.update_functions.BoundedRandomWalk(scheduler, mu, sigma, lo, hi, seed=None)

Bases: UpdateFn

Random walk clamped to [lo, hi].

Overview:

\[Y_t = \text{clip}(Y_{t-1} + \epsilon_t,\; lo,\; hi)\]

where \(\epsilon_t \sim \mathcal{N}(\mu, \sigma^2)\).

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• mu (float) – Mean of the noise.

• sigma (float) – Standard deviation of the noise.

• lo (float) – Lower bound.

• hi (float) – Upper bound.

• seed (Optional[int]) – Seed for the random number generator. Defaults to None.

class ns_gym.update_functions.CyclicUpdate(scheduler, value_list)

Bases: UpdateFn

Cycle through a list of values, wrapping around when exhausted.

Overview:

Each time the update fires, the parameter is set to the next value in value_list. After reaching the end, the index wraps back to 0.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• value_list (list) – Values to cycle through.

class ns_gym.update_functions.DecrementUpdate(scheduler, k)

Bases: UpdateFn

Decrement the probability of going in the intended direction by some k.

Overview:

\[Y_t = Y_{t-1} - k\]

where \(Y_t\) is the parameter value at time step \(t\) and \(k\) is the amount to decrement the parameter by.

Parameters:

• scheduler (Type[base.Scheduler]) – scheduler that determines when the update function fires.

• k (float) – The amount which the parameter is updated.

class ns_gym.update_functions.DeterministicTrend(scheduler, slope)

Bases: UpdateFn

Update the parameter with a deterministic trend.

Overview:

\[Y_t = Y_{t-1} + slope * t\]

where \(Y_t\) is the parameter value at time step \(t\) and slope is the slope of the trend.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• slope (float) – The slope of the trend.

class ns_gym.update_functions.ExponentialDecay(scheduler, decay_rate)

Bases: UpdateFn

Exponential decay of the parameter.

Overview:

\[Y_t = Y_0 * exp(-\lambda * t)\]

where \(Y_t\) is the parameter value at time step \(t\), \(Y_0\) is the initial parameter value, and \(\lambda\) is the rate of decay.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• decay_rate (float) – The rate of decay. i.e. \(\lambda\)

class ns_gym.update_functions.GeometricProgression(scheduler, r)

Bases: UpdateFn

Apply a geometric progression to the parameter.

Overview:

\[Y_t = Y_0 * r^t\]

where \(Y_t\) is the parameter value at time step \(t\), \(Y_0\) is the initial parameter value, and \(r\) is the common ratio.

class ns_gym.update_functions.IncrementUpdate(scheduler, k)

Bases: UpdateFn

Increment the the parameter by k.

Overview:

\[Y_t = Y_{t-1} + k\]

where \(Y_t\) is the parameter value at time step \(t\) and \(k\) is the amount to increment the parameter by.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• k (float) – The amount which the parameter is updated.

class ns_gym.update_functions.LinearInterpolation(scheduler, start_val, end_val, T)

Bases: UpdateFn

Linearly interpolate from start_val to end_val over T steps.

Overview:

\[Y_t = start\_val + (end\_val - start\_val) \cdot \min\!\left(\frac{t}{T},\; 1\right)\]

The output replaces the current parameter value entirely. After t >= T the value is clamped at end_val.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• start_val (float) – Value at t = 0.

• end_val (float) – Value at t = T.

• T (int) – Number of steps over which to interpolate.

class ns_gym.update_functions.NoUpdate(scheduler)

Bases: UpdateFn

Do not update the parameter but return correct interface

Overview:

This function does not update the parameter when called. It is useful for testing and debugging.

Parameters:

scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

class ns_gym.update_functions.OrnsteinUhlenbeck(scheduler, theta, mu, sigma=0.0, seed=None)

Bases: UpdateFn

Mean-reverting update via the Ornstein-Uhlenbeck process.

Overview:

\[Y_t = Y_{t-1} + \theta (\mu - Y_{t-1}) + \sigma \epsilon_t\]

where \(Y_t\) is the parameter value at time step \(t\), \(\theta\) controls the speed of mean reversion, \(\mu\) is the long-run mean, and \(\sigma\) scales the noise.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• theta (float) – Speed of mean reversion.

• mu (float) – Long-run equilibrium value.

• sigma (float) – Volatility / noise scale.

• seed (Optional[int]) – Seed for the random number generator. Defaults to None.

class ns_gym.update_functions.OscillatingUpdate(scheduler, delta)

Bases: UpdateFn

Update the parameter with an oscillating function.

Overview:

\[Y_t = Y_{t-1} + \delta * sin(t)\]

where \(Y_t\) is the parameter value at time step \(t\) and \(\delta\) is the amplitude of the sine wave.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• delta (float) – The amplitude of the sine wave.

class ns_gym.update_functions.PolynomialTrend(scheduler, coeffs)

Bases: UpdateFn

Deterministic polynomial trend.

Overview:

\[Y_t = Y_{t-1} + \sum_{i=1}^{n} a_i \, t^{i}\]

where coeffs = [a_1, a_2, ..., a_n]. A single-element list [slope] is equivalent to DeterministicTrend.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• coeffs (list) – Polynomial coefficients [a_1, a_2, ...] for powers t, t^2, ....

class ns_gym.update_functions.RandomWalk(scheduler, mu=0, sigma=1, seed=None)

Bases: UpdateFn

Parameter update function that updates the parameter with white noise.

Overview:

A pure random walk : \(Y_t = Y_{t-1} + \epsilon_t\) where \(Y_t\) is the parameter value at time step \(t\) and \(\epsilon\) is white noise.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• mu (Union[float, int]) – The mean of the white noise. Defaults to 0.

• sigma (Union[float, int]) – The standard deviation of the white noise. Defaults to 1.

• seed (Union[int,None], optional) – Seed for the random number generator. Defaults to None.

class ns_gym.update_functions.RandomWalkWithDrift(scheduler, alpha, mu, sigma, seed=None)

Bases: UpdateFn

A parameter update function that updates the parameter with white noise and a drift term.

Overview:

\[Y_t = \alpha + Y_{t-1} + \epsilon_t\]

where \(Y_t\) is the parameter value at time step \(t\), \(\alpha\) is the drift term, and \(\epsilon\) is white noise.

Parameters:

• alpha (float) – The drift term.

• mu (float) – The mean of the white noise.

• sigma (float) – The standard deviation of the white noise.

• seed (Optional[int]) – Seed for the random number generator. Defaults to None.

class ns_gym.update_functions.RandomWalkWithDriftAndTrend(scheduler, alpha, mu, sigma, slope, seed=None)

Bases: UpdateFn

Parameter update function that updates the parameter with white noise and a deterministic trend.

Overview:

\[Y_t = \alpha + Y_{t-1} + \text{slope} * t + \epsilon_t\]

where \(Y_t\) is the parameter value at time step \(t\), \(\alpha\) is the drift term, \(\text{slope}\) is the slope of the trend, and \(\epsilon\) is white noise.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• alpha (float) – The drift term.

• mu (float) – The mean of the white noise.

• sigma (float) – The standard deviation of the white noise.

• slope (float) – The slope of the trend.

• seed (Optional[int]) – Seed for the random number generator. Defaults to None.

class ns_gym.update_functions.SigmoidTransition(scheduler, a, b, k, t0)

Bases: UpdateFn

Smooth transition between two values via a logistic sigmoid.

Overview:

\[Y_t = a + \frac{b - a}{1 + \exp(-k (t - t_0))}\]

The output replaces the current parameter value entirely (it is independent of \(Y_{t-1}\)), producing a smooth S-curve from \(a\) to \(b\) centred at \(t_0\).

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• a (float) – Starting value (as \(t \to -\infty\)).

• b (float) – Ending value (as \(t \to +\infty\)).

• k (float) – Steepness of the transition. Larger = sharper.

• t0 (float) – Midpoint time of the transition.

class ns_gym.update_functions.StepWiseUpdate(scheduler, param_list)

Bases: UpdateFn

Update the parameter at specific time steps.

Overview:

This function updates the parameter to the next value in the param_list when called. If the param_list is empty, the parameter is not updated.

Parameters:

• scheduler (Type[Scheduler]) – scheduler that determines when the update function fires.

• param_list (list) – A list of parameters to update to.