neural_lam.utils.tensor#

Numerically stable inverses of common activation functions.

Functions#

`inverse_sigmoid`(→ torch.Tensor)	Inverse of `torch.sigmoid` with clamping for numerical stability.
`inverse_softplus`(→ torch.Tensor)	Inverse of `torch.nn.functional.softplus()`.

Module Contents#

neural_lam.utils.tensor.inverse_sigmoid(x: torch.Tensor) → torch.Tensor[source]#

Inverse of torch.sigmoid with clamping for numerical stability.

Sigmoid output takes values in [0, 1]; we clamp the input slightly within that open interval before applying log(x / (1 - x)).

Note that torch.clamp will make gradients 0 near the bounds, but this is not a problem as values of x that are this close to 0 or 1 have gradients of 0 anyhow.

Parameters:: x (torch.Tensor) – Input tensor assumed to contain logits after a sigmoid.
Returns:: Tensor containing log(x / (1 - x)) after clamping away from the saturation limits.
Return type:: torch.Tensor

Notes

torch.clamp zeroes gradients for values at the bounds, but values this close to 0 or 1 already have negligible gradients.

neural_lam.utils.tensor.inverse_softplus(x: torch.Tensor, beta: float = 1.0, threshold: float = 20.0) → torch.Tensor[source]#

Inverse of torch.nn.functional.softplus().

For most inputs this function is exact up to numerical precision. The input is clamped to ensure numerical stability: values above threshold / beta are treated as linear (which is exact in that regime), and values near zero are clamped to avoid log of non-positive numbers. Only near the lower clamping bound does the result deviate from the true inverse.

Parameters:

x (torch.Tensor) – Input tensor whose softplus inverse should be computed.
beta (float, optional) – Softplus beta parameter that controls the sharpness. Default 1.
threshold (float, optional) – Threshold above which the function is treated as linear for numerical stability. Default 20.

Returns:

Tensor containing the inverse-softplus values.

Return type:

torch.Tensor

Notes

torch.clamp will zero the gradients near the bounds, but values this close to zero or threshold / beta already have negligible gradients.