Smooth maximum

In mathematics, a smooth maximum of an indexed family x₁, ..., x_n of numbers is a differentiable approximation to the maximum function

\{x_{1},\ldots ,x_{n}\}\mapsto \max\{x_{1},\ldots ,x_{n}\},\,

and the concept of smooth minimum is similarly defined.

For large positive values of the parameter $\alpha >0$ , the following formulation is one smooth, differentiable approximation of the maximum function. For negative values of the parameter that are large in absolute value, it approximates the minimum.

{\mathcal {S}}_{\alpha }(\{x_{i}\}_{{i=1}}^{n})={\frac {\sum _{{i=1}}^{n}x_{i}e^{{\alpha x_{i}}}}{\sum _{{i=1}}^{n}e^{{\alpha x_{i}}}}}

${\mathcal {S}}_{\alpha }$ has the following properties:

${\mathcal {S}}_{\alpha }\to \max$ as $\alpha \to \infty$
${\mathcal {S}}_{0}$ is the average of its inputs
${\mathcal {S}}_{\alpha }\to \min$ as $\alpha \to -\infty$

The gradient of ${\mathcal {S}}_{{\alpha }}$ is closely related to softmax and is given by

\nabla _{{x_{i}}}{\mathcal {S}}_{\alpha }(\{x_{i}\}_{{i=1}}^{n})={\frac {e^{{\alpha x_{i}}}}{\sum _{{j=1}}^{n}e^{{\alpha x_{j}}}}}[1+\alpha (x_{i}-{\mathcal {S}}_{\alpha }(\{x_{i}\}_{{i=1}}^{n}))].

This makes the softmax function useful for optimization techniques that use gradient descent.

Another formulation is:

g(x_{1},\ldots ,x_{n})=\log(\exp(x_{1})+\ldots +\exp(x_{n}))

References

This article is issued from Wikipedia - version of the 6/30/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.