Policy

Definition

Policy

A policy $\pi$ is a mapping from states to probabilities of selecting each possible action. If an agent follows policy $\pi$ at time $t$, then $\pi (a|s)$ is the probability that $A_t=a$ given $S_t=s$.

$\pi (a|s)=P(A_t=a\mid S_t=s)$

• Deterministic policy: $\pi (s)=a$ — maps each state to exactly one action
• Stochastic policy: $\pi (a|s)\in [0,1]$ — a probability distribution over actions for each state, where ${\sum }_a\pi (a|s)=1$

Types of Policies

Greedy Policy

$\pi (s)=\arg {\max }_{a}q(s,a)$ Always picks the action with the highest estimated value. Pure exploitation, no exploration.

ε-Greedy Policy (Epsilon-Greedy Policy)

$\pi (a|s)=\{\begin{array}{ll}1-\epsilon +\frac{\epsilon }{|\mathcal{A}(s)|}& \text{if }a=\arg {\max }_{a^{\prime }}q(s,a^{\prime })\\ \frac{\epsilon }{|\mathcal{A}(s)|}& \text{otherwise}\end{array}$ Mostly greedy, but with probability $\epsilon$ picks a random action. Balances Exploration vs Exploitation.

Softmax / Boltzmann Policy

$\pi (a|s)=\frac{e^{q(s,a)/\tau }}{{\sum }_{a^{\prime }}{e}^{q(s,a^{\prime })/\tau }}$ Temperature $\tau$ controls exploration: high $\tau$ → uniform, low $\tau$ → greedy.

Optimal Policy

Optimal Policy

A policy ${\pi }_{\ast }$ is optimal if $v_{{\pi }_{\ast }}(s)\ge v_{\pi }(s)$ for all $s\in \mathcal{S}$ and all policies $\pi$. There always exists at least one optimal policy for any finite MDP.

All optimal policies share the same optimal value functions $v_{\ast }$ and $q_{\ast }$. Given $q_{\ast }$: ${\pi }_{\ast }(s)=\arg {\max }_a{q}_{\ast }(s,a)$

Policy in Different RL Methods

Method	How policy is used
Policy Iteration	Explicit policy, alternates evaluation and improvement
Value Iteration	Implicit policy (greedy w.r.t. current $V$)
Monte Carlo Methods	Generates episodes, improved via ε-greedy
SARSA	On-policy: follows and improves ε-greedy
Q-Learning	Off-policy: follows ε-greedy, learns about greedy
REINFORCE	Directly parameterized: ${\pi }_{\theta }(a\Vert s)$

On-Policy vs Off-Policy

Key Distinction

• Behavior policy $b$: the policy used to generate data (select actions)
• Target policy $\pi$: the policy being evaluated or improved
• On-policy: $b=\pi$ (same policy)
• Off-policy: $b\ne \pi$ (different policies, requires Importance Sampling correction)

See On-Policy vs Off-Policy for details.

Connections

• Acts within: Markov Decision Process
• Evaluated by: Value Function, Bellman Equation
• Improved by: Policy Iteration, Generalized Policy Iteration
• Parameterized: REINFORCE, Policy Gradient Theorem

Appears In

• RL-L01 - Intro, MDPs & Bandits, RL-L02 - Dynamic Programming, RL-L03 - Monte Carlo Methods, RL-L04 - Temporal Difference Learning