Target Network

Target Network

A target network is a separate, periodically-updated copy of the Q-network used to compute TD targets during training. Its weights ${\mathbf{\theta }}^{-}$ are held fixed for $C$ steps, then copied from the online network $\mathbf{\theta }$.

Why needed:

• Without it, the TD target $r+\gamma {\max }_{a^{\prime }}Q(s^{\prime },a^{\prime };\mathbf{\theta })$ changes with every gradient step → moving target problem
• Freezing the target provides a stable regression objective
• Alternative: soft update ${\mathbf{\theta }}^{-}\leftarrow \tau \mathbf{\theta }+(1-\tau ){\mathbf{\theta }}^{-}$ (Polyak averaging, used in DDPG/SAC)

Appears In

• RL-L08 - Deep RL Value-Based, Deep Q-Network (DQN)