Uses the current Q-table to select an action. During training it usually chooses the best-known action, but sometimes explores.
1 · The environment: vacuum-cleaner world
Untrained agent
Position: A1Dirty cells: 5
Episode: 0
The agent starts each training episode in the same room. It receives rewards for cleaning and a penalty for each action. Obstacles are fixed.
—Latest episode reward
—Success rate (last 25)
—Actions in latest episode
Learning progress
No training episodes yet.
Recent experience
- The agent is ready. Try one action or train an episode.
2 · Inside the learning agent (Russell & Norvig architecture)
AGENT
Critic
Compares observed outcomes with the performance standard
Compares observed outcomes with the performance standard
Sensors
Observe location and dirt in the current cell
Observe location and dirt in the current cell
↓ feedback / percept
Learning element
Updates Q-values from reward and experience
Updates Q-values from reward and experience
Performance element
Chooses an action using the current Q-values
Chooses an action using the current Q-values
↑ knowledge / policy update changes →
Problem generator
Suggests exploratory actions to discover better behaviour
Suggests exploratory actions to discover better behaviour
learning goals / exploratory experience
Actuators
Move or vacuum · action changes the environment
Move or vacuum · action changes the environment
↕ action and new percept
Environment
The room changes when the agent moves or cleans
The room changes when the agent moves or cleans
Ready
Train the agent or perform one action. The highlighted components show which parts of the learning-agent architecture are involved.
Performance standard and critic
Performance standard: clean all five dirty locations using as few actions as possible.
| Event | Critic's feedback |
|---|---|
| Clean a dirty cell | +10 reward |
| Each action | −1 cost |
| Clean the whole room | +20 completion bonus |
| Episode exceeds 80 actions | −10 timeout penalty |
What is the agent learning?
The performance element uses a Q-table: an estimate of the long-term value of each action in each state. The learning element updates it using the temporal-difference rule.
Q-learning update
Q(s,a) ← Q(s,a) + α [r + γ max Q(s′,a′) − Q(s,a)]
Recent learning signals
0Q-value updates
0Exploratory actions
0.00Mean learned Q-value
Observes the result and evaluates it against the performance standard, producing a numerical reward.
Uses the reward and next state to update Q-values, improving future action selection.
Encourages actions that gather useful experience. Here, ε controls the chance of trying a random action.
Teaching note: This is a small tabular Q-learning example that instantiates the four components in the Russell & Norvig learning-agent architecture. The performance standard is encoded as a reward function; the critic supplies feedback; the learning element updates the Q-table; and the problem generator is represented by ε-greedy exploration. The task is intentionally small so students can observe learning without needing an external library.