# Lab 6: Watch a learning agent improve — step-by-step guide

## Context

This laboratory accompanies Lecture 1 (slides 48–49) of 5DV248 Trustworthy Human-AI Interaction and follows AIMA Section 2.4.6, and it closes the series of labs. Until now, every agent program was designed by hand: you wrote the rules (Labs 1–3), the models (Lab 4) and the search (Lab 5). This lab removes the designer from that role. The agent starts with an empty table of values — no rules at all — and learns a policy purely from experience, using Q-learning.

The world is a five-by-five grid with four dirty squares, the same scenario as the Python notebook. Rewards are given by the environment: +10 for sucking up dirt, −1 for a wasted Suck, −0.2 per movement step. The agent's only knowledge is what it has tried and what came of it.

## Objective

The objective is to map the learning-agent diagram from slide 48 onto running code, component by component:

- the **performance element** is the greedy Q-policy: the action with the highest learned value in the current state;
- the **critic** is the reward function, which judges each action against the external performance standard;
- the **learning element** is the Q-value update, which modifies the performance element from experience;
- the **problem generator** is epsilon-exploration: occasional random actions that create informative experiences.

By the end of the lab you should be able to say precisely where each component lives, and why removing any one of them breaks the learning.

## The interface

- The grid shows the world: the blue dot is the agent (it always starts at the top-left corner), brown cells are dirt. Each new episode places four new dirty squares at random.
- The small letters in the cells show the current greedy action for that cell under the current dirt configuration: the arrows mean movement, S means Suck, and a dot means the agent has no learned value yet. This is the performance element, visible.
- Train 100 episodes runs one hundred learning episodes at full speed. You can press it repeatedly to accumulate more experience.
- Run 1 episode (animated) executes a single episode slowly, so you can watch the policy act — including exploratory detours, because exploration stays active during the run.
- Reset brain empties the Q-table and the learning history: the agent becomes a newborn again.
- The two sliders control the learning mechanism: epsilon (how often the problem generator suggests a random action) and alpha (the learning rate: how strongly each new experience rewrites old beliefs). The discount factor gamma is fixed at 0.9.
- The learning curve plots the number of steps per episode; lower is better. The statistics line compares the average of the first hundred episodes with the average of the last hundred, and reports how many distinct states the agent has learned about.

## Your goal as a student

You are the learning engineer: you do not program behaviour, you configure how behaviour is acquired. Proceed as follows:

1. Press Reset brain, then Run 1 episode without any training. Describe the behaviour of the empty-brain agent: why does it act essentially at random, and why does it sometimes win anyway?
2. Press Train 100 episodes three times. Watch the learning curve descend and the two averages separate. Then press Run 1 episode and compare the behaviour with step 1. Which of the four components changed the agent, and where is the change stored?
3. Find the components in the numbers: increase epsilon to 0.6, reset the brain and retrain. What happens to the learning curve, and why does heavy exploration make even the final policy look erratic in the animated run?
4. Set epsilon to 0 (pure exploitation), reset, and train. Explain the failure mode you observe: without a problem generator, which experiences can the agent never have?
5. Now vary alpha: train with 0.05 and with 0.6 on fresh brains. Describe the difference in terms of how strongly a single experience rewrites the agent's beliefs, and relate it to the notion of a performance standard being respected or overreacted to.
6. Question for the report, from the notebook: is this agent autonomous, social, reactive and proactive (Wooldridge and Jennings)? It is clearly reactive (it responds to percepts) and it achieves goals without step-by-step instruction (proactive in a weak sense). Argue about autonomy, and identify the missing property — there is no second agent and no human in the loop in this lab.
7. Bridge to the lecture and the exam: the slides ask the same question about LLM-based agents ("Are those LLM-based agents autonomous, social, reactive and proactive?"). Using the vocabulary of this lab — performance element, critic, learning element, problem generator — write three sentences on which components an LLM-based assistant has and which it lacks.
8. Bridge to the GAMA practical: the BDI firefighter has explicit, readable desires (`patrol_desire`) and plans (`stopFire`), but its plans do not change with experience; our Q-learner changes behaviour but stores nothing readable as a "desire". Discuss which of the two you would rather deploy near humans, and which Wooldridge property your trust argument hinges on.

## What to submit or discuss

A short reflection (half a page) containing: your observations from steps 3 and 4 (the roles of epsilon and the problem generator), your alpha comparison from step 5, and your answers to the two Wooldridge questions (steps 6 and 8). The learning curve is the critic's judgement of the agent over time — attach a screenshot of the curve before and after training as your evidence.