Skip to content

Iterative work for XCS stabilization #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: master
Choose a base branch
from
Open

Iterative work for XCS stabilization #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
7adf78c
MPX
May 6, 2021
6ecd96b
Clean Up + Tests
Metron45 May 7, 2021
62fc4d3
Experiments with new RP
Metron45 May 10, 2021
d50bff8
XCS Experiments
Metron45 May 18, 2021
7ec2fc6
Multiple varians pop comparison
Metron45 May 19, 2021
8a35998
Description
Metron45 May 19, 2021
11b6934
XNCS Test
Metron45 May 30, 2021
28b1d24
Further tests
Metron45 May 30, 2021
aaa6fb6
NXCS fix Woods
Metron45 May 30, 2021
f67f393
Child fix Tests
Metron45 May 31, 2021
f3dd4b4
Knowledge metric
Metron45 Jun 1, 2021
6c509d5
XNCS with most knowledge
Metron45 Jun 1, 2021
12e11bb
High update %
Metron45 Jun 1, 2021
2fbe434
Experiments with no comments
Metron45 Jun 12, 2021
4cd1780
Simple Maze
Metron45 Jun 20, 2021
d5de13c
XCS Comparisons
Metron45 Jun 20, 2021
cc7cb2b
Experiments Woods
Metron45 Jun 20, 2021
18417ea
Woods1 Experiments
Metron45 Jun 20, 2021
dca0c47
Experiments Woods not division by number of experiments
Metron45 Jun 20, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2,609 changes: 2,609 additions & 0 deletions XCS_Experiments/Experiment_XCS_XNCS_Maze5.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

3,025 changes: 3,025 additions & 0 deletions XCS_Experiments/Experiment_XCS_XNCS_Maze5_pre_populated.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

2,464 changes: 2,464 additions & 0 deletions XCS_Experiments/Experiment_XCS_XNCS_Woods1.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

2,870 changes: 2,870 additions & 0 deletions XCS_Experiments/Experiment_XCS_XNCS_Woods1_pre_populated.ipynb
View file
Open in desktop

Large diffs are not rendered by default.

41 changes: 41 additions & 0 deletions XCS_Experiments/TODOs.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "applied-topic",
"metadata": {},
"source": [
"# TODOs\n",
"\n",
"## XCS\n",
"\n",
"### Bugs\n",
"- There are duplicate classifiers in population. Please ensure that classes implement `__hash__` and `__eq__` methods. It should be unit tested as well\n",
"\n",
"### Enhancements\n",
"- Perform `assert` statements to ensure perception is in valid type"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.10"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
195 changes: 47 additions & 148 deletions XCS_Experiments/Untitled.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "code",
"execution_count": 120,
"execution_count": 13,
"id": "sitting-christmas",
"metadata": {},
"outputs": [
Expand All @@ -12,64 +12,27 @@
"text": [
"\n",
"\u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[31mA\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[33m$\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[33m$\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[31mA\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m \u001b[37m□\u001b[0m \u001b[30m■\u001b[0m\n",
"\u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m \u001b[30m■\u001b[0m\n"
]
}
],
"source": [
"from utils.xcs_utils import *\n",
"\n",
"scenario = MazeScenario(input_size=8)\n",
"scenario.maze.reset()\n",
"scenario.maze.render()"
"env = MazeScenario(input_size=8)\n",
"env.maze.reset()\n",
"env.maze.render()"
]
},
{
"cell_type": "code",
"execution_count": 121,
"id": "coastal-upper",
"metadata": {},
"outputs": [],
"source": [
"from xcs import XCSAlgorithm\n",
"import numpy as np\n",
"\n",
"algorithm = XCSAlgorithm()\n",
"\n",
"algorithm.max_population_size = 1600\n",
"algorithm.learning_rate = .1\n",
"algorithm.error_threshold = .01 # epsilon_0\n",
"algorithm.ga_threshold = 25\n",
"algorithm.crossover_probability = 0.5\n",
"algorithm.mutation_probability = 0.1\n",
"algorithm.initial_prediction = float(np.finfo(np.float32).tiny) # p_I\n",
"algorithm.initial_error = float(np.finfo(np.float32).tiny) # epsilon_I\n",
"algorithm.initial_fitness = float(np.finfo(np.float32).tiny) # F_I\n",
"algorithm.wildcard_probability = 0.0"
]
},
{
"cell_type": "code",
"execution_count": 122,
"id": "employed-laser",
"metadata": {},
"outputs": [],
"source": [
"model = algorithm.new_model(scenario)\n",
"for i in range(100):\n",
" scenario.reset()\n",
" model.run(scenario, learn=True)"
]
},
{
"cell_type": "code",
"execution_count": 123,
"execution_count": 14,
"id": "referenced-affiliation",
"metadata": {},
"outputs": [],
Expand All @@ -88,20 +51,19 @@
},
{
"cell_type": "code",
"execution_count": 124,
"execution_count": 15,
"id": "elementary-punch",
"metadata": {},
"outputs": [],
"source": [
"from lcs.agents.xcs import Configuration, XCS, GeneticAlgorithm\n",
"\n",
"agent = XCS(cfg)\n",
"population, metrics = agent.explore(scenario, 100, False)"
"agent = XCS(cfg)\n"
]
},
{
"cell_type": "code",
"execution_count": 125,
"execution_count": 16,
"id": "productive-recipe",
"metadata": {},
"outputs": [],
Expand All @@ -111,125 +73,62 @@
},
{
"cell_type": "code",
"execution_count": 136,
"execution_count": 17,
"id": "august-anatomy",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\n",
"0.05739385325559033\n"
]
}
],
"outputs": [],
"source": [
"average_fitness = sum(cl.fitness for cl in model) / sum(cl.numerosity for cl in model)\n",
"number = 0\n",
"for cl in model:\n",
" if cl.experience > algorithm.deletion_threshold and \\\n",
" cl.fitness / cl.numerosity < \\\n",
" algorithm.fitness_threshold * average_fitness:\n",
" number += cl.numerosity\n",
" print(str(cl) + f\" {population._deletion_vote(cl, average_fitness)}\")\n",
"print(number)\n",
"print(average_fitness)"
"prev_action_set = None\n",
"prev_reward = agent.reward\n",
"prev_state = None # state is known as situation\n",
"prev_time_stamp = agent.time_stamp # steps\n",
"done = False # eop\n",
"\n",
"raw_state = env.reset()\n",
"state = agent.cfg.environment_adapter.to_genotype(raw_state)"
]
},
{
"cell_type": "code",
"execution_count": 139,
"id": "isolated-chile",
"cell_type": "markdown",
"id": "freelance-austin",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0\n",
"0.05487554256814123\n"
]
}
],
"source": [
"average_fitness = sum(cl.fitness for cl in population) / sum(cl.numerosity for cl in population)\n",
"number = 0\n",
"for cl in population:\n",
" if cl.experience > cfg.deletion_threshold and \\\n",
" cl.fitness / cl.numerosity < \\\n",
" cfg.delta * average_fitness:\n",
" number += cl.numerosity\n",
" print(str(cl) + f\" {population._deletion_vote(cl, average_fitness)}\")\n",
"print(number)\n",
"print(average_fitness)"
"Main Loop"
]
},
{
"cell_type": "markdown",
"id": "potential-plaza",
"metadata": {},
"source": [
"Before Update Set"
]
},
{
"cell_type": "code",
"execution_count": 135,
"id": "welcome-instrument",
"execution_count": 20,
"id": "theoretical-projector",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Cond:09#1##1# - Act:6 - Num:2 [fit: 0.491, exp: 14.00, pred: 155.252]\n",
"Cond:0901#01# - Act:4 - Num:1 [fit: 0.274, exp: 5.00, pred: 142.435]\n",
"Cond:119#01#1 - Act:0 - Num:1 [fit: 0.308, exp: 7.00, pred: 133.162]\n",
"Cond:11900101 - Act:1 - Num:4 [fit: 0.285, exp: 5.00, pred: 129.223]\n",
"Cond:1#900101 - Act:4 - Num:2 [fit: 0.455, exp: 10.00, pred: 144.590]\n",
"Cond:11001111 - Act:0 - Num:1 [fit: 0.256, exp: 4.00, pred: 153.930]\n",
"Cond:1#011##0 - Act:3 - Num:6 [fit: 0.405, exp: 21.00, pred: 191.785]\n",
"Cond:10#1#10# - Act:4 - Num:5 [fit: 0.283, exp: 72.00, pred: 221.712]\n",
"Cond:0#010#1# - Act:3 - Num:4 [fit: 0.531, exp: 14.00, pred: 160.401]\n",
"Cond:11001#11 - Act:1 - Num:2 [fit: 0.494, exp: 43.00, pred: 153.595]\n",
"Cond:0#1#0010 - Act:0 - Num:4 [fit: 0.239, exp: 2.00, pred: 167.607]\n",
"Cond:01110100 - Act:6 - Num:1 [fit: 0.331, exp: 6.00, pred: 150.775]\n",
"Cond:#100000# - Act:2 - Num:1 [fit: 0.441, exp: 9.00, pred: 154.946]\n",
"Cond:09010010 - Act:1 - Num:3 [fit: 0.212, exp: 4.00, pred: 136.754]\n",
"Cond:10101000 - Act:5 - Num:1 [fit: 0.306, exp: 5.00, pred: 153.101]\n",
"Cond:11001111 - Act:4 - Num:5 [fit: 0.384, exp: 14.00, pred: 151.107]\n",
"Cond:000#1000 - Act:1 - Num:4 [fit: 0.228, exp: 4.00, pred: 149.296]\n",
"Cond:10000010 - Act:3 - Num:1 [fit: 0.380, exp: 7.00, pred: 158.779]\n",
"Cond:##000000 - Act:5 - Num:1 [fit: 0.575, exp: 15.00, pred: 174.065]\n",
"Cond:#010#01# - Act:1 - Num:1 [fit: 0.216, exp: 9.00, pred: 166.332]\n",
"Cond:0001#010 - Act:0 - Num:4 [fit: 0.212, exp: 4.00, pred: 132.836]\n",
"Cond:00000101 - Act:1 - Num:6 [fit: 0.227, exp: 5.00, pred: 149.043]\n",
"Cond:1#1#0101 - Act:1 - Num:1 [fit: 0.217, exp: 4.00, pred: 148.928]\n",
"Cond:01111110 - Act:0 - Num:3 [fit: 0.225, exp: 5.00, pred: 138.989]\n",
"Cond:01011101 - Act:0 - Num:3 [fit: 0.290, exp: 5.00, pred: 163.930]\n",
"Cond:00#11101 - Act:2 - Num:5 [fit: 0.240, exp: 4.00, pred: 187.022]\n",
"Cond:0011110# - Act:7 - Num:2 [fit: 0.240, exp: 4.00, pred: 180.617]\n",
"Cond:10101#0# - Act:6 - Num:4 [fit: 0.225, exp: 4.00, pred: 151.885]\n",
"Cond:0010000# - Act:7 - Num:3 [fit: 0.261, exp: 7.00, pred: 143.091]\n",
"Cond:1#00##11 - Act:5 - Num:3 [fit: 0.485, exp: 11.00, pred: 167.041]\n",
"Cond:011#0100 - Act:7 - Num:7 [fit: 0.385, exp: 9.00, pred: 167.215]\n",
"Cond:10100#1# - Act:1 - Num:3 [fit: 0.262, exp: 9.00, pred: 174.929]\n",
"Cond:10001000 - Act:3 - Num:1 [fit: 0.345, exp: 6.00, pred: 166.836]\n",
"Cond:00#010## - Act:4 - Num:8 [fit: 0.745, exp: 42.00, pred: 211.002]\n",
"Cond:0#0#00## - Act:5 - Num:7 [fit: 0.378, exp: 28.00, pred: 218.349]\n",
"Cond:1#00001# - Act:1 - Num:3 [fit: 0.314, exp: 6.00, pred: 196.347]\n",
"Cond:001#00## - Act:6 - Num:3 [fit: 0.312, exp: 7.00, pred: 179.387]\n",
"Cond:100111## - Act:7 - Num:4 [fit: 0.349, exp: 6.00, pred: 175.641]\n",
"Cond:0#01##01 - Act:2 - Num:8 [fit: 0.454, exp: 13.00, pred: 206.904]\n",
"Cond:1#001000 - Act:2 - Num:6 [fit: 0.288, exp: 5.00, pred: 158.223]\n",
"Cond:10011#00 - Act:5 - Num:8 [fit: 0.544, exp: 45.00, pred: 239.238]\n",
"Cond:01111110 - Act:4 - Num:3 [fit: 0.337, exp: 6.00, pred: 198.607]\n"
]
}
],
"outputs": [],
"source": [
"for cl in population:\n",
" if cl.fitness > 0.2:\n",
" print(cl)"
"assert len(agent.population) == len(set(agent.population)), 'duplicates found'\n",
"agent.population.delete_from_population()\n",
"# We are in t+1 here\n",
"match_set = agent.population.generate_match_set(state, agent.time_stamp)\n",
"prediction_array = match_set.prediction_array\n",
"action = agent.select_action(prediction_array, match_set)\n",
"action_set = match_set.generate_action_set(action)\n",
"# apply action to environment\n",
"raw_state, step_reward, done, _ = env.step(action)\n",
"state = agent.cfg.environment_adapter.to_genotype(raw_state)\n",
"if agent.cfg.multistep_enfiroment:\n",
" agent.reward = step_reward + agent.cfg.gamma * agent.reward"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "theoretical-projector",
"id": "patient-circle",
"metadata": {},
"outputs": [],
"source": []
Expand Down
Loading