Comments on the robotSupervisorSchemeTutorial

[tsampazk]

Link to the tutorial.

[ringpy]

Hi, we are trying to reproduce the find and avoid in our world. We are not able to transmit the created message from epuck to supervisor. Do you have a quick document on this program and how to use the json scripts for emitter/receiver.

[Mouses94]

Hi, I am writing a paper with the goal of implementing the stable_baselines algorithms in a Webots environment (using the IRB4600). The first step for me is now this tutorial.
I have gone through the tutorial but now I always get the following error message:

INFO: robotSupervisorController: Starting controller: python.exe -u robotSupervisorController.py
Traceback (most recent call last):
File "D:\Webots Projekte\controllers\robotSupervisorController\robotSupervisorController.py", line 88, in
env = CartpoleRobot()
File "D:\Webots Projekte\controllers\robotSupervisorController\robotSupervisorController.py", line 16, in init
self.robot = self.getSelf() # Grab the robot reference from the supervisor to access various robot methods
AttributeError: 'CartpoleRobot' object has no attribute 'getSelf'
WARNING: 'robotSupervisorController' Controller beendet mit Status: 1

Another question:
I already have the framework for the IRB4600, the motors and armChain, and above that the position sensors, I want to implement with the package ikpy as shown in the example Inverse kinematic.
For this I need the supervisor method: supervisor.getUrdf().encode('utf-8')

However, I cannot implement this at the same time as the RobotSupervisor. Is there another solution for this?

Is there already an example project for this?

Many thanks

[Mouses94]

However, now I have another Issue with the Deepbots Framework.

I have written a Controller for the IRB4600 and wanted to run it with the Deepbots Framework.

But theres now this Error:

Traceback (most recent call last):
  File "D:\Webots Projekte\controllers\ReachIRB\ReachIRB.py", line 293, in <module>
    selectedAction, actionProb = agent.work(observation, type_="selectAction")
  File "D:\Webots Projekte\controllers\ReachIRB\PPO_agent.py", line 72, in work
    c = Categorical(action_prob)
  File "C:\Users\Office\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\distributions\categorical.py", line 64, in __init__
    super(Categorical, self).__init__(batch_shape, validate_args=validate_args)
  File "C:\Users\Office\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\distributions\distribution.py", line 53, in __init__
    raise ValueError("The parameter {} has invalid values".format(param))
ValueError: The parameter probs has invalid values

What can I do about it?
Does it have anything to do with the observation space, or where could the Error come from?

Thank you

[eakirtas]

What type are the observation, could you please add some samples? Pytorch seems to waits Categorical values does this makes sense?

[caozx1110]

seems like only deepbots==0.1.3.dev2 can work

[tsampazk]

seems like only deepbots==0.1.3.dev2 can work

Hello, @caozx1110. The discussion you answered to is quite old now (May 2021).

Right now, indeed the tutorial is intended for version 0.1.3.dev2 as stated here. I am currently updating the tutorial for the latest changes in Webots, fixing some issues with rotations, etc.

[caozx1110]

ok, thanks for your response

[Kai-Mansfield]

Hi, great tutorial! Are there definitions for some of the methods, like getSelf(), getDevice(), getFromDef(), etc. anywhere please? I tried searching the Deepbots and Webots github pages, but couldn't find the definitions or their methods and attributes. I found a reference to them in webots\docs\reference\supervisor.md, but their methods and attributes aren't included. Thank you for your time :)

[tsampazk]

Hey @Kai-Mansfield, thanks!

As far as i know, you can work off of the documentation here which is the same as the one you linked but in a more readable format. In most cases, the Webots python bindings are more or less self-explanatory, and if needed, most information exists on the documentation texts describing the methods.This and a little bit of experimenting usually get me the results i need.

If you still face problems you can visit the official Webots discord (invite link) and search there for similar questions or even ask a question yourself. Usually, one of Webot's developers will answer.

As a side note there is a new python API in the works for Webots, which will probably make it a lot easier to work with python.

[Kai-Mansfield]

Hi @tsampazk, thank you for such a quick response. Excellent, I will have a look at the links, thank you!

[Kaijun101]

Hi, thanks for the awesome tutorial! However, when I run the tutorial I got the error, ModuleNotFoundError: No module named 'controller'. Would kindly require your assistance on this. The error is from supervisor_env.py.

[tsampazk]

Well that would be my guess!

[Kaijun101]

I was using the PPO algo from the tutorial, any ideas how output multiple actions @tsampazk hahaha?

[tsampazk]

If your problem/environment uses a continuous action space, then the PPO agent used in the tutorial is not suitable as it is implemented for discrete action space.

There's a wealth of continuous action space agent implementations online, as well as libraries like stable-baselines that you can use.

[Kaijun101]

Alright, and really thanks a lot @tsampazk for all your support! Appreciate it definitely.

[tsampazk]

No problem @Kaijun101, happy to help!

[ChrisSim01]

Great, crystal clear tutorial! Many thanks indeed for your work.
I plan to use this as the basis for my future work. I have been using Webots for a while and this tutorial is just what I needed as a guide into RL.
If you can offer any advice into the following it would be greatly appreciated:
-saving the ANN state so that it can be loaded again from file, without the need to retrain.
-restart the training from an already trained n/w, perhaps on a slightly different task?
Best regards, Chris

[tsampazk]

Hello @ChrisSim01, thank your for your kind words, very happy to hear that the tutorial proved useful for you! 😄

As for your first question, you can find the agent save/load methods here, which are basically saving the actor/critic neural nets. They can be called whenever you wish, per some number of episodes, every episode or at the end of training to save the agent.

As long as you saved the models once, you can load them after the agent is initialized here. To continue training on a different task with the same agent, you can modify your environment, webots worlds, etc. and just load the agent as discussed.

Hope this helps, let me know if you have other questions!

[ChrisSim01]

Awesome, thank you so much!

…

On Thu, 21 Mar 2024 at 21:25, Kostas Tsampazis ***@***.***> wrote: Hello @ChrisSim01 <https://github.com/ChrisSim01>, thank your for your kind words, very happy to hear that the tutorial proved useful for you! 😄 As for your first question, you can find the agent save/load methods here <https://github.com/aidudezzz/deepbots-tutorials/blob/48f76ecc9791d6b9cfc415623b4ad30d208efc9c/robotSupervisorSchemeTutorial/full_project/controllers/robot_supervisor_controller/PPO_agent.py#L93-L113>, which are basically saving the actor/critic neural nets. They can be called whenever you wish, per some number of episodes, every episode or at the end of training to save the agent. As long as you saved the models once, you can load them after the agent is initialized here <https://github.com/aidudezzz/deepbots-tutorials/blob/48f76ecc9791d6b9cfc415623b4ad30d208efc9c/robotSupervisorSchemeTutorial/full_project/controllers/robot_supervisor_controller/robot_supervisor_controller.py#L95>. To continue training on a different task with the same agent, you can modify your environment, webots worlds, etc. and just load the agent as discussed. Hope this helps, let me know if you have other questions! — Reply to this email directly, view it on GitHub <#12 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH4Z5XTCRSLFN6NZYL6KAK3YZNF3ZAVCNFSM4YCUZX52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQOBXGEYTENY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[ChrisSim01]

Hi again, You could make an online course out of this tutorial via Udemy.com and make money from it! Just an idea, Best regards, Chris

…

On Thu, 21 Mar 2024 at 21:25, Kostas Tsampazis ***@***.***> wrote: Hello @ChrisSim01 <https://github.com/ChrisSim01>, thank your for your kind words, very happy to hear that the tutorial proved useful for you! 😄 As for your first question, you can find the agent save/load methods here <https://github.com/aidudezzz/deepbots-tutorials/blob/48f76ecc9791d6b9cfc415623b4ad30d208efc9c/robotSupervisorSchemeTutorial/full_project/controllers/robot_supervisor_controller/PPO_agent.py#L93-L113>, which are basically saving the actor/critic neural nets. They can be called whenever you wish, per some number of episodes, every episode or at the end of training to save the agent. As long as you saved the models once, you can load them after the agent is initialized here <https://github.com/aidudezzz/deepbots-tutorials/blob/48f76ecc9791d6b9cfc415623b4ad30d208efc9c/robotSupervisorSchemeTutorial/full_project/controllers/robot_supervisor_controller/robot_supervisor_controller.py#L95>. To continue training on a different task with the same agent, you can modify your environment, webots worlds, etc. and just load the agent as discussed. Hope this helps, let me know if you have other questions! — Reply to this email directly, view it on GitHub <#12 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH4Z5XTCRSLFN6NZYL6KAK3YZNF3ZAVCNFSM4YCUZX52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQOBXGEYTENY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[tsampazk]

Good idea Chris, i will definitely consider it! Thank you.

[ChrisSim01]

Hi again Kostas, I do have one further question please: is there any way to 'lock', or encapsulate, a neural network module once it is learned, so that it is used by, but not changed by further neural network training? This would allow tasks to be broken down into subtasks, and chaining of the subtasks together I think. I'm thinking in the style of https://nn.cs.utexas.edu/downloads/papers/lessin.gecco13.pdf Best regards, Chris

…

On Thu, 21 Mar 2024 at 21:25, Kostas Tsampazis ***@***.***> wrote: Hello @ChrisSim01 <https://github.com/ChrisSim01>, thank your for your kind words, very happy to hear that the tutorial proved useful for you! 😄 As for your first question, you can find the agent save/load methods here <https://github.com/aidudezzz/deepbots-tutorials/blob/48f76ecc9791d6b9cfc415623b4ad30d208efc9c/robotSupervisorSchemeTutorial/full_project/controllers/robot_supervisor_controller/PPO_agent.py#L93-L113>, which are basically saving the actor/critic neural nets. They can be called whenever you wish, per some number of episodes, every episode or at the end of training to save the agent. As long as you saved the models once, you can load them after the agent is initialized here <https://github.com/aidudezzz/deepbots-tutorials/blob/48f76ecc9791d6b9cfc415623b4ad30d208efc9c/robotSupervisorSchemeTutorial/full_project/controllers/robot_supervisor_controller/robot_supervisor_controller.py#L95>. To continue training on a different task with the same agent, you can modify your environment, webots worlds, etc. and just load the agent as discussed. Hope this helps, let me know if you have other questions! — Reply to this email directly, view it on GitHub <#12 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH4Z5XTCRSLFN6NZYL6KAK3YZNF3ZAVCNFSM4YCUZX52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQOBXGEYTENY> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[tsampazk]

Very interesting paper!

I think that what you are describing can be achieved by "freezing" specific layers of your networks, something along the lines of this.

[ChrisSim01]

Thanks again Kostas. I'm on a steep learning curve!

…

On Fri, 22 Mar 2024 at 21:50, Kostas Tsampazis ***@***.***> wrote: Very interesting paper! I think that what you are describing can be achieved by "freezing" specific layers of your networks, something along the lines of this <https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088> . — Reply to this email directly, view it on GitHub <#12 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH4Z5XV3FCKQPHBAY2UESVDYZSRTRAVCNFSM4YCUZX52U5DIOJSWCZC7NNSXTOKENFZWG5LTONUW63SDN5WW2ZLOOQ5TQOBYGI4TANI> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[tsampazk]

You sure are, but with persistence everything is possible!