Week 6. Feb. 14: Reinforcement Learning - Orienting

[ShiyangLai]

Post your questions here about:

“Why Reinforcement Learning”, Reinforcement Learning, chapter 1.

“Reinforcement Learning”, Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow, chapter 18.

“Deep Reinforcement Learning from Human Preferences.” 2017. Christiano et al. NeurIPS.

[chychoy]

In the paper "Deep Reinforcement Learning from Human Preferences," it seems that a key trait of these models is allowing humans decisions to be the drivers of building a rewards function rather than mandating humans to build the rewards function themselves. A question that I have relates to an earlier reading, where we considered the ethics in making certain decisions for self-driving cars. How do we ensure that the "human preferences" we see here are also "ethical"? Furthermore, is it possible to retroactively extract and interpret what the machine derived as a "rewards function," especially when we are considering dilemmas that might be more contested morally and ethically?

[yangyuwang]

The paper “Deep Reinforcement Learning from Human Preferences” designed a reinforcement learning way of using human preferences to adjust the reinforcement function. But what I concerned is how does this human-AI interactive learning method capture the complexity of human preferences. From my perspective, human preferences would be refined by human's demographic attributes, and also the time and space. If one person who interact with the reinforcement learning changed their preference across time (whether intentionally or not on purpose), how could the machine learn about the "true" human preferences? Or moving forward, would this method assume that there is a universal or general human preference across human beings? And if we want to check the changes and differences in preferences, and debias them, how can we adjust the method?

[kiddosso]

For the RLHF paper, what are the specific processes through which humans give responses to the agent? In my understanding, men sometimes cannot judge which response is better than the other. The human feedback process may involve some personal preferences that are not that objective. So how can the RLHF workers prevent or exploit it? Does RLHF also pave the way for character AI?

[xpan4869]

In the paper 'Deep Reinforcement Learning from Human Preferences,' the authors provide an alternative method for agents to learn and achieve complicated goals compared to traditional reinforcement learning. While traditional RL approaches typically offer clear interpretability with their well-defined reward functions and decision-making processes, the introduction of human preferences seems to make the decision-making process less transparent, potentially introducing unintended biases and making it harder to detect and correct systematic errors. I am wondering what strategies or approaches you would suggest for balancing the benefits of human preference learning with the need for model interpretability?

[zhian21]

Christiano et al. (2017) demonstrate that human preferences over trajectory segments can be leveraged to train deep reinforcement learning agents without predefined reward functions. By presenting humans with pairwise trajectory comparisons and using these judgments to learn a reward model, the authors achieve comparable performance to traditional RL on complex tasks, such as Atari games and robotic locomotion, with significantly less human feedback. This scalable approach highlights the potential for applying RL to tasks where reward specification is difficult, while introducing new challenges related to preference consistency and reward model generalization. Given the observed discrepancies in human and synthetic feedback efficiency, how might future models adaptively calibrate human input quality to further reduce oversight requirements?

[ulisolovieva]

How do we create reward functions that lead to desired behaviors without unintended consequences? (e.g., predicting job candidate success might incentivize high immediate performance & lead to bias; optimizing for user engagement might promote polarization). What makes rewards change over time? And does it always reflect the agent’s changes in “better = more rewarding” policy?

[xiaotiantangishere]

In the paper's feedback collection process, humans are asked to compare two trajectory segments and select the preferable one. However, if neither is ideal, this might introduce bias into the learned reward function. Meanwhile, in specific applications like autonomous driving—resonating with discussions from previous readings—would it be more effective to incorporate explicit negative feedback to define "red lines" and prevent undesirable behaviors, rather than solely relying on preference-based optimization?

[Sam-SangJoonPark]

In the paper Deep Reinforcement Learning from Human Preferences approach, I find it fascinating that this method achieves strong performance with minimal human feedback. The idea of learning from human preference comparisons instead of predefined reward functions is compelling because it allows for more flexible and intuitive learning, especially in complex tasks where reward design is difficult.

However, since I haven’t worked with reinforcement learning before, one of my main concerns is how performance is evaluated in this setup. In traditional reinforcement learning, performance is often measured using explicit reward signals (e.g., total score in a game, cumulative reward over time). But in this case, since the reward function itself is learned from human feedback, I wonder how we can ensure that the resulting policy is actually optimal or aligned with human intent.

[Daniela-miaut]

“Deep Reinforcement Learning from Human Preferences” introduces a way to train the reinforcement learning algorithm without access to the reward function and with relatively lower cost of human oversight. The basic idea is to learn the reward function using the data from human comparison of possible trajectories of the reinforcement learning agent. This method seems promising in training algorithms to learn human preferences even without a clearly pre-defined goal.
I have a few questions on this topic:
Can reinforcement learning be used to learn more complicated human behaviors in agent-based modeling?
Would it be useful to simulate environments in societal settings in simulation?
Would it be useful to use reinforcement learning to learn human ethical preferences?
Can we (and how can we) get the embedding from the results of reinforcement learning? That is, how can we interpret the results of reinforcement learning?

[DotIN13]

How does the reward model in this approach handle conflicting human preferences, especially in scenarios where different human raters provide contradictory feedback on similar trajectory segments? Additionally, is there a mechanism for prioritizing or reconciling diverging human inputs to ensure the learned reward function remains stable and generalizable across different human perspectives?

[psymichaelzhu]

Can RLHF be used to infer human preferences in a systematic way? Since reinforcement learning from human feedback optimizes agent's behavior based on human-provided preferences, could this process be reversed to extract an implicit formal reward function that captures human values?

[haewonh99]

While the concept is very interesting, I was a bit confused about the 'curiosity-based exploration' introduced in Hands-On Machine Learning with Scikit-Learn, Keras & Tensorflow. I assume that in curiosity-based exploration, the 'surprised' AI would have been programmed to explore more of the discrepancies that caused the surprise. If so, Isn't it structurally similar to reward-based learning acting on rewards in the sense that they are different ways to refer to the the KPI that the AI should design their future action based on the number?

[JairusJia]

Can the efficiency of human preference feedback in deep reinforcement learning be further optimized to reduce reliance on manual labeling?

[tyeddie]

In what situations might a model-based approach be more advantageous than a model-free one? Does the selection resemble the choice between parametric and non-parametric methods as in stats?

[siyangwu1]

How can reinforcement learning systems adapt to evolving human preferences over time without requiring continuous manual oversight? Are there mechanisms to detect and adjust for preference drift, ensuring that learned policies remain aligned with long-term human goals rather than overfitting to transient preferences?