Unbalanced discounted reward in reinforcement learning : is it a problem?

Question

Unbalanced discounted reward in reinforcement learning : is it a problem?

Astariul

2019年8月8日 05:52

Discounted rewards seems unbalanced to me.

If we take as example an episode with 4 actions, where each action receive a reward of +1 :

+1 -> +1 -> +1 -> +1

The discounted reward for the last action is : 1
The discounted reward for the first action (considering gamma = 1 for simplicity) is : 4

Intuitively both action are as good as the other, because both received same reward.
But their total reward is different, unbalanced.

So when we will backpropagate, first action will be favored over last action ?

Topic discounted-reward reinforcement-learning

Category Data Science

Vamshi Kumar Kurva · Accepted Answer · 2019年8月8日 05:52

Most of the RL problems are sequential decision making processes. That is the action taken at time $t$, can influence the future rewards in a sequential way. From this perspective, the first action lead to a sequence of states/actions which produced positive rewards, which means it deserves more positive feedback than the last action. It only seems natural to prefer the first action in the first state unless we know some other action yields better cumulative rewards. Rewards are gains in short-term, an action might be good in the short term but not in the long run, which is why we use cumulative rewards called returns to estimate the value of an action in a given state.

Unbalanced discounted reward in reinforcement learning : is it a problem?

About