Reward and Observe Higher-Order Goalderivatives

The repository researches into possible improvements to Goal-Oriented Reinforcement Learning (RL) by evaluating the Differential Goalkinematic State (DGS), whose components are based on the distance to the goal - in the following called goal-directed derivatives or simply goalderivatives.

The probed improvements include sample-efficiency during training and generality of the resulting policy to unseen goals.

Goalderivatives can speed-up the training by factor $6$ (reward shaped), factor $14$ (reward designed) or factor $20$ (observation augmented/reduced) compared to sparse RL environments.

_{Keywords: Deep Reinforcement Learning, Robotics}

Contents

1. Definition
2. Goalderivative Potential-Based Reward Shaping
3. Goalderivative Reward Design
4. Goalkinematic Observation Augmentation
5. Goalkinematic Observation Reduction
6. Fluent Visual Imitation of Hand Gestures by a Robotic Hand (Case Study)

Definition

A goalderivative $d^{(k)}$ of order $k>1$ is the rate of change in the scalar distance $d$ (specific to e.g. the $L²$-norm) or its derivatives (velocity $d^{(1)}$, acceleration $d^{(2)}$, jerk $d^{(3)}$ etc.) towards a numerically defined goal.

In time-discrete environments, the goalderivatives at a time $t_i$ can be recursively estimated with the distance to the goal (goaldistance) at $t_i$ by backward difference:

In a vector, multiple goalderivatives of $k > 1$ at $t_i$ form a differential kinematic state vector

The idea is now to evaluate the vector for either reward design, observation augmentation, or both.

Summary

Instead of using only the distance to the goal, its derivatives are also considered - in the reward function or as part of the observation in RL.

RL algorithms are applied to a formal Markov Decision Process (MDP)

to find, for each state $s_k \in S$, the optimal action $a^*_{k} \in A$ that maximizes the expected $\gamma$ - discounted return

of experienced rewards ${r’, r’’,\dots} \subseteq R$ from its next states ${s’, s’’,\dots} \subseteq S$, which are reached with probabilities in $T$. However with RL, $T$ and $R$ are initially unknown and must be gradually discovered by exploration similar to trial and error. ¹

The result is an optimal policy $\pi^* : S \to A$ that assigns to each state its optimal action. Note that a higher-level goal is not formalized in the MDP - its optimality does not ensure the objective success in reaching (or keeping) an indirect goal. Careful and effective design of the underlying MDP is therefore crucial for the correct and efficient convergence of RL.

Goalderivative Potential-Based Reward Shaping

Research Claim 1

In goal-oriented RL training towards an optimal policy, by adding to the existent rewards a potential-based shaping term based on the DGS vector, the training can be more efficient without changing the original optimal policy.

In this work, rewards are deterministic, i.e. they are assigned to states by the real reward function $R:S\times A \to \mathbb{R}$, meaning at state $s_t$ the algorithm receives a reward $r_t = R(s_t, a_t)$. It is proven that by adding a strictly shaping function $F(s_t, a_t) = \gamma\Phi(s_{t+1}) - \Phi(s_t)$ with state-dependent potentials $\Phi$, the reward function can be modified to

without changing the optimal policy: By solving the modified MDP $M’ = (S, A, T, \gamma, R’)$ we also solve the original MDP $M$. ²

With a naive goalkinematic potential

achieving states that are not only close, but are also expected to be closer in the next states, will be rewarded higher - their goalderivatives are more favorable. On the other hand, achieving states that are only statically close, or even moving away, will be rewarded lower. However, such a frequent rewarding enables reward hacking, where the agent might oscillate between moving closer and farther from the goal without actually reaching the goal, i.e. the converged policy is not the optimal policy, let alone a successful one. With a conjunctive goalderivative potential (CGP)

    phi = 0
    if np.all(goalderivs < 0):
        phi = 1
    elif np.all(goalderivs > 0):
        phi = -1

that type of reward hacking is less probable with higher order $k$ , since isolated goalderivatives are not rewarded anymore. Since they are also not penalized, exploration is allowed - this is especially beneficial in multi-goal environments.

In 5 simulations à 50 epochs (one epoch consists of 200 episodes à 50 timesteps) within Gymnasium’s sparse HandReach environment ³, where a robotic hand is trained to reach different coordinates with its fingertips, the shaped reward ($k=3, \gamma = 0.95$)

increased the training sample efficiency, defined as the mean area under the curve (AUC) of the test goalprogress

by factor 2.75 (Fig. 1: yellow line) compared to the unshaped baseline reward (Fig. 1: blue line) (median). The details of the experiments are described in the full work.

_{Fig. 1: Median test goalprogress by reward shaping (yellow and orange line) with interquartile range (IQR, shaded area) and mean AUC (±s.d., label)}

Note: With potential-based shaping, this claim theoretically requires that the goaldistance and the DGS are part of the observable state space ², which is a separate focus in this research. However, in the experiments, the efficiency gains with non-observable goaldistance and DGS were even greater with factor 6 (Fig. 1: orange line) compared to the baseline, i.e. possibly justifying the violation of the Markov assumption.

Goalderivative Reward Design

Research Claim 2

In goal-oriented RL training towards an optimal policy, by designing rewards based on the DGS vector, the training can be more efficient.

    reward = 0
    if np.all(goalderivs < 0):
        reward = 1
    if np.all(goalderivs > 0):
        reward = -1

_{Fig. 2: Median test goalprogress by reward design (green line) with IQR (shaded area) and mean AUC (±s.d., label)}

Goalkinematic Observation Augmentation

Research Claim 3

In goal-oriented RL training towards an optimal policy, by adding goalkinematic information to the observation space, the training can be more efficient.

    obs = np.concatenate(obs, [
                    goaldist, # current eucl. distance
                    goaldists_recent, # eucl. distances from prev. steps
                    goalderivs, # k-derivatives of current eucl. distance

                    goaldelta, # current difference in each goal dims.
                    goaldeltas_recent, # differences from prev. steps
                    goaldeltas_velocity # 1-derivatives in each dims. 
                    ]) 

_{Fig. 3: Median test goalprogress by obs. augmentation (red line) with interquartile range (shaded area) and mean AUC (±s.d., label)}

Goalkinematic Observation Reduction

Research Claim 4

In goal-oriented RL training towards an optimal policy, by reducing the observation space to contain only goalkinematic information, the training can be successful, more efficient and more general.

    obs = np.array([]) # empty observations

    obs = np.concatenate(obs, [
                    goaldist,
                    goaldists_recent,
                    goalderivs,

                    goaldelta,
                    goaldeltas_recent,
                    goaldeltas_velocity 
                    ]) 

_{Fig. 4: Median test goalprogress by obs. reduction (purple line) with IQR (shaded area) and mean AUC (±s.d., label)}

_{Fig. 5: Median test goalprogress (line) with IQR (shaded area) and mean AUC (±s.d., label)}

_{Fig. 6: Median test success rate (mean goaldistance ≤ 1cm, line) with IQR (shaded area) and mean AUC (±s.d., label)}

Fluent Visual Imitation of Hand Gestures by a Robotic Hand (Case Study)

• goal-reduced end-to-end RL
• naive mapping between visually detected joints and robotic joints
• trained on 18 images of different gestures
• tested on 4 images of unseen gestures (static imitation)
• tested on webcam/video input with intermediate gestures (fluent imitation)

_{Real-time screen recording of 15k rendered training steps directly after learning of the networks started for the first time (progress from blank policy with a single gesture)}

_{Real-time screen recording of rendered evaluation steps after 1M training steps (~55k steps per gesture, trained policy with random evaluation samples)}

Hyperparameters

Anecdotes

“with enough goaldimensions, the goaldistance becomes meaningful goalprogress”

“efficiency does not necessarily mean success/effectiveness”

References

• https://openai.com/index/ingredients-for-robotics-research/
• https://wandb.ai/rodrigodelazcano/gym_robotics/runs/1s3fuwye?nw=nwuserrodrigodelazcano