Deep Dive: Robust Interplanetary Trajectory Design with RL

Mission setup & motivation

The case study is a time-fixed Earth→Mars rendezvous under Sun gravity, modeled as an $N=40$-step sequence of impulses (Sims–Flanagan-like discretization). Each step is roughly nine days. The goal is to maximize final mass while meeting terminal constraints on Mars-relative position and velocity. Uncertainties are injected as: (i) state/disturbance noise, (ii) observation (navigation) noise, (iii) thrust execution errors (small-angle pointing + magnitude), and (iv) single- or multi-step missed thrust events.

Problem setup & why robustness matters

Picture a small spacecraft leaving Earth, arcing around the Sun on a transfer toward Mars. Every few days a guidance computer decides whether to fire the thruster—just a nudge—or to coast. The mission isn’t only about getting there; it’s about arriving on target with as much propellant left as possible. This post studies a time-fixed Earth→Mars rendezvous, discretized into \(N=40\) decision steps (about nine days each). The objective is to maximize final mass while meeting terminal constraints on Mars-relative position and velocity, despite four uncertainty sources: (i) state/disturbance noise, (ii) observation (navigation) noise, (iii) execution errors (small pointing/magnitude errors), and (iv) single- or multi-step missed-thrust events (MTEs).

From physics to a POMDP

The guidance problem is cast as a Partially Observable Markov Decision Process (POMDP) so the policy can react to noisy navigation and imperfect actuation.

State, observations, actions, dynamics.

Reward and objective.

How PPO is used here

1) Goal

The aim is to learn parameters \( \theta \) that maximize discounted return:

2) Policy-gradient with an advantage baseline

Using the log-derivative trick and a baseline gives an estimator that scales with advantage \( \hat A_k \) (how much better an action was than typical in that state):

Here \( \hat A_k \) is computed from value estimates (GAE is used to reduce variance), \( \hat A_k \approx Q^\pi(s_k,a_k) - V^\pi(s_k) \).

3) Proximal update via clipping

To avoid destructive parameter jumps, PPO uses the probability ratio \( r_k(\theta)=\tfrac{\pi_\theta(a_k|s_k)}{\pi_{\theta_{\text{old}}}(a_k|s_k)} \) and clips it to stay within \( [\,1-\epsilon,\;1+\epsilon\,] \):

Intuition: if \( \hat A_k>0 \) and the new policy pushes \( r_k \) above \( 1+\epsilon \), the objective stops increasing—preventing an overly large update.

4) Full PPO objective

PPO augments the clipped term with a value-regression loss and an entropy bonus:

where

The authors linearly decay both the clip range \( \epsilon \) and learning rate during training to stabilize updates.

5) Paper-specific implementation - What was actually used?

1. Observations: noisy \((\mathbf r,\mathbf v,m)\) and a time/step index; channels are normalized to natural scales (e.g., 1 AU).
2. Actions: 3-D impulsive \( \Delta\mathbf v \) sampled from a Gaussian and projected to a per-step magnitude bound set by thrust/mass.
3. Dynamics & uncertainty: two-body propagation with impulses; state/observation/control noise; single- and multi-step missed-thrust events.
4. Reward: \( R_k = -\mu_k - \lambda_{eu}e_{u,k} - \lambda_{es}e_{s,k} \) with a two-stage \( \varepsilon \)-schedule on the terminal tolerance (loose early for exploration, tight later for accuracy). This curriculum was the most impactful trick reported by the authors.
5. Networks: separate actor/critic MLPs (tanh). Hidden widths follow the paper’s heuristic (e.g., \( h_1=10n_{in},\; h_2=\sqrt{h_1h_3},\; h_3=10n_{out} \)); compact enough for on-board inference.
6. Optimization: Learning rate and clip-range decay linearly, several epochs used per update and periodic Monte-Carlo evaluation are done which returns the best-performing checkpoint.

Tightening constraints with an $\varepsilon$-schedule

Reward shaping can accidentally encourage hacks. Here, a simple idea helps: start training with a looser terminal tolerance $\varepsilon$ so the policy learns to make progress, then tighten $\varepsilon$ halfway through training. This acts like curriculum learning for constraints and empirically stabilizes training.

At a high level, the per-step reward is of the form

$\displaystyle R_k = -\mu_k - \lambda_{\mathrm{eu}}\,e_{u,k-1} - \lambda_{\mathrm{es}}\,\max(0, e_{s,k} - \varepsilon)$

where $\mu_k$ is normalized fuel use, $e_{u,k}$ penalizes exceeding stepwise $\|\Delta v\|$ bounds, and $e_{s,N}$ penalizes terminal position/velocity error.

What made it work: implementation “knobs”

• Curriculum via an ε-schedule: start with looser terminal tolerances so the agent learns to make progress, then tighten mid-training.
• Normalization: nondimensionalize states and rewards to stabilize training and help generalization across scales.
• Policy/value nets: compact MLPs are sufficient for this task, and the actor outputs a mean and std for a Gaussian over Δv within per-step bounds.
• Policy and value networks: compact MLPs are sufficient for this task, and the actor produces a mean and standard deviation that parameterize a Gaussian distribution over Δv within the allowed bounds for each step.
• Optimization: decrease the learning rate and clip range linearly, tune hyperparameters with a lightweight search, and keep batch and mini-batch sizes consistent with typical on-policy PPO practice.

These small choices add up: together with the ε-schedule they noticeably improved stability and final performance in my presentation experiments.

PPO hyperparameters

Linear schedules used during training

Both the learning rate and the clip range are decayed linearly over the total training step budget \(T\):

Early in training, larger updates and wider exploration are beneficial. As convergence approaches, reducing the step size \( \alpha(t) \) and narrowing the effective trust region through a smaller \( \varepsilon(t) \) lowers variance and avoids policy collapse. This acts as a progressively tighter constraint on the KL divergence budget.

How these were tuned using Optuna's Tree-Structured Parzen Estimator (TPE)

Hyperparameters were tuned once on a deterministic (unperturbed) mission scenario using Optuna’s TPE (link in resources below) with a budget of 500 trials and a cap of \(2.4\times10^6\) training steps per trial. Categorical knobs (e.g., \( \gamma,\ \lambda,\ \varepsilon_0 \)) were searched over the finite sets in the table; scale-sensitive knobs (e.g., \( \alpha_0,\ c_2 \)) were sampled log-uniformly. Periodic Monte-Carlo evaluation (100 runs) promoted the best checkpoint found so far. See the exact Optuna library reference in References & resources.

What the Tree-Structured Parzen Estimator (TPE) is doing

TPE models \(p(x \mid y)\) instead of \(p(y \mid x)\) for a hyperparameter vector \(x\) and an observed score \(y\) (e.g., final return). Trials are split by a threshold \( y^\star \) (top-\( \gamma \) fraction). Two Parzen density models are fit:

where \( \mathcal{K} \) is a Parzen kernel (Gaussian for continuous variables; smoothed categorical kernels otherwise). The next suggestion maximizes a proxy for expected improvement:

In simple words, the “promisingness” of a candidate is \( \displaystyle \frac{p(x\mid\text{good})}{p(x\mid\text{bad})} \). See the Optuna TPE entry in References & resources.

Interactive: Tree-Structured Parzen Estimator “promisingness”

TPE chooses the next trial by sampling where the ratio of Parzen densities over the “good” and “bad” regions is largest.

Network architecture & sizing (actor)

The paper uses separate heads for actor (control policy) and critic (value function). With \(n_{\text{in}}=8\) (namely position and velocity, mass and time), the actor outputs \(n_{\text{out}}^{\text{act}}=3\) (Δv in 3 axes) and the critic \(n_{\text{out}}^{\text{crit}}=1\) (scalar value). All hidden layers use tanh and the output layers are linear. Both the actor and critic network uses three hidden layers sized by the paper’s heuristic:

Default below shows \(n_{\text{in}}=1\), \(n_{\text{out}}=3\) → 1–10–18–30–3.

Results and takeaways

What the trained policies actually do

• Near-optimal when clean: In unperturbed tests, the learned policy’s final mass is comparable to an indirect optimal method—evidence that PPO didn’t sacrifice too much efficiency to gain robustness.
• Robustness trade: With state/obs/control noise, the policy favors earlier partial corrections and avoids saturating $\|\Delta v\|$—leaving breathing room near rendezvous.
• MTEs are toughest: Multi-step missed-thrust episodes (late in the trajectory) degrade success rates most, motivating either more conservative policies or training curricula that include families of failure models.

Figures from the paper — what they show

Uncertainty	Parameter	Value
State	σr,s (km)	1.0
	σv,s (km/s)	0.05
Observation	σr,o (km)	1.0
	σv,o (km/s)	0.05
Control	σϕ (°)	1.0
	σϑ (°)	1.0
	σψ (°)	1.0
	σu (—)	0.05
Single-step MTE	pmte, nmte	0, 1
Multiple-step MTE	pmte, nmte	0.1, 3

Notes — Table 3

• State noise: σr,s (km), σv,s (km/s) — process/disturbance on dynamics.
• Observation noise: σr,o (km), σv,o (km/s) — navigation/measurement error.
• Control errors: pointing σφ, σϑ, σψ (deg); magnitude σu (fractional, e.g., 0.05 = 5%).
• Missed-thrust events (MTE): pmte = per-step miss probability; nmte = consecutive missed burns.
• All σ are 1-σ (Gaussian). Units: km, km/s, deg; probabilities unitless.

Monte-Carlo results

\(m_f\): final mass; \(e_r\), \(e_v\): terminal position/velocity errors (scaled); SR: success rate.

\(\pi^{unp}\) values are from the deterministic overview (Table 4 in the paper).

Table 5 in the paper also reports the tolerance \(\varepsilon_B\) needed to reach 1σ/2σ/3σ success.

Limitations & next steps

• Impulse model: \(N=40\) impulses is coarse for true low-thrust flight. A finite-thrust propagator is the natural upgrade or perhaps transitioning to fully continuous thrust modeling..
• Environment realism: the MTE training setups always include at least one miss, which can overfit policies to that specific failure pattern; broader failure families would improve generality.
• Verification: results rely on Monte-Carlo success. There are no formal guarantees (e.g., robust tubes or reachability) in this paper which is also a commong shortcoming in many other RL applications in aerospace. Having formal guarantees would increase trust, since it is often hard to reconcile why the agent acts in a certain way.

A compelling direction is to embed this robust leg solver inside a multi-gravity-assist (MGA) designer used for deepspace missions. Use a global search to pick sequences and epochs, then call the RL policy to realize each leg robustly. One such algorithm with which it could easily be combined to achieve a robust and fuel vs time balancing MGA trajectory is my master thesis project. For context and future work ties, see: my master’s thesis and the MGA trajectory-optimization project on the research page.

Compute & Reproducibility

Training follows a two-phase loop:

1. Policy rollout. Run the current policy \( \pi_\theta \) in nenv parallel environments for nb episodes of length \(N\), collecting on-policy trajectories \( \{\tau_i\}\).
2. Policy update. For each update, optimize the PPO loss for nopt epochs over mini-batches drawn from those fresh trajectories only.

With total training steps \(T\), the number of updates is $$ N_{\text{upd}} \;=\; \frac{T}{\,n_{\text{env}}\, n_b\, N\,}. $$

A strong student desktop can reproduce the runs: the authors report ~\(2\times 10^8\) steps in ≈12 hours on an 8-core i7-9700K with 8 envs. Exact per-policy settings (e.g., \(n_{\text{env}}, n_b\)) are in Table 4 of the paper, and the full code is available on GitHub: LorenzoFederici/RobustTrajectoryDesignbyRL.