OT-SGN: Geometric Navigation and Control of Large Language Model Hallucinations via Optimal Transport and Manifold Tunneling

Large Language Models (LLMs) often exhibit hallucinations or logical loops when traversing high-entropy semantic regions. Conventional decoding strategies rely solely on probabilistic likelihood, lacking global semantic foresight. In this research, we propose OT-SGN (Optimal Transport - Semantic Geometric Navigation), a novel framework that treats text generation as a dynamic trajectory optimization problem on a Riemannian manifold.

The Core Problem: Hallucination as Geometric Drift

The generation process of LLMs can be viewed as a random walk on a high-dimensional manifold. While models excel at local coherence, they often fail to maintain global semantic direction, especially when initialized in high-perplexity regions (paradoxes).

Our key hypotheses:

  1. Geometry is Semantics: The trajectory of hidden states encodes the reasoning process
  2. Hallucination is Drift: Deviations from the “Truth Manifold” can be corrected by external geometric forces
  3. Insight is Tunneling: Profound “Aha moments” correspond to non-linear jumps across low-probability regions

Methodology: Manifold Construction and Control

Semantic Potential Fields via Optimal Transport

We map high-dimensional hidden states $h \in \mathbb{R}^D$ to a low-dimensional control plane $\mathcal{M} \subset \mathbb{R}^2$ using PCA, calibrated by two sets of anchors:

  • Truth Manifold ($T$): Represented by embeddings of ArXiv abstracts (low entropy regions)
  • Paradox Singularity ($P$): Represented by logical paradoxes (high entropy regions)

We define a semantic potential field $U(z) = \min_{t \in T} |z - t|_2$, creating a gradient pointing towards logical consistency.

Geodesic Guided Decoding

Unlike standard autoregressive decoding, we introduce a dual-objective beam search where the score for a candidate token $w$ is:

\[S(w) = \log P_{LM}(w | w_{<t}) - \lambda(H) \cdot \| \phi(h(w)) - \gamma(t) \|_2\]

Where:

  • $\phi$: Projection function to 2D control space
  • $\gamma(t)$: Pre-calculated ideal geodesic curve (cubic spline)
  • $\lambda(H)$: Adaptive control coefficient based on predictive entropy $H$

Multi-Hop Tunneling and Spectral Noise

To overcome the “Eigen-word” phenomenon (where PC1 is dominated by stop words), we employ Spectral Noise Injection. We reconstruct high-dimensional vectors by injecting variance-scaled noise into components $PC_3 \dots PC_n$, enabling generation of specific semantic entities rather than syntactic repetition.

For cross-domain tasks, we implement Force Mode with extreme $\lambda$ values ($\lambda > 30$), sacrificing grammatical smoothness to force traversal of “Semantic Dead Seas” through “Stepping Stones.”

Experimental Results on Qwen2-7B

The Anisotropic Singularity

Visualization of Layer 14 attention reveals extreme anisotropy. Function words cluster at extreme coordinates ($x \approx -8000$), while semantic content is compressed near the origin. This confirms that geometric operations must account for the “Syntax-Semantics” principal component separation.

Controlled Hallucination (Geometric Dreaming)

By forcing models to follow curves passing through specific latent regions, we achieved Location-based Semantic Injection. A trajectory pulled towards “high PC2” regions spontaneously generated “The Greek Debt Crisis” from a generic finance prompt, demonstrating that specific memory retrieval can be triggered purely by geometric positioning.

The “Aha Moment” Tunneling

In our tunneling experiments, we forced collisions between disparate concepts like “Paradox” and “Phase Transition.” While raw output contained broken syntax with LaTeX fragments, post-processing via our Renormalization Refiner produced coherent insights: “The resolution… is unveiled through the innovative application of asymptotic analysis…”

High $\lambda$ acts as a particle collider, smashing concepts together to produce “semantic debris” containing profound, non-obvious links.

Key Findings and Limitations

The Uncertainty Principle of LLMs

Our experiments reveal a fundamental trade-off: Precision vs. Fluency.

  • High $\lambda$ (Force Mode): Ensures geometric target hits but causes “Semantic Aphasia” (broken grammar)
  • Low $\lambda$ (Adaptive): Preserves grammar but succumbs to “Manifold Stickiness”—refusing to jump between domains

The 7B Parameter Limit

In bridge-building experiments, the 7B model successfully hit geometric coordinates for “Information Theory” but generated generic mathematical terms rather than specific entities. We attribute this to manifold sparsity, hypothesizing that 70B+ models possess denser semantic manifolds for smoother transitions.

Implications for AI Control and Safety

OT-SGN demonstrates that LLM generation can be externally controlled via geometric constraints. We transformed the generation process from a probabilistic game into a navigational task, establishing “Geodesic Rails” that enable models to traverse semantic voids they would naturally avoid.

This work has significant implications for AI safety and alignment:

  • Controllable Emergence: Turning hallucinations into controlled creativity
  • Geometric Interpretability: Understanding LLM reasoning through manifold trajectories
  • Cross-Domain Reasoning: Forcing synthesis of disparate knowledge areas

Future Directions

Future work will focus on:

  1. Scaling to 70B+ models for stable Multi-Hop Tunneling
  2. Advanced manifold representations beyond PCA projections
  3. Real-time adaptive control for production deployment
  4. Integration with reinforcement learning for trajectory optimization

Experimental Summary

Version Feature Key Result Insight
v10 Trajectory Vis “The” Singularity at $x=-8000$ PC1 represents Syntax/Frequency, not Semantics
v10.1 Spectral Noise “I I I” → “Financial…” Noise injection in $PC_{3+}$ required for semantic decoding
v11 Geodesic Rails “Greek Debt Crisis” generation Semantic injection possible via spatial coordinate forcing
v12 Quantum Tunneling Broken syntax → Asymptotic Analysis Broken syntax contains high-value semantic sparks
v13 Adaptive Refiner Blue/Red Trajectory Control Adaptive $\lambda$ prevents crash but fails deep semantic inertia
v14 Bridge Builder Magenta Force Mode Geometric forcing works, but 7B models lack semantic density

This research establishes OT-SGN as a concrete step toward “computable hallucination”—moving from observing LLM failures to systematically engineering their semantic navigation. The geometric approach offers a promising path for controlling AI behavior in high-stakes applications.