🗺️ LingBot-Map

🔮 Run Inference Remotely

Upload a short video and this tab calls the LingBot-Map model remotely via the dennny123/lingbot-3d ZeroGPU Space API.

⚠️ Note: First run may take 2-5 minutes (cold start: GPU allocation + 4.3GB model download). Only 1 request at a time.

🗺️ LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstruction

LingBot-Map is a feed-forward 3D foundation model that reconstructs 3D scenes from video streams in real-time at ~20 FPS.

Given a continuous video stream, it recovers:

• 📷 Camera poses for each frame
• 🌊 Depth maps per frame
• ☁️ 3D point clouds of the scene

Unlike traditional SLAM systems that rely on iterative optimization, LingBot-Map does this in a single forward pass through a transformer.

Paper

"Geometric Context Transformer for Streaming 3D Reconstruction"
Lin-Zhuo Chen, Jian Gao, Yihang Chen, Ka Leong Cheng, Yipengjing Sun, Liangxiao Hu, Nan Xue, Xing Zhu, Yujun Shen, Yao Yao, Yinghao Xu

📄 arXiv:2604.14141 | 🌐 Project Page | 💻 GitHub

🧠 Architecture: Geometric Context Transformer (GCT)

The core innovation is Geometric Context Attention (GCA) — a carefully designed attention mechanism with three complementary context types:

1️⃣ Anchor Context — Coordinate Grounding

Provides a stable spatial reference frame. Without it, the model's coordinate system can drift or become ambiguous over long sequences. The anchor acts as a "north star" that grounds all predictions.

2️⃣ Pose-Reference Window — Dense Geometric Cues

A sliding window of recent frames that supplies rich local geometric information. This is where the model gets its precise relative pose estimates and dense depth predictions — analogous to how classical SLAM uses co-visibility frames for local bundle adjustment.

3️⃣ Trajectory Memory — Long-Range Drift Correction

A compact, long-range memory that prevents accumulated drift over thousands of frames. Unlike the dense pose-reference window, trajectory memory stores sparse keyframes that enable the model to "remember" places it saw long ago — similar to loop closure in traditional SLAM, but learned.

Key Design: Paged KV Cache

Like autoregressive LLMs, LingBot-Map caches KV states of processed frames. It uses a paged KV cache with sliding window + keyframe selection to keep memory bounded while retaining long-range context. This enables stable inference on sequences exceeding 10,000 frames.

Model Specs

📊 Benchmark Results

🏛️ Oxford Spires — Large-Scale Trajectory Estimation (Sparse: 320 frames)

🏆 LingBot-Map beats ALL offline, optimization, and online methods despite being streaming!

Dense Setting (3,840 frames) — Drift Resistance

LingBot-Map's ATE barely increases over 12× longer sequences!

3D Reconstruction (F1 Score)

ETH3D F1 of 98.98 — nearly perfect reconstruction!

🚀 Quick Start

Installation

conda create -n lingbot-map python=3.10 -y
conda activate lingbot-map
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
pip install -e .
pip install flashinfer-python -i https://flashinfer.ai/whl/cu128/torch2.9/

Inference from Images

python demo.py --model_path lingbot-map.pt \
    --image_folder /path/to/images/

Inference from Video

python demo.py --model_path lingbot-map.pt \
    --video_path video.mp4 --fps 10

Long Sequences (10,000+ frames)

python demo.py --model_path lingbot-map.pt \
    --image_folder /path/to/images/ \
    --keyframe_interval 6

Windowed Mode (>3000 frames)

python demo.py --model_path lingbot-map.pt \
    --video_path video.mp4 --fps 10 \
    --mode windowed --window_size 64

Without FlashInfer (CPU fallback)

python demo.py --model_path lingbot-map.pt \
    --image_folder /path/to/images/ --use_sdpa

Model Checkpoints

All available at robbyant/lingbot-map