Jetson Deployment

X-Edge AI runs on NVIDIA Jetson modules (arm64 / L4T) with the same three-service stack as x86 boxes — frontend, backend, and the Python inference service. What changes on Jetson is the inference image: it must be built against the JetPack-provided CUDA/TensorRT libraries, with an ONNX Runtime wheel that matches your JetPack version.

Note

Jetson deployment ships from the dedicated Jetson-device release line. Updates flow one-way from the main product line into it, so Jetson builds always lag the newest x86 release by design.

Supported profiles

Profile	Device / JetPack	Base image	ONNX Runtime	Execution provider
jetson	Orin / Xavier / Nano, JetPack 5+ (L4T r35+)	l4t-base:r35.4.1	JetPack wheel (per device)	TensorRT → CUDA → CPU
jetson-gpu	TX2, JetPack 4.5 (L4T r32.5)	l4t-base:r32.5.0	onnxruntime-gpu 1.10.0 (cp36 wheel)	CUDA (TensorRT EP not available on JetPack 4.5)
jetson-cpu	Any arm64 Jetson	python:3.11-slim (arm64)	onnxruntime (CPU)	CPU

Two implications of the TX2 GPU profile worth knowing before you start:

• ONNX opset ceiling. ONNX Runtime 1.10 supports up to opset 15. Export TX2-bound models with --opset 15 (the bundle seeds an opset-15 model for this reason). Newer Jetsons running current JetPack wheels accept opset 17.
• Frozen Python 3.6 dependency set. JetPack 4.5 is locked to Python 3.6, so the jetson-gpu image uses a pinned legacy dependency fork. Prefer jetson (Orin/JetPack 5) for new fleets.

Prerequisites (on the Jetson)

Images are built natively on the device — cross-building L4T images under QEMU is unreliable and unsupported.

docker version              # >= 20.10
docker compose version      # Compose v2 plugin required
sudo apt-get install -y zstd openssl
df -h ~                     # keep a few GB free (TX2 eMMC is small)
docker info | grep -i nvidia   # GPU profiles: NVIDIA container runtime must be wired in

For GPU profiles, pick the ONNX Runtime wheel matching your JetPack version from the Jetson Zoo — the x86 onnxruntime-gpu pip wheel does not work on Jetson (it links x86 CUDA libs; “missing library” at startup is the classic symptom).

Build → bundle → install

flowchart LR
  A["build-release-images.sh\n(build + tag 3 images)"] --> B["build-release-bundle.sh\n(docker save → dist/<profile>-<version>/)"]
  B --> C["install.sh on the target\n(load images → compose up → seed admin)"]
  A -. "build host: Jetson, internet once" .-> B
  C -. "target host: air-gapped OK" .-> C

1. Build the images (on the build Jetson, internet required once):

   # GPU profile — pass the JetPack-matched ONNX Runtime wheel
   ORT_WHEEL_URL="https://<jetson-zoo-wheel-for-your-jetpack>.whl" \
     ./scripts/build-release-images.sh <version> --profile jetson-gpu
   
   # CPU profile — no wheel needed
   ./scripts/build-release-images.sh <version> --profile jetson-cpu

   Before the first GPU build, de-risk the wheel and GPU access with the probe script: ./scripts/spike-jetson-gpu.sh (GO/NO-GO check).

2. Package the offline bundle:

   ./scripts/build-release-bundle.sh --profile jetson-gpu --version v<version>

   The output dist/jetson-gpu-v<version>/ contains the compressed image archive with an integrity manifest, a Tegra-correct compose file, the seed model, and the install.sh / update.sh / uninstall.sh runbook scripts — the same layout as the x86 bundle described in Offline Bundle Install.

3. Install on the target device (copy the bundle directory via USB/scp):

   cd dist/jetson-gpu-v<version>
   sudo ./install.sh                    # auto-detects arm64 and accepts jetson-* bundles
   # sudo ./install.sh --with-systemd   # optional: start on boot

   The installer verifies the archive checksum, loads images, generates a unique JWT secret and admin password, starts the stack, waits for health, and prints the dashboard URL and admin password once — store it immediately.

4. Verify GPU inference is real:

   docker compose -f compose/docker-compose.release.yml ps                   # 3 services healthy
   docker inspect aiboard-inference-real --format '{{.HostConfig.Runtime}}'  # expect: nvidia
   docker logs aiboard-inference-real 2>&1 | grep -iE "CUDA EP|provider"     # CUDA/TensorRT EP active
   curl -s http://localhost:5000/api/system/metrics | grep -oE '"gpu"[^}]*}' # real GPU metrics

Tegra-specific behavior

The Jetson bundle compose differs from the x86 one on purpose:

• runtime: nvidia and root inside the inference container — required for GPU device access on Tegra.
• /sys mount — GPU load and temperatures come from Tegra sysfs (there is no nvidia-smi on Jetson), feeding the dashboard’s GPU panel.
• OPENBLAS_CORETYPE pin — avoids an OpenBLAS illegal-instruction crash on Tegra cores.
• Per-field N/A metrics — dashboard health fields a Jetson cannot report show as N/A instead of fake zeros.

Execution-provider selection works the same as on x86: EXECUTION_MODE=auto tries TensorRT → CUDA → CPU and logs a warning if it degrades. For a hard guarantee that the box never silently falls back to CPU, set EXECUTION_MODE=cuda (or tensorrt) plus STRICT_EP=1 — the service then refuses to start instead of degrading. See Execution provider fell back to CPU.

Caution

The jetson-gpu / jetson-cpu bundles are build- and lab-verified; final end-to-end validation of the air-gapped install on a clean TX2 is still in progress. Keep a serial/SSH console available during your first on-device install.

Versioning on the Jetson line

Jetson releases are versioned by the VERSION file only — the Jetson line never creates product v* tags. When you see version drift between the dashboard’s About panel and your bundle, compare against the bundle manifest, not Git tags. See Versions & Updates.

If something goes wrong

Symptom	Fix
unknown shorthand flag: 'f' in -f	Compose v2 plugin missing — sudo apt-get install -y docker-compose-plugin
Cannot autolaunch D-Bus during build	Headless credential-helper issue — the build script isolates it automatically; for ad-hoc docker commands use an empty DOCKER_CONFIG dir
GPU container: no CUDA-capable device	Host is missing the NVIDIA container runtime, or the compose lacks runtime: nvidia (the bundle compose already sets it)
Inference exits with a missing-library error	Wrong ONNX Runtime wheel for your JetPack — pick the matching wheel from the Jetson Zoo and rebuild
install.sh: “profile does not match bundle”	Pass --profile jetson-gpu (or jetson-cpu) explicitly — both are valid on arm64
Model rejected at upload (TX2)	Model exported with opset > 15 — re-export with --opset 15 for ONNX Runtime 1.10

Next steps

Offline Bundle Install The full air-gapped install runbook shared by all profiles.

Hardware Setup Execution-provider modes and network prerequisites.

Preparing Models with modelctl Export ONNX models with the right opset for your device.

GPU / CUDA errors Troubleshoot GPU visibility and CUDA failures.