# gRPC Inference API

> The gRPC contract between the backend and the inference runtime — streaming inference, model loading, validation, health, and log streaming.

The backend and the Python inference runtime communicate over **gRPC on port
50051** (`protos/inference.proto`, service `InferenceService`). Operators rarely
call this API directly — the backend is the client — but the contract matters
when you debug the pipeline, integrate a custom client on the box, or read
inference logs.

The channel is **plaintext HTTP/2 (no TLS, no authentication)** by design: it is
intended to stay on the box's internal Docker network. Never publish port 50051
beyond the host or a trusted management network — see
[Security](/security/).

## Service methods

| RPC | Type | Purpose |
|---|---|---|
| `StreamInference` | bidirectional stream | Sensor windows in, predictions out — the real-time hot path |
| `LoadModel` | unary | Validate + register + hot-swap an ONNX model (zero downtime) |
| `ValidateModel` | unary | Non-mutating inspection: shapes + SHA-256, nothing loaded |
| `HealthCheck` | unary | Service status + CPU/GPU/memory metrics |
| `StreamLogs` | server stream | Tail inference logs filtered by minimum level |

### StreamInference

Each `InferenceRequest` carries `request_id`, a `timestamp` (Unix epoch ms,
stamped once by the backend at sensor-sample time and echoed back unchanged —
so there is no clock-sync problem), a flat `features` array, and a metadata map.

Each `InferenceResponse` returns `predictions` and `confidence_scores`, the
serving `model_id`/`model_version`, `inference_time_ms`, and three timestamps:

| Field | Meaning |
|---|---|
| `window_start_timestamp` | Oldest input sample in the prediction's window |
| `window_end_timestamp` | Newest input sample — the "as-of" time of the prediction |
| `emitted_at` | Wall-clock at emission; `emitted_at − window_end_timestamp` approximates end-to-end latency |

During model warm-up (the sliding window is not yet full) responses carry
`metadata["warming_up"] = "true"`; the backend suppresses these from the
dashboard.

### LoadModel

Registers a model version and hot-swaps it into the running session with
double-buffering — the new session is loaded and warmed **outside** the lock,
then the reference is swapped atomically, so streaming never pauses.

- **Idempotent by content hash:** re-sending the same `(model_id, version)` with
  an identical SHA-256 is a success no-op. The same version with a *different*
  SHA-256 is rejected as `version_conflict` — versions are immutable.
- The response echoes `input_shape`, `output_shape`, `registered_path`, and the
  computed `sha256`.
- If the new model's `(window_size, n_features)` differ from the previous one,
  the windowing pipeline is rebuilt and warm-up restarts.

### ValidateModel

Pure inspection used by the backend during model upload: opens a throwaway
runtime session, extracts input/output shapes and SHA-256, and returns
`{valid, message, input_shape, output_shape, sha256}` without registering
anything.

Validation enforces the model contract:

- ONNX file ≤ **500 MB**, loadable, and passing the ONNX checker
- Opset **13–18**
- Input tensor `input` shaped `(batch, window_size, n_features)` — 3-D required;
  the batch dimension may be dynamic
- Output tensor `output` shaped `(batch, 3)` =
  `[health_score, failure_probability, rul_normalized]`

### HealthCheck and StreamLogs

`HealthCheck` powers the dashboard health panel (GPU load, memory, uptime).
`StreamLogs` forwards the Python service's log stream to the backend, which
persists it for the LogViewer — filter with `min_level`.

## Runtime configuration

Environment variables the inference container honors (set in the compose file):

| Variable | Default | Effect |
|---|---|---|
| `EXECUTION_MODE` | `auto` | Execution provider: `auto` (TensorRT → CUDA → CPU), `tensorrt`, `cuda`, `cpu` |
| `STRICT_EP` | off | `1` = refuse to start instead of silently degrading a pinned GPU mode to CPU |
| `MODEL_PATH` | `/app/models/predictive_maintenance.onnx` | Seed model loaded at startup |
| `MODEL_STORAGE_PATH` | `/data/models` | Where registered model versions are stored |
| `LOG_DIR` | unset | Enables the rotating on-disk log file (`inference.log`); unset = console only |
| `FORCE_CPU` | off | Legacy alias for `EXECUTION_MODE=cpu` |

`EXECUTION_MODE=auto` probes TensorRT native libraries before trusting the
runtime's provider list, and logs a loud warning when a GPU mode degrades to
CPU. Watch for it after driver or JetPack updates — see
[Execution provider fell back to CPU](/troubleshooting/execution-provider-fallback/).

## Next steps

  - [API Overview](/api-reference/overview/) — REST API surface of the backend.
  - [Deploying Models](/configure/models/) — Upload, pair, and activate model versions from the UI.
  - [Preparing Models with modelctl](/configure/prepare-models/) — Author bundles that pass validation the first time.
  - [Security](/security/) — Network posture and hardening checklist.