gRPC Inference API

The backend and the Python inference runtime communicate over gRPC on port 50051 (protos/inference.proto, service InferenceService). Operators rarely call this API directly — the backend is the client — but the contract matters when you debug the pipeline, integrate a custom client on the box, or read inference logs.

Caution

The channel is plaintext HTTP/2 (no TLS, no authentication) by design: it is intended to stay on the box’s internal Docker network. Never publish port 50051 beyond the host or a trusted management network — see Security.

Service methods

RPC	Type	Purpose
StreamInference	bidirectional stream	Sensor windows in, predictions out — the real-time hot path
LoadModel	unary	Validate + register + hot-swap an ONNX model (zero downtime)
ValidateModel	unary	Non-mutating inspection: shapes + SHA-256, nothing loaded
HealthCheck	unary	Service status + CPU/GPU/memory metrics
StreamLogs	server stream	Tail inference logs filtered by minimum level

StreamInference

Each InferenceRequest carries request_id, a timestamp (Unix epoch ms, stamped once by the backend at sensor-sample time and echoed back unchanged — so there is no clock-sync problem), a flat features array, and a metadata map.

Each InferenceResponse returns predictions and confidence_scores, the serving model_id/model_version, inference_time_ms, and three timestamps:

Field	Meaning
window_start_timestamp	Oldest input sample in the prediction’s window
window_end_timestamp	Newest input sample — the “as-of” time of the prediction
emitted_at	Wall-clock at emission; emitted_at − window_end_timestamp approximates end-to-end latency

During model warm-up (the sliding window is not yet full) responses carry metadata["warming_up"] = "true"; the backend suppresses these from the dashboard.

LoadModel

Registers a model version and hot-swaps it into the running session with double-buffering — the new session is loaded and warmed outside the lock, then the reference is swapped atomically, so streaming never pauses.

• Idempotent by content hash: re-sending the same (model_id, version) with an identical SHA-256 is a success no-op. The same version with a different SHA-256 is rejected as version_conflict — versions are immutable.
• The response echoes input_shape, output_shape, registered_path, and the computed sha256.
• If the new model’s (window_size, n_features) differ from the previous one, the windowing pipeline is rebuilt and warm-up restarts.

ValidateModel

Pure inspection used by the backend during model upload: opens a throwaway runtime session, extracts input/output shapes and SHA-256, and returns {valid, message, input_shape, output_shape, sha256} without registering anything.

Validation enforces the model contract:

• ONNX file ≤ 500 MB, loadable, and passing the ONNX checker
• Opset 13–18
• Input tensor input shaped (batch, window_size, n_features) — 3-D required; the batch dimension may be dynamic
• Output tensor output shaped (batch, 3) = [health_score, failure_probability, rul_normalized]

HealthCheck and StreamLogs

HealthCheck powers the dashboard health panel (GPU load, memory, uptime). StreamLogs forwards the Python service’s log stream to the backend, which persists it for the LogViewer — filter with min_level.

Runtime configuration

Environment variables the inference container honors (set in the compose file):

Variable	Default	Effect
EXECUTION_MODE	auto	Execution provider: auto (TensorRT → CUDA → CPU), tensorrt, cuda, cpu
STRICT_EP	off	1 = refuse to start instead of silently degrading a pinned GPU mode to CPU
MODEL_PATH	/app/models/predictive_maintenance.onnx	Seed model loaded at startup
MODEL_STORAGE_PATH	/data/models	Where registered model versions are stored
LOG_DIR	unset	Enables the rotating on-disk log file (inference.log); unset = console only
FORCE_CPU	off	Legacy alias for EXECUTION_MODE=cpu

Note

EXECUTION_MODE=auto probes TensorRT native libraries before trusting the runtime’s provider list, and logs a loud warning when a GPU mode degrades to CPU. Watch for it after driver or JetPack updates — see Execution provider fell back to CPU.

Next steps

API Overview REST API surface of the backend.

Deploying Models Upload, pair, and activate model versions from the UI.

Preparing Models with modelctl Author bundles that pass validation the first time.

Security Network posture and hardening checklist.