Disk fills / database grows unbounded

Symptom

The Storage Health card on the Dashboard shows a warning, and its Database row is large compared with the data the box should hold. Deleting old rows, or shortening retention, does not shrink the file.

The cause: old rows are deleted on schedule, but a database created by an older release keeps the freed pages in an internal free list. It never returns them to the disk, so the file stays at its largest size. Updating the box does not change this. A database created by a current release returns freed space after each retention sweep.

Confirm

Run the compaction script and read what it measures. It reports before it changes anything.

1. Go to the release bundle directory you install and update from.

2. Run:

   sudo ./compact-db.sh

3. Read the report: database size, free list (the space it can reclaim), and free space on the disk.

4. If the free list is large, continue with the fix. To stop here, answer N at the prompt. Nothing has changed yet.

If there is less than 100 MB to reclaim and the database already returns space on its own, the script says so and exits.

Fix

1. Check free space. Compaction writes a full second copy of the database before it swaps it in, so the disk needs free space at least equal to the database size. The script checks this and refuses to start if there is not enough.

2. Plan a short maintenance window. The script stops the whole stack, so inference stops while it runs. On a large database this takes minutes.

3. Run the script from the release bundle directory and answer y at the prompt:

   sudo ./compact-db.sh

   It then:

   • takes a snapshot of the database into the backups folder of the install directory (/opt/aiboard by default);
   • stops the stack;
   • compacts the database and switches it to return freed space from now on;
   • starts the stack again.

   Do not interrupt it while it compacts.

4. Check the result. The script prints the size before and after and the space reclaimed. On the Dashboard, the Database row shows the new size.

Keep the snapshot

--no-backup skips the snapshot. Do not use it unless you have another backup: compaction rewrites the whole file, and the snapshot is the only way back. Use --yes to skip the prompt in an unattended run.

Note

You need to compact only once per database. After that the database returns freed space after each retention sweep.

Prevent

The backend deletes old data on a schedule. The defaults are:

Data	Kept for	Setting
Inference records (latency, throughput, pipeline events)	1 day	InferenceObservability:RetentionDays
Request and response payloads	6 hours	InferenceObservability:PayloadRetentionHours
Inference records, row cap	newest 5,000,000 rows	InferenceObservability:MaxRows
Logs page entries	newest 10,000 lines	LoggingService:Retention:MaxEntries

• Keep retention bounded. Longer retention means a larger database.
• Keep the row cap above normal volume. The row cap trims to the newest rows even inside the time window, so a burst cannot fill the disk. If it is lower than what a normal day produces, it cuts the time window short.
• Be careful with throughput tests. High-rate write bursts are what inflate the file. Do not leave a high-rate test running on a production box.
• Watch old backups. Each update keeps the last few database backups; see AIBOARD_BACKUP_KEEP in Environment Variables.

Related

• Monitoring → Health cards — the Storage Health card and its thresholds.
• Operating Best Practices — when to compact.