Snap & Track Vision Pipeline

Overview

Snap & Track is a mobile nutrition tracking app currently in the late stages of development. This page highlights its key feature: the food vision pipeline. It accepts a food photo with metadata and outputs identified foods matched to real USDA FoodData Central rows from a scraped database, along with estimated mass. Photo in, matched FDC foods + mass out.

The greater part of the pipeline is the alignment engine — taking a predicted food and reliably matching it to the correct entry in the FDC database. This is a deceptively hard problem with many edge cases, and the bulk of the engineering effort lives here.

The MVP runs in a Docker container, deriving matches from an entire 1M+ row scraped FDC database in Neon PostgreSQL. The pipeline has been evaluated against 459 ground-truth dishes prioritized for having the most accurate nutrition data. LiDAR-based volume detection is planned as a future enhancement.

The codebase includes a robust testing suite with 27+ test modules covering alignment guards, stage-specific behavior, form normalization, cooking conversions, telemetry schema validation, end-to-end pipeline tests, and batch evaluations. Documentation spans 110+ files including runbooks, deployment checklists, sprint plans, and phase completion reports.

The pipeline has been successfully integrated into the Snap & Track mobile app.