MLOps Is DevOps for Models
An ML system is a model plus data pipelines, evaluation, deployment, and monitoring. MLOps is the discipline that keeps the whole thing correct over time. Models degrade silently — no stack trace, no crash, just slowly wronger answers — so the operational bar is higher than for ordinary software.
Experiment Tracking
Every training run needs its parameters, code version, dataset, metrics, and artifacts recorded. Without tracking, “which run produced the model in production?” is unanswerable. A run is a tuple: (data commit, code commit, hyper-parameters, seed) → (metrics, model artifact). Track it all; reproducibility is the foundation of ML engineering.
Data Pipelines and the Data Leak
Models are downstream consumers of data. If the pipeline changes the meaning of a feature between training and serving — same column name, different definition — the model silently breaks. Rules:
- Version your datasets like code.
- Validate schema and distributions on every pipeline run.
- Never train on data from the future (features computed with the label, or with data that wouldn’t exist at prediction time).
Offline Evaluation
Before shipping, measure the candidate model against the production model on a frozen test set:
| Check | Question it answers |
|---|---|
| Metric improvement | Is it better on the chosen metric? |
| Segment breakdown | Is any user segment worse off? |
| Slice analysis | Is it fair across groups? |
| A/B plan | What will we measure in production? |
Evaluation should also stress the model on adversarial and edge cases, not just the happy path.
The Model Registry
A registry stores every candidate model with its metadata and a promotion workflow: candidate → validated → staged → production. Rollback is a registry operation: point production back at the previous model. Never hand-copy model files; always route through the registry.
Drift and Retraining
Models decay as reality changes:
- Data drift — the inputs shift (new products, new words, seasonality).
- Concept drift — the relationship between inputs and labels changes.
Monitor input distributions and prediction quality in production. When drift crosses a threshold, retrain and re-evaluate. The retraining cadence is a product decision: too frequent wastes compute, too rare degrades quality.
CI/CD for ML
Model pipelines get CI/CD too:
- CI — lint and test pipeline code; run data validation; train a small smoke model.
- CD — on merge, retrain, evaluate against the production model, and gate on the evaluation.
- Deploy — promote through the registry with automatic rollback on metric regression.
The model, not the code, is the deployable artifact — which is why evaluation gates replace unit tests as the release bar.
Guardrails in Production
Even with RAG and fine-tuning, LLM outputs need runtime guardrails: input checks (prompt injection, PII), output checks (policy violations, harmful content), and allowed-domain constraints. Log inputs and outputs for auditing. Guardrails are a separate, testable layer — not a prompt-wish.
Practice Trajectory
- List every artifact a training run must record for reproducibility.
- Describe one data-leak scenario that would silently destroy accuracy.
- Sketch a promotion and rollback workflow using a model registry.
- Define data drift vs concept drift with one example each.
- Design a guardrail layer for a customer-facing LLM chatbot.