Building a hands-on MLflow curriculum, and what actually breaks when you stop reading and start executing.
I had a roadmap for learning MLflow — a good one, phase by phase from “what is a run” to “how do I roll back production”. The kind of document you nod along to and then never act on, because reading about experiment tracking is not the same as having tracked an experiment.
So I turned it into a repo: 21 numbered folders, one per phase, each with runnable code. About 75 Python files, 22 READMEs, a full churn-prediction project with tests, and a Makefile so every phase is one command.
I set exactly one rule for myself:
Every example must actually run, and I have to read its output before I keep it.
That rule is the entire reason this post exists. Six examples that looked correct — that I would have published without a second thought — were quietly wrong. Not “wouldn’t compile” wrong. Produced plausible output and lied wrong.
Five of them are worth walking through. The sixth, and a handful of related traps, are in a table near the end.
1. The ensemble that was averaging one model with itself
I wrote a custom pyfunc that blends two models and falls back to a rules engine if they fail to load. Standard stuff:
mlflow.pyfunc.log_model(
name="model",
python_model=EnsembleWithFallback(weights=(0.6, 0.4)),
artifacts={"model_a": a_uri, "model_b": b_uri}, # two models:/ URIs
)Then I printed a table comparing every degradation path — both members, only A, only B, neither. This came out:
path roc_auc recall served_by degraded
both members 0.8403 0.5249 ensemble 0
model_a down 0.8403 0.5249 ensemble 0
model_b down 0.8403 0.5249 ensemble 0
both down 0.8403 0.5249 ensemble 0Four identical rows. Two separate bugs stacked on top of each other, and neither raised an exception.
Bug one: my simulate_failure parameter was being silently discarded. MLflow drops inference params that aren’t declared in the signature’s ParamSchema — it logs one warning and calls predict() with the defaults. My kill switch did nothing.
I added the schema. Now the paths differed… but the three model paths still reported identical metrics.
Bug two, the real one. I dumped the MLmodel manifest:
artifacts:
model_a: {path: 'artifacts/.', uri: 'models:/m-b7c4a5be...'}
model_b: {path: 'artifacts/.', uri: 'models:/m-f22537b3...'}Both models downloaded to artifacts/.. Same directory. The second overwrote the first, and context.artifacts["model_a"] and context.artifacts["model_b"] returned the same model. My ensemble was averaging a model with itself, at a 0.6/0.4 weighting, forever.
The fix is to materialise each model into its own directory first:
staging = Path(tempfile.mkdtemp())
a_local = mlflow.artifacts.download_artifacts(a_uri, dst_path=str(staging / "model_a"))
b_local = mlflow.artifacts.download_artifacts(b_uri, dst_path=str(staging / "model_b"))
mlflow.pyfunc.log_model(..., artifacts={"model_a": a_local, "model_b": b_local})And then the table says something:
both members 0.8288 0.5017 ensemble 0
model_a down 0.8403 0.5249 single:model_b 1
model_b down 0.8136 0.4917 single:model_a 1
both down 0.7137 0.1827 rules_fallback 1Think about how this fails in production. Your ensemble ships. It scores. Its metrics are plausible — a bit worse than you hoped, so you tune the weights, which also does nothing, because there is only one model. Nothing ever errors. You could run that for a year.
The lesson isn’t “MLflow has a bug”. It’s that I only caught it because I made the example print a comparison table instead of just a number. A single number would have looked fine.
2. int → double is refused, and everyone tells you otherwise
Schema enforcement is supposed to allow safe widening. Every tutorial I’ve read says int-to-float is a safe upcast. I wrote that in my README, then wrote a script that deliberately breaks the schema nine different ways to prove it.
Verified behaviour on MLflow 3.15:
| Input | Declared | Result |
|---|---|---|
int32 | long | allowed |
float32 | double | allowed |
int64 | double | raises — “Can not safely convert int64 to float64” |
float64 | long | raises (lossy, expected) |
The rule is widening within the same kind, not int→float. So a notebook where monthly_charges happens to be a float works, and a production caller that sends 70 instead of 70.0 gets a 400.
I had to rewrite the README table. If I’d shipped the version I wrote from memory, I’d have taught something false to whoever read it.
3. search_model_versions never returns aliases
This one costs an afternoon.
client.search_model_versions("name = 'churn-classifier'")
# -> [('3', []), ('2', []), ('1', [])] .aliases is ALWAYS empty
client.get_model_version("churn-classifier", "1").aliases
# -> ['champion'] here it is
client.get_registered_model("churn-classifier").aliases
# -> {'champion': '1', 'challenger': '2'} and hereBuild a promotion dashboard on search_model_versions — the natural choice, it’s the bulk API — and it reports that nothing is promoted. Forever. With no error.
I now have a common/registry.py with one function whose entire job is to work around this:
def alias_map(client, name) -> dict[str, list[str]]:
"""{version: [aliases]} -- because search_model_versions won't tell you."""
registered = client.get_registered_model(name)
out = defaultdict(list)
for alias, version in (registered.aliases or {}).items():
out[str(version)].append(alias)
return dict(out)4. “We promoted the model but nothing changed”
Aliases are the right abstraction. Your app loads models:/churn@champion, deploy moves the alias, no redeploy. Beautiful.
Except load_model resolves the alias once, at load time. A service that booted a month ago is still serving what was champion a month ago.
I wrote an example that demonstrates it end to end, because reading it isn’t the same as seeing it:
booted with champion = v1
predictions: [0 0 0 0 0]
...deployment moves champion to v3...
registry says champion = v3
the naive service still predicts: [0 0 0 0 0]Registry says v3. Service serves v1. Nothing errored, nothing warned. This is the mechanism behind every “the promotion didn’t take” ticket.
The fix is a model holder that re-resolves on an interval — and one detail that matters more than it looks:
wanted = version_for_alias(client, name, alias) # cheap metadata call
if wanted != self.current.version:
# Load by VERSION, not by alias. The alias could move between resolving
# and loading, and then your logs claim a version you aren't running.
model = mlflow.pyfunc.load_model(f"models:/{name}/{wanted}")Resolve the alias, then load the number. Otherwise your observability is subtly lying about what’s in memory.
5. My own assertion caught a bug in my own example
I built a pyfunc that wraps the model with business rules — the shape real inference actually takes:
- never target a customer on a two-year contract (contractually locked in)
- always target a customer with 5+ support calls (escalation)
- clamp probabilities to
[0.01, 0.99]
I wrote each rule, then wrote a verification line: “two-year customers targeted: {n} (must be 0)”.
62 two-year customers in this batch
targeted: 2 (must be 0)Rule 2 ran after rule 1 and overrode it. The two rules contradict each other on the overlap — customers who are both locked in and escalated — and I’d never decided which wins. Whichever I happened to write last won by accident.
The fix isn’t code, it’s a decision, written down:
# Rule PRECEDENCE is a business decision, and it has to be explicit.
# Order here, lowest priority first, so the last rule applied wins:
# 1. inclusion : support escalation forces targeting
# 2. exclusion : locked-in contracts are never targeted <- final sayEvery rules engine I’ve seen in production has this bug somewhere. It survives because nobody writes the assertion that would expose it.
The rest, briefly
| Gotcha | Consequence |
|---|---|
./mlruns is in maintenance mode in 3.15+ and raises; the default is now sqlite:///<cwd>/mlflow.db | every “just run it and check ./mlruns” tutorial is out of date — and the new default is relative to your working directory, so running the same script from two folders gives you two databases and a mysteriously empty UI |
| Param values over 6000 chars are silently truncated | your config blob is “in” the UI with the tail quietly gone |
| sklearn models now serialise with skops, not pickle | an ordinary Pipeline(ColumnTransformer(...)) fails to log until you pass skops_trusted_types=["numpy.dtype"] |
mlflow.models.evaluate scores at the model’s implicit 0.5 cut-off | if you serve at 0.45, the precision you gated on is not the precision you’ll observe |
eval_results_table only appears if some metric emits per-row scores | your error-analysis artifact silently doesn’t exist |
custom_artifacts functions receive only prediction and target | not your features — so half the plots you’d want to write there can’t be written |
autolog’s max_tuning_runs defaults to 5 | your 24-candidate grid search logs 5 children and you never notice |
What this changed about how I structure ML projects
The bugs were the fun part. The durable part is what kept recurring across twenty-one phases:
Failures that raise are cheap. Failures that return plausible numbers are expensive. Every single item above was in the second category. Not one of them threw an exception. That reframes what defensive code is for: not to prevent crashes, but to convert silent wrongness into loud wrongness.
Preprocessing belongs inside the model artifact. Pipeline(preprocess → estimator) logged as one object. The moment imputation lives in a separate script, whoever serves the model has to reimplement it, and training/serving skew is a matter of time.
Separate the three verbs: train, evaluate, promote. My training script never touches the champion alias. It registers a version tagged validation_status=pending. A separate gate decides, a separate script acts. That separation is what makes it safe to run the pipeline automatically — and it’s why evaluate.py exiting non-zero is the CI integration, with no log-parsing glue.
Aggregate metrics hide the failure that gets you paged. A model at 0.81 overall was at 0.74 for month-to-month customers — the exact segment the retention campaign targets. The slice table takes twenty lines to write and should be in every evaluation run.
Pin last-known-good during the promotion, not during the incident. MLflow keeps no alias history. At 3 a.m. you want a pointer, not a forensic reconstruction of what used to be champion.
What I did not verify
Since this post is partly an argument for running your examples, it would be poor form to hide the ones I didn’t.
Phases 1–14, 17–19, 21 and the capstone ran end to end and I read every output. The churn project’s 28 tests pass (10 skip without a running server, by design). The local CI pipeline runs 7 stages in ~35 seconds.
Not executed: the Docker Compose stack (Postgres + MinIO — the compose file validates, but I never brought the daemon up), the model-serving scripts, and the GitHub Actions workflows. Those READMEs say so at the top. The repo also isn’t a git repo yet, which is why the capstone self-check honestly reports 7/8 on “answer these questions without asking a human” — the missing one is “which commit produced this model?”
That last detail is a nice accident. The tooling correctly refuses to claim lineage it doesn’t have.
If you’re learning MLflow
Three things I’d tell myself at the start:
- Pick your tracking URI explicitly, on day one. Half of all “my runs disappeared” confusion is a working-directory-relative default.
- Learn signatures before you learn the registry. A model without a signature is a model without an API, and phase 7 is where the production failure modes actually live.
- Type the examples and read the output. Not the code — the output. Six times in twenty-one phases, the output disagreed with what I’d written directly above it.
The repo is structured so you can do exactly that: make server, then make phase1 through make phase21, each phase a folder with its own README, its own gotchas, and three “check yourself” questions that are harder than they look.
Verified against MLflow 3.15.1, scikit-learn 1.9, Python 3.12.
If you’re reading this on a later MLflow version and something contradicts it: trust the docs, re-run the example, and let me know which one moved.
Find the repo inside my github: https://github.com/kmerkuri/mlops-learning
