Choosing the metric is choosing the mistake

The model that looked better

One problem was hourly air quality in the Monterrey metropolitan area: predict the hours when fine particulate goes over the legal limit. Those hours are the minority, a bit under one in five.

Trained without class weights, the random forest had the better accuracy of the two versions. It also let 92% of the pollution events go by unflagged. It was accurate because it learned to say "the air is fine", which is true most of the time and useless exactly when it matters.

Accuracy rewards whoever agrees with the majority. When the expensive case is the rare one, it is the wrong number to optimise, and it will not tell you so.

The average that hides the class

Another was classifying skin lesions from images with PyTorch. The model reached 75% accuracy on the test set, which reads well until you open the per-class report: on dermatofibroma, 17 test images, the F1 score was zero. The model never got one right.

Same model, same predictions, two honest summaries. The weighted F1 was 0.74 and the macro F1 was 0.48. The first answers "how often is it right"; the second answers "is it right for every kind of case". Which one goes on the slide is a decision about who you are willing to fail, and it deserves to be made out loud.

The baseline that almost won

The reinforcement learning problem was assigning gifts to a group of people from a limited inventory. The trained agent reached 96% of the best possible allocation. That sounds like a result until you put next to it a greedy rule that anyone could write in ten lines: 94%.

With five people the gap was three tenths of a point. It only opened up with eight people and scarce stock, when one early choice starts to constrain the rest. That is the real finding: the agent is worth its complexity in some conditions and not in others, and without the simple baseline nobody would have known which.

This is the same argument as the control group in note 02. A number with nothing to compare against is not a result yet.

The number that did not pass

The last one is the one I am proudest of. A convolutional network trained from scratch to classify aerial threats reached 0.49 accuracy. The rubric asked for 0.70. The pretrained model did get there, at 0.71.

Getting the scratch model over the line was a matter of reshuffling the sample until it looked better. I delivered it at 0.49, with the reason it fell short. A score tuned to pass the test measures the tuning, not the model, and that habit is much harder to unlearn once there is real money on the other side.

What changes when an agent acts on it

In a notebook, a bad metric produces a misleading chart. In production, an agent that acts on a metric inherits every blind spot that metric has, and it acts on them at scale, every cycle, without asking.

So before any agent of mine touches a real decision, four things are written down. The primary metric, declared before the first result. The mistake it is allowed to make, stated in business terms: an idle store or an empty shelf, a missed alert or a false alarm. A deliberately simple baseline it has to beat. And the result broken down by segment, because the average is where the class that matters goes to hide.