The Best Things I Found, I Wasn't Looking For

A surprising number of the most valuable things I took from this project, I was not looking for when I found them. They did not arrive as the reward for a deliberate search. They arrived sideways — out of a bug I was annoyed by, a failed experiment, an odd result I very nearly dismissed. The things I planned to find, I mostly found, more or less as I had pictured them. But the things that genuinely changed how I understood the problem tended to be things I had not known existed until they ambushed me.

This took a while to notice, because it runs against the story I liked to tell about my own work — that progress came from my intelligent, intentional searching. A lot of it came from something much closer to paying attention when reality handed me something I had not asked for.

A search can only find what you aimed at

A deliberate search is a powerful thing, but it has a specific and absolute limitation: it can only find what you already conceived of clearly enough to go looking for. You define a target, you orient yourself toward it, you go and get it. That is exactly what you want when you know what you are after, and it is necessary, and I am not knocking it.

But by its very construction, a directed search cannot surface the things you did not know to seek. They are not in the space you are searching, because you did not put them there — you could not have, because you did not know about them. And in a genuinely hard, unfamiliar problem, the most important things are very often precisely there: outside your current frame, in the region you have no reason to point your search at, invisible to any amount of diligent looking-for-what-you-expect.

The bug that turned out to be telling me something true

More than once, a bug that I was simply trying to make go away turned out, when I finally stopped and actually investigated it, to be revealing something true about the problem that I would never have thought to ask about. I had been treating it as an annoyance, an obstacle between me and my plan. It was, underneath, reality disagreeing with one of my assumptions — and the disagreement was information of exactly the kind I most needed and least expected.

This reframed how I thought about the irritating, inexplicable results. A thing that behaves in a way your model says it should not is not merely broken. It is a message: it means your model is wrong somewhere, and the misbehavior is pointing, however cryptically, at where. The thing I was trying to fix was, on closer inspection, a thing I needed to learn. I just had to be willing to read the bug as a teacher rather than only as a defect.

The failure that taught more than a success would have

There is a related asymmetry I came to rely on. When a result comes out exactly as you expected, you have confirmed what you already believed, and you have learned almost nothing — you knew that already, and now you know it slightly more confidently. When a result comes out wrong, surprising, flatly contrary to what you predicted, that is where the actual learning lives. Because a surprise means one of your beliefs was false, and now you get the chance to discover which one.

So the failed experiments and the baffling outcomes consistently carried more information than the clean confirmations, precisely because they contradicted me. A confirmation tells you that you and reality agree, which is pleasant and nearly content-free. A contradiction tells you that you and reality disagree, which is uncomfortable and enormously informative, because it is the only signal that can actually move you off a wrong belief you did not know you held.

The real danger is dismissing the weird

Here is the trap that nearly cost me most of these gifts. The unexpected result is desperately easy to wave away. It looks like noise. It looks like a mistake. It looks like one more thing to suppress so you can get back to the plan you were executing. Every instinct, especially when you are busy and focused, is to make the strange thing disappear and move on.

But the strange thing is often the entire message. The discipline I had to build, slowly and against my nature, was to pause on the anomaly instead of erasing it — to ask “what if this is actually real?” before assuming it was an error to be cleaned up. Most anomalies, examined, do turn out to be nothing. But the ones that are not nothing are wildly disproportionately valuable, and you only get them if you are in the habit of looking before you dismiss. The cost of investigating a false alarm is small. The cost of erasing a real signal because it was inconvenient is enormous and invisible.

You cannot plan serendipity, but you can make room for it

You cannot schedule an accidental discovery. That is what makes it accidental. But I came to believe you can absolutely create the conditions in which these things are more likely to happen, and cultivate the alertness to catch them when they do.

It meant leaving some slack rather than optimizing every single moment toward the immediate goal, because a fully optimized process has no room for a detour into something unexpected. It meant actually looking at things — at the full texture of what was happening — rather than only at the one number I was chasing, because the gift is never in the metric you are staring at. It meant staying curious about the side observations, the things that happened while I was busy doing something else. The accidents cannot be summoned, but they can be starved or fed, and a process built purely for ruthless efficiency tends to starve them.

The humility in all of it

There is a genuinely humbling lesson buried in this, about the limits of planning and intelligence. I wanted to believe my progress was the product of my deliberate, clever search — that I was advancing by being smart about where to look. A large share of it, on honest reflection, came instead from being present and paying attention when reality offered me something I had not requested and could not have requested.

The world, it turned out, knew a great many things I did not. And it kept trying to tell me, through bugs and surprises and results that refused to behave. My most important job, much of the time, was not to be cleverer about my search. It was simply to be listening — to not be so busy executing my plan that I failed to hear the far more interesting thing reality was saying underneath it.

The deeper lesson: stay alert, not just on-task

What I ultimately concluded is that the most valuable working posture is not pure, relentless focus on the plan. It is focus held loosely enough that you still notice when something unexpected and important wanders into the edge of your view. Being perfectly on-task makes you efficient at obtaining exactly what you aimed for — and blind to everything you did not aim for, which is where the things you did not know to want are quietly waiting.

So I try, now, to hold the plan firmly enough to make progress and loosely enough to be interrupted by something better. The deliberate search gets me what I know to chase. Staying alert is what lets reality hand me the things I never would have thought to chase at all — and those, over and over, turned out to be the ones that mattered most.

— No signals, no returns, not investment advice.