Nobody would say the camera is lying

12 Aug, 2026

For several years at Srishti I facilitated courses where students wrote generative code. p5.js, small algorithmic systems that turned a few numbers into a pattern on screen. Some students came in expecting the code to be the hard part. It never was. Most of the design happened before anything ran. You'd sit with a student and ask: out of everything this system could control, what are you going to expose as a parameter, and what are you going to bury as a fixed constant? That question among others decided the whole piece. Expose the angle and hide the colour, and you get one kind of artwork. Do it the other way and you get something nobody would recognise as the same system. Students rarely wrote those decisions down anywhere. The choices sank into the finished piece, and the piece hung there looking inevitable, every version it beat out invisible behind it.¹

I didn't think I'd run into that same problem again in a materials mechanics paper about honeycomb structures.

Before the honeycombs, a simpler example, since it might make this easier to see. Say you photograph a sunset. You pick an aperture, which decides how much of the scene stays in focus. You pick an exposure, which decides whether the clouds keep their detail or blow out to white. You crop afterward, which decides what's even in the frame to begin with. None of these choices are neutral. Two photographers standing in the same spot, same sunset, can hand you two completely different photos, and both might call theirs "accurate."

Nobody would say the camera is lying. The photographer made choices, and those choices are the photo. The trouble starts when you forget that a photo was ever a choice at all, and start treating it as the sunset itself.²

That's basically the whole argument of the paper Dr. R. S. Dhari and I just published.³ Just replace "sunset" with "strain field from a simulation," and "aperture" with "the threshold that decides what counts as a plastic hinge."

Two years earlier we had built a pipeline⁴ to classify how honeycomb auxetic structures deform under load. The structures, and the six named deformation modes they can show, came out of Dr. R. S. Dhari's earlier work with Dr. Zia Javanbakht.⁵ The pipeline itself isn't complicated. A finite element simulation produces a field of plastic strain. That field gets converted to grayscale, thresholded into a binary image, cropped, resized, and clustered so a model has something to learn from.

None of these steps look like research decisions. They look like preprocessing, the boring part before the "real" work starts. That's the whole problem, really.

Take the threshold. Somewhere in the pipeline we take the grayscale image and pick a value, τ. Anything above τ gets called a plastic hinge. Anything below becomes background. In the original paper, τ was 200. Not because we'd tested it against other values, just because 200 sat "towards the higher end of the grayscale range," which is a sentence and not really a reason.

Once that line is drawn, it decides everything after it. A pixel carrying 10% strain and a pixel carrying 80% strain both become the same white dot: "slightly bent" and "about to snap", filed in the same drawer. Whatever gradient might have told you a hinge is just starting to form, versus already collapsed, is gone before the model even sees the image.

So this time we went back and swept it properly. We tried τ across {120, 150, 175, 200, 225, 250, 280}, ran the full pipeline each time, and compared the resulting clusters to the τ = 200 version using the adjusted Rand index (ARI), which measures how much two groupings of the same data agree once you correct for the agreement you'd expect from pure chance. Two people sorting the same box of photographs into piles, scored pair by pair. A value of 1 means the two groupings are identical; 0 means chance-level agreement, and it can dip slightly below zero when two groupings agree even less than chance would.

What we found was a plateau, then a cliff. Between τ = 200 and τ = 250, ARI held at 1.00. Nothing moved. Then at τ = 280, accuracy collapsed to 27%, down from around 70% across the whole plateau. τ = 200 had been standing on a wide, flat ledge the whole time; nobody had ever walked over to see where the edge was. This is the accuracy of the audit's own reproduction of the pipeline, not the 88.08% headline number from the original paper. Those are two different runs measuring two different things, which is exactly the kind of mix-up this whole framework is trying to prevent, so it's worth being precise about it here too.

There's a quieter number worth mentioning. How much of the original grayscale signal even survives the binary cut? You can measure this directly with mutual information, I(binary; grayscale), which tells you how many bits about the grayscale image you can still recover from the binary one. Across the whole dataset it comes out to about 0.03 bits per pixel, against a ceiling of 1 bit (a binary variable can't hold more, by definition). Of everything the grayscale image knew, the binary version remembers about 3%. We kept the book by tearing out a note that says it was mostly about a whale. That can be a completely fine thing to do. But it's a claim about the physics, and it should get treated like one, not folded quietly into a preprocessing script.

This isn't a problem specific to honeycombs, either. Fischer et al. looked at powder bed monitoring in metal 3D printing and found a near-perfect correlation between image resolution and model accuracy across a 16-fold resolution range, along with a 12-percentage-point gap in recall between two different lighting setups on the same physical process.⁶ Stuckner, Harder and Smith found that switching the source of a representation, from generic pretraining to a domain-specific microscopy dataset, cut relative error by over 70% in low-data settings.⁷ Neither of these is a modelling change. Both are representation changes, and both moved the result more than most architecture choices would have.

The finding that stung a little more was about our own paper's headline number. We'd reported 88.08% test accuracy. Real result, published, and it held up. What we hadn't said clearly enough at the time is: 88% accuracy against what, exactly? The labels weren't hand-annotated deformation modes. They were K-means clusters, eight groups an algorithm found in a 4096-dimensional pixel space (one dimension per pixel, every frame a single point in it), which we had quietly assumed lined up with the six named physical modes from Dr. R. S. Dhari's earlier work. Assumed, not checked.

So this time we hand-labelled 82 frames ourselves, spread across different loading angles and strain levels, and actually checked. We used two agreement measures rather than one. Normalized mutual information (NMI) asks how much knowing a frame's cluster tells you about its true physical mode. ARI, again, corrects for the agreement you'd get by chance, which for both of these measures sits near zero when two labellings have nothing real in common. Both numbers came back modest: NMI = 0.56, ARI = 0.30. Well above chance, but nowhere close to the near-1.00 you'd want before calling "cluster" and "physical mode" the same thing.

In practice, the 'X' deformation mode (two diagonal hinge arms crossing) got split across two different clusters. A distorted version of it split across three more. Two of the six known modes weren't detected as their own cluster at all. The 88% was real. It just measured agreement with what the clustering algorithm decided, not agreement with the physical reality Dr. R. S. Dhari had carefully worked out in a separate paper. Two different claims, sitting behind one number.

None of this makes the original result wrong, to be clear. It just means it was a narrower claim than we realised we were making at the time.

We wanted to know if this was a habit particular to us, or something more widespread. So we reviewed 28 published studies across materials mechanics and computational materials science, covering microstructure classification, fracture prediction, defect detection in 3D printed parts, and a handful of other areas. Every one of those studies told you what representation it used. Every one gave the parameter values. That part of the culture is in decent shape.

Only six of the 28, about 21%, had ever swapped the representation for a different one and reported what changed. I don't think this is because the other 22 were careless, or lazy, or cutting corners. It's that a representation choice doesn't feel like a decision while you're making it. It feels like formatting. It's the paragraph you write in passive voice, three days after the actual model results are sitting in a spreadsheet and everyone's already moved on to the discussion section.

This lines up with what other people have already been saying about ML pipelines more broadly, just not usually about this exact layer. Sculley et al. described how upstream changes in a pipeline cascade through the whole system in ways nobody fully tracks, what they called the CACE principle: changing anything changes everything.⁸ Sambasivan et al. interviewed practitioners across high-stakes AI projects and found that data and representation work is treated as lower status and gets fewer resources than model work, even when it's the thing actually driving the result.⁹ Our review just gives that observation a number, in one specific corner of engineering ML.

Deep learning has spent a decade getting extremely good at representations the network learns for itself, in the sense Bengio, Courville and Vincent meant when they framed representation learning as ML's central challenge.¹⁰ Almost nothing exists for the representations a person builds by hand, with a threshold slider and a crop tool, before any of that learning even starts. Mitchell showed back in 1980 that any model capable of generalizing at all has to carry some inductive bias, some built-in assumption about what counts as a reasonable answer.¹¹ Battaglia et al. catalogued what those biases look like for architectures: CNNs assume nearby pixels are related, RNNs assume order matters, graph networks assume relationships are the point.¹² What we're arguing is that the same kind of bias gets baked in one step earlier, into the data itself, by someone who never had to defend the choice the way an architecture choice gets defended. Facilitating generative code for several years was largely about making that kind of hand-built decision, what to expose and what to fix, visible and defensible to an eighteen-year-old. It's strange to find applied machine learning, as a field, still treating the equivalent decision mostly as plumbing.

The paper doesn't claim to have solved this. It ends with a documentation format we're calling a representation datasheet, which sits next to the existing datasheets for datasets¹³ and model cards for models,¹⁴ and covers the layer nobody had named: the point where a physical field turns into a picture, and someone should be able to answer for how. And it ends with ten audit questions. In the paper each one comes with a metric you can actually compute. In plain words, they ask:

1. What quantity is the picture actually showing, and what did the drawing throw away?
2. Is the encoding faithful to the data, or is it adding artefacts of its own? Wiggle the threshold and watch what moves.
3. Could a different encoding reveal structure this one hides?
4. What does the choice of crop and window assume about where the interesting physics lives?
5. Is the resolution enough for the phenomena you claim to detect?
6. What symmetries does your augmentation assume the physics has?
7. Does augmentation actually improve generalisation, or just shift the distribution?
8. Where did the labels come from: the world, or an algorithm?
9. If you changed the method that generates the labels, how much would the labels change?
10. Which different things do your labels file in the same drawer, and which single thing do they split across two?

If you never touch a pipeline, the takeaway is shorter. A benchmark, a dashboard figure, an 88%: every number that reaches you was made the way the photo was made. Somebody set the aperture, somebody left the threshold at 200 because it looked about right. So ask the one question I wish we had asked ourselves two years ago. What got buried to make this number?

If you work with simulation images, microstructure data, or any pipeline where a physical field gets turned into something a model can see, I'd genuinely like you to read it and run the ten questions above against your own pipeline. Even a rough pass tends to surface something. And if any part of it ends up useful in your own work, a citation goes a long way for a paper like this one, still finding its readers.

@article{singh2026vml,
  title   = {A framework for visual-mathematical literacy in applied machine learning: why representation choices shape what models can learn},
  author  = {Singh, Gaurav and Dhari, Rahul Singh},
  journal = {Machine Learning: Engineering},
  volume  = {2},
  pages   = {025004},
  year    = {2026},
  doi     = {10.1088/3049-4761/ae7df3}
}

#auxetics #machine-learning #representation #srishti