Reframing the target
"Sounds AI-generated" is not synonymous with "bad writing," yet boring AI prose shares distinct pathologies. To ground the target in actual reading experience, I selected 30 essays in my domain, built a custom pairwise comparison tool, and completed all 435 head-to-head judgments myself—yielding a ground-truth ranking of what holds my attention.
Testing signals instead of relying on vibes
With a ranking in hand, I tested candidate linguistic features. Mean token-level surprisal proved useless because averaging flattened rhythmic contrast; switching to the variance of token-level surprisal immediately yielded a 0.825 rank correlation, confirming that flat pacing is a primary culprit. After testing semantic trajectory and logical graph structures—and pruning redundant metrics—the combined interpretable signals reached a Spearman correlation of 0.899. I then wired these diagnostics into my writing agent's self-check loop.
Who this ruler is for
This is a calibrated personal-taste instrument, not a universal AI detector. It rests on 30 tech essays and a single annotator, and rank correlation is not classification accuracy. Adapting it to other writers or genres requires fresh calibration. While it now serves as a practical guardrail in my own workflow, I have not yet run controlled studies to quantify its downstream editing gains.