Inferentia
Practical writing on measuring AI systems and the decisions built on them, for teams running AI in commerce.
Most teams can ship an LLM feature in weeks. Far fewer can say whether it’s working, whether it’s getting worse, or whether last week’s change actually helped. Inferentia is about closing that gap: evaluating AI output, monitoring it once it’s live, and keeping the measurement honest enough to make decisions on.
What I write about
AI evals: how to build evaluations you can trust. That covers designing test sets, using LLMs as graders without inheriting their mistakes, how many examples you need before a difference is real, and when a “better” score isn’t better.
AI observability: what to watch once a model is in production. That means tracing, drift, failure patterns, and catching quality problems before users or stakeholders do.
Measurement and experimentation: tracking, metrics, experiment design, and diagnosing results. This is the statistical groundwork that AI evaluation depends on.
Conversion and UX: earlier writing on CRO, product pages and using AI for conversion work.
Who I am
I’m Piyush Ranjan, based in Bengaluru. I’ve spent about ten years in analytics and growth across Flipkart, Paytm, PharmEasy and a D2C brand. I’ve built and run a production AI analytics system, so most of what I write comes from problems I’ve had to solve on live systems rather than in demos.
Start here
AI Evals: grader errors, eval sample sizes, and building evals that hold up
Measurement Guides: tracking, metrics, experiments and diagnosis
CRO Lab: conversion optimisation, UX and AI for CRO
Contact
If you’re working on AI evals or observability and want to compare notes, reach me on LinkedIn.

