<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Inferentia: Research documentation]]></title><description><![CDATA[Project documentation and research standards, temporarily excluded from the homepage.]]></description><link>https://www.inferentia.in/s/research-documentation</link><image><url>https://substackcdn.com/image/fetch/$s_!Ydl9!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20523561-1faf-45ef-81af-96c7aeb07db6_1120x1120.png</url><title>Inferentia: Research documentation</title><link>https://www.inferentia.in/s/research-documentation</link></image><generator>Substack</generator><lastBuildDate>Tue, 17 Nov 2026 19:59:12 GMT</lastBuildDate><atom:link href="https://www.inferentia.in/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Piyush Ranjan]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[inferentia@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[inferentia@substack.com]]></itunes:email><itunes:name><![CDATA[Piyush Ranjan]]></itunes:name></itunes:owner><itunes:author><![CDATA[Piyush Ranjan]]></itunes:author><googleplay:owner><![CDATA[inferentia@substack.com]]></googleplay:owner><googleplay:email><![CDATA[inferentia@substack.com]]></googleplay:email><googleplay:author><![CDATA[Piyush Ranjan]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[AI shopping reliability: the backpack pilot]]></title><description><![CDATA[Can we check whether recommendations meet shoppers' requirements?]]></description><link>https://www.inferentia.in/p/ai-shopping-reliability-the-backpack</link><guid isPermaLink="false">https://www.inferentia.in/p/ai-shopping-reliability-the-backpack</guid><dc:creator><![CDATA[Piyush Ranjan]]></dc:creator><pubDate>Sat, 03 Oct 2026 06:55:24 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Ydl9!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20523561-1faf-45ef-81af-96c7aeb07db6_1120x1120.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Exploratory pilot &#183; not pre-registered &#183; AI-assisted checks; independent human review pending &#183; page updated 3 October 2026</p><p>Observations described here were collected on 28 September 2026. Updating this page does not mean new observations have been collected.</p><h2>The question</h2><p>Do AI shopping recommendations satisfy shoppers&#8217; stated requirements, and can that be judged reliably?</p><p>I am starting with backpacks sold online in India. Before counting errors or comparing systems, I want to establish whether a reviewer can identify the exact recommended product, find relevant evidence and make a defensible judgment.</p><h2>Why it matters</h2><p>A recommendation can name a product while leaving a shopper unable to confirm its specifications, price or purchase route. For a commerce team, the practical question is which information problems can be verified and which might justify a correction.</p><p>The pilot will inform a first decision: whether this review method is feasible enough to extend, needs a narrower question, or should change direction. It does not yet establish demand for an audit service, the effectiveness of a merchant correction, or an effect on sales.</p><h2>Current scope</h2><p>The planned design has five researcher-written English prompts, two consumer search surfaces and two runs per prompt: 20 answers in total. The surfaces are ChatGPT Search and Perplexity Search. These are buying tasks designed for a feasibility pilot, not a representative sample of Indian shoppers.</p><p>Each prompt asks for up to three specific products, a current listed price and a purchase link. Conditional bank or membership discounts are excluded. The five requirement families are:</p><ul><li><p>Laptop fit: below &#8377;4,000 and compatible with a 15.6-inch laptop.</p></li><li><p>Capacity: below &#8377;3,000 and a stated capacity of at least 25 litres.</p></li><li><p>Weight: below &#8377;4,000 and manufacturer-published product weight no more than 800 grams.</p></li><li><p>Warranty: below &#8377;5,000 and a manufacturer warranty of at least one year applicable in India.</p></li><li><p>Dimensions: below &#8377;4,000 and manufacturer-listed external dimensions no greater than 45 cm high, 35 cm wide and 20 cm deep. This does not establish airline approval.</p></li></ul><p>Every prompt also requires availability for online purchase in India. Availability is checked for the recommended merchant; it does not establish delivery to every postcode.</p><p>Currently, three Perplexity answers have been saved, containing nine product recommendations. Independent human review has not been completed for any answer. One further prompt was submitted but its answer could not be recovered; another has not been submitted. The planned ChatGPT Search observations have not been collected.</p><h2>What exists</h2><p>The working materials include:</p><ul><li><p>A versioned protocol and five fixed prompts.</p></li><li><p>A requirement rubric separating supported constraints, constraint violations, insufficient evidence and pending review.</p></li><li><p>Saved answer transcripts, collection records and dated verification excerpts.</p></li><li><p>Provisional review records, including unresolved judgments and correction history.</p></li><li><p>Code for capturing answers, protecting records from accidental overwrite, checking review completeness and generating descriptive reports.</p></li></ul><p>The saved answers are transcripts from consumer interfaces, not native exports or API responses. Records include the surface, displayed mode or model label and collection timing. An exact underlying model version is not assumed when the interface does not disclose it.</p><p>Public repository links: pending. The working materials exist locally, but this page does not claim they have already been publicly released. Reviewer-agreement calculations have not yet been implemented or completed.</p><h2>What we have learned so far</h2><p>These are preliminary lessons about the review process, not validated estimates of platform accuracy:</p><ul><li><p>Exact product and variant matching matters before a specification can support a judgment.</p></li><li><p>A price or stock page checked after an answer establishes its state at verification. It may not establish its state when the answer was generated.</p></li><li><p>Missing evidence does not demonstrate that a product claim is false. An unverifiable claim needs a different label from an established violation.</p></li><li><p>Collection and source-access failures must remain visible. They must not be turned into apparent recommendation errors or silently replaced with different interfaces.</p></li></ul><p>The current records are insufficient for an accuracy estimate, platform ranking or causal claim. The original provisional annotations also include unresolved timing interpretations and are not pooled into a headline result.</p><h2>Open questions</h2><p>Can another person apply the requirement rubric consistently? Which ambiguous cases need clearer rules? How much active verification effort does each answer require? Would a commerce practitioner identify a plausible action from the evidence?</p><p>A later question is how often a brand appears in sampled answers, and how many prompts, runs and days would be needed to detect a meaningful change. That is a separate measurement problem. Visibility alone would not establish recommendation quality, customer reach or incremental sales.</p><h2>When this is wrong</h2><p>The conclusions would be too strong if they treated these five prompts as representative of shoppers, counted repeated answers or product checks as independent people, or mistook later stock information for generation-time truth.</p><p>Manufacturer statements establish published specifications, not independently tested product performance. Merchant offers do not establish nationwide availability. The prompts request sources and explicit uncertainty, so findings apply to that wording.</p><p>The current review is AI-assisted and provisional. Independent human review, reviewer agreement, generation-time stock and price, some exact variant matches, and per-answer verification effort remain unresolved. Account context, location and underlying model versions are incompletely known. None of this pilot establishes the cause of an answer, the effect of changing a merchant page, or business value.</p><h2>Change log</h2><ul><li><p>28 September 2026: protocol v0.1 and the working capture/review process were documented. The pilot remains exploratory and is not registered as a confirmatory study.</p></li><li><p>2 October 2026: this project introduction was prepared from the existing records.</p></li><li><p>3 October 2026: the project page was published. No additional observations or completed human reviews were added.</p></li></ul><p>The next checkpoint is a five-answer evidence table and a decision about whether to continue, narrow the question or change direction. Expansion depends on that checkpoint.</p><h2>Background</h2><p>This project follows earlier work on the limits of AI-assisted conversion optimisation: <a href="https://www.inferentia.in/p/i-built-an-ai-cro-system-heres-where">I Built an AI CRO System. Here&#8217;s Where It Failed</a>. That article is retained as background, separate from the current pilot&#8217;s evidence.</p><p>For the lab&#8217;s documentation and review policies, see <a href="https://www.inferentia.in/p/research-standards">Research Standards</a>.</p>]]></content:encoded></item><item><title><![CDATA[Research Standards]]></title><description><![CDATA[How Inferentia labels, checks and corrects its work.]]></description><link>https://www.inferentia.in/p/research-standards</link><guid isPermaLink="false">https://www.inferentia.in/p/research-standards</guid><dc:creator><![CDATA[Piyush Ranjan]]></dc:creator><pubDate>Sat, 03 Oct 2026 06:54:20 GMT</pubDate><enclosure url="https://substackcdn.com/image/fetch/$s_!Ydl9!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20523561-1faf-45ef-81af-96c7aeb07db6_1120x1120.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Last updated 3 October 2026. The current AI commerce reliability project is an exploratory pilot. Its checks are AI-assisted and provisional; independent human review is pending.</p><h2>Exploratory and pre-registered work</h2><p>Each project states its research question, design, evidence status and review status. Exploratory work can develop a question or method without being presented as a confirmatory test.</p><p>A pre-registration records a dated question, design and analysis plan before the relevant evidence is examined. Publishing a protocol after looking at results does not make those results pre-registered. An exploratory pilot may inform a later test using fresh or genuinely held-out evidence. The pilot keeps its exploratory label.</p><p>Research updates and methods explainers are labelled by their purpose. They do not become confirmatory research simply because they are published here.</p><h2>Amendments</h2><p>Changes to a research question, prompt set, rubric or analysis are recorded with a date, a reason and the affected version. Changes are made before collecting or analysing the observations to which the revised rule will apply.</p><p>Original observations are preserved. A new rule does not silently rewrite what was collected under an earlier design. The project page links to the relevant protocol and change history when those materials are publicly released.</p><h2>Review</h2><p>Review credits state who checked the work and what they checked: the code, evidence, rubric, annotations or conclusions. Those roles are distinct, and a review of one part does not imply approval of everything.</p><p>AI assistance is disclosed. Software completeness checks and AI-assisted judgments do not establish that an annotation is true and do not count as independent human validation.</p><p>For the current pilot, independent human review has not been completed. Its protocol calls for a second person to review at least four complete answers across surfaces and difficult labels before the pilot&#8217;s results are presented as validated.</p><h2>Corrections</h2><p>Errors in an explanation or annotation are corrected with a dated record of what changed and why. Raw responses remain unchanged. A correction preserves the distinction between the original observation and a revised interpretation.</p><p>Material changes to a conclusion are noted on the project page and linked to the underlying correction history. The repository change log, once released, is the authoritative record; the project page carries a dated summary.</p><h2>Disagreements</h2><p>Substantive methodological objections and unresolved reviewer disagreements are documented alongside the relevant work. Responses explain the evidence and reasoning rather than treating agreement as a condition for publication.</p><p>Outside contributors and reviewers are credited for their actual contribution. Private correspondence is attributed only with permission.</p><h2>Simulation</h2><p>Simulations are used where they fit the question, to examine how a method behaves when the generating process is known. Their assumptions and sensitivity to alternative assumptions are stated.</p><p>A result from simulated commerce data is a result about that simulation. It does not establish what happened in a real business. Simulated observations are clearly labelled and kept separate from real observations.</p><h2>Data</h2><p>Current research uses public, purpose-collected, synthetic or consented data. It uses no data from any current or former employer.</p><p>Data records identify their source, collection timing, scope, missingness and limitations. Public sharing respects the applicable consent and rights. A project&#8217;s materials are explicitly marked as available, withheld or not yet released.</p><p>Earlier practitioner articles are retained in Measurement Guides and CRO Lab. They were written before these standards and are not presented as research validated under them.</p><h2>What a completed study includes</h2><p>A completed study states its question, method, findings and limitations, with review credits and a section titled &#8220;When this is wrong.&#8221; It links to its registration where applicable and to publicly released code, evidence and data where available. Missing materials or unfinished checks are stated explicitly.The first project, <a href="https://www.inferentia.in/p/ai-shopping-reliability-the-backpack">AI shopping reliability: the backpack pilot</a>, asks whether AI shopping recommendations meet shoppers&#8217; requirements and can be judged reliably.</p>]]></content:encoded></item></channel></rss>