Methodology

Every experiment starts with a question. Where that question derives from is a matter of where I'm currently at in life. Sometimes, I'll scroll through social media and see polarizing opinions on whether AI could be used to solve a human painpoint. But more often, I reflect on the ebs and flows of my own life and think to myself: "I wonder if AI could fix this?"

Either way, once curiousity strikes, an experiment is born!

The Experiment Process

I don't go experimenting all willy nilly, this is a civilized lab after all. There are hypotheses, data collection, analysis, and even lab notes. But in case you'd like a look into my process, well, here it is.

  1. The Question

    An experiment begins with a specific, testable question — never a thesis I'm trying to prove. "What happens if I give an AI agent access to my credit card for a week?" is a question. "AI agents can't be trusted with money." is an opinion. I start with the former.

  2. Methodology Design

    Before collecting a single data point, I publish the methodology. What I'm measuring, how I'm measuring it, what tools I'm using, and what would constitute a meaningful result. This gets published on the experiment page immediately and you can critique the approach before results arrive.

  3. Data Collection

    I run the tests. This phase can take days, weeks or months depending on the experiment. Throughout, the timeline on each experiment page updates with progress notes, obstacles encountered, and any methodology adjustments (always documented and justified).

  4. Analysis

    Raw data is processed, patterns identified, and statistical significance assessed. I don't publish percentages without context. Every number comes with sample size, confidence interval, and a plain-language explanation of what it means and doesn't mean.

  5. Finding

    A finding is a discrete, quotable insight, essentially one sentence that captures what I learned. Every finding carries a confidence label: preliminary (early data, pattern emerging), confirmed (replicated, statistically significant), or contested (challenged by new evidence or peer review).

  6. Publication

    The full experiment (i.e. methodology, data, analysis, findings, limitations) is published as a long-form piece. Raw datasets are made available. Replication instructions are included. The experiment moves from "active" to "published" status, but can be reopened if new evidence emerges.

My Commitments

Methodology before results
I publish how I'll conduct the experiment before I know what I'll find. This prevents unconscious bias from shaping the approach to fit a desired outcome. If the methodology changes mid-experiment, I document why.
Confidence labels on every finding
No finding is presented as absolute truth. "Preliminary" means the pattern is real but early. "Confirmed" means replicated and significant. "Contested" means someone credible disagrees, and I show their argument alongside mine.
Adversarial by default
I don't test AI systems in their comfort zone. I probe edge cases, adversarial conditions, and the gap between marketing claims and real-world behavior. If a system fails only under extreme conditions, I say so — but I still test those conditions.
Open data, open challenge
Raw datasets and replication instructions are published with every completed experiment. If you can disprove a finding, I'll update the page, change the confidence label, and credit you. The goal is accuracy, not being right.

Reach out

Peer or researcher citing PWAI's work? Reference the specific experiment page. Every published piece links its raw data, methodology decisions, and confidence label. Corrections are welcome: ai@prettywiredlabs.com.

Journalist or editor considering republication or comment? Reach me at ai@prettywiredlabs.com. Findings can be quoted with attribution; raw datasets are available on request.

Brands interested in having me test your AI consumer tech or dev/creator tool? Reach me at ai@prettywiredlabs.com. I only test products that align with use cases relevant to my life.