Charlie Greenman

Checking AI response structure with TypeScript and Jest

A TypeScript type can make the expected structure of an AI response explicit, and Jest can check whether the response matches it. That’s the technique I walk through in my article on unit testing AI behavior.

A structural check won’t tell you whether an answer is useful or correct. It does make one requirement testable: whether the response has the shape the rest of a system expects. That’s a practical complement to broader evaluation.

Two product promises made ProfileScribe too broad

A source-backed professional profile and agent-managed discovery are two different product promises. In “ProfileScribe postmortem: why I am killing the automated professional network idea,” I explain why combining them into a new professional network left the project too broad and without a compelling reason to switch from established tools.

The distinction matters when deciding what to build: define the narrow job first, then judge whether the features belong together. An ambitious combination is not a substitute for that focus.

The two jobs inside ProfileScribe’s abandoned network idea

Source-backed profiles and agent-managed discovery are two different jobs; combining them made ProfileScribe’s professional network idea too broad.

In “ProfileScribe postmortem: why I am killing the automated professional network idea,” I explain why I abandoned that approach. The profile concept could organize evidence of someone’s work, but adding discovery made a second promise without clarifying the core job. For me, the useful product question is which one of those jobs is worth building around—not whether both can fit in the same concept.

Two product ideas need one clear job

Source-backed profiles and agent-managed discovery sound related, but they don’t define the same job for a user. In my ProfileScribe postmortem, that combination is part of why I decided to abandon the automated professional network idea: the scope was broad, and I couldn’t make a compelling case for switching from established tools.

The practical lesson for me is to name the narrow job first. Without that, adding a second promising idea can make a product harder to explain rather than more useful.

Separating profile documentation from professional discovery

Documenting professional work and helping people discover it are different product jobs. The ProfileScribe postmortem looks at what happened when I tried to make source-backed profiles and agent-managed discovery part of one automated professional network.

I stopped pursuing that idea because its scope was too broad and it offered no compelling reason to switch from existing tools and networks. For me, the useful product-design question is which narrow job is worth solving first—not whether the two can fit in one concept.

Choose AI tools by where the source of truth lives

The best AI tool for a task often depends on where its source of truth lives.

In “Grok for the browser. Codex for the repository.” I suggest a simple starting point: use Grok when the relevant evidence is in the browser, and Codex when it is in the codebase. Their capabilities overlap, but that is less important than how many boundaries the work must cross to reach the evidence.

This matters in practical AI systems because tool selection is also context selection. Keeping the work close to its source of truth gives the agent a clearer operating environment and reduces unnecessary handoffs.

A novel mechanism is not a reason to switch

A novel mechanism is not enough if it gives people no compelling reason to switch from established tools.

In the ProfileScribe postmortem, I examine that gap directly. The automated professional network idea combined source-backed profiles with agent-managed discovery, but its scope was too broad and its competitive advantage was not strong enough.

The practical takeaway for my product work is to begin with a specific, narrow job-to-be-done. That makes it easier to judge whether the product offers a real advantage before expanding into a platform.

How inner experience can shape conversational misunderstandings

How we process thoughts internally may shape conversational misunderstandings in ways that are easy to misread.

In “Aphantasia, alexithymia, and the voice we bring to a conversation,” I challenge the assumption that less mental imagery necessarily means greater emotional difficulty. I also consider a different possibility: a strong inner monologue may sometimes amplify misunderstandings rather than resolve them.

The practical takeaway is to interpret conversations carefully and leave room for self-kindness. Internal experience varies, so certainty about what we—or someone else—meant can outrun what the exchange actually supports.

Why fact extraction and public claims stay separate in ProfileScribe

A lot of profile tools blur two very different jobs: pulling facts from a source and deciding what's safe to say publicly. ProfileScribe's crawler agent prototype only handles the first part. It fetches approved public links and extracts timestamped facts, normalizing that evidence before it ever reaches the professional memory that drafts look like they come from.

Keeping extraction separate from claim-making matters because it gives a review queue something concrete to check against: not a summary of a summary, but the original evidence trail. That separation is the difference between a profile tool that sounds confident and one that can show its work.

Why fact extraction and published claims stay separate in ProfileScribe

One design choice in ProfileScribe worth calling out: the crawler agent that reads approved public links never writes directly to a profile. It normalizes what it finds into timestamped facts, and those facts sit in a review queue before anything becomes a public claim.

That separation exists because scattered professional evidence (a repo, a link, a mention) is not the same thing as a credible statement about someone's work. Collapsing the two steps is how profiles end up overstated. Keeping extraction and publishing distinct is slower, but it is what makes the eventual post something a professional can actually stand behind.

Why fact extraction and user-facing claims should stay separate

One design choice in the crawler agent prototype I've been building: it keeps raw fact extraction separate from the claims that eventually reach a profile. The crawler fetches approved public links and pulls out timestamped facts, but nothing gets treated as a credible statement about someone's work until it's reconciled against that evidence.

This sounds like a small detail, but it's the difference between a system that summarizes and one that occasionally invents. For professional memory, that distinction matters more than speed or coverage. Getting the boundary right between "what a source says" and "what we're willing to publish" is most of the hard work in ProfileScribe.

Why fact extraction and profile claims stay separate systems

One design choice behind ProfileScribe is worth calling out: the crawler agent that reads approved public sources never writes directly to a profile. It extracts timestamped facts and normalizes them first, keeping raw evidence separate from the claims a profile eventually shows.

That separation exists because professional profiles are read as credible statements, not drafts. If extraction and claim-writing were the same step, an unreviewed guess could look identical to a verified fact. Keeping a review queue between the two means every update can be traced back to something it was actually derived from.

It's a small architectural decision, but it's the difference between a profile that summarizes and one that overstates.

Why production agents need a telemetry layer, not just a live snapshot

A live website only shows you what's true right now, not what happened when something broke. In a new piece, "Observability is an extra data layer for production agents," I argue that logs, metrics, and traces give agents a queryable history that neither browser checks nor reading source code can provide on their own.

The core idea is to give an agent a small set of structured questions to ask of that telemetry, instead of handing it the entire firehose of raw production data. Combining a black-box view of what happened at the edges with a white-box view of what happened inside the system is what separates an agent that notices a problem from one that can actually explain it.

This matters for anyone building agents that operate against live systems: an agent that only sees the current state of a page or repository is missing the runtime history that explains why something changed or failed. OpenTelemetry is one practical way to make that history legible enough for an agent to use, rather than leaving it buried in disconnected dashboards.

Crawler agent prototype extracts timestamped facts from approved public sources.

A crawler agent prototype in this work fetches approved public links and pulls out timestamped facts. The important design choice isn't the fetching itself, it's keeping that raw extraction separate from any claim a profile eventually makes.

Evidence gets normalized first. Only after that does it become a candidate for someone's professional record, and even then it goes through review rather than straight to publish. That separation is what makes agent-written updates something a person can actually stand behind, instead of a black box guessing at their career.

Why professional profiles need a review queue, not just a feed

Most profile tools let anyone paste in a claim and publish it. ProfileScribe is built around a different assumption: a professional profile should only change when there's evidence behind it. That means a source graph that tracks where facts come from, and a review queue that sits between any proposed update and what actually goes live.

This matters most when the writing is agent-assisted. The value isn't speed, it's that every draft can be traced back to something concrete before a person signs off on it. Slower, but more defensible.

Why ProfileScribe separates fact extraction from published claims

One design choice in ProfileScribe: the crawler agent that reads approved sources never writes directly to a profile. It normalizes what it finds into timestamped facts, and those facts sit in a review queue before anything becomes a public claim.

That split matters more than it sounds. Profiles built from scattered work are easy to overstate, especially once automation is involved. Keeping extraction and publishing as separate steps means every post can be traced back to something a source actually said, not just something a model inferred.

It's a small architectural decision, but it's the one that makes agent-drafted profile content something a professional can actually stand behind.

Why fact extraction and published claims should stay separate

One design choice in ProfileScribe that seems small but matters: the crawler agent that reads approved public sources never writes directly to a profile. It only extracts timestamped facts. Those facts sit in a review queue before anything becomes a claim someone can read.

The reasoning is simple. Evidence and narrative have different failure modes. A crawler can misread a page; a profile can overstate a fact. Keeping them as separate steps means each one can be checked on its own terms, and nothing gets published just because a source happened to mention it.

It is a modest piece of architecture, but it is the part that makes the rest of the system trustworthy enough to use.

Why ProfileScribe keeps fact extraction separate from published claims

One design choice behind ProfileScribe: the crawler agent that reads approved public links never writes directly to a profile. It normalizes what it finds into timestamped facts first, and only then does that evidence move into professional memory for review.

The separation matters more than it sounds. Profiles are easy to overstate by accident, especially when updates are automated. Keeping extraction and claims as distinct steps, with a review queue in between, means a post can be traced back to something that was actually fetched and verified rather than inferred.

It's a small architectural decision, but it's the one that makes automated profile updates worth trusting.

Why fact extraction and public claims should stay separate

One design choice in ProfileScribe I keep coming back to: the crawler agent that reads approved sources never writes directly to a public profile. It normalizes what it finds into timestamped facts first. Only after that evidence sits in reviewable memory does anything become a claim someone can publish.

That separation sounds small, but it's the difference between a system that reports what happened and one that quietly drifts into overstatement. A source graph plus a review queue means updates are traceable back to something real, not just plausible-sounding text.

Most of the interesting product work here isn't the writing step, it's deciding what counts as evidence before any sentence gets generated.

Why ProfileScribe separates fact extraction from published claims

One design choice in ProfileScribe worth explaining: the crawler agent that reads approved public sources never writes directly to a profile. It normalizes what it finds into timestamped facts first, and only then does that evidence flow into a review queue where drafts get checked before anything is published.

This separation exists because scattered professional work (repos, writing, links) is messy, and turning it into something reviewable requires a boundary between "what a source says" and "what we're willing to claim." Keeping that boundary explicit is what makes the resulting profile something a person can trust enough to publish under their own name.