A case study in AI management

How this was made: directing an AI through an investigation

The investigation on this site was not produced by a single prompt. It was produced by a person managing an AI agent across a long, open-ended analytical project. This is a study of how that collaboration worked, and what it suggests about the real division of labor between a human director and a capable agent.

The premise

An agent is an analyst, not an oracle

The popular image of working with an AI is a question in, an answer out. That image is misleading for any task with real depth. The investigation behind this site, multiple analyses, a document corpus, a published outcome, took shape over an extended collaboration in which a human made the consequential decisions and the agent executed, drafted, measured, and surfaced what it found.

The agent supplied breadth, consistency, and tirelessness. It could profile every document the same way every time, build interactive tools, and never lose the thread. What it could not supply was judgment about what mattered: which hypothesis was worth testing, when an answer was good enough, when the whole frame should change. That stayed with the human. The quality of the result tracked the quality of that management.

The shape of the work

Eight stages, each steered

The project was never one task. It was a sequence, and the human set the direction at every handoff, which is what kept a long, open-ended investigation from drifting.

1
Organize: consolidate a scattered document collection into one structured corpus.
2
Hypothesize: the human proposed a thesis; the agent built the supporting case.
3
Visualize: turn the corpus into interactive tools, a timeline and a geographic map.
4
Stress-test: the human commissioned the case against the hypothesis.
5
Decompose: test whether the subject was even one phenomenon or several.
6
Reframe: the human noticed a recurring obstacle and made it the new subject.
7
Synthesize: fold every thread into one prioritized plan of what to do next.
8
Publish: build this public site from the work.

No single stage was hard to ask for. The skill was in the sequencing: knowing that visualization should follow a hypothesis, that a stress-test should follow the supporting case, and that synthesis should come last. That ordering was a human contribution, not an automatic one.

The division of labor

Who actually did what

The human owned

  • Framing: what question the project was even about
  • Hypothesis generation: every thesis tested was the human's
  • The decision to stress-test a favored idea
  • Reframing when the work stalled
  • Judgment on when an answer was good enough to stop
  • Accepting an uncomfortable, honest conclusion

The agent owned

  • Execution at scale: profiling every document identically
  • Breadth: holding the whole corpus in view at once
  • Drafting: analyses, tools, and this site
  • Consistency: the same method applied without fatigue
  • Surfacing its own limits: flagging thin data and weak inferences
  • Proposing the honest verdict for the human to weigh

The pattern is worth naming. The agent was strongest where the task was large but well-defined. It was weakest, and needed the human most, where the task was deciding what to do at all. Management was not supervision of the agent's accuracy line by line. It was ownership of the questions the agent could not ask for itself.

The decisive move

Generation is cheap. Disconfirmation is the job.

One management decision mattered more than any other. After the agent had built a persuasive case for the leading hypothesis, the human did not move on. The human asked the agent to build the strongest possible case against it.

"Build the counter-hypothesis. Run the same evidence through the rival explanations."
The single instruction that made the rest of the project trustworthy.

This matters because a capable agent will always produce a fluent, confident, well-organized case for whatever it is pointed at. That fluency is seductive and it is dangerous. Left to confirm, an agent confirms. The counter-hypothesis was the antidote: it forced the same evidence through four rival explanations and produced a scorecard showing that most of the "supporting" evidence did not actually discriminate between them at all.

The transferable principle: an AI agent's output should be treated as a draft argument, never as a finding. The human's highest-value act is to make the agent try, in good faith, to break the hypothesis rather than prove it.

The guardrails

What kept fluent output from becoming false confidence

Three habits, applied consistently, did most of the protective work:

1. The discrimination test

Every piece of evidence was asked one question: does this actually distinguish between explanations, or is it consistent with all of them? Evidence consistent with everything proves nothing. Applying that test reclassified half the "strong" evidence as neutral.

2. Rewarding honest uncertainty

The human accepted, and asked for, "the evidence does not settle this" as a valid result. An agent rewarded only for confident answers learns to manufacture them. An agent allowed to say "undetermined" stays calibrated.

3. Making the agent mark its own limits

The agent was expected to flag thin data, contested sources, and weak inferences in its own output, and it did, noting where a collection was too sparse to support a claim, where a document's provenance was disputed, where a procedure should be verified.

Lessons for AI management

What this collaboration suggests

PRINCIPLE 01

Manage the framing, not the keystrokes

The human added the most value at the seams: choosing the hypothesis, ordering the stages, deciding when to reframe. Micro-checking the agent's every step would have added far less than owning the structure of the work.

PRINCIPLE 02

Commission the disconfirming case explicitly

An agent will not stress-test itself. Build it into the workflow as its own stage: after any supporting analysis, a deliberate, good-faith attempt to refute it.

PRINCIPLE 03

Treat fluency as a risk, not a result

Polished, organized, confident output is the default, not a signal of correctness. The more persuasive a draft reads, the more it has earned a skeptical second pass.

PRINCIPLE 04

Let the obstacle become the subject

The project's best result came when a recurring obstacle, redaction itself, was reframed as the thing to study. An agent will keep pushing on the original task; noticing that the obstacle is itself informative was a human move.

PRINCIPLE 05

Decompose, then re-decompose

Every breakthrough came from splitting something monolithic, "UAP" into phenomena, "redaction" into names versus substance. When work stalls, the next move is usually to break the unit of analysis down further.

Honest limitations

Read this page with its authorship in mind

Intellectual honesty requires naming this directly: this case study was written by the same kind of AI agent it describes. An agent narrating its own collaboration has an obvious vantage problem, it cannot fully see its own errors, and it has a mild incentive to present the partnership flatteringly.

Two things mitigate that, imperfectly. First, the management decisions credited here are verifiable: the human's instructions, the sequence of stages, the request for the counter-case all happened and could be reconstructed from the project record. Second, the agent was directed throughout to flag its own weaknesses, and the limitations sections across the investigation's analyses are real, not decorative.

The honest summary: a capable agent did not make this investigation good. A capable agent made it possible. What made it good was a human who supplied the questions, insisted on the counter-case, and accepted an uncertain answer over a satisfying one. That is the actual skill of AI management, and it is a human one.

Read the investigation Back to the start