There is a particular kind of demo every WV attorney has now sat through.
A vendor in a polo shirt opens a laptop. He types a question into a clean white box. The screen fills with confident-sounding prose, a few citation cards, maybe a chart. He turns the laptop around and waits for the reaction. The reaction he is hoping for is that looks impressive. The reaction he is rarely prepared for is what is actually happening underneath this.
Most attorneys do not push on the second question. Not because they do not want to, but because the vocabulary feels foreign and the demo is already moving on to the next slide. So the evaluation ends where the marketing ends, which is the worst possible place for a tool that is going to sit between the attorney and her bar license.
This post is the list of questions to ask in that demo. They apply to our tool. They apply to every competitor. They apply to whatever launches next year that nobody has seen yet. If a vendor cannot answer them clearly, the answer to the larger question — should I trust this with my work — is no, regardless of how good the demo looked.
1. Where do your citations come from?
This is the first question and the most important one. The answer separates the tools that can fabricate cases from the tools that cannot.
A generative system produces citations the same way it produces every other word — by predicting what would come next in a plausible legal answer. Sometimes the prediction matches a real case. Sometimes it does not. The system has no mechanism to tell the difference because it has no corpus to check against.
A retrieval-augmented system begins with a database of actual opinions. It pulls the relevant ones in response to a question, and the AI's job is to summarize what was retrieved. The citations exist because the documents exist. The system structurally cannot cite a case that is not already in the corpus.
The right answer to this question is specific. "We have ingested every West Virginia Supreme Court of Appeals opinion since 1991 into a searchable index. When you ask a question, we retrieve from that index first, then synthesize." If the answer is some version of "we use the latest large language model," the question has not been answered.
2. Can I open every cited case directly?
A citation that links to nothing is not a citation. It is a string of text that resembles one.
Every cited opinion in a trustworthy tool should be one click away from the actual document — the PDF on the court's website, a verified reproduction, a source the attorney can read in full. If the vendor's demo shows citation cards without click-through, ask why. If the answer is "that's coming in a future release," the tool is not ready.
This is not a paranoid question. It is the operational mechanism that makes verification possible in two minutes instead of forty-five. Without it, the tool has not actually saved any time — it has just moved the hunt from before the answer to after.
3. What does your corpus actually cover, and what does it not?
A vendor who claims to cover "all U.S. law" is either being imprecise or is wrong about what coverage means. Real coverage has dates, jurisdictions, document types, and update frequencies.
For a tool an attorney is going to rely on, the corpus answer should sound something like this: "Every published opinion from these specific courts, in this date range, updated on this cadence. These statutes, current as of this date. These regulations, sometimes; ask before you rely." The vendor who can describe the boundaries of the corpus precisely is the vendor whose tool will not surprise the attorney with a gap she did not know existed.
The corollary question is what is not in the corpus. A tool that knows what it does not know is a tool that will tell the attorney when her question is outside its competence. A tool that pretends to know everything will produce confident answers in cases where the truth is there is no controlling authority on this point, and confident answers in those cases are the dangerous kind.
4. What happens when there is no clear authority?
This is the question most demos are not designed to handle, which is exactly why it is worth asking.
Ask the tool a question you already know the answer to — one where the controlling authority is thin, contested, or genuinely open. Watch what comes back. A trustworthy system will tell you the authority is limited, will flag the uncertainty, will lower its precedent strength rating, or will say outright "there is no controlling WV authority on this question."
An untrustworthy system will produce the same confident-sounding memo it produces for every other question. The fluency will be identical. The citations may be thinner or stretched, but the tone will not give that away. The attorney has to know to look.
If the tool's behavior on a known-hard question is indistinguishable from its behavior on a known-easy question, the tool has no internal sense of its own limits. That is the Mata v. Avianca failure mode, dressed up in better packaging.
5. Can I see how the answer was constructed?
A black-box answer is a liability. The attorney has no way to evaluate whether the synthesis applied the right authority to the right question, and the right authority to the right question is the entire job.
A reasoning panel — even a simple one — converts the output from a conclusion into an audit trail. The attorney can read the chain, identify the controlling opinions, and check whether the AI weighed them the way a court would weigh them. This is the same review the attorney would conduct on a junior associate's memo. The tool should not earn an exemption from that review because it is software.
Ask to see the reasoning. If the vendor cannot show it, the attorney has been asked to substitute the tool's judgment for her own. She should not.
6. How does the tool grade its own confidence?
Different questions produce different levels of certainty. A tool that returns every answer with the same flat confidence is hiding information from the attorney.
A precedent strength badge, a confidence score, a freshness indicator, an authority-level flag — the specific implementation varies, but the underlying signal is the same. The tool is telling the attorney here is how much of this you can lean on. That signal sets the attorney's reading posture before she reads a word of the memo. Without it, every answer feels equally reliable, and that uniformity is the lie that hides the gaps.
If the tool grades itself, the attorney can calibrate her trust per answer. If it does not, she has to bring her own calibration to every interaction — which is workable, but which removes one of the main reasons to use the tool in the first place.
7. Where does my data go, and who can see it?
The research the attorney runs is, in the aggregate, a map of her practice. The questions she asks reveal the cases she is working on, the matters that are coming up for hearing, the issues her clients are facing. That information is privileged in spirit if not always in technical doctrine, and the tool that stores it should treat it that way.
The vendor's answer should specify where the data is stored, who has access, what is shared between accounts, and what is shared with the AI providers underneath the tool. If the tool is built on top of a commercial API, ask whether the vendor's contract with that provider prohibits training on the attorney's queries. If the vendor cannot answer cleanly, the attorney has reason to be cautious about what she asks.
Inside a firm, the same question applies horizontally. Is research history private to the researcher, or visible to everyone in the firm? Can the admin see what each attorney is researching? The answers are not inherently right or wrong — different firms have different cultures — but the attorney should know the answers before she runs the first query.
8. What is the verification workflow, and how long does it take?
Every cited authority in every memo should be verified before it is relied on. That is true with AI, true with a junior associate, true with Westlaw, true with a treatise. The question is not do I have to verify — the answer to that is always yes — but how long does verification take with this tool.
A tool with one-click citation access and visible reasoning supports two-or-three-minute verification on a typical memo. A tool that hides its citations behind logins, paywalls, or unclickable text turns verification into a thirty-minute exercise, which means the attorney either stops verifying or stops using the tool. Both outcomes are bad.
Ask the vendor to walk through a verification, end to end, on a real answer. Time it. If verification takes longer than the answer was supposed to save, the math does not work.
What this checklist is for
The eight questions are not a gotcha. They are the working evaluation framework that any responsible legal-tech buyer should apply to any AI-assisted research tool in 2026.
Most of the vendors selling into the WV bar this year are going to fail at least three of these questions. Some will fail all eight. The few who pass cleanly are the few worth taking seriously, and the attorney who runs the checklist has earned the right to her decision — yes or no — because she made it with the information she actually needed instead of the information the vendor wanted to highlight.
Print this. Bring it to the next demo. Push on every question until the answer is specific. The tool that survives the eight questions is the tool that will survive contact with your real work. The tool that does not is the tool that was going to fail you eventually, and finding out in a demo costs nothing. Finding out in a sanctions hearing costs the practice.
West Virginia Case Search answers legal questions in plain English, grounded in every WV Supreme Court of Appeals opinion since 1991, every WV Intermediate Court of Appeals opinion since 2022, and the full West Virginia Code. Every citation links to the source PDF on the court's website. Every answer is graded by precedent strength. You verify before you rely.
Start a 7-day free trial — no credit card required.