Continuous Active Learning: How CAL Works in eDiscovery Review

Continuous active learning in e-discovery is a technology-assisted review method that re-ranks every unreviewed document after each coding decision a reviewer makes, so the most likely relevant files keep rising to the top of the queue as the model learns. In controlled studies, CAL has matched or exceeded the recall of exhaustive manual review while requiring attorneys to examine only about 2% of the total collection. Federal courts have repeatedly endorsed it, with one widely cited opinion calling it “black letter law” that producing parties may use technology-assisted review when they choose to.1Justia. Rio Tinto PLC v Vale SA et al

How CAL Differs From the Older Train-Then-Apply Model

The first generation of technology-assisted review, often called TAR 1.0, worked in two separate phases. A legal team would code a fixed batch of documents known as a seed set, the algorithm would learn from those examples, and then it would classify the rest of the collection in a single pass. Once training ended, learning stopped. If the seed set missed a whole category of relevant evidence, the model had no way to correct itself.

CAL erases that line between training and review. The algorithm updates after every single document a reviewer codes, so its understanding of relevance evolves throughout the project. There’s another practical difference. In TAR 1.0, many documents the computer classified as relevant were never seen by a human, which created real risk around privileged or sensitive material slipping into a production. In CAL, review continues until every document the algorithm identifies as likely relevant has been looked at by a person.

The seed set also matters far less. Because the model keeps retraining, early missteps in training-document selection get corrected as more reviewer decisions come in. The court in Rio Tinto PLC v. Vale S.A. made exactly this point, observing that when the methodology uses continuous active learning, the contents of the initial seed set become “much less significant.”1Justia. Rio Tinto PLC v Vale SA et al That flexibility is what has pushed CAL to the center of modern e-discovery practice.

How the Algorithm Learns From Each Coding Decision

At the mechanical level, the software converts the text of every document into a numerical representation, mapping words, phrases, and patterns into vectors. That allows the system to measure how similar an unreviewed document is to files already coded as relevant. Because the comparison is mathematical rather than keyword-based, the algorithm picks up conceptual similarity even when the actual vocabulary differs across emails, contracts, and memos.

Each time a reviewer marks a document responsive or non-responsive, that decision becomes a new training example. The algorithm recalculates the relevance score of every remaining document in the collection and pushes the most likely evidence toward the top of the queue. This isn’t a periodic batch update. The ranking shifts in real time, so the next document a reviewer sees already reflects the model’s current state.

Early in a project, the model’s sense of relevance is rough. After thousands of coding decisions, it has built a detailed internal picture of what responsive material looks like for this specific case. Accuracy improves the longer the review runs.

How Documents Reach the Reviewer’s Queue

CAL uses two complementary strategies to decide what a reviewer sees next. The primary approach exploits what the system already believes: it serves whatever it ranks as most likely relevant. That front-loads the discovery of important evidence, which is exactly what teams need when deadlines are tight or when early production shapes settlement talks.

The second strategy is exploration. The system periodically slips randomly selected documents into the queue with no particular predicted relevance score. The point is to expose the algorithm to topics or document types it hasn’t encountered yet. Without that randomness, the model could develop a blind spot for an entire category of relevant material simply because nothing in the early training data pointed toward it.

Seed sets still exist in CAL, but they carry much less weight than they did under TAR 1.0. Teams build them either by random sampling from the full collection or by judgmental selection using keyword searches or clustering. Judgmental seeding can accelerate the earliest rounds, especially when relevant material is rare in the overall population, but random seeds give broader coverage and reduce bias toward what the legal team expected to find. Either way, the stream of reviewer decisions that follows tends to correct an imperfect start.

The Reviewer’s Workflow

The core of a CAL project is a tight loop. A document appears on screen. The reviewer reads it and assigns a code, typically responsive or non-responsive, often with more granular tags like “hot” or “privileged.” That decision feeds back into the engine, the rankings update, the next document appears. No separate training phase. No waiting for a batch to run.

This loop means reviewers spend most of their time on documents that actually matter. In linear review, an attorney might wade through thousands of irrelevant files before finding anything useful. CAL inverts that experience. The highest-value documents come first, and the percentage of irrelevant material in the queue only grows as the review nears the end.

Real-time feedback also lets a team adapt mid-review. If a reviewer stumbles onto a category of relevant evidence no one anticipated at the start, the algorithm picks up on that signal and begins surfacing similar documents on its own. There’s no need to retrain from scratch or rewrite search terms.

Knowing When to Stop

One of the harder judgment calls in a CAL review is deciding when enough is enough. Because the algorithm prioritizes the most relevant material first, the yield of responsive documents drops as the review progresses. Early batches might run 60% or 70% responsive. Deep into the project, that rate can fall into the single digits.

The practical stopping point is where the yield curve flattens and the cost of finding each additional relevant document gets out of proportion to its value. Rule 26 of the Federal Rules of Civil Procedure builds this concept into the discovery standard itself: parties are entitled to discovery that is “proportional to the needs of the case,” considering factors like the amount in controversy, the parties’ resources, and whether the burden of continued review outweighs the likely benefit.2Legal Information Institute. Federal Rules of Civil Procedure Rule 26 – Section: (b) Discovery Scope and Limits At some point, continuing to review documents that are overwhelmingly non-responsive fails that proportionality test.

Stopping, though, isn’t the same as declaring the review complete. The team still has to prove statistically that the algorithm didn’t leave significant pockets of relevant evidence behind.

Validating the Review

Once active coding stops, the next step is showing the review was thorough enough. The standard tool is an elusion test. The team draws a random sample from the documents the algorithm classified as non-relevant, and a senior reviewer examines every file in the sample to check whether anything responsive was missed.

The proportion of relevant documents found in that non-relevant pile is the elusion rate. A sample of 2,600 that turns up 26 responsive documents means an elusion rate of 1%, roughly 1 in 100 documents left behind actually mattered. An elusion rate below 1% to 2% is widely considered defensible, though no court has adopted a single mandatory threshold. Sample size matters too. Standard statistical practice calls for a 95% confidence level with a margin of error no greater than plus or minus 5%, which requires a minimum sample of about 385 documents. Larger samples tighten the margin and strengthen the result.

Elusion is one piece. The two metrics opposing counsel focus on most are recall and precision. Recall measures completeness: of all the relevant documents in the collection, what percentage did the review find? Precision measures accuracy: of all the documents the review flagged as relevant, what percentage actually were?

No specific recall target is mandated, but industry consensus and existing case law treat recall above 80% as generally sufficient, and anything above 90% is very hard for an opposing party to attack. For context, research has shown that exhaustive manual review by attorneys averages recall of only about 59%, while CAL workflows in controlled studies hit recall between 77% and 97%. The assumption that attorney eyes on every page produces a more complete review doesn’t hold up against the data.

Federal Rule of Civil Procedure 34 lets a party request documents within the scope of Rule 26, which means the requesting side can challenge a production as incomplete if it looks like significant responsive material was missed.3Legal Information Institute. Federal Rules of Civil Procedure Rule 34 – Section: (a) In General Strong validation metrics are the best defense against that kind of attack.

Protecting Privilege in a Fast Review

Speed creates risk. When CAL is surfacing thousands of documents for rapid coding, the odds of accidentally producing a privileged communication go up. Attorney-client emails, litigation strategy memos, and work-product documents can get swept into a production set, especially when the relevance model scores them highly because they discuss the same subject matter as the dispute.

The most important safeguard is a Rule 502(d) order. Federal Rule of Evidence 502(d) allows a court to order that producing privileged documents during litigation does not waive the privilege, either in the current case or in any other federal or state proceeding.4Legal Information Institute. Federal Rules of Evidence Rule 502 – Section: (d) Controlling Effect of a Court Order Without that order, an inadvertent production can destroy the privilege permanently. With it, the producing party can claw the document back and the disclosure has no lasting consequence.

Requesting a 502(d) order should be one of the first moves in planning a CAL review. The Southern District of New York’s model order provides standard language establishing that production of privileged material, “whether inadvertent or otherwise, is not a waiver of the privilege or protection from discovery in this case or in any other federal or state proceeding.”5United States District Court Southern District of New York. Rule 502(d) Order The order doesn’t eliminate the need for a privilege review before production. It provides a safety net when mistakes happen.

Many teams add another layer. Automated filters flag documents containing known attorney names, law firm domains, or privilege-related terminology, routing them to a separate track with more experienced reviewers. That reduces the chance a first-level coder marks a privileged document responsive.

What to Discuss at the 26(f) Conference

Federal Rule of Civil Procedure 26(f) requires the parties to confer early and develop a discovery plan that addresses “any issues about disclosure, discovery, or preservation of electronically stored information, including the form or forms in which it should be produced.”6Legal Information Institute. Federal Rules of Civil Procedure Rule 26 – Section: (f) Conference of the Parties; Planning for Discovery For cases using CAL, the 26(f) conference is where the parties work out the review protocol.

How much a producing party must disclose about its CAL process is still an open question, and courts are split. The District of Minnesota’s e-discovery guide suggests the parties talk through which custodians and data sources will go through CAL, what software and vendor will run the review, how the algorithm will be trained and who will code the training documents, what quality control measures the team will use, whether validation metrics like recall and sample size will be shared, and how any disputes about the process will be handled.7United States District Court, District of Minnesota. Discussion of Electronic Discovery at Rule 26(f) Conferences – A Guide for Practitioners

Disclosure of all those details isn’t strictly required. In Rio Tinto, the court observed that a requesting party can test the adequacy of a production through other means, such as statistical recall estimates at the end of the review and whether gaps appear in what was produced.1Justia. Rio Tinto PLC v Vale SA et al The court was explicit that it is “inappropriate to hold TAR to a higher standard than keywords or manual review,” meaning demands for transparency beyond what would be expected for a keyword search may be unreasonable. In practice, the smart move is to be more transparent than the rules strictly demand. Early agreement on validation protocols avoids expensive fights later.

The Case Law That Makes CAL Defensible

Two federal opinions do most of the work in establishing CAL’s legal defensibility.

Da Silva Moore v. Publicis Groupe

This 2012 decision was the first judicial opinion approving technology-assisted review.8Federal Judicial Center. Technology-Assisted Review for Discovery Requests Magistrate Judge Andrew Peck of the Southern District of New York held that “computer-assisted review now can be considered judicially-approved for use in appropriate cases” and said it “should be seriously considered for use in large-data-volume cases where it may save the producing party (or both parties) significant amounts of legal fees.” The opinion grounded its approval in the parties’ agreement to use the technology, a collection of over three million documents, the method’s superiority to manual review or keyword searching, proportionality under Rule 26, and the transparency of the proposed process. It also set an expectation that still governs: “the Federal Rules of Civil Procedure do not require perfection.”9Justia. Da Silva Moore v Publicis Groupe et al

Rio Tinto PLC v. Vale S.A.

Three years later, Judge Peck went further, declaring that “it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.” The opinion drew an important line: courts let a producing party choose TAR, but have refused requests by the opposing side to force a party to use it. The choice belongs to the producing party. The decision also directly addressed CAL’s advantages over earlier TAR methods, noting that continuous learning reduces the significance of the initial seed set.1Justia. Rio Tinto PLC v Vale SA et al Judge Peck was careful to note that the approved protocol in that case was not a universal template and did not endorse any particular vendor or software.

Taken together, these opinions give producing parties wide latitude to use CAL. Courts will not demand perfection, and the methodology cannot be held to a stricter standard than the alternatives. What they do expect is a reasonable process with appropriate quality control, transparency proportional to the circumstances, and statistical validation that the production is substantially complete. A legal team that documents its protocol, runs proper validation, and cooperates with opposing counsel on methodology has built a defensible review.