Calibrate a Carrier-Research Rule with Independent Reviews

To check whether two researchers apply a carrier-selection rule consistently, give them the same saved evidence, the same written criterion and separate answer sheets. Compare their labels and the evidence behind each label before they discuss the cases. Resolve unclear wording separately from missing facts. Then test the revised rule again without turning an unresolved case into a confirmed match.

This guide answers one question: Will two researchers reach the same supported decision from the same carrier evidence? The output is a versioned decision rule, a six-packet exercise and a short record of what changed. It is an editorial method for research quality review, not a carrier rating or a USDOTData product feature.

By John Hauler, USDOTData's editorial persona. Sources reviewed October 22, 2026. All companies, records and review results in the exercise are fictional.

Choose one decision that reviewers can actually test

Start with a narrow criterion. “Good trucking prospect” combines too many judgments: identity, location, equipment, service capability, relationship ownership and current need. Reviewers can agree on that label while answering different questions.

For this exercise, the decision concerns one named operating yard for one carrier entity during October 2026. A fictional mobile-service company has already defined its service area. The research question is whether the evidence establishes that the selected yard is in that area during the stated period. Call this yard-location fit. It does not establish that a visit is feasible or that the carrier wants service.

Keep three results available:

• C — confirmed within scope: the packet supports the stated yard-location criterion.
• O — outside scope: the packet supports a specific location mismatch with that criterion.
• U — unresolved: the evidence cannot settle the criterion, including a material conflict.

These are local worksheet labels. U is a completed research judgment when a decisive fact is missing. Keep “review not completed” as a separate work status; do not automatically put it in U.

Use the prospect-exclusion policy when the resulting decision needs a campaign-specific reason and a re-entry condition. This exercise checks how people apply one criterion before the team uses it across a larger batch.

Freeze the evidence before comparing the reviewers

Build one bounded packet per case. Both reviewers need the same source versions, complete relevant passages and stated cutoff. If one reviewer opens a refreshed source or uses a private conversation the other cannot see, their results no longer test interpretation of identical evidence.

Each packet should identify:

1. The carrier entity, named yard and period being assessed.
2. The exact documents or observations available to both reviewers.
3. The date of each observation and any separate date or period stated in the source.
4. The source labels, definitions and permitted business-record context.
5. Deliberate gaps, such as an unknown statement date or unresolved site identity.

Give the packet a stable ID and revision. Keep the evidence itself separate from the answer sheet. Do not include an expected label in the material the reviewers receive.

An address needs its purpose attached. FMCSA defines the SAFER physical address as the principal-office location and the mailing address as the place where the company receives mail. Neither definition says that it identifies a maintenance yard. That is why a local mailing address alone cannot settle the example's operating-yard question. FMCSA SAFER field definitions

If a working field is called only “address,” resolve that mapping with the carrier-data field dictionary before running the exercise. Otherwise a definition problem can look like a disagreement about judgment.

Collect independent answers before the discussion

Each reviewer records a label, the decisive evidence reference, the relevant rule clause and the most important uncertainty. A bare C, O or U is not enough. Two matching labels can conceal opposite interpretations.

Do not show the first reviewer's answer to the second before both submit. Do not ask one person to explain the case while the other completes it. Keep the original answers after discussion so the team can see what the instruction actually produced.

Independent reviews mean separate application of the rule. They do not create independent sources for the underlying fact. Two people reading copies of one carrier filing still rely on that filing. Use the source-independence check when the question concerns the origin of evidence.

If a reviewer needs information outside the packet, record the question first. The coordinator can later add the same permitted material to both copies as a new revision. That separates a changed evidence set from a changed interpretation.

Worked example: six packets and two different problems

The following records and results are invented teaching cases. Assume the entity match, stated service-area boundary and relevant business role are established unless the case says otherwise. An existing business response is part of the packet; this exercise does not require new outreach.

The first written rule, Yard rule v1, says: “Confirm fit when current evidence places the operating yard inside our service area; mark outside when it places the yard outside; otherwise leave unresolved.” The words “current evidence” are not defined well enough.

Packet A: dated confirmation for the named yard

A permitted business record, dated October 20, identifies Yard A as the operating yard for the October period. Its location falls inside the declared service area. Both reviewers select C and cite that record. They agree on the criterion and its evidence.

Packet B: the selected yard is outside the area

A dated response confirms that the selected work site is Yard B, outside the service boundary, for the same period. Both select O. This concerns Yard B under this offer. It does not establish that every location of the carrier is unsuitable.

Packet C: a relevant-looking company without yard evidence

The packet contains a carrier record with a reported power-unit count and a principal-office address. It contains no evidence identifying the operating yard for this work. Both select U. The missing yard fact is explicit; the carrier has not failed the location requirement.

Packet D: a distant office and a confirmed local yard

The principal office is outside the service area. A dated business record identifies Yard D inside the area for the October work period. Both select C and cite the yard evidence. Neither uses the office location as a substitute for the work site.

Packet E: a mailing address mistaken for a yard

The only location evidence is a SAFER mailing address inside the service area. Reviewer A selects C, citing the local address. Reviewer B selects U because the packet does not identify an operating yard.

The source definition resolves the interpretation problem. A mail destination does not establish the relevant yard. The supported result remains U. Add an explicit address-purpose example to the criterion so this mistake is visible before the next batch.

Packet F: a business statement with no usable date

A copied business statement identifies Yard F outside the service area. Its date and applicable operating period are absent. Reviewer A selects O, treating it as current. Reviewer B selects U because it cannot establish the October location.

The packet is incomplete for this decision. Declaring the copy “recent enough” would invent evidence. Retain U until the relevant date and scope are established. The revised rule can explain the gap; it cannot fill it.

The important distinction is between E and F. E exposes misuse of a known source field. F exposes missing temporal evidence and an undefined rule for handling it. E needs actual yard evidence; repeating the mailing address will not help. F needs the date and scope of the business statement established. Both remain unresolved under the present packets.

Count the original decisions without calling them accuracy

Use a small cross-tabulation to retain the direction of disagreement. Rows show Reviewer A's original label. Columns show Reviewer B's original label.

Original reviews: six packets

                 Reviewer B
Reviewer A       C   O   U   Total
C                2   0   1     3
O                0   1   1     2
U                0   0   1     1
Total            2   1   3     6

Same label:       4 packets
Different label:  2 packets

The diagonal contains A, B, C and D. The C/U cell is E; the O/U cell is F. All six packets remain in the denominator. Excluding U would erase the very cases the exercise should expose.

This is an agreement table, not an accuracy table against established real-world truth. NIST's measurement guidance makes a related distinction: if a measurement process is improperly specified or calibrated, control procedures may establish comparability without establishing the quality intended. Here, the practical editorial lesson is to examine the rule and evidence even when the labels match. We are not applying a NIST statistical-control test to these six cases. NIST measurement-process guidance

The four agreements do not establish how often the team will agree on future accounts. The six deliberately chosen examples are diagnostic cases, not a representative estimate or a staff-performance score. Their usefulness lies in the specific defects they reveal.

Revise the rule and keep the unresolved evidence visible

The coordinator should inspect the cited evidence before deciding whether a disagreement requires a wording change, an evidence request or a correction to application. Seniority or majority preference is not a substitute for the criterion.

For this exercise, Yard rule v2 adds these instructions:

• Evaluate the named yard, resolved entity and stated work period.
• C requires evidence identifying that yard and placing it inside the declared service boundary for the relevant period.
• O requires equally scoped evidence that places the selected yard outside the boundary.
• A mailing or principal-office address alone does not establish the yard's purpose.
• A business statement must have enough date and scope context to support the relevant period. A recent retrieval date does not supply a missing statement date.
• When a decisive fact is missing or conflicting, record U and name the gap.

Retest the same six packet revisions with v2. In the fictional retest, both reviewers return C, O, U, C, U and U for A through F. Record this as agreement on six reused teaching packets after clarification. Do not rewrite the original four-agreement result or describe the retest as proof of general reliability.

The retest checks whether the clarification is understood. It can also reflect familiarity with the examples. Before wider use, include fresh permitted cases that test the same boundaries, such as a dated statement that covers a different site or an earlier operating period. Keep their results separate from the teaching retest.

No universal pass threshold follows from this exercise. The decision owner should set the required review scope and unresolved-case handling before adopting the rule. If a formal population estimate is needed, use an appropriate carrier-research sampling plan and analysis rather than extending the six-case count.

Copy this calibration worksheet

Use one batch card and one decision card per reviewer and packet. The cards fit ordinary text records; they do not require a new tool.

BATCH CARD
Decision / entity-site-period scope:
Criterion ID and version:
Field dictionary version:
Evidence cutoff / packet manifest:
Reviewers / coordinator:
Labels and meanings:
Unfinished-review handling:

INDEPENDENT DECISION CARD
Packet ID / revision:
Reviewer / submitted at:
Review complete: yes or no
Label: C / O / U
Decisive evidence reference:
Rule clause applied:
Missing or conflicting fact:

RESOLUTION AND RETEST
Original labels retained where:
Same label and same reasoning?
Disagreement type:
  rule / field meaning / missing evidence
  changed packet / application error
Evidence-based resolution / owner:
Revised clause or example:
Rule version / packet revision retested:
Retest labels / remaining uncertainty:
Fresh-case check / adoption decision:
Affected earlier decisions to review:

Before applying the rule to another batch, check that both reviewers used identical packets, all completed and unfinished reviews reconcile, every decision cites evidence, and changes preserve the original answers. Confirm that unresolved cases have an owner and a specific next question.

If v1 was already used on real accounts, identify decisions that relied on the affected clause. Recheck those decisions against permitted evidence. Do not assume every old C is wrong or silently relabel an entire carrier list.

Frequently asked questions

Should reviewers see each other's answers before they start?

No, if the purpose is to check independent application. Collect the first decisions separately, then discuss them. Joint training is useful, but its results answer a different question from an independent first pass.

Does agreement prove that a carrier fact is correct?

No. Both reviewers may use the same mistaken definition or incomplete source. Check the evidence and reasoning behind matching labels. Agreement describes the reviewers' decisions on the supplied material.

How should missing evidence be labeled?

Use an explicit unresolved result when the missing fact can change the decision. Name the gap. Keep that result separate from an unfinished review and from evidence showing a genuine mismatch.

How many packets are enough?

There is no universal count supplied by this workflow. Select cases that test the actual rule boundaries for a diagnostic exercise. A formal claim about future agreement or accuracy needs a suitable design, outcome definition and analysis; six teaching packets do not supply those.

When should we change the criterion?

Change it when the intended decision is unclear, a source definition was misused, or an important evidence state has no stated treatment. Preserve the old version and explain the change. Do not change the business requirement just to make reviewers agree.

Can the result measure staff performance?

This exercise primarily diagnoses the instruction and evidence process. A difficult packet, unclear rule or unequal information can cause disagreement without showing poor work. Do not rank staff from the six fictional cases or a batch designed to concentrate on ambiguities.

Choose one rule your team will use next. Save its evidence packets, collect separate answers and repair the most consequential ambiguity. A supported yard location is an observed fit fact under this criterion. Service suitability remains a broader hypothesis, and confirmed buying need requires separate business evidence.