Disaster Recovery Testing for NIST SP 800-34: Prove the Decision Before the Outage

If a Modesto public-safety team cannot demonstrate who declares an outage, which systems recover first, and how dispatch verifies the restored data, its contingency plan is only paperwork. NIST SP 800-34 points toward a more useful standard: test the operating decisions, technical recovery steps, and evidence needed to improve the plan.

At 2:13 a.m., a Modesto emergency communications supervisor notices that the computer-aided dispatch map is showing stale unit locations. The radio console still works, but the dispatch application cannot reliably assign the next call. A database administrator is checking replication status, the patrol lieutenant is asking whether officers can see active incidents, and the IT lead is deciding whether restoring the latest backup will overwrite potentially corrupted data.

The decision is not simply “restore the server.” The team must decide whether to fail over, how to keep dispatch moving during the transition, who authorizes the change, how evidence and call records remain intact, and what “operational” means before the system is handed back to dispatchers.

That is the practical value of testing for NIST SP 800-34 contingency-planning requirements. A meaningful exercise turns a plan into an observable sequence of decisions. It shows whether the named people, recovery priorities, backup controls, alternate systems, and communications procedures work together under pressure.

What does NIST SP 800-34 actually expect from a recovery test?

NIST SP 800-34 is guidance for developing information-system contingency plans, evaluating recovery requirements, and setting priorities; it also explains how contingency planning relates to disaster recovery, incident response, emergency management, and organizational resilience.¹ For a California public-safety organization, that makes it a useful structure for asking a practical question: can the organization continue essential operations while technology is unavailable, compromised, or being rebuilt?

The guide separates three activities that are often mistakenly treated as one:

• Testing verifies that systems and recovery procedures operate as intended.
• Training prepares personnel to perform their assigned responsibilities.
• Exercising exposes gaps in the plan, coordination, and decision-making.

NIST describes testing as validation, training as preparation, and exercising as a way to identify planning gaps; together, those activities improve plan effectiveness and organizational preparedness.²

That distinction matters. A backup job showing “successful” is not a test of an emergency workflow. A tabletop where everyone agrees that “IT will restore the system” is not proof that restoration works. And a technical failover without dispatch supervisors, records personnel, vendors, and communications staff does not establish that the organization can operate.

A strong program uses the right type of event for the question being answered:

Test or exercise	Decision it validates	What Datapath would look for
Tabletop discussion	Do leaders understand activation authority, priorities, and communications?	Named decision-makers, escalation paths, manual workarounds, and unresolved assumptions
Walk-through	Can each role follow the written procedure?	Current runbooks, correct contacts, access to credentials, and usable vendor instructions
Functional recovery test	Can a system or component be recovered without affecting production?	Restore sequence, dependencies, authentication, network access, application health, and data validation
Parallel recovery	Can the alternate environment run while production remains available?	Application functionality, integrations, performance, user access, and operational acceptance
Full interruption or failover	Can the organization actually operate from the alternate environment?	Technical recovery, human coordination, communications, service continuity, and return to normal

NIST says contingency-plan testing should use explicit test objectives and success criteria and may examine notification, recovery on an alternate platform, connectivity, performance, restoration of normal operations, and coordination with related plans.² That is the difference between “we conducted a drill” and “we proved that the dispatch database can be restored, authenticated, queried, and accepted by its users.”

How do you turn a contingency plan into a testable decision?

Start with the business service, not the backup console. In the Modesto example, the service is emergency dispatch—not merely the CAD virtual machine. The recovery chain could include:

1. Identity and privileged access
2. Core network, DNS, and firewall services
3. CAD application and database
4. GIS or mapping integration
5. Radio and telephony interfaces
6. Mobile data or officer-facing access
7. Evidence, incident, and call-record retention
8. Supervisor verification and operational acceptance

If the test only restores the database, it may miss a broken identity connector or an unavailable GIS feed. If it only proves the application opens, it may miss the permissions that prevent dispatchers from viewing active incidents. If it does not include the supervisor who accepts the service, IT may declare success while operations remain unusable.

Write the objective before choosing the scenario

A useful objective is observable and bounded. For example:

“Within 20 minutes of an authorized failover decision, restore the dispatch application to the alternate environment, authenticate three test users, display current test incidents and unit locations, and obtain written acceptance from the dispatch supervisor.”

The 20-minute target in this example is an organizational decision, not a universal NIST requirement. The point is to make the target measurable. If the team cannot agree on the target, the test has already identified a governance gap.

Create a separate objective for data loss. For example, the recovery team might decide that the restored environment must contain all test records committed before a defined cutoff. That is a recovery-point decision. It should be agreed with the service owner before the exercise—not discovered after the restore begins.

Then define failure conditions. The test should not be scored as successful merely because a server boots. Failure might include:

• The wrong person can activate the plan.
• The backup is available but cannot be decrypted.
• The restored database lacks required records.
• Dispatchers cannot authenticate or reach a dependent service.
• The alternate environment is slower than the operational threshold.
• A vendor contact or integration credential is obsolete.
• No one knows whether to continue operating in degraded mode.

What should a real NIST-aligned test include?

1. A controlled scenario with a decision point

Do not begin with “restore from backup.” Begin with a credible operating condition: ransomware indicators on a file server, a failed storage array, a corrupted dispatch database, a facility network outage, or loss of the primary hosting environment. Introduce a decision point where the incident lead must choose between containment, continued degraded operations, and failover.

The scenario should identify what participants know at each stage. If the team is told immediately that the backup is clean, the test removes the uncertainty that makes real recovery difficult. A better scenario may reveal that the latest restore point is questionable and require the team to select an earlier validated copy.

2. A recovery sequence with dependencies

Document the order in which services must return. For a public-safety system, restoring the application before identity, network routing, or time synchronization may produce a technically “running” system that cannot support users.

The recovery sequence should name the owner of each step, the evidence of completion, and the person authorized to proceed. This is where a named Datapath team can add value: our disaster recovery work can connect the recovery plan to infrastructure, cloud services, vendors, security controls, and the people who operate the environment.

3. User validation, not just infrastructure validation

A system owner or operational supervisor must test realistic transactions. For dispatch, that could mean opening a test incident, assigning a unit, viewing the map, sending a status update, and confirming that records appear where expected. For a clinic, it could mean validating the downtime workflow and controlled return to the EHR. For a school district, it could mean verifying identity, bell-schedule access, and communications before classes resume.

Technical staff should record the evidence: timestamps, screenshots, restore logs, error messages, test accounts used, and the acceptance decision. This creates accountability without exposing production records or treating an informal verbal confirmation as proof.

4. Security checks during recovery

Recovery is not the time to bypass every control indefinitely. Test how privileged access is obtained, how restored systems are isolated, how malware checks are performed, how credentials are rotated, and how logging resumes. If the scenario involves suspected compromise, the recovery environment should have a documented trust decision rather than an assumption that the newest backup is safe.

For local government and public safety, our government and public safety team can help align recovery testing with the operational reality of dispatch, law-enforcement systems, records, and interdependent departments. Where security leadership is missing internally, a vCISO engagement can help define test objectives, evidence requirements, and corrective actions.

How often should recovery testing happen?

NIST guidance says organizations should conduct test, training, and exercise events periodically, after organizational or system changes, after new guidance is issued, or when otherwise needed. It also emphasizes testing as close to the operating environment as possible and testing individual system components for accurate recovery procedures.²

That does not mean every organization needs a disruptive full failover every month. A practical annual rhythm might look like this:

• Monthly: review backup jobs, failures, retention, immutability or offline-copy status, and alert ownership.
• Quarterly: conduct a focused restore of selected files, databases, or application components in an isolated environment.
• Twice yearly: run a tabletop or functional exercise involving business owners, IT, security, vendors, and communications.
• After major change: retest when an application, identity platform, network, cloud dependency, vendor, facility, or recovery procedure changes.
• Annually: perform a broader recovery exercise with operational acceptance and a documented return-to-normal procedure.

The cadence should follow risk and change—not a calendar disconnected from the environment. NIST also recommends keeping the contingency plan current as procedures change and as lessons emerge from tests, exercises, and actual disruptions.²

CISA gives a similarly practical message to state and local governments: test whether teams can restore data fully and partially, verify rollback capability, and ensure staff know how to access critical information when normal connectivity is unavailable³. For a county or city, that means the recovery plan should include degraded operations and manual workarounds, not just a successful server restore.

What evidence should an auditor, board, or county IT director expect?

A credible test package should be easy for a nontechnical decision-maker to understand. Keep:

• The scenario, scope, date, participants, and systems included
• The business and technical objectives
• Recovery time and data-loss targets approved by service owners
• The notification and escalation record
• Step-by-step results with timestamps
• Screenshots, logs, restore reports, and validation results
• Deviations from the plan and why they occurred
• An after-action report
• Corrective actions with owners and due dates
• The date for retesting unresolved findings

Do not report only “pass” or “fail.” Record what was proven, what was not tested, and what remains an assumption. A failed test is useful when it identifies an expired vendor credential, an incomplete dependency map, or a recovery target that the current architecture cannot meet.

NIST also calls for coordination between contingency testing and related plans such as business continuity, disaster recovery, continuity of operations, crisis communications, and cyber-incident response plans. In practice, that means the incident-response lead should not end the exercise when the system is restored. Communications, legal, records, department leadership, and service owners may still have decisions to make.

CISA’s tabletop exercise resources reinforce this approach by providing customizable scenarios, objectives, discussion questions, and after-action-report materials covering incident response and post-incident recovery⁴. A tabletop is therefore most valuable when it produces a change to the plan, not when it simply produces attendance.

How Datapath makes recovery testing operational

A contingency plan should identify more than technology. It should identify accountable people, service priorities, recovery evidence, and the conditions for declaring operations restored.

Datapath supports organizations in Modesto, Ceres, Manteca, Merced, Fresno and the wider Central Valley, Modesto, and communities throughout California with a named team rather than a ticket queue. Depending on your environment, that may include managed IT services, managed cybersecurity, co-managed IT, or a vCIO strategy engagement.

The conversation should begin with one question: which business service would be hardest to operate tomorrow if its systems failed tonight? From there, we can map dependencies, set an organization-owned recovery target, design a test that does not create unnecessary production risk, and produce evidence that leadership can use.

The goal is not to perform a ceremonial drill. It is to know—before the outage—who decides, what happens first, what can be restored, how users verify it, and which gaps must be fixed before the next incident.

---

Footnotes

1. SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems | CSRC ↩

2. Fetched web page ↩ ↩² ↩³ ↩⁴

3. Back Up Government Data | CISA ↩

4. CISA Tabletop Exercise Packages | CISA ↩