Noblis has developed a set of standard tools that help it manage the information generated while estimating cost-effectiveness transition points (CETP) for various countermeasure technologies during the consequences management phase and evaluating the overall risk posed by scenarios during the threat assessment phase. These tools are data bases that capture the thousands of individual ratings that make up the evaluation and combine them to generate the overall assessments. These tools also allow Noblis to perform quality assurance (QA) at various stages of the project. Sensitivity analyses can also be performed to identify the elements in the evaluation that are the main drivers of the study results; these are generally performed at decision points in the process, i.e., when selecting threats that will be carried forward in the analysis and when determining which countermeasures are cost-effective.

Quality Assurance

Using its standard tools, Noblis can perform QA on the evaluation. The QA checks primarily involve completeness and consistency. The standard tools allow Noblis to identify which elements have been evaluated and which have not been evaluated at any given time. This ensures that the final analysis is not missing key pieces of information, which is otherwise quite possible given the very large number of individual elements that must be evaluated. Consistency checks allow Noblis to place all the individual evaluations for any particular element side by side so that analysts can check that items are valuated in a consistent fashion. Ideally, this consistency check is done before the final analysis is completed, so that the evaluators focus on the consistency within individual elements and are not tempted to make changes in order to influence the final results. These QA checks mean that the threat assessment and CETP results are based on complete and consistent evaluations of all the elements considered by the expert panel working groups. 

Sensitivity Analysis

Using its standard tools, Noblis can determine the sensitivity of the analysis to a variety of parameters, including:

• Weightings assigned to evaluation elements - The sensitivity analysis for the threat assessment typically includes varying the weightings to see how this changes the relative ranking of the threats. If changes in weights make only modest changes in the relative ranking, the analysis can be described as robust. If small changes in weights make dramatic changes in the relative rankings, then the analysis is not robust, and the choice of elements to be evaluated and the items considered when evaluating them needs to be carefully reviewed.
• Assignments made to any particular element - The sensitivity analysis for the CETP typically involves varying the costs assigned to various incidents to see how the CETP depends on those assignments. This identifies which cost elements drive the CETP, and allows the analysts to review the evaluations of those elements with particular care.
  

Given that many of the individual elements of any analysis related to terrorist use of WMD necessarily depend on professional judgment, knowing how sensitive the final results are to such elements and which elements drive the conclusions is of great importance to decision makers.