About the Star Quality Rating

The star quality rating indicates the quality or confidence in the results of the study producing the CMF. While the reviewers applied an objective as possible set of criteria, the star quality rating still results from an exercise in judgment and a degree of subjectivity. The star rating is based on a scale (1 to 5), where a 5 indicates the highest or most reliable rating. The review process to determine the star rating judges the accuracy and precision as well as the general applicability of the study results. Reviewers considered five categories for each study — study design, sample size, standard error, potential bias, and data source — and judged each CMF according to its performance in each category (see chart below).

Relative Rating Excellent Fair Poor
Study Design Statistically rigorous study design with reference group or randomized experiment and control Cross sectional study or other coefficient based analysis Simple before / after study
Sample Size Large sample, multiple years, diversity of sites Moderate sample size, limited years, and limited diversity of sites Limited homogeneous sample
Standard Error Small compared to CRF Relatively large SE, but confidence interval does not include zero Large SE and confidence interval includes zero
Potential Bias Controls for all sources of known potential bias Controls for some sources of potential bias No consideration of potential bias
Data Source Diversity in States representing different geographies Limited to one State, but diversity in geography within State (e.g., CA) Limited to one jurisdiction in one State

To provide a more quantitative translation from these categories to the star rating, a point-based system was developed. Points are assigned to each CMF characteristic based on the level of rigor (excellent = 2 points, fair = 1 point, or poor = 0 points). While the points decrease from excellent to poor, not all characteristics receive equal weight. For example, the study design is more important than the data source. Therefore, the final quality rating is based on a weighted score. Study design and sample size categories receive twice the weight of the other characteristics (see equation below).

Score = (2 * study design) + (2 * sample size) + standard error + potential bias + data source

The star rating is assigned based on the score and the ranges in the table below. It should be noted that information may be missing from a study report for specific characteristics such as sample size. In these cases, the rating is based on available information and the CMF will likely receive a lower rating due to the lack of information.

Score Star Rating
14 (max possible) 5 Stars
11 – 13 4 Stars
7 – 10 3 Stars
3 – 6 2 Stars
1 – 2 1 Star
0 0 Stars

 

The information contained in the Crash Modification Factors (CMF) Clearinghouse is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the use of the information contained in the CMF Clearinghouse. The information contained in the CMF Clearinghouse does not constitute a standard, specification, or regulation, nor is it a substitute for sound engineering judgment.