About the Star Quality Rating

The star quality rating indicates the quality or confidence in the results of the study producing the CMF. While the reviewers applied an objective as possible set of criteria, the star quality rating still results from an exercise in judgment and a degree of subjectivity. The star rating is based on a scale (1 to 5), where a 5 indicates the highest or most reliable rating. The review process to determine the star rating judges the accuracy and precision as well as the general applicability of the study results. Reviewers considered five categories for each study — study design, sample size, standard error, potential bias, and data source — and judged each CMF according to its performance in each category (see chart below).

Relative Rating Excellent Fair Poor
Study Design Statistically rigorous study design with reference group or randomized experiment and control Cross sectional study or other coefficient based analysis Simple before / after study
Sample Size Large sample, multiple years, diversity of sites Moderate sample size, limited years, and limited diversity of sites Limited homogeneous sample
Standard Error Small compared to CRF Relatively large SE, but confidence interval does not include zero Large SE and confidence interval includes zero
Potential Bias Controls for all sources of known potential bias
*See below for a list of potential biases
Controls for some sources of potential bias No consideration of potential bias
Data Source Diversity in States representing different geographies Limited to one State, but diversity in geography within State (e.g., CA) Limited to one jurisdiction in one State

To provide a more quantitative translation from these categories to the star rating, a point-based system was developed. Points are assigned to each CMF characteristic based on the level of rigor (excellent = 2 points, fair = 1 point, or poor = 0 points). While the points decrease from excellent to poor, not all characteristics receive equal weight. For example, the study design is more important than the data source. Therefore, the final quality rating is based on a weighted score. Study design and sample size categories receive twice the weight of the other characteristics (see equation below).

Score = (2 * study design) + (2 * sample size) + standard error + potential bias + data source

The star rating is assigned based on the score and the ranges in the table below. It should be noted that information may be missing from a study report for specific characteristics such as sample size. In these cases, the rating is based on available information and the CMF will likely receive a lower rating due to the lack of information.

Score Star Rating
14 (max possible) 5 Stars
11 – 13 4 Stars
7 – 10 3 Stars
3 – 6 2 Stars
1 – 2 1 Star
0 0 Stars

 

Potential biases for cross-sectional studies may include:

· Control of confounders
· Unobserved heterogeneity and omitted variable bias
· Accounting for state-to-state differences if using multiple states
· Selection of appropriate functional form
· Correlation or collinearity among the independent variables
· Overfitting of prediction models
· Low sample mean and small sample size
· Bias due to aggregation, averaging, or incompleteness in data
· Temporal and spatial correlation
· Endogenous independent variables
· Misspecification of structure of systematic variation and residual terms
· Correlation between crash types and injury severities

The following biases should be addressed for before-after studies:

· Regression-to-the-mean
· Changes in traffic volumes
· History trends
· Other safety treatments
· Changes in crash reporting
· Accounting for state-to-state differences if using multiple states
· Suitability of comparison or reference groups

The information contained in the Crash Modification Factors (CMF) Clearinghouse is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the use of the information contained in the CMF Clearinghouse. The information contained in the CMF Clearinghouse does not constitute a standard, specification, or regulation, nor is it a substitute for sound engineering judgment.