Table 3

Methods of validation for the USP cases

DomainIndicatorData collectionStatistical analysis
PhaseMethod
Content validityContent Validity Index (CVI)USP case reviewExpert panel review of SP cases, measured by a 4-point Likert scale (1=lowest, 4=highest).CVI for SP case and for specific USP, where CVI=number of raters giving a rating of 3 or 4 divided by the total number of raters.
Face validityAuthenticity of SP role-playValidation studyTranscripts of the recording of the USP–clinician encounter to be assessed by a member of the project team for accuracy of portraying the clinical case by a 5-point Likert scale (1=100% inaccurate, 5=100% accurate).Accuracy score=per cent of positive evaluations (ie, evaluation ≥4).
Detection ratioClinicians receiving an SP visit to complete a ‘detection form’ afterwards to report any suspected USP visits: 0=not suspected; 1=somehow suspected; 2=suspected with certainty).Detection ratio=number of detected USP visit divided by the total number of USP visits (for case-specific detection ratio and all-case detection ratio, respectively). Detection ratio of 10% and less is considered acceptable.
Criterion validityLin’s concordance correlation coefficient (rc); kappa statisticValidation studySP-completed checklist against that by a clinician based on the transcript of the visit (ie, the clinician rating as the ‘reference standard’).The concordance of the quality scores based on SP-completed checklist against that based on the reference standard. rc used for continuous process quality scores, and kappa for dichotomous diagnoses and treatment and management measures.
Test–retest reliabilityLin’s concordance correlation coefficient (rc); kappa statisticValidation study.The same SP to score his own recorded encounter in a month.The concordance to be examined by rc for continuous process quality scores, fees charged (yuan) and time spent (min), and kappa for dichotomous diagnoses and treatment and management measures.
Inter-rater reliabilityMultiple SPs to complete the checklist for the same recorded transcript.

  • SP, standardised patient; USP, unannounced standardised patient.