How accurate are your residents’ AN-ACC assessments?

June 16, 2023 | AN-ACC

The AN-ACC Assessment Inter-rater Reliability Analysis report, published by the Department on 21 April 2023, prepared by Ernst & Young details the statistical checks undertaken to confirm the repeatability and reliability of AN-ACC assessment outcomes. The findings in this report support our recommendations to the sector and are empirical evidence that accurate remote assessments can be achieved in the forecasting of AN-ACC classes for our clients’ residents.

For some context, please note that the report presented the results of analysis conducted on 656 residents involving 1,312 individual assessments (2 assessors for each resident) occurring in November and December 2022. Across all findings, we note that where the report mentions “pure rates of agreement” or “exact agreement” between assessors, this result is an identical score from both assessors with no discrepancies between scores; hence, can be deemed an accurate assessment of the resident. The report differentiates “exact agreement” from “partial agreement”, which instead allows a higher degree of tolerance for disagreement and thus, includes small discrepancies in scoring between assessors.

When pooling “exact agreement” and “partial agreement” together, the report highlights that most agreement results were above the threshold to be considered “excellent”. This demonstrated that in situations where assessors do not agree exactly, discrepancies in scoring tend to be small.

However, it is worth noting that where discrepancies are seen, it is most often and most significant for the following assessments which also have larger scoring ranges compared to other assessments:

FIM Cognition
FIM Motor
Braden Scale and
BRUA.

Certain AMOs experienced lower “exact agreement” rates than others, and these tended to be the AMOs with the more “experienced” assessors (i.e., those who had completed >350 AN-ACC assessments to date). See Figure 1 below for the details.

^{Figure 1: Agreement statistics for each assessment by AMO (Source: AN-ACC Assessment Inter-rater Reliability Analysis report)}

When segmenting the agreement by assessor experience, the report found that overall agreement rates and “exact agreement” rates for all assessments were lower for “extensively” experienced assessors, largely driven by AMOs 1 and 5 (see Figure 2).

^{Figure 2: Agreement statistics for each assessment by assessor experience (Source: AN-ACC Assessment Inter-rater Reliability Analysis report)}

This same finding is also noted for the “RN only” profession pairing, which is the most common pairing across all assessments conducted – given the limited number of occupational therapists and physiotherapists conducting assessments (see Figure 3).

^{Figure 3: Agreement statistics for each assessment by assessor profession (Source: AN-ACC Assessment Inter-rater Reliability Analysis report)}

We have known for the last few months, through our own lived experiences, that some assessors are more accurate in their assessments of a provider’s residents than others. This is due to the professional judgement required to complete these assessments. However, to determine how accurate an assessor’s assessment is, requires the provider themselves to have a thorough understanding of the tool itself, forecast an AN-ACC classification of their resident using the different assessments, and keep a continuous record of an accurate representation of all residents in their facility at any given time.

The variations in scoring and accuracy of assessments based on AMOs, assessor experience, and assessment type support our focus on providers managing the AN-ACC assessment process at each of their facilities proactively. Managing this process well can have significant impacts on the operational viability and sustainability of providers, ensuring that providers can continue to provide the greatest care that they are capable of.

Want to find out more? Reach out to our team.