Aviation Security X-ray Image Interpretation Competency Assessment & Certification
In spite of all technological improvements and newest-generation equipment the human operator is still the last decision instance in the chain of aviation security procedures. And without appropriate measures he is the weakest link. In X-ray baggage screening the aviation security screening officer is the one to take the final decision if a passenger is allowed to take his baggage on board of the airplane or not. This decision is usually made on the basis of an X-ray image interpretation of the bag. And this is exactly where actions can be taken to turn the human operator into the strongest link: competency of X-ray image interpretation is the critical factor. X-ray image interpretation not only demands specific visual abilities, but is also highly dependent on training. Therefore, reasonable investments have to be made into selection, training, and certification of aviation security screening officers. In order to have aviation security screeners with the needed visual abilities for the task of X-ray image interpretation they have to be carefully selected with a pre-employment assessment. It is recommended to use an X-ray image test (e.g., the X-Ray Object Recognition Test, ORT) for this purposes, where these visual abilities are specifically tested. For more information on pre-employment assessment see the chapter on pre-employment assessment. Training is another important prerequisite for the enhancement and high-level-maintenance of X-ray image interpretation competency. After employment initial training is very important to increase X-ray image interpretation competency up to a certain level. Studies showed that on average 10-20 hours of adaptive computer-based training are needed to achieve a minimum level of x-ray image interpretation competency. Thereafter, recurrent computer-based training should be conducted to further improve and maintain x-ray image interpretation competency. For more information on training see the chapter on training. Competency assessment underpins the workforce certification process. The main aim of certification procedures is to ensure that adequate standards in aviation security are consistently and reliably achieved. Certification can be regarded as providing quality control over the screening process to that effect that a screener must be able to detect threat objects reliably.
The following international documents refer to certification and competency assessment of aviation security staff:
EU Regulation 2320/2002
ICAO Annex 17, 3.4.3
ICAO-Manual on Human Factors in Civil Aviation Security Operations (Doc 9808)
ICAO Human Factors Training Manual (Doc 9683), Part 1, Chapter 4, and Appendix 6, Appendix 32
ICAO Security Manual for Safeguarding Civil Aviation Against Acts of Unlawful Interference, Doc 8973, Chapter 4, I-4-45
ECAC Doc 30, Chapter 12, and Annex IV-12A
As specified in ECAC Doc 30 by the European Civil Aviation Conference and ICAO Annex 17, 3.4.3, individuals carrying out screening operations should be certified initially and periodically thereafter. ECAC Doc 30 by the European Civil Aviation Conference specifies three elements for initial certification of airport security screeners:
an X-ray image interpretation exam
a theoretical exam
a practical exam
Periodic certification should contain a theoretical exam and an X-ray image interpretation exam. Practical exams can be conducted if considered necessary.
Relevance of certification of aviation security screeners
Certification can not only be considered as providing quality control over the screening process, but also as a valuable basis for awarding personnel a qualification, measuring training effectiveness, improving training procedures, and increasing motivation. Certification data provides important information on strengths and weaknesses in aviation security procedures in general as well as on an individual basis. Furthermore, standardized certification can help in achieving international standardization in aviation security. However, this demand is very challenging since many countries, organizations, and companies develop their own certification and quality control systems. The goal therefore is to define an international standard for the certification process which allows that the level of performance is comparable between airports from different countries and also within one country.
Requirements for assessing competency
The main aim of the X-ray image interpretation task is for the screener to detect prohibited items within X-ray images of passenger baggage. Hit rates, false alarm rates and time per bag are important measures that can be used to assess the effectiveness of screeners at this task. To correctly detect a prohibited item in an X-ray image (i.e. a "NOT OK" answer for a bag containing a threat) counts as a hit. "NOT OK" responses on images that do not contain a prohibited item count as false alarms. The time to process each bag is also important as it helps determining throughput rates.
The results of an X-ray image interpretation test are very important for appropriate authorities, aviation security institutions, and companies. Moreover, failing a test can have serious consequences, depending on the regulations of the appropriate authority. Therefore, it is essential that the tests are fair, reliable, valid, and standardized. In the last 50 years, scientific criteria have been developed that are widely used in psychological testing and psychometrics. These criteria are essential for the development of tests for measuring human performance. In the following, a summary of the three most important concepts shall be provided: reliability, validity, and standardization. Reliability refers to the "consistency" or "repeatability" of measures and relates to the quality of measurement. If, for example, an IQ test yields a score of 90 for an individual today and 125 a week later, it is not reliable. Reliability can vary between 0 (no reliability) and 1 (perfect reliability). When applying a test for competency assessment purposes it is crucial to have a test featuring high reliability scores. Acceptable tests usually have reliability coefficients between 0.7 and 0.9. For individual performance to be measured reliably, reliability coefficients of at least 0.85 are recommended as minimum test statistics requirement. Another important factor that has to be accounted for is test difficulty. Especially if different tests are compared with each other it is important to pay attention to these attributes.
Two tests may feature similar reliability coefficients but completely different difficulty levels. Validity indicates whether a test is able to measure what it was intended to measure. Standardization involves administering the test to a representative sample of people in order to establish norms. For more detailed information on these three important concepts in testing please refer to the chapter General Information.
How To Conduct Certification
As mentioned above, certification of screeners should contain a theoretical exam and an X-ray image interpretation exam. For the periodic certification, practical exams can be conducted if considered necessary, unlike the initial certification, where it is required. The exams should meet the requirements of high reliability and validity and standardization.
The theoretical and the practical exam are provided and carried out by the national airport authority, X-ray image interpretation exams can be obtained from www.viaproject.eu. The theoretical exam should test the screener´s knowledge of the regulations on aviation security screening. Apart from national rules and specifications, individual airports may enunciate questions covering special conditions. This exam, normally in the form of a questionnaire, should be acceptably reliable. It stands to reason to develop the exam as a multiple choice questionnaire. Good questions with qualitatively high answer possibilities (including distractor answers) are the basis for a good questionnaire which differentiates between knowledgeable screeners and those who have limited knowledge.
The X-ray image interpretation exam can be adapted to the location in which a screener is employed, i.e., cabin baggage screening, hold baggage screening, or both. A so far rather neglected operational domain, which should receive more attention also regarding training and certification, is cargo and staff screening.
Since not every threat object always constitutes a threat during the flight, depending on where aboard the aircraft it is transported, screeners should be certified according to their location. The certification of cabin baggage screeners should be based on cabin baggage images which contain all kinds of threat objects that are prohibited from being carried on in cabin baggage (e.g., guns, knives, improvised explosive devices, and other threat items). Objects that are prohibited from being transported in the cabin of an aircraft do not necessarily pose a threat when transported in the hold or cargo. Furthermore, different types of bags are transported in the cabin and the hold, respectively. The certification of hold baggage screeners should be carried out using images of hold baggage. Hold baggage screeners only have to detect threat objects which are prohibited in the hold of an aircraft, like explosive materials. Similar specificities apply to cargo screeners. Persons working in more than one location should be certified with the respective test versions. The VIA projects provides two X-ray image interpretation tests developed for competency assessment and certification procedures, X-Ray Competency Assessment Test for Cabin Baggage Screening and X-Ray Competency Assessment Test for Hold Baggage Screening. Both versions of X-Ray CAT were developed on the basis of scientific findings regarding visual information processing. Image based factors influencing detection of (threat) objects, that is, viewpoint, bag complexity, and superposition, were accounted for in the design of the tests. An X-Ray Competency Assessment Test for Cargo is currently under development.
X-Ray CAT CBS
X-Ray Competency Assessment Test for Cabin Baggage Screening (X-Ray CAT CBS) is used for certification of personnel working in cabin baggage screening. The test consists of X-ray images of passenger carry-on baggage which have to be visually inspected on the monitor and judged if they contain a prohibited item or not. The prohibited items belong to four categories according to current threat image projection systems: guns, knives, improvised explosive devices (IEDs), and other threat objects.
X-Ray CAT CBS features excellent reliability coefficients with Cronbach´s alpha and split-half values of over 0.95 (for details about reliability see Statistical Background Information), and good validity. And with several thousands of data points that were collected meanwhile a representative reference group can provide a solid norm.
X-Ray CAT HBS
X-Ray Competency Assessment Test for Hold Baggage Screening (X-Ray CAT HBS) is used for certification of personnel working in hold baggage screening. The test consists of X-ray images of passenger baggage which contain only improvised explosive devices (IEDs). Since in hold baggage other threat objects like knives or guns do not pose a threat, the focus lies on IED detection. The task is the same as for X-Ray CAT, namely to visually inspect the images and to decide if they contain an IED or not.
Also X-Ray CAT HBS features excellent reliability scores with Cronbach´s alpha of over 0.90 and split-half values well above 0.85. Concerning validity and standardization the same applies as for X-Ray CAT CBS.
Screeners should be kept up-to-date regarding new and emerging threats. In order to verify whether this is consistently achieved, it is recommended that a recurrent certification should be conducted every year. The minimum threshold that should be achieved in the tests in order to pass certification should be defined by the national air transportation authority and should be based on a large and representative sample (see Statistical Background Information for detailed information about standardization).
Take Home Message
Even with the most expensive technology the human operator is the weakest link in the aviation security chain. But with appropriate measures like pre-employment assessment and specific training he can become the strongest link.
Certification procedures ensure that adequate standards in aviation security, especially in X-ray image interpretation are consistently and reliably achieved
Certification can be regarded as providing quality control in aviation security, especially over the screening process
Standardized certification can help in achieving international standardization in aviation security
The X-Ray Competency Assessment Tests (X-Ray CAT) for CBS and HBS were specifically designed as highly reliable and valid certification tools
Measures whether screeners know which items are prohibited and what they look like in X-ray images of passenger bags
Exist in domain-specific versions
Feature excellent reliabilities, good validity and standardized norms
Order Pre-Employment Assessment Tools for Free
If you are interested in implementing the VIA Project Quality Management Tools in your country/ airport please go to the Ordering Form.
ECAC Doc 30Annex IV-12A Certification criteria for screeners and ECAC Doc 30, Chapter 12, 12.2.3 Certification of security staff 220.127.116.11
ICAO Annex 17, 3.4.3 (Each Contracting State shall ensure that the persons carrying out screening operations are certified according to the requirements of the national civil aviation security program)
ICAO-Manual on Human Factors in Civil Aviation Security Operations (Doc 9808)
ICAO Human Factors Training Manual (Doc 9683), Part 1, Chapter 4, and in Appendix 6 - Guidance on recruitment, selection, training and certification of aviation security staff and Appendix 32 - Guidance on the use of threat image projection
ICAO Security Manual for Safeguarding Civil Aviation Against Acts of Unlawful Interference, Doc 8973, Chapter 4, I-4-45 (Recruitment, selection, training and certification of security staff)
MacMillan, N.A., & Creelman, C.D. (1991). Detection theory: A user´s guide. Cambridge: University Press.