HZ Assessments

Frequently Asked Questions

How can we tell if a credential is credible?

A recent article entitled "What's behind that credential?" published in the American Psychological Association's Monitor on Psychology (September 2000, Volume 31, Number 8) discusses this very issue. The author approaches the topic from a logical perspective by asking key questions about:

Integrity of the organization granting the credential
Credibility of the individuals who are currently certified by the organization
Linkage between the credentialing examination and a practice (or task) analysis
Expertise of the professionals who developed the examination
Documentation of the procedures used to construct and validate the credentialing examination
Eligibility requirements to sit for the credentialing examination
Mechanisms for determining continued competence

The author comments that viable organizations should be willing to provide detailed materials for prospective certificants that include a comprehensive summary or outline of the knowledge or functions covered on the examination.

How do we know if our certification program meets legal, technical and professional standards?

There are five benchmarks of a defensible certification program in terms of legal, technical and professional standards.

First, the examination should be based on a practice (or task) analysis. A practice analysis is a comprehensive description of required job competencies that adheres to industry standards such as the Standards for Educational and Psychological Testing (1999).

Second, the examination should be linked to competencies defined in a practice analysis. The practice analysis will result in a blueprint or specifications from which the examination is constructed. Every examination will contain specific percentages of questions from the same topic areas and thus have consistent content.

Third, the persons who develop the questions for the examination must be knowledgeable of the procedures for writing questions and have a command of the subject matter to be tested. There are specific formats for writing questions so that they can be universally understood regardless of where persons were trained or educated.

Fourth, the passing standards must be based on a standard of competence related to success in the profession.

Fifth, there is a mechanism for continuous improvement. There may be procedures for statistical evaluation of test results and administrative processes as well as comprehensive review of eligibility, training and experience requirements.

Who should be recruited to participate in the development of a certification program?

The group of individuals who write the test questions should reflect the level of expertise of the persons who are qualified to obtain the credential. The best group includes persons familiar with expertise of entry-level persons as well as educators who are charged with responsibility of providing training. The key to recruiting a balanced group of individuals is to tap multiple sources, e.g., professional organizations, committee members, academic organizations, and management.

We have been directed by our agency to update the content of our examination. We would like to conduct a practice analysis, however, our licensee population is relatively small (<1000). How can we develop content specifications for a new examination?

Focus groups can provide a cost-effective solution that maximizes use of resources and practitioner participation. Research shows that outcomes from focus groups closely match outcomes from large-scale studies (Tannenbaum & Wesley, 1993). A typical focus group has 6-8 licensees that are representative of the experience and practice specialty mix of the profession.

The success of your focus groups depends largely upon the expertise of your testing consultant. The consultant should have expertise in your profession as well as expertise with credentialing examinations. You should expect the consultant to review existing practice analyses of the profession and your laws and regulations governing your profession.

An experienced consultant can synthesize the materials into a format that is conducive to group discussion of the essential components of the profession (tasks and knowledge) and their relative importance of each content domain (weight).

How can we produce the highest quality test items? We had licensees with a great deal of experience in the field write questions, yet our candidates complain that many of the items are poorly worded and have more than one correct answer.

We had licensees with a great deal of experience in the field write quesitons, yet our candidates complain that many of the items are poorly worded and have more than one correct answer. There are no shortcuts for producing quality items. Developing items is time-consuming, and, of course, costly. The cost per item is driven upward because many of the items do not survive review and pretesting.

There are some procedures, if followed, improve the quality of items and increase the number of useable items.

Develop items according to content specifications. Close adherence to the content specifications will increase the number of items that survive since the items are likely to be viewed as relevant.

Select focus group participants that are representative of the profession, e.g., geographic location, practice specialty, experience, etc.

Provide thorough training for the item writers. At a minimum the training should encompass a full day orientation, initial item development, group feedback to item writers before the item writers are ready to produce quality items.

Have a focus group conduct an independent review of the items. Independent review is essential to critical evaluation of the item for technical accuracy and relevance to job-related contexts.

How should we select the items for a new examination? It seems like a simple enough task.

The best way to publish (select items) an examination is to use a focus group. Publication of an examination may seem like a simple job for one or two people, however, there are a number of factors that should be considered when selecting items.

Does the item assess entry-level minimum acceptable competence?
Is the content of the item job related?
Is the wording of the item clear and unambiguous?
Is the content of the item assessed in other content domains?
Are the percentages of items selected from the content domains consistent with the content specifications?