USCIS Naturalization Test Redesign — Flaws in Design and Transparency
Excerpt of remarks prepared for a webinar on September 20, 2023, offered by the Immigrant Legal Resource Center. My thanks to ILRC Executive Director Eric Cohen and Senior Staff Attorney Peggy Gleason for the opportunity to join them for this session focused on changes that will make the English/Civics evaluation more difficult for applicants and raise new barriers to naturalization. A free recording of the complete webinar is available from ILRC.
Can you tell me how low-literacy adults prepare for the English/civics requirements for naturalization?
For the English speaking requirement, applicants practice answering questions about their own personal information as it appears on the N-400 citizenship application form. They also practice very basic communication skills for the USCIS interview, everything from “Raise your right hand” to “Show me your identification”.
For the reading requirement, applicants study a very short official list of words and practice reading them aloud in sentences like the ones that will be on the reading test. Some low-literacy applicants memorize these as sight words; others decode and sound out the words if they can recognize letter-sound correspondences.
For the writing requirement, applicants study another very short list of words and practice writing sentences with them like the ones that will be on the writing test. Some low-literacy applicants prepare using handwriting worksheets where they practice tracing and copying the letters of the alphabet to make sure that their writing is legible.
For the civics requirement, they usually rote-learn each of the 100 questions and an acceptable answer for each one. If their reading skills are limited, they can practice just by listening to the questions and speaking the answers. With basic reading skills, they might use flash cards, an app, or study sheets with questions and answers in two columns left and right. Or without basic reading skills, they can use those materials by having a family member, friend, neighbor, or co-worker practice with them. Just using their emerging listening and speaking skills — and the ability to memorize — low-literacy applicants often do very well on the current civics exam.
How would you evaluate the level of difficulty of a written multiple choice civics test compared to the current oral evaluation of civics?
There are at least two factors to consider here — first, that it’s a written test; and second, that it’s in a multiple-choice format. I’ll focus on the written aspect first. The proposed civics exam would require reading comprehension skill at a significantly higher level of English proficiency and reading vocabulary knowledge than is currently required for naturalization. Keep in mind there’s already the reading test I just described. A written civics test would therefore be a second reading test — and a much more difficult one.
To emphasize again: The current reading test is an oral test that evaluates an applicant’s ability to read aloud short sentences composed of a limited number of words. USCIS provides a study sheet that lists the words — just 64 of them — common words for people, places, some verbs, and some simple civics terms. Applicants can study the words, and even if they have limited ability to read in English, they may be able to recognize and say them as sight words that they’ve memorized. Or, if they have some very basic reading ability in English, they can decode and sound out the words by applying their knowledge of letter-sound relationships — much the same way that a child begins to read.
In contrast, a reading-required civics test — if based on the current 100 questions and answers — could potentially contain over 500 words — up to eight times more than the current read-aloud oral reading test. And these words would be much more difficult.
Let me share with you a yardstick for measuring how much more difficult a reading-based civics test would be. The US Department of Education has a National Reporting System for describing different levels of English proficiency. In my opinion, the level of English required for applicants to pass the current oral reading test is ESL Level 2 — Low Beginning English as a Second Language. It involves some basic reading skill and limited knowledge of simple and familiar vocabulary. In the current test, that vocabulary consists of the short list of words on the USCIS study sheet.
But a written, reading-based civics exam would be significantly more challenging. Reading skill would be a prerequisite and determine if the applicant can demonstrate the required civics knowledge. Using the National Reporting System yardstick, I believe the proposed civics exam would involve a two-level jump from Level 2 reading proficiency to Level 4. Why two levels? Well, the reading skills at Level 3 — High Beginning ESL — only involve sight words, common words, familiar phrases, and simple sentences. However, written exam questions about civics would involve reading subject matter vocabulary about U.S. government, history, geography, and civics. This goes way beyond common vocabulary and is at least at Level 4 — the Low Intermediate ESL Level.
What impact do you think this would have on low-literacy adults? Why?
Many low-literacy adults who are able to pass an oral civics exam using their basic listening and speaking skills won’t have the reading skills to pass a written version. Consider the number of hours of instruction it takes for students to move from one level of English proficiency to the next. In Los Angeles, for example, the adult education program provides up to 200 class hours for each level. If applicants need to advance one or two levels in reading skill to succeed on a written civics exam, that could set them back a year or more on their path to citizenship. And acquiring the requisite skills will be especially challenging for adults who have limited reading proficiency in their native language, or whose native language doesn’t use our Roman alphabet.
Thanks to our experience with students who were legalization applicants in the 1980s under the Immigration Reform and Control Act, our profession has developed the curricula and instructional techniques to prepare low-literacy adults for the current civics exam. If we’re faced with preparing students for a new written civics exam, I fear that we’ll set the clock back to the days when citizenship prep programs wouldn’t even allow students to enroll until they had already reached a high-beginning or intermediate level of reading ability so they could prepare for the test.
And from a historical perspective, implementing a higher-level reading requirement for the civics test harkens back to Reconstruction-era literacy tests in some southern states, or New York State’s amendment in 1921 that established English literacy requirements to vote. Hanging on my office wall is the New York State certificate issued to my immigrant grandfather after he passed his literacy exam in 1923. It reads, in the lower-left corner: “Keep this Certificate of Literacy. Show it to the election inspectors.” Next month, that certificate will be 100 years old. We shouldn’t go back to that time.
Does the current evaluation of English/civics in naturalization require an applicant to have standardized test-taking ability?
No, it doesn’t. The English evaluation happens during the normal course of the interview — it isn’t a separate test. And as I’ve described, the civics test is a set of simple direct questions with one or more acceptable answers. The examiner asks a question; the applicant gives an answer orally.
Many students consider preparing for the civics exam as being similar to learning the catechism of their faith — but in this case, it’s a set of 100 questions and answers about civics that they can commit to memory and recite orally to demonstrate their basic knowledge of US government, history, national holidays, and geography. Compared to taking a standardized test, this is straightforward, and it’s oral.
Also, I should mention that many applicants who are English learners take great pride in being able to recite these basic facts in their new language as they prepare to become citizens.
Would the redesigned test require applicants to learn new test-taking skills? Does the law require that for naturalization?
It would absolutely require new test-taking skills. And no, the law doesn’t require this.
If the reading requirement is the first key problem with the redesigned civics test, the second major problem is the language and cognitive skill required for low-level applicants to deal with the test’s multiple-choice format. This involves reading through a question and multiple answer alternatives, including a correct answer and distractors — the incorrect choices. This format arguably evaluates the applicant’s test-taking skills as much as it evaluates civics knowledge. These skills include predicting an answer from the question in order to evaluate the answer alternatives as you read them, eliminating distractors that you know are incorrect, and then selecting an answer, or if necessary, guessing. Preparing for a civics exam in this contrived format would involve taking practice tests that include civics answers that are correct and, by multiples, other answer choices that are wrong.
Furthermore, the amount of civics content that applicants need to know to succeed on a multiple-choice test would exceed the current civics knowledge requirement. The current test includes 100 questions and answers. In a multiple-choice format, applicants would need to know the correct answer but would also need to read and understand the distractors — the incorrect answers — in order to rule them out. And this would become even more challenging and confusing if USCIS proceeds with a plan to construct the test items so that distractors in one question would contain vocabulary that appears in other questions or answers on the exam.
Let’s take one current question as an example: “Name one war fought by the United States in the 1900s.” There are five acceptable answers. Let’s say that the applicant has practiced one of them: “the Vietnam War”. A contemplated multiple-choice version of this question might be: “Name one war fought by the United States in the 1900s”, with three answer choices:
A. the Civil War
B. the Spanish-American War
C. the Vietnam War
The applicant would need to know that the distractors are incorrect because they are possible answers for a different civics question about wars in the 1800s.
So, understand the task of the item writer when constructing a multiple-choice civics test: Your job is to surround correct civics facts with a greater number of plausible but incorrect distractors. Compared with the current simple and straightforward direct-question and short-answer civics exam, this format is more complicated and difficult to prepare for, and applicant performance is as likely to be an indicator of test-taking skill as civics knowledge. And no, this is not a skill required for naturalization.
Would the redesigned civics test require an applicant to have a greater knowledge of civics content than is required currently?
It could, but not necessarily. One of the concerns about the rescinded 2020 version of the test was that it included a lot of new civics content that was considered more difficult. Whether that happens in the current revision depends on how many current civics exam questions are replaced, what the content consists of, and whether the list of questions remains at 100 or if it increases. For the current proposed pilot testing, USCIS is planning to use the existing questions as the content — so the concerns right now are about the test formats, not content.
In fact, the current test, with its multiple acceptable answers for many of the questions, allows citizenship education programs to better develop their students’ civics knowledge. Take for instance one of the key questions on the current test: “What is one right or freedom from the First Amendment?” Several correct answers are possible: Freedom of speech, religion, assembly, freedom of the press, and the right to petition the government. A good civics lesson would include them all, and the applicant can choose which one to use if asked that question. But in a multiple-choice version of that kind of question, the test item writer would choose one of those freedoms and then surround it with incorrect answers.
But to get back to your question: Yes, there’s the potential for new civics questions — or an increase in the number of possible questions — to require a greater amount of civics knowledge.
What impact would this have on adult learners who have had few years of formal schooling but who are eager to naturalize?
This would add to the level of difficulty. But to be honest, a little more civics content would not be as imposing a barrier as the standardized testing formats and reading requirements that are being proposed in the redesign.
Describe the redesigned photo-based English speaking test.
The speaking test would consist of photographs from everyday life — such as weather, food, shopping, and daily routines. Applicants would need to describe the photos using simple words and phrases, or short sentences. The expectation is that to pass the speaking test, applicants would need to score at High Beginner ESL Level 3.
Does this format require a greater knowledge of English than is currently needed to naturalize?
Yes, it does. First, keep in mind that this format would add an English testing requirement that doesn’t currently exist. The current evaluation of an applicant’s English speaking skills occurs during the normal course of the interview as the officer asks for and verifies information pertinent to the application for citizenship.
An English speaking test in which applicants describe photographs showing life situations wouldn’t align with the authentic context of the naturalization interview. It would require applicants to study general vocabulary to answer questions not related to civics or the application for citizenship. Picture-prompt speaking tests like this are commonly used to place students into general English language classes or evaluate their progress, but they don’t elicit the level of meaningful performance of English skills that applicants demonstrate when communicating their most important personal information during the normal course of the interview.
As I mentioned earlier, applicants at Low Beginning ESL Level 2 have sufficient skills to pass the current oral read-aloud test — if they adequately prepare. In a similar manner, applicants at this level can pass the current English speaking evaluation if they practice the listening and speaking skills that are needed during the interview. According to the National Reporting System standards, at Level 2 they can understand simple commands and questions related to personal information, and they can use vocabulary to meet immediate needs — in this case, the interaction with the officer. They can answer all questions satisfactorily using phrases and incomplete sentences.
In contrast, a photo-based speaking test would include vocabulary and topics beyond the applicant’s immediate needs, and this puts us into the content of High Beginning ESL Level 3. In addition, I should warn that scoring rubrics for such speaking tests tend to expect Level 3 examinees to show some limited ability to use English grammar correctly. But grammatical correctness has no bearing on the ability of applicants to use English sufficiently during the interview. And therefore, it shouldn’t factor at all in evaluating English ability for naturalization.
I should also mention that educators are raising concern about the cultural content that is included in any photographs used in a test, and the potential for wide variation in how a photo is seen and described by applicants from different backgrounds.
As an alternative to a photograph-based test, our Citizenship Test Working Group proposed to USCIS two years ago that the existing English-speaking evaluation should be retained, but improved, by standardizing some of the key interactions between the applicant and the officer that serve as a basis for evaluating English proficiency. Officers can be provided with a list of questions that elicit basic personal information — asking for or verifying the applicant’s name, address, date of birth, country of birth, country of citizenship or nationality, contact information, family members (parents, children, spouse), marital status, current employment, and current residence. Paraphrased versions of the questions should also be provided. And for purposes of the English evaluation, we recommended that these questions should NOT include any of the Form N-400 Part 12 topics — the “Have you ever…” questions and others that are difficult to answer due to their very difficult, rarely used, and often legal vocabulary.
As part of keeping but standardizing the existing English evaluation, a simple scoring rubric could be used to evaluate the applicant’s ability to communicate information meaningfully and sufficiently — without regard to completeness of sentences or control of grammar. And a training module for officers should be developed to implement these improvements and to provide periodic retraining to assure consistency and reliability.
So you see, some of the most important aspects and benefits of standardized testing — set questions, consistent paraphrased questions, scoring guidance, and officer training — all of these can be implemented as improvements while retaining the current English evaluation rather than replacing it.
What do you think the impact would be of having a test that is separate from the naturalization interview?
If the English/civics test is removed from the interview, the questions are: Where would it occur? Who would administer it? And what would be the logistics and costs of this in terms of time and expense? Would a separate test require a separate appointment at a different location? Would these locations be operated by USCIS or by a private vendor? And given how far applicants might have to travel, why would we add a new appointment for a test that takes just a few minutes during the interview, especially at a time that we’re trying to streamline naturalization with same-day oath ceremonies?
How do you think applicants will perform on tests that are delivered via technology?
The technology will probably be a computer or tablet device that presents one question at a time on a screen. For many applicants, this wouldn’t be a problem. For many others with low literacy skill or less experience with technology, interacting with a device will present a new barrier to naturalization. For the civics exam, applicants would tap a button to answer a multiple-choice civics question and then tap an arrow to move to the next question. For each question in the English speaking exam, applicants would tap a record button to record their answer, and then tap an arrow to submit their response and move to the next question. So there would be a good amount of interaction with the device’s interface in order to take the exams. For many of the applicants we’re most concerned about due to low-literacy levels, this would raise the barrier even higher.
Can you describe how USCIS developed the format for the redesign?
According to USCIS descriptions of this during engagements, the redesign was developed in-house by staff with subject matter expertise — with the stated goal being to align the English/civics evaluation with best practices in standardized testing. Some existing adult education tests were used as models for developing the reading-intensive multiple-choice civics exam and the photograph-based speaking test. This is at the root of the problem we’re facing — the premise that the existing 2008 evaluations conducted during the naturalization interview should be replaced by standardized testing of any sort.
Was there adequate stakeholder input before the format was developed and announced?
No, not at all. The test redesign process currently underway arguably repeats mistakes of the current 2008 test’s first three years of development from 2001 to 2004, and in some respects it’s worse. Back in 2001, Legacy INS contracted with an assessment company to revamp the naturalization test formats and content. This resulted in a contemplated civics test that contained multiple-choice questions, and an English test consisting of photographs. Some citizenship service organizations were invited to provide input to the contractor, and many of my education colleagues and I had the opportunity to review sample test materials and offer feedback to INS staff directly during their attendance at professional conferences. The prevailing critique was that the test design was faulty in content and format, and it would impose new barriers to naturalization.
So as a result of considerable pushback, the newly established USCIS contracted with the National Academies Board on Testing and Assessment in 2004 to evaluate the redesign process. The Board formed a committee that was so concerned that it issued an unplanned interim report in December that year. It called for suspending any further work on test development until a better process was in place that was, in the words of the report, more “open, transparent, and accountable.” A follow-up letter report from the DHS Office of Inspector General agreed on the main points of concern, and the test development was suspended.
The outcome of that three-year mis-start was a much-improved redesign roadmap starting in mid-2005 under the Office of Citizenship. This included a significant number of stakeholder meetings and discussion group sessions, substantial input by subject matter specialists into the design of test formats and content prior to pilot testing, and ongoing internal USCIS working group meetings including representatives from Field Operations, Policy and Strategy, and General Counsel.
I attended many of the stakeholder sessions, and one of the most valuable aspects of these meetings was the participation of Field Operations staff from many different offices around the country. Having frontline field personnel at the table with service providers during these meetings resulted in important reality-checks about the logistical implications of test recommendations for interview length, officer training requirements, and the impact on overall application processing. The current 2008 test is the result of this very careful development process.
In our current situation, not only was there no stakeholder input into the test format decisions, there was also no Technical Advisory Group of external subject matter experts available to guide the process. The TAG was formed after the formats were decided.
What have you observed in the public participation in this process overall?
It hasn’t been adequate or appropriate. Let’s start with the Federal Register Notice announcing the trial test process on December 15 of last year. The notice directed public comments to be sent to an internal email inbox set up for the redesign process. So the comments may be from the public; but they aren’t public. Stakeholders can’t review that input or potentially engage with others who have weighed in. For transparency, it would have been better to start with a publicly available process. USCIS should implement one now for communication going forward.
Stakeholder input is also problematic for at least three other reasons:
First, as I’ve already mentioned, it’s too late. The test formats have been redesigned in-house already.
Second, although USCIS has conducted strong outreach and scheduled many engagements about the redesign, these have essentially been information sessions to explain and answer questions about the already-established pilot test formats and procedures. These are not, to date, an opportunity or invitation for stakeholder input that would change the direction of the pilot. For example, at various sessions, input has included suggestions that the pilot written civics test should have audio support, that there should be experimental and control groups of low-proficiency level applicants taking the current test or the revised test to determine the impact on pass rates, and that the English test should also be piloted with standardized personal information questions rather than photographs. At the present time, such suggestions are apparently too late for the redesign process.
A third concern is whether communication with stakeholders will occur prior to implementing a second-phase pilot test in field offices. The Technical Advisory Group is supposed to provide a report on first-phase pilot results. This should presumably include impact statements from the hundreds of community-based organizations serving as pilot testing sites, so there’s an opportunity to review how the test redesign has affected their students and their instructional programs, including pass-fail rates on the tests. But there’s no assurance that such impact statements — if they exist — will be shared with members of the Technical Advisory Group, and there’s no indication that the Group’s report will be shared with stakeholders prior to moving to a second phase of testing. This needs to happen. In fact, this was the urgency that led the National Academies Board to recommend suspending test development in 2004. The Board was concerned that the pilot would enter a second phase in field offices and then be difficult for implemented changes to be withdrawn.
Let me close with this recommendation: I believe there’s an opportunity for USCIS to reimagine the pilot testing before it begins and consider this a formative research phase for longer-term improvements in English and civics testing. The agency should consider pausing the pilot and reopening the test redesign process for educator and stakeholder input that can have an impact on rethinking decisions about formats, content, and the use of technology. For example, Canada is now giving eligible citizenship applicants an option to take their civics test via a computer, laptop, or tablet at home or in a library or other public location. Perhaps such options are worth exploring here. But we need to be realistic about this: If we don’t want to raise barriers to naturalization, we need to retain the current English/civics evaluation during the interview as an option for any applicant who can pass the existing requirements but would fail the new ones.
Bill Bliss has worked in English language and civics education since 1974 as a teacher, trainer, curriculum developer, and consultant, and has provided technical assistance on citizenship training and testing since the 1980s legalization program. He is the author of two civics courses: Voices of Freedom, for learners preparing for the naturalization process; and Basic Civics for U.S. Citizenship, a review text for secondary school students in states that require a civics exam for high school graduation. The views expressed in this article are solely those of the author.