I. REVIEW AND APPROVE RECOMMENDATION REGARDING THE RELEASE OF
CALIFORNIA SUPPLEMENTAL EXAMINATION (CSE) RESULTS
Mr. Gutierrez asked Justin Sotelo to present this agenda item.
Mr. Sotelo stated that at the previous two Board meetings there had been discussion about the release
of CSE results to candidates. He stated historically, oral examination results had been released to
candidates 30 days after the date of their examination, and periodically there were wait periods for
results due to necessary statistical analyses. He added that it was now possible with the test vendor,
Psychological Services, LLC (PSI), for results to be released to candidates at the test center
immediately after their exam is finished.
Mr. Sotelo reminded the Board that at its September 15, 2011, meeting the Board voted to continue
releasing results after 30 days until February 1, 2012, (one year after the launch of the computerdelivered examination), and then begin releasing results immediately to candidates at test centers.
Additionally, he stated the Board asked staff to provide a recommendation at its December 7-8, 2011,
meeting on the release of the CSE results based on the issues discussed at the September 2011 Board
meeting. Mr. Sotelo informed the Board that staff met with the exam development vendor, Office of
Professional Examination Services (OPES), and due to the items/issues listed on the agenda item
cover sheet, staff recommended beginning the immediate release of CSE results to candidates
effective on June 1, 2012. He also added that periodic statistical analysis would still be required in
the future which would occasionally cause CSE results to be held, but that candidates are made aware
of this in the CSE Handbook and on the Board’s website.
Mr. Gutierrez clarified the recommendation as presented by Mr. Sotelo as asking the Board to
consider moving the implementation date of the immediate release of CSE results from
February 1, 2012, to June 1, 2012. Mr. Sotelo confirmed Mr. Gutierrez’ clarification as accurate.
Mr. Baker strongly disagreed with the recommendation and stated that there are numerous multiplechoice exams for certificate programs where the candidates receive their results immediately after
finishing the exam. He indicated the CSE was different from the Architect Registration Examination
(ARE) graphic divisions and does not believe the immediate release of CSE results should be
postponed. He stated his belief that the information from the statistics is being over analyzed by the
test vendor and unnecessarily delaying the release of CSE results to candidates. He opined that
candidates should be provided results the minute they finish the CSE and it should commence
immediately. Ms. Lyon agreed with Mr. Baker’s conclusions and opinion and stated the process for
immediately releasing results is taking too long.
Vickie Mayer explained that the vendor would have to complete some required programming to
begin immediate release of CSE results and it could potentially be finished by February 1, 2012. She
further explained that because of the timing for launch of the next new test form, the immediate
release of CSE results would only occur for approximately 30 days before a new analysis phase
would begin due to the release of the new March exam form, which will delay the release of CSE
results for 60 to 90 days while the analysis is being performed. She suggested that if the Board did
not wish to have the new test items analyzed, then it could be possible to begin the immediate release
of CSE results February 1, 2012.
Mr. Baker asked why new test items, which are only released with every new version of the CSE,
could not be introduced as nonscorable items during prior test administrations and separately
analyzed for performance, thus allowing those items that do count to be scored more timely.
Ms. Mayer explained that during each release of the CSE there are approximately 100 new scorable
items and a certain number of nonscorable items, which must be analyzed. She added that the
analysis process requires a minimum number of candidates before the analysis can be performed,
after which the results can be released to candidates. She further explained that during the release of
the first CSE test form, a pool of 400 candidates was necessary before conducting the analysis.
Ms. Mayer stated there was an issue in reaching this amount of candidates and that created the delay.
She added that the pool of candidates was reduced in the release of the second test form, but still
required more than 60 days to complete the analysis due to issues that arose. Ms. Mayer further
stated the next test form is scheduled for March 2012 and would also require an analysis phase.
Mr. Baker asked why the new form (Form C), which will be launching in March, could not use
previously tested and analyzed questions from the previous test forms. Ms. Mayer explained that
there are an insufficient number of items in the test item bank. She stated the plan was to launch
Form A followed by Form B, and then concurrent with the release of Form C reintroduce Form A.
She further stated, the Form A reintroduction may be delayed to allow the development of a sufficient
candidate pool for the Form C statistical analysis. Ms. Mayer indicated that gradually a sufficient
pool of test items would be developed and there could be a reduced need for future statistical
analyses. She stated this approach was agreed to by the Examination Committee and PSI.
Mr. Baker asked how frequently the CSE is administered and how long a candidate must wait to
retake the CSE should they fail. Mr. McCauley responded that the CSE is continuously administered
and that candidates must wait six months before it may be retaken. He added this is similar to the
National Council of Architectural Registration Boards’ (NCARB) ARE requirement. Mr. Baker also
asked how over exposure could occur when candidates must wait at least six months between retakes
and why were new test items not included as part of the pre-test to assist in developing the test bank.
Ms. Mayer responded that new test items are being pre-tested during CSE administrations. She also
informed the Board the number of candidates who are taking the CSE is declining, which can
contribute to a longer analysis process.
Mr. Baker expressed his concerns about the process for how the CSE is being administered and
cycled. He stated a better plan should have been developed for how new test items would be
integrated with existing ones so it would be easier to initialize the immediate release of CSE results to
candidates. Ms. Mayer explained that OPES stated the analysis may eventually no longer be
necessary after a sufficient test bank had been developed and is performing adequately. Mr. Baker
stated his displeasure with OPES for having initiated a process where entirely new test forms are
being administered and a year later candidates still cannot get their results more timely.
Mr. Gutierrez asked how many times the release of CSE results could be interrupted by analysis
should the Board delay the implementation of the immediate release of CSE results until June and is
it conceivable the delay or the analysis could be eliminated altogether by that time. Mr. Sotelo
responded that it is recommended an analysis be performed each time a new test form is released and
depending on how the test bank develops and its performance, there may be a reduced necessity for
analysis. Ms. Mayer added that staff could contact the vendor to see whether the analysis could be
eliminated. She indicated, however, that there could arise potential issues related to scoring and test
Mr. Gutierrez stated that candidates may not necessarily be impacted by delay should the Board
decide to wait until June for the immediate release of CSE results. He added that if the test vendor
can grant assurances there would be no further delays after June 1, 2011, then the Board could show
candidates how the CSE has evolved into a modern computer-delivered exam with no waiting for
results (streamlining the licensing process) and is a vast improvement over the older format.
Kurt Cooknick posed two questions to the Board: 1) how would the Board communicate information
about the immediate release of CSE results to candidates; and 2) what about reciprocity candidates
for whom receiving a commission may depend on the immediate release of CSE results.
Mr. Gutierrez responded to Mr. Cooknick‘s second question by indicating there are alternate methods
an individual can utilize to work on a California project such as partnering with a California licensee.
Mr. Baker made a recommendation that staff work with the test vendor to develop an improved
process for implementing new test forms and test items to prevent future delays in the release of CSE
results to candidates. He opined that a new process should allow for multiple forms to be
simultaneously administered without the need to withhold results for statistical analyses. Ms. Mayer
added that holding results is not unique to the CSE and that NCARB also does not immediately
release test results.
Sheran Voigt moved to accept the recommendation to release CSE results on-site at the test
centers beginning June 1, 2012, and request the test vendor to develop a process for future
analysis of test items that does not create an interruption or delay in the release of results.
Marilyn Lyon seconded the motion.
Mr. Cooknick stated the Little Hoover Commission just released a report about how the regulatory
process in California is hampering business. He added that primarily candidates taking the CSE now
have an expectation the CSE results should be immediately released when their exam is finished. He
also added these types of candidates view the delay in releasing results an unnecessary artificial
hindrance in obtaining their license and advancement.
Mr. Baker stated his disagreement with the motion. He added that the benchmark of “it’s better now”
is unacceptable. He noted the release plan developed by OPES is ineffective and a better plan which
would have minimized or eliminated the delay to candidates for their results should have been
implemented. He opined the delay to candidates was completely avoidable. Ms. Mayer explained
that the vendor believed there would be a higher number of candidates testing in order to begin the
statistical analysis. Mr. Baker further added that candidates are getting licensed in other states and
later taking the CSE because of the artificial delays and the bureaucratic process in California.
Mr. Gutierrez offered an amendment to the motion “requiring” the test vendor to develop a
process for future analysis of test items rather than “request.” Marilyn Lyon accepted this
amendment to the motion.
The motion passed 5-1 (Jon Baker opposed).