California Supplemental Exam Preparation header image

I. REVIEW AND APPROVE RECOMMENDATION REGARDING THE RELEASE OF 

CALIFORNIA SUPPLEMENTAL EXAMINATION (CSE) RESULTS 
Mr. Gutierrez asked Justin Sotelo to present this agenda item. 
Mr. Sotelo stated that at the previous two Board meetings there had been discussion about the release 
of CSE results to candidates.  He stated historically, oral examination results had been released to 
candidates 30 days after the date of their examination, and periodically there were wait periods for 
results due to necessary statistical analyses.  He added that it was now possible with the test vendor, 
Psychological Services, LLC (PSI), for results to be released to candidates at the test center 
immediately after their exam is finished. 
Mr. Sotelo reminded the Board that at its September 15, 2011, meeting the Board voted to continue 
releasing results after 30 days until February 1, 2012, (one year after the launch of the computerdelivered examination), and then begin releasing results immediately to candidates at test centers.  
Additionally, he stated the Board asked staff to provide a recommendation at its December 7-8, 2011, 
meeting on the release of the CSE results based on the issues discussed at the September 2011 Board 
meeting.  Mr. Sotelo informed the Board that staff met with the exam development vendor, Office of 
Professional Examination Services (OPES), and due to the items/issues listed on the agenda item 
cover sheet, staff recommended beginning the immediate release of CSE results to candidates 
effective on June 1, 2012.  He also added that periodic statistical analysis would still be required in 
the future which would occasionally cause CSE results to be held, but that candidates are made aware 
of this in the CSE Handbook and on the Board’s website. 
Mr. Gutierrez clarified the recommendation as presented by Mr. Sotelo as asking the Board to 
consider moving the implementation date of the immediate release of CSE results from 
February 1, 2012, to June 1, 2012.  Mr. Sotelo confirmed Mr. Gutierrez’ clarification as accurate. 
Mr. Baker strongly disagreed with the recommendation and stated that there are numerous multiplechoice exams for certificate programs where the candidates receive their results immediately after 
finishing the exam.  He indicated the CSE was different from the Architect Registration Examination 
(ARE) graphic divisions and does not believe the immediate release of CSE results should be 
postponed.  He stated his belief that the information from the statistics is being over analyzed by the 
test vendor and unnecessarily delaying the release of CSE results to candidates.  He opined that 
candidates should be provided results the minute they finish the CSE and it should commence 
immediately.  Ms. Lyon agreed with Mr. Baker’s conclusions and opinion and stated the process for 
immediately releasing results is taking too long. 
Vickie Mayer explained that the vendor would have to complete some required programming to 
begin immediate release of CSE results and it could potentially be finished by February 1, 2012.  She 
further explained that because of the timing for launch of the next new test form, the immediate 
release of CSE results would only occur for approximately 30 days before a new analysis phase 
would begin due to the release of the new March exam form, which will delay the release of CSE 
results for 60 to 90 days while the analysis is being performed.  She suggested that if the Board did 
not wish to have the new test items analyzed, then it could be possible to begin the immediate release 
of CSE results February 1, 2012.  
Mr. Baker asked why new test items, which are only released with every new version of the CSE, 
could not be introduced as nonscorable items during prior test administrations and separately 
analyzed for performance, thus allowing those items that do count to be scored more timely.  
Ms. Mayer explained that during each release of the CSE there are approximately 100 new scorable 
items and a certain number of nonscorable items, which must be analyzed.  She added that the 
analysis process requires a minimum number of candidates before the analysis can be performed, 
after which the results can be released to candidates.  She further explained that during the release of 
the first CSE test form, a pool of 400 candidates was necessary before conducting the analysis.  
Ms. Mayer stated there was an issue in reaching this amount of candidates and that created the delay.  
She added that the pool of candidates was reduced in the release of the second test form, but still 
required more than 60 days to complete the analysis due to issues that arose.  Ms. Mayer further 
stated the next test form is scheduled for March 2012 and would also require an analysis phase. 
Mr. Baker asked why the new form (Form C), which will be launching in March, could not use 
previously tested and analyzed questions from the previous test forms.  Ms. Mayer explained that 
there are an insufficient number of items in the test item bank.  She stated the plan was to launch 
Form A followed by Form B, and then concurrent with the release of Form C reintroduce Form A.  
She further stated, the Form A reintroduction may be delayed to allow the development of a sufficient 
candidate pool for the Form C statistical analysis.  Ms. Mayer indicated that gradually a sufficient 
pool of test items would be developed and there could be a reduced need for future statistical 
analyses.  She stated this approach was agreed to by the Examination Committee and PSI.   
Mr. Baker asked how frequently the CSE is administered and how long a candidate must wait to 
retake the CSE should they fail.  Mr. McCauley responded that the CSE is continuously administered 
and that candidates must wait six months before it may be retaken.  He added this is similar to the 
National Council of Architectural Registration Boards’ (NCARB) ARE requirement.  Mr. Baker also 
asked how over exposure could occur when candidates must wait at least six months between retakes 
and why were new test items not included as part of the pre-test to assist in developing the test bank.  
Ms. Mayer responded that new test items are being pre-tested during CSE administrations.  She also 
informed the Board the number of candidates who are taking the CSE is declining, which can 
contribute to a longer analysis process. 
Mr. Baker expressed his concerns about the process for how the CSE is being administered and 
cycled.  He stated a better plan should have been developed for how new test items would be 
integrated with existing ones so it would be easier to initialize the immediate release of CSE results to 
candidates.  Ms. Mayer explained that OPES stated the analysis may eventually no longer be 
necessary after a sufficient test bank had been developed and is performing adequately.  Mr. Baker 
stated his displeasure with OPES for having initiated a process where entirely new test forms are 
being administered and a year later candidates still cannot get their results more timely.   
Mr. Gutierrez asked how many times the release of CSE results could be interrupted by analysis 
should the Board delay the implementation of the immediate release of CSE results until June and is 
it conceivable the delay or the analysis could be eliminated altogether by that time.  Mr. Sotelo 
responded that it is recommended an analysis be performed each time a new test form is released and 
depending on how the test bank develops and its performance, there may be a reduced necessity for 
analysis.  Ms. Mayer added that staff could contact the vendor to see whether the analysis could be 
eliminated.  She indicated, however, that there could arise potential issues related to scoring and test 
item performance. 
Mr. Gutierrez stated that candidates may not necessarily be impacted by delay should the Board 
decide to wait until June for the immediate release of CSE results.  He added that if the test vendor 
can grant assurances there would be no further delays after June 1, 2011, then the Board could show 
candidates how the CSE has evolved into a modern computer-delivered exam with no waiting for 
results (streamlining the licensing process) and is a vast improvement over the older format.   
Kurt Cooknick posed two questions to the Board:  1) how would the Board communicate information 
about the immediate release of CSE results to candidates; and 2) what about reciprocity candidates 
for whom receiving a commission may depend on the immediate release of CSE results.  
Mr. Gutierrez responded to Mr. Cooknick‘s second question by indicating there are alternate methods 
an individual can utilize to work on a California project such as partnering with a California licensee. 
Mr. Baker made a recommendation that staff work with the test vendor to develop an improved 
process for implementing new test forms and test items to prevent future delays in the release of CSE 
results to candidates.  He opined that a new process should allow for multiple forms to be 
simultaneously administered without the need to withhold results for statistical analyses.  Ms. Mayer 
added that holding results is not unique to the CSE and that NCARB also does not immediately 
release test results. 
 Sheran Voigt moved to accept the recommendation to release CSE results on-site at the test 
centers beginning June 1, 2012, and request the test vendor to develop a process for future 
analysis of test items that does not create an interruption or delay in the release of results. 
Marilyn Lyon seconded the motion. 
Mr. Cooknick stated the Little Hoover Commission just released a report about how the regulatory 
process in California is hampering business.  He added that primarily candidates taking the CSE now 
have an expectation the CSE results should be immediately released when their exam is finished.  He 
also added these types of candidates view the delay in releasing results an unnecessary artificial 
hindrance in obtaining their license and advancement. 
Mr. Baker stated his disagreement with the motion.  He added that the benchmark of “it’s better now” 
is unacceptable.  He noted the release plan developed by OPES is ineffective and a better plan which 
would have minimized or eliminated the delay to candidates for their results should have been 
implemented.  He opined the delay to candidates was completely avoidable.  Ms. Mayer explained 
that the vendor believed there would be a higher number of candidates testing in order to begin the 
statistical analysis.  Mr. Baker further added that candidates are getting licensed in other states and 
later taking the CSE because of the artificial delays and the bureaucratic process in California. 
Mr. Gutierrez offered an amendment to the motion “requiring” the test vendor to develop a 
process for future analysis of test items rather than “request.”  Marilyn Lyon accepted this 
amendment to the motion. 
The motion passed 5-1 (Jon Baker opposed).
Begin Studying for the NEW 2017 Test RIGHT NOW! Click to Start Studying TODAY!