Case Information

JAMES T. BRADY, APPELLANT-RESPONDENT, v. DEPARTMENT OF PERSONNEL, RESPONDENT-APPELLANT.

Supreme Court of New Jersey

Argued February 4, 1997-Decided May 22, 1997.

693 A.2d 466

ORDER

The Disciplinary Review Board on January 22, 1997, having filed with the Court its decision concluding that JAMES A. MAJOR, II, of HACKENSACK, who was admitted to the bar of this State in 1960, should be reprimanded for violating RPC 1.3 (lack of diligence) and RPC 1.4(a) (failure to communicate), and good cause appearing;

It is ORDERED that JAMES A. MAJOR, II, is hereby reprimanded; and it is further

ORDERED that the entire record of this matter be made a permanent part of respondent's file as an attorney at law of this State; and it is further

ORDERED that respondent reimburse the Disciplinary Oversight Committee for appropriate administrative costs incurred in the prosecution of this matter.

June K. Forrest, Senior Deputy Attorney General, argued the cause for appellant (Peter Verniero, Attorney General of New Jersey, attorney; Mary C. Jacobson, Assistant Attorney General, of counsel).

D. William Subin argued the cause for respondent (Mr. Subin, attorney; Mr. Subin and Steven P. Scheffler, on the briefs).

Paul L. Kleinbaum argued the cause for amicus curiae New Jersey State Policemen's Benevolent Association (Zazzali, Zazzali, Fagella & Nowak, attorneys).

Dennis J. Alessi submitted a brief on behalf of amici curiae New Jersey State Firemen's Mutual Benevolent Association and Newark Firefighters Union (Fox and Fox, attorneys).

The opinion of the Court was delivered by

HANDLER, J.

In this case, a municipal police sergeant, seeking to attain the rank of captain, took a State promotional civil-service examination. His examination score reduced his chances of promotion. The officer challenges the results of his examination, claiming that in order to determine the accuracy of his score, he must have complete access to his testing materials, including the actual exam questions and the standards used to grade his answers. He contends that without full discovery of the examination materials, he will not be able to obtain meaningful administrative and judicial review. The State allowed the officer only limited access to the materials. It based this limitation on the need to preserve the integrity of the civil-service examination process and the security and confidentiality of civil-service examinations, which it believed would be compromised by permitting full access to the materials.

I

In October 1992, plaintiff James Brady, who is currently a police sergeant in the Atlantic City Police Department, took a civil-service examination to attain the rank of captain. The examination is administered by the Department of Personnel ("DOP") and is designed to test a candidate's ability to respond to specific situations that may arise in the course of duty. It consists of two parts, a written component, known as the "in-basket" exercise, and an oral component based on a video exercise. Candidates first take the written portion, which lasts for three hours, and those who exceed a particular score then qualify to take the oral portion, although scoring highly enough on the written part to proceed to the oral part does not guarantee that the examinee has performed exceptionally on the written portion.

The written portion, consisting of ten questions, is designed to evaluate candidates based on nine behavioral characteristics or "dimensions." The dimensions include analysis, judgment, decisiveness, ability to delegate, community sensitivity, leadership, planning and organizational ability, management capability, and written-communication skills. These characteristics are not tested in isolation; instead, each of the ten questions tests multiple characteristics.

Although written answers respond to issues raised in the context of each of the ten questions, the actual grading of the written portion does not proceed question by question, but rather dimension by dimension, with each dimension receiving a score ranging from one (lowest) to five (highest). In assessing each dimension, graders compare examinee responses to a list of "possible courses of actions" ("PCAs"), crediting responses for the PCAs that examinees recognize and marking off for the PCAs that they miss.

The oral portion of the examination tests six categories: analytic ability, judgment, decisiveness, leadership ability, community sensitivity, and oral-communication skills. Like the written portion, the oral portion employs a grading system of one (lowest) to five (highest).

Once a candidate has taken both parts of the examination, he or she receives an overall score that ranks the examinee among all exam-takers. This ranking determines a candidate's relative eligibility for promotion.

Plaintiff performed sufficiently well on the written portion of the examination to proceed to the oral component, but his written score was not outstanding.1 In contrast, he excelled on the oral portion.2 His overall score, although passing, reflected his lackluster written performance, thus reducing his promotion opportunities.

Unhappy with his overall score and with his written score in particular, plaintiff appealed through the DOP's administrative channels. He was permitted, approximately one year after he had taken the test, to review a portion of his written test materials, including his answers, a very brief summary of each question, brief comments by the grader about the PCAs that plaintiff had missed, an Orientation Background Guide (which he previously had received), and an explanation of the scoring process. He also received an audio tape of the oral component of his examination.

Pursuant to its internal policy, however, the DOP placed significant limitations on plaintiff's ability to review the test materials. First, he received only one hour to review them, which was actually fifteen minutes longer than DOP guidelines provided. Second, he did not gain access to the actual test questions or the answer key containing all potential PCAs. Finally, he could not copy any of the materials, although he could take notes. Those restrictions apparently derive from the DOP's "Examination Review Policy for the Police Promotional Assessment Process," issued in Fall 1992, which allows examinees "to review their own responses to the examination (written/tape), the examination instructions and the scores they achieved on each dimension for each examination exercise, along with a brief descriptive statement of their performance on each dimension" and which provides for a forty-five-minute review period.

Based on his brief review of the limited materials, plaintiff wrote a letter to the DOP Selection Appeals Unit on September 13, 1993, in which he expressed his disagreement with his scores on several of the written dimensions and requested that the examination be regraded. Nine months later, Anita Mathes, a supervisor in the Selection Appeals Unit, replied by letter, dated May 31, 1994, with an analysis of plaintiff's score. In the analysis, she addressed the concerns that he had raised and broke down his score for each dimension. She also noted many of the PCAs that he had missed on the written component. Mathes then concluded:

In reply to your appeal of the scoring of your in-basket examination, a review of your in-basket dimension scores revealed that in each case, significant opportunities to demonstrate behaviors associated with specific dimensions were missed. As stated previously, these possible courses of action were generated by a panel of subject matter experts consisting of senior command personnel from police departments. After reviewing your responses in light of the guidelines and parameters established, we conclude that your assigned scores are accurate.

After expressing this conclusion, the letter informed plaintiff that he could appeal the decision to the Merit System Board ("the Board"). The letter, however, contained the caveat that "the Board will only consider the proofs, arguments and issues presented at the previous level of appeal. No new or additional proofs, arguments or issues will be considered at the next level of appeal." Apparently relying on that advice, plaintiff appealed to the Board but did not advance any new arguments, including his belief that the supervisor erroneously had relied on information (i.e., the complete list of PCAs) to which plaintiff had not had access. The Board subsequently denied his appeal, noting that "Mr. Brady provides no arguments, submissions or issues in support of his appeal, but relies on proofs and arguments presented below" and concluding that because the supervisor had "addressed all arguments and contentions raised by appellant[,] ... [t]he present record fails to provide a basis to disturb [the Supervisor's] decision."

Plaintiff then appealed the Board's determination to the Appellate Division, which ordered production of all test materials, 289 N.J. Super. 557, 674 A.2d 616 (1996), concluding that "procedural fairness requires that a person who is entitled to appeal from the grade awarded him or her on a written civil service examination must be furnished with a copy of the questions and of his or her answers." Id. at 565, 674 A.2d 616. The court based its decision on the need of both a court reviewing and a party challenging an administrative determination to have access to the record upon which the agency has acted:

The limited disclosure afforded [plaintiff] was not sufficient to satisfy the due process requirement that all of the evidence on which an agency bases its decision must be revealed to the appellant. Without custody of the questions and answers, he was not in a position to obtain the advice of an attorney or of an expert on the subject matter of the examination. Without that advice and some reasonably complete explanation of the basis for the grader's evaluation of his answers, he is not able to intelligently dispute or acquiesce in the grade he was awarded. [Id. at 563, 674 A.2d 616.]

The Appellate Division recognized that its "appellate duties [did] not include grading civil service examination questions," ibid., but nevertheless reasoned that

after we have accorded the utmost deference to the Department of Personnel and to its technical expertise, ... [we must] judge whether there is a reasonable basis for its determinations.... We cannot perform that review function without the questions, the answers, and some expert explanation to tell us in what respects the questions were well or poorly answered. [Id. at 563-64, 674 A.2d 616.]

The court left the initial "good faith determination" to the DOP of "how or subject to what conditions" the information was to be provided. Id. at 565, 674 A.2d 616. The Appellate Division subsequently refused to stay its order, thus allowing plaintiff immediate access to the materials subject to a protective order.

We granted the State's petition for certification. 146 N.J. 565, 683 A.2d 1161 (1996).

II

As an initial matter, we reject plaintiff's suggestion that, because the DOP already has provided him with the test materials, the case is moot. Although the controversy between the parties about access to these particular materials is technically moot, the issue presented is of sufficient importance that review by this Court is appropriate. See In re Geraghty, 68 N.J. 209, 212, 343 A.2d 737 (1975) ("[W]e have often recognized that courts may hear and decide cases which are technically moot where issues of great public importance are involved."). Moreover, as evidenced by the Appellate Division's refusal to stay its decision in this case and the pendency of similar cases, the issue presented is capable of repetition yet likely to evade review. See In re J.I.S. Indus. Serv. Co. Landfill, 110 N.J. 101, 104-05, 539 A.2d 1197 (1988); In re Conroy, 98 N.J. 321, 342, 486 A.2d 1209 (1985). Consequently, the case's technical mootness is not a bar to our exercise of jurisdiction.

III

A.

New Jersey's Constitution and the New Jersey Civil Service Act ("the Act"), N.J.S.A. 11A:1-1 to 12-6, express a general policy that selection for State employment should be dependent on considerations of merit. N.J. Const. art. VII, § 1, ¶ 2; N.J.S.A. 11A:1-2 c; see Connors v. City of Bayonne, 36 N.J. Super. 390, 116 A.2d 48 (App. Div.), certif. denied, 19 N.J. 362, 117 A.2d 203 (1955). In order to carry out this policy, the Act creates the DOP and endows it with broad power over the selection and retention of State employees. N.J.S.A. 11A:2-6.

Regarding employee hiring and promotion, the Act vests the DOP with substantial authority over the civil-service examination process. It states:

The commissioner shall provide for:

a. The announcement and administration of examinations which shall test fairly the knowledge, skills and abilities required to satisfactorily perform the duties of a title or group of titles. The examinations may include, but are not limited to, written, oral, performance and evaluation of education and experience;

b. The rating of examinations;

c. The security of the examination process and appropriate sanctions for a breach of security;

*

*

*

*

*

*

*

*

e. The right to appeal adverse actions relating to the examination and appointment process....

[N.J.S.A. 11A:4-1.]

Thus, in keeping with the Act's general policy of encouraging a system of state employment that focuses on merit and its delegation of authority to the DOP to further that end, the Act vests the DOP with the authority to devise a fair, secure, merit-based testing process by which candidates are selected for employment and promotion. In exercising its authority under the Act, the DOP has promulgated several regulations dealing with access by examinees to testing materials. N.J.A.C. 4A:4-2.16 requires retention of the bulk of test materials for a set time period after an examination. Moreover, N.J.A.C. 4A:4-6.4 provides for a degree of access by examinees to test materials, but it provides the DOP with substantial discretion in determining the appropriate level of access. N.J.A.C. 4A:4-6.4(e) ("In order to maintain the security of the examination process, the Commissioner may, on a particular examination, modify or eliminate the review of examination questions and answers.").

The DOP has exercised its discretion under N.J.A.C. 4A:4-6.4(e) to preclude examinee access to the actual test questions and the PCA scoring key. It bases those restrictions on the need for security and confidentiality. Apparently, the DOP reuses significant numbers of exam questions from test to test, either in identical or similar form. Its reasons for this practice are the expense involved in formulating entirely new questions for each test, the consistency that recycling questions achieves among exams given at different times, and the ability to evaluate the effectiveness of questions that results from providing the questions to a large pool of examinees. Given the massive number of civil-service examinees and the hundreds of tests administered each year, the DOP has concluded that its reuse of test questions would render it impossible to ensure the integrity of the testing process if candidates were to have full access to exam materials. Moreover, according to the DOP, full access would lead to significantly increased costs in propounding test questions and would impair the DOP's ability to contract with private testing firms to provide this service.

B.

Plaintiff's basic contention is that he was entitled to greater access to his examination questions and to the specific testing criteria that were used to evaluate his examination. That contention must be considered against the standard of review of whether the DOP's limitation of access was arbitrary, capricious, or unreasonable. Public Serv. Electric & Gas Co. v. Department of Envtl. Protection, 101 N.J. 95, 103, 501 A.2d 125 (1985). In applying that standard of review, we are mindful of the deference that courts must accord agency action that purports to effectuate statutory and regulatory authority. Id. As we recently stated:

Courts have only a limited role to play in reviewing the actions of other branches of government. In light of the executive function of administrative agencies, judicial capacity to review administrative actions is severely limited.... Courts can intervene only in those rare circumstances in which an agency action is clearly inconsistent with its statutory mission or other state policy. Although sometimes phrased in terms of a search for arbitrary or unreasonable action, the judicial role is generally restricted to three inquiries: (1) whether the agency's action violates express or implied legislative policies, that is, did the agency follow the law; (2) whether the record contains substantial evidence to support the findings on which the agency bases its action; and (3) whether, in applying the legislative policies to the facts, the agency clearly erred in reaching a conclusion that could not reasonably have been made on a showing of the relevant factors. [In re Musick, 143 N.J. 206, 216, 670 A.2d 11 (1996) (citations omitted).]

The deference that courts accord an agency's actions that fall within its delegated authority is not without limits. One of those limits is embodied in the general principle of judicial review that an agency decision may not be based on undisclosed evidence. See High Horizons Dev. Co. v. Department of Transp., 120 N.J. 40, 53, 575 A.2d 1360 (1990) ("[A]n agency is never free to act on undisclosed evidence that parties have had no opportunity to rebut."); Brotherhood of R.R. Trainmen v. Palmer, 47 N.J. 482, 487, 221 A.2d 721 (1966) ("The [administrative] determination cannot rest upon undisclosed evidence which the parties have had no opportunity to test for trustworthiness or to explain or rebut."); see also In re Dep't of Ins. Order, 129 N.J. 365, 383, 609 A.2d 1236 (1992) ("One of the core values of judicial review of administrative action is the furtherance of accountability.").

In conducting judicial review of the DOP's administration of the civil-service system and its determinations regarding civil-service testing processes, we note its broad regulatory authority and the limited role allotted to the judiciary. We long have recognized that

[t]he preparation and administration of civil service examinations is an administrative function delegated most liberally to the authorized examiners of the Department ... by the Legislature.... The fulfillment of that function is a matter requiring special expertise, involving as it does the determination of what job knowledge, skills and abilities are necessary or desirable in a candidate for a particular position, and the highly technical problem of devising suitable examination questions which will demonstrate as accurately as possible whether an applicant possesses those requirements sufficiently to qualify for the position.... In view of the above, the courts cannot intervene to nullify a civil service examination unless it is clearly shown that the Department has abused its discretion. [Zicherman v. Department of Civil Serv., 40 N.J. 347, 350-51, 192 A.2d 566 (1963) (quotations and citations omitted).]

See also Artaserse v. Department of Civil Serv., 37 N.J. Super. 98, 105, 117 A.2d 22 (App. Div. 1955) ("[T]he judiciary is not disposed to engage in a critical supervisory examination of the composition of the questions to be propounded to candidates in promotional tests ... or to exercise an appellate review of the scoring of the answers....").

In this case, the gravamen of plaintiff's complaint is that he did not receive a fair score. The question, then, is what evidence relating to his examination is plaintiff entitled to have disclosed in order to substantiate that claim. The answer to that question is that plaintiff is entitled to obtain only such evidence that reasonably may enable an examinee to assess the correctness of his or her answers and to demonstrate that the DOP's grading of his or her examination constituted an abuse of discretion. Zicherman, supra, 40 N.J. at 351, 192 A.2d 566.

Consistent with that standard for determining the validity of the DOP's grading process, courts in only very rare cases have invalidated agency scoring, namely, cases in which the person challenging the score affirmatively has shown that the scoring method was arbitrary. E.g., Rox v. Department of Civil Serv., 141 N.J. Super. 463, 358 A.2d 819 (App. Div. 1976) (annulling test scores based on unsound and overly subjective testing practices). Such cases, however, simply express the narrow proposition that obvi-ously arbitrary grading systems are invalid. They in no way counter the general rule that courts will defer to an agency's grading of a civil-service examination except in the most exceptional of circumstances that disclose a clear abuse of discretion.

The court in Lavash v. Kountze, 604 F.2d 103 (1st Cir. 1979), aff'g 473 F. Supp. 868 (D. Mass.) rejected an invitation to broad judicial review of agency test scoring:

What appellant wants is the right to have his test results reviewed and to have input into that review to foreclose the possibility that his score might have been adversely affected by the form of the questions and/or the preparer's determination of the correct answers. The private interest affected here, failure to be promoted through an erroneous determination, is almost chimerical. Opening up ... examination results to inspection and review by all dissatisfied applicants for promotion would impose fiscal and administrative burdens out of all proportion to the ends sought. Such inspection and review would involve a challenge to the substantive validity of the examination. Court proceedings would require the opinion testimony of experts and submerge the court in the testing and grading process. [Id. at 105, 501 A.2d 125.]

The Appellate Division, in ordering full revelation of the testing materials, was attempting to ensure agency accountability and procedural fairness in the grading process. Although this goal is an important one, we believe that the Appellate Division did not afford sufficient weight to the limitations on judicial review in the context of the DOP's administration of civil-service examinations. Consequently, the Appellate Division, in stating that, in order to perform its "review function," it required access to "the questions, the answers, and some expert explanation to tell us in what respects the questions were well or poorly answered," 289 N.J. Super. at 564, 674 A.2d 616 (emphasis added), imposed on itself a decisional responsibility that the court was not required to assume. Courts may not routinely review the contents of civil-service examinations and answers and, aided by expert testimony, determine whether the questions were "well or poorly answered." "Such inspection and review would involve a challenge to the substantive validity of the examination." Lavash, supra, 604 F.2d at 105. Instead, reviewing courts may conduct only a limited review of the reasonableness of a grading system and determine simply whether the testing and grading were clearly arbitrary. The limited role of judicial review in this context applies to the two DOP determinations that are challenged by plaintiff. First, the DOP has determined that test security, and hence the need for confidentiality, are essential to preserve the integrity of the examination process and to discharge the State's constitutional duty to hire and to promote based on merit. N.J.A.C. 4A:4-6.4(e). Second, it has concluded that the optimal accommodation of its competing obligations to guarantee "[t]he security of the examination process," N.J.S.A. 11A:4-1 c, and to ensure "[t]he right to appeal adverse actions relating to the examination and appointment process," N.J.S.A. 11A:4-1 e, is to provide aggrieved examinees with certain materials but not others.

Regarding the first determination, the Act specifically requires the DOP to "provide for the security of the examination process," N.J.S.A. 11A:4-1 c, and the disclosure of test questions and scoring keys that are likely to be reused in future tests clearly implicates exam security. Although plaintiff contends that the DOP, in administering civil-service examinations, has abused its discretion by reusing exam questions, that decision is clearly within the range of responsibilities that the Legislature has delegated to the DOP to implement the most effective and efficient procedure to assure public hiring and promotion based on merit. See Brotspies v. Department of Civil Serv., 66 N.J. Super. 492, 495-96, 169 A.2d 484 (App. Div. 1961). Because we are unable to conclude that the DOP has abused its discretion in deciding to recycle test questions, based on significant concerns about achieving exam consistency and avoiding unnecessary cost, we cannot take issue with its determination that confidentiality in some form is the most effective means of fulfilling the Act's mandate that test security be ensured. Cf. Brotspies v. Department of Civil Serv., 72 N.J. Super. 334, 342, 178 A.2d 367 (App. Div. 1962) (citing Brotspies, supra, 66 N.J. Super. at 495-96, 169 A.2d 484, and approving of its statement that "[s]ince the Department may desire to use many of the items for examinations for other positions, then once a particular set of questions and correct answers is publish-ed, all of the value resulting from the empirical evidence so carefully collated by the Department is destroyed").

The second determination by the DOP, relating to the proper balance between test security and an examinee's right to challenge his or her score, poses a more difficult issue because it concerns the proper procedure for ensuring fair, secure, merit-based testing. The DOP has implemented a procedure by which an examinee has access to all test materials except for the actual test questions and the PCA answer key; in lieu of these items, the examinee receives a summary of the questions and a brief explanation of the PCAs that the examinee actually missed. Moreover, the examinee is given only forty-five minutes to review and to take notes on the materials and may not copy them. Finally, at the examinee's request, a supervisor reviews the entire exam and, in a detailed letter, describes many of the PCAs that the examinee missed. Under these procedures, according to the DOP, an examinee has sufficient access to the relevant materials and a reasonable opportunity to evaluate the grading process. The examinee, however, is not provided additional access to the materials because that access would increase the risk of misuse through dissemination, thus undermining the security and integrity of the testing process. Plaintiff complains that without full access to examination materials and a greater opportunity to assess the examination methodology and results, examinees and courts cannot ensure that the DOP does not misgrade examinations.

We conclude that the DOP's accommodation of the competing goals of examination security and examinee access is neither arbitrary, capricious, nor unreasonable. Plaintiff essentially seeks full access to the testing materials as a basis for challenging and obtaining broad judicial review of the accuracy of the agency's scoring of the examination. However, we already have rejected the proposition that courts should engage in full judicial review under these circumstances, given that "the judiciary is not disposed to engage in a critical supervisory examination of the composition of the questions to be propounded to candidates in promotional tests ... or to exercise an appellate review of the scoring of the answers...." Artaserse, supra, 37 N.J. Super. at 105, 117 A.2d 22.

In the vast majority of cases, full disclosure thus would confer little or no administrative or litigational benefit on the examinee, given the general prohibition against judicial regrading of examinations. However, full disclosure would wreak havoc with the DOP's legitimate efforts to maintain security. This balance clearly supports the reasonableness of the accommodation that the DOP has chosen between test security and examinee access. Cf. Mathews v. Eldridge, 424 U.S. 319, 96 S. Ct. 893, 47 L. Ed. 2d 18 (1976) (balancing government and private interests and determining that due process did not require additional process where process would not benefit individual interest substantially and would harm government interest).

Other courts have reached similar conclusions and thus have approved limitations on examinee access to civil-service examination materials based on security and confidentiality concerns. See, e.g., Lavash, supra, 604 F.2d 103; Civil Serv. Comm'n v. Pinder, 812 P.2d 645 (Colo. 1991) (en banc); Garner v. Department of Personnel, 835 P.2d 527 (Colo. Ct. App. 1992) (same), cert. denied, 507 U.S. 917, 113 S. Ct. 1274, 122 L. Ed. 2d 669 (1993); Patch v. Civil Serv. Comm'n, 295 N.W.2d 460 (Iowa 1980). Federal regulations also parallel the DOP's approach of limiting access to test materials. 5 C.F.R. § 300.201; 14 C.F.R. § 1212.205 (National Air and Space Administration's application of 5 C.F.R. § 300.201).

We note that in Martin v. Educational Testing Service, 179 N.J. Super. 317, 326, 431 A.2d 868 (Ch. Div. 1981), the court held that because the plaintiff in that case had a protectible interest in accurate grading, due process required full revelation of testing materials in order to evaluate whether the scoring was in fact correct. Id. at 326-27, 431 A.2d 868. Martin may be distinguishable in that the test was developed and administered by Educational Testing Service, a private entity. However, to the extent that Martin suggests a requirement of full disclosure of civil-service examinations without regard to security and confidentiality concerns, it is overruled.

We thus conclude that the DOP has chosen a reasonable balance between its interest in the confidentiality of the examination process and examinees' interest in reviewing the grading of examinations, and that the provision for partial or limited access to examination materials is a valid exercise of the agency's regulatory authority.

C.

Our conclusion regarding nondisclosure should not be read to insulate the civil-service scoring process from judicial review entirely. If the DOP were, in the name of confidentiality and security, to deny examinees all access to testing materials, such a decision almost surely would be arbitrary, capricious, or unreasonable because it would allow the DOP to conduct the testing and grading process without any accountability and would foreclose any opportunity on the part of an examinee to demonstrate the unreasonableness or unfairness of his or her examination. The DOP, however, has not gone so far in its efforts to ensure the integrity of the process, instead allowing limited access to most of the materials and providing sufficient opportunity for examinees to discover grading irregularities, while not jeopardizing test security.

We note that despite the general reasonableness of the DOP's limitation on examinee access, even this amount of access may prove insufficient if an examinee makes a prima facie showing of more than the mere possibility of misgrading, thus justifying more extensive disclosure. Even the DOP admits that certain cases call for greater access. For example, a candidate may be able to make a prima facie showing of arbitrariness or discrimination in grading that is so obvious and rises to such a high level that the full materials must be produced. Although we do not impute any precedential authority to unpublished decisions, R. 1:36-3, we are aware that some courts have acknowledged instanc-es of such demonstrated arbitrariness. See, e.g., In re Setkiewicz, No. A-2123-94T2, slip op. at 5 (N.J. App. Div. Nov. 27, 1995) ("We think it plain that some prima facie showing must be made to justify ... access [by examinees to full test materials]. We are satisfied, however, that the rescoring here justified appellant's demand."); Dellaventura v. Department of Personnel, No. A-3957-91T1 (N.J. App. Div. Mar. 11, 1993) (allowing access based on specific prima facie showing of arbitrariness in grading).

We stress that such a prima facie showing requires more than an allegation of the mere possibility of erroneous scoring. Instead, it must be based on a specific allegation of arbitrariness or discrimination. See Artaserse, supra, 37 N.J. Super. at 105, 117 A.2d 22 ("The courts ... will intervene to nullify such an examination where it is affirmatively shown to have been manifestly corrupt, arbitrary, capricious, or conspicuously unreasonable."). Under DOP procedures, examinees appear to have a reasonable opportunity to make such a showing. They are provided with a summary of the questions, their complete answers, and a summary of the PCAs. Further, they are entitled to an internal review in which their answers are reexamined, and they are furnished with personalized explanations of the basis for their grade on each dimension. An examinee should be able to determine from this process whether the DOP appears to have acted arbitrarily in scoring his or her examination.

In this case, however, plaintiff has made no such prima facie showing of arbitrariness despite his substantial access to the examination materials. This failure is especially stark given plaintiff's presumed knowledge and experience as a police officer and his access to other police officers and union officials who doubtlessly have a degree of expertise in the civil-service process. With those resources, plaintiff should have been able, after consultation, to make some sort of proffer about how the DOP had acted arbitrarily in scoring his exam. Instead, without offering any particular allegation of arbitrariness, he simply asserts that his

score is too low and that he therefore should have unfettered access to the materials.

We agree with the separate opinion that, under our holding, the possibility exists that examinees and reviewing courts will not catch each and every instance of misgrading by the DOP. Post at 267-268, 693 A.2d at 478. However, despite the validity of that concern, we believe that it overstates the problem posed by limited access. Examinees receive more than "bare-bones access," post at 267, 693 A.2d at 477, to their exams and to the PCAs used to grade them. The DOP, as we have noted, initially provides examinees with their complete answers and brief summaries of the test questions and missed PCAs. In addition, the supervisor's subsequent evaluation of examinees' performance contains significantly more detailed analysis of each dimension, including specific descriptions of many of the missed PCAs. Consequently, although examinees do not have full access to their exam materials, they have sufficient access to judge whether the DOP grading process is arbitrary.

The DOP has chosen to sacrifice a small degree of scoring accountability in order to bolster the security, and hence the integrity, of the entire examination process. This policy may or may not be the wisest one. That determination, however, has been entrusted by the Legislature to the DOP, not to the judiciary. Our role is simply to ensure that the DOP has not acted arbitrarily, capriciously, or unreasonably in drawing on its expertise to strike a balance between the Legislature's dual command that the civil-service examination process be secure and that it allow for examinees to appeal their scores. The DOP has exercised its delegated authority and has struck a balance that accommodates both of these mandates. Whether we agree or disagree with that balance, we are persuaded that it has a basis in reason and, hence, is not arbitrary.

We also believe that the separate opinion understates the role that reviewing courts would have to assume in routinely reviewing in camera "the examinee's answers, the correct answers, and a sufficient explanation of the grading standards to enable the court to make a preliminary assessment of arbitrariness and determine whether further disclosure to the examinee, an evidentiary hearing or other relief may be appropriate." Post at 267, 693 A.2d at 477. Although that level of judicial review would not be quite as expansive as that endorsed by the Appellate Division, it nevertheless would require routine judicial second-guessing of the DOP's grading decisions, thus conflicting with our long jurisprudential tradition of not reviewing the accuracy of individual grades without a prima facie showing by the examinee that the process was arbitrary and constituted a clear abuse of discretion. The DOP's procedures allow examinees to make such a showing, but plaintiff simply has failed to do so.

IV

Plaintiff also contends that he was treated unfairly and denied procedural fairness in the administrative review of his examination results. The supervisor in the DOP's Selection Appeals Unit who responded to plaintiff's concerns about his score advised him that "the [Merit System] Board will only consider the proofs, arguments and issues presented at the previous level of appeal. No new or additional proofs, arguments or issues will be considered at the next level of appeal." Plaintiff claims to have followed that advice by not advancing any new arguments, most notably his contention that the supervisor should not have relied on undisclosed evidence, namely, the PCA scoring key. The Board, in dismissing plaintiff's appeal, noted that he had not submitted any "arguments, submissions or issues in support of his appeal," but instead had "relie[d] on proofs and arguments presented below." The Board thus denied the appeal based on the supervisor's having adequately addressed all of plaintiff's contentions. Plaintiff now argues that the supervisor's statement deprived him of a real opportunity to appeal his score because, in reliance on that statement, he did not raise before the Board the issue of the supervisor's use of undisclosed information in rejecting his appeal.

We are unable to discern a specific legal basis for the supervisor's assertion that plaintiff could not raise any new arguments in his appeal to the Board.3 N.J.A.C. 4A:4-6.4(g)1 and 2 and N.J.A.C. 4A:4-6.6(b)1 and 2 both state that "[t]he appeal shall contain all information which was presented to the first level, plus a copy of the decision below" and that "[t]he Board shall decide any appeal on the written record or such other proceeding as the Board deems appropriate." However, nowhere do the regulations state that an appellant may not submit any new arguments, especially new arguments that take issue with the supervisor's decision. Even the Board implied that plaintiff could have advanced new "arguments, submissions or issues" on appeal by stating in its decision that he had chosen not to do so in favor of relying "on the proofs and arguments presented below."

We infer that the supervisor was attempting to inform plaintiff that the Board would reach an independent determination of the facts and legal conclusions based on the record below, cf. Henry v. Rahway State Prison, 81 N.J. 571, 579, 410 A.2d 686 (1980) (holding that Civil Service Commission had to review disciplinary determinations de novo), and that it would follow the established principle that an appellate tribunal generally will not consider issues and arguments that could have been raised below but were not. See, e.g., Nieder v. Royal Indem. Ins. Co., 62 N.J. 229, 234, 300 A.2d 142 (1973) ("It is a well-settled principle that our appellate courts will decline to consider questions or issues not properly presented to the trial court when an opportunity for such a presentation is available....").

In any event, we need not determine whether the supervisor's statement was misleading and unfair because it did not prejudice plaintiff. As we have noted, plaintiff takes issue with the statement not because it prevented him from challenging a specific aspect of his test score, but rather because it allegedly deprived him of the ability to challenge the supervisor's reliance on undisclosed information. As we already have determined, however, the DOP acted reasonably in not disclosing that information. Supra at 259-262, 693 A.2d at 473-474. Consequently, the supervisor's potentially erroneous statement was harmless and provides no basis for a reversal of the Board's denial of plaintiff's appeal.

V

The judgment of the Appellate Division is reversed.

STEIN, J., concurring in part and dissenting in part.

In attempting to arrive at a proper balance between the interest of civil-service test takers in obtaining review of their test grades and that of the Department of Personnel (DOP) in safeguarding the confidentiality of test questions, the Court concludes that the Appellate Division erred in holding that as a matter of procedural fairness all persons challenging their test scores must be provided with copies of the questions, their answers, and the grading standards. I do not disagree with that aspect of the Court's opinion.

However, in my view the Court overshoots the mark and tips the balance too far toward the interests of confidentiality when it precludes disclosure of relevant test materials to the reviewing court, as well as to the applicant, unless the applicant makes a prima facie showing that the test results are arbitrary. That holding creates a prototypical Catch-22 for a test taker seeking to challenge his or her test results: the bare-bones access to testing materials that the DOP customarily makes available is an insuffi- cient basis on which to make even a prima facie showing of arbitrariness, and absent such a showing neither the test taker nor the reviewing court will receive sufficient access to the test materials to be able to evaluate whether the test was graded arbitrarily.

The Court can and should strike a fairer balance. It could do so by requiring the DOP to furnish the reviewing court, in camera, with the test questions, the examinee's answers, the correct answers, and a sufficient explanation of the grading standards to enable the reviewing court to make a preliminary assessment of arbitrariness and determine whether further disclosure to the examinee, an evidentiary hearing or other relief may be appropriate.

The Court implies that the Appellate Division conceived that its responsibility over the test grade was plenary, ante at 258, 693 A.2d at 473, but a fair reading of that court's opinion demonstrates that it intended its review to be properly deferential to the DOP, seeking only to determine whether the grading was arbitrary. "[O]ur review function does require us, after we have accorded the utmost deference to the Department of Personnel and to its technical expertise, to judge whether there is a reasonable basis for its determinations." 289 N.J. Super. 557, 563–64, 674 A.2d 616 (1996). With the truncated materials provided, the Appellate Division could not perform its review function. The Court's disposition of this appeal should assure that reviewing courts are not unduly impeded from performing the review that is mandated by statute.

I

The pertinent facts are summarized adequately in the Court's opinion. I highlight certain facts to emphasize the difficulty encountered by the Appellate Division in attempting to provide even the most cursory review of respondent Brady's written examination grade. As noted by the Court, the examination for police captain consisted of an oral and written examination, each component using a grading system of one (lowest) to five (highest). Brady received the highest possible grade on five of the six characteristics, or dimensions, tested on the oral examination. His oral scores were: analytic ability (4); judgment (5); decisiveness (5); leadership ability (5); community sensitivity (5); oral-communication skills (5).

The written examination graded the applicants on the same dimensions as the oral examination, with the obvious exception of oral-communication skills, and on four additional dimensions. Brady's scores on the written examination were not outstanding, and his grades on the very same dimensions tested in the oral component were significantly lower. His written examination grades were: analysis (2); judgment (3); decisiveness (4); leadership ability (2); community sensitivity (3); ability to delegate (3); planning and organizational ability (3); management capability (2); written-communication skills (3).

The written portion of the examination required the candidate to assume the role of a Captain on the police force of the hypothetical Township of Grandview. The testing materials included essential factual information about the Township, the police force, and its personnel. Each examinee was required to respond in writing to ten hypothetical situations, and the examinees were evaluated on the basis of the various dimensions set forth above. In grading the examinations, graders were provided with a list of "possible courses of action" ("PCAs") that had been compiled by a panel of senior-command personnel from various police departments. The list of PCAs was used by each grader to ascertain the minimum level of acceptable responses related to each dimension. When an examinee's response omitted reference to a PCA on the master list, that omission was noted as a deficiency or missed opportunity, and the aggregate number of such deficiencies was the basis for the numerical grade awarded on each dimension tested. Brady, dissatisfied with his score on the written portion of the examination, exercised his statutory right of appeal. N.J.S.A. 11A:4-1 e. Pursuant to the DOP's published review policy, Brady was afforded only limited access to the test materials for the purpose of preparing his appeal. He was permitted to examine his answers to each question, and a brief summary of each question, but not the question itself. He was not allowed to examine the PCAs, on which his grades were based. He was permitted to review his grades on each dimension, a brief description of his performance on each dimension, the general instructions for the written examination, and a summary sheet describing how his score was computed. Brady was allotted one hour to review those materials, allowed to take notes, but not allowed to remove or photocopy any of the materials that he examined.

As noted by the Appellate Division, 289 N.J. Super. at 559, 674 A.2d 616, Sergeant Brady initially appealed to the DOP's Supervisor of the Selection Appeals Unit, communicating by letter dated September 15, 1993, the reasons why he believed the grader's scoring on each of the dimensions was too low. The response, dated May 31, 1994, noted generally that "significant opportunities to demonstrate behaviors associated with specific dimensions were missed," that those "behaviors" were based on the PCAs compiled by experts, that an item-by-item review was unwarranted, and that a review of Brady's responses in comparison with the PCAs demonstrated that his scores were accurate.

In accordance with a statement in the Supervisor's letter that any appeal to the Merit System Board could not be based on new or additional proofs or argument, Brady appealed to the Merit System Board and apparently relied only on his initial letter to the Selection Appeals Unit. The Merit System Board affirmed.

Brady then appealed to the Appellate Division, and the record before that court was sparse. Brady's submission consisted of his Notice of Appeal, his letter to the Selection Appeals Unit, the Supervisor's response, and a copy of the DOP's one-page "Examination Review Policy for the Police Promotional Assessment Pro- cess, Fall 1992," which essentially described the limited review rights of a candidate seeking to appeal his or her examination score. The Merit System Board's submission consisted of an eleven-page orientation guide, containing background information about the Captain/Lieutenant examination, four pages of background facts relating to the Township of Grandview and its police department on which the written examination was based, and Brady's one-page score sheet for both the written and oral examinations, together with a comment sheet containing cryptic comments about each of the graded dimensions.

Conspicuously absent from the record furnished to the Appellate Division were the examination questions, Sergeant Brady's answers, the PCAs to which his answers were compared for grading, and any explanation of the grading methodology sufficient to assure the Appellate Division that the grading was fair, impartial, and free from arbitrariness. As the Appellate Division observed, "[T]he Department of Personnel must provide a sufficient explanation for its grading decisions to enable a reviewing court, which is obligated to give due deference to the authority, responsibility and expertise of the agency, to determine whether the agency's decisions have a reasonable basis." 289 N.J. Super. at 565-66.

II

The standard by which courts exercise their obligation to review challenges to civil-service test results is not a matter in dispute. The majority opinion correctly observes that courts generally are obliged to recognize the broad regulatory authority of the DOP and to defer to the agency's grading of an examination absent a showing of arbitrariness. Ante at 257-258, 693 A.2d at 472-473. As this Court stated in Zicherman v. Department of Civil Service, 40 N.J. 347, 351, 192 A.2d 566 (1963), "the courts cannot intervene to nullify a civil-service examination unless it is clearly shown that the Department has abused its discretion." In Zicherman, in which the plaintiff challenged the appropriateness of a civil-service promotional examination for the position of Clerk of the District Court, this Court reviewed the examination questions and the correct answers. Although observing that the "correct" answers to two questions appeared to be erroneous and that the relevancy of a few questions was debatable, the Court nevertheless upheld the validity of the examination. Id. at 352, 192 A.2d 566.

A similarly deferential standard of review is reflected in other reported decisions involving challenges to results of civil-service examinations. See, e.g., Rox v. Department of Civil Serv., 141 N.J. Super. 463, 467-69, 358 A.2d 819 (App. Div. 1976) (acknowledging deferential standard of review, but invalidating as too subjective oral testing format for police captains and lieutenants, noting that one of the seven grading teams awarded consistently lower scores to examinees than did the other six grading teams); Brotspies v. Department of Civil Serv., 66 N.J. Super. 492, 494-99, 169 A.2d 484 (App. Div. 1961) (acknowledging that intervention to nullify civil-service examination is warranted only where examination is corrupt, arbitrary, capricious, or unreasonable, and concluding that questions were fair and appropriate, and that answers were neither unreasonable nor implausible, following hearing at which single judge reviewed objections to questions and to Department's version of correct answers); Artaserse v. Department of Civil Serv., 37 N.J. Super. 98, 102-05, 117 A.2d 22 (App. Div. 1955) (holding that plaintiffs had failed to prove civil-service examination for police lieutenant was arbitrarily administered or graded where department, after consultation with plaintiff's counsel, eliminated six of nine challenged questions and regraded all papers, but plaintiffs nevertheless failed to attain minimum requisite score).

The experience of other courts confronted with the responsibility of reviewing challenges to civil-service examinations but hampered in discharging that duty by insufficient information concerning the test confirms the wisdom of mandating in camera submission to the reviewing court of all pertinent material. Like the majority, ante at 264, 693 A.2d at 476, I impute no prece- dential authority to unpublished decisions, see R. 1:36-3, but the factual context of the two unreported opinions referred to by the majority is illustrative of the need to provide adequate information to a reviewing court. In Dellaventura v. Department of Personnel, No. A-3957-91T1 (App. Div. Mar. 11, 1993), the plaintiff appealed the Merit System Board's decision upholding his failing grade on the written portion of a promotional examination for Fire Lieutenant. Plaintiff's initial score was thirty-one and the passing grade was thirty-five. Only examinees who passed the written test were eligible to take an oral examination.

After reviewing his test papers, plaintiff appealed his score, relying on his memory of the questions and answers in preparing his appeal. The initial review process resulted in no change in his score. A second-level review raised his score by one point. The Merit System Board then found other errors in the grading of his examination, recalculated his score, and changed the final score to thirty-four, one point below a passing grade. On appeal, plaintiff challenged the Merit System Board's evaluation of his answers and also challenged the Board's scoring methodology.

The Appellate Division rejected the challenge to the correctness of his answers but remanded for further proceedings that would generate fact findings and conclusions concerning the scoring methodology. Acknowledging the highly deferential standard of review that it was applying, the court nevertheless observed that "we cannot tell on the record that presently exists whether the method of scoring appellant's test results was or was not arbitrary, capricious or unreasonable."

An analogous issue was presented in In re Setkiewicz, No. A-2123-94T2 (App. Div. Nov. 27, 1995), in which plaintiff, a Hoboken Fire Captain, took the promotional test for Deputy Fire Chief along with several other candidates. Based on the examination, the DOP's promotional list placed plaintiff fourth, and a candidate named Blohm second. Blohm was promoted to Deputy Chief in October 1992. Approximately two years later the DOP decided plaintiff's appeal of his test score, and awarded him sufficient additional points to place him second on the list and drop Blohm to fourth. Subsequently, Blohm revived his own appeal, which had been dismissed after his promotion, and that appeal resulted in an increase in Blohm's score again sufficient to rank him higher than plaintiff.

Plaintiff then sought access to Blohm's test file, including his answers and the scoring key, arguing that without such access he could not possibly challenge the rescoring of Blohm's exam. Relying on its regulations and the need for confidentiality, the DOP denied access. The Appellate Division reversed, concluding that the regulations permitted the DOP to exercise discretion in resolving the access issue, but that under the unique circumstances presented the denial of access to Blohm's file precluded plaintiff from obtaining adequate review. The court permitted the DOP to impose appropriate conditions on access to assure confidentiality.

III

This appeal does not directly present the Court with the question of the validity of the DOP's regulation limiting an appealing examinee's access to test materials. That regulation, N.J.A.C. 4A:4-6.4(e), provides in part: "In order to maintain the security of the examination process, the Commissioner may, on a particular examination, modify or eliminate the review of examination questions and answers." Exercising that regulatory discretion based on considerations of examination security and confidentiality, the DOP has decided to preclude Brady's access to the actual questions, the PCAs, and the scoring guide. This record is entirely inadequate for a court to make a competent determination about the reasonableness of the DOP's discretionary decision.

Nevertheless, the Court not only defers to the DOP's undocumented need for confidentiality in upholding the denial to examinees of access to test records, but also inhibits the statutorily mandated function of reviewing courts by denying them access to test records absent a preliminary showing of arbitrariness. The DOP's interest in the confidentiality of its examinations surely would not be compromised if the reviewing court were permitted to conduct an in camera review of all pertinent test materials, not for the purpose of second-guessing the DOP's grading, but merely to verify that the test of the examinees seeking review was not administered or graded arbitrarily.

I would modify the Appellate Division's judgment to require that, in addition to those materials to which the examinee is given access by the DOP, the DOP must provide the reviewing court with the full text of the test questions, the examinee's answers, the "correct" answers, and a sufficient explanation of the grading process to enable the reviewing court to determine preliminarily whether the administration and grading of the test were conducted arbitrarily, and whether further proceedings are necessary to resolve the examinee's appeal.

For reversal—Chief Justice PORITZ and Justices HANDLER, POLLOCK, O'HERN, GARIBALDI and COLEMAN—6.

Concur in part; dissent in part—Justice STEIN—1.

Footnotes

1 Plaintiff's written scores were: analysis (2), judgment (3), decisiveness (4), ability to delegate (3), leadership ability (2), community sensitivity (3), planning and organizational ability (3), management capability (2), and written-communication skills (3).

2 Plaintiff's oral scores were: analytic ability (4), judgment (5), decisiveness (5), leadership ability (5), community sensitivity (5), and oral-communication skills (5).

3 We note that the supervisor cited N.J.A.C. 4A:4-6.6(c) as a basis for informing plaintiff of his general right to appeal and the requirement that he do so within twenty days of receipt of the supervisor's decision. However, the supervisor appears to have miscited the correct administrative provision, because N.J.A.C. 4A:4-6.6(c) was irrelevant, simply stating that "[t]he Board may bypass any other level of appeal for its direct review." Ibid. We assume that the supervisor intended to cite either N.J.A.C. 4A:4-6.4(f) and (g) or N.J.A.C. 4A:4-6.6(a)3 and (b), which require the DOP to notify the appellant of the right to appeal to the Board and of the twenty-day requirement.