Gonzales v. Galvin

151 F.3d 526 | 6th Cir. | 1998

OPINION

BATCHELDER, Circuit Judge.

Plaintiffs appeal the district court’s June 1996 order terminating a 1974 consent decree entered into by Plaintiffs and Defendants the City of Toledo, Ohio; the Fraternal Order of Police, Toledo Lodge No. 40; the Toledo Police Patrolmen’s Association, Inc.; and those individually-named referred to collectively as (“Defendants” or “City”). For the reasons that follow, we VACATE and REMAND this ease for further proceedings consistent with this opinion.

I.

In 1972, Plaintiffs filed a class action suit challenging the employment practices of the Toledo Police Department, claiming these practices violated 42 U.S.C. § 1983. In 1974, the parties entered into a consent decree, subsequently approved by the district court, that established two goals: (1) ensuring “integration” of the police department’s workforce, and (2) ensuring that police candidates are hired based on non-discriminatory selection criteria. To meet the first goal, the City agreed to implement an affirmative action plan to remedy the effects of past discrimination. Both parties now agree that the City has met its affirmative action obligations. The provision of the consent decree at issue here pertains to the second goal and provides:

1. The defendants shall begin immediately the process required to validate all employment examinations for the Toledo Police Department consistent with EEOC guidelines 29 C.F.R. Sec. 1607.1-1607.14J¹] The validation procedures shall be of a quality to insure that such examinations used in the police selection process do not discriminate against Blacks, Mexican-Americans, or any other person and that the results obtained will provide a reasonable prediction of job performance.

The consent decree gave Plaintiffs the right to engage an “expert on the construction and administration of tests of relative ability” to assist the City in choosing testing areas and types of questions and in the development of a job-related examination.² After administration of the test, Plaintiffs’ expert and “an expert mutually agreed upon by defendants’ associations” were to be permitted to review the actual test and challenge the job-relatedness of any specific question. Court supervision was to continue “for the entry of such further orders as may be appropriate to effectuate the provisions of this Order, and to monitor the progress of the Defendants in meeting its Affirmative Action obligations and its stated employment goal.”

In October 1995, Plaintiffs filed a motion for a temporary restraining order and preliminary injunction to prevent the City from swearing in a new police class. Plaintiffs argued that the City had used discriminatory selection procedures that resulted in a disproportionately low number of African-Americans applying for entry into the new police class. In November 1995, the district court denied the motion, finding that Plaintiffs had presented no evidence that the lower percentage was due to selection procedures rather than mere chance.

In February 1996, the district court ordered the parties to show cause why the court should not vacate the 1974 consent decree and terminate its jurisdiction. Plaintiffs served a discovery request on Defendants asking for a copy of the 1993 written recruitment examination (“Written Examination”), which had been administered to the 1995 police class.³ Defendants refused to provide the examination. After two unsuccessful telephone conferences brokered by the magistrate judge, Plaintiffs moved to compel production of the 1993 examination, arguing that it was necessary for their expert, Dr. Barrett, to review the examination and determine its “content validity.”⁴ The magistrate judge denied Plaintiffs’ motion to compel, stating that Dr. Ryan, the court-appointed expert,⁵ had already reviewed the Written Examination and therefore having Dr. Barrett do the same would be “cumulative and duplicative.” Plaintiffs appealed this decision to the district court.

Meanwhile, Defendants filed a response to the show cause order. Defendants included the affidavit of Dr. Ryan, wherein she concluded that the written examination process met the standards set forth in the Uniform Guidelines. Dr. Ryan stated that the “content validity evidence consists of documentation relating test components to job analysis data;” that she had reviewed this evidence; and that she had concluded that it was sufficient to support the job-relatedness of the test components. She also stated that the “criterion-related evidence consists of a series of studies relating test scores to academy performance,” and that “[ljooking across all of the years for which data are available and considering issues of sampling error, the written examination appears to relate to performance in the academy.” (emphasis added). She noted that the data suggested stronger relationships in earlier examinations than in later ones, and stated that she had made suggestions to counteract this trend.

Plaintiffs filed a response to the show cause order and moved the court to enforce the consent decree. Plaintiffs requested an evidentiary hearing and submitted an affidavit of their expert, Dr. Barrett, setting forth several alleged deficiencies in the City’s processes for criterion-related and content validation. Plaintiffs also submitted a copy of the 1983 criterion-related study, the only other criterion-related study in the record, conducted by then-court-appointed expert Dr. Outtz. In this study, Dr. Outtz had concluded that the City had shown the examination was related to success in the Police Academy, but “additional evidence” was needed to show that “the selection procedure is directly related to job performance or that the selection procedure is valid with regard to training performance, which in turn is directly related to success on the job.”

In addition, Plaintiffs submitted a February 1993 letter from Dr. Ryan to the City; the letter stated that the proposed 1993 Written Examination was content valid, but went on to point out several deficiencies in the job analysis used to support her finding of validity. In particular, Dr. Ryan noted that the skills and aptitudes to be measured on the examination were not clearly defined in terms of “observable aspects of work behavior,” as required by the Uniform Guidelines. See 29 C.F.R. § 1607.14(C)(2) (1997). Dr. Ryan also pointed to several areas in the proposed 1993 examination where it was not clear that the questions tested job-related skills. In her June 1993 “Final Review of the 1993' Police Officer Examination,” however, Dr. Ryan stated that the City had incorporated her suggestions for revisions into the 1993 Written Examination before administering the test. Finally, in her letter Dr. Ryan briefly mentioned the 1983 criterion-related validity study and concluded that it would be insufficient support of the validity of the examination process. In her deposition, Dr. Ryan stated that the City’s criterion-related strategy linked performance on the examination with performance in the Police Academy. Contrary to the implication of statements in her letter, Dr. Ryan stated that while there was no further attempt to relate academy performance with job performance, there is “an assumption” that one must perform well at the Academy to-perform well on the job. Nevertheless, she also testified that she' did not look at any evidence which would test that assumption.

Defendants responded to the court’s show cause order by submitting the City’s own validation study report detailing the development, finalization, administration, and results of the 1985, 1987, 1990, and 1993 Written Examinations. This report included “Job Analyses” that summarized the “knowledges, skills, abilities, and personality characteristics” (“KSAPs”) identified by the City as necessary to perform various duties of the police officer position. Attached to the report was a “criterion-related” validation study conducted to assess the predictive validity of selection systems used from 1983-1990. This study, which was conducted by Michele Grisez under Dr. Ryan’s supervision, concluded that there was “some evidence of predictive validity for the selection system, although some inconsistencies in the data exist.” The study recommended that the City continue its validation efforts and stated that there was a “relationship (in some cases) between exam scores and training success,” but that “[t]he validity of these exams in predicting job success is dependent upon the critical link between training content and job content.”

Plaintiffs responded' to Defendants’ validation evidence, arguing'that Defendants’ evidence did not satisfy the Uniform Guideline requirements for either content-validity or criterion-related validity and did not insure that the examination results would provide a reasonable prediction of job performance. In particular, Plaintiffs argued' that Defendants had not provided any evidence linking performance at the Academy to performance on the job. Further, Plaintiffs asserted that Defendants had not adequately documented which job-related skills, aptitudes, or work behaviors the Written Examination purported to measure, as required for content-validity. Defendants replied, submitting a summary of the courses taught at the Police Academy, a copy of the basic curriculum required by the Ohio Peace Officer Training Counsel, and a copy of the Job Analysis on which they circled the KSAPs acquired at the Academy.

Two days later, the district court⁶ denied Plaintiffs’ request for time to respond to this new evidence, and entered a memorandum and order vacating the consent decree and terminating the court’s jurisdiction. The court, however,, included in its order neither its reasoning nor ,any findings supporting its conclusion that the Written Examination “complies with the consent decree, Title VII, and the Uniform Guidelines.” Noting that Defendants had submitted Dr. Ryan’s affidavit, the order simply recited her conclusions; observed that Plaintiffs objected to several areas of Dr. Ryan’s report; “question[ed] the propriety of Plaintiffs’ attack on their own experts;” and explicitly refused to address all but one of Plaintiffs’ arguments, because they were “lacking in factual or legal support.” Referring to the factors set forth in Heath v. DeCourcy, 992 F.2d 630 (6th Cir. 1993), the court concluded that the decree’s purposes had been fulfilled and that court supervision should be terminated. The court then denied as moot Plaintiffs’ appeal of the magistrate’s decision denying Plaintiffs’ motion to compel production of the Written Examination.

After the court entered its order but prior to the entry of judgment, Plaintiffs moved the court to reconsider its decision to terminate the decree, and renewed their request for an evidentiary hearing, but the court entered its judgment terminating jurisdiction over the 1974 decree. This appeal followed.

II.

A.

We review a district court’s termination of supervision and jurisdiction over a consent decree for an abuse of discretion. Heath v. DeCourcy, 992 F.2d 630, 633 (6th Cir.1993). A court abuses its discretion when it relies on clearly erroneous factual findings, improperly applies the law, or uses an erroneous legal standard. See Black Law Enforcement Officers Ass’n v. City of Akron, 824 F.2d 475, 479 (6th Cir.1987). The appellant has the burden of proof on appeal to show that there was no reasonable basis for the district court’s termination order. Heath, 992 F.2d at 633 (citing United States v. W.T. Grant Co., 345 U.S. 629, 633, 73 S.Ct. 894, 97 L.Ed. 1303 (1953) (reviewing decision on preliminary injunction)). The court’s findings of fact will be upheld unless clearly erroneous. Brunet v. City of Columbus, 58 F.3d 251, 255 (6th Cir.1995).

B.

Injunctive relief after a period of compliance should not extend beyond the time necessary to remedy the violation. See Board of Educ. v. Dowell, 498 U.S. 237, 248-49, 111 S.Ct. 630, 112 L.Ed.2d 715 (1991); Youngblood v. Dalzell, 925 F.2d 954, 961 (6th Cir.1991). A district court must look to the specific terms of a consent decree in determining whether and when to terminate supervision or jurisdiction over it. Youngblood, 925 F.2d at 960-61. Factors to be considered include (1) any specific terms providing for continued supervision and jurisdiction over the consent decree; (2) the consent decree’s underlying goals; (3) whether there has been compliance with prior court orders; (4) whether defendants made a good faith effort to comply; (5) the length of time the consent decree has been in effect; and (6) the continuing efficacy of the consent decree’s enforcement. Heath, 992 F.2d at 633 (citing Board of Educ. v. Dowell, 498 U.S. 237, 111 S.Ct. 630, 112 L.Ed.2d 715 (1991), and Youngblood, 925 F.2d at 960-61).

Notwithstanding this list of “factors,” a district court may not terminate its jurisdiction until it finds both that Defendants are in compliance with the decree’s terms and that the decree’s objectives have been achieved. See Heath, 992 F.2d at 633 (citing Youngblood, 925 F.2d at 957-58 (6th Cir.1991)). A “consent decree, although in effect a final judgment, is a contract founded on the agreement of the parties.... It should be construed to preserve the position for which the parties bargained_” Jansen v. City of Cincinnati, 977 F.2d 238, 241 (6th Cir.1992) (quoting Vogel v. City of Cincinnati, 959 F.2d 594, 598 (6th Cir.1992)); see also Hadix v. Johnson, Nos. 93-1555, 93-1559, 93-1560, 93-1642, 93-1643, 93-1551, 66 F.3d 325, 1995 WL 559372, at *3 (6th Cir. Sept. 20, 1995) (unpublished). ‘While we recognize that a consent decree operating in perpetuity may not be the most effective way to eradicate the evils of discrimination in the municipal workplace, we must remain true to the wording and intent of such decrees, until their goals are met.” Jansen, 977 F.2d at 246.

Before terminating a consent decree, a district court “should make sufficiently detailed findings of fact and conclusions of law under Rule 52(a), Fed.R.Civ.P., to advise the parties of the factual basis for its decision and permit informed appellate review.” Bradley v. Milliken, 772 F.2d 266, 272 (6th Cir.1985). Because the same deferential standard of review applies in an appeal of a preliminary injunction, this Court’s statements concerning the adequacy of the lower court’s findings in that context are instructive here:

It is of the highest importance to proper review of the action of a court in granting or refusing a preliminary injunction that there should be fair compliance with Rule 52(a).... [T]he findings of the trial court must be sufficiently detailed to give us a clear understanding of the analytical process by which ultimate findings were reached and to assure us that the trial court took care in ascertaining the facts. These findings should not be merely con-clusory, nor, on appeal, may the arguments of the parties substitute for findings of fact by the district court.

The absence of adequate findings is highlighted in this case by our deferential standard of review. We review the district court’s findings of fact for clear error.... The sparseness and conclusory nature of the trial court’s findings would require us to overstep the bounds of our duty [under Rule 52] and, essentially, to duplicate the role of the district court, substituting the arguments of the parties for considered findings of the district court_

It has long been clear that [w]hen the court does not make findings which are sufficient to indicate the factual basis for its ultimate conclusion, the appropriate procedure is to vacate the judgment and remand for such findings.

Stotts v. Memphis Fire Dep’t, No. 85-5001, 774 F.2d 1164, 1985 WL 13722, at *2-3 (6th Cir. Sept. 11, 1985) (unpublished)⁷ (emphasis added) (collecting cases) (internal citations and quotation marks omitted); see also United States v. Claycraft Co., 408 F.2d 366, 366 (6th Cir.1969); Deal v. Cincinnati Bd. of Educ., 11 Ohio Misc. 184, 369 F.2d 55, 63-64 (6th Cir.1966) (there must be subsidiary findings to support the ultimate conclusions of the court). In addition, before a district court dissolves a consent decree, it must consider and resolve all' objections to such dissolution. Youngblood, 925 F.2d at 961; Jansen, 977 F.2d at 245. When the district court terminates its supervision and jurisdiction before making findings concerning compliance with all terms of a decree, the court abuses its discretion. See Heath, 992 F.2d at 634.

C.

The district court correctly held that the Heath factors must be considered in deciding whether time had come to terminate the decree. The court’s error lies in its purporting to terminate jurisdiction over the consent decree without first making, explicit findings concerning Defendants’ compliance with the decree’s goals and specific terms. See Heath, 992 F.2d at 633-34; Bradley, 772 F.2d at 272.

The decree before us specifically requires that in order to realize the ultimate goal of a non-discriminatory selection process predictive of job performance, Defendants must •validate the Written Examination in accordance with the Uniform Guidelines. The Uniform Guidelines require a user to conduct a “validity study” when a particular selection procedure has an adverse impact⁸ on a minority group; otherwise, the selection procedure will be deemed discriminatory. 29 C.F.R. §§ 1607.1(B), 1607.3. The City conceded that the Written Examination adversely impacted on minorities and that to comply with the decree’s ultimate goal, it therefore must support the Written Examination with an appropriate validation study.

To comply with the Uniform Guidelines, validation studies must conform to several technical requirements; the minimum standards are set forth at 29 C.F.R. § 1607.14. In addition, the Uniform Guidelines set forth specific documentary requirements for each type of study. See § 1607.15. Thus, before terminating the decree, the court was obligated to explain how the Defendants’ validation studies comported with the Uniform Guidelines requirements. In particular, the court needed to determine: (1) whether Defendants followed the Uniform Guidelines’ technical standards in conducting their validation studies, see § 1607.14; (2) whether Defendants properly documented their evidence of validity, see § 1607.15; and (3) whether the evidence actually demonstrates criterion-related or content validity, i.e., that the examination is either “predictive of or significantly correlated with important elements of work behavior” or “representative of important aspects of performance on the job,” § 1607.16(D), (F).

Here, the district court summarily stated that “Defendants have met their burden of proving that the exam complies with the consent decree, Title VII, and the Uniform Guidelines,” and concluded that the decree’s second goal had been satisfied. To permit proper appellate review, the court was required to set forth its “subsidiary findings” that support its ultimate conclusions. Deal, 369 F.2d at 64. The court, however, failed to articulate even a single reason. The order contains no discussion of any of the technical standards for validity studies^ much less how the City satisfied these standards, or for that matter, which type of validity the City purportedly demonstrated. For example, the court does not explain whether it found a correlation between higher test scores and better job performance (and, if so, what that correlation might be), see Albemarle Paper Co. v. Moody, 422 U.S. 405, 430-32, 95 S.Ct. 2362, 45 L.Ed.2d 280, (1975); Brunet v. City of Columbus, 58 F.3d 251, 253 (6th Cir.1995); Bernard v. Gulf Oil Corp., 841 F.2d 547, 566-67 (5th Cir.1988) (remanded to district court to make findings on correlation to job performance and explain specifically what it relied upon in determining significant correlation); whether the test itself was sufficiently representative of the job (and if so, how), see Police Officers for Equal Rights v. City of Columbus, 916 F.2d 1092, 1098-1100 (6th Cir.1990); Guardians Ass’n of the New York Police Dep’t, Inc. v. Civil Service Comm’n, 633 F.2d 232, 243 (2d Cir.1980), aff'd, 463 U.S. 582, 103 S.Ct. 3221, 77 L.Ed.2d 866 (1983); or upon what data the court relied. Cf. Griggs v. Duke Power Co., 401 U.S. 424, 431, 91 S.Ct. 849, 28 L.Ed.2d 158 (1971) (to be valid, successful performance on a selection test must “bear a demonstrable relationship to successful [job] performance”) (emphasis added). In fact, the court did not even refer to the actual validation studies themselves, but instead referred only to Dr. Ryan’s assessment of them.

Plaintiffs argde before us that the court’s cursory conclusion was clearly erroneous. They allege numerous deficiencies in the City’s validation evidence, objecting both to the criterion-related and content-based studies, and set forth in great detail the statutory and common-law bases for their arguments. The vast majority of these arguments were also presented to the district court. Defendants, of course, counter that their studies do meet the Uniform Guideline requirements and that they have clearly provided the district court with evidence sufficient to find in their favor. On appeal, both parties are essentially arguing factual matters. But due to the court’s complete failure to set forth the grounds, for its ultimate conclusion that the City met its burden of proving that the Written Examination complies with the -consent decree, we are wholly unable to determine whether this conclusion is clearly erroneous.

At one point in its memorandum, the district court does state that Defendants submitted Dr. Ryan’s affidavit wherein she concluded that the written examination is valid under the Uniform Guidelines. At no point, however, does the district court credit her conclusions as the basis for its own. Even were we to assume that the court based its conclusions on Dr. Ryan’s affidavit and assume that an expert’s affidavit alone is sufficient support, we still could not properly evaluate the court’s conclusions. In' her affidavit, Dr. Ryan stated that the examination’s validity is' supported by both content and criterion-related validity studies conducted in accordance with the Uniform Guidelines; however, she did not state with any specificity upon what evidence she relied. Moreover, her statements regarding the criterion-related evidence are arguably deficient as a matter of law.⁹ We can only guess whether the district court credited the content-based evidence, the criterion-related evidence, or both.

Further, as we noted earlier in this opinion, the district court may not dissolve a consent decree without addressing all of the Plaintiffs’ objections to dissolution. See Youngblood, 925 F.2d at 961; Jansen, 977 F.2d at 245. Had the court in this case made findings of fact and set forth its reasoning, it might well have addressed all of the Plaintiffs’ objections in the course of its opinion.¹⁰ Instead, rather than addressing Plaintiffs’ contentions, the court summarily concluded that the examination was valid and explicitly refused to address Plaintiffs’ objections. This too was an abuse of the court’s discretion.

Accordingly, we conclude that the district court’s order dissolving the consent decree must be vacated, and this matter remanded to the district court with instructions that the court reconsider the matter and, consistent with this opinion, issue appropriate findings of fact and conclusions of law in support of its ultimate determination of the show cause order.

III.

A.

We review for an abuse of discretion a district court’s decision not to hold an evidentiary hearing before terminating a consent decree. See Alberti v. Klevenhagen, 46 F.3d 1347,1358 (5th Cir.1995).

B.

Although they concede that the standard of review is abuse of discretion, Plaintiffs argue at length that the Sixth Circuit requires a district court to hold a formal evidentiary hearing prior to terminating a consent decree. Plaintiffs, however, are mistaken.

It is true that we have said that modification of a consent decree requires a “ ‘complete hearing and findings of fact.’ ” See, e.g., Heath v. DeCourcy, 992 F.2d 630, 634, 635 (6th Cir.1993) (quoting Brown v. Neeb, 644 F.2d 551, 560 (6th Cir.1981)); Bradley v. Milliken, 772 F.2d 266, 271 (6th Cir.1985) (“A hearing is generally required before a district court may modify a consent decree.”). We have not, however, held that termination of a consent decree requires such a hearing. More importantly (and putting aside Defendants’ argument that modification cases are inapposite because a court has the inherent power to terminate its own jurisdiction), we think the point here is that a “complete hearing” of an issue does not necessarily require a full-blown evidentiary hearing. See Patterson v. Newspaper & Mail Deliverers’ Union, 797 F.Supp. 1174, 1182 (S.D.N.Y.1992) (full evidentiary hearing not required or necessary before dissolving consent decree), aff'd, 13 F.3d 33 (2d Cir.1993). Evi-dentiary hearings are not necessary where the parties’ briefs clearly set forth the relevant facts and arguments of a case such that a hearing would not add anything to the briefs, and where the court has sufficient evidence before it to make detailed factual findings. See FDIC v. Morely, 915 F.2d 1517, 1522-23 (11th Cir.1990) (deficiency judgment); United States v. Gross, No 92-4105, 1 F.3d 1242, 1993 WL 300393, at *3 (6th Cir. Aug. 3, 1993) (unpublished) (motion to dismiss); Heflin v. Stewart, Nos. 91-6219, 91-6160, 968 F.2d 1214, 1992 WL 162554, at *3 (6th Cir. July 13, 1992) (unpublished) (attorney fee award).¹¹

Plaintiffs expend a significant amount of energy attempting to create a factual dispute that would require an oral hearing by noting that Dr. Barrett’s expert opinion as to the Written Examination’s content validity conflicted with that of Dr. Ryan. They argue that an evidentiary hearing is necessary for the court to evaluate the credibility, qualifications, and differing opinions of these two experts. In cases such as this, experts provide factual information in the form of data compilations, statistical analyses, and reports expressing and explaining their opinions. These core facts are statistical in nature, and, unlike the case where two eyewitnesses to an event present different stories, the court’s need to assess the expert witness’s . demeanor is peripheral at best. While, in reaching its conclusion, a court might find it useful to hear live testimony from experts espousing different opinions, a court may just as reliably evaluate these opinions on a paper record. Furthermore, we note that the district court had before it Dr. Ryan’s deposition, in which her opinions were subjected to cross-examination by Plaintiffs’ counsel.

Defendants argue correctly that in making the technical and complex determination of whether a validation study conforms to the Uniform Guidelines requirements, a court is entitled to rely on expert opinions. See, e.g., Police Officers for Equal Rights v. City of Columbus,. 916 F.2d 1092, 1103 (6th Cir.1990). The court may not, however, merely rubberstamp the conclusions reached by a court-appointed expert. To provide a “complete hearing” in this case, the court was required to consider all of the evidence before it, including the views and opinions of Plaintiffs’ expert, Dr.'Barrett, before reaching any conclusions. For this reason, we conclude that the court should have permitted Plaintiffs time to respond to the evidence that Defendants submitted two days before the court issued its order terminating the consent decree, which evidence, according to Defendants, established the critical criterion-related link theretofore missing between predictions of Academy performance and job performance. Although not required to hold an' evidentiary hearing, the court was still obliged to review Plaintiffs’ evidence and hear Plaintiffs’ arguments on these matters. See Heath, 992 F.2d at 634-35.

C.

Finally, we note that the district court declined to address Plaintiffs’ appeal of the Magistrate Judge’s order denying their motion to compel production of the Written Examination, holding that because the decree was being dissolved, the issue was moot. The issues surrounding the need for the Examination to be admitted into evidence,¹² including any confidentiality issues that may be associated with Plaintiffs’ access to the examination, are better evaluated in the first instance by the district court. Accordingly, we conclude that the court’s order denying the Plaintiffs’ appeal of the Magistrate Judge’s refusal to compel production of the Written Examination must be vacated and that issue remanded to the district court for further proceedings consistent with this opinion.

CONCLUSION

Accordingly, we VACATE the district court’s order, and REMAND this case to the district court for further proceedings not inconsistent with this opinion.

1 . The Uniform Guidelines on Employee Selection Procedures (“Uniform Guidelines”), which were adopted on August 25, 1978, by the EEOC, the Civil Service Commission, the Department of Labor, and the Department of Justice, supercede the employee selection-procedure guidelines previously issued by the EEOC. See 29 C.F.R. §§ 1607.1(A), (C), 1607.18 (1997). Although the 1974 consent decree predaLes the adoption of the Uniform Guidelines, the parties and the district court have consistently assessed the City's selection procedures pursuant to them.

2 . Defendants were given the right to object to the chosen expert, and the court was to serve as the final arbiter.

3 . Successful performance on the Written Examination was required for entry-level positions in the Toledo Police Department. The other components of the hiring process were a background investigation, a structured oral interview, a medical examination, and a psychological examination.

4 . The Uniform Guidelines, set forLh at 29 C.F.R. 1607.1-.18 (1997), are designed to provide a framework for determining that tests and other employee-selection procedures do not discriminate on the grounds of race, color, religion, sex, or national origin. 29 C.F.R. § 1607.1(B). A test demonstrating an adverse impact on hiring of members of any race or ethnic group is considered "discriminatory” unless it is "validated" in accordance with the guidelines. 29 C.F.R. § 1607.3(A).

The guidelines set forth three types of permissible validation studies, two of which are relevant herc: "criterion-related validity” studies and "content validity” studies. 29 C.F.R. § 1607.5(A). Criterion-related studies are statistical studies, based on empirical data demonstrating that the test is predictive ol or significantly correlated with important elements of job performance, while content studies are based on data showing that the content of the test is representative of important aspects of performance on the job for which the candidates are to be evaluated. 29 C.F.R. § 1607.16(D), (F); aw also §§ 1607.5(B), 1607.14(B),(D). An example frequently cited as a content-based test is a typing test given to a secretary.

5 .The court appointed Dr. Ann Marie Ryan in 1993; she was recommended by the City and approved by Plaintiffs. Dr. Ryan is the third expert that the court has appointed over the life of the consent decree; the other two, Drs. Albert Palmer and James Outtz, were recommended by Plaintiffs and approved by the City.

6 . In May 1996, Judge Young, who had presided over this case from its inception, passed away, and Judge Katz inherited this case.

7 . Although unpublished, the Stotts opinion aptly pulls together the law of this and several other circuits. Therefore, we borrow Stotts’s summation instead of creating one of our own.

8 . The Uniform Guidelines define "adverse impact” in part as

A selection rate for any race, sex, or ethnic group which is less than four-fifths (4/5) (or eighty percent) of the rate for the group with the highest rate will generally be regarded ... as evidence of adverse impact, while a greater than four-fifths rate will generally not be regarded ... as evidence of adverse impact.

29 C.F.R. § 1607.4(D).

9 . Section 1607.14(B)(3) provides: "Where performance in training is used as a criterion, success in-training should be properly measured and the relevance of the training should be shown either through a comparison of the content of the training program with the critical or important work behaviors(s) of the job(s), or through a demonstration of the relationship between measures of performance in training and measures of job performance." This Court, as well as several other circuits, requires that where selection tests are shown to predict successful performance in a training program, the study must further demonstrate a link between successful performance in the training program and on the job. See Police Officers for Equal Rights, 916 F.2d at 1101 ("employers may not test for candidates' abilities to perform in a training program unless they can show that the program is job-related”); Harless v. Duck, 619 F.2d 611, 616-17 (6th Cir.1980) ("The [selection procedure] related only to training performance at the police academy and not to actual job performance; consequently, it is deficient as a matter of law.”); Guardians Ass’n, 633 F.2d. at 244-47; Ensley Branch of NAACP v. Seibels, 616 F.2d 812, 817-22 (5th Cir.1980).

In her 1993 letter, Dr. Ryan stated that the 1983 criterion-related validity study would be "insufficient support of the validity of the examination process." In her deposition, Dr. Ryan stated that the City’s criterion-related strategy linked performance on the examination with performance in the Police Academy. She further stated that while there was no further attempt to relate academy performance with job performance, there is "an assumption” that one must perform well at the academy to perform well on the job, but that she did not look at any evidence that would test that assumption. In her affidavit, Dr. Ryan stated only that the "written examina-lion appears to relate to performance in the academy.” (emphasis added).

Even if there were no conflict among her various statements. Dr. Ryan's "assumptions” cannot support criterion-validity. The Uniform Guidelines specifically require documentary evidence, and explicitly state that "casual reports” of validity are not acceptable in lieu of evidence of validity and that professional supervision is not a substitute for documented evidence. See 29 C.F.R. §§ 1607.9, 1607.15; see also Albemarle Paper Co., 422 U.S. at 428 n. 23, 95 S.Ct. 2362 (job relatedness cannot be proved through vague and unsubstantiated hearsay).

10 . The court did address Plaintiffs' argument that "alternative selection procedures” exist and points to evidence supporting its conclusion. But whether or not an alternative selection procedure exists becomes relevant only after a finding that a test is valid. See 29 C.F.R. § 1607.3(B); Albemarle Paper Co., 422 U.S. at 425, 95 S.Ct. 2362 (Once plaintiffs make out case of disparate impact, the employer must show that its tests are job-related. If the employer makes such a showing, it remains open to the complaining party to show that other tests or selection devices would also serve the employer's interests without having a similar adverse im- • pact.) (citations omitted). The district court's finding with regard to alternative selection procedures does not address compliance with Uniform Guidelines validity-testing requirements or supporL a conclusion that the test used is nondiscriminatory and predicts job performance.

11 . We recognize that the citation of unpublished per curiam opinions is disfavored. We cite Gross and Heflin, however, because "there is no published [Sixth Circuit] opinion that would serve as well.” See 6th Cir. R. 24(c).

12 . Plaintiffs point out that because the district court did not address their demand to have access to the Written Examination, the document was nexer put into evidence. Plaintiffs contend that the district court erred in failing to rexiew the 1993 Written Examination before dissolving the consent decree. Assuming, without deciding, that the district court could make a final determination regarding content-validity without reviewing the actual test questions, wc nevertheless are skeptical that a thorough and proper review of an examination to determine content validity can be accomplished without looking at the examination itself. By definition, content-valid examination questions essentially replicate actual job activities and requirements, mental and phx'sical. See 29 C.F.R. § 1607.14(C)(4), 1607.16(Y).' Content validation allows employers to show job relevance, not by relying on statistical analysis, but instead, by demonstrating that the contents of a wriuen examination (or other selection procedure) represent important aspects of job per-lormance. .See 29 C.F.R. §§ 1607.5(B), 1607.14(C)(4); cf. Police Officers for Equal Rights, 916 F.2d at 1100 (to be content valid, the test must measure important attributes of the job, although it need not be precisely proportional to importance and frequency of use in the job). Defendants argue that the court is not an expert in psychometric analysis and that it could rely on the validation evidence presented to it and Dr. Ryan's opinion to determine content-validity. But surely the court need not be an expert in this area to be capable of following an expert's logic and making an intelligent evaluation of the actual questions asked and the expert’s conclusions that those questions are representative of important aspects of the job for which the applicants are being tested.