KANNE, Circuit Judge.

Thirteen employees of the Cook County Sheriffs Department, including Samuel Scaife, sued Cook County, Sheriff Michael Sheahan, and several Department supervisors including William Wallace (the “defen *737 dants”), alleging numerous violations of federal law. The district court granted the defendants’ motion for summary judgment and dismissed the case. Only Scaife appeals. For the following reasons, we affirm.

I. HISTORY

Samuel Scaife, an African-American male, has been employed by the Cook County Sheriffs Department since June 1, 1988. Scaife began as a correctional officer and in 1990 became an investigator in the Department’s electronic monitoring unit. In 1994, Scaife was promoted to supervisor, and, after returning from injury leave, was permanently placed in the permissions section of the electronic monitoring unit.

On March 18, 1999, Scaife signed Grievance No. 99-003 along with several coworkers (who were plaintiffs in the proceedings below but did not appeal). The grievance alleged “that only particular people are allowed to leave early. All investigators should be allowed to leave early or All made to stay until the end of their 8-hour shift. The permission office was told that if we complained, that ‘you know what’s going to happen to you.’ We took that as intimidation.” (emphasis in original). On March 31, a threatening note left on Scaife’s desk stated, ‘Your [sic] next Inv. Scaife.” Scaife requested an internal investigation of the incident. Chief Randy Pietrowski conducted an investigation and on April 19, found the allegations to be inconclusive based on the lack of supporting evidence.

On May 30, Scaife and three others were transferred to patrol from the permissions unit where they were placed on a six-and-two schedule (six consecutive working days followed by two days off). Scaife, who had been working a five-and-two schedule (Monday through Friday with weekends off), did not request the transfer. On August 10, Scaife filed Grievance No. 99-012, alleging that he was transferred against his will, and that he had been harassed for being an African-American single parent. The grievance also alleged that Scaife had not been informed of open job posts for which he otherwise could have applied. This grievance was “granted” on November 4. 1

Scaife allegedly arrived late to work on September 7, 1999, resulting in a recommended suspension of three days. Scaife filed a grievance. 2 After Scaife failed to attend the grievance hearing on October 14, the three-day suspension was imposed. Scaife subsequently was written up for arriving late to work on October 17, this time incurring a recommended suspension of 29 days. Again Scaife filed a grievance (No. 99-023) and again did not attend the hearing. On December 3, executive director William Wallace, citing Scaife’s absence from the grievance hearing scheduled for that day, imposed the recommended 29-day suspension in full.

On November 16, deputy director Michael Ricci notified Department personnel that the electronic monitoring unit would be reorganized. Under the plan, the permissions section would be combined with another section to form a new detainee management unit. The new unit would employ a six-and-two standard schedule, with preferences given according to seniority. Personnel in the unit would also have patrol duties on weekends. On November *738 19, Ricci gave notice that the reorganization would be effective November 21. Apparently the effect of the reorganization was to put Scaife on a six-and-two schedule, only a few weeks after his grievance (No. 99-012) was granted which presumably gave him weekends off. The next day, Scaife requested a transfer from patrol to the records unit with Saturdays and Sundays off, and his request was granted November 28.

On November 30, Scaife filed a charge before the Illinois Labor Relations Board (“Labor Board”) alleging that the redeployment of November 19 violated state law and seeking enforcement of Grievance No. 99-012. In October 2001, Scaife prevailed before the Labor Board, which affirmed the Administrative Law Judge’s award of damages and recommendation that Scaife’s Grievance No. 99-012 be granted. Pursuant to the Labor Board’s order, on February 2, 2002, Scaife was reassigned to the detainee management unit with weekends off.

Scaife filed a charge with the Equal Employment Opportunity Commission (“EEOC”) on September 18, 2001, alleging discrimination on the basis of race and retaliation for filing a union grievance. On December 14, 2001, Scaife and several others filed a complaint in federal court. The district court allowed the plaintiffs to amend their complaint several times. The operative complaint is the third amended complaint, which contained eleven counts, of which Scaife alleged seven.

Specifically, Scaife joined Count I (42 U.S.C. § 1981 race discrimination and retaliation); Count II (42 U.S.C. § 1983 equal protection claims based on race); Count III (Title VII race discrimination); Count VI (42 U.S.C. § 1983 first amendment retaliation); Count VII (Title VII retaliation claims naming only Cook County and Sheahan officially); Count IX (29 U.S.C. § 157 et seq. unfair labor practices); and Count X (29 U.S.C. § 207 unfair wage and hour claims) 3 . The district court dismissed Count IX pursuant to the defendants’ motion to dismiss. The portion of the complaint relating solely to Scaife included numerous allegations regarding his scheduling, transfers, grievances, and suspensions.

The defendants moved for summary judgment seeking to dismiss with prejudice all of the remaining claims, including Scaife’s. The defendants’ Local Rule 56.1 statement of facts and supporting materials referred to Scaife’s transfers and filing of grievances but not the suspensions. The defendants argued, inter alia, that Scaife had suffered no adverse action. In his response, Scaife pointed to his suspensions as instances of adverse action. On June 28, 2004, the district court granted summary judgment for the defendants on all claims. In its opinion, the court noted that the defendants did not address the suspensions but concluded in any event that Scaife could not meet his prima facie burden for discrimination and retaliation.

The scope of Scaife’s appeal is limited to the district court’s judgment in favor of defendants Michael Sheahan in his official capacity as Cook County Sheriff, Cook County, and William Wallace in his individual capacity, and only for the claims of discrimination and retaliation insofar as they relate to the suspensions.

II. ANALYSIS

We review the district court’s grant of summary judgment de novo, viewing the facts and drawing all inferences in the *739 light most favorable to Seaife, the nonmoving party. Volovsek v. Wis. Dep’t of Agric., Trade & Consumer Prot., 344 F.3d 680, 686 (7th Cir.2003) (citation omitted); Waggoner v. Olin Corp., 169 F.3d 481, 483 (7th Cir.1999). Summary judgment is appropriate only if the record shows that there is no genuine issue as to any material fact and that the moving party is entitled to judgment as a matter of law. Fed. R.Civ.P. 56(c). Initially, the burden of production is upon the moving party to identify “those portions of ‘the pleadings, depositions, answers to interrogatories, and admissions on file, together with the affidavits, if any,’ which it believes to demonstrate the absence of a genuine issue of material fact.” Celotex Corp. v. Catrett, 477 U.S. 317, 323, 106 S.Ct. 2548, 91 L.Ed.2d 265 (1986) (quoting Fed.R.Civ.P. 56(c)). “[T]he burden on the moving party may be discharged by ‘showing’ — that is, pointing out to the district court — that there is an absence of evidence to support the nonmoving party’s case.” Celotex, 477 U.S. at 325, 106 S.Ct. 2548. “Once the moving party satisfies this burden, the nonmovant must ‘set forth specific facts showing that there is a genuine issue for trial.’ ” Vitug v. Multistage Tax Comm’n, 88 F.3d 506, 512 (7th Cir.1996) (quoting Fed.R.Civ.P. 56(e)). If “the record taken as a whole could not lead a rational trier of fact to find for the non-moving party,” the nonmovant has not demonstrated a dispute of material fact exists. Matsushita Elec. Indus. Co., Ltd. v. Zenith Radio Corp., 475 U.S. 574, 587, 106 S.Ct. 1348, 89 L.Ed.2d 538 (1986) (citation omitted).

Discrimination may be proven either directly, such as by an admission by the defendant, or indirectly under the burden-shifting method established in McDonnell Douglas Corp. v. Green, 411 U.S. 792, 802-04, 93 S.Ct. 1817, 36 L.Ed.2d 668 (1973); Herron v. DaimlerChrysler Corp., 388 F.3d 293, 299-302 (7th Cir.2004); Alexander v. Wis. Dep’t of Health & Family Servs., 263 F.3d 673, 682 (7th Cir.2001). For his discrimination claim, Seaife makes no argument relating to the direct method, opting instead to proceed only under the indirect method. McDonnell Douglas places the initial burden on the plaintiff to establish a prima facie case, which is the same whether the claim is brought under Title VII or § 1981. See Cerutti v. BASF Corp., 349 F.3d 1055, 1060-61 n. 4 (7th Cir.2003) (citations omitted). A prima fa-cie case of discrimination is established if the employee provides evidence that: (1) he is a member of a protected class; (2) he was meeting his employer’s legitimate expectations at the time of the alleged adverse action; (3) he was subjected to an adverse employment action; and (4) the employer treated similarly situated employees not in the protected class more favorably. Herron, 388 F.3d at 299 (citation omitted).

Similarly, retaliation may be proven directly or indirectly. A prima fa-cie case for retaliation under the indirect method requires the plaintiff to show that: (1) he engaged in statutorily protected activity; (2) he was performing his job according to his employer’s legitimate expectations; (3) he suffered a materially adverse action; and (4) he was treated worse than a similarly situated employee who did not engage in statutorily protected activity. Firestine v. Parkview Health Sys., Inc., 388 F.3d 229, 233 (7th Cir.2004) (citations omitted). In both the discrimination and retaliation contexts, once the plaintiff has established his prima facie case, the burden shifts to the defendant to provide a legitimate, nondiscriminatory reason for the adverse employment action. Id. (citation omitted). If the defendant has provided a legitimate reason, the burden shifts back to the plaintiff to show *740 that the defendant’s proffered reason is merely pretextual. Id. (citation omitted). Scaife’s claims are based upon his suspensions, which he claims were motivated by his race and by his filing of grievances.

The highlight of Scaife’s appeal is the defendants’ omission of Scaife’s suspensions in their summary judgment motion and reply. Seaife argues that the district court interjected arguments on the defendants’ behalf so that its granting of summary judgment for the defendants amounted to a sua sponte act without notice to Seaife. In Scaife’s view, the suspensions were not within the scope of the defendants’ motion for summary judgment even though the defendants did move to dismiss all of Scaife’s claims with prejudice. In effect, Seaife mischaracterizes the suspensions to be legal claims, not factual allegations, so that he may rest on his pleadings which, of course, he may not do. Fed.R.Civ.P. 56(e); see Celotex, 477 U.S. at 325-26, 106 S.Ct. 2548. But Scaife’s claims are for discrimination and retaliation; the suspensions are mere factual allegations supporting these claims. The defendants’ motion for summary judgment seeking dismissal of these claims (all claims actually) was sufficient to put Seaife on notice that no longer could he rest on the allegations in his pleadings, including the suspensions. The burden was upon Seaife to show a material dispute of fact existed as to these claims. Additionally, Seaife did attempt to provide evidentiary support for his suspensions in his response, indicating there was no deficiency of notice or lack of opportunity to provide factual support for his claims. We turn to whether Seaife has shown a material factual dispute exists.

For his discrimination claim, Seaife cannot show an issue of fact as to whether similarly situated employees were treated more favorably. Seaife points to his affidavit, which states: “Other investigators, including Chris Helms, Dan Nadolski and Ray Villa, all white male investigators with DCSI, regularly came in late to work and left early from work without discipline from 1999 through and including 2003.” The affidavit continues, “Between November 1999 to June, 2002 the terms and conditions of my employment were significantly different and less favorable than for white employees.” This may be sufficient to prevail against a motion to dismiss under Rule 12(b)(6), but to survive summary judgment, Seaife needed to do better than to make such broad-brushed, conclusory allegations. See Larsen v. City of Beloit, 130 F.3d 1278, 1282 (7th Cir.1997) (“ ‘The object of [Rule 56(e) ] is not to replace conclusory allegations of the complaint or answer with conclusory allegations of an affidavit.’ ” (quoting Lujan v. Nat’l Wildlife Fed’n, 497 U.S. 871, 888, 110 S.Ct. 3177, 111 L.Ed.2d 695 (1990))); In re Morris Paint & Varnish Co., 773 F.2d 130, 136 (7th Cir.1985) (noting “conclusory assertions are insufficient to raise a genuine issue of material fact”). Seaife did not list instances in which the three white men were late, did not indicate whether they had the same supervisor as he did, nor did he say how the terms and conditions of his employment differed.

Seaife produced time sheets for the days he was allegedly late to work, a review of his disciplinary history recommending further action, and the notices of his suspension. Seaife claims that he was never late and that his time sheets were altered. The time sheets do indicate that on several days Seaife signed in to work on time at 8:00 A.M. and that someone subsequently wrote a later time, initialing the change. But Seaife did not produce the time sheets of the three white employees who Seaife asserts received preferential treatment, nor do their names appear on the time *741 sheets Scaife did submit. Additionally, the time sheets show that the same supervisor altered the sign-in times of other employees who purported to sign in at 8:00 A.M. to reflect that they had been late to work as well, but Scaife does not identify these people. Finally, Scaife makes no reference to the several employees listed on the time sheets whose sign-in times on or before 8:00 A.M. were not subsequently altered. Presumably these are the people who would have received the type of preferential treatment which Scaife alleges, and the omission of them creates an evi-dentiary gap which Scaife’s mere self-serving allegations are insufficient to fill. See Albiero v. City of Kankakee, 246 F.3d 927, 933 (7th Cir.2001) (“Self-serving affidavits without factual support in the record will not defeat a motion for summary judgment.” (quotations and citations omitted)).

Scaife’s evidence shows at most that he was suspended at least twice. Even if we were to assume that the suspensions constituted an adverse action, there is nothing in the record whatsoever to suggest that Scaife’s suspensions were improperly motivated. Scaife has not produced sufficient evidence that similarly situated employees not in Scaife’s protected class were treated any better than Scaife. Without this evidence, Scaife cannot meet his prima facie burden on his discrimination claim.

For his retaliation claim, Scaife relies upon the same evidence, buttressed by his own conclusory allegation that the suspensions were the result of the grievances he had filed. Scaife cannot satisfy the fourth element of the prima facie case for retaliation because he has made no attempt to describe the treatment of employees who did not file grievances, which dooms his retaliation claim under the indirect method.

Scaife argues alternatively that he has put forth sufficient evidence to prove retaliation under the direct method using circumstantial evidence. Scaife may prove retaliation under the direct method by showing that he engaged in a statutorily protected activity, that the defendants subjected him to adverse employment action, and a causal connection exists between the two events. Lang v. Ill. Dep’t of Children & Family Servs., 361 F.3d 416, 418-19 (7th Cir.2004) (citations omitted); McKenzie v. Ill. Dep’t of Transp., 92 F.3d 473, 483 (7th Cir.1996) (citation omitted); Brenner v. Brown, 36 F.3d 18, 19 (7th Cir.1994) (citation omitted). Putting aside the first two elements, Scaife cannot show causation.

Scaife refers to the affidavit of Louis Mustari, Scaife’s former supervisor. Mustari stated that Ricci told him that he would never hire or promote blacks, and that Ricci “blamed African American employees of DCSI for failures in the department which were not their responsibility ... [and] that he would never promote any investigator who complained of race discrimination or associated with blacks ... [and] that if it weren’t for whites, the ‘black bastards’ wouldn’t do any work.” The only context Mustari’s affidavit provides is that Ricci made the remarks “to me in my capacity as a chief in that department .... ” Mustari’s affidavit, much like Scaife’s, is conclusory, but most importantly the comment is not sufficiently connected to the suspensions to support a finding of retaliation.

When a plaintiff offers an employer’s stray remark in a discrimination case, it is necessary to demonstrate “some nexus” between the remark and the challenged employment decision. See Cowan v. Glenbrook Sec. Servs., Inc., 123 F.3d 438, 444 (7th Cir.1997). At most, the record shows that Ricci was one of Scaife’s superiors, but what is more definite is that Wallace was the decisionmaker because *742 the suspensions were not effective until he upheld them. Scaife has not shown that Ricci was a decisionmaker; indeed, he makes no attempt to do so. Even if Ricci was a decisionmaker, there is no evidence to connect the statement to the suspensions. The statements, therefore, cannot carry the day for Scaife.

Next, Scaife argues that the timing of the suspensions is suspicious. “Close temporal proximity provides evidence of causation and may permit a plaintiff to survive summary judgment provided that there is other evidence that supports the inference of a causal link.” Lang, 361 F.3d at 419 (citations omitted). But rather than create an inference of causation, the timing of Scaife’s suspensions weighs against him. The notices of Scaife’s suspensions, which Scaife provided to supplement his response, were issued on the dates of hearings which had been scheduled to resolve Scaife’s grievances challenging the recommended suspensions for Scaife’s alleged tardiness. Had Scaife prevailed on these grievances, the suspensions would have been abated or not imposed at all. The notices set out in detail the circumstances surrounding the hearings and Scaife’s failure to attend. The notices also state that the recommended suspensions were upheld not on the merits, but because Scaife had defaulted upon the very grievances he had filed to contest them. Scaife’s absences from the grievance hearings are conspicuously missing from his argument and, when taking into account that Scaife did prevail on other grievances, provide a compelling and unrebutted explanation of the suspensions. Thus, Scaife’s retaliation claim cannot go forward.

Finally, Scaife’s claims against Cook County and Michael Sheahan in his official capacity as Cook County Sheriff do not warrant detailed discussion. Scaife cannot proceed on these claims because, at this juncture, it is impossible to find a constitutional deprivation, nor does Scaife argue any differently. See McCormick v. City of Chicago, 230 F.3d 319, 324 (7th Cir.2000).

III. CONCLUSION

For the foregoing reasons, the district court’s grant of summary judgment in favor of the defendants was proper, and its judgment of dismissal is Ajffirmed.

. The remedy Scaife obtained by prevailing on Grievance No. 99-012 is not clear, particularly in light of his subsequent labor-relations claim seeking enforcement of the grievance.

. The exact contents of this grievance are not included in the record.

. Although Count IX of the complaint referred to Chapter 27 of the United States Code, it is clear from the allegations that the claim actually arises under Chapter 29.