OAKES, Circuit Judge:

Richard Stratton appeals from an order of the United States District Court for the Southern District of New York, Constance Baker Motley, Judge, denying his motion for reduction of sentence pursuant to Fed.R.Crim.P. 35(b).

On November 17, 1983, Stratton was indicted with five others for conspiring to import and distribute over seven tons of hashish in violation of 21 U.S.C. §§ 846, 963. He was also charged with the substantive count of importing hashish in violation of 21 U.S.C. §§ 952, 960, and with engaging in a criminal enterprise in violation of 21 U.S.C. § 848. Stratton immediately moved to dismiss the indictment, arguing that his prior conviction in the United States District Court for the District of Maine on a charge of conspiring to distribute marijuana and hashish raised a double jeopardy bar to all subsequent charges arising out of the “single criminal agreement.” The motion was denied, United States v. Stratton, 583 F.Supp. 1234 (S.D.N.Y.1984), and this court summarily affirmed on an interlocutory appeal, United States v. Stratton, 751 F.2d 373 (2d Cir.1984). Thereafter, a trial was conducted resulting in a verdict against Stratton and two codefendants on all counts.

On December 7, 1984, Stratton was sentenced by Judge Motley to the statutory minimum of ten years’ imprisonment, a $100,000 fine, and a special parole term of five years. The ten-year sentence was to be served consecutively with the fifteen-year sentence that Stratton had already been given in connection with his conviction in the District of Maine. The court gave the following reasons for imposing a consecutive sentence:

I think that your sentence probably should be made consecutive for the reason that it might convince you that cooperation with the government is in your best interest, and so I intend to make your sentence consecutive for this reason — that is, I expect that you will reflect on your conduct since you are in a reflective mood at this time, and I understand that the government can benefit from your cooperation in respect of other people who were involved in this, so if you are interested in getting out of prison soon and really rehabilitating yourself the best way to demonstrate that is to cooperate with the government with respect to this matter.

Now, Mr. Stratton, if you decide to cooperate with the government in this matter you have 120 days to apply to the court for reduction of your sentence, and the court will consider reducing your sentence based upon the nature and the extent of your cooperation with the government. So that in addition to the 15 year sentence you now have a sentence of ten years to follow that.

Thereafter, Stratton moved for a reduction of sentence pursuant to Fed.R.Crim.P. 35(b). In his motion he argued first that Judge Motley impermissibly increased his sentence because of his refusal to cooperate with the Government. Second, he argued that the structure of his sentence *564 violated the Sentencing Reform Act of 1984, Title II of the Comprehensive Crime Control Act, Pub.L. 98-473, 98 Stat. 2015. He claimed that his sentence was vastly disproportionate to sentences generally imposed in cannabis-related offenses. See 18 U.S.C. § 3553(a)(6) (Supp. II 1984). Judge Motley denied the motion, and this appeal followed, with Stratton raising essentially the same issues that the district court considered. The only change is that while Stratton alleged in his motion that his refusal to cooperate with the Government was due to a “deeply ingrained” moral conviction that it is wrong to blame or inform on others for one’s own mistakes, he now contends that he refused to cooperate out of fear of self-incrimination. Thus, he claims that his sentence was increased as a result of exercising his Fifth Amendment right.

We write here to address the issue of whether Stratton’s sentence was based on an impermissible factor, i.e., his refusal to cooperate. In Roberts v. United States, 445 U.S. 552, 100 S.Ct. 1358, 63 L.Ed.2d 622 (1980), the Supreme Court held that a defendant’s failure to cooperate was a valid sentencing consideration. This court, however, has drawn a distinction between increasing the severity of a sentence for a defendant’s failure to cooperate and refusing to grant leniency. “ ‘It is one thing to extend leniency to a defendant who is willing to cooperate with the government; it is quite another thing to administer additional punishment to a defendant who by his silence has committed no additional offense.’ ” United States v. Bradford, 645 F.2d 115, 117 (2d Cir.1981) (quoting United States v. Ramos, 572 F.2d 360, 363 n. 2 (2d Cir.1978)). This distinction may be difficult to apply. See Roberts, 445 U.S. at 557 n. 4, 100 S.Ct. at 1362 n. 4; Mallette v. Scully, 752 F.2d 26, 30 (2d Cir.1984) (taking Roberts n. 4 into account). Nevertheless, “even though the distinction is somewhat illusory, it is the only rule that recognizes the reality of the criminal justice system while protecting the integrity of that system.” Mallette, 752 F.2d at 30.

When applying these standards to the instant case, it is clear that Stratton’s sentence was impermissibly enhanced. Judge Motley specifically stated that the sentences would run consecutively because of Stratton’s refusal to cooperate. In this case, Judge Motley’s comments crossed that fine line between showing leniency, see id., 752 F.2d at 31 (“if Mr. Mallette were willing to assist us to bring the other person to justice, I would find it very easy to be reasonable and lenient”); Bradford, 645 F.2d at 118 (defendants’ failure to cooperate is permissible factor along with other relevant considerations), and punishing a defendant for his silence. In DiGiovanni v. United States, 596 F.2d 74 (2d Cir.1979), the sentencing court made comments very similar to those made by Judge Motley in the present case. At sentencing, the district court stated:

[I]n the light of what I feel is an unwarranted reluctance on your part to assist the Government and so your desire to turn away from this kind of crime, I feel that I must impose on you a more serious sentence____

[I]f ... you change your mind ... about the desirability of helping the Government do some thing about this drug scene which has had its consequences on you, as well as many others, you may make an application to the Court for a reduction of sentence, but not otherwise____

Id. at 75. This court vacated the sentence and remanded for sentencing before a different judge. Here, as in DiGiovanni, the district court improperly enhanced the defendant’s sentence.

We find it unimportant that Stratton did not raise his Fifth Amendment claim in the district court. A defendant’s position would of course be much stronger if his failure to cooperate was based on assertion of his Fifth Amendment rights; but this court has not limited to the Fifth Amendment context its rule on improper sentence enhancement for refusal to cooperate. For example, in at least four cases before this court the defendant’s silence was based on an alleged fear of physical reprisal if cooperation were given. See Mallette, 752 F.2d at 30; Bradford, 645 F.2d at 116; DiGiov *565 anni, 596 F.2d at 75; Ramos, 572 F.2d at 361. It is improper to increase a defendant’s sentence due to his silence regardless of his motivations.

Stratton’s remaining contention that his sentence violated the Sentencing Reform Act is frivolous. The issue of whether the Maine conviction and New York prosecution were based on distinct facts was fully litigated. Thus, concurrent sentences would in any event have been permissible.

Having found that Judge Motley improperly increased Stratton’s sentence because of his silence, we vacate the sentence and remand the case for resentencing before a different judge.