Milter-greylist

Hi.  So, I've been getting a lot of spam that is not tagged by 
spamassassin.  These messages have a header that says:

X-Greylist: Default is to whitelist mail not delayed by milter-greylist

That seems problematic.  I've been getting a bunch of mail which passed 
untrammeled by milter-greylist, but it still needs to be checked.

How do I turn off this whitelisting behavior?

--Ryan