Are you trying to set a resolution above your displays native resolution? That is what AA is for...

Unless your having some strange driver related issue when usign AA which is what your talking about.