You could do a tracert to see where it is picking up the latency.

Normally when this occurs it is b/c your ISP is doing something to your packets. I once saw a case where they routed this one guy to NYC to DET back to NYC and then back to London and then back to NYC and then to DC he called his ISP daily to complain, it took about 6 months before they fixed it.