1.5 second cooldown? 1.0 cooldown? Just so that they have to spend that tiny bit of time NOT invis to give people a chance... If you think that vagabond is a weak race because its cooldown goes up ONE second, then you are clearly using the ult as a crutch. Its a powerful race without non-stop invis-teleporting and much less infuriating. With other races you feel you have a chance - if you are good and fast you can usually pop off a headshot and win. With my level 16 human I often take down maxed Santa and the like, but basically just avoid all fights with vegabonds (same goes with the other invis teleport races). I use radar/lace/wards/roots/etc but it rarely works.
How about restricting some items for vaga? No lace would be a great start!