I'd recommend having some admins on your friends list so that if its an issue you can have someone come in and fix the situation right there. And then if the guy still won't give it back he will likely be kicked. And then if he keeps doing it again and again and again then yeah, you might have something there for a short ban.