Peter Dinklage admits walking away from Game of Thrones was the hardest thing to do