The expression getting someone's goat is based on the custom of keeping a goat in the stable with a racehorse as the horse's companion. The goat becomes a settling influence on the thoroughbred. If you owned a competing horse and were not above some dirty business, you could steal your rival's goat (seriously, it's been done) to upset the other horse and make it run a poor race. From goats and horses it was linguistically extended to people: in order to upset someone, get their goat.