Does dns-prefetching ever have negative performance implications?

Our site fetches resources from various first and third parties, like this:

<link rel="dns-prefetch" href="http://ift.tt/1fotH9P">

Some of these URLs aren't used immediately on page load. Is it worth while to not DNS prefetch ever for some reason? Is the recommendation to just prefetch everything you'd possibly use?