I ran into a interesting situation where the server was running super heavy on apache, maxing it’s connections, and server load. It turned out varnish was not caching items when it should, and was not giving a expiration time either.
In varnishstat it would show the miss count almost as great as what it was attempting to cache.
I was able to tell that it was using vary: User-Agent from looking at the content in the Apache Access logs, and running a curl against a item that was static curl -Iv http://blah.com/image.jpg
In that you would see the Header, and I was missing an expiration date as well which was a bit strange.
We checked out his .htaccess file and saw that W3 Total Cache for WordPress was setting the header to !dont-vary.
Header append Vary User-Agent env=!dont-vary
Commenting out this line helped solve this problem, varnishstat seems to have near 100 percent hit rate after this point.
A note on what vary: User-Agent does with varnish is store different cache for every different browser/client and browser/client version you are using. So you are going to see a ton of misses, if everyone is using a different version of firefox for instance, and varnish is creating a cache for each.