If you set the TTL for a particular origin to 0, CloudFront will still cache the content from that origin. It will then make a GET request with an If-Modified-Since header, thereby giving the origin a chance to signal that CloudFront can continue to use the cached content if it hasn't changed at the origin.
I wonder how well this works for content that is truly dynamic. Seems like it would necessarily be slower for those pages that change on every request.
Not necessarily. The networks used by CloudFront may outperform direct paths. More importantly, CloudFront edge locations try to maintain a persistent connection to the origin and use a large initial TCP congestion window. This saves you from the delay caused by setting up a TCP connection over a long network path.
Simplified example (ignoring DNS latency, assuming symmetric paths):
User to CloudFront RTT is 30ms
User to Origin RTT is 100ms
CloudFront to Origin RTT is 100ms
It seems clear that User to Origin is faster than User to CloudFront to Origin, but not if you consider TCP mechanics.
If the User makes an HTTP request for a 4KB file to the Origin directly, it will take 100ms to set up the connection, 50ms for the request to reach the Origin, and another 50ms for the first response byte to arrive. Total: 200ms. If the origin does not have a big initial congestion window, it will take another 100ms for the last byte to arrive. Total: 300ms.
If the User makes an HTTP request for a 4KB file through CloudFront, it will take 30ms to set up the TCP connection. The request packet(s) will take 15ms to reach CloudFront. CloudFront tries to maintain a persistent TCP connection to the Origin, which avoids set-up time and slow start. The request to the Origin will take 100ms to complete, and another 15ms to reach the User. Total: 160ms.
Using CloudFront as an intermediary could reduce latency by a lot, even if no caching is going on.
I bet they use a grace period, like Varnish does. The grace period would let the cache serve stale data for a few seconds or more while the if-modified-since call is made, and if necessary, the cache refreshed.
I don't think that's what the post said. Doing so would break the semantics of the 0 second cache time. They must wait for your 304 Not Modified response before serving from their cache.
There's nothing stopping you from using a traditional CDN vendor as an application CDN. I spoke to the folks at Edgecast about this a year or two ago, and they didn't have a problem with it. However, it sounds like Amazon made some specific optimizations for application content which could be more appealing.
Also, CloudFlare offers the same service, but with added security and anti-spam features.
I wonder how well this works for content that is truly dynamic. Seems like it would necessarily be slower for those pages that change on every request.