Another rule of thumb that's used is that responses within a tenth of a second feel instant, within a second will keep the user focussed, and you can get up to about ten seconds before they'll want to do something else while they wait.

Source: http://www.nngroup.com/articles/response-times-3-important-l...