Roughly speaking, CPUs are highly optimized for sequential execution, with long pipelines, high power consumption, and complex implementations of arithmetic/logic instructions (I'm not an expert, but I _believe_ these are much _better_ than on GPUs, except possibly for certain graphics-related GPU primitives), but relatively few threads. GPUs are optimized for parallel execution, and run many more threads than CPUs, each with higher latency, but nonetheless with higher overall throughput.