I've worked this way for several years, in a few different teams.
I guess it sounds weird if you've never done it, but it works really well.
Now and then, someone will ask "Hey, does anyone know X?" or something else, and issues will be resolved quickly. One counter intuitive thing that makes this possible is that since you're already engaged in a conversation with your pair, this turns out to not be disruptive.