I really think that this could work. It reminds me of a "Web of Trust" system, where you have a few known trustworthy individuals, and the reputation propagates out from them.