In 2003 and 2005, one party had complete control of the US federal government. I do agree that Fannie and Freddie, not to mention banks and mortgage service companies in general, needed more oversight. One party had the opportunity to fix it both years and didn't. This baby rests with everybody in DC. They're all crooks and incompetents, regardless of affiliation.
The political party in question in power in 2005 did not do what their supporters sent them to Washington to do, and as a consequence, they did not receive a whole lot of support when they ran. Gay congressman scandals (really, of lower magnitude than Jerry Studds or Barney Frank's pecadilloes, but if one considers how Gingrich got dumped by his own side after he was caught cheating on his wife, one side is less forgiving of immorality, IMHO)) didn't help any, but when sent to Washington to limit spending and control growth of government, and a Republican Congress and President did neither, they are not going to get enthusiastic support in fund raising and get out the vote and the such.
CRA was just one ingredient, private and public sector greed went along way. I hope we get Enron style hearings, and maybe criminal prosecutions, both from the private banks and Freddie and Fannie.