If we had tried nation-building, we would have been blamed for imposing our institutions on the Libyan people. See: previous attempts in Somalia, Afghanistan, Iraq, Mali, the CAR, Chad, Cote D'Ivore. Or see the constant criticism of foreign aid to any number of developing countries.
In the end, while we probably could have done some more to help stabilize Libya, the fundamental fault was that Gadaffi had intentionally made himself the lynchpin of all Libyan national institutions, and without him, the process of negotiating a national settlement was reduced to... force.
Yet a nation can not (and should not, for that matter) remain under a dictator forever; and the longer the dictator remains, the more rotted even non-government institutions become, making the transition ever-harder.