The phrase is, "didn't use to be." The word "weren't" would be correct had he been responding to "I was free", but he is responding to "I use to be free", so "didn't" is correct.