I believe you are miss reading the correlation, having more children isn't making people poorer, it's part of a more complicated cycle of poverty that involves lower levels of education and attainment which leads to poorer income and poorer fertility planning. The strongest correlation is with female education levels.