Assembly of Intermediate-Mass Black Holes Along Star Formation