Category-selective regions in ventral temporal cortex (VTC) have a consistent anatomical organization, which is hypothesized to be scaffolded by white matter connections. However, it is unknown how white matter connections are organized from birth. Here we scanned newborn to 6-month-old infants and adults to determine the organization of the white matter connections of VTC. We find that white matter connections are organized by cytoarchitecture, eccentricity and category from birth. Connectivity profiles of functional regions in the same cytoarchitectonic area are similar from birth and develop in parallel, with decreases in endpoint connectivity to lateral occipital, parietal and somatosensory cortex, and increases in connectivity to lateral prefrontal cortex. In addition, connections between VTC and early visual cortex are organized topographically by eccentricity bands and predict eccentricity biases in VTC. These data show that there are both innate organizing principles of white matter connections of VTC, and capacity for white matter connections to change over development.