Despite the significant role of the gut microbiota in infant health and development, little is known about the ecological processes determining gut microbial community assembly. According to ecology theory, the timing and order of arrival of microbial species into an ecosystem affect microbial community assembly, a phenomenon termed priority effects. Bifidobacterium species are recognized as highly abundant early colonizers of the infant’s gut, partly due to their ability to selectively utilize human milk oligosaccharides (HMOs) from breast milk. However, the role of priority effects in Bifidobacterium community assembly remains unclear. Here, we investigated the Bifidobacterium community assembly in the gut of 25 breastfed Danish infants longitudinally sampled throughout the first 6 months of life. Our results showed that the breastfed infants were often initially, but temporarily, dominated by suboptimal HMO-utilizing Bifidobacterium taxa, such as B. longum subsp. longum, before more efficient HMO-utilizers such as B. longum subsp. infantis, replaced the first colonizer as the dominant Bifidobacterium taxon. Subsequently, we validated this observation using gnotobiotic mice sequentially colonized with B. longum subsp. longum and B. longum subsp. infantis or vice versa, with or without supplementation of HMOs in the drinking water. The results showed that in the absence of HMOs, order of arrival determined dominance. Yet, when mice were supplemented with HMOs the strength of priority effects diminished, and B. longum subsp. infantis dominated regardless of colonization order. Our data demonstrate that the arrival order of Bifidobacterium taxa and the deterministic force of breast milk-derived HMOs, dictate Bifidobacterium community assembly in the infant’s gut.