The brain has a critical window for language development between the ages of two and four, brain scans suggest.
Environmental influences have their biggest impact before the age of four, as the brain's wiring develops to process new words, say UK and US scientists.
The research in The Journal of Neuroscience suggests disorders causing language delay should be tackled early.
It also explains why young children are good at learning two languages.
The scientists, based at King's College London, and Brown University, Rhode Island, studied 108 children with normal brain development between the ages of one and six.
They used brain scans to look at myelin - the insulation that develops from birth within the circuitry of the brain.
To their surprise, they found the distribution of myelin is fixed from the age of four, suggesting the brain is most plastic in very early life.
Any environmental influences on brain development will be strongest in infanthood, they predict.
This explains why immersing children in a bilingual environment before the age of four gives them the best chance of becoming fluent in both languages, the research suggests.
It also suggests that there is a critical time during development when environmental influence on cognitive skills may be greatest.
Dr Jonathan O'Muircheartaigh, from King's College London, led the study.
He told the BBC: "Since our work seems to indicate that brain circuits associated with language are more flexible before the age of four, early intervention for children with delayed language attainment should be initiated before this critical age.
"This may be relevant to many developmental disorders, such as autism, since delayed language is a common early trait."
Early childhood is a time when language skills develop very rapidly.
Babies have a vocabulary of up to 50 words at 12 months but by the age of six this has expanded to about 5,000 words.
Language skills are localised in the frontal areas of the left-hand side of the brain.
The researchers therefore expected more myelin to develop in the left-hand side of the brain, as the children learned more language.
In fact, they found it remained constant, but had a stronger influence on language ability before the age of four, suggesting there is a crucial window for interventions in developmental disorders.
"This work is important as it is the first to investigate the relationship between brain structure and language across early childhood and demonstrate how this relationship changes with age," said Dr Sean Deoni from Brown University, a co-researcher on the study.
"This is important since language is commonly altered or delayed in many developmental disorders, such as autism."
Commenting on the study, Prof Dorothy Bishop of the department of Developmental Neuropsychology at the University of Oxford said the research added important new information about early development of connections in brain regions important for cognitive functions.
"There is suggestive evidence of links with language development but it is too early to be confident about functional implications of the findings," she said.
"Ideally we would need a longitudinal study following children over time to track how structural brain changes relate to language function."
The study was funded by the National Institutes for Mental Health (US) and the Wellcome Trust (UK).