But they caution that more investigation is needed to understand whether the clear pattern they found among the hundreds of people in the United Kingdom they studied did indeed mean long-term damage.
In one of the largest studies of its kind to date, and published today in prestigious scientific journal Nature, a team of researchers led by Oxford University's Professor Gwenaëlle Douaud analysed changes in the brains of 785 participants in the UK Biobank.
People in the study had two brain scans around 38 months apart, and also underwent cognitive tests.
A total 401 participants tested positive with mostly-mild Covid-19 infection between their two scans, while the remaining 384 non-infected people were used as age and sex-matched controls for the study.
When the researchers compared those two groups over time, they found "significant, deleterious" long-term effects among those people infected with the virus: notably a reduction in grey matter thickness in regions associated with smell and memory of events.
They also observed tissue damage in regions linked to smell, while post-infected people showed larger cognitive decline than controls between scans.
"On average, the participants who were infected with Sars-CoV-2 also showed greater cognitive decline between their two scans, associated with the atrophy of a brain region known as the cerebellum, which is linked to cognition," the researchers said in a press statement.
"The findings may indicate the degenerative spread of Covid-19, either via smelling pathways, inflammation of the nervous system or a lack of sensory input owing to a loss of smell."
Importantly, the documented effects were still seen after excluding the 15 people who had been hospitalised with Covid-19 - implying even mild illness may have consequences for the brain.
The future vulnerability of those affected regions, they concluded, warranted further investigation.
University of Auckland neuroscientist Professor Maurice Curtis said the shrinkage or loss of brain volume shown in the study was "significant" - exceeding a six per cent difference on average.
"Furthermore, those who demonstrated this shrinkage, performed significantly worse on executive function, visual searching and mental flexibility tests," he said.
"We know that loss of the sense of smell very early on in Covid-19 is a key sign of infection and some people never get their sense of smell back.
"The smell pathway and the memory pathway in the brain are connected and these are the same pathways affected in some dementias, including Alzheimer's disease."
He also pointed out the study authors' conclusion that Covid-19 caused the shrinkage - whereas being infected with the seasonal flu didn't result in such an effect.
"This study demonstrates that there is a long-term consequence to getting Covid-19 and it highlights the importance of taking all measures possible to reduce Covid-19's impact on the body and especially the brain."
Another Kiwi brain expert, Dr Indranil Basak of Otago University's Neurodegenerative and Lysosomal Diseases Laboratory, said the study findings were "very interesting, but worrying as well".
His own research team was exploring what happened to our brains - at a molecular and cellular level – when it was exposed to the virus.
"The results from our experiments will help us understand what is happening inside the cells, which could be leading to the changes in the brain that are highlighted in this study," he said.
"Our preliminary data shows some infection in brain cells, including neurons. However, we still don't know if the virus can enter these cells after crossing the blood-brain barrier, or if the symptoms we're seeing inside the cells are because of some other reason."
For example, he said, they'd observed an immune response that occurred when the virus was infecting our bodies, which could be causing secondary effects that affected the brain cells as well.
"There are definitely several cases where patients infected with the virus started getting neurological symptoms like dizziness, disturbed consciousness, headache, loss of smell and taste, seizure, encephalitis and there have been links with Parkinson's disease," Basak said.
"From this research and our own, it is clear there is an effect on the brain from Covid-19 infection, and this could lead to some Long Covid effects.
"We still don't know how to treat this, because no one has looked at it yet. But we do know that the virus directly, or indirectly, can affect the human brain."