Some schools may be moving pupils off their registers as an "easy way" to improve exam data, the watchdog Estyn has warned.
The proportion of pupils who do not move from year 10 to the final year of GCSEs has nearly doubled in six years.
Estyn said data suggested the majority of pupils repeating year 10 were "not doing so for genuine reasons".
The Welsh Government said the increase in pupils not progressing to year 11 was worrying.
And a head teachers' union said there were many different reasons why a pupil might be taken off a school's register.
The report looks at "off-rolling" - pupils being moved from a school's roll without being formally excluded - and is often referred to in relation to improving exams data.
Using data from the annual schools' census, the watchdog found 1,352 pupils did not progress from year 10 to year 11 in 2017-18.
Pupils eligible for free school meals and those with special educational needs were far more likely not to progress up to year 11 than their peers.
Around 10% of year 10 pupils eligible for free school meals did not move up to year 11 last year.
What happens to these pupils?
Estyn looked at pupils who repeated year 10 and those who moved into other provision, known as Education Other than at School (EOTAS).
"Of the 91 pupils that repeated year 10 in 2018-2019, over half of these pupils were at a very few schools," it said.
Pupil referral units are the main providers of EOTAS but it also covers a range of other provision by further education colleges, independent schools or tuition.
Some children split their time between school and EOTAS.
Estyn found the numbers transferring to EOTAS had not changed significantly.
But there had been a marked increase in the percentage of these pupils whose main education provider was listed as EOTAS, rather than the school.
That meant their results would not count towards their school's exam performance.
The watchdog said there were inconsistencies in how well local authorities were monitoring pupil registrations, and arrangements in a few had "not been robust enough".
The report says some council officials said schools were eager for pupils to be mainly registered with the other education setting so their results would not count.
"It is seen by schools as an easy way for a school to improve its performance data," the report said.
It calls on local authorities to monitor how schools register pupils "to provide assurance that schools are always acting in the best interests of individual pupils".
Because of the small numbers, the analysis is incomplete for some local authorities.
But the most recent figures showed Gwynedd, Wrexham, Anglesey and Carmarthenshire had the highest percentages of pupils not moving on to year 11, while Merthyr Tydfil and Denbighshire had the highest percentage of pupils moved to EOTAS as the main provider.
What is the response of ministers and teachers?
The Welsh Government said it was currently developing a new approach to pupil registration and school evaluation to prevent off-rolling.
"It is worrying that the number of children who do not progress from year 10 to year 11 has increased," said a spokesman.
"We will be consulting on draft regulations which will require each local authority to establish a database of all school-aged children, in order to identify those who are not on a school roll, EOTAS register or independent school roll and are not receiving a suitable education."
Head teachers' union NAHT said there were many different reasons why a pupil might leave a school's roll.
"The practice of off-rolling a pupil to 'game' the data is wrong. But not all unexplained pupil exits are off-rolling," NAHT Cymru director Rob Williams said.
"If we can identify the true extent of questionable practice, and its root cause, we can ensure that schools are supported to make better decisions.
"But this mustn't be at the expense of the majority of schools, who may have exits which are unexplained but are actually for valid reasons."
Eithne Hughes, director of the Association of School and College Leaders Cymru, said: "The vast majority of school leaders would only ever suggest that a pupil remains in year 10 for a further year in exceptional circumstances and when it is clearly in the best interests of the pupil.
"Most school leaders will be concerned to see that there has been a sharp rise in the percentage of pupils who repeat year 10 and will be more keen than anybody to ensure that this is never done as a means of manipulating performance tables."