Your right, it really is a question of physics. There seems to be a correlation between pixel size and light gathering ability. That is probably why Cannon and Nikon went for FF35. But where is the true limit of digital sensors?
But in a way, it does not really matter what is the absolute limit of sensors. What matters, is where that limit will be for the next 10 years when film transfers to digital and new standards are created based on digital sensors. Once a standard is set for Digital Cinematography, all the accessories, lenses and talent will adjust to that standard. Even if the light gathering drastically improves, larger sensors and larger lenses will probably always gather more light. So the improvements, may just be in image quality, rather than sensor size.
We might have some small sensors for specific action shots, but the main cameras will always be based on the new standard.
Again, this is the opinion of someone with no technical knowledge.