Training for instruction following, reasoning, reflection, and self-monitoring may have forced models to learn resource allocation: compress the currently useful concepts into a sparse shared workspace instead of keeping all computation equally active.
This is actually interesting observation, what if they add resource allocation constraints in post training, will it force to add more workspaces?
I believe this may impact on post training.
Training for instruction following, reasoning, reflection, and self-monitoring may have forced models to learn resource allocation: compress the currently useful concepts into a sparse shared workspace instead of keeping all computation equally active.
This is actually interesting observation, what if they add resource allocation constraints in post training, will it force to add more workspaces?