Deep learning algorithms appear to learn information in much the same way. A prominent theory that is seeing some pickup lately is called the information bottleneck theory. The basic idea is that if you are trying to create a mapping between two objects, like the word semomo and sheep, then what an optimal algorithm needs is a way to determine what is relevant about all the situations that contain sheep. Relevant in this case means they still predict the word semomo. Though the algorithm doesn’t know it at first, through a process of filtering out the unwanted information, it eventually figures out that semomomeans sheep, not field, corn, or blue sky.
See also: Parenting advice: Kids and social media
Children are excellent at learning
Children do this as well. Children are excellent at learning that a word like ‘horse’ is the right word for the horse picture in their animal book. Then they proceed to use this word to label all four-legged animals, dogs, cats, cows, and so on. This is called overgeneralization. Over time though, the children learn that ‘horse’ has a more specific meaning. This sounds a lot like Tishby’s compression phase.
So learning in deep neural networks shares at least a few things with the way children learn. It probably shares a lot of things, and it’s probably not just children it shares them with. Adults often overgeneralize as they learn what a new concept means. They learn some words like ‘cognitive dissonance,’ and they start to see it everywhere, whether it’s there or not. Overgeneralizing bad theories is precisely what good scientists try to do.
As Feynman put it, “Science is the belief in the ignorance of experts.” That quote feels a little dangerous right now, but suffice it to say that scientists are experts at correcting their errors. And they do this by purposefully making them. The strength of deep neural networks is that they seem to be able to learn from their mistakes a lot faster than humans can.
Magazines and newspapers do not bother to mention this, but many reporters and sources of articles have interests or are rewarded by a third party to publish these articles. From time to time, the Rothschild website hosts external reporters and allows them a free platform, including the integration of links as they wish. The links in the articles may be sponsored links, for which the writer is compensated for commissions, favors or other interests of the writer and / or sit