Gang Fang's Blog

DL by IG, YB and AC

I have been reading the Introduction of the book and here are some notes:

• A shallow understanding of mine re DL is that of “depth”. I thought “deep” simply refers to the bigger size or the larger number of hidden layers in a neural net, compared to a not-so-deep network. But when it’s discussed in the book, it refers more to the larger number of LAYERS of compositions of features (i.e., complex features composed of simpler ones) or abstractions we achieve in the system, which doesn’t even need to be neural inspired. And to measure the depth of a model, the depth of the computational graph or the concept graph is used.
• Representation learning is the immediate superset of DL. Only after representation learning comes ML. I didn’t know representation learning or feature learning is such a big deal until reading it, and apparently distributed learning, even though directly related to representation learning, doesn’t CAUSE representation learning.
• The orientation of the field of DL is about BUILDING COMPUTER SYSTEMS. So I think this is also system design but just a very different kind than that I do at Amazon. But since they are both system design, I bet there are commonalities.
  • On a related note, I am surprised to learn that DL borrows in a limited fashion from neuroscience.
• “The age of ‘Big Data’ has made machine learning much easier because the key burden of statistical estimation – generalizing well to new data after observing only a small amount of data – has been considerably lightened.”