Category: AI
-
Seeds for research
I adopted the concept of seeds from the book, the creative act by Rick Rubin here to document all interesting ideas that I want to pursue scientifically: (they will be organized into blog posts) Most seeds above come from https://yoshuabengio.org/2023/03/21/scaling-in-the-service-of-reasoning-model-based-ml/ -> GFlowNet is a manifestation of all these ideas. More seeds
-
Inductive Biases
“”” Inductive bias refers to the set of assumptions that a learner (human or machine) makes to predict outputs for previously unseen inputs when trying to learn from a given set of training examples. In other words, it’s the bias or set of preferences a learning algorithm has when inferring a hypothesis or function from…
-
More concepts
“Convex Combination” The term “convex” in mathematics usually refers to a set of points that, if you take any two points within the set and draw a line segment between them, every point on that line segment also belongs to the set. However, in the context of the paragraph, “convex” refers to a specific type…
-
Reductionism Can’t Address Complex Systems
A colleague once said that drones are not innovative because they are just four motors and some plastic put together, arguing that these are existing components. However, I don’t believe that this reductionist argument fully explains the true nature of these devices. This particular argument stuck with me, as I had recently been contemplating a…
-
Current AI as Artificial Intuition
Looking at state-of-the-art (SOTA) artificial intelligence (AI), which is based on deep learning (DL), through the lens of dual process theory, it’s easy to see that what SOTA AI truly accomplishes is the intuitive aspect of intelligence (in the broadest sense). This is akin to our ability to automatically coordinate our bodies when we run…
-
Distributed representation
Distributed representation is at the core of why ANNs work. What is distributed representation in ML? Distributed representation in machine learning refers to a way of representing data where each input is described by numerous elements or features, and each feature can contribute to the representation of many inputs. In other words, the information is…
-
Compositionality
It helps IID and OOD generalization because lower level features in smaller quantities can be combined in different way to generalize to something out of distribution of the training data. It has forms such as: And the Bengio’s goal is to introduce more forms of compositionality.
-
Dual process theory and key questions in SOTA DL
As I understood more about how Type 1 / Type 2 processing is defined and what the Global Workspace Theory for consciousness is about. I came to realized while Type 2 cognitive processing and consciousness explained by GWT share a striking similarity, namely, they both require the working memory to hold and process elements coming…
-
World Model
Bengio’s world model Bengio’s concept of world model is a generative model that learns the abstract structure, or abstraction, of x. And this is similar to how human learns and understands the world. This is powerful because this enables training with far less data (because we don’t operate at the low level anymore) and better…
-
Discriminative function/model vs. Generative model
Do “discriminative function” and “discriminative model” refer to the same thing? In general, yes, they refer to models that define the decision boundary between classes, like logistic regression, SVM and most ANNs. The only difference, in a more specific context, is a discriminative function can refer to the actual mathematical function that takes the input…
Gang Fang's Blog 