Gang Fang's Blog

Do “discriminative function” and “discriminative model” refer to the same thing?

In general, yes, they refer to models that define the decision boundary between classes, like logistic regression, SVM and most ANNs. The only difference, in a more specific context, is a discriminative function can refer to the actual mathematical function that takes the input and makes a prediction.

How are they different from generative models?

Discriminative models do not model how the data was generated, they just differentiate between different classes of data. They aim to directly model the conditional probability of the target given the input, P(y|x), while generative models model the joint probability of the target and the input, P(x, y), or equivalently, P(x|y)P(y).

What does it mean when we say “you need to be able to generate samples from the model.”?

It means that the model can generate data within the same distribution.

Let’s consider an example using a generative model. Suppose you have a generative model that has been trained on a dataset of images of handwritten digits (like the MNIST dataset). This model has learned the joint probability distribution of the pixel intensities and the class labels of the digits.

Now, let’s say you want to generate a new image of a handwritten ‘3’. You could sample a new image from your model by first picking the label ‘3’, and then sampling pixel intensities based on the conditional distribution of pixel intensities given that the digit is ‘3’. The result would be a new image that the model “believes” could reasonably be a handwritten ‘3’.