Discriminative vs. Generative model

总是听到这两个术语，但是又一直不清楚它们最本质的区别。今天花了一小点时间来彻底的弄清楚了。得到的结论如下：
Discriminative Model是判别模型，又可以称为条件模型，或条件概率模型。
Generative Model是生成模型，又叫产生式模型。
二者的本质区别是
discriminative model 估计的是条件概率分布(conditional distribution)p(class|context)
generative model 估计的是联合概率分布（joint probability distribution）p()

常见的Generative Model主要有：
– Gaussians, Naive Bayes, Mixtures of multinomials
– Mixtures of Gaussians, Mixtures of experts, HMMs
– Sigmoidal belief networks, Bayesian networks
– Markov random fields

常见的Discriminative Model主要有：
– logistic regression
– SVMs
– traditional neural networks
– Nearest neighbor

Successes of Generative Methods：
? NLP
– Traditional rule-based or Boolean logic systems
Dialog and Lexis-Nexis) are giving way to statistical
approaches (Markov models and stochastic context
grammars)
? Medical Diagnosis
– QMR knowledge base, initially a heuristic expert
systems for reasoning about diseases and symptoms
been augmented with decision theoretic formulation
? Genomics and Bioinformatics
– Sequences represented as generative HMMs

主要应用Discriminative Model：
? Image and document classification
? Biosequence analysis
? Time series prediction


Discriminative Model缺点：
? Lack elegance of generative
– Priors, structure, uncertainty
? Alternative notions of penalty functions,
regularization, kernel functions
? Feel like black-boxes
– Relationships between variables are not explicit
and visualizable


Bridging Generative and Discriminative：
? Can performance of SVMs be combined
elegantly with flexible Bayesian statistics?
? Maximum Entropy Discrimination marries
both methods
– Solve over a distribution of parameters (a
distribution over solutions)