Link to topic assignments, presentation dates, and guest speaker schedules: Google Doc
Link to project presentation dates: Google Doc
Link to presentation videos (subject to removal): Google Doc
1.
Introduction
and course overview (1 lecture)
2.
Graphical
models and approximate inference (2 lectures)
Read/re-visit Bishop,
chapters 8-11.
A.
Variational inference
Tutorial: M.I. Jordan, Z. Ghahramani,
T.S. Jaakkola, L.K. Saul. "An Introduction to Variational Methods for Graphical Models." Machine Learning 37(2), 1999: pp 183-233. [PDF]
Video lecture: http://videolectures.net/mlss04_bishop_gmvm/
B.
Sampling based approaches
Tutorial: C. Andrieu, N.
de Freitas, A. Doucet, M.I.
Jordan. "An Introduction to MCMC for Machine Learning." Machine Learning 50(1), 2003: pp 5-43. [PDF]
Video lecture: http://videolectures.net/mlss09uk_murray_mcmc/
Radford NealÕs book http://www.cs.utoronto.ca/~
Software: Infer.NET
is a framework for running Bayesian inference in graphical models.
Topic 3 onwards are covered
through student led presentations. * denotes a lead
paper.
3.
Topic
models (3 lectures)
Bibliography and pointers to code
at
http://www.cs.princeton.edu/~mimno/topics.html
David Blei's resource page
:
http://www.cs.princeton.edu/~blei/topicmodeling.html
David Blei's KDD 2011 tutorial
:
[PDF]
Data sets:
http://www.cs.umass.edu/~mccallum/data.html
http://cs.nyu.edu/~roweis/data.html
http://www.zjucadcg.cn/dengcai/Data/TextData.html
A.
Survey/overview: LDA and
extensions
Slides: http://www.cs.umd.edu/class/spring2008/cmsc828g/Slides/topic-models.pdf
Video lecture: http://videolectures.net/mlss09uk_blei_tm/
D.M. Blei.
"Introduction to Probabilistic Topic Models." Communications of the ACM
(to appear 2011). [PDF]
* D.M. Blei, A. Ng, M. Jordan. "Latent Dirichlet allocation." Journal of Machine Learning Research 3, 2003: pp 993-1022. [PDF]
* D.M. Blei,
J.D. Lafferty. " A Correlated Topic Model of Science." Annals of Applied Probability, 2007. [PDF].
C Code at www.cs.princeton.edu/~blei/ctm-c/
A. Agovic, A.
Banerjee. "Gaussian Process Topic Models." UAI 2010. [PDF]
B.
Hierarchical approaches
* D.M. Blei, T.L. Griffiths, M.I. Jordan. "The Nested Chinese
Restaurant Process and Bayesian Nonparametric Inference of Topic
Hierarchies." Journal of the ACM 57(2), 2010: 1-30. [PDF]
John Paisley, Chong
Wang, David Blei. "The Discrete Infinite
Logistic Normal Distribution for Mixed-Membership Modeling." AISTAT 2011. [PDF]
Y. Teh, M. Jordan, M. Beal, and D. Blei,
Hierarchical Dirichlet Processes. Journal of American Statistical Association, 2006. [PDF].
Tutorial by Teh at mlg.eng.cam.ac.uk/tutorials/07/ywt.pdf
C.
Evaluation, inferencing, and online/parallel
* H.M.
Wallach, I. Murray, R. Salakhutdinov, D. Mimno. "Evaluation Methods for Topic Models."
ICML 2009. [PDF]
* A. Asuncion, M.
Welling, P. Smyth, Y-W. Teh. "On Smoothing and
Inference for Topic Models." UAI
2009. [PDF]
M. Hoffman, D.M. Blei, F. Bach. "Online Learning for Latent Dirichlet Allocation." NIPS 2010. [PDF]
* D. Newman, A.
Asuncion, P. Smyth, M. Welling. "Distributed Algorithms for Topic
Models." Journal of Machine Learning
Research 10, 2009: pp 1801-1828. [PDF]
4.
CRFs
and information extraction (1 lecture, guest speaker)
* C. Sutton, A. McCallum.
"An Introduction to Conditional Random Fields."
http://arxiv.org/abs/1011.4088 [PDF]
CRF Software: http://www.inference.phy.cam.ac.uk/hmw26/crf/#software
CRF resources http://iic.arizona.edu/search/by_tag?tag=crf
J. Zhu, N. Lao, N. Chen, E.P. Xing. "Conditional Topical Coding: An Effcient Topic Model Conditioned on Rich Features." KDD 2011. [PDF]
5.
Mining
Multi-relational Data for Affinity Estimation (6 lectures)
Data sets:
Several
Affinity Datasets
$1 Million Prize to Speed Innovation in Retail
Personalization (Overstock) http://overstockreclabprize.com/
ECML/PKDD 2011 Discovery Challenge: VideoLectures.Net recommender system.
A.
Matrix factorization
(probabilistic or otherwise)
Tutorial for PMF
and RBMs for CF: http://www.cs.toronto.edu/~hinton/csc2515/notes/pmf_tutorial.pdf
* R. Salakhutdinov, A. Mnih. "Probabilistic Matrix Factorization." NIPS 2008. [PDF]
R. Salakhutdinov,
A. Mnih. "Bayesian Probabilistic Matrix
Factorization using MCMC." ICML 2008.
[PDF]
[Code: http://www.mit.edu/~rsalakhu/BPMF.html]
*R. Gemulla, P.J.
Haas, E. Nijkamp, Y. Sismanis. "Large-Scale Matrix Factorization with
Distributed Stochastic Gradient Descent." KDD 2011 [PDF]
*A.P. Singh, G.J. Gordon. "A Unified View of Matrix Factorization Models." ECML/PKDD 2008. [PDF]
G. Tak‡cs, I. Pil‡szy, B. NŽmeth, D. Tikk. "Scalable collaborative filtering approaches for
large recommender systems." Journal
of Machine Learning Research 10,
2009: pp 623-656. [PDF]
B.
Block models
* E.M. Airoldi, D.M. Blei, S.E.
Fienberg, E.P. Xing. "Mixed Membership Stochastic Blockmodels."
Journal of Machine Learning Research 9, 2008: pp
1981-2014. [PDF]
I. Sutskever, R. Salakhutdinov, and
J. Tenenbaum. "Modelling
Relational Data using Bayesian Clustered Tensor Factorization." NIPS 2010. [PDF]
* L. Mackey, D.
Weiss, M.I. Jordan. "Mixed Membership Matrix Factorization." ICML
2010 [PDF
slides] [MATLAB code at
http://code.google.com/p/m3f/]
C.
Multi-relational data
analysis adding "side information"
* H. Shan, A.
Banerjee. "Generalized Probabilistic Matrix Factorizations for
Collaborative Filtering." ICDM 2010.
[PDF]
* R.P. Adams, G.E.
Dahl, I. Murray. "Incorporating side information into probabilistic matrix
factorization using Gaussian Processes." UAI 2010. [PDF]
*A.K. Menon, K-P
Chitrapura, S. Garg, D. Agarwal, N. Kota. "Response Prediction Using
Collaborative Filtering with Hierarchies and Side-information." KDD 2011 [PDF]
I. Porteous, A. Asuncion, M. Welling. "Bayesian Matrix
Factorization with Side Information and Dirichlet
Process Mixtures." AAAI 2010. [PDF]
M.S. Handcock, A.E. Raftery, J.M.
Tantrum. "Model-based clustering for social networks." J. of the Royal
Statistical Society: Series A (Statistics in Society) 170, 2007: pp 301-354. [PDF]
D.
Multi-level Bayesian
modeling
*D. Agarwal, B-C
Chen, B. Long. "Localized Factor Models for Multi-Context
Recommendation." KDD 2011 [PDF]
* D. Agarwal, B-C Chen. "Regression-based latent factor
models." KDD 2009. [PDF]
P. Hoff.
"Hierarchical multilinear models for multiway data." 2009. [PDF]
E.
Supervised learning in
multi-relational data; extracting structure/meaning
* D. Agarwal, B-C Chen. "fLDA: Matrix Factorization through Latent Dirichlet Allocation." WSDM 2010. [PDF]
* D.M. Roy, C.
Kemp, V.K. Mansinghka, J.B. Tenenbaum.
"Learning annotated hierarchies from relational data." NIPS 2006. [PDF]
I. Porteous, E. Bart, M. Welling. "Multi-LDA/HDP A Non
Parametric Bayesian Model for Tensor Factorization." AAAI 2008 [PDF]
A.K. Menon, C. Elkan. "A
log-linear model with latent features for dyadic prediction." ICDM 2010. [PDF] [Slides] [Code]
F.
Dynamics
*D. Stern, R. Herbrich, and T. Graepel.
"Matchbox: Large Scale Bayesian Recommendations." WWW 2009. [PDF]
*Q. Ho, L. Song, E.
Xing. "Evolving Cluster Mixed-Membership Blockmodel
for Time-Evolving Networks." AISTAT 2011. [PDF]
L Tang, H Liu, J
Zhang, H. Liu. "Community evolution in dynamic multi-mode networks."
KDD 2008. [PDF]
L. Xiong,
X. Chen, T. Huang, J. Schneider, J. Carbonell, ÒTemporal Collaborative Filtering with
Bayesian Probabilistic Tensor FactorizationÓ, SIAM Data Mining (SDM),
2010. [PDF]
6.
Transfer
and multi-task learning (3 lectures)
Data sets and resources:
Transfer Learning Resources: Code, Data,
Surveys [Homepage]
Unsupervised and Transfer Learning Challenge (http://www.causality.inf.ethz.ch/unsupervised-learning.php#cont)
A.
Overview
*S.J. Pan, Q. Yang.
"A Survey on Transfer Learning." IEEE
Transactions on Knowledge and Data Engineering. 22(10), 2010: pp 1345-1359. [PDF].
Also see slides by S.J. Pan. [PPT]
E.H. Zhong, W. Fan,
Q. Yang, O. Verscheure, J. Ren. "Cross Validation Framework to Choose
amongst Models and Datasets for Transfer Learning." ECML/PKDD 2010. [PDF]
B.
Transfer Learning Challenge
*
"Unsupervised and Transfer Learning Challenge." (http://www.causality.inf.ethz.ch/unsupervised-learning.php#cont)
I. Guyon, G. Dror, V.
Lemaire, G. Taylor, D.W. Aha. "Unsupervised and Transfer Learning
Challenge." IJCNN 2011 [PDF]
C.
Multitask learning
B. Cao, N. Liu, Q.
Yang. "Transfer Learning for Collective Link Prediction in Multiple
Heterogeneous Domains." ICML 2010
[PDF]
*Laurent Jacob,
Francis Bach, Jean-Philippe Vert "Clustered Multi-Task Learning: a Convex
Formulation." NIPS 2008 [PDF]
7.
Distributed and Privacy
Preserving Data Mining
(2 lectures)
A.
Privacy preserving learning
*S. Merugu, J. Ghosh. "Privacy-perserving
distributed clustering using generative models." ICDM 2003. [PDF] (also see paper at KDD 2005)
*M. Kearns, J. Tan,
J. Wortman. "Privacy-Preserving Belief
Propagation and Sampling." NIPS 2007. [PDF]
B.
Measures and limitations
*S. Chawla, C. Dwork, F. McSherry, A. Smith, H. Wee. "Toward Privacy in Public
Databases." Lecture Notes in
Computer Science, 2005: pp 363-385. [PDF]
[Extended
PDF]
C. Dwork. "Differential Privacy: A Survey of
Results." Lecture Notes in Computer
Science, 2008: pp 1-19. [PDF]
* E. Zheleva, L. Getoor. "To join
or not to join: the illusion of privacy in social networks with mixed public
and private user profiles." WWW 2009.
[PDF] [Slides]
N. Mohammed, R.
Chen, B.C.M. Fung, P.S. Yu. "Differentially Private Data Release for Data
Mining." KDD 2011 [PDF]
8.
Scaling to large data sets and
parallel/cloud computing (2
lectures)
A.
ADDM
* S. Boyd, N.
Parikh, E. Chu, B. Peleato, J. Eckstein.
"Distributed Optimization and Statistical Learning via the Alternating
Direction Method of Multipliers." To appear in Foundations and Trends in Machine Learning.
[Webpage]
[PDF]
[Slides]
MATLAB scripts: http://www.stanford.edu/~boyd/papers/admm/
B.
Articles
Choose chapters
from R. Bekkerman, M. Bilenko,
and J. Langford (Eds) "Scaling Up Machine
Learning", Cambridge University Press, 2011. [http://www.cs.umass.edu/~ronb/scaling_up_machine_learning.htm]
Recent workshop: http://lccc.eecs.berkeley.edu/
9.
Additional
resources
General:
A Halevy, P. Norvig,
F. Periera, "The Unreasonable Effectiveness of
Data." IEEE Computer 24, 2009: pp
8-12. [Abstract]
Topic
Models:
C. Wang, D.M. Blei. Lafferty.
"Collaborative Topic Modeling for Recommending Scientific Articles." KDD 2011 [PDF]
D.M. Blei, J.D.
Lafferty. "Dynamic Topic Models." ICML
2006. [PDF]
G. Doyle, C. Elkan.
"Accounting for Burstiness in Topic
Models." ICML 2009. [PDF]
Volker Tresp. "Multivariate Models for Relational
Learning." Tutorial, ILP
2010. [PDF]
Multirelational and Network Models:
A. Goldenberg, A.X.
Zheng, S.E. Fienberg, E.M. Airoldi.
"A Survey of Statistical Network Models." [PDF]
X. Su, T.M. Khoshgoftaar. "A survey of collaborative filtering
techniques." Advances in AI 2009, 2009: pp
1-19. [Abstract] [PDF]
E. Zheleva, G. Namata.
"Stochastic Blockmodels: A Survey." [PDF
slides]
J. Tenenbaum. "How to Grow a Mind: Statistics, Structure
and Abstraction". NIPS 2010 Posner
Lecture. [Video]
J.B. Tenenbaum, C. Kemp, T.L. Griffiths, N.D. Goodman. "How
to Grow a Mind: Statistics, Structure, and Abstraction." Science 331, 2001: pp 1279-1285. [PDF] [Supplementary material]
Kai Yu's Homepage: http://www.dbs.informatik.uni-muenchen.de/~yu_k/
Other Challenges
Active Learning
Challenge [Homepage]
I. Guyon, G. Cawley, G. Dror, V.Lemaire. "Results of
the Active Learning Challenge." JMLR
2011. [PDF]
Heritage Health
Prize Challenge [Homepage]
Check out http://www.kaggle.com/