605.744: Information Retrieval, Spring 2009

Lecturer

Textbooks

Introduction to Information Retrieval by Manning, Raghavan, and Schütze, Cambridge University Press, 2008. An online version of the text is available in digital form at the companion web site.

Course Times and Location

Lecture: Thursday, 7:20pm - 10:00pm, Kossiakoff Center Room K-5.
Email should be the primary means of out-of-class communication; however I can meet with students by appointment.

Course Overview: This course covers the storage and retrieval of unstructured digital information. Topics include automatic index construction, retrieval models, textual representations, efficiency issues, search engines, text classification, and multilingual retrieval.

Grading Policy: Grades will be given based on the A,B,C, etc..., scale, per university policy. Work for the class will include homework assignments, an independent research project, a midterm exam, and classroom participation (e.g., quizzes, oral presentations). Refer to the course outline for details.

Academic Integrity: Work for this class is expected to be the result of individual effort; however, unless explicitly prohibited, it is perfectly acceptable to make use of published examples and source code from the literature or public domain - but only if attribution is given. Furthermore, while it is permissible to discuss the general nature of lecture material and assignments with your peers, this does not extend to discussing or revealing solutions or source code. Students are expected to uphold the academic integrity of the university. Students using without reference, published material or copying the work (i.e., particulary source code) of another individual will face consequences such as receiving a zero on the assignment and having the matter referred to the dean. Contact me if you have any questions, no matter how slight, about this policy, or if you have questions about a particular assignment.

Assigned Readings

1/29/09 Chapters 1 and 2 in Manning, Raghavan, and Schütze.
Michael Lesk, The Seven Ages of Information Retrieval
2/5/09 Chapters 3 and 4 in Manning, Raghavan, and Schütze.
2/12/09 Chapters 5 - 7 in Manning, Raghavan, and Schütze.
2/12/09 G. Salton and C Buckley, Term-Weighting Approaches in Automatic Text Retrieval, IPM 24(5), pp. 513-523, 1988.
2/19/09 Chapters 8 and 9 in Manning, Raghavan, and Schütze.
2/26/09 Chapters 11 and 12 in Manning, Raghavan, and Schütze.
3/5/09 Chapters 13, 14, and 15 in Manning, Raghavan, and Schütze.
3/5/09 Goodman et al., Spam and the on-going battle for the inbox., CACM 50(2), pp. 24-33, 2007.
3/12/09 Chapters 19, 20, and 21 in Manning, Raghavan, and Schütze.
4/2/09 K. Kashida, Technical issues of cross-language information retrieval: a review, IPM 41, pp. 433-455, 2005. (Handed out in class)
4/16/09 M. Sanderson, Retrieving with Good Sense, Information Retrieval 2(1), pp. 49-69, 2000.

Handouts

1/29/09 Course Description
1/29/09 Lecture 1 Course Overview, Boolean Model
2/5/09 Lecture 2 Tolerant Retrieval, Inverted File Construction
2/12/09 Lecture 3 Efficiency Issues, Vector-Space Model
2/19/09 Lecture 4 Evaluation / Relevance Feedback
2/19/09 Reading List
2/26/09 Lecture 5 Alternate models for retrieval
3/5/09 Lecture 6 Text Classification
3/5/09 Paper Review Presentation Schedule
3/12/09 Lecture 7 Web IR
3/12/09 A sample exam was given out in class; contact me if you missed the lecture and want a copy.
4/2/09 Lecture 8 Multilingual Retrieval
4/9/09 Lecture 9 Distributed Retrieval; Multimedia
4/16/09 Lecture 10 NLP and IR
4/16/09 Lecture 11 NLP Applications: Information Extraction and Question Answering

Assignments

[Due 2/12/09] HW#1 Corpus Linguistics hw1.pdf. Text data for the first assignment. (Note: consider each paragraph with a 'P' tag to be a document for the purposes of this assignment.)
[Due 2/26/09] HW#2 Inverted Files hw2.pdf. Use the same dataset from HW#1.
[Due 3/12/09] HW#3 Processing Queries hw3.pdf. Use the Time magazine dataset with these queries.
Assigned: 2/19/09 Class-Project
[Due 3/26/09] HW#4 Text Classification hw4.pdf. Data files: coffee.train.txt.bz2, coffee.test.txt.bz2, ship.train.txt.bz2, ship.test.txt.bz2, wheat.train.txt.bz2, wheat.test.txt.bz2.
[Due 4/2/09] HW#5 Web IR hw5.pdf. Data: search engine query log (large file).
[Due 4/16/09] HW#6 Multilingual Issues hw6.pdf. Data: en.train es.train fr.train en.test es.test fr.test
[Due 5/7/09] HW#7 NLP and IR hw7.pdf.

Course related web-links

Software Resources

Cornell's SMART system
Managing Gigabytes mg system
Lemur: a language modelling IR toolkit
Lucene:popular open-source search engine software
Very nice list of NLP, IR, CL, resources (i.e. parsers, taggers) at Stanford.
University of Michigan tool suite: Clairlib
Trigrams-n-Text (TnT) toolkit, a visible markov model tagger written by Thorsten Brants (now of Google).
QTag a probabilistic POS-tagger.
On-line translators: Systran, FreeTranslation.com
WordNet, a lexical database for English
Andrew McCallum's MALLET toolkit, a Java-based API for machine learning applications using Conditional Random Fields
Wget
Perl LWP library (at CPAN).
Public Domain Speech Recognition Software (from Miss State Univ).

Frivolity

Journal of Machine Learning Gossip: A tongue-in-cheek (yet curiously informative) career guide for young researchers

Cool Demos

A 'meta' search engine: Dogpile
A question-answering system: START
An online joke recommendation system that demonstrates collaborative filtering: JESTER
A faux computer science paper generator, SCIgen, from MIT

IR Test collections

Web Search Engines

JHU Links

The Johns Hopkins University's Center for Language and Speech Processing. During the Spring and Fall semesters CLSP runs a weekly seminar series which meets Tuesday afternoons at 4:30 on the Homewood campus. You can see the schedule here.
Human Language Technology Center of Excellence
The Whiting school's part-time program home page.