Information Theory

Dr. Cong Ling
Spring Term

Syllabus

This course will set out the fundamental concepts of information theory. Expressions for the information generated by discrete memoryless sources and sources with memory will be established and the loseless source coding theorem will be proved and the asymptotic equipartition theorem will be presented. The practical significance of the source coding theorem will be examined and examples of source coding will be given. The concept of channel capacity will be introduced and the calculation of the capacity of important communication channels and systems will be dealt with. The capacity theorem will be proved for various cases and its practical significance will be examined, and simple examples of coding aimed at achieving the results promised by the capacity theorem will be outlined. The concept of source coding, subject to fidelity criteria, (rate distortion theory) will be introduced. Finally, the basics of network information theory will be given, which is presently the most dynamic research area in information theory.

Textbooks

Course textbook: Elements of Information Theory, T. M. Cover and J. A. Thomas, Wiley.

Reference: Information Theory and Network Coding, Raymond W. Yeung, Springer, 2008. Free at http://iest2.ie.cuhk.edu.hk/~whyeung/book2/

Reference: Lecture Notes on Network Information Theory, A. E. Gamal and Y.-H. Kim. http://arxiv.org/abs/1001.3404

Reference: Information Theory, Inference, and Learning Algorithms, D MacKay, CUP, or free at http://www.inference.phy.cam.ac.uk/mackay/itila/

Shannon’s original paper: “A mathematical theory of communication,” Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656, July, October, 1948.

Lecture Slides

Slides

Problem Sheets

Problems

Past Exam Papers

2006, 2007, 2008, 2009, 2010, 2011

Frequently Asked Questions

Is topic "xyz" examinable? All topics included in the above lectures are examinable with the exception of the Kuhn Tucker conditions for constrained optimization.

Do I need to remember long complicated proofs? You will not be asked to reproduce long proofs from the notes. However you will be expected to understand the proofs and to justify the steps they take.

Do I need to remember long formulas? In general you are not expected to memorize long complicated formulas. However you will be expected to remember some basic and important formulas.