This book grew out of three series of lectures given at the summer school on "Modular Forms and their Applications" at the Sophus Lie Conference Center in Nordfjordeid in June 2004. The first series treats the classical one-variable theory of elliptic modular forms. The second series presents the theory of Hilbert modular forms in two variables and Hilbert modular surfaces. The third series gives an introduction to Siegel modular forms and discusses a conjecture by Harder. It also contains Harder's original manuscript with the conjecture. Each part treats a number of beautiful applications, and together they form a comprehensive survey for the novice and a useful reference for a broad group of mathematicians.