Advanced Parallel Computing for Scientific Applications

With the growing complexity of computer simulations and the availability of multi-core processors, parallel computing becomes important to all fields of science and technology. This course covers parallel high-performance computing on all levels: from the basics to high-level parallelism and grid computing. It is a hands-on course with practical programming exercises.

Contents
Parallel programming paradigms, vectorization, shared-memory and multi-core programming, OpenMP, multi-threading, the Message Passing Interface (MPI), non-determinism in parallel programs, parallel debugging, domain decomposition schemes, communication scheduling methods, parallel linear algebra and parallel solvers, data structures and abstractions, parallel algorithms and libraries, grid computing, resource allocation models.

Learning goals
- Choosing the proper programming paradigm for an application
- Shared memory implementation using OpenMP
- Distributed memory implementation using MPI
- Knowing parallel algorithms, data structures, and numerical solvers
- Implementing loosely coupled applications on a grid of workstations

Lecture language: ENGLISH

Time table and study material
The lecture takes place Tuesdays, 09-12h, ML F 38. (Start: Sep 28, 2010)
The exercises take place Tuesdays 13-15h, HG E 21. (Start: Sep 28, 2010)

Please notice that there is NO lecture and NO exercise on September 21.

Please find below the lecture syllabus. Teaching is shared between Prof. Ivo Sbalzarini (IS) and Prof. Peter Arbenz (PA). Lecture slides and teaching material will be posted here during the semester:

week 1, 28.9.2010 (IS): admin/intro; parallel programming paradigms; performance evaluation; definitions; vectorization and pipelining; loop unrolling

Course material: 1 slide per page, 3 slides per page
Self-test questions

week 2, 5.10.2010 (IS): MPI 1: message passing basics; point-to-point communications; collective communications

Course material: 1 slide per page, 3 slides per page
Self-test questions

week 3, 12.10.2010 (PA): shared memory programming model; threads vs. processes; generation/administration/coordination of threads

Course material: 1 slide per page, 4 slides per page
Self-test questions

week 4, 19.10.2010 (PA): basic OpenMP, loop vectorization on multi-cores, the fork-join principle of OpenMP; Variations on the parallel construct; shared vs. private variables; beyond loop-parallelism

Course material: 1 slide per page, 4 slides per page
Self-test questions

week 5, 26.10.2010 (PA): Hybrid multi-threading/multi-processing by combining OpenMP with MPI

Course material: 1 slide per page, 4 slides per page (updated!)
Self-test questions

week 6, 2.11.2010 (PA): parallel matrix-vector product

Course material: 1 slide per page, 4 slides per page
Self-test questions

week 7, 9.11.2010: guided tour to the HPC center of ETH

week 8, 16.11.2010 (IS): MPI 2: derived data types; data packing/unpacking; process groups; communicators; MPI topologies; non-determinism; error handling; parallel debugging.

Course material: 1 slide per page, 3 slides per page
Self-test questions

week 9, 23.11.2010 (IS): domain decomposition; load balancing; communication scheduling; graph partitioning algorithms

Course material: 1 slide per page, 3 slides per page
Self-test questions

week 10, 30.11.2010 (PA): parallel iterative solvers; PCG/GMRES; Newton iteration

Course material: 1 slide per page, 4 slides per page

week 11, 7.12.2010 (IS): parallel data and operator abstractions; parallel algorithms 1: sorting

Course material: 1 slide per page, 3 slides per page
Self-test questions

week 12, 14.12.2010 (IS): parallel algorithms 2: tree search, distributed termination detection, fast N-body solvers; overview of parallel libraries (ScaLAPACK, PetSc, POOMA, PPM, ALPS)

Course material: 1 slide per page, 3 slides per page
Self-test questions

week 13, 21.12.2010 (IS): grid computing; performance indicators; resource allocation; the Gamma model

Course material: 1 slide per page, 3 slides per page
Self-test questions

 

Please find below exercise sheets and material needed for or shown in the exercises:

week 1: Exercise 01, Brutus guide, Terminal guide, Tutorial, Solution 01

week 2: Exercise 02, Tutorial, Solution 02

week 3: Exercise 03, Source 03, Tutorial, Solution 03

week 4: Exercise 04, Source 04, Tutorial, BLAS quick reference, OpenMP API , Solution 04

week 5: Exercise 05, Source 05, Tutorial 05, Solution 05

week 6: Exercise 06, Source 06, Tutorial 06/07, Solution 06

week 7: Exercise 07, Source 07, Solution 07 (pdf), Solution 07 (src)

week 9: Exercise 09, Tutorial 09, Solution 09

week 10: Exercise10, Source 10, Tutorial 10, Solution 10

week 11/12: Exercise 11/12, Tutorial 11 (updated), Solution 11 (pdf),, Solution 11 (src)

week 13 (opt): Exercise 13, Tutorial 13, Solution 13

Teaching assistance
For questions regarding the exercises please contact Janick Cardinale.