Course 02407: Stochastic processes Fall 2023

Lecturer and instructor: Professor Bo Friis Nielsen
Instructor: Nikolaj Mathias Primault
Contact: bfni@dtu.dk

Textbook: Mark A. Pinsky and Samuel Karlin An Introduction to Stochastic Modelling - can be bought at Polyteknisk Boghandel, DTU. The bookstore offers a 10% discount off the announced price.
Supplementary text on conditional expectation: Zdzisław Brzeźniak and Tomasz Zastawniak, Basic stochastic processes : a course through exercises

Lectures are held in Building 414, Room 061E Lectures will be held tuesdays between 8.15 and 10 (E3A). Exercises and Problems sessions are held in the same room from 10 to 12.

Course plan:

Lecture 1 - August 29: Markov Chains: Introduction

Reading: Pinsky & Karlin 3.1-3.5
A set of slides recapitulating basic probability here, for printing here.
Slide show (draft) in pdf, printable slides (draft) in pdf
Week 1. Teaching notes (only Markov chain part).
Exercises and problems from Pinsky & Karlin .
In class exercises: KP 3.1.1, KP 3.2.1, KP 3.4.1, KP 3.4.2, (KP 3.1.4), (KP 3.2.4), (KP 3.4.6). KP exercises 3.1 , KP problems 3.1 , KP exercises and problems 3.2 and 3.4 ,
In class problems: KP 3.1.4 , KP 3.2.1 , KP 3.4.4 , (KP 3.2.5) , (KP 3.4.7) . Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 3.1.1 , KP 3.2.3 , KP 3.4.12 , (KP 3.5.2) .

Lecture 2 - September 5: Markov Chains: Random walk, First Step Analysis, Random sums, Branching Processes

Reading: Pinsky & Karlin 3.6 up to 3.6.1 you only need to understand the setup and the results, 3.7, 2.3, 3.9.1-3.9.2, 3.8 can be read superficially. You can omit: 3.6.1,3.6.2,3.6.3, 3.8.1, 3.9.3
Slide show (draft) in pdf, printable slides (draft) in pdf
Week 2.
In class exercises: KP 3.6.1, KP 3.7.1, KP 2.3.3, KP 3.9.2.
In class problems: KP exercises and problems 3.6 , KP exercises and problems 3.7 , KP exercises and problems 3.8 , KP exercises and problems 3.9 , KP 2.3.4, KP 3.6.7 , KP 3.8.3 , (KP 3.9.4) , (KP 3.7.1) . Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 3.6.2 ([derive equations 3.65 only and discuss how you think the non-zero value of r will effect the mean time to absorption]), KP 3.7.4 , KP 3.8.2 , KP 3.9.7 .

Lecture 3 - September 12: Markov Chains: Limiting and invariant distributions, classification of states and chains

Reading: Pinsky & Karlin 4.1,(4.2),4.3,4.4,(4.5)
Slide show (draft) in pdf, printable slides (draft) in pdf
Week 3.
In class exercises: KP 4.1.1, KP 4.3.1, KP 4.4.2, (KP 4.3.2).
In class problems: KP 4.1.1 , KP 4.3.3 , (KP 4.1.13) , (KP 4.4.7 - to be simplified) . Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 4.1.12 , KP 4.3.2 , KP 4.4.3 .

Lecture 4 - September 19: Poisson Processes

Reading: Pinsky & Karlin Chapter 5 - for details see weekly note
Slide show (draft) in pdf, printable slides (draft) in pdf
Week 4 .
In class exercises: KP 5.1.1, KP 5.3.2, KP 5.4.1, KP 5.4.3
In class problems: KP 5.3.7 , KP 5.1.9 , KP 5.2.6 , (KP 5.4.8) , (KP 5.3.9) , (KP 5.6.3) . Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 5.1.7 , KP 5.4.7 , KP 5.5.6 .

Lecture 5 - September 26: Birth and Death Processes

Reading: Pinsky & Karlin 6.1-6.4 The last example of 6.4 can be omitted.
Additional Reading: note in time reversed Markov chains (discrete time)
Slide show (draft) in pdf, printable slides (draft) in pdf
Week 5 .
In class exercises: KP 6.1.2, KP 6.2.2, 6.3.1, KP6.4.4.
In class problems: (KP 6.1.3), KP 6.1.4, KP 6.2.1, KP 6.2.3, (KP 6.2.6), KP 6.3.1, (KP 6.3.2), KP 6.3.3, KP 6.4.2. Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 6.1.2 , KP 6.2.4 , (KP 6.3.4) , KP 6.4.4 .

Lecture 6 - October 3: Finite State Continuous Time Markov Chains, Queueing systems; basic and intermediate queueing theory

Reading: Pinsky & Karlin 6.6, 9.1-9.2,(9.3).
Slide show (draft) in pdf, printable slides (draft) in pdf.
Week 6. Teaching notes.
In class exercises: KP 6.6.2.
In class problems: Exercise 27, (solution), KP 9.2.2 , KP 9.2.6 , KP 9.2.4 , KP 9.3.2 . Zip file with hand written solutions for exercises and problems.
Problems for self study: Exercise 30, (solution) KP 9.2.5 , KP 9.2.7 , (KP 9.3.1) .

Lecture 7 - October 10: Pinsky & Karlin 6.5, Phase-type distributions

Reading: note
Additional Reading: Bladt & Nielsen Section 1.2.6 (discrete case) and Chapter 3,

(Note on Markovian arrival processes).

Slide show(DRAFT!) in pdf, printable slides(DRAFT!) in pdf
Week 7 (draft still to be finalised). Teaching notes.
In class exercises:
In class problems: KP 6.5.2 , Exercise 1 (solution), Exercise 13 Q.1-Q.2 (solution), Exercise 14, (solution). Zip file with hand written solutions for exercises and problems.
Problems for self study: Exercise 2 (solution), Exercise 19. (solution).

Lecture 8 - October 24: Renewal Phenomena

Reading: Pinsky & Karlin 7.1,(7.2),7.3-7.5
Slide show (draft) in pdf, printable slides (draft) in pdf
Short note on moment generating function and Laplace transform .
Week 8. Teaching notes.
In class exercises: KP 7.2.1, KP 7.4.1.
In class problems: KP 7.1.3 , 2.9 from Wolff (solution), 2.19 from Wolff, (solution), 2.33 from Wolff. (solution). Exercise 13 Q.3-Q.4 (solution), 5.22 from Bladt & Nielsen. Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 7.3.5 , KP 7.5.4 .

Lecture 9 - October 31: Markov Decision Processes

Reading: Martin L. Puterman: Markov Decision Processes: 1,1, (1.6), - 2.1 This section is notationally heavy, but we will only focus on the simplest type of model, which has; - in 2.1.1: Finite horizon, - in 2.1.2: finite state and action sets, - in 2.1.4 and 2.1.5: Markovian and deterministic (MD) decision rules and policies - 2.1.6 can be read superficially, but note the connection to Markov chains on the bottom of page 24. - 2.2 (except page 27), 4.1.2, 4.5 (Page 92), 4.6.4.
printable slides (draft) in pdf.
A precorona student relevant example of the secretary problem (youtube video).
In class problems: MDP exercise (solution), Puterman 3.9 - Optimal parking, Puterman 3.26 - The lazy adaptable lion, (Puterman 2.5 (only part a)). Zip file with hand written solutions for exercises and problems.
Problems for self study: Puterman 3.10 & 3.11 Variants of secretary problem

Lecture 10 - November 7: Brownian Motion

Reading: Pinsky & Karlin 8.1-8.2
Slide show (draft) in pdf, printable slides (draft) in pdf

Small movie with Brownian motion

Week 10. Teaching notes.
In class exercises: KP 8.1.1, KP 8.2.1, KP 8.1.4, (KP 8.1.6), ((KP 8.1.5)).
In class problems: KP 8.1.1 , KP 8.1.2 , KP 8.1.7 , KP 8.2.1 , KP 8.2.4 , Zip file with hand written solutions for exercises and problems.
Problems for self study: Surplus exercises and problems from "In class".

Lecture 11 - November 14: Brownian Motion with Drift and Variations and Extensions

Reading: Pinsky & Karlin 8.3-8.4.1 (to Page 424)
Slide show (draft) in pdf, printable slides (draft) in pdf
Week 11. Teaching notes.
In class exercises:
In class problems: KP 8.3.3 , (KP 8.3.6) , KP 8.4.1 , KP 8.4.2 , (KP 8.4.5) . Zip file with hand written solutions for exercises and problems.
Problems for self study: Brownian motion . KP 8.3.1 , KP 8.4.4 there is a typo in this exercise you should replace 1/2 with t/2, KP 8.4.10 .

Lecture 12 - November 21: Black-Scholes Option Pricing Formula and Ohrnstein-Uhlenbeck Process

Reading: Pinsky & Karlin 8.4.1 from P.424 (Black Scholes), 8.5 (we might not make it to 8.5.4)
Short note on stationary Ornstein-Uhlenbeck process .

Week 12 (draft only)

Teaching notes

In class exercises: KP 8.4.5 , KP 8.5.1 , KP 8.5.2 , KP 8.5.3 ,
In class problems: ( KP 8.4.6 ). KP 8.4.7 , KP 8.4.8 , KP 8.4.10 , KP 8.5.2 , KP 8.5.3 . Zip file with hand written solutions for exercises and problems.
Problems for self study: KP 8.5.1 Brownian motion . KP 8.3.1 , KP 8.4.4 , KP 8.4.10 .

Lecture 13 - November 28: General course overview

Zip file with selected exercises. The style of these exercises are somewhat similar to the type of questions you can expect in the take home exam.. Zip file with selected solutions..

Deadline for reports on the take home exam is Saturday December 23^nd 2023 3 pm. The exam will be available December 15 8 am in DTU Learn.

The report has to made individually or in groups of two. It is not allowed to discuss the exercise with other students or third parties. Supplementary questions can be posted on DTU Learn in relation to the message with the announcement of the exam. If you hand in as a group there will be a short supplementary oral examination in January. If it is impossible for you to be physically present at DTU, then the oral exam can be held online. The report has to be handed in electronically. If you submit a scanned version of a handwritten report I will need the original not later than January 10th. The original can be sent using ordinary mail.

Last change: 29/11 2023, by bfn