Question 9/10 fast.ai v3 lecture 8
When initializing weights for multiplying relu activations, do you need to take any additional precautions?
Answer
Yes, in your formula you need to account for the halved variance after relu.
Relevant part of lecture