Question 11/13 fast.ai v3 lecture 12
What are the two activation functions used inside the LSTM cell? What happens to the input after it gets multiplied with the weight matrix?
The two activation functions are simgoid and tanh. The input, after multiplication with the weights matrix, gets added to the hidden state from the earlier time step (the hidden state also gets multiplied with a weight matrix first). The result then gets separated into four different pieces which follow different paths (play different roles) inside the cell.
Relevant part of lecture