No, we cannot do that, because fp16 is not accurate enough, its hard to get good gradients with fp16 (things often round off to zero). Instead, we do the heavy lifting in fp16 - the forward pass and the backward pass. Everywhere else, we convert tensors to fp32 and perform the operations in full precision (calculating the loss, subtracting gradients).