You Don't Need Matrix Calculus

Matrix calculus is unnecessary for differentiating functions of vectors or matrices; writing out index-level expressions and applying ordinary differentiation suffices. For the ReLU-style activation \(\mathsf{activation}(x)=\max(0,Ax+b)\), the partial derivatives are \(\partial \mathsf{activation}_i/\partial A_{kl}=x_l\), \(\partial \mathsf{activation}_i/\partial x_k=A_{ik}\), and \(\partial \mathsf{activation}_i/\partial b_k=1\) when the pre-activation is non-negative, otherwise zero.