Reinforcement learning has been the standard paradigm for modeling sensorimotor adaptation and classical conditioning in animal studies. However, I will argue that traditional reinforcement learning is incapable of explaining two classes of experiments. First, in sensorimotor learning, the motor correction is often dependent on the error nonlinearly (a specific example is that of a songbird compensating for an experimental perturbation to the pitch of the produced song). Second, extinguishing of associations sometimes shows oscillatory dynamics not predicted by reinforcement models (here a specific example is the food-temperature association in a roundworm). I will show that extensions of traditional models to probabilistic Bayesian filtering and to multi-objective reinforcement (both with multiple time scales) account for these diverse experimental data. Further, I will discuss how such models of learning are implemented in real animals.