Market consistent forecasts of financial drivers require our clients in the insurance and banking industry to perform repeated calibrations of financial models. Among the financial models at stake, those dedicated to interest rates have reached a major complexity within the market practice compared to those dedicated to other financial drivers. For interest rate stochastic modelling, the LIBOR Market Model with Stochastic Volatility (SVLMM) is now widely used as it has proven its ability to reproduce volatility smile and market prices in a satisfactory way. Additionally, in a very low interest rate regime, the use of a displacement coefficient allowing to forecast interest rates in the negative region is becoming a market standard, which takes the form of the so-called Displaced Diffusion SVLMM. As it is now crucial to get fast calibration procedures for such a model, Milliman developed an advanced parameter inference strategy allowing to set such interest rate models in a significantly faster way compared to the classical methods available today. This paper details the related mathematical and numerical toolkit.