Place a saucepan over medium heat. Add the cocoa powder and ¼ cup of the milk and whisk until the cocoa powder is mostly integrated. Whisk in the remaining milk until smooth.
Add the remaining ingredients and heat over medium heat until the chocolate is melted, about 2 minutes. Do not boil.
Stir in the Baileys. Serve immediately. Marshmallows are optional.