You would need to purchase on-premise Windows Server licenses with Software Assurance (SA), for the whole duration of the reserved VM unless you are using Linux which I believe doesn’t require purchasing (may need to check). You should be able to purchase Windows Server Standard Core licenses from Techsoup since they do come with free two years Software Assurance.
You can save a lot of money moving to reserved instances but do note that you would need to pay upfront which might bypass the Azure sponsorship credits you enjoy annually. You should have USD$3,500 Azure credits granted annually. You would need to do some calculations to see which path cost less.
Once you have the Windows Server licenses with SA and purchased upfront the reserved instances, you will need to switch your existing VMs to use the Azure Hybrid Benefit by changing an option in your portal; if not you will still be made to pay for you Windows Server licensing through Azure.
FYI that the Azure Reserved Instance once purchased will automatically look for the exact instance in your subscription to discount. Make sure they are in the same subscription, VM size and region.