Recently I did a blog post on the Azure VM size limit and what disk performance you get. Picking the right VM in Azure is important when you use Azure Storage SSD disks, as machines are limited in throughput. With a D64s_V3 and ultra SSD disk I get the 81544 IOPS that’s good but costly the VM is $5K then the disk will cost you also some $$.
This post is not about costs or good or bad but it will show you that picking random resources will cost you more that a selective menu and get even better performance ad a lower cost.
This blog post is just as a reference, measurements in your config may be different, read the blog comments.
below is my biggest IOPS number I have seen in Azure with a Diskspd test and this is not bad at all and the cost are not even worse that the D64 this is done with a H16r cost $1,472 per month
###########################################
Good comments are made below in the blog comment the measurements could point to caching. Also always build a solution that is supported by the vendor.
##################################
Getting massive IOPS* in a VM is still not plug and play. (with Caching)
In my former blog post I tested some VM to get the maximum speed and selecting the right Azure VM can save some costs
When using VM’s with disk attached then the storage throughput is important, in a Azure VM cluster with Storage spaces direct the network is also important. Now with SMB storage the VM well it’s a VM but the setup of this VM is more important, what is the Network bandwidth and can it do #RDMA ? Basically a 1Gbps can do 95 Mb/s that’s not bad but we want to do more right.
But what if we use other storage will this be the same or different ? to give a good answer on this I’m doing the same test as I did in the blog post
here I got 80.000 IOPS not bad but at what costs. Storage is cheap in Azure but Performance cost a lot. and then the latency of the disks. It is all part of your solution.
In this case I’m using the new Azure NetApp files It’s NFS or SMB and the pricing is different than the Azure disks. Cheaper or more Expensive it al depends on your solution and performance what you need. but your solution may need a different setup, as this is a SMB solution and not a direct attached disk.
A comparison between the Azure Files and the Azure Netapp Files and Azure Disks
Big numbers but it comes with a BUT for 320K IOPS on a 500TB pool times the $0.39 per GB that’s also a big cost So handle with care. but it’s a lot off IOPS.
Azure NetApp Files Comes in 3 options Standard, Premium and Ultra Seems fast.
Well lets test this and see where the difference is between premium and ultra. We add the AzureNetapp files to Our Azure subscription.
For starters the subscription need to be white listed before we can use this. We create NetApp Account and then we create some pools and Disks
Now we have a virtual NetApp Storage device. We can create pools and volumes to work with.
For SMB Connections we need to join the Active Directory first. As we need to create a Fileserver name similar like a SOFS server. This is created in your own Active directory and not in the Azure AD. Fileserver Computer Accounts will be created in your AD
Whit these settings we have a similar configuration as the on premise config with RDMA Hyper-v or S2D or Azurestack HCI with these thoughts we can get big performance goals.
A user account that can create computer objects in the OU. The LDAP structure is without the DC= part and My OU is in the Root so a short syntax.
Here you can see the just created link to the Domain.
Our next step is creating a Capacity pool These can be Large but remember pay per use!
Remember you will pay for the provisioned capacity, and here is the catch the larger the Pool the more IOPS you get. So if you have a tiny application that need performance you need to see what is the right choice.
Now that the Pool is created we can create a volume in the pool, all the basics are similar to the Microsoft Storage Pools.
Create Volume off 100GB in the 4 TB pool. the Costs are 4TB and not the 100GB.
I create a new Storage network, this is also a must a dedicated storage network for the NetApp files. If you are familiar with Storage spaces direct and multichannel SMB then you have a new playground, but less building.
Network is done File server is created and share is created, at this time no ACL’s are available on the share.
When the setup is done we see a File server in the AD structure. Netapp is using only the given name but will add a –XXXX to make the name unique. that makes sense as the deployment will fail on this.
My first test on the 100GB share. Testing this with a Azure VM D2s_v3 Capable of delivering 4000 IOPS on the Disks
That is impressive almost 29.000 IOPS on a simple VM. well in my volumes I have just 4TB so 4x 4000 IOPS not bad
So a bigger machine is not needed than, well now the VM selection should not be on the Disk size but on the network throughput and more network adapters is needed the max out the storage as the D machine is having a 1Gbps nic this is to low for this.
Impressed by the low latency numbers and how easy you got the performance.
Hitting the Network adapter limit 
29K IOPS on a cheap Azure D2 VM that’s not bad at all.
Good performance but I want more lets create some bigger pools and better network machines. Thinking about SMB direct or some big pool to hit real hard the storage to get big numbers. Like the demo’s of Jeff Woolsey @WSV_GUY
Looking into the resource group, here you can see the created resources and two NIC’s that’s resource I created a premium and an Ultra pool.
I can Imagine that this nic will be a throttle, in a big Pool hitting all the data to one NIC.
Let me start a H8 Azure VM this is a new VM better network performance. Should be good. pricing below $1000 monthly
Azure NetApp Files with a Premium 16TB disk
Low Latency good performance, with Azure disk I needed more disks and a bigger VM to get these results.
Same test with Ultra 16T Storage pool
No big changes between the Premium and Ultra and the cost are $0.10 more per GB
So a new test this time with a H16R nic with RDMA with this we can optimize the network performance a bit
https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-hpc#h-series
Well that’s good eh better than good big performance, but I must say this is not the default H16r machine I tweaked it a bit and used the same technique as when you build a Azurestack HCI or Hyper-v over SMB*
the measure results could be partly cached, for facts I should do multiple runs, but all the tests are done only once, as these setups are expensive and testing takes time, I have no benefit on any result good or bad it is just my setup and opinion,
I was to late to grab a decent CPU and nic performance screenshot.
I was planning to the same results on a A9 VM on my Windows Virtual desktop with the FSLogix profiles (user profile disks) but Running these is a A machine compaired a H or N series an NC24r or a ND40 or 24 cost some $ but huge performance. This is something for a next blog post.
But initial test where not better and the A9 VM is more expensive, and gives me thoughts on different options. what and how Can I extend the Azure networks to get the ultimate sweet spot cost vs performance.
I think this is a real good replacement for Azure S2D, its cheaper and faster, Suppose you run a RDS site with lots of User profile disks or Windows virtual desktop with FsLogix, build profiles high available will take you to a storage space direct cluster at least two nodes and 6 Disk. Replacing this with Azure NetApp files could save you some $$ and sure lots of options are in the cloud there and yes you may need a extra network, but if you can offload your network with some extra network adapters and have a storage network to boost the performance, then you have a great solution in the Cloud.
Read the comments below, and always test your setup to see the results before you go into production. Also read the requirements so that your config is supported.
I’m saving my Azure credits to do a massive S2D run with the Azure Ultra disks. Keep in mind all my configs are different and nothing is next next finish and my be not supported.
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Share this:
Robert Smit MVP Blog 
Hi Robert, would you mind stating what configuration change you used to get a single Azure VM (H8) to 4700MiB/s on reads. I’d like to reproduce what you’ve done here.
Hi Robert,
My name is Chad Morgenstern and I am the lead performance solutions architect for Azure NetApp files. Your blog was referred to me by a few colleagues and customers. Your testing methodology is not quite clear in this paper and as such I have some questions. Would you be willing to engage in a call with me to go over ANF, testing methodology, and results.
Chad
HI Chad,
The config is real simple multiple nic’s , network config and the netapp files. Test tool is Diskspd. as shown on the blog and screenshots.
Hi Robert,
I asked because some things do not add up for me:
1) ANF does not support SMB Direct (ONTAP Does not have support for this technology).
2) At present, there is no support for SMB multichannel in Azure NetApp Files though we are always considering modifications and this could change on a moment’s notice.
3) Although the VM’s you used do have support for RSS (and in the case of the H16r RDMA), without support for the same within ANF multiple NICS are really of no use.
4) Further, a single Azure NetApp Files Volume is capable of ~450,000 IOPS.
With the above said, take a look again at the results you generated. The 45,000 to 50,000 IOPS you saw with other VM types were rate limited primarily by the affinitizing of SMB to one CORE on the Virtual Machine. If you look at Perfmon’s Processor stats in the previous cases you will see that the idle time is hovering at 0 for this one core.
I’d like to point out however that the underlying ANF volume has plenty of bandwidth left, had you stood up additional VM’s and driven workload therein against the same volume you would not notice much if any impact of the workload of the other VM’s to this volume.
Further, you will notice that the previous workloads saw ~2ms of latency at around 50,000 IOPS? I point this out as when you ran the H16r instance (if I am reading your blog correctly) you observed latency of 0.09ms for ~1.2Million IOPS. What is very likely occurring is cached reads on the VM.
At NetApp we are all about setting proper expectations, while I am very happy that your testing generated such outstanding results with ANF, I do want to set the expectations properly for our fans, analysts and customers.
Chad
P.S.
I’d love to show you around the ANF environment, please hit me up on email if you are interested.
Hi Chad, absolutely the blog is not about tricking into something that is not true, I need to do some retesting in a setup, as testing these config are costly, all the screenshots are real and not changed in any kind. caching could be a valid point as I did not pay attention to that, the only point in this blog post was how to get as much iops as you can. and VM does mater in that case. Also it was tested in a preview situation.
As vm I did test with the A and H series with the RDMA, supported or not customers build strange configs in Azure.
I’ll add some extra comments to the blog post and reference to the comments section
Interesting comparison. Firstly, we would love for you to add reviews to our site (IT Central Station) for these products, so that it can help others when they are researching their options. It might also be interesting for people to look at a review-based comparison of Azure NetApp Files to get a feel for the relative pros and cons based on user’s experiences and specific use cases: https://www.itcentralstation.com/products/comparisons/azure-netapp-files_vs_microsoft-storage-spaces-direct/tzd/c1122-sbc-39
Hi Rony, For the product choice it depending on the usage. With pros and cons it often look that one is better than the other but I’m a strong believer that there is no one size fits all. Workload is key. and issues are shifting when using a different product as both “storage” solutions are different and need different care.