Archive for the ‘Azure’ Tag

Azure VM vs Disk vs Costs, Does Size matter ? or a Higher price for better specifications #Azure #Storage #Performance   Leave a comment

Building in Azure is easy and the wizard takes you to all the steps and you have a working VM. choosing the right size is different often it has a link to the on premise world 4 core CPU and 8 GB memory. and the disk I need 1 TB disk space. All simple but then things get complicated the performance needs to be better CPU is fine Memory well 60 %  plenty of disk space. Bigger VM perfect.

Still slow Whole VM runs at 20-60% users are complaining must be this Azure thing someone else his computer runs slow.

I often hear this. But is it really slow or is your measurement wrong ?

When you pick a machine on premise what do you take performance or Cost ? <> performance and then cost right and at the end you settle with the cost vs. performance.

But in Azure what do you take performance or Cost ?<> 100% Costs, VM’s are expensive.  This is not always wrong but sometimes is paying a bit more the best approach

In my sample here I show you the performance in a Disk with different machine types, and not picking the right components doesn’t give you the right performance. but it may well function on your workload, but then you may pay to much for you over sized configuration.

In my sample I need a VM with 300 IOPS and one with 4000 IOPS and I need one with 27000 IOPS CPU and Memory are in this case not important as it is more i/o intensive.

I pick a default Azure VM an D machine, put some disks to the machine one HDD-S30 ,SSD-E30 ,SSD-P30,SSD-P60 

 

VM Type Disk Type MiB/s I/O per s
Standard D2s v3 (2 vcpus, 8 GiB memory) HDD-S30 2.01 514.23
  SSD-E30 2.21 566.27
  SSD-P30 13.29 3403.51
  SSD-P60 12.33 3157.46

 

First goal met 500 IOPS and an cheap machine but this could also an Azure B type VM much cheaper. then I wonder why use SSD over HDD for the IOPS it’s the same speed and latency there is a point SDD are performance steady, but for normal workload. Costs If you have a lot of transactions then SDD may be cheaper. A fact is nobody knows how expensive the HDD disk are, have you ever calculated the Storage transactions ?

image

below is a overview of the disk latency.

25th |    100.325 |    N/A |    100.325 HDD-S30

25th |    100.012 |   N/A |    100.012 SSD-E30

25th |      4.545 |    N/A |      4.545   SSD-P30

Comparing all the SSD disks and pick the right performance is not hard Microsoft did a great job on explaining this. on Microsoft docs

Disk size

Premium SSD sizes P30 P40 P50 P60 P70 P80
Disk size in GiB           1,024 2,048 4,096 8,192 16,384 32,767
IOPS per disk           Up to 5,000 Up to 7,500 Up to 7,500 Up to 16,000 Up to 18,000 Up to 20,000
Throughput per disk           Up to 200 MiB/sec Up to 250 MiB/sec Up to 250 MiB/sec Up to 500 MiB/sec Up to 750 MiB/sec Up to 900 MiB/sec

When you provision a premium storage disk, unlike standard storage, you are guaranteed the capacity, IOPS, and throughput of that 

 

When you provision a premium storage disk, unlike standard storage, you are guaranteed the capacity, IOPS, and throughput of that

that is interesting In my D2 machine and with a P30 I got only 3400 IOPS, so this is wrong ? Well according to the disk but the VM can only deliver 3200 IOPS with the 3400 IOPS delivered its perfectly normal then.

image

 

The same test again with a better Azure VM and the same disks.

 

VM Type Disk Type MiB/s I/O per s
Standard DS3 v3 (4 vcpus, 14 GiB memory) HDD-S30 2.01 514.01
  SSD-E30 2.21 566.63
  SSD-P30 21.58 5523.51
  SSD-P60 51.00 13056.39

 

The requirements are there 5500 Iops for a disk that need to deliver 5000 IOPS that’s good. but what about the P60 disk , again a had cap to the VM max of 12800 IOPS

The latency is not that different for this you need a different kind of VM

25th |    100.256 |        N/A |    100.256  HDD-S30

25th |    100.008 |        N/A |    100.008 SSD-E30

25th |      4.416 |        N/A |      4.416 SSD-P30

25th |      2.135 |        N/A |      2.135  SSD-P60

Comparing the Azure VM’s selected on IOPS and select the right machine

imageimage

 

selecting the F4 VM that can deliver 16000 lops according the sheet .

VM Type Disk Type MiB/s I/O per s
Standard F4s (4 vcpus, 8 GiB memory) HDD-S30 2.01 514.01
  SSD-E30 2.21 566.63
  SSD-P30 21.58 5523.51
  SSD-P60 50.85 13018.46

 

Did not get the 16.000 lops in fact it produce almost the same results ad the DS3 only double the costs.

SSD-P60 latency measurement 4k blocks vs 64K blocks

25th |      2.171 |        N/A |      2.171

25th |      3.088 |        N/A |      3.088  <> 64kblocs

So this strange big machine still not hitting the limits CPU and memory is low. Seems good but not the performance

image

image

Checking the Microsoft site : https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-compute

You can see a different specs result. this means the machine can’t deliver the IOPS and the Size table thinks he can. Results are bad.

Standard_F4s_v2 4 8 32 8 8000 / 63 (64) 6400 / 95 2 / 1750

 

Then lets pick a Azure VM than can deliver the iops. a F16 big VM costly but can it deliver I compare both tables In the Azure portal and the Docs

  But on the other side on the Docs https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-compute

Standard_F16s_v2 16 32 128 32 32000 / 255 (256) 25600 / 380 4 / 7000

 

VM Type Disk Type MiB/s I/O per s
Standard F16s v2 (16 vcpus, 32 GiB memory) HDD-S30 2.01 514.09
  SSD-E30 2.21 566.63
  SSD-P30 21.60 5529.96
  SSD-P60 63.76 16321.29

 

This looks OK now 16000 IOPS.

But what If I build a stripe set from the SSD-P30 and SSD-P60 and HDD-S30 and SSD-E30 what would be the iops ? (it’s a bad idea to mix different disk types this is just a sample)

What if we create a stripe set ?

image

Worse performance than if I user the SSD-P60 alone. Bad config to do this. 

 

HDD and SSD

image

Both Disks have around 500 IOPS each and now they can produce a 1000 IOPS that’s not bad

But what happens if I combine all the disks into a Storage space direct ? combining all the disk you have and build a new disk JBOD.

image

Also a Bad Idea and a waste of resources and Money an P60 disk combined with a S30

That’s all about the little side step, but it keeps me thinking…. -What if

Below is a list with similar iops performance  And Instead of using 1 SSD-P60 I’ll use 3 disks on paper I should have 3x 16000 IOPS = 48000 IOPS and 3x 500MB/s =1500 MB/s that is massive right. stripe set or Storage space or storage space direct ? all valid options but what machine do I need to handle the performance.

image

I selected 3 types a E32,DS5 and a DS14 all with big price difference but similar specs .

Standard_E32s_v3 2 32 256 512 32 64000 / 512 (800) 51200 / 768 8 / 16000
Standard_DS5_v2 16 56 112 64 64000 / 512 (688) 51200 / 768 8 / 12000
Standard_DS14_v2 3 16 112 224 64 64000 / 512 (576) 51200 / 768 8 / 12000

 

First I build a Storage Pool on the DS5_V2

image

Nice Capacity good latency and decent performance a round 29000 IOPS of 3 disks, in a Mirror set I’ll loose a disk so the performance is good better than I expected.  To hit the limits I should add 2 more disks to this config and see if they can handle the performance.

25th |      2.025 |        N/A |      2.025

image

I’ll run the same test on a E32-8s_v3

Bigger VM much more performance, higher price.

image

So overall the cheaper VM can produce the same disk performance. but the machine is $1000 cheaper per month. Again it depends what you are doing with the VM

Now the same configuration with Storagespaces Direct just to see if the performance is better, keep in mind that every run the machine performance can be a bit different so in the same range I see this as the same performance.

The S2D results on a E32 VM

image

And even a step higher an expensive VM with 432 GB memory. With an S2D Cluster.

 

image

So same performance when Running a StorageSpace or S2D cluster and no change on the machine type. in fact the DS5 machine is slightly better. it saves $2000 per month. If you don’t need the CPU and memory from the VM.

image

image

So size does matter but it depends on what size you are looking right. Azure is like Lego but different. Combining the pieces makes a great solution.

Below I created a table Cost vs performance, I also compared the datasheet in the azure portal to the DOC pages and I think you should keep this page as a reference. https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-memory

image

This shows you that in complex configurations there is no one size fits all and it comes to testing and adjusting, Tools may help you but picking the right VM size and choose the right storage can take some time. As in this I only compared disks but what if I choose Netapp files or some other disks like ultra SSD’s

And Now I did this config with 3 P60 disk that cost  $1000 each = $3,121.92 (in azure Calculator) it gets me ~30.000 IOPS

Now On the DS5 machine a 2 way mirror Auto created.

image

It nags me that I can’t get the max from the VM, the must be something wrong in my configuration. lets do some quick testing change VM and Disk types

With 6 times a SSD-P30 disk  I’ll get 27.000 IOPS on the DS5 Machine

image

When using a Stripeset this hits the VM limit of 768 true put. Less IOPS but more speed. So Configuration is also KEY in the used hardware.

image

Lets tweak the config a bit and see if we can pass the 50.000 Iops and hit the machine limit.

image

With read cache enabled and 8 P30 disks. that’s not bad right.

image

The P40 disks have 7500 IOPS each will this break the record ?  (6x P40 disk storage space)

image

First test same result a bit lower, but there is more to get. Testing now With 8 P40 disks

(8x P40 disk storage space)

image

(8x P40 disk storage space) Manual configuration.

image

(8x P40 disk storage space) Manual configuration. with 6 columns

image

That’s not bad the DS5 hits the limit.

On Microsoft Ignite 2015 Mark Russinovich did a demo, where he showed a virtual machine with Premium Storage that hit over 64,000 IOPS. Well This beats the record but the Azure hardware is much better now right.

Lets Switch to some big Azure VM

image

64 Cores lets see If I can use some of these cores in the S2D config.

image

image

Oh ok it seems I need more cores or less workload on this.  But easily hit the IOPS limit on this machine.

image

image

 

Overall in this is what do you need and test this also with a different configuration. Not only on price but also on performance.  In the first section I used 3x a P60 disk cost $3.000 a even better result I get with 8x P30 disk cost $1.000

Picking the right configuration can only be don based on testing and create some references for you. Azure machines and storage is changing all the time its getting better all the time. It all depends on your workload but there is no one size fits all !

 

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted July 9, 2019 by Robert Smit [MVP] in Azure

Tagged with , ,

How to start with Microsoft #Azure #Bastion Service, secure VM access #AzureBastion #jumpserver #PaaS #WAC   Leave a comment

In case you may missed this Azure has released a new service called Bastion. So what is the fuzz about this new service and why should you use this ?

Bastion can Manage RDP/SSH to VMs over SSL using private IP on the VM.

Azure Bastion is a fully managed PaaS service that provides secure and seamless RDP and SSH access to your virtual machines directly through the Azure Portal. Azure Bastion is provisioned directly in your Virtual Network (VNet) and supports all VMs in your Virtual Network (VNet) using SSL without any exposure through public IP addresses.

image

 

So basically it is the old Jump server that you already used to get into the Azure VM’s if needed. It can access all virtual machines within a virtual network through a single hardened access point. Exposing the bastion host as primary exposed public access helps lockdown of public Internet exposure and limit threats such as port scanning and other types of malware targeting your VMs.

A jump server as PaaS services.

image

This seems nice but as always is it free or is it costly ? Well in the Azure Calculator you can see the Costs.

https://azure.microsoft.com/en-us/pricing/details/azure-bastion/

image

 

Ho do we start with Bastion.

 

First we need to register the new resource in Azure this is always needed to get to work with the new Azure components.

Keep in mind this can take some time to register

Get-AzProviderFeature -ProviderNamespace Microsoft.Network

image

With the Powershell command below we are registering the Bastion service into our subscription and network.

Register-AzProviderFeature -FeatureName AllowBastionHost -ProviderNamespace Microsoft.Network

image

Now that we triggered to register the Bastion services we need to wait

Check if it is done

image

 

image

Then register the network again. with your subscription and the Microsoft.Network provider namespace

Register-AzResourceProvider -ProviderNamespace Microsoft.Network

image

 

Now that this is done we can start with the Configuration, and there a multiple ways on how to get there. by the market place or directly in the VM

image

In the VM almost all the items are pre defined and ready to go if you want to go with the defaults.

image

 

In the marketplace you need to find the bastion and select the new resource.

imageimage

Select and create the resource. Configure this accordantly and select the proper network.

image

The starting point is almost the same the first one is already in the VM network and the one from the market place is just a blank one , where you need to select your network.

In this LAB I’ll go for connection directly from the VM.

Lets start in the VM go to connect and select bastion and use Bastion

image

As I want to move forward quickly I already see some red lines. I need a /27 Subnet.  This is currently not in my network so I need to create a new subnet in the used Azure network.

image

As shown below the extra subnet is created to connect to the AzureBastion

image

 

The subnet inside your virtual network to which Bastion resource will be deployed. The subnet must be created with the name AzureBastionSubnet. This lets Azure know which subnet to deploy the Bastion resource to. This is different than a Gateway subnet. Click Manage subnet configuration to create the Azure Bastion Subnet. We highly recommend that you use at least a /27 or larger subnet (/27, /26, etc.). Create the AzureBastionSubnet without any Network Security Groups, route tables, or delegations. Click Create to create the subnet, then proceed with the next settings.

image

image

Now that the Subnet is added we can creating the Bastion service.

image

The validation started a it is created.

image

Now that it is created we can connect to the VM with HTML5 the connection is similar with WVD RDP connection to the VM.

image

You can see the created subnet.

image

Connecting With chrome or with Microsoft Edge is no problem you do need to configure the popup blocker

image

 

Web based RDP connection keep in mind the background is filtered out.

For connection with the browser you will need to allow the popup showing

 

image image image

now that the portal has access the connection will proceed. Unless your VM is in the Wrong region

image

Currently only the following regions are supported :

  • West US
  • East US
  • West Europe
  • South Central US
  • Australia East
  • Japan East
 

This is a nice feature but if you running already a hybrid site why not using the Windows admin center here you can also connect with the HTML5 browser to the Azure VM. the only thing here is you will need to connect to an external IP with proper NSG or to the internal IP with a S2S VPN connection.

image

 

https://azure.microsoft.com/en-us/services/azure-bastion/

 

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted June 20, 2019 by Robert Smit [MVP] in Azure

Tagged with

Windows server 2019 Upgrade virtual machine version in Hyper-V #hyperv #winserv #hybrid   Leave a comment

Why should I upgrade the virtual machine configuration version?

image

When you move or import a virtual machine to a computer that runs Hyper-V on Windows Server 2019, Windows Server 2016, or Windows 10, the virtual machine"s configuration isn’t automatically updated. This means that you can move the virtual machine back to a Hyper-V host that runs a previous version of Windows or Windows Server. But, this also means that you can’t use some of the new virtual machine features until you manually update the configuration version. You can’t downgrade the virtual machine configuration version after you’ve upgraded it.

The virtual machine configuration version represents the compatibility of the virtual machine’s configuration, saved state, and snapshot files with the version of Hyper-V. When you update the configuration version, you change the file structure that is used to store the virtual machines configuration and the checkpoint files. You also update the configuration version to the latest version supported by that Hyper-V host. Upgraded virtual machines use a new configuration file format, which is designed to increase the efficiency of reading and writing virtual machine configuration data. The upgrade also reduces the potential for data corruption in the event of a storage failure.

 

With PowerShell we check what versions I have running

Get-VM * | Format-Table Name, Version

image

As you can see I have version 5.0 – 9.0 running time for some upgrading.

This VM has version 5 and I’m upgrading this to version 9.0 , Windows server 2019 default.

Microsoft Windows 10 October 2018 Update/Server 2019 9.0     True

Update-VMVersion HYD-DC1 

image

image

Confirming and done.

image

If you want to upgrade all vm’s   then use a *

Update-VMVersion *

Get-VMHostSupportedVersion –Default

image

 

Microsoft Windows 10 October 2018 Update/Server 2019 9.0     True

In the table below you can see the versions between the OS versions and LTSC and SAC.

Supported VM configuration versions for long-term servicing hosts

The following table lists the VM configuration versions that are supported on hosts running a long-term servicing version of Windows.

Hyper-V host Windows version 9.1 9.0 8.3 8.2 8.1 8.0 7.1 7.0 6.2 5.0
Windows Server 2019
Windows 10 Enterprise LTSC 2019
Windows Server 2016
Windows 10 Enterprise 2016 LTSB
Windows 10 Enterprise 2015 LTSB
Windows Server 2012 R2
Windows 8.1

Supported VM configuration versions for semi-annual channel hosts

The following table lists the VM configuration versions for hosts running a currently supported semi-annual channel version of Windows.

Hyper-V host Windows version 9.1 9.0 8.3 8.2 8.1 8.0 7.1 7.0 6.2 5.0
Windows 10 May 2019 Update (version 1903)
Windows Server, version 1903
Windows Server, version 1809
Windows 10 October 2018 Update (version 1809)
Windows Server, version 1803
Windows 10 April 2018 Update (version 1803)
Windows 10 Fall Creators Update (version 1709)
Windows 10 Creators Update (version 1703)
Windows 10 Anniversary Update (version 1607)

 

 

 

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted June 5, 2019 by Robert Smit [MVP] in Windows Server 2019

Tagged with ,

How to install Azure Portal app on Windows server 2019 #ws2019 #Azure #portal #winserv #Cloud #Hybrid   Leave a comment

As Windows Server 2019 Still holds Internet Explorer and no Edge Chromium or other browser. therefore all initial internet contact is done by the Internet Explorer. This can be annoying when you want to do something on the server and connect to Azure and first you need to install another browser.

This is just a quick blog on the Azure portal app, as this could be handy on any machine without using the browser.

Or you can download the Azure portal app.

When opening the IE browser and go to https://Portal.azure.com

You will see this, the option to download the Application to manage the portal.

image

Agreeing on the Terms and download

image

The Azueportalinstaller can also be deployed by SCCM or intune if you want. its not only an application that can be used on older machines.

image

The setup is easy and you only need to logon.

image

Use your Azure credentials and you good to go.

image

 

image

 

 

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted June 4, 2019 by Robert Smit [MVP] in Windows Server 2019

Tagged with ,

Configure Azure Service Endpoints for Web Applications #Azure #ASE #Endpoints #AzureServiceEndpoints #webapp #AzureDevOps   Leave a comment

Sometimes you are building things in Azure and thinking if this is possible than that would be a cool feature. Suddenly you are building this and noticed that it is already there in Azure. How Cool is that.

Today I was building a demo website but I did not want to expose this directly to the web, play with this and still get the use of Azure Cloud over the internet. Reading the Azure Endpoint services there is no WebApp Endpoint services. Using a NSG or enable the Azure Firewall well it is just a test so lets see what we can do with all the basic stuff. But during the test I saw this option Microsoft.Web in the service endpoints.

image

More security is needed in everything you expose to the internet. And in Azure it all starts with a Vnet.

Virtual Network (VNet) service endpoints extend your virtual network private address space and the identity of your VNet to the Azure services, over a direct connection. Endpoints allow you to secure your critical Azure service resources to only your virtual networks. Traffic from your VNet to the Azure service always remains on the Microsoft Azure network.

First we create a new Vnet, while we creating this wen can enable an pick the right service endpoints. this can also be done afterwards.

imageimageimage

Azure service traffic from a virtual network uses public IP addresses as source IP addresses. With service endpoints, service traffic switches to use virtual network private addresses as the source IP addresses when accessing the Azure service from a virtual network. This switch allows you to access the services without the need for reserved, public IP addresses used in IP firewalls.

In case you have already a Vnet, just go to the Service points and add the selected service you want to add or select it all.

image image image

At this point there is no Configuration, it is just adding a services to the network or subnet.

image

Below is a list of the Azure services that are currently available.

Generally available

Public Preview

The Web app is not listed but the option is there, and working. The Azure service Endpoint is not a Firewall, as the Azure Firewall this is a totally different service.

image

For Samples you have a Web application, and it needs to have connection to storage or SQL server and connection to an other Web services, without setting this open to Any – Any you can restrict this with the Azure Service Endpoints

image

Creating the Rules is a quick process, these are similar as in the NSG.

  • Network security groups (NSGs) with service endpoints:

    • By default, NSGs allow outbound Internet traffic and so, also allow traffic from your VNet to Azure services. This continues to work as is, with service endpoints.
    • If you want to deny all outbound Internet traffic and allow only traffic to specific Azure services, you can do so using service tags in your NSGs. You can specify supported Azure services as destination in your NSG rules and the maintenance of IP addresses underlying each tag is provided by Azure.

First we go the the Web App Service. in Networking and the non readers will click the VNet integration. #Wrong 

image

image

In this case I don’t want a premium network, So we go to Configure Access Restrictions

image

Here we create a access rule, on who gets access to this web application.

image

I created a deny rule for a specific IP.

image

image

And the pages shows an error webapp is stopped. here you can also see the difference between a complete port block and no access to the application.

image

Changing this to Allow the App is visible

image

Also for the KUDU SCM you can have different rules or apply the same rules. with the little check box

image

With these options you can create a more secure environment again this is a great add on.

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted May 29, 2019 by Robert Smit [MVP] in Azure

Tagged with ,

First look on the new Azure Sentinel cloud-native with Azure Notebooks free service #Jupyter #SIEM #SIEMaaS #Azure #Sentinel   2 comments

Azure Sentinel is Microsoft’s cloud-native SIEM that provides intelligent security analytics for your entire enterprise at cloud scale.
This SIEM as a Service (SIEMaaS) solution is designed as a cloud-based security-monitoring platform that leverages the power of the cloud for analytics and detections.

https://azure.microsoft.com/en-us/services/azure-sentinel/

there is a good video  https://www.youtube.com/watch?v=XXZp6LQZSJU&feature=youtu.be 

Limitless cloud speed and scale
Azure Sentinel is the first SIEM built into a public cloud platform to help your security analysts focus on what really matters.

Easily connect your data sources
Azure Sentinel provides simple and easy integration with signals and intelligence from security solutions whether they are on premises, in Azure, or in other clouds.
Azure Sentinel provides seamless integration with Microsoft 365, Azure, and other Microsoft products, including Microsoft’s security products.

Detect suspicious activities in your organization
Azure Sentinel fuses together unique machine learning algorithms, world-class security research, and the breadth and depth of the critical security data available to Microsoft as a major enterprise vendor. Azure Sentinel helps you detect both known and unknown attack vectors, detecting threats across all stages of the kill chain.

Investigate and remediate breaches
Azure Sentinel gives you visibility into all the entities involved in an alert and provides a simple and instinctive UI to investigate the detection, helping you easily understand the scope of the breach.
To cut down on the volume of alerts you get, Azure Sentinel automatically investigates alerts to help you determine what action to take, enabling you to move from alert to remediation in minutes, at scale.
Leveraging the power of Logic Apps, Azure Sentinel helps you respond to incidents instantly, using built-in orchestration and automation playbooks.

Joining the Preview program give you the enable option and you will need some configuration in the Azure portal. Overall a great overview in the new dash boarding. one thing is I need more screens to show all this.

Azure Sentinel cloud-native Azure Notebooks  Azure Sentinel cloud-native Azure Notebooks

You will need a workspace I you have already one you can use this or just create a new one

Azure Sentinel cloud-native Azure Notebooks

I’ll pick my current one as all my VM’s are reporting into this.

Azure Sentinel cloud-native Azure Notebooks  

Now we can install the add-on for data collection, there is already a big list.

Azure Sentinel cloud-native Azure Notebooks

As I already had a workspace there is already some content to use, at this point I don’t have any incidents, so no cases and alerts

Azure Sentinel cloud-native Azure Notebooks

I think this is a grate feature the “hunting” predefined query’s ready to run and adjustable to your need.

Azure Sentinel cloud-native Azure Notebooks

Reuse the custom query, for better adjustment in your site.

Azure Sentinel cloud-native Azure Notebooks

You can find more samples on github https://github.com/Azure/Azure-Sentinel

Azure Sentinel cloud-native Azure Notebooks

Also the Azure Notebooks for Azure Sentinel is a new option, create your Project in Jupyter

 

image

Azure Notebooks for Azure Sentinel

What is Azure Notebooks?

Azure Notebooks is a free hosted service to develop and run Jupyter notebooks in the cloud with no installation. Jupyter is an open source project that lets you easily combine markdown text, executable code (Python, R, and F#), persistent data, graphics, and visualizations onto a single, sharable canvas called a notebook.

How do Azure Notebooks work?

Interactive Azure Notebooks provides security insights and actions to investigate anomalies and hunt for malicious behaviors. Each Azure Notebook is purpose-built with a self-contained workflow for a specific use case. Visualizations are included in each Azure Notebook for faster data exploration and threat hunting. Click on the button below to clone our prebuilt investigation and hunting Azure Notebooks into projects that belong to you. Modify and tailor your projects to your environment. Either run the Azure Notebooks for free or, for better performance, run them on a dedicated virtual host. Click here to learn more.

Using the Notebooks locally or in other environments

Azure Sentinel will provision notebooks and supporting modules for you in Azure Notebooks. You can also download the notebooks and modules and use them locally in a supported Python environment (Anaconda is recommended) or another notebook hosting environment such as Azure Databricks or a JupyterHub environment that supports Python 3.6 or later.

 

image 

With the import a copy will be made from the Github to your own repository to get you started.

image 

this take some time after this the project page is opening for you.

image

You can check the samples and adjust them for your needs

image 

Checking the Logs in the Azure Sentinel will give you a nice dashboard with all the content. I have limited amount of data in this so no big lines or exceptions.

image

A Sample dashboard with the infrastructure query in Azure Sentinel

A Sample dashboard with the infrastructure query in Azure Sentinel

A Sample dashboard with the infrastructure query in Azure Sentinel

A Sample dashboard with the infrastructure query in Azure Sentinel

A Sample dashboard with the general overview query in Azure Sentinel

A Sample dashboard with the infrastructure query in Azure Sentinel

Some are based on multiple pages, big screens are needed or smaller fonts but overall this is a nice addition to the Azure Family.

A Sample dashboard with the infrastructure query in Azure Sentinel

Azure Sentinel will take some time to get this running and configuring but once there is data you will see a very nice new tool that can help you to solve your problems in Azure better an quickly.

See and stop threats before they cause harm, with SIEM reinvented for a modern world. Azure Sentinel is your birds-eye view across the enterprise. Put the cloud and large-scale intelligence from decades of Microsoft security experience to work. Make your threat detection and response smarter and faster with artificial intelligence (AI). Eliminate security infrastructure setup and maintenance, and elastically scale to meet your security needs—while reducing IT costs.

 

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted February 28, 2019 by Robert Smit [MVP] in Azure

Tagged with ,

Secure DevOps Kit for Azure (AzSK) With Security Monitoring #Devops #Azure #AzSK #Security #LogAnalytics #PowerShell   Leave a comment

The Secure DevOps Kit for Azure is a collection of scripts, tools, extensions, automations.

image

The kit is based on Powershell and can be extended to Azure log analytics with some nice dashboarding. But if you have a large subscription the Powershell query can take some time. With this toolkit Devops teams using extensive automation and smoothly integrating security into native Devops workflows helping accomplish secure Devops with these 6 focus areas:

  • Secure the subscription: A secure cloud subscription provides a core foundation upon which subsequent development and deployment activities can be conducted. An engineering team should have the capabilities to deploy and configure security in the subscription including elements such as alerts, ARM policies, RBAC, Security Center policies, JEA, Resource Locks, etc. Likewise, it should be possible to check that all settings are in conformance to a secure baseline
  • Enable secure development: During the coding and early development stages, developers should have the ability to write secure code and to test the secure configuration of their cloud applications. Just like build verification tests (BVTs), we introduce the concept of security verification tests (SVTs) which can check for security of various resource types in Azure
  • Integrate security into CICD: Test automation is a core tenet of devops. We emphasize this by providing the ability to run SVTs as part of the VSTS CICD pipeline. These SVTs can be used to ensure that the target subscription used to deploy a cloud application and the Azure resources the application is built upon are all setup in a secure manner
  • Continuous Assurance: In the constantly changing dev ops environment, it is important to move away from the mindset of security being a milestone. We have to treat security as a continuously varying state of a system. This is made possible through capabilities that enable continuous assurance using a combination of automation runbooks, schedules, etc.
  • Alerting & Monitoring: Visibility of security status is important for individual application teams and also for central enterprise teams. We provide solutions that cater to the needs of both. Moreover, the solution spans across all stages of dev ops in effect bridging the gap between the dev team and the ops team from a security standpoint through the single, integrated views it generates
  • Cloud Risk Governance: Lastly, underlying all activities in the kit is a telemetry framework that generates events capturing usage, adoption, evaluation results, etc. This allows us to make measured improvements to security targeting areas of high risk and maximum usage before others.

Keep in mind that The OMS portal will is retired on January 15, 2019. You can continue to use your existing services and licensing in the Azure portal.

Even in the Azure portal you can still connect to OMS

Complete feature set of Secure DevOps Kit for Azure

Feature Area Secure DevOps Kit Feature
Secure the subscription
  • Subscription Health Check
  • Subscription Provisioning
    • Alerts Configuration
    • ARM Policy Configuration
    • Azure Security Center Configuration
    • Access control (IAM) Hygiene
Enable secure development
  • Security Verification Tests (SVT)
  • Security IntelliSense- VS Extension
Integrate security into CICD
  • AzSK VS Extension-executes SVTs in a CICD pipeline
Continuous Assurance
  • Security scanning via Azure Automation Runbooks
Alerting & Monitoring
  • OMS Solution for AzSK containing:
    • Security dashboard views covering security state/actions
    • Alerts with pertinent search queries
Cloud Risk Governance
  • Control/usage telemetry through Insights

Setting up Secure DevOps Kit for Azure (AzSK)

First make sure you have the right Azure modules installed, I noticed the automation module failed So I added this manualy.

Import-Module AzureRM.Automation

Get-AzSKAzureServicesSecurityStatus -SubscriptionId

image

Installing the Secure DevOps Kit for Azure (AzSK)

Install-Module AzSK -Scope CurrentUser

image

Now that the Powershell modules are installed we can start the (AzSK) Scan

Get-AzSKAzureServicesSecurityStatus –SubscriptionId  ID

image

In this subscription there are 44 items that are been checked

image

Items are been checked on the security issues

image

Nice detailed overview is shown. Also a log folder is been created with all the issues. per resource Item.

image

As you can see I have some failed items and with a High, so I need to take a good look at this and fix this.

image

This maybe one of the best Items here an excel sheet with al the issues listed with the solution mentioned and if this can be automated.

If needed there is an URL that points you to the right solution.

image

As Azure log analytics is great and it can be integrated with some OMS (Azure monitoring Dashboards)

The OMS portal will be retired on January 15, 2019. You can continue to use your existing services and licensing in the Azure portal. So the current documentation need some updating.

image

Pressing the OMS button in the Azure portal brings you to the OMS portal but then nothing. As it is now all Azure portal.

Setting up the dashboards failed on me during the first installation but when I did run this a second time the dashboard was there.  (Timing) 

image

Creating the OMS default dashboard we need to run some powershell scripts.

$omsSubId =”id”   #subscription hosting the OMS workspace

$omsWSId =’OMS ID’

$omsRGName =’omsrsg’     #RG where the OMS workspace is hosted

$azSkViewName = ‘MVP_AzSK_view’ #This will identify the tile for AzSK view in OMS.


    #This command will deploy the AzSK view in the OMS workspace.  
    Install-AzSKOMSSolution -OMSSubscriptionId $omsSubId `

                    -OMSResourceGroup $omsRGName `

                    -OMSWorkspaceId $omsWSId `

                    -ViewName $azSkViewName

image

Note:

1) The blades of the OMS view created by this command will start populating only after AzSK scan events become available in the corresponding OMS workspace.

To understand how to send AzSK events to an OMS workspace see https://aka.ms/devopskit/oms.

2) The OMS view installed contains a basic set of queries over DevOps Kit scan events. Please feel free to customize them once you get familiar with the que

ries.

We also periodically publish updated/richer queries at: https://aka.ms/devopskit/omsqueries.

image

Checking the OMS – log analytics workspace it has not much issues as this is a test subscription and if it was all perfect then there is no fun.

image

image

and with longer logging and more Items in azure you will get a different overview.

image

There are lots of options you can set and there is a detailed description on how to use this on Github

Setting up ARM policys is also one of the options

Set-AzSKARMPolicies –SubscriptionId

image

So get started with the DevOpsKit https://github.com/azsk/DevOpsKit-docs 

image

https://github.com/azsk/DevOpsKit-docs/tree/master/05-Alerting-and-Monitoring

AzSK Security Controls Portal @https://aka.ms/azskosstcp

With this it’s a nice tool and yes a bit time consuming but learned a lot and make me see things different in the Azure Subscription 

And If you combine this directly and not afterwards then this could be your time saver to fix all the security items

image

 

Follow Me on Twitter @ClusterMVP

Follow My blog https://robertsmit.wordpress.com

Linkedin Profile Robert Smit MVP Linkedin profile

Google  : Robert Smit MVP profile

Posted January 24, 2019 by Robert Smit [MVP] in Azure

Tagged with , , ,

  • Twitter

  • RSS Azure and Microsoft Windows Server Blog

  • %d bloggers like this: