The Secure DevOps Kit for Azure is a collection of scripts, tools, extensions, automations.
The kit is based on Powershell and can be extended to Azure log analytics with some nice dashboarding. But if you have a large subscription the Powershell query can take some time. With this toolkit Devops teams using extensive automation and smoothly integrating security into native Devops workflows helping accomplish secure Devops with these 6 focus areas:
- Secure the subscription: A secure cloud subscription provides a core foundation upon which subsequent development and deployment activities can be conducted. An engineering team should have the capabilities to deploy and configure security in the subscription including elements such as alerts, ARM policies, RBAC, Security Center policies, JEA, Resource Locks, etc. Likewise, it should be possible to check that all settings are in conformance to a secure baseline
- Enable secure development: During the coding and early development stages, developers should have the ability to write secure code and to test the secure configuration of their cloud applications. Just like build verification tests (BVTs), we introduce the concept of security verification tests (SVTs) which can check for security of various resource types in Azure
- Integrate security into CICD: Test automation is a core tenet of devops. We emphasize this by providing the ability to run SVTs as part of the VSTS CICD pipeline. These SVTs can be used to ensure that the target subscription used to deploy a cloud application and the Azure resources the application is built upon are all setup in a secure manner
- Continuous Assurance: In the constantly changing dev ops environment, it is important to move away from the mindset of security being a milestone. We have to treat security as a continuously varying state of a system. This is made possible through capabilities that enable continuous assurance using a combination of automation runbooks, schedules, etc.
- Alerting & Monitoring: Visibility of security status is important for individual application teams and also for central enterprise teams. We provide solutions that cater to the needs of both. Moreover, the solution spans across all stages of dev ops in effect bridging the gap between the dev team and the ops team from a security standpoint through the single, integrated views it generates
- Cloud Risk Governance: Lastly, underlying all activities in the kit is a telemetry framework that generates events capturing usage, adoption, evaluation results, etc. This allows us to make measured improvements to security targeting areas of high risk and maximum usage before others.
Keep in mind that The OMS portal will is retired on January 15, 2019. You can continue to use your existing services and licensing in the Azure portal.
Even in the Azure portal you can still connect to OMS
Complete feature set of Secure DevOps Kit for Azure
Setting up Secure DevOps Kit for Azure (AzSK)
First make sure you have the right Azure modules installed, I noticed the automation module failed So I added this manualy.
Import-Module AzureRM.Automation
Get-AzSKAzureServicesSecurityStatus -SubscriptionId
Installing the Secure DevOps Kit for Azure (AzSK)
Install-Module AzSK -Scope CurrentUser
Now that the Powershell modules are installed we can start the (AzSK) Scan
Get-AzSKAzureServicesSecurityStatus –SubscriptionId ID
In this subscription there are 44 items that are been checked
Items are been checked on the security issues
Nice detailed overview is shown. Also a log folder is been created with all the issues. per resource Item.
As you can see I have some failed items and with a High, so I need to take a good look at this and fix this.
This maybe one of the best Items here an excel sheet with al the issues listed with the solution mentioned and if this can be automated.
If needed there is an URL that points you to the right solution.
As Azure log analytics is great and it can be integrated with some OMS (Azure monitoring Dashboards)
The OMS portal will be retired on January 15, 2019. You can continue to use your existing services and licensing in the Azure portal. So the current documentation need some updating.
Pressing the OMS button in the Azure portal brings you to the OMS portal but then nothing. As it is now all Azure portal.
Setting up the dashboards failed on me during the first installation but when I did run this a second time the dashboard was there. (Timing)
Creating the OMS default dashboard we need to run some powershell scripts.
$omsSubId =”id” #subscription hosting the OMS workspace
$omsWSId =’OMS ID’
$omsRGName =’omsrsg’ #RG where the OMS workspace is hosted
$azSkViewName = ‘MVP_AzSK_view’ #This will identify the tile for AzSK view in OMS.
#This command will deploy the AzSK view in the OMS workspace.
Install-AzSKOMSSolution -OMSSubscriptionId $omsSubId `
-OMSResourceGroup $omsRGName `
-OMSWorkspaceId $omsWSId `
-ViewName $azSkViewName
Note:
1) The blades of the OMS view created by this command will start populating only after AzSK scan events become available in the corresponding OMS workspace.
To understand how to send AzSK events to an OMS workspace see https://aka.ms/devopskit/oms.
2) The OMS view installed contains a basic set of queries over DevOps Kit scan events. Please feel free to customize them once you get familiar with the que
ries.
We also periodically publish updated/richer queries at: https://aka.ms/devopskit/omsqueries.
Checking the OMS – log analytics workspace it has not much issues as this is a test subscription and if it was all perfect then there is no fun.
and with longer logging and more Items in azure you will get a different overview.
There are lots of options you can set and there is a detailed description on how to use this on Github
Setting up ARM policys is also one of the options
Set-AzSKARMPolicies –SubscriptionId
So get started with the DevOpsKit https://github.com/azsk/DevOpsKit-docs
https://github.com/azsk/DevOpsKit-docs/tree/master/05-Alerting-and-Monitoring
AzSK Security Controls Portal @https://aka.ms/azskosstcp
With this it’s a nice tool and yes a bit time consuming but learned a lot and make me see things different in the Azure Subscription
And If you combine this directly and not afterwards then this could be your time saver to fix all the security items
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
This blog is about creating a new SQL cluster with a Failover instance based on a storage space direct configuration. There are a lot off opinions on do’s and don’t for on prem and in Azure with S2D. This is not the blog post on this discussion. This is just a post on SQL 2019 on Windows server 2019 storage space direct because I can!
And in a later blog post we are extending this to Azure. Still it always depends on running a Full SQL server or use a managed instance in Azure. But for now the new SQL server 2019 is the basic of our cluster running on a Windows Server 2019 cluster. As this time the installation will be GUI based as I got often the question on the screens. normally I run some PowerShell or a command line script that will do the job in several minutes
What are the new features in SQL Server 2019, the list below is just a short list for a full updated list go to the link below.
OR if you want to run a managed SQL server in Azure : https://azure.microsoft.com/en-us/pricing/details/sql-database/managed/
The Database Engine is the core service for storing, processing, and securing data. The Database Engine provides controlled access and rapid transaction processing to meet the requirements of the most demanding data consuming applications within your enterprise. The Database Engine also provides rich support for sustaining high availability.
https://docs.microsoft.com/en-us/sql/sql-server/sql-server-technical-documentation?view=sql-server-ver15
What’s new in SQL Server 2019
- Installation of SQL Server is supported on x64 processors only. It is no longer supported on x86 processors.
- SysPrep is supported for all installations of SQL Server. SysPrep now supports failover cluster installations
- Always On Availability Groups – secondary replica connection redirection
- SQL Server Machine Learning Services failover clusters
- the operating system requirements for the principal editions of SQL Server
- SQL Server Management Studio (SSMS) 18.0 (preview)
- Azure Data Studio
- Azure Data Studio
Always On Availability Groups – more synchronous replicas (CTP 2.0)
-
Up to five synchronous replicas: SQL Server 2019 preview increases the maximum number of synchronous replicas to 5, up from 3 in SQL Server 2017 (14.x) . You can configure this group of 5 replicas to have automatic failover within the group. There is 1 primary replica, plus 4 synchronous secondary replicas.
-
Secondary-to-primary replica connection redirection: Allows client application connections to be directed to the primary replica regardless of the target server specified in the connection string. This capability allows connection redirection without a listener. Use secondary-to-primary replica connection redirection in the following cases:
- The cluster technology does not offer a listener capability.
- A multi subnet configuration where redirection becomes complex.
- Read scale-out or disaster recovery scenarios where cluster type is
NONE.
SQL Server Enterprise /SQL Server Standard :
- Windows Server 2019
- Windows Server 2016
- Windows Server 2012 R2
- Windows Server 2012
The supported storage types for data files are:
As the SQL server 2019 is still in preview I download the evaluation version. Get your SQL server here
Pick the version you want, in our case I select the download media.
Pick the ISO format do download this.
The SQL installation is done on my cluster. described in the blog post below
https://robertsmit.wordpress.com/2018/11/29/step-by-step-windows-server-2019-file-server-clustering-with-powershell-or-gui-cluster-ha-azure-windowsadmincenter-windowsserver2019/
In this Windows server 2019 cluster I created a S2D formatted with ReFS. There is plenty of info to find on my blog how to create the S2D in the cluster.
https://robertsmit.wordpress.com/2016/01/05/using-windows-storage-spaces-direct-with-hyper-converged-in-microsoft-azure-with-windows-server-2016/
https://robertsmit.wordpress.com/2017/11/09/azure-storage-spaces-direct-s2d-standard-storage-vs-premium-storage/
We have a Windows server 2019 Cluster in place with the Storagespace direct configuration.
The specific cluster configuration is debatable, in a typical SQL configuration you would have multiple disks.
Our Cluster with storage spaces. In storage spaces all disks come from the same disk pool if you create things default. then multiple disk has a different purpose. As you want to manage the SQL different than others.
Our SQL Installation is done this time by the GUI, show you the screens and options. you can always use the one liner to install the SQL server unattended.
And no I do not use the normal cluster installation but I use the advanced Cluster prep and completion. these two steps gives me a bit more freedom and flexibility it the installation fails.
The choice is developer but can also be the one with a product key.
Just do the updates direct as it is a new version and on a new OS, just to make sure that I run the latest bits.
I got a warning on the Firewall as I turned this off during the installation. This is configured by a GPO when I place the Cluster in the Right AD OU
Selecting the features and the installation folders.
Do I go for a default instance or a named instance. I always go for a named instance. Its easy and better management. In the old days some applications just want to run on a default instance.
My service account that has JEA Just enough Access
C:\Program Files\Microsoft SQL Server\150\Setup Bootstrap\Log\20181217_034408\ConfigurationFile.ini
If you want to use a unattended install you can use this ini file to get the same or adjusted values during the install
Ini file content.
Now that the first step is completed we can do the completion step.
Now the second step is needed to finish the SQL 2019 installation on the first node
But running the wizard if failed on me, as the error said the Cluster is not verified!
I just need to run the Cluster validation.
Now with the Passed Cluster validation we try again.
Now this looks good lets stat the installation of the SQL 2019
With the Named instance and usage of an Alias, I can easy change this or move the SQL to another cluster.
The installation will show me what cluster groups I can use and which are used at the moment.
Selecting the right disks for the installation remember These are the Storage space direct disks that are created with the cluster installation. See my other blog post about creating S2D on Windows Server 2019
A fixed IP is needed unless you use the DHCP checkbox.
Some applications needs special Collation, when changing this it is an Instance setting and can’t be changed again.
Normally you will place the SQL admins or some groups. this is a demo cluster so admin access only
Select the right disks and change this in the location fields.
Selecting multiple files for the Temp DB and the DB location
Also in this step we get a Ini file as output this can be reused if needed.
Now that the installation on one node is ready you can see the SQL resources in the cluster. As I did only the install on one node we need to do the Add node installation to get the installation done for a full FCI.
The Step to add an extra cluster node with the SQL installation is an easy step.
Follow the Wizard and use the Same account that you used before.
As the SQL studio is no longer a default installation you need to download the bits from Microsoft.
SSMS 18.0 Public Preview 6 is now available, and is the latest generation of SQL Server Management Studio that provides support for SQL Server 2019 preview!
SSMS 17.9.1 is the current General Availability (GA) version of SSMS
Download SQL Server Management Studio 17.9.1
Download SQL Server Management Studio 17.9.1 Upgrade Package (upgrades 17.x to 17.9.1)
Version Information
- Release number: 17.9.1
- Build number: 14.0.17289.0
- Release date: November 21, 2018
As we are using the 2019 We need to have the latest version
Now we have the studio ready and can access the SQL instance.
In the next blogs I’ll show you how to extend the SQL to Azure.
Below is an overview of the SQL disks based on storage spaces direct.
With All these steps I hope you can build your own cluster and play with this. As for production never use a next next Finish installation there is always some custom tweaks needed to get the best performance.
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
Olympia V2 is the next step for enabling Windows Insiders to try new and pre-release Windows 10 Enterprise features. Windows Insider Lab for Enterprise v2 provides a complete Microsoft 365 deployment and management testing environment that can be run directly on your own machines. The lab features both client and administrative functionality, including System Center Configuration Manager Preview plus connectivity to Office 365 and Enterprise Mobility Security evaluation trials. Customers can also add the latest Windows 10 Insider Preview Enterprise build to the lab.
This is a great lab toolkit, to start with the new features. It is easy to setup with a great learning curve.
First we download the entire Lab. it around 14GB
The table below lists the virtual machines, which will be imported and created in Hyper-V:
|
Server Name
|
Roles & Products
|
|
HYD -DC1
|
Active Directory Domain Controller, DNS, DHCP, Certificate Services
Windows Server 2016
|
|
HYD-CM1
|
System Center Configuration Manager Technical Preview Branch – Version 1808 (Note: After installing a baseline version, you can then use in-console updates to bring your installation up-to-date with the most recent preview version. See Section 4.)
Windows Deployment Services
Microsoft Deployment Toolkit
Windows 10 ADK
Windows Software Update Services
Microsoft SQL Server 2014
Windows Server 2016
|
|
HYD-APP1
|
Microsoft BitLocker Administration and Monitoring
Microsoft SQL Server 2014
Windows Server 2016
|
|
HYD-GW1
|
Remote Access for Internet Connectivity
Windows Server 2016
|
|
HYD-CLIENT1 (Optional)
|
If the Windows 10 Insider Preview ISO image is imported, this machine will be created with windows installed and will be Domain Joined
|
|
HYD-CLIENT2 (Optional)
|
If the Windows 10 Insider Preview ISO image is imported, this machine will be created with windows installed and will be Domain Joined
|
|
HYD-CLIENT3 (Optional)
|
If the Windows 10 Insider Preview ISO image is imported, this machine will be created with windows installed and will be on Workgroup
|
|
HYD-CLIENT4 (Optional)
|
If the Windows 10 Insider Preview ISO image is imported, this machine will be created with windows installed and will be on Workgroup
|
The VM list in Hyper-v
The table below lists the credentials and access type available in the default implementation:
After that just extract the files, keep in mind the setup extract the files at the current location of the setup files. You can move the VM’s afterwards
Starting the setup and extracting the VM’s
Select your Vswitch on the Hyper-v server
Select a insiders ISO or download one,
Plenty of room in the Windows Server 2019 Hyper-v server with Storage Spaces direct.
The extracting can take up some time depends on the disks and CPU speed for extraction
After the Extraction Several VM’s are added to the Hyper-v Server
The Gateway will route all the data to internet.
The setup is done the full lab is installed, there are several laps that you can do and setup
The domain structure that is created is the basic for all the labs
A SCCM site is created and ready for use. As this is the Technical preview I already got the 1812 Build
In the Azure Active directory we set some custom pictures.
Customize these screens is easy done in the Azure portal
Next step is use SCCM and Intune to manage your systems. This lab is perfect for showing all the options.
The Setup is Complete and ready to use, this lab is a great way to self explore the new features.
Lab Objectives
This guide is designed to provide step-by-step guidance in demonstrating the basic functionality of the feature.
· Lab Setup
o On-Premises Environment
o Cloud Environment
o On-Premises Environment Post Setup Manual Steps
· Servicing
o Windows Analytics Update Compliance
· Deployment & Management
o Modern Device Deployment
o Modern Device Management with AutoPilot
o Co-Management
o Modern Application Management with Intune
o Enterprise State Roaming
· Security
o Windows Information Protection
o Windows Defender Advanced Threat Protection
o Windows Defender Application Guard
o Windows Defender Exploit Guard
o Windows Hello
o Credential Guard
o Device Encryption (MBAM)
o Device Guard – User Mode Code Integrity
· Compatibility
o Windows Analytics Upgrade Readiness
o Browser Compatibility
o Desktop Bridges
· Additional Labs
o MDM WINS over GP
o MAM FAQ
The Windows Insider Lab for Enterprise was designed for Windows Insiders who want to try new experimental and pre-release Enterprise Privacy and Security features. There are two versions of the lab:
· Windows Insider Lab for Enterprise v1 – provides a client-side view of the latest Microsoft 365 enterprise features through access to Olympia Corp – a virtual corporation has been set up to reflect the IT infrastructure of real world business.
· Windows Insider Lab for Enterprise v2 – provides a complete Microsoft 365 deployment and management testing environment that can be run directly on your own machines. The lab features both client and administrative functionality, including System Center Configuration Manager Preview plus connectivity to Office 365 and Enterprise Mobility + Security evaluation trials.
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
In the former blog post :https://robertsmit.wordpress.com/2018/11/29/step-by-step-windows-server-2019-file-server-clustering-with-powershell-or-gui-cluster-ha-azure-windowsadmincenter-windowsserver2019/
I created a File share on a Cluster to make the share HA. This is more the traditional way to make the share HA. But what if you have multiple locations and you want to use this share in Azure. Big internal lines between the Datacenter and copy the files to Azure (DFS) method. but that’s old. Better use the Azure File Sync option the files are synced to all the Server and available in Azure. Better and faster.
#bettertogether
With Azure File Sync , shares can be replicated on-premises or in Azure and accessed through SMB or NFS shares on Windows Server. Azure File Sync is useful for scenarios in which data needs to be accessed and modified far away from an Azure datacenter, such as in a branch office scenario. Data may be replicated between multiple Windows Server endpoints, such as between multiple branch offices. Azure File Sync transforms Windows Server into a quick cache of your Azure file share. You can use any protocol that’s available on Windows Server to access your data locally, including SMB, NFS, and FTPS. You can have as many caches as you need across the world.
To get started with the Azure File Sync we need a Storage account in Azure.
We create a storage account in Azure.
Remember this works only on Windows Servers ! System Requirements:
-
A server running Windows Server 2012 R2, Windows Server 2016 or Windows Server 2019:
Version
Supported SKUs
Supported deployment options
Windows Server 2019
Datacenter and Standard
Full (server with a UI)
Windows Server 2016
Datacenter and Standard
Full (server with a UI)
Windows Server 2012 R2
Datacenter and Standard
Full (server with a UI)
Now that the storage account is created we are starting with the Azure File Sync creation in Azure.
Name the Storage Sync Service , and create a resource group.
The next step is register the Onpremise server to Azure with the Azure File Sync Agent
Azure File Sync Agent download https://go.microsoft.com/fwlink/?linkid=858257
The installation is in two steps.
- Installing the agent
- Configuring the Agent
After the download install the Agent on the File server, As I use a Cluster install the Agent on every node of the Cluster.
Now that the agent is installed the Second wizard pops up for the configuration and if needed a update.
So far so good. As the Agent is connecting to Azure there are some additional components needed.
As this Cluster was a fresh installation and I did not used the PowerShell command for Azure here I need to install the AzureRM modules (or AZ module)
https://go.microsoft.com/fwlink/?linkid=856959
Installing and updating the modules.
Install-Module -Name AzureRM –AllowClobber
With this command you can see the current Powershell version
Get-Module -Name AzureRM -List | select Name,Version
Now that the PowerShell commands are installed we can refresh the page and the installation continues
If you are using a CSP subscription in Azure then you need to set this check box. and use your tenant ID
In all other subscriptions keep this default
Pick the right Resource group the one with the created Storage Sync services in it. else the field will be empty.
Select a resource group that contains a Storage Sync Service, or use the Azure portal to create one in this resource group.
When this process is done we can configure the rest in the Azure portal.
As you can see the Cluster CNO object is named here
In the pane that opens, enter the following information to create a sync group with a cloud endpoint:
- Sync group name: The name of the sync group to be created. This name must be unique within the Storage Sync Service, but can be any name that is logical for you.
- Subscription: The subscription where you deployed the Storage Sync Service.
- Storage account: If you select Select storage account, another pane appears in which you can select the storage account that has the Azure file share that you want to sync with.
- Azure file share: The name of the Azure file share with which you want to sync.
Next is creating the Sync group.
Pick a name for the Sync group name. and the proper Storage account that we created earlier. In this storage account we did not create a File share this is needed to hold the Files. so the azure file share check box is not showing you anything.
Go the the storage account and create a File share
With this created the creation of the Sync group can be completed.
Next step is creating some endpoints. this means bind the local share to the services and sync this to the Azure storage account share.
Adding the endpoint and pick the registered server and the file share that will be synced.
If you want to enable cloud Tiering and fill in the values. In this demo I don’t use this.
Note:
Only NTFS volumes are supported. ReFS, FAT, FAT32, and other file systems are not supported.
Failover Clustering
Windows Server Failover Clustering is supported by Azure File Sync for the "File Server for general use" deployment option. Failover Clustering is not supported on "Scale-Out File Server for application data" (SOFS) or on Clustered Shared Volumes (CSVs).
The Azure File Sync agent must be installed on every node in a Failover Cluster for sync to work correctly.
In my demo the Share is not listed, I already know why, As I used ReFS for the cluster disk.
This can be painful as you need to format that disk and move all the data to a temp location.
After changing the disk format and a refresh you can see that the deployment is pending and working.
After this you have a full Hybrid file share Fully redundant on premise and a off load to Azure.
As last the best option to get the data into this HA file share is using the Windows Admin Center
In Windows Admin Center there is a great options Storage Migration Services
Opening Windows admin Center and select the source this will be scanned and when done the files can be migrated. (the scanning can take some time)
When the scanning is done the files and shares are listed. more info can be found here https://youtu.be/WCWxAp27ERk
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
Installing the Cluster is easy now days. But just this I post a little blog on how to do this, In my blog stats it shows that the 2012 post is still very active , so time for an update to Windows Server 2019. in the creation there isn’t much changed, it gets only easier. but If you still not in PowerShell you got more clicks to do an less Coffee. And Windows Admin Center is also a great addition to manage a cluster. This blog post is also usable in Azure Only you need to add Storagespacesdirect and a CSV file share.
Just install a bare metal (VM) windows Server 2019 and do a domain join and the fun can start.
Installing the Cluster Feature in powershell
Install-WindowsFeature –Name Failover-Clustering –IncludeManagementTools
#Create cluster validation report
Test-Cluster -Node MVP19-01,MVP19-02
#Create new Cluster
New-Cluster -Name MVP1911-27 -Node MVP19-01,MVP19-02 -NoStorage -StaticAddress "10.255.255.45"
#place witness file on USB device from my router
Set-ClusterQuorum -FileShareWitness \\SERVER\SHARE -Credential $(Get-Credential)
Now that the basic cluster is ready we start with the HA share
File share witness enhancements We enabled the use of a file share witness in the following scenarios:
- Absent or extremely poor Internet access because of a remote location, preventing the use of a cloud witness.
- Lack of shared drives for a disk witness. This could be a Storage Spaces Direct hyperconverged configuration, a SQL Server Always On Availability Groups (AG), or an * Exchange Database Availability Group (DAG), none of which use shared disks.
- Lack of a domain controller connection due to the cluster being behind a DMZ.
-
A workgroup or cross-domain cluster for which there is no Active Directory cluster name object (CNO). Find out more about these enhancements in the following post in Server & Management Blogs: Failover Cluster File Share Witness and DFS.
We now also explicitly block the use of a DFS Namespaces share as a location. Adding a file share witness to a DFS share can cause stability issues for your cluster, and this configuration has never been supported. We added logic to detect if a share uses DFS Namespaces, and if DFS Namespaces is detected, Failover Cluster Manager blocks creation of the witness and displays an error message about not being supported.
that’s it the cluster is created, we can start with the File server
Next is installation of the file server role
A restart is needed! After the restart we can build the cluster with the HA file share
$servers = ("MVP19-01", "MVP19-02")
foreach ($server in $servers) {Install-WindowsFeature -Name file-services -ComputerName $server}
Now that the File Server Role is added we can add the Disk. Or use a disk that you already added before.
First we need to add a disk this can be done in the Failover Cluster manager or with PowerShell
Get-ClusterAvailableDisk | Add-ClusterDisk
The Roles are there and the Disk is added
Next step is adding the File server Role to the Cluster and add the HA File Share.
In this case I have a fail over disk and I use the File Server for general use.
So when adding the Disk it is not showing the disk. This is The disk is added to the cluster but the disk isn’t formatted!
Keep in mind that formating the cluster disk while it is online is not possible. You need to set the disk in maintenance mode else the format will fail.
So after the disk format we will see the Disk appear and can be added to the File server
After this the File server is up and running. As you can see the setup is screen intense, building this with PowerShell is a lot faster.
Powershell
add-ClusterFileServerRole -Storage "Cluster Disk 1" -Name MyFiles
New-SmbShare -Name "Data" -Path "J:\Data" -EncryptData $True
Quick steps with powershell and even the share is created and encrypted
Next step is adding the file share.
go for the Quick setup
Pick the disk and select the folder with the data on the disk, if there is no data then create a folder that will hold the data later.
as you can see the UNC path from the File Server.
As you can see the settings can be adjusted for you needs and also set the right access, and keep in mind this needs to be don on the Cluster Level!
All Done
So creating a File Server and 2 file shares is Click intensive if you don’t use PowerShell.
But What about Windows Admin Center ? yes that would be an option also except here you can’t create a cluster role.
You can create a new role but no file server /share etc.
But when the share is created and running like now you can use Windows Admin Center for migration the data to the file share.
But more and more options are coming in Windows Admin Center below are some links that you can use to add your request to the UserVoice
https://docs.microsoft.com/en-us/windows-server/manage/windows-admin-center/use/manage-failover-clusters
More Coming
Failover cluster management in Windows Admin Center is actively under development and new features will be added in the near future. You can view the status and vote for features in UserVoice:
Feature Request
Show more clustered disk info
Support additional cluster actions
Support converged clusters running Hyper-V and Scale-Out File Server on different clusters
View CSV block cache
See all or propose new feature
+++++++++++++++
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
Distributed denial of service (DDoS) attacks are some of the largest availability and security concerns facing customers that are moving their applications to the cloud. A DDoS attack attempts to exhaust an application’s resources, making the application unavailable to legitimate users. DDoS attacks can be targeted at any endpoint that is publicly reachable through the internet.
What is DDoS Protection? Protecting applications from DDoS attacks has been one of the top security concerns for Azure customers. Azure DDoS protection service is an Azure Networking offering aimed at protecting publicly accessible endpoints from DDoS attacks. The offering gives customers access to the same protection that is used to protect Microsoft’s online assets, such as Xbox Live and Office 365. Azure DDoS protection service provides constant network flow monitoring of the protected endpoints, and when detecting a DDoS attack, automatically applies traffic scrubbing to make sure only legitimate requests are forwarded to the application.
Azure DDoS protection, combined with application design best practices, provide defense against DDoS attacks. Azure DDoS protection provides the following service tiers:
- Basic: Automatically enabled as part of the Azure platform. Always-on traffic monitoring, and real-time mitigation of common network-level attacks, provide the same defenses utilized by Microsoft’s online services. The entire scale of Azure’s global network can be used to distribute and mitigate attack traffic across regions. Protection is provided for IPv4 and IPv6 Azure public IP addresses.
- Standard: Provides additional mitigation capabilities over the Basic service tier that are tuned specifically to Azure Virtual Network resources. DDoS Protection Standard is simple to enable, and requires no application changes. Protection policies are tuned through dedicated traffic monitoring and machine learning algorithms. Policies are applied to public IP addresses associated to resources deployed in virtual networks, such as Azure Load Balancer, Azure Application Gateway, and Azure Service Fabric instances, but this protection does not apply to App Service Environments. Real-time telemetry is available through Azure Monitor views during an attack, and for history. Rich attack mitigation analytics are available via diagnostic settings. Application layer protection can be added through the Azure Application Gateway Web Application Firewall or by installing a 3rd party firewall from Azure Marketplace. Protection is provided for IPv4 Azure public IP addresses.
So how to start with DDoS in Azure.
First go to the Virtual Networks.
Next selecting the Network and in the left pane there is a section DDoS Protection.
Selecting the DDoS Protection there is the Basic and the Standard Setting
Pricing Details
There the Basic is the default and comes with free pricing.
The Standard is a different option and Cost you some real money! and these are monthly costs. For a demo I turned it on and forget to turned it of and spend 10K in 4 months so keep a track on your Azure costs.
The DDoS Protection service will have a fixed monthly charge, as well as a charge for data processed. The fixed monthly charge includes protection for 100 resources. Protection for additional resources will be charged on a monthly per-resource basis.
Monthly price for DDoS Protection (includes protection for 100 resources): €2,483/month
Overage charges (more than 100 resources): €25 per resource per month
When Enabling the DDoS Standard we need to create a DDoS protection plan first, if you have already one you can add the ID.
Check the create DDoS protection Plan
Now that we created a plan witch is more a resource place holder, we can add this to the DDoS protection plan
Now that the DDoS and the plan is in place we can create an alert rule in case we have a DDoS attack.
In the Azure Monitor we can create the alert rule and we can see the logging.
To see telemetry for a DDoS attack, log into the Azure Portal and navigate to the “Monitor” blade.
Within the monitor blade, click on “Metrics”, select the appropriate subscription, resource group, resource type of “Public IP” and the Public IP that was the target of the attack. After selecting the resource, a series of Available Metrics will appear on the left side. These metrics are selected and then will be graphed.
The metric names are relatively self-explanatory and the basic construct is that there are tag names on each metric as follows: • Dropped tag name (e.g. Inbound Packets Dropped DDoS): The number of packets dropped/scrubbed by the DDoS system
• Forwarded tag name (e.g: Inbound Packets Forwarded DDoS): The number of packets forwarded by the DDoS system to the destination VIP – traffic that was not filtered • No tag name (e.g: Inbound Packets DDoS): The total number of packets that came into the scrubbing system – representing the sum of the packets dropped and forwarded
The traffic shown in the Monitor dashboard.
To create a dashboard there are some options with counters. It all depends on your need.
now we create an alert rule.
Email Alerting To configure an email alert for a metric, click on the “Click to add an alert” text. An email alert can be created on any metric, but the most obvious metric to create an alert on is “Under DDoS attack or not”. This is a boolean value 1 or 0. “1” means you are under attack. “0” means you are not under attack. To be emailed when under attack, set the Metric for “Under DDoS attack or not” and “Condition” to “Greater than” zero (0) over the last 5 minutes. Similar alerts can be set up for other metrics. An example screenshot is provided below.
To divine the Severity I keep this as this is also be used in SCOM
Azure Monitor Alert Severity Levels
Sev 0 = Critical
Sev 1 = Error
Sev 2 = Warning
Sev 3 = Informational
Sev 4 = Verbose
Last part in selecting the email for this alert.
With this setup you got a good protection against DDoS attacks. below is the workflow how DDoS protection works.
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
In the old days all File servers where on one place, and if you want to replicate data you needed a extra tool to do this. Now days its already build in into Windows server. Storage replica can be used in several ways, replicate data from one Cluster to another or to Azure. but in this case I do a server to server replication as not everyone has a cluster.
For moving data to the Cloud there are currently several other applications like Azure file sync or Azure Migrate https://docs.microsoft.com/en-us/azure/migrate/migrate-overview Blog about Azure File Sync https://robertsmit.wordpress.com/2017/09/28/step-by-step-azure-file-sync-on-premises-file-servers-to-azure-files-storage-sync-service-afs-cloud-msignite/
Storage Replica is Windows Server technology that enables replication of volumes between servers or clusters for disaster recovery. It also enables you to create stretch failover clusters that span two sites, with all nodes staying in sync.
Storage Replica supports synchronous and asynchronous replication:
-
- Synchronous replication mirrors data within a low-latency network site with crash-consistent volumes to ensure zero data loss at the file-system level during a failure.
- Asynchronous replication mirrors data across sites beyond metropolitan ranges over network links with higher latencies, but without a guarantee that both sites have identical copies of the data at the time of a failure.
Storage Replica allows more efficient use of multiple datacenters. By stretching clusters or replicating clusters, workloads can be run in multiple datacenters for quicker data access by local proximity users and applications, as well as better load distribution and use of compute resources. If a disaster takes one datacenter offline, you can move its typical workloads to the other site temporarily.
Storage Replica may allow you to decommission existing file replication systems such as DFS Replication that were pressed into duty as low-end disaster recovery solutions. While DFS Replication works well over extremely low bandwidth networks, its latency is very high – often measured in hours or days. This is caused by its requirement for files to close and its artificial throttles meant to prevent network congestion. With those design characteristics, the newest and hottest files in a DFS Replication replica are the least likely to replicate. Storage Replica operates below the file level and has none of these restrictions.
Storage Replica also supports asynchronous replication for longer ranges and higher latency networks. Because it is not checkpoint-based, and instead continuously replicates, the delta of changes will tend to be far lower than snapshot-based products. Furthermore, Storage Replica operates at the partition layer and therefore replicates all VSS snapshots created by Windows Server or backup software; this allows use of application-consistent data snapshots for point in time recovery, especially unstructured user data replicated asynchronously.
The Setup I used two servers both domain joined, And there are different ways to configure the Storage Replica, the easy way and the 10 second way.
First we are installing the Storage replica feature and the File server Role. The Storage replica feature needs a reboot.
Or use Powershell
install-WindowsFeature “Storage-Replica” –IncludeAllSubFeature
If you don’t know the module name you can find it easily
A reboot is needed.
Doing this server by server is not handy, So placing this together saves us some time.
$Servers = “Building-5”,”Building-9”
$Servers | ForEach { Install-WindowsFeature -ComputerName $_ -Name Storage-Replica,FS-FileServer -IncludeManagementTools -restart }
The –restart does an automatic restart if this is needed.
Storage Replica prerequisites
-
- Active Directory Domain Services forest.
- Storage Spaces with SAS JBODs, Storage Spaces Direct, fibre channel SAN, shared VHDX, iSCSI Target, or local SAS/SCSI/SATA storage. SSD or faster recommended for replication log drives. Microsoft recommends that the log storage be faster than the data storage. Log volumes must never be used for other workloads.
- At least one Ethernet/TCP connection on each server for synchronous replication, but preferably RDMA.
- At least 2GB of RAM and two cores per server. (with less memory the replication won’t start)
- A network between servers with enough bandwidth to contain your IO write workload and an average of 5ms round trip latency or lower, for synchronous replication. Asynchronous replication does not have a latency recommendation.
As there is no Gui on the replica part we need to configure this by PowerShell or with the new Windows Admin Center
Both our servers had Two extra disks. One log and Data Disk.
-
- You must create two volumes on each enclosure: one for data and one for logs.
- Log and data disks must be initialized as GPT, not MBR.
- The two data volumes must be of identical size.
- The two log volumes should be of identical size.
- All replicated data disks must have the same sector sizes.
- All log disks must have the same sector sizes.
- The log volumes should use flash-based storage, such as SSD. Microsoft recommends that the log storage be faster than the data storage. Log volumes must never be used for other workloads.
- The data disks can use HDD, SSD, or a tiered combination and can use either mirrored or parity spaces or RAID 1 or 10, or RAID 5 or RAID 50.
- The log volume must be at least 9GB by default and may be larger or smaller based on log requirements.
- The File Server role is only necessary for Test-SRTopology to operate, as it opens the necessary firewall ports for testing.
As you can see there are some needs for the Replication As I show you below with the performance test why you need this.
First we are configuring the Disks on both servers. with some PowerShell commands but this can also be done with Disk manager.
Get-Disk | Where FriendlyName -eq ‘Msft Virtual Disk’
Get-Disk | Where FriendlyName -eq ‘Msft Virtual Disk’|Initialize-Disk -PartitionStyle GPT –PassThru
1..2 | % { Get-Disk $_ }| Where FriendlyName -eq ‘Msft Virtual Disk’|New-Partition -AssignDriveLetter -UseMaximumSize | Format-Volume -FileSystem ReFS -NewFileSystemLabel “SR01-disk” -Confirm:$false
I formatted the disk with ReFS and not with NTFS.
Now that the disks are in place we can start but before we start building the replica I want to make sure the connection and the network is fast and the server can deliver the performance we need.
Therefor I download a test tool Diskspd. https://aka.ms/diskspd
Important is that the network speed between the server is good as this is the life line for the storage replica. We can test the replication before the build things for real.
With this test tool we bring up a small load to test the server.
Using the Diskspd with the line below.
Diskspd.exe -c1g -d600 -W5 -C5 -b8k -t2 -o2 -r -w5 –i100 –j2 E:\test
Storage replica has a great test tool report. So with this we configure the test. Using Powershell
MD c:\temp
Test-SRTopology -SourceComputerName “Building-5” -SourceVolumeName “e:” -SourceLogVolumeName “f:” -DestinationComputerName “Building-9” -DestinationVolumeName “e:” -DestinationLogVolumeName “f:” -DurationInMinutes 30 -ResultPath c:\Temp
#set output file
$outputfile=”$Env:TEMP”
Test-SRTopology -SourceComputerName “Building-5” -SourceVolumeName “e:” -SourceLogVolumeName “f:” -DestinationComputerName “Building-9” -DestinationVolumeName “e:” -DestinationLogVolumeName “f:” -IntervalInSeconds 5 -DurationInMinutes 30 -ResultPath $outputfile
#open output file
If (Test-Path $outputFile) { Invoke-Item $outputFile\TestSrTopologyReport.html } Else { Write-Host “FAILED: Output file not found: $url” -fore red }
Write-Host “Done” -ForegroundColor Cyan
while running the Test-SRTopology with the -DurationInMinutes 30 option we also run Diskspd.
Diskspd.exe -c1g -d600 -W5 -C5 -b8k -t2 -o2 -r -w5 –i100 –j2 E:\test
It is a 1 Gb file placed on our E drive that is our Data disk for replication.
As you can see I have just one network adapter and no RDMA and in this config I hit the limit of the CPU and the network card max 4.4 Gbps not bad for a test config. (if you use a better machine in Azure Pick a Azure H-series those have RDMA
One CPU with 99% usage.
When the test is done the is a log file created in -ResultPath c:\Temp
Open the log file and detailed information is there about the test. this is why I choose 30 min duration.
Nice graph about the Data throughput, in this case not bad.
the Latency is always a issue this could change you from sync to async or more network adapters or better disks. But for now it is good.
Log Volume Free Disk Space Test: The log volume F: in Building-5 has enough free space to hold the recommended log volume size of 8GB
Log Volume Free Disk Space Test: The log volume F: in Building-9 has enough free space to hold the recommended log volume size of 8GB
Storage replica has not that much PowerShell commands
#list all the commands
get-command *sr*
Setting up the actual replica is done with a long PowerShell command
The default log size is 8GB. Depending on the results of the Test-SRTopology cmdlet, you may decide to use -LogSizeInBytes with a higher or lower value.
New-SRPartnership -SourceComputerName “Building-5” –SourceRGName rg01 -SourceVolumeName “e:” -SourceLogVolumeName “f:” -DestinationComputerName “Building-9” –DestinationRGName rg02 -DestinationVolumeName “e:” -DestinationLogVolumeName “f:”
The default log size is 8GB. Depending on the results of the Test-SRTopology cmdlet, you may decide to use -LogSizeInBytes with a higher or lower value.
New-SRPartnership -SourceComputerName “Building-5” –SourceRGName rg01 -SourceVolumeName “e:” -SourceLogVolumeName “f:” -DestinationComputerName “Building-9” –DestinationRGName rg02 -DestinationVolumeName “e:” -DestinationLogVolumeName “f:” -LogSizeInBytes 1gb
here you can see the disk setup between both servers, the active side you can access the data disk, on the passive side the disk is not accessible.
Don’t place files on the Log disk.
To get replication source and destination state, use Get-SRGroup and Get-SRPartnership
Get-SRGroup
Get-SRGroup |fl *
Get-SRPartnership
(Get-SRGroup).replicas
This is just after the creation so no data yet for the last time in sync.
New-SRPartnership -SourceComputerName “Building-5” –SourceRGName rg01 -SourceVolumeName “e:” -SourceLogVolumeName “f:” -DestinationComputerName “Building-9” –DestinationRGName rg02 -DestinationVolumeName “e:” -DestinationLogVolumeName “f:”
For troubleshooting there are some events that you can check, go to the event viewer and check for the Storage replica events.
Or check the events with PowerShell
Get-WinEvent -ProviderName Microsoft-Windows-StorageReplica -max 20
On the destination server, we can do the same or look for the events in the eventlog.
Get-WinEvent -ProviderName Microsoft-Windows-StorageReplica | Where-Object {$_.ID -eq “1215”} | fl
(Get-SRGroup).Replicas | Select-Object numofbytesremaining
There are also a lot of performance counters that can be viewed with PowerShell
Get-Counter -Counter “\Storage Replica Statistics(*)\Total Bytes Received”
Get-Counter -Counter “\Storage Replica Statistics(*)\Total Bytes Sent”
Get-Counter -Counter “\Storage Replica Statistics(*)\Avg. Network Send Latency”
Get-Counter -Counter “\Storage Replica Statistics(*)\Replication State”
Get-Counter -Counter “\Storage Replica Statistics(*)\Last Recovery Elapsed Time”
Get-Counter -Counter “\Storage Replica Partition I/O Statistics(*)\Number of times flush paused”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Flushed Recovery Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Recovery Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Flushed Replication Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Replication Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Messages Received”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Messages Sent”
Get-Counter -Counter “\Storage Replica Partition I/O Statistics(*)\Avg. App Write Latency”
Get-Counter -Counter “\Storage Replica Partition I/O Statistics(*)\Avg. App Read Latency”
Get-Counter -Counter “\Storage Replica Statistics(*)\Target RPO”
Get-Counter -Counter “\Storage Replica Statistics(*)\Current RPO”
Get-Counter -Counter “\Storage Replica Statistics(*)\Avg. Log Queue Length”
Get-Counter -Counter “\Storage Replica Statistics(*)\Current Log Queue Length”
Get-Counter -Counter “\Storage Replica Statistics(*)\Total Bytes Received”
Get-Counter -Counter “\Storage Replica Statistics(*)\Total Bytes Sent”
Get-Counter -Counter “\Storage Replica Statistics(*)\Avg. Network Send Latency”
Get-Counter -Counter “\Storage Replica Statistics(*)\Replication State”
Get-Counter -Counter “\Storage Replica Statistics(*)\Avg. Message Round Trip Latency”
Get-Counter -Counter “\Storage Replica Statistics(*)\Last Recovery Elapsed Time”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Flushed Recovery Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Recovery Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Flushed Replication Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Replication Transactions”
Get-Counter -Counter “\Storage Replica Statistics(*)\Max Log Sequence Number”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Messages Received”
Get-Counter -Counter “\Storage Replica Statistics(*)\Number of Messages Sent”
these counters look like this
To remove the Replication we run the following command :
Get-SRPartnership Get-SRPartnership | Remove-SRPartnership Get-SRGroup | Remove-SRGroup
Or change the direction of the replication just run the PowerShell command
#move the replication direction from one site, use the
Set-SRPartnership -NewSourceComputerName “Building-9” -SourceRGName rg02 -DestinationComputerName “Building-5” -DestinationRGName rg01
Why not use Windows Admin Center ?
But all this PowerShell my fear you on using this. Good news than when using Windows Admin Center
Windows Admin Center is a locally deployed, browser-based app for managing servers, clusters, hyper-converged infrastructure, and Windows 10 PCs. It comes at no additional cost beyond Windows and is ready to use in production.
Get it here
When opening the Source Storage Replica server you will see a quick over view of you configuration
Easy switch replication direction.
Notifications on the preformed actions
With an overview of the current configuration.
But the best part of Windows Admin Center is creating a new Replica. I removed the old replica and create a new one with the WAC.
Fill in the source and destination and your done.
With the Admin center you got a GUI wrapper for creating the Storage replica, No PowerShell needed
So removing the replication or in case one server is dead.
Normaly you would do
Get-SRPartnership | Remove-SRPartnership –confirm:$false
this removes the replication and both locations will show the Data.
But if source server is no longer there this will not work
Remove-SRPartnership –Name RG02 -IgnoreRemovalFailure so that it breaks the partnership completely
Remove-SRPartnership [[-SourceComputerName] <String>] [-SourceRGName] <String> [-DestinationComputerName] <String> [-DestinationRGName] <String> [-IgnoreRemovalFailure] [-Force] [-CimSession <CimSession[]>] [-ThrottleLimit <Int32>] [-AsJob] [-WhatIf] [-Confirm] [<CommonParameters>]
here is the source link
https://docs.microsoft.com/en-us/powershell/module/storagereplica/remove-srpartnership?view=win10-ps
Clear-SRMetadata Removes unreferenced Storage Replica metadata.
There are more options in Windows Admin Center that could be useful to you just try it.
And if you want to use file replication to Azure take a look at the Azure File Sync https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction
https://robertsmit.wordpress.com/2017/09/28/step-by-step-azure-file-sync-on-premises-file-servers-to-azure-files-storage-sync-service-afs-cloud-msignite/
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Robert Smit MVP Linkedin profile
Google : Robert Smit MVP profile
Like this:
Like Loading...
