Azure Arc-enabled servers enables you to manage your Windows and Linux physical servers and virtual machines hosted outside of Azure, on your corporate network, or other cloud provider. This management experience is designed to be consistent with how you manage native Azure virtual machines. When a hybrid machine is connected to Azure, it becomes a connected machine and is treated as a resource in Azure. Each connected machine has a Resource ID enabling the machine to be included in a resource group. Now you can benefit from standard Azure constructs, such as Azure Policy and applying tags.
When running Azure Arc for some time and suddenly the response stopped you need to dig a bit deeper into the how things are working instead of just kicking off an MSI and the issue is still not fixed.
This is all test So it may look different in your site.just to say so.
Here I have my two servers managed by Arc
As you can see “Something went wrong while getting your resources. Please try again later.”
yes let me get more info about this as currently I know nothing about the error.
So It is all OK according to the Azure troubleshooter and still it doesn’t work
Let me click around and see if there is and error ( I could see the local event log of the server but that’s no fun Who uses this ? post some comments in the blog post) Eventlogs are extremely helpful on finding issues or hidden issue’s Often people for get to look at his and see the problem right there. and yes it needs to be fixed also.
Will that be the issue ? checking already running the latest version, so what is this error or did it go wrong when updating the agent, well I did skip patching for some time on these servers and upgraded these to Windows server 2022
Let me check the agent version, well the latest version for now..
How is this Azure arc be configured anyway, there is no console other than in azure and an MSI with an agent,
let me check the configuration of this and see if I can find something there.
C:\ProgramData\GuestConfig
Perfect lots of log files and a config let me check this all
time="2021-09-01T16:32:17+02:00" level=error msg="Could not acquire token from cert: FromAssertion(): http call(https://login.windows.net/-d391a79950b1/oauth2/v2.0/token)(POST) error: reply status code was 401:\n{\"error\":\"invalid_client\",\"error_description\":\"AADSTS700027: Client assertion contains an invalid signature. [Reason – The key used is expired., Thumbprint of key used by client: ‘C2FA453DD43C16E584868C1C762DC91EBEC63232’, Found key ‘Start=11/12/2019 15:45:00, End=02/10/2020 15:45:00’, Please visit the Azure Portal, Graph Explorer or directly use MS Graph to see configured keys for app Id ‘a16df9d0-f012-45ae-8a92-1d0ad72e045e’. Review the documentation at https://docs.microsoft.com/en-us/graph/deployments to determine the corresponding service endpoint and https://docs.microsoft.com/en-us/graph/api/application-get?view=graph-rest-1.0&tabs=http to build a query request URL, such as ‘https://graph.microsoft.com/beta/applications/].\\r\\nTrace ID: 932-2ba8-4098-813e-05a2900\\r\\nCorrelation ID: 66a070fe-6ae4-4a25-ad3f-\\r\\nTimestamp: 2021-09-01 14:32:07Z\",\"error_codes\":[700027],\"timestamp\":\"2021-09-01 14:32:07Z\",\"trace_id\":\"932e7194-2ba8-4098-813e-343df05a2900\",\"correlation_id\":\"-4a25-ad3f-160f98c9fd9e\",\"error_uri\":\"https://login.windows.net/error?code=700027\"}"
Seeing the Config and also see the issue here — Client assertion contains an invalid signature. [Reason – The key used is expired–
As I did not update the agent the certificate got expired make sense.
But the device has already the new agent So reconnect ? but how ?
Looking at the Config I see all the details how the agent is been registered and the resource group etc
C:\ProgramData\AzureConnectedMachineAgent\Config
agentconfig.json
{"subscriptionId":"f34","resourceGroup":"AzureBackupRG_westeurope_1","resourceName":"Hyperv1201","tenantId":"0b1","location":"westus2","vmId":"9659193c-f4d8-4a77-b8f9baad507ce9a9","certificateThumbprint":"c2fa453dd43c16e584868c1c762dc91ebec63232","clientId":"0-f012-45ae-8a92-1045e"}
Let me open powershell and maybe I got more details. and reactivate the Agent
With the azcmagent command you can get more details.
let me get all the logs
azcmagent logs
now we have all the logs in a zip file this could be handy for a next time.
As I reconfigure the agent with the following command
& "$env:ProgramFiles\AzureConnectedMachineAgent\azcmagent.exe" connect –resource-group "AzureBackupRG_westeurope_1" –tenant-id "your tenant id" –location "westus2" –subscription-id "errryh934" –verbose
With the reconnect we need to log in again and all goes well
But in the logging there is suddenly another error
When looking here I see there is an Azure Policy that demands a TAG and this is currently not available on the resource group So I Can’t onboard my Azure Arc server.
Thought this was about an Agent that has an expired Certificate.
Seems there is a Azure policy that is blocking as the hyperv1201 has no tags set the mvpdc02 has only a tag set.
After a quick change I rerun the command line and it worked perfectly and it showed up in the console again.
& "$env:ProgramFiles\AzureConnectedMachineAgent\azcmagent.exe" connect –resource-group "MVPRSG-Azure-Arc" –tenant-id "3078684f-d143-440a-ae40-d391a79950b1" –location "West US 2" –subscription-id "df1e2f32-7adf-48f6-b969-f02376152934" –verbose
Starting client connection on: \\\\.\\pipe\\himds"
time="2021-09-01T17:12:53+02:00" level=debug msg="Awaiting status message from agent…"
time="2021-09-01T17:12:53+02:00" level=debug msg="Status Message received"
As I have a second machine with the same issue I removed the machine directly in the arc portal and rerun the registration as the agent was also already installed. (this would be the quick fix for this)
Perfect reconnecting and waiting for the Agent.
Now I can look at the Azure Arc Insights again.
Flickr Tags: Windows Server 2022,CloudOS
Follow Me on Twitter @ClusterMVP
Follow My blog https://robertsmit.wordpress.com
Linkedin Profile Http://nl.linkedin.com/in/robertsmit
Share this:
Robert Smit MVP Blog 