While working with a customer we were leveraging Custom logs in Log Analytics to pull in and dashboard\workbook a .NET application they built and it’s own unique diagnostic logs on a server. The application was a data pull mechanism that loaded jobs from one system to another. We deployed and configured the Microsoft Monitoring Agent to the server and configured Custom Logs in Log Analytics. It normally takes 15 minutes or so for the first logs and events to come into the Custom log. However they never did. In troubleshooting we went to the Server Event Viewer and Operations Manager and found several 4502 errors pertaining to the Custom log upload.
If you are ever troubleshooting client side Microsoft Monitoring Agent (MMA) Issues this event log can be a gold mine.
We checked and the network path from server to Log analytics was clear the Firewall it passed through was able to packet capture and actually see 3 way handshake and data transmission channel open and then retry a few times. So back to the server and client software. Within the OS it turned out we needed to update a registry setting to ensure proper TLS protocols were being used in the Microsoft Monitoring Agents communication.
To fix we ran in Admin mode of PowerShell :
New-ItemProperty -path 'HKLM:\SOFTWARE\Microsoft.NetFramework\v4.0.30319' -name 'SchUseStrongCrypto' -value '1' -PropertyType 'DWord' -Force | Out-Null
New-ItemProperty -path 'HKLM:\SOFTWARE\WOW6432Node\Microsoft.NetFramework\v4.0.30319' -name 'SchUseStrongCrypto' -value '1' -PropertyType 'DWord' -Force | Out-Null
Log Name: Operations Manager Source: HealthService Date: 4/23/2020 3:30:16 PM Event ID: ** 4502** Task Category: None Level: Error Keywords: Classic User: N/A Computer: AServer.contoso.com Description: A module of type "Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction" reported an exception Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.FileUploadException: Unable to get blob container for CustomLog from https://0470718c-bb05-4eba-a66d-c249e934c655.ods.opinsights.azure.com/ContainerService.svc. Will keep trying according to the specified policy. ---> System.Net.WebException: **The underlying connection was closed: An unexpected error occurred on a receive. --->** System.ComponentModel.Win32Exception: **The client and server cannot communicate, because they do not possess a common algorithm** at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface SecModule, String package, CredentialUse intent, SecureCredential scc) at System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse credUsage, SecureCredential& secureCredential) at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint) at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output) at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count) at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback asyncCallback, Object asyncState) at System.Net.ConnectStream.WriteHeaders(Boolean async) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) at Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction.GetRequestStreamCallback(IAsyncResult asynchronousResult) --- End of inner exception stack trace --- which was running as part of rule "Microsoft.IntelligencePacks.CustomLogUpload.UploadCustomLog" running for instance "" with id:"{997E3931-D7CD-5D40-E2BB-4279FDCBC850}" in management group "AOI-0470718c-bb05-4eba-a66d-c249e934c655". Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="HealthService" /> <EventID Qualifiers="49152">4502</EventID> <Level>2</Level> <Task>0</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2020-04-23T20:30:16.091338100Z" /> <EventRecordID>77015</EventRecordID> <Channel>Operations Manager</Channel> <Computer>AServer.contoso.com</Computer> <Security /> </System> <EventData> <Data>AOI-0470718c-bb05-4eba-a66d-c249e934c655</Data> <Data>Microsoft.IntelligencePacks.CustomLogUpload.UploadCustomLog</Data> <Data> </Data> <Data>{997E3931-D7CD-5D40-E2BB-4279FDCBC850}</Data> <Data>Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction</Data> <Data>Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.FileUploadException: Unable to get blob container for CustomLog from https://0470718c-bb05-4eba-a66d-c249e934c655.ods.opinsights.azure.com/ContainerService.svc. Will keep trying according to the specified policy. ---> System.Net.WebException: **The underlying connection was closed: An unexpected error occurred on a receive**. ---> System.ComponentModel.Win32Exception: **The client and server cannot communicate, because they do not possess a common algorithm** at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface SecModule, String package, CredentialUse intent, SecureCredential scc) at System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse credUsage, SecureCredential& secureCredential) at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint) at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count, Byte[]& output) at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count) at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest) at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result) at System.Net.TlsStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback asyncCallback, Object asyncState) at System.Net.ConnectStream.WriteHeaders(Boolean async) --- End of inner exception stack trace --- at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult, TransportContext& context) at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult) at **Microsoft.EnterpriseManagement.Mom.Modules.CloudFileUpload.CloudFileUploadWriteAction.GetRequestStreamCallback(IAsyncResult asynchronousResult)** --- End of inner exception stack trace ---</Data> </EventData> </Event>