We left off in the last article Part 3 making a call to Log Analytics to find the latest date time in the IDCSAudit_CL table to use later and pass it in your Oracle IDCS AuditEvents call.
In this article you will now take the authorization and latest date time values and make a call to the Oracle IDCS Audit Events API. You will also work through pagination of the results.
You will first need to load some additional PowerShell functions to allow your script to POST data to Azure Sentinel. Azure Sentinel's Log analytics workspace contains a HTTP Data Collector API that allows you to send and ingest data. What is even better is the precise examples they give for PowerShell which you can use.
We won't spend much going over this as it is documented here:
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api#powershell-sample
# HTTP DATA Collector Functions # Create the function to create the authorization signature Function Build-Signature ($customerId, $sharedKey, $date, $contentLength, $method, $contentType, $resource) { $xHeaders = "x-ms-date:" + $date $stringToHash = $method + "`n" + $contentLength + "`n" + $contentType + "`n" + $xHeaders + "`n" + $resource $bytesToHash = [Text.Encoding]::UTF8.GetBytes($stringToHash) $keyBytes = [Convert]::FromBase64String($sharedKey) $sha256 = New-Object System.Security.Cryptography.HMACSHA256 $sha256.Key = $keyBytes $calculatedHash = $sha256.ComputeHash($bytesToHash) $encodedHash = [Convert]::ToBase64String($calculatedHash) $authorization = 'SharedKey {0}:{1}' -f $customerId,$encodedHash return $authorization }
# Create the function to create and post the request Function Post-LogAnalyticsData($customerId, $sharedKey, $body, $logType) { $method = "POST" $contentType = "application/json" $resource = "/api/logs" $rfc1123date = [DateTime]::UtcNow.ToString("r") $contentLength = $body.Length $signature = Build-Signature ` -customerId $customerId ` -sharedKey $sharedKey ` -date $rfc1123date ` -contentLength $contentLength ` -method $method ` -contentType $contentType ` -resource $resource $uri = "https://" + $customerId + ".ods.opinsights.azure.com" + $resource + "?api-version=2016-04-01" $headers = @{ "Authorization" = $signature; "Log-Type" = $logType; "x-ms-date" = $rfc1123date; "time-generated-field" = $TimeStampField; } $response = Invoke-WebRequest -Uri $uri -Method $method -ContentType $contentType -Headers $headers -Body $body -UseBasicParsing return $response.StatusCode }
Now that you have the PowerShell functions to be used to POST data to Azure Sentinel you can call Oracle IDCS Audit Events.
To begin with you will need to form the headers to pass to the IDCS API Audit Events endpoint. In part 1 and 2 you generated $signature bearer token to use for authorization.
# Invoke to OCI AuditEvents using start time from tableresult $headers = @{ 'Authorization' = 'Bearer ' + $sig 'Content-Type' = 'application/scim+json' }
Next you will need to format the URI you are going to pass in a Invoke-WebRequest
# Create the URI needed to invoke AuditEvents in OCI | BASE URI for testing $audituri = "https://" + $IDCS + "/admin/v1/AuditEvents" + "?sortBy=timestamp&sortOrder=descending&filter=timestamp ge " + '"' + $starttime + '"' + " and timestamp le " + '"' + $endtime + '"'
You will get a result like:
https://idcs-XXXXXXXXXXXX.identity.oraclecloud.com/admin/v1/AuditEvents?sortBy=timestamp&sortOrder=descending&filter=timestamp ge "2020-07-22T10:33:31Z" and timestamp le "2020-07-23T22:33:43Z"
With the headers and the the constructed URI you can make your first call with Invoke-WebRequest.
$auditresults = (Invoke-WebRequest -Uri $audituri -Method GET -Headers $headers).Content | ConvertFrom-Json
Again you are wrapping the cmdlet () so that you can . source the Content that comes back in the variable $auditresults. In addition piping the content and converting from JSON into PS Table.
Wait a sec ... In our auth token request in Part 2 you used Invoke-RestMethod, so why change to Invoke-WebRequest ? In a lot of APIs that you may call the event results will be too many to return and pagination will come into play. Pagination can occur in the Header response sometime in the form of NextPage: https://xxxxx/xxxxxx?resultspage=2. Invoke-WebRequest will allow you to capture and replay the response Header returned results where Invoke-RestMethod will not capture the response header results from the API Endpoint. You could then pass .header.nextpage in a loop to continue collecting the results until finished.
In Oracle IDCS Audit Events API however the Header response next page concept is not used.
PS C:\Users\> $auditresults = Invoke-RestMethod -Uri $audituri -Method GET -Headers $headers
PS C:\Users\> $auditresults.Headers
PS C:\Users\> $auditresults = Invoke-WebRequest -Uri $audituri -Method GET -Headers $headers
PS C:\Users\> $auditresults.Headers
Key Value
Cache-Control {no-store, must-revalidate, no-cache}
Date {Fri, 24 Jul 2020 14:39:14 GMT}
Pragma {no-cache}
Server {Oracle, Identity, Cloud, Service}
Strict-Transport-Security {max-age=315360000; includeSubdomains}
Vary {Accept-Encoding}
Via {1.1 net-idcs-config}
X-Content-Type-Options {nosniff}
X-Oracle-Dms-Ecid {bXkTE0YsI00000000}
X-Oracle-Dms-Rid {0:1}
X-XSS-Protection {1; mode=block}
Transfer-Encoding {chunked}
Content-Type {application/json; charset=utf-8}
Expires {Sat, 01 Jan 2000 00:00:00 GMT}
In the case of Oracle IDCS the results by default will come 50 at a time. You can increase the results returned up to 1000 if needed by forming an additional parameter in the URI call by:
&count=1000
It is always helpful to review the API documentation on how results and pagination will occur so that you can build a sound script framework to handle multiple requests needed at the endpoint. In the Oracle ICDS case when you make a Audit Events call you get back some key information you can use.
$auditresults
schemas : {urn:ietf:params:scim:api:messages:2.0:ListResponse}
totalResults : 447
Resources : {}
startIndex : 1
itemsPerPage : 50
The way this Audit Events API Works is that the results come in a batch of 50 by default you see that by the itemsPerPage. You also receive a totalResults in this case 447. Finally a startIndex of 1 which shows where you are starting off in the totalResults to show the current 50.
It is that startIndex that also can be formed in the URI and you could start on the next 50 by using ?startIndex=51.
Good stuff here Oracle is kindly giving you what you need to programmatically ensure you get all results in the event there are more than 50 audit events in the given time period you execute the Audit Events API query.
You will want to create a conditional check to ensure that the results are 50 or less than to just execute the POST of the audit event results to Azure Sentinel and exit the script. Else if the results are more than 50 or really more than the itemsPerPage (*remember you can alter items per page to 1000) then you can use a Do While loop to cover all your results.
To start with you can define a variable $loopend to capture the value of totalResults (447). In addition set a variable $counter = 1.
# define loop end upper boundary of pagination in this call $loopend = $auditresults.totalResults # define the index start at result 1 for loop $counter = 1
Next you can use the If (check something){do something}
Here you can check if total results (447) are less than or equal to itemsPerPage (50) then define a $jsonbody variable that are the actually value of Events them selves stored in $auditresults.Resources | and convert them back to JSON.
*The Azure Sentinel log Analytics workspace HTTP Data Connector API can only receive results and logs in JSON format to ingest.
In our example 447 <= 50 is False and will now pass on to the Else{do something} portion.
# Conditional check are the total results less than the pagination (defualt 50 results per query, using index to start at), do we need to loop ?, default is 50 items per page but can be changed to 1000 in URI | count=1000 if ($loopend -le $auditresults.itemsPerPage) { # format the the first OCI AuditEvents API call into a JSON body $jsonbody = $auditresults.Resources | ConvertTo-Json # Use for testing uncomment below to writeout files to ensure while loop for pagination is working correctly. BE SURE TO COMMENT OUT Post-LogAnalyticsData line when tshooting. #$jsonbody | Out-File -FilePath c:\temp\jsonindex$counter.json # Send the JSONBody results to Azure Sentinel Post-LogAnalyticsData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($jsonbody)) -logType $logType } else {
Here you will construct a do {something} while (condition exists)
do { # Create the URI needed to invoke AuditEvents in OCI using the StartIndex=$counter $audituri = "https://" + $IDCS + "/admin/v1/AuditEvents?startIndex=" + $counter + "&sortBy=timestamp&sortOrder=descending&filter=timestamp ge " + '"' + $starttime + '"' + " and timestamp le " + '"' + $endtime + '"' # Invoke OCI AuditEvents API to obtain results $auditresults = (Invoke-WebRequest -Uri $audituri -Method GET -Headers $headers).Content | ConvertFrom-Json # format the the first OCI AuditEvents API call into a JSON body $jsonbody = $auditresults.Resources | ConvertTo-Json # Use for testing uncomment below to writeout files to ensure while loop for pagination is working correctly. BE SURE TO COMMENT OUT Post-LogAnalyticsData line when tshooting. #$jsonbody | Out-File -FilePath c:\temp\jsonindex$counter.json # Send the JSONBody results to Azure Sentinel Post-LogAnalyticsData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($jsonbody)) -logType $logType # update pagination counter for the next set of results (defualt is 50 unless specified in URI to count=xxxx where xxxx is max value of 1000) $counter = $counter + $auditresults.itemsPerPage # Continue the loop untill we are greater than the total AuditEvents pagination results } while ($counter -lt $loopend)
Let's examine some of the mechanisms here.
You start by changing the $audiuri to now include ?startIndex= it will look like:
$audituri = "https://" + $IDCS + "/admin/v1/AuditEvents?startIndex=" + $counter + "&sortBy=timestamp&sortOrder=descending&filter=timestamp ge " + '"' + $starttime + '"' + " and timestamp le " + '"' + $endtime + '"'
https://idcs-XXXXXX.identity.oraclecloud.com/admin/v1/AuditEvents?startIndex=1&sortBy=timestamp&sortOrder=descending&filter=timestamp ge "2020-07-17T01:24:34Z" and timestamp le "2020-07-24T13:29:05Z"
In PowerShell you are forming the URI string so that after ?startIndex= you pass a programmatic $counter variable.
The rest of the execution in the loop will be:
Getting the $auditresults
# Invoke OCI AuditEvents API to obtain results $auditresults = (Invoke-WebRequest -Uri $audituri -Method GET -Headers $headers).Content | ConvertFrom-Json
Setting a variable for $jsonbody so that the .Resources are in JSON format and all 50 audit events will POST to Azure Sentinel
# format the the first OCI AuditEvents API call into a JSON body $jsonbody = $auditresults.Resources | ConvertTo-Json
Finally invoking the PowerShell Function Post-LogAnalyticsData you defined earlier from sample code, this will take the properly formed URI, and JSON Body of results and POST them to Azure Sentinel's Log Analytics Workspace.
# Send the JSONBody results to Azure Sentinel Post-LogAnalyticsData -customerId $customerId -sharedKey $sharedKey -body ([System.Text.Encoding]::UTF8.GetBytes($jsonbody)) -logType $logType
At the end of the Do While loop you will increase by the itemsPerPage (50) so that on the next loop you start at 51 and get the next 50 results, so on and so forth.
# update pagination counter for the next set of results (defualt is 50 unless specified in URI to count=xxxx where xxxx is max value of 1000) $counter = $counter + $auditresults.itemsPerPage
Finally you will define the conditions to exit and stop running the loop. Recall that in some APIs you will be examining the response header returned for nextPage field with a URI to pass. In this case Oracle IDCS gives us the totalResults to look for when you should exit.
$counter (1,50,51,101...401) <= $loopend (447)
while ($counter -lt $loopend)
As you come towards that last URI call ?startIndex=401 and grab the .itemsPerPage (50) and add that to the counter at the very end before WHILE
$counter will then be 451 which is not less than or equal to 447 the condition is false and the DO WHILE loop will now exit.
In the last article of this series you will clone an example O365 Data connector from Azure Sentinel Github and make modifications in the ARM deployment templates, and this scripts that it can execute a 1 click deploy to Azure as a Azure Function timer triggered job.
As a reminder the complete script can be found here: https://github.com/swiftsolves-msft/PowerShell-Scripts/blob/master/Get-OIDCSAuditEvents.ps1
Sources:
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/data-collector-api#powershell-sample