Troubleshooting with timestamps and other time fields

Overview

In parsed logs, there are time-related fields that are relevant when reviewing the data in Report Builder and help clarify the sequence of events in log shipping. We recommend checking these fields if you believe that your logs are delayed or you are trying to understand the sequence of events around certain activity.

The time related fields include the following:

• timestamp:
  • In the majority of logs, this is the time that the log file was generated on the device.
  • If original_timestamp is also populated, timestamp reflects the time that Blumira's infrastructure received the log.
• original_timestamp:
  • Important: This field is only ever populated when logs are delayed and is the time the log was initially created on the device.
  • We use original_timestamp in addition to timestamp for delayed logs so that our infrastructure can properly process the log and account for the delay.
• read_time:
  • For Cloud Connector integrations, this is the time that Blumira received the log.
  • For sensor integrations, this is the time that the sensor received the log.

Troubleshooting with time fields

The standard flow of a log is as follows:

device (timestamp)  >  the sensor (read_time)  >  Blumira infrastructure (timestamp_parsed)

A delay between the device's timestamp and the sensor's read_time usually indicates that the device had trouble sending the log to the sensor.

If timestamp or read_time are the same and the delay is for the timestamp_parsed then the delay was between the sensor and Blumira's infrastructure.

If original_timestamp is not null, that means timestamp was overwritten.

Time fields and troubleshooting Azure Event Hub logs

Logs generated from the Azure Event Hub include several fields that can help you to troubleshoot delays and understand processing that occurs on Microsoft's side prior to reaching Blumira's infrastructure.

• creation_time:

  • This is the time that the event actually occurred. Azure sometimes is delayed, whether in processing your log or in receiving your log from a third-party service or integration. The creation_time is the time of the event before it made it to Azure's log processing.

  • This value is different from the timestamp field in Blumira, which is the time of the log message as it appears in the raw unprocessed Azure log. The timestamp field tells us that Azure processed and shipped the log from their systems through the event hub.

  • Normally, the disparity between the creation_time and the timestamp is negligible, but depending on your tenant or tier, it can be delayed significantly. Microsoft does not have an official SLA for this. If you think a log is delayed, it is important to first compare these two fields to determine if there was a delay on Microsoft's side.

• read_time:

  • This is the time that Blumira saw the event from the source. The read_time should never have a huge disparity than that of the timestamp field.

  • If you notice a significant disparity between these times, you should check your event hub to see if it is running out of resources or hitting a resource cap.

• correlation_id:

  • In Azure, events can be related as part of a chain of event logs. For instance, you have a user sign in to the portal to perform a Graph API query. These events are part of the same chain, so they will have the same correlation_id. Searching by this ID can be helpful in an investigation where you are trying to find related events.

  • You can also filter by this in the Report Builder and in the Azure portal when looking through activity or Entra ID logs.

• operation_name:

  • This field is the actual operation or action taken. For example, if a user's license has been changed, the operation_name could be "Change user license."

  • Knowing what operations are generally being taken could help with investigations and also allow for searching for logs in the Report Builder within Blumira or the Azure portal.

• info and additional_fields:

  • The info field is a longer array type field that usually contains helpful information like which properties were changed for a given operation or what methods were used for MFA authentications. Refer to the info field whenever you want to see additional information about a given log.

  • There is also an additional_fields field that may also contain more information about a given log.