The List

Vendor	Gating Mechanism	Log Issues
1Password	Higher Tier (Business+)	Does not use UTC or standardized time for timestamps Lack of documentation on what event types represent Inconsistent language and meaning for event types in Usage Logs and the Admin Activity Logs All logs do not include the name of the object and which vault was accessed Major audit logging difference between versions above and below 8.x.x.
Atlassian Cloud	Higher Tier (Standard+)	None noted at this time.
AWS Cloudtrail	Usage Based Billing	Nested unordered arrays that are inconsistently present in the same event IDs Unknown log delivery delays Additional expense for specific event types, such as S3 data events Lack of metadata surrounding specific event types, such as failed authentication attempts to the console Not all events are logged, which are demonstrated by exploits found by attackers
Datadog	Usage Based Billing	2% of the monthly spend, which is a lot comparing with fixed contract add in or even free
DocuSign	Additional SKU	Collection: Small page limits
GitHub	Tiered quality	Logs don't link to company email Standard logs don't include IPs Does not share all user activity logging with the enterprise organization Logged changes don't include both the new & old values
Gitlab	Higher Tier (Premium+)	Forces you to pay for the highest tier to stream events Does not include audit events for project settings, group settings, or deployment approval activity. The timezone used differs based on where you view audit logs (local time vs. UTC logged)
Terraform Cloud	Higher Tier (Plus/Enterprise only)	Lacking audit data (Requests: source IP, user-agent. Changes: previous and current values) Lacking response documentation (Possible resource values, meta KVs) No documentation of expected event types within Audit Trail logging Audit Trail logs and application logs are omitted together.
JAMF PRO	Higher Tier (Enterprise+) & Additional SKUs (Compliance Reporter)	Can only export logs via the UI, in batches of 500 messages per export. If you don't want to use the UI, you have to manually poll each machine in your environment to get the machines audit logs. JAMF Compliance Reporter has better audit logs. (Not part of the base JAMF install) Admin audit log events extremely sparse for the events that have them.
LastPass	Higher Tier (Business+)	Poor support for Enterprise API to pull audit logs Lack of external documentation on what event types are available and what they represent All logs do not include the name of the object and which vault was accessed No log events when TOTP codes are accessed No log events when items are Moved or Cloned from an enterprise shared folder to a personal folder Lastpass CLI allows one to view EVERY secret in all shared folders (via `lpass show --expand-multi --basic-regexp '.*' --json`) without generating any events Undocumented API behavior where HTTP status code 429 is returned and data is returned in CSV format
Notion	Higher Tier (Enterprise only)	No API collection
Salesforce	Additonal SKU	24 hour delay in log generation Output to CSV via SOQL API is not user friendly
Slack	Higher Tier (Enterprise Grid only)	None noted at this time
Stripe	No documentation	Logs only a subset of activity, no administrative audit logging Lack of documentation on what security logging is available
Zendesk	Higher Tier (Enterprise only)	No issues noted at this time.

What is an audit log?

DataDog provides good context on what qualifies as an audit vs. system log.

The difference between audit logs and system logs (e.g., error logs, operational logs, etc.) is the information they contain, their purpose, and their immutability. Whereas system logs are designed to help developers troubleshoot errors, audit logs help organizations document a historical record of activity for compliance purposes and other business policy enforcement.

Why does this exist?

Audit logs are a core resource for any defensive security teams to successfully detect, investigate, or find behavioral patterns related to their organization. These audit logs are most commonly generated by external parties, such as SaaS vendors, hardware vendors, on-premise solution based companies etc.

Audit logs designed with security professionals in mind are critical to threat detection engineering and incident response. Considering their significance, vendors should be upheld to following core audit log practices.

The practices might fall into the buckets of:

• Data collection & its technical implementation
• Log quality & consistency
• Log content & documentation

Let’s try to answer the question, how does a security engineer judge what qualifies as a “good” audit log? What do we need as professionals to effectively use these logs?

Here is an attempt to define an answer.

What can be added to this list?

Any vendor can be added to the list if they charge a premium for audit log data.

The purpose is to push those building solutions to think of the security engineering customer and give us the information to better protect our organizations.

For example, added sources may only provide audit logs to high paying enterprise tiers (Zendesk), provide higher quality logs with increased pricing tiers (GitHub), or have audit logs that are a separate package from even enterprise tiers (Salesforce).

Audit Log Quality

Improving security engineers experience with audit logs spans across: the log content, how engineers can collect them.

Log Content

• Event types cover all actions taken in the system and include critical fields, such as source ip address.
• Audit logs have external facing documentation on event types.
• Logs contain enough information to attribute activity to a user within the platform.
• The ability to get detailed audit logs is part of the core product or reasonably priced.

Log Collection

• The ability exists to stream logs to a cloud storage or SIEM provider (such as logpush to S3). Otherwise the API to self-retrieve logs is straightforward, documented and allows engineers to easily retrieve their logs.
• Log collection makes it possible for event IDs to be sorted and straightforward in order to not miss log events or get duplicate events in the pipeline.
• There is good log formatting and data structure choice, making it easy to parse logs once they are retrieved.

Quality & Consistency

• There is log consistency across product versioning and operating systems, including when pulling the logs. Backwards compatibility introduced when needed.
• Low rate of log quality related incidents. Logs are reliable and can be taken as a source of truth.
• There is limited latency between when an action occurs and when the log event is available.

What are some examples of qualities that make a log “bad”?

• Lack of information to attribute activity to a user or IP address.
• Poor formatting and structure that makes it difficult to access required information.
• Ex: Unordered arrays inside of nested JSON
• Ex: Random dicts inside of variable length arrays so the same event may have mixed lengths.
• Different formats depending on which internal team built each event type.
• Inconsistent event type definitions based on how a user is accessing the system.
• Inconsistent formats and naming conventions that differ if you pull it via API or view it in the UI.
• Lack of correlation indicators between two related log events.
• Consistency in naming throughout logs (choosing to use ip_address OR source_ip)

So how do you write good logs?

This is a question I pose to the greater security community, this framework is open to suggestions and edits via the Github repository.

References

1. Creating Audit Logs for Security Professionsals
2. The SSO Wall of Shame
3. Elastic Common Schema, Open Source Logging Fields