The Audit Log Wall of Shame

The List

Vendor Gating Mechanism Log Issues
1Password Higher Tier (Business+)
  • Does not use UTC or standardized time for timestamps
  • Lack of documentation on what event types represent
  • Inconsistent language and meaning for event types in Usage Logs and the Admin Activity Logs
  • All logs do not include the name of the object and which vault was accessed
  • Major audit logging difference between versions above and below 8.x.x.
Atlassian Cloud Higher Tier (Standard+)
  • None noted at this time.
AWS Cloudtrail Usage Based Billing
  • Nested unordered arrays that are inconsistently present in the same event IDs
  • Unknown log delivery delays
  • Additional expense for specific event types, such as S3 data events
  • Lack of metadata surrounding specific event types, such as failed authentication attempts to the console
  • Not all events are logged, which are demonstrated by exploits found by attackers
Datadog Usage Based Billing
  • 2% of the monthly spend, which is a lot comparing with fixed contract add in or even free
DocuSign Additional SKU
  • Collection: Small page limits
GitHub Tiered quality
  • Logs don't link to company email
  • Standard logs don't include IPs
  • Does not share all user activity logging with the enterprise organization
  • Logged changes don't include both the new & old values
Gitlab Higher Tier (Premium+)
  • Forces you to pay for the highest tier to stream events
  • Does not include audit events for project settings, group settings, or deployment approval activity.
  • The timezone used differs based on where you view audit logs (local time vs. UTC logged)
Terraform Cloud Higher Tier (Plus/Enterprise only)
  • Lacking audit data (Requests: source IP, user-agent. Changes: previous and current values)
  • Lacking response documentation (Possible resource values, meta KVs)
  • No documentation of expected event types within Audit Trail logging
  • Audit Trail logs and application logs are omitted together.
Hugging Face Higher Tier (Enterprise Hub only)
  • No API collection
JAMF PRO Higher Tier (Enterprise+) & Additional SKUs (Compliance Reporter)
  • Can only export logs via the UI, in batches of 500 messages per export.
  • If you don't want to use the UI, you have to manually poll each machine in your environment to get the machines audit logs.
  • JAMF Compliance Reporter has better audit logs. (Not part of the base JAMF install)
  • Admin audit log events extremely sparse for the events that have them.
LastPass Higher Tier (Business+)
  • Poor support for Enterprise API to pull audit logs
  • Lack of external documentation on what event types are available and what they represent
  • All logs do not include the name of the object and which vault was accessed
  • No log events when TOTP codes are accessed
  • No log events when items are Moved or Cloned from an enterprise shared folder to a personal folder
  • Lastpass CLI allows one to view EVERY secret in all shared folders (via `lpass show --expand-multi --basic-regexp '.*' --json`) without generating any events
  • Undocumented API behavior where HTTP status code 429 is returned and data is returned in CSV format
Notion Higher Tier (Enterprise only)
  • No API collection
Salesforce Additonal SKU
  • 24 hour delay in log generation
  • Output to CSV via SOQL API is not user friendly
Slack Higher Tier (Enterprise Grid only)
  • None noted at this time
Stripe No documentation
  • Logs only a subset of activity, no administrative audit logging
  • Lack of documentation on what security logging is available
Tines Higher Tier (Paid+)
Zendesk Higher Tier (Enterprise only)
  • No issues noted at this time.

What is an audit log?

DataDog provides good context on what qualifies as an audit vs. system log.

The difference between audit logs and system logs (e.g., error logs, operational logs, etc.) is the information they contain, their purpose, and their immutability. Whereas system logs are designed to help developers troubleshoot errors, audit logs help organizations document a historical record of activity for compliance purposes and other business policy enforcement.

Why does this exist?

Audit logs are a core resource for any defensive security teams to successfully detect, investigate, or find behavioral patterns related to their organization. These audit logs are most commonly generated by external parties, such as SaaS vendors, hardware vendors, or on-premise solution based companies. As noted by CISA’s 2024 “Secure by Design” pledge, it’s imperative for organizations to provide these in the base-tier of products, with sane storage limits of the telemetry provided at no extra cost.

Audit logs designed with security professionals in mind are critical to threat detection engineering and incident response. Considering their significance, vendors should be upheld to following core audit log practices.

The practices might fall into the buckets of:

  • Data collection & its technical implementation
  • Log quality & consistency
  • Log content & documentation

Let’s try to answer the question, how does a security engineer judge what qualifies as a “good” audit log? What do we need as professionals to effectively use these logs?

Here is an attempt to define an answer.

What can be added to this list?

Any vendor can be added to the list if they charge a premium for audit log data.

The purpose is to push those building solutions to think of the security engineering customer and give us the information to better protect our organizations.

For example, added sources may only provide audit logs to high paying enterprise tiers (Zendesk), provide higher quality logs with increased pricing tiers (GitHub), or have audit logs that are a separate package from even enterprise tiers (Salesforce).

Audit Log Quality

Improving security engineers experience with audit logs spans across: the log content, how engineers can collect them.

Log Content

  • Event types cover all actions taken in the system and include critical fields, such as source ip address.
  • Audit logs have external facing documentation on event types.
  • Logs contain enough information to attribute activity to a user within the platform.
  • The ability to get detailed audit logs is part of the core product or reasonably priced.

Log Collection

  • The ability exists to stream logs to a cloud storage or SIEM provider (such as logpush to S3). Otherwise the API to self-retrieve logs is straightforward, documented and allows engineers to easily retrieve their logs.
  • Log collection makes it possible for event IDs to be sorted and straightforward in order to not miss log events or get duplicate events in the pipeline.
  • There is good log formatting and data structure choice, making it easy to parse logs once they are retrieved.

Quality & Consistency

  • There is log consistency across product versioning and operating systems, including when pulling the logs. Backwards compatibility introduced when needed.
  • Low rate of log quality related incidents. Logs are reliable and can be taken as a source of truth.
  • There is limited latency between when an action occurs and when the log event is available.

What are some examples of qualities that make a log “bad”?

  • Lack of information to attribute activity to a user or IP address.
  • Poor formatting and structure that makes it difficult to access required information.
  • Ex: Unordered arrays inside of nested JSON
  • Ex: Random dicts inside of variable length arrays so the same event may have mixed lengths.
  • Different formats depending on which internal team built each event type.
  • Inconsistent event type definitions based on how a user is accessing the system.
  • Inconsistent formats and naming conventions that differ if you pull it via API or view it in the UI.
  • Lack of correlation indicators between two related log events.
  • Consistency in naming throughout logs (choosing to use ip_address OR source_ip)

So how do you write good logs?

This is a question I pose to the greater security community, this framework is open to suggestions and edits via the Github repository.

References

  1. Creating Audit Logs for Security Professionsals
  2. The SSO Wall of Shame
  3. Elastic Common Schema, Open Source Logging Fields
  4. OCSF
  5. CISA Secure By Design Pledge