CloudTrail limitations: what log forensics cannot tell you
What CloudTrail, VPC Flow Logs and S3 access logs do not record, where baselines fail, and how to write honest conclusions when evidence is missing.
TL;DR. Logs record API calls and network metadata — not intent, not payloads, not what happened inside an instance. CloudTrail does not log S3 object reads unless data events were configured; event history keeps only 90 days of management events; flow logs skip DNS to the Amazon resolver and instance-metadata traffic; S3 access logs are best effort. Detections built on baselines are blind without quiet days before the incident. Write conclusions that match coverage: "no evidence of X in the sources available, which cover Y", never "X did not happen" when X could not have been seen.
A log-driven verdict — from a SIEM, from Athena queries or from this site's analyzer — is a statement about the logs, not about the account. This post lists the gaps that come up in almost every AWS investigation, so they end up in the report instead of in a surprise six months later.
What CloudTrail does not see
| Gap | Why | What to do |
|---|---|---|
| Object-level S3 reads and writes | Data events are off by default and billed separately | Check trail selectors; use S3 server access logs if enabled |
| Lambda invocations, DynamoDB item reads | Also data events | Same |
| Anything older than 90 days without a trail | Event history limit | Trail or CloudTrail Lake for the future |
| What ran inside an instance or container | CloudTrail logs the control plane only | Host forensics, EDR, OS logs; for EKS the Kubernetes audit log (kubernetesforensics.com) |
The full user data of RunInstances | Recorded as <sensitiveDataRemoved> | Retrieve it from the instance |
| Events from services or actions not covered | Coverage varies per service | CloudTrail supported services |
| Order within a file | Events are not delivered in order (CloudTrail concepts) | Always sort by eventTime |
Delivery is also not instantaneous: CloudTrail typically delivers within about five minutes, without guarantee (how CloudTrail works). A collection made seconds after the last attacker action may miss it.
What the network logs do not see
VPC Flow Logs never record traffic to the Amazon DNS server, to the instance metadata service at 169.254.169.254, Time Sync, DHCP or ARP (limitations). So:
- DNS tunnelling through the VPC resolver is invisible (Route 53 Resolver query logs are the source for that);
- theft of instance-role credentials from IMDS leaves no flow — its trace, if any, is the credentials used elsewhere, which GuardDuty flags as
InstanceCredentialExfiltrationfindings; - S3 and other API downloads made from the internet never cross your VPC.
S3 server access logs are delivered on a best-effort basis; AWS says a record may arrive late or not at all (documentation). Good for volume and patterns, weaker as proof that one specific request did not happen.
Identity blind spots
- Federated and SSO users appear as assumed-role sessions; who the human was lives in the identity provider's logs (for Okta, see oktaforensics.com; for Microsoft Entra ID, m365forensics.com).
- Role chaining splits one actor across sessions; follow
AssumeRoleresponses andsessionContext.sessionIssuer. - AWS services acting for you show a service name in
sourceIPAddress; an attacker using a service (CloudFormation, Lambda) to act indirectly looks like that service. - Shared egress. An office NAT or a CI provider's IP range makes "new IP" detections noisy; a VPN endpoint the attacker also uses makes them blind.
- No geolocation in the logs. CloudTrail records IP addresses, not countries or ASNs. The analyzer does not enrich IPs either, so "sign-in from a new country" is not detected; GuardDuty findings carry that context when available.
Where detections fail
Rule-based detection, including the analyzer's, has predictable failure modes:
- Baselines need history. "Key used from a new IP" and "new region" compare with the first day of activity in the logs you load. Load only the incident day and they see nothing — or everything.
- Thresholds can be stayed under. 29 distinct List calls in ten minutes will not trigger a "30 in 10 minutes" rule; 99 downloads per hour will not trigger a "100 per hour" rule. A patient attacker is harder to catch than a scripted one.
- Legitimate admin looks like attack. Creating users, attaching policies, launching GPU instances are normal jobs. Detections point; people confirm.
- Unknown techniques. Rules cover known attack paths. The analyzer's rule file lists what is covered; anything else needs manual review of the raw events.
Writing honest conclusions
Tie every conclusion to coverage. A template that works:
Between [start] and [end], CloudTrail management events for accounts […] in all regions, S3 data events for buckets […], and VPC Flow Logs for VPCs […] were reviewed. S3 data events were not enabled for buckets […]; for those buckets, object access cannot be determined. No evidence of [activity] was found in the reviewed sources.
Also record: time zone (UTC), hashes of the exports, the tools and versions used, and every gap — stopped trails, missing regions, deleted log groups. The defense evasion post covers how to bound a gap created by the attacker.
Closing the gaps for next time
- A multi-region (or organization) trail with log file validation, stored in a separate, locked-down account.
- S3 data events for buckets holding sensitive data, at least for reads.
- VPC Flow Logs on every VPC, including default VPCs in regions you do not use — or a service control policy denying those regions.
- GuardDuty in every enabled region, with findings exported to S3.
- Retention that outlives your detection time.
AWS's Security Incident Response Guide covers the preparation side in depth. For the investigation itself, start from the incident response overview.