Linux does not give a SOC analyst one universal security log with identical paths on every distribution. Modern systems may use the systemd journal, traditional syslog files, the Linux Audit system, application logs, or a combination of all of them.
That makes Linux investigation partly a source-discovery problem: first learn where this host records authentication, services, scheduled tasks, and audit events; then build the timeline.
Cisco’s current 200-201 CCNACBR v1.2 training explicitly includes Linux host analysis and advanced log analysis, so analysts need to understand both the common evidence patterns and the distribution differences.
Start with the systemd journal on modern distributions
On systemd-based systems, journalctl is a central way to query logs collected by the journal.
Useful filters include time, boot, service unit, priority, and process or identifier fields. Rather than scrolling through everything, scope the query to the incident window and the service involved.
For example, authentication investigation often begins with the SSH service, while a persistence investigation may focus on an unfamiliar systemd unit.
The journal is especially useful because it can bring together service output and system context. It should not be treated as the only source, however. Traditional files under /var/log, audit records, application logs, and security-agent telemetry may contain additional evidence.
The journal can be queried by time range, boot, systemd unit, priority, process, and other structured fields, which makes it useful for reconstructing a short incident window without reading an entire text file. journalctl -b can focus on the current boot, while previous boots may be available if persistent journal storage is configured. That persistence detail matters after a reboot: a volatile journal can lose the evidence an analyst expected to review.
Treat journal output as one view of system activity, not as proof that every security-relevant event is present. Services can also write dedicated application logs, auditd records use a separate audit trail, and containers may have their own logging path. Begin with the journal because it connects many services by time, then pivot to the source that records the detail required by the hypothesis.
Know the distribution-specific authentication files
Traditional log paths vary.
On Red Hat Enterprise Linux, /var/log/secure contains security and authentication-related messages, while /var/log/cron records periodically executed tasks. Other distributions may use different files, such as /var/log/auth.log.
Do not write a detection rule that assumes one filename represents “Linux authentication” everywhere.
The stronger approach is to understand which logging stack the host uses and normalize the resulting events centrally.
For SSH, useful context can include the account, source address, authentication method, success or failure, and session lifecycle.
If key-based access is relevant, an explanation of SSH key files helps clarify why a new authorized key can matter during an investigation.
On Red Hat-derived systems, authentication and authorization messages are commonly found in /var/log/secure; Debian/Ubuntu systems commonly use /var/log/auth.log. The exact source depends on distribution, logging configuration, and whether events are being forwarded elsewhere. Analysts should confirm the host’s logging architecture before assuming a missing file means missing evidence.
Within those records, the message text matters. An SSH failure, an accepted public-key login, a sudo command, and a PAM authentication error describe different stages of access. The same account name can appear in all of them, so correlate the source address, terminal or service, authentication method, and time instead of counting usernames alone.
Authentication patterns matter more than one failed login
Internet-facing Linux systems can receive many failed SSH attempts. A single failure is usually not enough to declare an incident.
Look for patterns:
- repeated failures followed by a success
- a successful login from a new network
- use of a privileged or service account at an unusual time
- a change from password to key-based access
- logins followed immediately by privilege escalation
- simultaneous or geographically implausible sessions
The meaning depends on the environment.
A successful SSH login from a deployment system may be routine. The same login from an unmanaged external address to a server that normally accepts connections only from a bastion host is a different finding.
The Linux Audit system can preserve deeper security evidence
Linux Audit can record system calls, authentication activity, changes to watched files, and security-relevant events.
On RHEL, audit records are normally stored in /var/log/audit/audit.log. Tools such as ausearch help query the records and interpret fields.
Audit data can answer questions that ordinary service logs cannot, such as which process accessed or changed a watched security-sensitive file.
It is also useful for login analysis. Red Hat documents USER_LOGIN audit records that can show successful or failed user login activity.
The tradeoff is volume and configuration. Audit evidence exists only for the events and rules the system was configured to record.
Audit records are often multi-record events. A single action can produce SYSCALL, PATH, CWD, and other message types tied together by an audit event identifier. Reading only one line can therefore hide which executable ran, which object it touched, and which user identity was responsible. Tools such as ausearch and aureport help assemble those related records.
Red Hat documents USER_LOGIN searches as a direct way to review login activity. Audit rules can also watch sensitive files or classes of system calls, but the visibility depends on the rules that were actually active at the time. During an investigation, preserve the audit configuration along with the events so the team knows both what was logged and what could never have been logged.
Persistence often appears as an ordinary administration mechanism
Linux persistence does not require exotic malware.
An attacker with sufficient access can reuse mechanisms administrators rely on every day:
- cron or systemd timers
- systemd service units
- shell startup files
- SSH
authorized_keys - modified startup scripts
- new users or changes to privileged groups
That is why a cron job is not malicious by itself. The question is whether the job is expected, who created it, what it executes, and where the executable came from.
Likewise, a new systemd service may be legitimate software installation or persistence. Examine the unit definition, executable path, user, dependencies, creation time, and associated package or change record.
Cron is only one persistence surface. Systemd services and timers, modified shell startup files, new SSH authorized_keys entries, and altered application startup configuration can all create legitimate or malicious recurring execution. The investigative question is whether the mechanism, owner, command, and creation time fit the host’s administrative history.
For a suspicious cron job, inspect more than the schedule. Resolve the referenced executable or script, check its ownership and permissions, review recent modifications, and determine which user context will execute it. A harmless-looking schedule that launches a writable script from /tmp presents a different risk from a root-owned maintenance command in a controlled path.
File permissions and ownership provide investigation context
Linux permissions can explain whether an account was realistically able to modify a suspicious file.
Linux ownership and permissions become direct investigation evidence when the analyst needs to answer questions such as:
- Who owns the file?
- Which group can write it?
- Is the directory user-writable?
- Is an executable running with elevated permissions?
- Could a low-privilege account replace a script executed by a privileged service?
Special permission bits can also matter. setuid, setgid, and the sticky bit have legitimate purposes, but unexpected permission changes deserve context.
Correlate process, authentication, and network activity
A Linux investigation becomes much stronger when host evidence is aligned with network evidence.
Suppose DNS monitoring flags a server querying a suspicious domain. The analyst might correlate:
- an SSH login from an unusual source
- use of
sudo - a shell or script execution in the same time window
- creation of a cron job or systemd unit
- the DNS query and outbound connection
- an archive or staging file written shortly beforehand
A centralized SIEM can help align those records, but the analyst still needs to understand what each Linux source means.
Time consistency becomes especially important when evidence spans several Linux hosts. Check the system time, time-zone interpretation, boot time, and synchronization status before building a cross-host sequence. A few minutes of clock drift can make a remote login appear to occur after the process it actually launched.
Host identity also matters in centralized logs. Reused hostnames, ephemeral cloud instances, containers, and rebuilt systems can cause records from different machines to look related. Keep stable identifiers such as cloud instance IDs, machine IDs, IP history, or asset inventory references where available so the timeline is tied to the correct system.
Do not assume absence of a log means absence of activity
Logging configuration varies widely between systems.
A process may execute without process-level auditing. A short-lived container may send logs somewhere other than the host. An application may keep its own logs. Journal retention can be limited. Authentication may pass through centralized identity infrastructure.
When evidence is missing, document the visibility gap rather than interpreting silence as proof that nothing happened.
That is a critical SOC habit: distinguish no evidence observed from evidence that the event did not occur.
Logs can disappear for benign reasons such as rotation, retention limits, a reboot with volatile journal storage, or a service that writes somewhere unexpected. They can also disappear because auditing was never enabled, a forwarder failed, or an attacker tampered with local evidence. The absence itself becomes a question to investigate: should this host have produced this event, and where should it have gone?
Compare local and remote copies when possible. A centralized collector may preserve an authentication or audit record that has already rotated off the endpoint, while the endpoint may retain process or filesystem details that were never forwarded. Differences between the two can reveal a collection gap and should be documented rather than silently treated as proof that nothing happened.
A repeatable Linux investigation workflow
Begin with time synchronization and the incident window. Then:
- identify the relevant host and distribution
- determine which logging stack is active
- review authentication and privilege-escalation evidence
- inspect process/service/task activity
- check relevant files, ownership, and permissions
- correlate with network and security-tool telemetry
- preserve the original records before making investigative changes
Linux logs become manageable when they are treated as multiple evidence sources describing one timeline, not as a single file an analyst is expected to memorize.