- Site Collector Overview
- Get Started with Site Collectors
- Install Site Collector
- Set Up Collectors
- Sign Up for the Early Access Program: Site Collectors
- Choose the Right Collector based on Data Sources
- Set Up Archive Windows Collector
- Set Up Archive Linux Collector
- Set Up EStreamer Collector
- Set Up Fortinet Collector
- Set Up IBM Security QRadar Collector
- Set Up Kafka Collector
- Set Up Splunk Collector
- Set Up Linux File Collector
- Set Up Microsoft SQL Collector
- Set Up MySQL Collector
- Set Up Oracle Collector
- Set Up Syslog Collector
- Set Up Windows Active Directory Collector
- Set Up Windows Event Log Collector
- Set Up Windows File Collector
- Manage Site Collectors
- Apply Antivirus Exclusions
- Migrate to the New-Scale Site Collectors Service
- Modify Collector Configuration
- Modify a Site Collector Instance
- Manage Templates
- Monitor Log Sources
- Add Filters to Set Egress Log Filtering Conditions
- New Site Collector Management Service NGSCD
- Regenerate Certificates for Collectors
- Upgrade the Site Collector
- Upgrade the Site Collector Specifications
- Site Collector Monitoring
- Troubleshoot the Site Collector
- Pre-checks failed during Site Collector installation and upgrade
- Site Collector UI shows the status INSTALLATION_ERROR
- Download Support Packages for Troubleshooting
- How to reboot the Virtual Machine (VM) successfully to apply security updates?
- What information must be added while creating a support ticket to resolve an issue?
- Site Collector UI is not displaying the heartbeats
- Splunk Collector can't be set up
- Splunk Collector is set up however, logs are not reaching DL/AA
- Only a few of the installed Splunk Collectors are processing logs or EPS has dropped by 50% as compared to last hour
- The Windows Active Directory Collector (formerly known as LDAP Collector) is set up, however, the context data is not reaching DL/AA
- The Windows Active Directory Collector (formerly known as LDAP Collector) is stuck in the ‘Update’ mode after deployment
- Installation is initiated; however, the collector shows the status as ‘Setting Up’ for some time
- Data Lake and Advanced Analytics Does Not Show Context Data
- Context Data from Windows Active Directory Collector is Segmented
- Minifi Permission Denied - Logback.xml File Missing and Config File Update - Failed Error Occurred while Installing the Windows Event Log Collector
- Where should I upload proxy certificates if I am running proxy with TLS interception?
- How to upgrade Linux collector instance?
Choose the Right Collector based on Data Sources
Selecting a type of collector—either File Collector or Archive Collector—depends on how the data source is generated and updated, not on the total size of your data.
File Collector – Select File Collector for the files that continuously grow. For example, log files where data keeps getting added and applications continuously append new data. The File Collector tracks the last line it read in a file and continues collecting data from that point during each fetch interval.
File Collector is designed to:
Track updates and continue processing from the last read position in each file.
Resume collection from the last read point during each scheduled fetch.
Support large files, up to 20 GB per individual file.
You can choose File Collector for monitoring and collecting data from active, evolving files.
Archive Collector – Use the Archive Collector for files that are static and do not change after being created. These files are typically compressed backup files such as .zip, .gzip, or .tar.
Archive Collector is designed to:
Monitor for new files being added to a specified location.
Work with compressed files that are not modified after creation.
Support compressed files up to 2 GB each, which can expand to 15–20 GB uncompressed depending on the compression algorithm used.
Note
When the collector processes large files, the collector can process fewer number of files in parallel.
You can choose Archive collector for environments where data is stored in periodic, packaged archives rather than being appended to continuously.
Note
If you have a unique scenario where a process generates new files and appends them to existing ones using a script, you can use both type of collectors with minor adjustments. To utilize Archive collector in this case, you can generate the new files and compress them into gzip files, and instead of appending them to existing files, place the new .gzip files into a designated location. The Archive Collector can then detect and process these new files automatically.