All Collections
Send your logs
Filebeat
Filebeat

Use Coralogix's prospector to ship your logs directly from Filebeat

Amnon Shahar avatar
Written by Amnon Shahar
Updated over a week ago

Coralogix provides seamless integration with Filebeat so you can send your logs from anywhere and parse them according to your needs.

Prerequisites

General

Private Key - A unique ID which represents your company, this Id will be sent to your mail once you register to Coralogix.

Company Id - A unique number which represents your company. You can get your company id from the settings tab in the Coralogix dashbaord.

Application Name - The name of your main application, for example, a company named “SuperData” would probably insert the “SuperData” string parameter or if they want to debug their test environment they might insert the “SuperData– Test”.

SubSystem Name - Your application probably has multiple subsystems, for example: Backend servers, Middleware, Frontend servers etc. in order to help you examine the data you need, inserting the subsystem parameter is vital.

Installation

For a quick setup of Filebeat on your server, you can use prepared scripts.

Go to the folder with your Filebeat configuration file (filebeat.yml) and execute (as root):
Deb:

$ curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-deb.sh | bash

rpm:

$ curl -sSL https://raw.githubusercontent.com/coralogix/integrations-docs/master/integrations/filebeat/scripts/install-rpm.sh | bash

This script will install Filebeat on your machine, prepare configuration and download Coralogix SSL certificates.

Note: If you want to install a specific version of Filebeat you should pass version number with environment variable before script run:

$ export FILEBEAT_VERSION=6.6.2

Configuration

Open your Filebeat configuration file and configure it to use Logstash (Make sure you disable Elastic output). For more information about configuring filebeat to use logstash please refer to: https://www.elastic.co/guide/en/beats/filebeat/current/config-filebeat-logstash.html

Point your filebeat to output to Coralogix logstash server: 

logstashserver.coralogix.com:5044

or if you want to use an encrypted connection (recommended):

logstashserver.coralogix.com:5015


Here is a basic example of filebeat.yml:

#============================== Filebeat Inputs ===============================

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"

fields_under_root: true
fields:
  PRIVATE_KEY: "YOUR_PRIVATE_KEY"
  COMPANY_ID: YOUR_COMPANY_ID
  APP_NAME: "APP_NAME"
  SUB_SYSTEM: "SUB_NAME"

#----------------------------- Logstash output --------------------------------

output.logstash:
  enabled: true
  hosts: ["logstashserver.coralogix.com:5015"]
  tls.certificate_authorities: ["<path to folder with certificates>/ca.crt"]
  ssl.certificate_authorities: ["<path to folder with certificates>/ca.crt"]

Note: If you want to send all additional metadata, the fields_under_root option should be equals to true.

If you have multiline logs like:

2019-08-31 14:27:33 [main] ERROR Main - Exception
javax.management.RuntimeErrorException: null
    at Main.main(Main.java:16) ~[bin/:na]
Copy

You can use multiline pattern:

filebeat.inputs:
- type: log
  paths:
  - "/var/log/your_app/your_app.log"
  multiline:
    pattern: '^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} \['
    negate: true
    match: after

The pattern should match the beginning of each record.

Docker

FROM docker.elastic.co/beats/filebeat:6.6.2

LABEL description="Filebeat logs watcher"

# Adding configuration file and SSL certificates for Filebeat
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
COPY ca.crt /etc/ssl/certs/Coralogix.crt

# Changing permission of configuration file
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml

# Return to deploy user
USER filebeat

Before deploying your container don’t forget to mount volume with your logs.

Did this answer your question?