3.4. Logs Developer Guide¶
3.4.1. Ansible Client-side¶
3.4.1.1. Ansible File Organisation¶
Files Structure:
ansible-client
├── ansible.cfg
├── hosts
├── playbooks
│ └── setup.yaml
└── roles
├── clean-td-agent
│ └── tasks
│ └── main.yml
└── td-agent
├── files
│ └── td-agent.conf
└── tasks
└── main.yml
3.4.1.2. Summary of roles¶
Roles |
Description |
---|---|
|
Install Td-agent & change configuration file |
|
Unistall Td-agent |
3.4.1.3. Configurable Parameters¶
File (ansible-client/roles/) |
Parameter |
Description |
---|---|---|
|
host |
Fluentd-server IP |
|
port |
Fluentd-Server Port |
3.4.2. Ansible Server-side¶
3.4.2.1. Ansible File Organisation¶
Files Structure:
ansible-server
├── ansible.cfg
├── group_vars
│ └── all.yml
├── hosts
├── playbooks
│ └── setup.yaml
└── roles
├── clean-logging
│ └── tasks
│ └── main.yml
├── k8s-master
│ └── tasks
│ └── main.yml
├── k8s-pre
│ └── tasks
│ └── main.yml
├── k8s-worker
│ └── tasks
│ └── main.yml
├── logging
│ ├── files
│ │ ├── elastalert
│ │ │ ├── ealert-conf-cm.yaml
│ │ │ ├── ealert-key-cm.yaml
│ │ │ ├── ealert-rule-cm.yaml
│ │ │ └── elastalert.yaml
│ │ ├── elasticsearch
│ │ │ ├── elasticsearch.yaml
│ │ │ └── user-secret.yaml
│ │ ├── fluentd
│ │ │ ├── fluent-cm.yaml
│ │ │ ├── fluent-service.yaml
│ │ │ └── fluent.yaml
│ │ ├── kibana
│ │ │ └── kibana.yaml
│ │ ├── namespace.yaml
│ │ ├── nginx
│ │ │ ├── nginx-conf-cm.yaml
│ │ │ ├── nginx-key-cm.yaml
│ │ │ ├── nginx-service.yaml
│ │ │ └── nginx.yaml
│ │ ├── persistentVolume.yaml
│ │ └── storageClass.yaml
│ └── tasks
│ └── main.yml
└── nfs
└── tasks
└── main.yml
3.4.2.2. Summary of roles¶
Roles |
Description |
---|---|
|
Pre-requisite for installing K8s, like installing docker & K8s, disable swap etc. |
|
Reset K8s & make a master |
|
Join woker nodes with token |
|
EFK & elastalert setup in K8s |
|
Remove EFK & elastalert setup from K8s |
|
Start a NFS server to store Elasticsearch data |
3.4.2.3. Configurable Parameters¶
File (ansible-server/roles/) |
Parameter name |
Description |
---|---|---|
Role: logging |
||
|
storage |
Increase or Decrease Storage size of Persistent Volume size for each VM |
|
version |
To Change the Kibana Version |
|
count |
To increase or decrease the replica |
|
version |
To Change the Elasticsearch Version |
|
nodePort |
To Change Service Port |
|
storage |
Increase or Decrease Storage size of Elasticsearch data for each VM |
|
nodeAffinity -> values (hostname) |
In which VM Elasticsearch master or data pod will run (change the hostname to run the Elasticsearch master or data pod on a specific node) |
|
stringData |
Add Elasticsearch User & its roles (Elastic Docs) |
|
replicas |
To increase or decrease the replica |
|
nodePort |
To Change Service Port |
|
index_template.json -> number_of_replicas |
To increase or decrease replica of data in Elasticsearch |
|
fluent.conf |
Server port & other Fluentd Configuration |
|
replicas |
To increase or decrease the replica |
|
nodePort |
To Change Service Port |
|
kibana-access.key, kibana-access.pem |
Key file for HTTPs Connection |
|
Nginx Configuration |
|
|
replicas |
To increase or decrease the replica |
|
elastalert.key, elastalert.pem |
Key file for HTTPs Connection |
|
run_every |
How often ElastAlert will query Elasticsearch |
|
alert_time_limit |
If an alert fails for some reason, ElastAlert will retry sending the alert until this time period has elapsed |
|
es_host, es_port |
Elasticsearch Serivce name & port in K8s |
|
http_post_url |
Alert Receiver IP (Elastalert Rule Config) |
Role: nfs |
||
|
line |
Path of NFS storage |