Deploy an Elasticstack with SIEM with docker-compose for lab

Edits:

2021-07-06 - First Init

Caveats:

This IS NOT a production deployment of any sort. This is for LAB environment
This is a “Mental note to self” for the next time i want to fire up an current Elasticsearch stack for lab .

Like the last guides:
This installation is not made for production or public facing servers, there is no built in security in this setup.
I take no responsibility if this guide bork you server, burn your house down to ashes, make your christmas tree self combust or makes your cat to leave you..
It’s under the “it worked for me[tm]” clause.
This is as always a work in progress.

preparation is key

Preparation is key

Prepare a can of coffee and (if at home) put on your best comfy clothes.
When writhing this, Rage Beats was playing in the speakers. And not everyone in the family is so happy about this :)

Requirements

Some kind of VM is suggested.
I use ubuntu LTS releases as vm’s, but any linux distro is usable (i guess?)
Some free time :)

Install the VM with minimal options.

Nothing special here, just remember that this vm must have more than 4G of virtualmemory for this example.
https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html

Config vm-virtualmemory

First run this as a sudo enabled user

sudo sysctl -w vm.max_map_count=262144

Then add this line in /etc/sysctl.conf

vm.max_map_count=262144

Install docker and docker-compose

Beq this is a lab I use the docker and docker-compose in the ubuntu repos.
If this was a production docker environment I would use this guide » https://docs.docker.com/engine/install/ubuntu/

sudo apt install docker-compose

Then add your user to the docker group. Again this is a temporary lab, so I just add my local user to the docker group.
If this would be a production environment I would add a dedicated elastic-docker user with nologin set.
Something like this » https://sadsloth.net/post/install-librenms-docker/ (Create user that runs librenms)

sudo usermod -aG docker $USER

Start to configure the Elastic stuff in docker.

It’s almost a blueprint of the docs from here
» https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
» https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls-docker.html And when I did this 7.13.2 was the current version.,

Create directory

mkdir ~/elastic && cd ~/elastic

Create instances.yml

vim instances.yml

instances:
  - name: es01
    dns:
      - es01
      - localhost
    ip:
      - 127.0.0.1

  - name: es02
    dns:
      - es02
      - localhost
    ip:
      - 127.0.0.1

  - name: es03
    dns:
      - es03
      - localhost
    ip:
      - 127.0.0.1

  - name: 'kib01'
    dns:
      - kib01
      - localhost

Create environment file

vim .env

COMPOSE_PROJECT_NAME=es
CERTS_DIR=/usr/share/elasticsearch/config/certificates
VERSION=7.13.2

Create kibana config

vim kibana.yml

server.name: "localhost"
server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: ["https://es01:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "CHANGEME"
server.ssl.enabled: true
server.ssl.certificate: /usr/share/elasticsearch/config/certificates/kib01/kib01.crt
server.ssl.key: /usr/share/elasticsearch/config/certificates/kib01/kib01.key
elasticsearch.ssl.certificateAuthorities: [ "/usr/share/elasticsearch/config/certificates/ca/ca.crt" ]
logging.dest: stdout
logging.silent: false
logging.quiet: false
logging.verbose: false
xpack.encryptedSavedObjects.encryptionKey: "ar87tpeqJ$u8XByVzR%HJY5jSMvMDTnZhM5tYnYtUp!D*@GK&@j"

Create the certfile

vim create-certs.yml

version: '2.2'

services:
  create_certs:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: create_certs
    command: >
      bash -c '
        yum install -y -q -e 0 unzip;
        if [[ ! -f /certs/bundle.zip ]]; then
          bin/elasticsearch-certutil cert --silent --pem --in config/certificates/instances.yml -out /certs/bundle.zip;
          unzip /certs/bundle.zip -d /certs;
        fi;
        chown -R 1000:0 /certs
      '
    working_dir: /usr/share/elasticsearch
    volumes:
      - certs:/certs
      - .:/usr/share/elasticsearch/config/certificates
    networks:
      - elastic

volumes:
  certs:
    driver: local

networks:
  elastic:
    driver: bridge

Ant then docker-compose file

vim docker-compose.yml

version: '2.2'

services:
  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=basic 
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true 
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true 
      - xpack.security.transport.ssl.verification_mode=certificate 
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    ports:
      - 9200:9200
    networks:
      - elastic

    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es02/es02.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es02/es02.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=basic
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es03/es03.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es03/es03.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic
  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    volumes:
      - certs:$CERTS_DIR
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - elastic
volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
  certs:
    driver: local

networks:
  elastic:
    driver: bridge

Generate certificates

docker-compose -f create-certs.yml run --rm create_certs

Bring up the dev cluster

docker-compose up -d

Create passwords and such stuff

docker exec es01 /bin/bash -c "bin/elasticsearch-setup-passwords \
auto --batch --url https://es01:9200"

write down the output

Changed password for user apm_system
PASSWORD apm_system = DPDRZDRCtYTmNf2xIAs2

Changed password for user kibana_system
PASSWORD kibana_system = lT24jlUpKrSakDLoM5ch

Changed password for user kibana
PASSWORD kibana = lT24jlUpKrSakDLoM5ch

Changed password for user logstash_system
PASSWORD logstash_system = G6GIv3myblm5Jf2hzjsc

Changed password for user beats_system
PASSWORD beats_system = ar0vnXOHuqpTHmfYxyWR

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = FE7q18DepQUq0fitxR9q

Changed password for user elastic
PASSWORD elastic = i9oCA6HjMohVosH8pKlq

Replace the “CHANGEME” to kibana_system in docker-compose.yml and kibana.yml

sed -i 's|CHANGEME|\"lT24jlUpKrSakDLoM5ch\"|g' docker-compose.yml
sed -i 's|CHANGEME|lT24jlUpKrSakDLoM5ch|g' kibana.yml

Restart the lab cluster

docker-compose down
docker-compose up -d

use elastic / IfxCNUv0b0qolGjzGOS6 from the output above to login.

If you have any questions just make a shoutout in the comments, or join the slackchannel.

To be continued….

And of course.. Get some coffee and go outside and get some fresh air when the installation is running.