elasticsearch reference

Tools
– head for Chrome (ElasticSearch Head – Chrome Web Store)
– Postman (link)
– Insomenia (link)
– elasticdump – nodejs (link)

Monitoring
– ps_mem.py – monitor real memory utilization (github link)
ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x< =NF ; x++ ) { printf("%s ",$x) } print "" }' |cut -d "" -f2 | cut -d "-" -f1 | head -n 40
from : here
- netdata, dockerable too - (link)

System tuning
sysctl -w vm.max_map_count=262144
sysctl -w vm.swappiness = 0

verify
sysctl vm.max_map_count
sysctl vm.swappiness

Reference
https://stefanprodan.com/2016/elasticsearch-cluster-with-docker/

Memory tuning
https://qbox.io/blog/memory-considerations-in-elasticsearch-deployment

GC Tuning: In Practice

Stuck shards
https://thoughts.t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode-ce196e20ba95
https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/

mysqldump script (per tables)


#!/bin/bash

# This is PER TABLE backup. Each table will be backed up individually

## to enable passwordless mysqdump, put your password in /etc/mysql/[mysqld|percona|maria].conf.d/client.conf
# [client]
# user=""
# pass=""

THEDB="mydbname" # THE DATABASE
OUTPUT="/home/backup/mysqldump"
DATE=`date +%Y%m%d`
OPTS="--max_allowed_packet=512M "

tables=`mysql -e "use $THEDB; show tables;" | tr -d "| " | grep -v -E "^Tables_in_" `
mkdir -p $OUTPUT/$THEDB-$DATE

for table in $tables; do
STAGEDDATE=`date +%Y%m%d`
echo "Dumping table : " $table
mysqldump -a $OPTS $THEDB $table > $OUTPUT/$THEDB-$DATE/$STAGEDDATE-$table.sql
gzip $OUTPUT/$THEDB-$DATE/$STAGEDDATE-$table.sql
done

echo "Backup done at " $OUTPUT/$THEDB-$DATE

Web server tuning (apache and nginx)

Key points

  • enable http2 & change mpm prefork to event
  • php-fpm

 

Enable http2

On apache + prefork to event – https://techwombat.com/enable-http2-apache-ubuntu-16-04/

sudo add-apt-repository ppa:ondrej/apache2
sudo apt update
sudo apt upgrade
sudo apt install php7.0-fpm 
sudo a2enmod proxy_fcgi setenvif
sudo a2enconf php7.0-fpm 
sudo a2dismod php7.0 
sudo a2dismod mpm_prefork 
sudo a2enmod mpm_event 
sudo service apache2 restart
sudo service php7.0-fpm restart

Add in <VirtualHost>… </VirtualHost> for individual site, or in apache.conf file, for global settings.

Protocols h2 h2c http/1.1

sudo a2enmod http2
sudo service apache2 restart

Nginx – Need to compile nginx with http2 module
./configure –with-compat –add-dynamic-module=../ModSecurity-nginx –with-http_ssl_module –with-stream_ssl_module –prefix=/etc/nginx –with-http_v2_module

** mod_security for nginx, follow this : https://www.nginx.com/blog/compiling-and-installing-modsecurity-for-open-source-nginx/

 

This post content is under development.. new content will be added in the future.

Elasticsearch notes

Useful links

Elasticdump – http://blog.ruanbekker.com/blog/2017/11/22/using-elasticdump-to-backup-elasticsearch-indexes-to-json/

sudo npm install n -g
sudo n stable

 

https://www.digitalocean.com/community/tutorials/how-to-interact-with-data-in-elasticsearch-using-crud-operations

Delete by time
https://hobo.house/2016/02/18/how-to-manually-clean-indexes-from-elasticsearch/

 

Slow HTTP Dos Verify and mitigate

What it is
https://en.wikipedia.org/wiki/Slowloris_(computer_security)

Tools to check
Slowloris : https://github.com/gkbrk/slowloris
SlowHttpTest : https://github.com/shekyan/slowhttptest

Mitigatation

mod_qos
apt-get update && apt-get -y install libapache2-mod-qos && a2enmod qos && /etc/init.d/apache2 restart
* configuration file in /etc/apache2/mods-enabled/

mod_reqtimeout
a2enmod reqtimeout && /etc/init.d/apache2 restart
* configuration file in /etc/apache2/mods-enabled/

Securing apache

Some notes on securing apache..

A few key points

  • Disable access to .
  • Disable banner
  • disabled PHP functions
  • Slow dos mitigation (link)

Disable access to .
https://stackoverflow.com/questions/4352737/apache-configuration-regex-to-disable-access-to-files-directories-beginning-wit

<filesmatch "^\.ht">
Order allow,deny
Deny from all

<filesmatch “^\.”>
Order allow,deny
Deny from all

<directorymatch “^\.|\/\.”>
Order allow,deny
Deny from all

Disable banner
http://www.ducea.com/2006/06/15/apache-tips-tricks-hide-apache-software-version/

Usually found in /etc/apache/conf-enabled/security.conf

Disable PHP functions
https://www.cyberciti.biz/faq/linux-unix-apache-lighttpd-phpini-disable-functions/

disable_functions =exec,passthru,shell_exec,system,proc_open,popen,pcntl_exec,curl_exec,curl_multi_exec,parse_ini_file,show_source,phpinfo,eval,assert,create_function

Slow DOS mitigation
http://www.farhanfaisal.com/slow-http-dos-verify-and-mitigate/

Easiest way to install ffmpeg on mac os

Just keeping reference.

https://www.oodlestechnologies.com/blogs/Easiest-Way-To-Install-FFmpeg-On-Mac-OS-X
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

https://gist.github.com/clayton/6196167
brew install ffmpeg --with-vpx --with-vorbis --with-libvorbis --with-vpx --with-vorbis --with-theora --with-libogg --with-libvorbis --with-gpl --with-version3 --with-nonfree --with-postproc --with-libaacplus --with-libass --with-libcelt --with-libfaac --with-libfdk-aac --with-libfreetype --with-libmp3lame --with-libopencore-amrnb --with-libopencore-amrwb --with-libopenjpeg --with-openssl --with-libopus --with-libschroedinger --with-libspeex --with-libtheora --with-libvo-aacenc --with-libvorbis --with-libvpx --with-libx264 --with-libxvid

Installing cuckoo sandbox on Mac OS

Cuckoo sandbox is an automated malware analysis system. Its does utilize virtualization engine to isolate malware execution and analysis. You will be given a web-interface, as well as CLI tools to communicate with cuckoo, such to upload, and reviewing all reports.

I’ve found a complete tutorial on this.

http://advancedmalwareprotection.blogspot.com/2012/03/installing-cuckoo-on-max-os-x-lion.html

Hope this would help anyone to have this. The tutorial is a bit technical. If you’re not familiar with Mac system or Terminal, you might want just to use any public sandbox.