Shell: Functions

In this tutorial I will show you how to work with functions.

Define Function

You return 0 for success and 1 for failure

function test()
{
    return 0
}

Call function with no arguments

test

Call function with arguments

val1='test'
val2='test2'
test $val1 $val2

Call function with arguments that have spaces in the value

val1='test'
val2='test2 test23'
test "${val1}" "${val2}"

 

 

Shell: Misc Commands

In this tutorial I will show a few useful commands when working with linux shell.

Check Directory Exists:

if [ -d /opt/test/ ]; then
    echo 'Directory Exists'
fi

Check Directory Not Exists:

if [ ! -d /opt/test/ ]; then
    echo 'Directory Does Not Exist'
fi

Check File Exists:

if [ -f /opt/test/test.log ]; then
    echo 'File Exists'
fi

Check File Not Exists:

if [ ! -f /opt/test/test.log ]; then
    echo 'File Does Not Exist'
fi

Lowercase Variable:

val='TEXT'
echo "${val,,}"

Echo Variable:
This will print the value of “test”. Notice we use double quotes.

test='My Test Val'
echo "$test"

Echo String:

echo 'My test string'

Split:
This will split on the comma into an array list and then loop through it.

test='test 1,test 2'
split_test=(${test//,/ })

for val in "${split_test[@]}"
do
    echo $val
done

Date:
This will print the date in the format YYYY-MM-dd

my_date="$(date +Y-%m-%d)"
echo "$my_date"

Remove Space From Variable:

VAL='test string'
echo "${VAL//\ /}"

Increment Variable:

index=0
index=$((index+1))

Substring

VAL='test string'
echo "${VAL:4:4}"

If value is equal to

VAL='hello'
if [ "$VAL" == 'hello' ] ; then
    echo 'Is Equal'
fi

If with OR

VAL='hello'
if [ "$VAL" == 'hello' ] || [ "$VAL" != 'hi' ] ; then
    echo 'Is Hello'
fi

If Variable is Empty

VAL=''
if [ -z "$VAL" ] ; then
    echo 'Is Empty'
fi

Append to File

echo 'Hi' >> file_to_log_to.log

Write to File

echo 'Hi' > file_to_log_to.log

While Loop: Increment to 10

This will loop till the value is 9 then exit.

i=0
while [ $i -lt 10 ];
do
    echo "$i"
done

whoami

USER=$(whoami)

If Variable Contains Text

VAL='my Test String'
if [[ "${VAL,,}" == *"test"* ]] ; then
    echo "Found test"
fi

Color Coding

NoColor=$'\033[0m'
READ=$'\033[0;31m'
GREEN=$'\033[0;32m'
YELLOW=$'\033[1;33;40m'

printf "%s Variable Not Set %s\n" "${RED}" "${NoColor}"

Get Log to a logfile and console

SOME_COMMAND 2>&1 | tee -a "${LOG_FILE_PATH}"

Read a JSON config

JSON=$(cat "/path/to/json/file.json")
export MY_VAR=$(echo "${JSON}" | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["MyKey"])')

Extract tar to Folder

sudo tar -xvf /the/location/file.tar -C /to/location/ --force-local --no-same-owner

Update Certificates

This will update certificates. After you put a certificate in /usr/local/share/ca-certificates/

update-ca-certificates

PipeStatus

somecommand
RETURN_CODE=${PIPESTATUS[0]}

Shell: AWK Command

In this tutorial I will show you some useful awk uses.

Determine Line Number

This will find the line number of the text_to_find.

lineNumber=`awk '/TEXT_TO_FIND/{print NR}' /file/path/name.extension`
Split Text By Space

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk '{ print $2 }'
Split Text By Comma

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk -F, '{ print $2 }'

 

Shell: SED Command

In this tutorial I will show a few useful examples of using “sed”.

-i: inline replace
-e: command to run

Add Line At End

Notice ($) and (a after $). Match last line (‘$’) and append (‘a’)

sudo sed -i -e "\$amy new line" /file/path/name.extension
Match Replace

This will replace the text on the left with the text on the right. Notice the “/” at the beginning and “/c\” in the middle.

sudo sed -i '/myCustomText=/c\myCustomText=newtext' /file/path/name.extension
Remove Matched Line

This will remove the line that matches the text. Notice the “/d” at the end and the “/” at the beginning.

sudo sed -i '/RemoveMyCustomText/d' /file/path/name.extension
Remove All Occurrences

This will remove all occurrences of “NIFI.APACHE.ORG” and replace with “REALM.CA”. Notice “/g” at end for global replace. Notice “s/” at the beginning. If means substitute. The “/” in the middle is used to denote the separation of the text to find and the text to replace.

sudo sed -i 's/NIFI.APACHE.ORG/REALM.CA/g' /file/path/name.extension
Handle ” Quotes / Handle XML / Add Tab

So the below example will go a few things. It will show you how to escape double quotes. It shows you how to add a tab and shows you how to handle xml. It also will add a line under text.

sudo sed -i -e "/<property name=\"Default Realm\">REALM.CA<\/property>/a \ \t<property name=\"Kerberos Config File\">/etc/krb5.conf<\/property>" /file/path/name.extension
Delete Lines By Number
sudo sed -i -e "$lineNumber,$maxLineNumber d" /file/path/name.extension

NGINX + UWSGI + Ubuntu 16

If you use Python as part of your stack in developing a web architecture you most likely will also be utilizing NGINX with UWSGI together. In the following article I will give step by step instructions on how to set this up assuming you are using sockets as part of your installation.

What is UWSGI?: A full stack for building application servers for python and various other languages.

What is NGINX?: Is simply a web server or proxy server for various protocols but can also do load balancing, etc.

NGINX Installation:

sudo apt-get install nginx

#The next command checks that nginx is running
ps -auxw | grep nginx

#Stop the service
sudo service nginx stop
#Start the service
sudo service nginx start
#Restart the service
sudo service nginx restart
#nginx status
sudo service nginx status

Once it is installed you will have two folders. One called “sites-enabled” and one called “sites-available” located under /etc/nginx/ folder. The “sites-enabled” folder usually holds a symbolic link to the actual file sometimes in “sites-available” folder. But really can be hosted anywhere on the machine.

If you want to remove the default site link you can

sudo rm /etc/nginx/sites-enabled/default

Create our symbolic link in sites-enabled if we already have a nginx.conf file ready.

cd /etc/nginx/sites-enabled/

sudo ln -s ~/path/to/nginx/conf/nginx.conf nginx.conf

It’s important to note that you may run into permissions errors depending on where you are running your nginx.conf from. So you may need to grant permissions to your users home folder.

sudo chmod 755 /home/user_your_run_your_app_from/

UWSGI Installation:

sudo apt-get install uwsgi

#Stop uwsgi
sudo service uwsgi stop
#Start uwsgi
sudo service uwsgi start
#Restart uwsgi
sudo service uwsgi restart
#uwsgi status
sudo service uwsgi status

It you want to log to a custom folder your uwsgi.ini file will need “logto” folder specified and the correct permissions set.

cd /var/log/
sudo touch /var/log/mylogfile
sudo chmod 666 /var/log/mylogfile

You uwsgi.ini will also have a pid file “pidfile” which we will need to create.

cd /tmp/
touch mypid.pid

#Ownership
sudo chown THEUSER:www-data mypid.pid

#Ensure permissions are set to -rw-r--r--

UWSGI INI Options:

You can add whatever options you prefer as part of your uwsgi.ini setup file. However the below are usually the ones you should use.

[uwsgi]
socket = :5006 #or whatever socket you want
chmod-socket = 777
pidfile=/tmp/mypid.pid
master = True
chdir = /home/user_your_run_your_app_from/

#NOTE that you may or may not need this depending on how you setup your server.
plugin = python3
wsgi_file = my_python_class.py #File that runs your application
module = whatever
callable = app #Set default WSGI callable name.
logto = /var/log/mylogfile

These are not all the options you can pick. You can also set “stats”, “buffer-size”, “processes”, “threads”, “env”, etc. Just review the docs and pick what is right for you.

The option “env” you can add many times and for each time you use it you can set environment variable for use with your application.

NOTE: If you are running flask and use send_file and run as Python3.5 you may get the error “SystemError: <built-in function uwsgi_sendfile> returned a result with an error set”. If you do then you must add the following to your uwsgi.ini file “wsgi-disable-file-wrapper = true”.

NGINX Conf Example:

upstream flask {
    server 0.0.0.0:5006;
}
server {
    client_max_body_size 1G;
    listen 0.0.0.0:8081;
    server_name localhost;
    access_log /var/log/nginx/access_log combined;
	
    location / {
        include uwsgi_params;
        uwsgi_pass flask;
    }
    location /static/ {
        alias /path/to/my/app/static/;
    }
}
fastcgi_intercept_errors on;

UWSGI Service:

Sometimes we want to run our app as a service if you do then below are the steps for that.

sudo nano /etc/systemd/system/myawesomeservice.service

#Enter the contents below

[Unit]
Description=uWSGI instance to serve My Service
After=network.target
 
[Service]
User=user_to_run_as
Group=www-data
WorkingDirectory=/path/to/my/app/
ExecStart=/usr/bin/env bash -c 'uwsgi --ini /path/to/my/app/uwsgi.ini --uid user_to_run_as --gid www-data'
Restart=always
RestartSec=3
 
[Install]
WantedBy=multi-user.target

Next you need to ensure the user has access to your folder with correct ownerships.

chown -R user_to_run_as:www-data /path/to/my/app/*

Next enable the service and start the service.

sudo systemctl enable myawesomeservice
sudo systemctl daemon-reload
sudo systemctl start myawesomeservice
sudo service myawesomeservice status

Ubuntu: tmux

Tmux (Terminal Multiplexer) is really helpful for running many virtual terminals that stay around even after you exit out of the server.

Some quick cheats are:

  • tmux -V: What version you are running.
  • CTRL b %: Split terminal horizontally.
  • CTRL b “: Split terminal vertically.
  • CTRL b LEFT: Switch between panes to the left.
  • CTRL b RIGHT: Switch between panes to the right.
  • CTRL b DOWN: Switch between panes to the down.
  • CTRL b UP: Switch between panes to the up.
  • CTRL b z: Toggle a pane.
  • CTRL b x: Kill the pane you are on.
  • CTRL b c: Create a session window.
  • CTRL b NUMBER: Enter 1, 0, 2, etc and you can switch to that session.
  • CTRL b d: Detach from tmux keeping it running.
  • tmux ls: Lists all sessions currently running.
  • tmux attach -t 0: Attach to the session by number. If you have two you can select 1 instead of 0.
  • tmux kill-session -t 0: This kills a session by session number.

Postgres: Setup

If you want to setup a database server for Postgresql 9.6 these are the basic steps that should be done when working with Ubuntu 16.04.

Installation:

  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo reboot (if necessary)
  • sudo add-apt-repository “deb https://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main”
  • tee -a /etc/apt/sources.list.d/pgdg.list
  • wget –quiet -O – https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add –
  • apt-get upgrade
  • sudo apt-get install postgresql-9.6
  • cd /var/
  • mkdir pg_log (This is where the logs will be stored)
  • chmod o+wxr /var/pg_log/
  • cd /etc/postgresql/9.6/main
  • nano postgresql.conf:
    • CONNECTIONS AND AUTHENTICATION
      • set “listen_addresses” = ‘*’
      • Set “max_connections” to 250 (or a reasonable number that works for your setup)
      • logging_collector = on
      • log_directory = ‘/var/pg_log’
      • log_file_mode = 0506
    • AUTOVACUUM PARAMETERS
      • autovacuum = on
        • Remove #
      • autocavuum_analyze_threshold = 100
      • autovacuum_vacuum_threshold = 100
      • track_counts = on
  • nano pg_hba.conf
    • host all all 0.0.0.0/0 md5
    • local all postgres trust
  • systemctl enable postgresql
  • systemctl start postgresql
  • /etc/init.d/postgresql restart
  • -u postgres psql template1
  • ALTER USER postgres with encrypted password ‘SET A PASSWORD’;
  • \q
  • -u postgres psql
  • CREATE EXTENSION adminpack;
    • To ensure it was installed correctly run “select * from pg_extension;”
  • If you want to view the postgresql service to find what directories are currently being using then run “ps auxw | grep postgres | grep — -D”

Create DB:

  • -u postgres psql
  • CREATE DATABASE ##DATABASE NAME##;
    • First run \l to check that DB doesn’t exist already.
  • CREATE ROLE ##ROLE NAME## LOGIN ENCRYPTED PASSWORD ‘##PASSWORD##’;
  • GRANT ##REQUIRED## PRIVILEGES ON DATABASE ##DATABASE NAME##TO ##ROLE NAME##;

Useful PSQL Commands

  • \l = list database
  • \q = quit
  • \c DBNAME = switch to db
  • \dt = list tables
  • ALTER DATABASE name OWNER TO new_owner
  • ALTER TABLE test OWNER TO test;