Shell: Functions

In this tutorial I will show you how to work with functions.

Define Function

You return 0 for success and 1 for failure

function test()
{
    return 0
}

Call function with no arguments

test

Call function with arguments

val1='test'
val2='test2'
test $val1 $val2

Call function with arguments that have spaces in the value

val1='test'
val2='test2 test23'
test "${val1}" "${val2}"

 

 

Shell: Misc Commands

In this tutorial I will show a few useful commands when working with linux shell.

Check Directory Exists:

if [ -d /opt/test/ ]; then
    echo 'Directory Exists'
fi

Check Directory Not Exists:

if [ ! -d /opt/test/ ]; then
    echo 'Directory Does Not Exist'
fi

Check File Exists:

if [ -f /opt/test/test.log ]; then
    echo 'File Exists'
fi

Check File Not Exists:

if [ ! -f /opt/test/test.log ]; then
    echo 'File Does Not Exist'
fi

Lowercase Variable:

val='TEXT'
echo "${val,,}"

Echo Variable:
This will print the value of “test”. Notice we use double quotes.

test='My Test Val'
echo "$test"

Echo String:

echo 'My test string'

Split:
This will split on the comma into an array list and then loop through it.

test='test 1,test 2'
split_test=(${test//,/ })

for val in "${split_test[@]}"
do
    echo $val
done

Date:
This will print the date in the format YYYY-MM-dd

my_date="$(date +Y-%m-%d)"
echo "$my_date"

Remove Space From Variable:

VAL='test string'
echo "${VAL//\ /}"

Increment Variable:

index=0
index=$((index+1))

Substring

VAL='test string'
echo "${VAL:4:4}"

If value is equal to

VAL='hello'
if [ "$VAL" == 'hello' ] ; then
    echo 'Is Equal'
fi

If with OR

VAL='hello'
if [ "$VAL" == 'hello' ] || [ "$VAL" != 'hi' ] ; then
    echo 'Is Hello'
fi

If Variable is Empty

VAL=''
if [ -z "$VAL" ] ; then
    echo 'Is Empty'
fi

Append to File

echo 'Hi' >> file_to_log_to.log

Write to File

echo 'Hi' > file_to_log_to.log

While Loop: Increment to 10

This will loop till the value is 9 then exit.

i=0
while [ $i -lt 10 ];
do
    echo "$i"
done

whoami

USER=$(whoami)

If Variable Contains Text

VAL='my Test String'
if [[ "${VAL,,}" == *"test"* ]] ; then
    echo "Found test"
fi

Color Coding

NoColor=$'\033[0m'
READ=$'\033[0;31m'
GREEN=$'\033[0;32m'
YELLOW=$'\033[1;33;40m'

printf "%s Variable Not Set %s\n" "${RED}" "${NoColor}"

Get Log to a logfile and console

SOME_COMMAND 2>&1 | tee -a "${LOG_FILE_PATH}"

Read a JSON config

JSON=$(cat "/path/to/json/file.json")
export MY_VAR=$(echo "${JSON}" | python -c 'import json,sys;obj=json.load(sys.stdin);print(obj["MyKey"])')

Extract tar to Folder

sudo tar -xvf /the/location/file.tar -C /to/location/ --force-local --no-same-owner

Update Certificates

This will update certificates. After you put a certificate in /usr/local/share/ca-certificates/

update-ca-certificates

PipeStatus

somecommand
RETURN_CODE=${PIPESTATUS[0]}

Shell: AWK Command

In this tutorial I will show you some useful awk uses.

Determine Line Number

This will find the line number of the text_to_find.

lineNumber=`awk '/TEXT_TO_FIND/{print NR}' /file/path/name.extension`
Split Text By Space

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk '{ print $2 }'
Split Text By Comma

If you want to split a word by a space and then select the second one

val='test test2'
echo $val | awk -F, '{ print $2 }'

 

Shell: SED Command

In this tutorial I will show a few useful examples of using “sed”.

-i: inline replace
-e: command to run

Add Line At End

Notice ($) and (a after $). Match last line (‘$’) and append (‘a’)

sudo sed -i -e "\$amy new line" /file/path/name.extension
Match Replace

This will replace the text on the left with the text on the right. Notice the “/” at the beginning and “/c\” in the middle.

sudo sed -i '/myCustomText=/c\myCustomText=newtext' /file/path/name.extension
Remove Matched Line

This will remove the line that matches the text. Notice the “/d” at the end and the “/” at the beginning.

sudo sed -i '/RemoveMyCustomText/d' /file/path/name.extension
Remove All Occurrences

This will remove all occurrences of “NIFI.APACHE.ORG” and replace with “REALM.CA”. Notice “/g” at end for global replace. Notice “s/” at the beginning. If means substitute. The “/” in the middle is used to denote the separation of the text to find and the text to replace.

sudo sed -i 's/NIFI.APACHE.ORG/REALM.CA/g' /file/path/name.extension
Handle ” Quotes / Handle XML / Add Tab

So the below example will go a few things. It will show you how to escape double quotes. It shows you how to add a tab and shows you how to handle xml. It also will add a line under text.

sudo sed -i -e "/<property name=\"Default Realm\">REALM.CA<\/property>/a \ \t<property name=\"Kerberos Config File\">/etc/krb5.conf<\/property>" /file/path/name.extension
Delete Lines By Number
sudo sed -i -e "$lineNumber,$maxLineNumber d" /file/path/name.extension

VirtualBox

In this post I will just walk through a couple settings that you may need to configure. For these examples I am using Ubuntu 16.04.

Networking

If you want your virtualbox os to have it’s own IP on the network and accessable outside the vm then use “Bridged Adapter”.

  • Select “Enable Network Adapter”
  • Attached to: Bridged Adapter
  • Expand Advanced
  • Promiscuous Mode: Allow All
  • Select Cable Connected
  • Adapter Type: Desktop adapter.

If you want to just forward a port to localhost then use “NAT”.

  • Select “Enable Network Adapter”
  • Attached to: NAT
  • Expand Advanced
  • Adapter Type: Desktop adapter.
  • Click “Port Forwarding”
  • Add new Rule
    • Host IP: 127.0.0.1
    • Host Port: 22
    • Guest IP: Leave empty
    • Guest Port: 22

Shared Folders

If you want to share a folder from Host to Guest.

  • Click Add Share
  • Folder Path: Select folder from Host machine
    • I will us C:\data
  • Folder Name: What the share will be on the Guest
    • data
  • Turn on VM
  • Download https://download.virtualbox.org/virtualbox/5.2.12/VBoxGuestAdditions_5.2.12.iso
  • Mount VBoxGuestAdditions_5.2.12.iso
  • On the Guest Ubuntu server run
    • sudo mount /dev/cdrom /media/cdrom
  • Install build-essential
    • sudo apt-get install build-essential linux-headers-`uname –r`
  • Run VBoxGuestAdditions
    • sudo /media/cdrom/VBoxLinuxAdditions.run
  • sudo mkdir -p /mnt/data
  • sudo reboot
  • Mount Share
    • Ubuntu Server
      • sudo mount –t vboxsf data /mnt/data
    • Ubuntu Desktop
      • sudo mount -t vboxsf -o uid=$UID,gid=$(id -g) data /mnt/data/
  • cd /mnt/data
    • Now we can share files between host and guest.

NGINX + UWSGI + Ubuntu 16

If you use Python as part of your stack in developing a web architecture you most likely will also be utilizing NGINX with UWSGI together. In the following article I will give step by step instructions on how to set this up assuming you are using sockets as part of your installation.

What is UWSGI?: A full stack for building application servers for python and various other languages.

What is NGINX?: Is simply a web server or proxy server for various protocols but can also do load balancing, etc.

NGINX Installation:

sudo apt-get install nginx

#The next command checks that nginx is running
ps -auxw | grep nginx

#Stop the service
sudo service nginx stop
#Start the service
sudo service nginx start
#Restart the service
sudo service nginx restart
#nginx status
sudo service nginx status

Once it is installed you will have two folders. One called “sites-enabled” and one called “sites-available” located under /etc/nginx/ folder. The “sites-enabled” folder usually holds a symbolic link to the actual file sometimes in “sites-available” folder. But really can be hosted anywhere on the machine.

If you want to remove the default site link you can

sudo rm /etc/nginx/sites-enabled/default

Create our symbolic link in sites-enabled if we already have a nginx.conf file ready.

cd /etc/nginx/sites-enabled/

sudo ln -s ~/path/to/nginx/conf/nginx.conf nginx.conf

It’s important to note that you may run into permissions errors depending on where you are running your nginx.conf from. So you may need to grant permissions to your users home folder.

sudo chmod 755 /home/user_your_run_your_app_from/

UWSGI Installation:

sudo apt-get install uwsgi

#Stop uwsgi
sudo service uwsgi stop
#Start uwsgi
sudo service uwsgi start
#Restart uwsgi
sudo service uwsgi restart
#uwsgi status
sudo service uwsgi status

It you want to log to a custom folder your uwsgi.ini file will need “logto” folder specified and the correct permissions set.

cd /var/log/
sudo touch /var/log/mylogfile
sudo chmod 666 /var/log/mylogfile

You uwsgi.ini will also have a pid file “pidfile” which we will need to create.

cd /tmp/
touch mypid.pid

#Ownership
sudo chown THEUSER:www-data mypid.pid

#Ensure permissions are set to -rw-r--r--

UWSGI INI Options:

You can add whatever options you prefer as part of your uwsgi.ini setup file. However the below are usually the ones you should use.

[uwsgi]
socket = :5006 #or whatever socket you want
chmod-socket = 777
pidfile=/tmp/mypid.pid
master = True
chdir = /home/user_your_run_your_app_from/

#NOTE that you may or may not need this depending on how you setup your server.
plugin = python3
wsgi_file = my_python_class.py #File that runs your application
module = whatever
callable = app #Set default WSGI callable name.
logto = /var/log/mylogfile

These are not all the options you can pick. You can also set “stats”, “buffer-size”, “processes”, “threads”, “env”, etc. Just review the docs and pick what is right for you.

The option “env” you can add many times and for each time you use it you can set environment variable for use with your application.

NOTE: If you are running flask and use send_file and run as Python3.5 you may get the error “SystemError: <built-in function uwsgi_sendfile> returned a result with an error set”. If you do then you must add the following to your uwsgi.ini file “wsgi-disable-file-wrapper = true”.

NGINX Conf Example:

upstream flask {
    server 0.0.0.0:5006;
}
server {
    client_max_body_size 1G;
    listen 0.0.0.0:8081;
    server_name localhost;
    access_log /var/log/nginx/access_log combined;
	
    location / {
        include uwsgi_params;
        uwsgi_pass flask;
    }
    location /static/ {
        alias /path/to/my/app/static/;
    }
}
fastcgi_intercept_errors on;

UWSGI Service:

Sometimes we want to run our app as a service if you do then below are the steps for that.

sudo nano /etc/systemd/system/myawesomeservice.service

#Enter the contents below

[Unit]
Description=uWSGI instance to serve My Service
After=network.target
 
[Service]
User=user_to_run_as
Group=www-data
WorkingDirectory=/path/to/my/app/
ExecStart=/usr/bin/env bash -c 'uwsgi --ini /path/to/my/app/uwsgi.ini --uid user_to_run_as --gid www-data'
Restart=always
RestartSec=3
 
[Install]
WantedBy=multi-user.target

Next you need to ensure the user has access to your folder with correct ownerships.

chown -R user_to_run_as:www-data /path/to/my/app/*

Next enable the service and start the service.

sudo systemctl enable myawesomeservice
sudo systemctl daemon-reload
sudo systemctl start myawesomeservice
sudo service myawesomeservice status

unRaid: Midnight Commander

If you are not familiar with unRaid go here. unRaid is a raid 4 implementation. If is basically a software raid.

If you want to work with the file system of unRaid whether it to be to move files or for another reason use midnight commander.

Start putty and connect to the unRaid server. Then once logged in run the following command for midnight commander.

mc

 

Ubuntu: tmux

Tmux (Terminal Multiplexer) is really helpful for running many virtual terminals that stay around even after you exit out of the server.

Some quick cheats are:

  • tmux -V: What version you are running.
  • CTRL b %: Split terminal horizontally.
  • CTRL b “: Split terminal vertically.
  • CTRL b LEFT: Switch between panes to the left.
  • CTRL b RIGHT: Switch between panes to the right.
  • CTRL b DOWN: Switch between panes to the down.
  • CTRL b UP: Switch between panes to the up.
  • CTRL b z: Toggle a pane.
  • CTRL b x: Kill the pane you are on.
  • CTRL b c: Create a session window.
  • CTRL b NUMBER: Enter 1, 0, 2, etc and you can switch to that session.
  • CTRL b d: Detach from tmux keeping it running.
  • tmux ls: Lists all sessions currently running.
  • tmux attach -t 0: Attach to the session by number. If you have two you can select 1 instead of 0.
  • tmux kill-session -t 0: This kills a session by session number.

Various Commands

Below are some useful commands for those either just learning Ubuntu 14.04 or a resource for various commands. I will continually expand on them.

Directory Exists:
if [ ! -d "##DIRECTORY##" ]; then
fi
Format a Number as two digits even if it is 1:
$(printf %02d $variable)
Increment a Number:
variable=`expr $variable + 1`
Create a new file:
touch ##FILENAME##
Check if a file contains a string:

This will return 0 if it is not found.

$(echo `grep -c '##TextToSearch##' ##FileNameToSearch##`)
Create a link to a file:
ln -s ##OriginalFile## ##LinkedFile##
List all packages installed:
dpkg -l
Remove installed package:
sudo apt-get --purge remove ##PACKAGENAME##
Install Package:
sudo apt-get install ##PACKAGENAME##
Package Exists:

This will return 0 if it is not found.

$(echo `dpkg-query -l | grep -c ##PACKAGE_NAME##`)
Python Package Exists:

This will return 0 if it is not found.

$(echo `pip freeze | grep -c ##PACKAGE_NAME##`)
Get IP Address:
ip addr show
Find a file:
find / -type f -name "##FILENAME##"
Restart Service:
/etc/init.d/postgresql restart
invoke-rc.d postgresql restart
Kill a process:
kill ##PID##
Terminal Loop:

In the terminal let’s say you want to loop and display a value. You can do it like below. I am just printing free memory. But really you can do anything.

while true; do free -m | grep /+ | gawk '{ print $4 }'; sleep 2; done
Switch to root:
sudo su root
Add Text to File:
echo "##TextToAdd##" >> ##FileNameToAddTo##
Add Text Above Other Text:
sed -i 's/^##LookForText##/##TextToAdd##\n&/' ##FileNameToAddTo##
Loop:
until [ $Variable -gt 3 ]
do
done