Oliver

NGINX + UWSGI + Ubuntu 16

If you use Python as part of your stack in developing a web architecture you most likely will also be utilizing NGINX with UWSGI together. In the following article I will give step by step instructions on how to set this up assuming you are using sockets as part of your installation.

What is UWSGI?: A full stack for building application servers for python and various other languages.

What is NGINX?: Is simply a web server or proxy server for various protocols but can also do load balancing, etc.

NGINX Installation:

sudo apt-get install nginx
 
#The next command checks that nginx is running
ps -auxw | grep nginx
 
#Stop the service
sudo service nginx stop
#Start the service
sudo service nginx start
#Restart the service
sudo service nginx restart
#nginx status
sudo service nginx status

Once it is installed you will have two folders. One called “sites-enabled” and one called “sites-available” located under /etc/nginx/ folder. The “sites-enabled” folder usually holds a symbolic link to the actual file sometimes in “sites-available” folder. But really can be hosted anywhere on the machine.

If you want to remove the default site link you can

sudo rm /etc/nginx/sites-enabled/default

Create our symbolic link in sites-enabled if we already have a nginx.conf file ready.

cd /etc/nginx/sites-enabled/
 
sudo ln -s ~/path/to/nginx/conf/nginx.conf nginx.conf

It’s important to note that you may run into permissions errors depending on where you are running your nginx.conf from. So you may need to grant permissions to your users home folder.

sudo chmod 755 /home/user_your_run_your_app_from/

UWSGI Installation:

sudo apt-get install uwsgi
 
#Stop uwsgi
sudo service uwsgi stop
#Start uwsgi
sudo service uwsgi start
#Restart uwsgi
sudo service uwsgi restart
#uwsgi status
sudo service uwsgi status

It you want to log to a custom folder your uwsgi.ini file will need “logto” folder specified and the correct permissions set.

cd /var/log/
sudo touch /var/log/mylogfile
sudo chmod 666 /var/log/mylogfile

You uwsgi.ini will also have a pid file “pidfile” which we will need to create.

cd /tmp/
touch mypid.pid
 
#Ownership
sudo chown THEUSER:www-data mypid.pid
 
#Ensure permissions are set to -rw-r--r--

UWSGI INI Options:

You can add whatever options you prefer as part of your uwsgi.ini setup file. However the below are usually the ones you should use.

[uwsgi]
socket = :5006 #or whatever socket you want
chmod-socket = 777
pidfile=/tmp/mypid.pid
master = True
chdir = /home/user_your_run_your_app_from/
 
#NOTE that you may or may not need this depending on how you setup your server.
plugin = python3
wsgi_file = my_python_class.py #File that runs your application
module = whatever
callable = app #Set default WSGI callable name.
logto = /var/log/mylogfile

These are not all the options you can pick. You can also set “stats”, “buffer-size”, “processes”, “threads”, “env”, etc. Just review the docs and pick what is right for you.

The option “env” you can add many times and for each time you use it you can set environment variable for use with your application.

NOTE: If you are running flask and use send_file and run as Python3.5 you may get the error “SystemError: <built-in function uwsgi_sendfile> returned a result with an error set”. If you do then you must add the following to your uwsgi.ini file “wsgi-disable-file-wrapper = true”.

NGINX Conf Example:

upstream flask {
    server 0.0.0.0:5006;
}
server {
    client_max_body_size 1G;
    listen 0.0.0.0:8081;
    server_name localhost;
    access_log /var/log/nginx/access_log combined;
	
    location / {
        include uwsgi_params;
        uwsgi_pass flask;
    }
    location /static/ {
        alias /path/to/my/app/static/;
    }
}
fastcgi_intercept_errors on;

UWSGI Service:

Sometimes we want to run our app as a service if you do then below are the steps for that.

sudo nano /etc/systemd/system/myawesomeservice.service
 
#Enter the contents below
 
[Unit]
Description=uWSGI instance to serve My Service
After=network.target
 
[Service]
User=user_to_run_as
Group=www-data
WorkingDirectory=/path/to/my/app/
ExecStart=/usr/bin/env bash -c 'uwsgi --ini /path/to/my/app/uwsgi.ini --uid user_to_run_as --gid www-data'
Restart=always
RestartSec=3
 
[Install]
WantedBy=multi-user.target

Next you need to ensure the user has access to your folder with correct ownerships.

chown -R user_to_run_as:www-data /path/to/my/app/*

Next enable the service and start the service.

sudo systemctl enable myawesomeservice
sudo systemctl daemon-reload
sudo systemctl start myawesomeservice
sudo service myawesomeservice status

JavaScript: Download Object Data

Sometimes you just want to send an object to be downloaded in which you don’t have any special requirements just download the data. To do so is really straight forward. Create the blob with the type then create the blobs object url.

var blob = new Blob([JSON.stringify(myDataObject)], {type: "application/force-download"});
url  = URL.createObjectURL(blob);
 
<a target="_blank" href={url} download="file.extension" />

1974 Mustang II Center Console Redone

In July 2017 I had the center console redone. I got some of the replacement parts from Phil Schmidt at Mustang II Speciality Shop New Used Obsolete 1974-1978 Ford Mustang II Parts (ash tray, coin tray, emergency brake rubber, automatic selector surround, center council brackets and the placement diagram). I also had Brett McHugh refinish the center console so it would be back to it’s original state and it did!

Before Pics:

After Pics:

Python: Get First Occurrence of Value in Array

If you are searching for the first value in an array. There are a few ways to do this. I will show you the easiest to me.

You can do this with the following. It will return 34 as the result.

val_list = [0,0,34,21,0,657,345,32,34,5,8,9]
 
result = next(x for x in val_list if x > 0)
print(result)

Python: Working with DateTimes

In this tutorial I will show you the different ways of working with dates and times in python. Which includes working with milliseconds. You should note this isn’t all available options just some that I have encountered over the years.

Install Python Packages:

Open cmd/terminal and if required navigate to your sites working folder. (note: if you are working in a virtual env you should ensure you source it first).

pip install python-dateutil

There are many different packages that we can use to work with date and times. You need to decide what is right for you.

dateutil:

The following will convert the date string you give it fast and easily. This gives you back the datetime object. Notice how we don’t need to pass it a date time format. To me this is very convenient.

from dateutil import parser
 
date_str = '2017-06-06'
date_time_str = '2017-06-07 12:34'
date_time_str_2 = '2017-06-07 12:34:46'
date_time_str_3 = '2017-06-07 12:34:42.234'
 
result = parser.parse(date_str)
print(result) #2017-06-06 00:00:00
result = parser.parse(date_time_str)
print(result) #2017-06-07 12:34:00
result = parser.parse(date_time_str_2)
print(result) #2017-06-07 12:34:46
result = parser.parse(date_time_str_3)
print(result) #2017-06-07 12:34:42.234000

datetime:

The following will convert the date string you give it fast and easily. This gives you back the datetime object. Notice how we need to pass the format of the datetime. If you don’t you will get an exception. This is a convenient way if you know the format before hand. But that might not always be the case.

import datetime
 
date_str = '2017-06-06'
date_time_str = '2017-06-07 12:34'
date_time_str_2 = '2017-06-07 12:34:46'
date_time_str_3 = '2017-06-07 12:34:42.234'
 
result = datetime.datetime.strptime(date_str, "%Y-%m-%d")
print(result) #2017-06-06 00:00:00
result = datetime.datetime.strptime(date_time_str, "%Y-%m-%d %H:%M")
print(result) #2017-06-07 12:34:00
result = datetime.datetime.strptime(date_time_str_2, "%Y-%m-%d %H:%M:%S")
print(result) #2017-06-07 12:34:46
result = datetime.datetime.strptime(date_time_str_3, "%Y-%m-%d %H:%M:%S.%f")
print(result) #2017-06-07 12:34:42.234000

The above all works however the following example will not. Why do you think this is?

import datetime
 
date_time_str = '2017-06-07 12:34:46'
 
try:
    datetime.datetime.strptime(date_time_str, "%Y-%m-%d %H:%M:%S")
except:
    pass #just for this example don't do this lol

The reason is because datetime expects the correct format to be supplied. We gave it hour minute second but not milliseconds. You will get the following exception (ValueError: unconverted data remains: .234)

Timestamps:

Sometimes we want to convert the date to unix (epoch) time or vise versa.

From Date:

from dateutil import parser
from datetime import timezone
 
date_time_str = '2017-06-07 17:34:42.234'
result = parser.parse(date_time_str)
 
timestamp = result.replace(tzinfo=timezone.utc).timestamp()
print(timestamp) #1496856882.234

This gives us the timestamp as a float as 1496856882.234.

From Timestamp:

from dateutil import parser
import datetime
 
timestamp = 1496856882.234
 
result = datetime.datetime.fromtimestamp(timestamp)
print(result) #2017-06-07 13:34:42.234000
 
result = datetime.datetime.utcfromtimestamp(timestamp)
print(result) #2017-06-07 17:34:42.234000

Get Date Parts:

If you want to get specific date parts such as the year, month, day, hour, etc.

import datetime
from dateutil import parser
 
result = parser.parse(date_time_str_3)
print(result) #2017-06-07 12:34:42.234000
 
year = result.year #2017
month = result.month #6
day = result.day #7
hour = result.hour #12
minute = result.minute #34
second = result.second #42
millisecond = result.microsecond #234000

Add To Date:

If you want to add time to a date.

import datetime
from dateutil import parser
from datetime import timezone, timedelta
 
date_time_str = '2017-06-07 17:34:42.234'
result = parser.parse(date_time_str)
print(result) #2017-06-07 17:34:42.234000
 
timestamp = result.replace(tzinfo=timezone.utc).timestamp()
print(timestamp) #1496856882.234
 
#Add 10 seconds to datetime
new_time = int((datetime.datetime.fromtimestamp(timestamp) + timedelta(milliseconds=10000)).timestamp() * 1000)
print(new_time) #1496856892234

As you can see you can 10 seconds has been added the datetime.

datetime strftime

from datetime import datetime
 
now = datetime.now()
datetime_str = now.strftime("%Y-%m-%d %H:%M:%S")
print(datetime_str)

datetime fromisoformat

from datetime import datetime
 
print(datetime.fromisoformat("2024-04-09 13:48:20"))

1974 Mustang II Timing Cover Fix

In June 2017 while getting my tach adapter put on I noticed that I was leaking coolant yet again. I had just had the water pump replaced but this time it was coming from the timing cover itself. In order to take off the timing cover I had to take off all the puly’s, the water pump the radiator, the front grill, the bumper. It was a lot of work that as I am writing this is still ongoing. I also found that a head light was cracked so I am also waiting on that. Determining the right gasket set was a little difficult as my engine seems to be made up of a lot of different year parts. For example my water pump came from a 70’s (D00E-D). The timing cover is RP-E5AE-6058-FA, Top timing cover sprocket E3AE-A3.

Before Pics:

Once I got it all back together the pressure was good had a tiny leak on the water pump but after I snugged it a little more and added a missing bolt I didn’t notice till after all worked well.

After Pics:

Javascript: Find

If we want to find and return the first occurrence from our object array.

var rec = myObjectArray.find(value, index) { return value.id==mySearchId; });

Javascript: Filter

If you want to filter an array of objects for only specific records that have a key/value, etc.

var records = myObjectArray.filter(function(value, index) { return value.id==id});

Javascript: JSON

This post shows you how to parse string to json and then back again.

String to Json

var doc = null;
	
try {
	doc = JSON.parse(value);
}
catch(ex) {
	//handle exception
}

Json to string

JSON.stringify(value));

Javascript: Math Functions

In this post I will show you how to perform math operations such as min, max, etc.

Min:

//Array
var maxValue = Math.min.apply(Math, myArray);
 
//Object Array
Math.min.apply(Math,myObjectArray.map(function(v){return v,key;}));

Max:

//Array
var maxValue = Math.max.apply(Math, myArray);
 
//Object Array
Math.max.apply(Math,myObjectArray.map(function(v){return v,key;}));

Sum:

var sumValue = myArray.reduce(function(a, b) { return a + b; }, 0);

Average:

var avgValue = sumValue / myArray.length;

Standard Deviation:

var stdevValue = Math.sqrt(sumValue);

Python: CSV from Array

In this tutorial I will explain how to turn an array to a csv file. I will show you two ways. One is in memory and the other is to a file.

For both ways you need to import csv and io package.

import csv, io

Way 1 Write (In Memory):

#Create the string buffer
output = io.StringIO()
 
#Setup the csv writer to write the results to a string buffer
wr = csv.writer(output, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

Way 2 Write (File):

#Crate the file itself in write mode
f = open('filename.csv', 'w')
 
#Setup the csv writer to write the results to a file.
wr = csv.writer(f, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)

Technically both ways have the same setup for the csv writer. Then to write results to the csv writer you then pass an array of values like below.

wr.writerow(['123',5,4,'value'])

To Read the contents of the file or string buffer depends on which way you chose. I show you those ways below.

Way 1 Read (In Memory):

b = bytes(output.getvalue(), 'utf-u')

Way 2 Read (File):

f.close()
file_data = open('filename.csv', 'r').read()

If you want to send the file down using something like flask send_file then you need to convert it to BytesIO.

buffer = BytesIO()
buffer.write(b)
#You must seek to beginning otherwise it won't send anything back.
buffer.seek(0)

Now if you are sending it as a file back to the user and are using something like flask this is how you do that. Pretty straight forward.

return send_file(buffer, mimetype='application/octet-stream', as_attachment=True, attachment_filename='myFile.csv')

Python: Flask Resource

This tutorial helps setup flask restful api’s.

Install Python Packages:

Open cmd and navigate into your testApp folder and run the following commands.

pip install flask-RESTful && pip freeze > requirements.txt

init.py:

On the init of your application you will need to setup flask_restful. There are config options you could set for config.py. Look into it!

from flask_restful import Api
api = Api(app)
 
#Add api endpoints
#Get
api.add_resource(home.views.MyResource, '/home/getMyData/')
 
#Post
api.add_resource(home.views.MyResource, '/home/getMyData/', methods=['POST'])

Setup home views.py:

You need to import Resource in it’s most simplistic form. However if you want to deal with request parameters add in reqparse and inputs. Inputs give you access to boolean that way a boolean can be parsed into a python boolean easily.

from flask_restful import Resource, reqparse, inputs

You can now use get, post, etc. I will give you three examples below.

Get:

class MyResource(Resource):
	def get(self):
		return {}

Get /w Parameter:

class MyResource(Resource):
	def get(self, var):
		return {}

Get /w Parameter & Request Parameter:

class MyResource(Resource):
	def get(self, var):
        	parser = reqparse.RequestParser()
        	parser.add_argument('new_request_var', type=str, default='')
 
        	#If you want to have a boolean request parameter do the following.
        	parser.add_argument('new_request_var_bool', type=inputs.boolean, default=False)
 
        	args = parser.parse_args(strict=True)
        	new_request_var = args['new_request_var']
        	new_request_var_bool = args['new_request_var_bool']
 
		return {}

Post:

class MyResource(Resource):
	def post(self):
		return {}

CSS: Bootstrap Panel

In this tutorial I will show you how how to use bootstrap with React/Node to make a panel using just bootstrap.

You will need bootstrap and jquery.

npm install bootstrap@3.3.7 --save
npm install jquery@3.2.1 --save
npm install file-loader --save
npm install url-loader --save

You will also need to update webpack.config.js and add the following under “loaders”.

{ test: /\.eot(\?v=\d+\.\d+\.\d+)?$/, loader: "file-loader" },
{ test: /\.woff(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?limit=10000&mimetype=application/font-woff" },
{ test: /\.woff2(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?limit=10000&mimetype=application/font-woff" },
{ test: /\.ttf(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?limit=10000&mimetype=application/octet-stream" },
{ test: /\.svg(\?v=\d+\.\d+\.\d+)?$/, loader: "url-loader?limit=10000&mimetype=image/svg+xml" }

Next you need to require bootstrap and jQuery.

window.jQuery = window.$ = require("jquery");
require("bootstrap");

You will need to also require the bootstrap css.

require("bootstrap/dist/css/bootstrap.min.css");

The following is all you then need to display a css like panel.

<div className="panel panel-default">
	<div className="panel-heading">
		<h3 className="panel-title">Panel Title</h3>
	</div>
	<div className="panel-body">
		Panel Text
	</div>
</div>

unRaid: Midnight Commander

If you are not familiar with unRaid go here. unRaid is a raid 4 implementation. If is basically a software raid.

If you want to work with the file system of unRaid whether it to be to move files or for another reason use midnight commander.

Start putty and connect to the unRaid server. Then once logged in run the following command for midnight commander.

mc

Postgres: String to Rows

In this tutorial I will show you how to convert a string to rows delimited by a character.

There are three ways to do this all lead to the same answer.

--METHOD 1
SELECT split_data FROM regexp_split_to_table('I love programming!', E' ') AS split_data;
 
--METHOD 2
SELECT split_data 
FROM unnest(string_to_array('I love programming!',' ')) AS split_data;
 
--METHOD 3
SELECT split_string.arr[i] as split_data
FROM (
	SELECT generate_series(1, array_upper(arr, 1)) AS i, arr
	FROM (SELECT ARRAY['I','love','programming!'] arr) t
) split_string;

Results:

Each executes in 11ms based on my analysis. Preferably I would go with “regexp_split_to_table”

1974 Mustang II Dash Wiring Repair & Update

In 2016 I also decided to redo a lot of the dash. The cluster cover was not correct and had a big hole in it, missing cluster plastic cover. No vents. Gauges in wrong areas. Bad wiring and interior lights not working.

I got a lot of parts from from Phil Schmidt at Mustang II Speciality Shop New Used Obsolete 1974-1978 Ford Mustang II Parts such as cluster plastic, cover, wire connectors, tach reducer and vents.

Here are the before pictures.

I decided to get rid of the custom tachometer that I had mounted to the steering column and go with the original tach. In order to do this I needed a tach reducer and a MSD tach adapter 8920. I don’t have the tach adapter yet but I did find a tach reducer and had that mounted in preparation. Also my dash cluster’s connector was broken so I had to get that replaced. I also had to replace the tach connector as that was not the correct one as well.

Here is it finally all done.

1974 Mustang II Engine Harness Replacement

In 2016 I decided to completely redo the engine harness. I was constantly getting battery draining issues. Everytime I turned on my blinker the volts gauge would jump all over the place.

I will admit when I first started this project I thought ah this shouldn’t take too long to do. I was definitely wrong. It took almost a month of work. I had to find wiring diagrams and get them blown up so I could read them. I also got them laminated because I would be full of grease and they would be easily cleaned.

I had the car turned around in the garage so the front was facing out so I could work easier. Mike from Auto Electric MD taught and helped me redo all of the electrical in the engine bay and straight through to the dashboard. If you want to see the dashboard electrical restore I will be putting that up shortly.

Here are the before pictures.

After removing all the electrical tape from the wires I found that a good majority of the wire was corroded, connected improperly and completely overused. The voltage regulator was not setup properly. The 4th wire on the voltage regulator was why the car wasn’t charging.

Little by little we would trace wires in and compare to what the diagram said. I think labelling took the longest. Because you have to ensure everything is correctly mapped as to how it came off. We even had to disconnect the dashboard area to ensure wires went to where they actually said they went.

In trying to do replacements we had to disconnect the starter to replace and re-route cabling. That was a major part of the redo. We identified early on that a lot of what had been done could be routed better. The less cable the better. In disconnecting the starter it cracked the solenoid. Yes it actually cracked… The heat from the headers made the starter brittle. The plan was to put a heat cover on the starter next summer. So we had to replace the starter now. If you have ever worked on a 1974 Mustang II with a 302 engine. You will find that this is not done easily. We had to jack the engine in order to pull out the starter. What a nightmare to get that done but we did it. The alternator wasn’t setup correctly as well because the contact points were too close to the valve covers. So Mike had to rebuild the starter and alternator before we could continue. The voltage regulator seemed to not be correct as well so we put in a better one. All the head lights were redone. new connectors, etc.

Here are some pictures of the work as we were going through it.

By the time we were done we had probably 5 pounds of over used wire. No joke! At this time the tack still doesn’t work but it will summer 2017!

1974 Mustang II Exhaust Replacement

In summer 2016 I decided to get the exhaust replaced. I was looking pretty manky. It was never done right so it was time.

Here are the before pictures. Pretty shotty job if I do say so myself.

I brought it over to Carlane Auto Centre in Guelph, Ontario. They estimated and I got it done the right way. I believe it took them a day to get it all done. It looked amazing!

Here are the after pictures.

All in all very happy with the new exhaust system!

JavaScript: Node & Lodash

In this tutorial we will be giving out some quick examples of Lodash functionality. To know more go here. There are so many examples and ways of doing Lodash. Check out the documentation.

Q: What is Lodash you ask?
A: It is a toolkit of Javascript functions that provides clean, performant methods for manipulating objects and collections. It is a “fork” of the Underscore library and provides additional functionality as well as some serious performance improvements.

First thing we need to do is install it. You will need a node site already ready to go. If you don’t have one you can follow this tutorial on setting a basic one up.

npm install lodash --save

On whatever page you are working with all you need to do is add the following to where your requires are.

var _ = require('lodash');

Now we can use the functionality as we wish. I will do some basic uses below.

Array difference:

If we want to find the difference of an array to the second array. The result would be “1” because 1 is not in the second array. Notice how it does not compare the second array to the first. It’s only checking which values 2 or 1 don’t exist in the second array.

_.difference([2, 1], [2, 3])

Array uniqWith:

If you want to get the unique items in an array you could use the following. It would return “2 45 3 7 8 1” only notice that the additional 45 is not displayed. It has been removed.

_.uniqWith([2, 45, 3, 7, 8, 45, 1], __.isEqual)

NGINX Installation:

UWSGI Installation:

UWSGI INI Options:

NGINX Conf Example:

UWSGI Service:

Before Pics:

After Pics:

Install Python Packages:

dateutil:

datetime:

Timestamps:

From Date:

From Timestamp:

Get Date Parts:

Add To Date:

datetime strftime

datetime fromisoformat

Before Pics:

After Pics:

String to Json

Json to string

Min:

Max:

Sum:

Average:

Standard Deviation:

Way 1 Write (In Memory):

Way 2 Write (File):

Way 1 Read (In Memory):

Way 2 Read (File):

Install Python Packages:

__init__.py:

Setup home views.py:

Results:

Array uniqWith:

init.py: