Javascript: Map

You can use the map function to return different results from an array to return a new array with new data.

For example if you want to build an array of controls you could do the following.

  1. var newControls = myDataArray.map(function(rec, index){
  2. return <div></div>;
  3. });

 

AWS: Send Simple Email Service

This entry is part 5 of 5 in the series AWS & Java

If you want to send an email using AWS’ Simple Mail then you need to do the following. This is a very basic example.

Import the following:

  1. import com.amazonaws.auth.AWSStaticCredentialsProvider;
  2. import com.amazonaws.auth.BasicAWSCredentials;
  3. import com.amazonaws.services.simpleemail.AmazonSimpleEmailService;
  4. import com.amazonaws.services.simpleemail.AmazonSimpleEmailServiceClientBuilder;
  5. import com.amazonaws.services.simpleemail.model.Body;
  6. import com.amazonaws.services.simpleemail.model.Content;
  7. import com.amazonaws.services.simpleemail.model.Destination;
  8. import com.amazonaws.services.simpleemail.model.Message;
  9. import com.amazonaws.services.simpleemail.model.SendEmailRequest;

Setup Connection to AWS Simple Email Service

  1. final AmazonSimpleEmailService simpleEmailService = AmazonSimpleEmailServiceClientBuilder.standard().withRegion(myRegion)
  2. .withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(accessKeyId, secretKey)))
  3. .build();

Setup Email:

  1. final SendEmailRequest request = new SendEmailRequest().withDestination(new Destination().withToAddresses(TO)).withSource(FROM)
  2. .withMessage(new Message().withSubject(new Content().withCharset("UTF-8").withData(SUBJECT))
  3. .withBody(new Body().withText(new Content().withCharset("UTF-8").withData(BODY))));

Send Email:

  1. simpleEmailService.sendEmail(request);

AWS: Java Post to Kinesis Queue

This entry is part 4 of 5 in the series AWS & Java

Posting to an AWS Kinesis Queue is rather simple and straight forward. As always you should refer to AWS Documentation.

Put Multiple Records On Queue

Import the following

  1. import com.amazonaws.auth.AWSStaticCredentialsProvider;
  2. import com.amazonaws.auth.BasicAWSCredentials;
  3. import com.amazonaws.services.kinesis.AmazonKinesis;
  4. import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
  5. import com.amazonaws.services.kinesis.model.PutRecordsRequest;
  6. import com.amazonaws.services.kinesis.model.PutRecordsRequestEntry;
  7. import com.amazonaws.services.kinesis.model.Record;

Put Records

  1. AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard().withRegion(myRegion).withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(myAccessKeyId, mySecretKey)));
  2. AmazonKinesis kinesisClient = clientBuilder.build();
  3. PutRecordsRequest putRecordsRequest = new PutRecordsRequest();
  4. putRecordsRequest.setStreamName(myQueue);
  5. List putRecordsRequestEntryList = new ArrayList<>();
  6.  
  7.  
  8. //You can put multiple entries at once if you wanted to
  9. PutRecordsRequestEntry putRecordsRequestEntry = new PutRecordsRequestEntry();
  10. putRecordsRequestEntry.setData(ByteBuffer.wrap(myData));
  11. putRecordsRequestEntry.setPartitionKey(myKey);
  12. putRecordsRequestEntryList.add(putRecordsRequestEntry);
  13.  
  14.  
  15. putRecordsRequest.setRecords(putRecordsRequestEntryList);
  16. PutRecordsResult putResult = kinesisClient.putRecords(putRecordsRequest);

Put Single Record On Queue

Import the following

  1. import com.amazonaws.auth.AWSStaticCredentialsProvider;
  2. import com.amazonaws.auth.BasicAWSCredentials;
  3. import com.amazonaws.services.kinesis.AmazonKinesis;
  4. import com.amazonaws.services.kinesis.AmazonKinesisClientBuilder;
  5. import com.amazonaws.services.kinesis.model.PutRecordRequest;
  6. import com.amazonaws.services.kinesis.model.Record;

Put Record

  1. AmazonKinesisClientBuilder clientBuilder = AmazonKinesisClientBuilder.standard().withRegion(myRegion).withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(myAccessKeyId, mySecretKey)));
  2. AmazonKinesis kinesisClient = clientBuilder.build();
  3. PutRecordRequest putRecordRequest = new PutRecordRequest();
  4. putRecordRequest.setStreamName(myQueue);
  5.  
  6. putRecordRequest.setData(ByteBuffer.wrap(data.getBytes("UTF-8")));
  7. putRecordRequest.setPartitionKey(myKey);
  8.  
  9. PutRecordResult putResult = kinesisClient.putRecord(putRecordRequest);

You now have put a record(s) onto the queue congratulations!

HortonWorks: Kerberize Ambari Server

This entry is part 7 of 7 in the series HortonWorks

You may want to integrate Kerberos authentication into your Ambari Server implementation. If you do follow the next few steps. It’s that easy.

Step 1: Stop Ambari Server

  1. sudo ambari-server stop

Step 2: Create keytab file

  1. ktutil
  2. addent -password -p ##USER##@##DOMAIN##.COM -k 1 -e RC4-HMAC
  3. # Enter password
  4. wkt ##USER##.keytab
  5. q
  6. $ sudo mkdir /etc/security/keytabs
  7. $ mv ##USER##.keytab /etc/security/keytabs

Step 3: Test Keytab. You should see the ticket once you klist.

  1. kinit -kt /etc/security/keytabs/ambarisa.keytab -a ambarisa@AERYON.COM
  2. klist

Step 4: Run Ambari Server Kerberos Setup

  1. sudo ambari-server setup-kerberos

Follow the prompts. Say true to enabling kerberos. The keytab file will be the /etc/security/##USER##.keytab file. You should be able to leave the rest defaults. Save the settings and you are done.

Step 5: Remove the kinit ticket you created that way you can make sure you kerberos authentication is working correctly.

  1. kdestroy

Step 6: Start Ambari Server

  1. sudo ambari-server start

Step 7: Validate Kerberos. You should see your ticket get created and you should now be able to login with no issues.

  1. klist

HortonWorks: Install YARN/MR

This entry is part 6 of 7 in the series HortonWorks

This tutorial guides you through installing YARN/MapReduce on Hortonworks using a multi node cluster setup with Ubuntu OS.

Step 1: Go to “Stack and Version”. Then click “Add Service” on YARN. You will notice that “MapReduce2” comes with it.

Step 2: Assign Masters I usually put the ResourceManager, History Server and App Timeline Server all on the secondary namenode. But it is totally up to you how you setup your environment.

Step 3: Assign Slaves and Clients I put NodeManagers on all the datanodes and Client’s on all servers. Up to you though. This is what worked for me and my requirements.

Step 4: During Customize Services you may get the warning that Ambari Metrics “hbase_master_heapsize” needs to be increased. I recommend doing this change but it’s up to you and what makes sense in your environment.

Step 5: Follow the remaining steps and installation should complete with no issues. Should an issue arise review the error and if it was just a turning on connection error then you may not have any issues and it just needs all services to be stopped and started again. Please not Ambari Metrics may report errors but they should clear in around 15 minutes.

 

HortonWorks: Install Hadoop

This entry is part 4 of 7 in the series HortonWorks

This tutorial guides you through installing Hadoop on hortonworks using a multi node cluster setup with Ubuntu OS.

Hosts File:

Ensure every server has the FQDN of all the servers to be in the cluster.

  1. sudo nano /etc/hosts

SSH (Ambari Server)

You will do the following to all the servers in the Hadoop cluster.

  1. ssh-keygen -t rsa -P ""
  2. cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
  3. ssh-copy-id -i ~/.ssh/id_rsa.pub ##USER##@##FQDN##
  4. ssh ##USER##@##FQDN##

Pre-Requisites: (not Ambari Server)

Java 8:

  1. sudo apt-get install openjdk-8-jdk

Chrony:

  1. sudo apt-get install chrony

Disable HugePage:

  1. sudo su
  2. echo never > /sys/kernel/mm/transparent_hugepage/enabled
  3. exit

Install HDFS Service:

You will need to login to Ambari Server and click “Launch Install Wizard”. For the most part you will just follow the prompts. The major hurdles is that in the “Install Options” section make sure you put the FQDN (IE: host@domain.com). You will also need to get the SSH Private Key from the Ambari Server you just did during pre requisites from this location /home/##USER##/.ssh/id_rsa. Make sure you also set the SSH User Account to what you used during SSH creation. If for any reason it fails you can click the status to find out what failed and rectify the problem. As long as you did the pre-requisites you should be fine.

ZooKeeper / Ambari Metrics

As you install HDFS you will notice that Ambari Metrics and ZooKeeper get installed automatically. This is a good thing and you want it. ZooKeeper keeps all configs in sync and Ambari Metrics lets you easily monitor the system.

Assign Masters

You will need to setup how you want your masters to look. I usually have three zookeepers. Your secondary name node should go on a separate server. But it is totally up to you how you design your cluster. Have fun!

Assign Slaves / Clients

Your slaves (aka DataNodes) I don’t put any on my namenode or secondary namenode or my zookeeper servers. I leave my datanodes to perform that action alone. I also install clients on namenode, secondary namenode and all datanodes. Up to you how you configure it just have fun while doing it!

Key Config Optional Changes

Once you get to the customize services section. You can for the most part leave this as is and just do the password areas. But I do recommend reviewing the following and update as needed.

  1. HDFS: NameNode/DataNode/Secondary NameNode directories
  2. ZooKeeper: ZooKeeper directory

Deploy

Deploy should work with no issues. If there is issues sometimes you don’t need to worry about it. Such as connection issue. As long as it installed if it didn’t start right away and that was the connection issue then it may start once completed. You should also note that Ambari Metrics shows errors directly after starting. That is expected and no need to worry it will clear itself.

HortonWorks: SSL Setup

This entry is part 3 of 7 in the series HortonWorks

If you want to use SSL with Ambari Server (note this is not with Hadoop yet) then follow the below steps. Please note this does not cover the creation of a SSL Cert as there are many tutorials on how to create self signed certs, etc available.

Step 1: Stop the Ambari Server

  1. sudo ambari-server stop

Step 2: Run Ambari Server Security Setup Command

  1. sudo ambari-server setup-security

Select option 1 during the prompts and note that you cannot use port 443 for https as that is reserved in Ambari. The default is 8443 and that is what they recommend. Enter path to your cert /etc/ssl/certs/hostname.cer file. Enter path to your encrypted key /etc/ssl/private/hostname.key file. Follow the rest of the prompts.

Step 3: Start Ambari Server

  1. sudo ambari-server start

Step 4: Login to Ambari Server now available at https://hostname:8443

HortonWorks: Ambari LDAP Integration

This entry is part 2 of 7 in the series HortonWorks

If you want to use LDAP with your Ambari Server then follow the below steps.

Step 1: Stop the Ambari Server to setup LDAP integration

  1. sudo ambari-server stop

Step 2: Run Ambari Server LDAP Setup Command. This will require a bunch of settings to be set. Consult your IT department for your specific settings.

  1. sudo ambari-server setup-ldap

Step 3: Create the groups and users text files and add the users you want to add comma separated to users and groups comma separated to the groups file.

  1. nano ~/groups.txt
  2. nano ~/users.txt

Step 4 (Optional): You may need to adjust the “SERVER_API_HOST” value to your ambari server hostname. Default is 127.0.0.1 which is technically your host but sometimes it complains and you need to make this modification.

  1. sudo nano /usr/lib/python2.6/site-packages/ambari_server/serverUtils.py

Step 5: Import Groups/Users from the text files created in step 3. You will need to start the ambari server first.

  1. sudo ambari-server start
  2. #Import groups
  3. sudo ambari-server sync-ldap --groups groups.txt
  4. #Import users
  5. sudo ambari-server sync-ldap --users users.txt

Step 6: Login to Ambari and got to manage ambari and you will see your new users and groups.

HortonWorks: Ambari Server Installation

This entry is part 1 of 7 in the series HortonWorks

I have been playing around with HortonWorks sandbox and thought it about time I attempt installation on a multi node cluster. Feel free to reach out to me for further support or information. I will be documenting more in the coming weeks.

Add Repo:
  1. wget -O /etc/apt/sources.list.d/ambari.list http://public-repo-1.hortonworks.com/ambari/ubuntu16/2.x/updates/2.6.0.0/ambari.list
  2. sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
  3. sudo apt-get update
Java 8
  1. sudo apt-get install openjdk-8-jdk
Ambari Install
  1. sudo apt-get install ambari-server
Configure Ambari

I recommend installed as non root. In fact do not do any of the defaults for users and passwords. But it is totally up to you. I will document more as I learn more.

  1. sudo ambari-server setup
Ambari Start / Stop / Restart
  1. sudo ambari-server restart
  2. sudo ambari-server start
  3. sudo ambari-server stop
Ambari Server Log Directory

I suggest changing the ambari log directory as well.

  1. sudo vi /etc/ambari-server/conf/log4j.properties
  2.  
  3. #Look for the property "ambari.log.dir" and change it.
  4. #Don't forget to create the folder you point to and ensure that chown to the user that is running ambari.
Ambari Agent Log Directory

As well change the ambari agent log directory.

  1. sudo vi /etc/ambari-agent/conf/ambari-agent.ini
  2.  
  3. #Look for "logdir" and change to a directory that exists and has permissions.
Non-Root Install

Running as non-root user you will have to change the run directory because otherwise during a system reboot you may get folders not being able to be created. Because /var/run/ gets deleted on each restart and it get’s rebuilt. Do the following:

  1. sudo mkdir -p /home/##NONROOTUSER##/run/ambari-server
  2. sudo chown ##NONROOTUSER##:root /home/##NONROOTUSER##/run/ambari-server

Then you have to edit the ambari.properties file.

  1. sudo vi /etc/ambari-server/conf/ambari.properties
  2.  
  3. #Change the following to the folder you created above.
  4. bootstrap.dir
  5. pid.dir
  6. recommendations.dir

Postgres: dblink

Sometimes we need to connect to external database from within a query. There is an easy way to do this however before doing this make sure it meets your business requirements.

  1. SELECT *
  2. FROM dblink('dbname=##DBNAME## port=##PORT## host=##DBSERVER## user=##DBUSER## password=##PWD##', 'SELECT my fields FROM ##TABLE##') AS remote_data(my integer, fields varchar)

Hadoop: Secondary NameNode

By default a secondary namenode runs on the main namenode server. This is not ideal. A secondary namenode should be on it’s own server.

First bring up a new server that has the exact same configuration as the primary namenode.

Secondary NameNode:

  1. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

Remove property “dfs.namenode.secondary.http-address” and “dfs.namenode.name.dir” as they are unneeded.

Then add the following property. Making sure to change to the path you will store your checkpoints in.

  1. <property>
  2. <name>dfs.namenode.checkpoint.dir</name>
  3. <value>file:/usr/local/hadoop_store/data/checkpoint</value>
  4. </property>

NameNode:

  1. nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml

Then add the following property. Making sure to change ##SECONDARYNAMENODE##

  1. <property>
  2. <name>dfs.namenode.secondary.http-address</name>
  3. <value>##SECONDARYNAMENODE##:50090</value>
  4. <description>Your Secondary NameNode hostname for http access.</description>
  5. </property>

Now when you stop and start the cluster you will see the secondary name node now start on the secondary server and not on the primary namenode server. This is what you want.

 

Hadoop: Rack Awareness

If you want your multi node cluster to be rack aware you need to do a few things. The following is to be done only on the master (namenode) only.

  1. nano /home/myuser/rack.sh

With the following contents

  1. #!/bin/bash
  2.  
  3. # Adjust/Add the property "net.topology.script.file.name"
  4. # to core-site.xml with the "absolute" path the this
  5. # file. ENSURE the file is "executable".
  6.  
  7. # Supply appropriate rack prefix
  8. RACK_PREFIX=myrackprefix
  9.  
  10. # To test, supply a hostname as script input:
  11. if [ $# -gt 0 ]; then
  12.  
  13. CTL_FILE=${CTL_FILE:-"rack.data"}
  14.  
  15. HADOOP_CONF=${HADOOP_CONF:-"/home/myuser"}
  16.  
  17. if [ ! -f ${HADOOP_CONF}/${CTL_FILE} ]; then
  18. echo -n "/$RACK_PREFIX/rack "
  19. exit 0
  20. fi
  21.  
  22. while [ $# -gt 0 ] ; do
  23. nodeArg=$1
  24. exec< ${HADOOP_CONF}/${CTL_FILE}
  25. result=""
  26. while read line ; do
  27. ar=( $line )
  28. if [ "${ar[0]}" = "$nodeArg" ] ; then
  29. result="${ar[1]}"
  30. fi
  31. done
  32. shift
  33. if [ -z "$result" ] ; then
  34. echo -n "/$RACK_PREFIX/rack "
  35. else
  36. echo -n "/$RACK_PREFIX/rack_$result "
  37. fi
  38. done
  39.  
  40. else
  41. echo -n "/$RACK_PREFIX/rack "
  42. fi

Set execute permissions

  1. sudo chmod 755 rack.sh

Create the data file that has your rack information. You must be very careful not to have too many spaces between the host and the rack.

  1. namenode_ip 1
  2. secondarynode_ip 2
  3. datanode1_ip 1
  4. datanode2_ip 2

The last step is to update core-site.xml file located in your hadoop directory.

  1. nano /usr/local/hadoop/etc/hadoop/core-site.xml

Set the contents to the following of where your rack.sh file is located.

  1. <property>
  2. <name>net.topology.script.file.name</name>
  3. <value>/home/myuser/rack.sh</value>
  4. </property>

Python: Run Process

If you want to run a jar from python or really any process. You do so by leveraging subprocess package.

  1. from subprocess import Popen, PIPE

Then you need to call Popen. If you want to set java memory you can do so using -Xms and -Xmx in between java and -jar.

  1. #bufsize of 1 is line buffered
  2. #stdout and stderr to PIPE is to pipe the output of std out and std error to the PIPE so you can get the output
  3. result = Popen(['java -jar myapp.jar'], stdout=PIPE, stderr=PIPE, shell=False, bufsize=1)

If you want your process to wait until finished you will need to call wait.

  1. result.wait()

If you pushed the stderr and stdout then you can check the output.

  1. if result.stdout is not None:
  2. for line in result.stdout:
  3. print(line)
  4.  
  5. if result.stderr is not None:
  6. for line in result.stderr:
  7. print(line)

Python: Logging

If you want to do some basic logging to a file, etc. You can use the logging package that comes with python. Here are some of the basic ways to log.

You first have to import the package.

  1. import logging

You can setup your own logging configuration but for this we will just use the basic setup and log to a file.

  1. #If you are going to have multiple handlers you should setup your handler
  2. logging.root.handlers = []
  3.  
  4. #The file to log to
  5. log_file = /mnt/log/
  6.  
  7. #Setup the config with the level to log up to
  8. logging.basicConfig(filename=log_file, level=logging.INFO)

Then you setup your logger

  1. logger = logging.getLogger('my_awesome_log')

If you want your log to truncate after a certain size then you must add the handler for truncating the log and back. If you do not use the rotatingfilehandler then the log will increase till your drive runs out of space.

  1. handler = RotatingFileHandler(log_file, maxBytes=1024, backupCount=1)
  2. logger.addHandler(handler)

If you also want to log to console you will need to add an additional handler for the console setting the level to log.

  1. console = logging.StreamHandler()
  2. console.setLevel(logging.INFO)
  3. logger.addHandler(console)

That’s it a basic example of how to use the logging package.

 

Python: Multiprocessing Pool

Sometimes we want to run a method using multiple processors to process our code due to a costly function. Below is an example of how you could do it. There is other api’s you could use like ‘map’ but here is just one example.

  1. from multiprocessing import Pool
  2. # Sets the pool to utilize 4 processes
  3. pool = Pool(processes=4)
  4. result = pool.apply_async(func=my_method, args=("some_info",))
  5. # Performs the aync function
  6. data = result.get()
  7. pool.close()

Python: Selenium Tests

Selenium is a great way to test your UI. It is compatible with different browsers. I will show you two.

Gecko Driver Installation:

Make sure you are using latest version. At the time of this writing it is 0.19.0.

  1. wget https://github.com/mozilla/geckodriver/releases/download/v0.19.0/geckodriver-v0.19.0-linux64.tar.gz
  2. sudo tar -xvzf geckodriver-v0.19.0-linux64.tar.gz
  3. sudo chmod +x geckodriver
  4. cp geckodriver /usr/local/bin/
  5. sudo cp geckodriver /usr/local/bin/

You can use phantomjs, firefox, chrome, etc.

PhantomJS Installation:

  1. sudo mv phantomjs-2.1.1-linux-x86_64.tar.bz2 /usr/local/share/.
  2. cd /usr/local/share/
  3. sudo tar xjf phantomjs-2.1.1-linux-x86_64.tar.bz2
  4. sudo ln -s /usr/local/share/phantomjs-2.1.1-linux-x86_64 /usr/local/share/phantomjs
  5. sudo ln -s /usr/local/share/phantomjs/bin/phantomjs /usr/local/bin/phantomjs

Firefox Installation:

  1. sudo apt-get update
  2. wget https://ftp.mozilla.org/pub/firefox/releases/50.0/linux-x86_64/en-US/firefox-50.0.tar.bz2
  3. sudo tar -xjf firefox-50.0.tar.bz2
  4. sudo rm -rf /opt/firefox
  5. sudo mv firefox /opt/firefox
  6. sudo mv /usr/bin/firefox /usr/bin/firefoxold
  7. sudo ln -s /opt/firefoxX/firefox /usr/bin/firefox

Firefox Headless Installation:

  1. sudo apt-get install xvfb
  2. pip3 install pyvirtualdisplay==0.2.1

Selenium Installation:

  1. pip3 install selenium==3.6.0

PyUnit Selenium Test Examples:

Setup:

  1. #If you are using headless firefox
  2. from pyvirtualdisplay import Display
  3. #The selenium imports
  4. from selenium import webdriver
  5. from selenium.webdriver.common.by import By
  6. from selenium.webdriver.support import expected_conditions as EC
  7. from selenium.webdriver.support.ui import WebDriverWait
  8. import unittest, os, time
  9.  
  10. class MySeleniumTests(unittest.TestCase):
  11. @classmethod
  12. def setUpClass(self):
  13. self.server_url = "http://" + os.getenv("WEBSITE_URL", 'localhost:5000')
  14.  
  15. def setUp(self):
  16. #if you are using firefox headless browser
  17. display = Display(visible=0, size=(1080, 720))
  18. display.start()
  19. #Firefox selenium driver.
  20. self.driver = webdriver.Firefox()
  21. #PhantomJS selenium driver
  22. self.driver = webdriver.PhantomJS()
  23. self.driver.implicitly_wait(60)
  24. self.driver.set_page_load_timeout(60)
  25. self.driver.set_window_size(1080, 720)
  26. self.base_url = self.server_url
  27.  
  28. self.driver.get(self.base_url + "/")
  29. #If your site has a login then you need to set the username and password first.
  30. self.driver.find_element_by_id("user").clear()
  31. self.driver.find_element_by_id("user").send_keys(USERNAME)
  32. self.driver.find_element_by_id("password").clear()
  33. self.driver.find_element_by_id("password").send_keys(PWD)
  34. self.driver.find_element_by_id("submit").click()
  35. time.sleep(1)
  36.  
  37. def tearDown(self):
  38. self.driver.quit()
  39.  
  40. if __name__ == "__main__":
  41. unittest.main()

Test Title:

  1. self.driver.get(self.server_url)
  2. self.assertIn("MySite", self.driver.title)

Find Class:

  1. self.assertTrue(WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.CLASS_NAME, "my-awesome-class"))))

Find ID:

  1. self.assertTrue(WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.ID, "myId"))))

Find Partial Text:

  1. self.driver.find_element_by_partial_link_text("My Text On Page")

Find Element Contains Text:

  1. self.assertTrue('MyText' in self.driver.find_element_by_id('container').text)

Click Element:

  1. self.driver.find_element_by_id('myId').click()

Wait Element To Show:

  1. self.assertTrue(WebDriverWait(self.driver, 10).until(EC.text_to_be_present_in_element((By.ID, 'MyID'), "Text To See")))

xPath Click Second Element:

  1. self.driver.find_element_by_xpath("(//div[@class='my-awesome-class'])[1]").click()

Clear Input:

  1. self.driver.find_element_by_id("myId").clear()

Send Data To Input:

  1. self.driver.find_element_by_id("myId").send_keys('My New Data')

 

 

AWS: Java S3 Upload

This entry is part 3 of 5 in the series AWS & Java

If you want to push data to AWS S3 there are a few different ways of doing this. I will show you two ways I have used.

Option 1: putObject

  1. import com.amazonaws.AmazonClientException;
  2. import com.amazonaws.services.s3.model.ObjectMetadata;
  3. import com.amazonaws.ClientConfiguration;
  4. import com.amazonaws.auth.AWSCredentialsProvider;
  5. import com.amazonaws.regions.Regions;
  6. import com.amazonaws.services.s3.AmazonS3;
  7. import com.amazonaws.services.s3.AmazonS3ClientBuilder;
  8.  
  9. ClientConfiguration config = new ClientConfiguration();
  10. config.setSocketTimeout(SOCKET_TIMEOUT);
  11. config.setMaxErrorRetry(RETRY_COUNT);
  12. config.setClientExecutionTimeout(CLIENT_EXECUTION_TIMEOUT);
  13. config.setRequestTimeout(REQUEST_TIMEOUT);
  14. config.setConnectionTimeout(CONNECTION_TIMEOUT);
  15.  
  16. AWSCredentialsProvider credProvider = ...;
  17. String region = ...;
  18.  
  19. AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(credProvider).withRegion(region).withClientConfiguration(config).build();
  20.  
  21. InputStream stream = ...;
  22. String bucketName = .....;
  23. String keyName = ...;
  24. String mimeType = ...;
  25.  
  26. //You use metadata to describe the data.
  27. final ObjectMetadata metaData = new ObjectMetadata();
  28. metaData.setContentType(mimeType);
  29.  
  30. //There are overrides available. Find the one that suites what you need.
  31. try {
  32. s3Client.putObject(bucketName, keyName, stream, metaData);
  33. } catch (final AmazonClientException ex) {
  34. //Log the exception
  35. }

Option 2: MultiPart Upload

  1. import com.amazonaws.AmazonClientException;
  2. import com.amazonaws.event.ProgressEvent;
  3. import com.amazonaws.regions.Regions;
  4. import com.amazonaws.services.s3.AmazonS3;
  5. import com.amazonaws.services.s3.AmazonS3ClientBuilder;
  6. import com.amazonaws.ClientConfiguration;
  7. import com.amazonaws.auth.AWSCredentialsProvider;
  8. import com.amazonaws.event.ProgressEventType;
  9. import com.amazonaws.event.ProgressListener;
  10. import com.amazonaws.services.s3.AmazonS3;
  11. import com.amazonaws.services.s3.model.ObjectMetadata;
  12. import com.amazonaws.services.s3.transfer.TransferManager;
  13. import com.amazonaws.services.s3.transfer.TransferManagerBuilder;
  14. import com.amazonaws.services.s3.transfer.Upload;
  15.  
  16. ClientConfiguration config = new ClientConfiguration();
  17. config.setSocketTimeout(SOCKET_TIMEOUT);
  18. config.setMaxErrorRetry(RETRY_COUNT);
  19. config.setClientExecutionTimeout(CLIENT_EXECUTION_TIMEOUT);
  20. config.setRequestTimeout(REQUEST_TIMEOUT);
  21. config.setConnectionTimeout(CONNECTION_TIMEOUT);
  22.  
  23. AWSCredentialsProvider credProvider = ...;
  24. String region = ...;
  25.  
  26. AmazonS3 s3Client = AmazonS3ClientBuilder.standard().withCredentials(credProvider).withRegion(region).withClientConfiguration(config).build();
  27.  
  28. InputStream stream = ...;
  29. String bucketName = .....;
  30. String keyName = ...;
  31. long contentLength = ...;
  32. String mimeType = ...;
  33.  
  34. //You use metadata to describe the data. You need the content length so the multi part upload knows how big it is
  35. final ObjectMetadata metaData = new ObjectMetadata();
  36. metaData.setContentLength(contentLength);
  37. metaData.setContentType(mimeType);
  38.  
  39. TransferManager tf = TransferManagerBuilder.standard().withS3Client(s3Client).build();
  40. tf.getConfiguration().setMinimumUploadPartSize(UPLOAD_PART_SIZE);
  41. tf.getConfiguration().setMultipartUploadThreshold(UPLOAD_THRESHOLD);
  42. Upload xfer = tf.upload(bucketName, keyName, stream, metaData);
  43.  
  44. ProgressListener progressListener = new ProgressListener() {
  45. public void progressChanged(ProgressEvent progressEvent) {
  46. if (xfer == null)
  47. return;
  48. if (progressEvent.getEventType() == ProgressEventType.TRANSFER_FAILED_EVENT || progressEvent.getEventType() == ProgressEventType.TRANSFER_PART_FAILED_EVENT) {
  49. //Log the message
  50. }
  51. }
  52. };
  53.  
  54. xfer.addProgressListener(progressListener);
  55. xfer.waitForCompletion();

PIG: Testing

Apache PIG analyzes large data sets. There are a variety of ways of processing data using it. As I learn more about it I will put use cases below.

JSON:

  1. REGISTER 'hdfs:///elephant-bird-core-4.15.jar';
  2. REGISTER 'hdfs:///elephant-bird-hadoop-compat-4.15.jar';
  3. REGISTER 'hdfs:///elephant-bird-pig-4.15.jar';
  4. REGISTER 'hdfs:///json-simple-1.1.1.jar';
  5.  
  6. loadedJson = LOAD '/hdfs_dir/MyFile.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (json:map[]);
  7. rec = FOREACH loadedJson GENERATE json#'my_key' as (m:chararray);
  8. DESCRIBE rec;
  9. DUMP rec;
  10.  
  11. --Store the results in a hdfs dir. You can have HIVE query that directory
  12. STORE rec INTO '/hdfs_dir' USING PigStorage();

Python: MRJob

If you use hadoop and you want to run a map reduce type job using Python you can use MRJob.

Installation:

  1. pip install mrjob

Here is an example if you run just the mapper code and you load a json file. yield writes the data out.

  1. from mrjob.job import MRJob, MRStep
  2. import json
  3.  
  4. class MRTest(MRJob):
  5. def steps(self):
  6. return [
  7. MRStep(mapper=self.mapper_test)
  8. ]
  9.  
  10. def mapper_test(self, _, line):
  11. result = {}
  12. doc = json.loads(line)
  13.  
  14. yield key, result
  15.  
  16. if __name__ == '__main__':
  17. MRTest.run()