Wednesday, July 19, 2017

Use of jq in aws cli

I wrote a post with use of query option in aws cli for filtering.
Another powerful approach for filtering is jq.

jq is resourceful and helps in getting all the operations done, which query options fails

for example, I was struggling to pass a shell variable to cli command which I was able to do though jq

below command lets me get the exact name of the emr cluster and passing cluster ID as shell variable.

aws emr list-clusters --region $region |jq -r --arg cid "$cid"  '.Clusters[] | select(.Id == $cid ) | .Name'


Example to query the instaances tag

aws ec2 describe-instances  --region us-east-1 --instance-ids i-02426396b622c2b23   2> /dev/null | jq -r '.Reservations[].Instances[].Tags[] | select(.Key=="owner") | .Value'

Thursday, June 29, 2017

Restart Hadoop Services on EMR

https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/


to check all the services which you can manage

initctl status

 [root@host conf.dist]# initctl list
rc stop/waiting
tty (/dev/tty3) start/running, process 4452
tty (/dev/tty2) start/running, process 4450
tty (/dev/tty1) start/running, process 4448
tty (/dev/tty6) start/running, process 4458
tty (/dev/tty5) start/running, process 4456
tty (/dev/tty4) start/running, process 4454
gmetad start/running, process 10054
update-motd stop/waiting
hadoop-mapreduce-historyserver start/running, process 19861
hadoop-yarn-timelineserver start/running, process 19056
hive-server2 start/running, process 17579
plymouth-shutdown stop/waiting
control-alt-delete stop/waiting
hive-hcatalog-server start/running, process 18745
rcS-emergency stop/waiting
kexec-disable stop/waiting
quit-plymouth stop/waiting
hadoop-yarn-resourcemanager start/running, process 19206
hadoop-kms start/running, process 11467
rcS stop/waiting
hue start/running, process 19671
prefdm stop/waiting
hadoop-httpfs start/running, process 11750
init-system-dbus stop/waiting
print-image-id stop/waiting
oozie start/running, process 15441
ganglia-rrdcached start/running, process 9933
elastic-network-interfaces stop/waiting
splash-manager stop/waiting
hadoop-yarn-proxyserver start/running, process 18922
start-ttys stop/waiting
spark-history-server start/running, process 20044
hadoop-hdfs-namenode start/running, process 12003
rcS-sulogin stop/waiting
serial (ttyS0) start/running, process 4447
gmond start/running, process 10073



to restart a service
[root@server conf.dist]# status hive-hcatalog-server
hive-hcatalog-server start/running, process 18745
[root@server conf.dist]# restart hive-hcatalog-server
hive-hcatalog-server start/running, process 18745
[root@server conf.dist]# restart hive-server2
hive-server2 start/running, process 17579




show databases;

Wednesday, June 28, 2017

Remote JDBC connection to Hive on EMR

1. SSH tunneling :
ssh -o ServerAliveInterval=10 -N -L 10000:localhost:10000 username@masternode


for automating this tunning
we can use autossh

2. use local url which will use above tunnel:
jdbc:hive2://localhost:10000/default

Monday, June 26, 2017

IBM Fluid Query installation

[root@ip-xxxxxxxxx Fluid_1.7]# ./fluidquery_install.sh --datamove_path /mnt/fluid/
-------------------------------------------------------------------------------
IBM Fluid Query Installer
(C) Copyright IBM Corp. 2015, 2016. All rights reserved.
Version 1.7.0.1 [Build 161026-10]
-------------------------------------------------------------------------------
Installing data movement...
Data movement path: /mnt/fluid/
Checking for previous installation of IBM Fluid Query data movement...
Previous installation of data movement does not exist in /mnt/fluid/...
Creating HDFS directory /fluidquery...
HDFS directory /fluidquery created.
Copying files to /mnt/fluid/...
Copying license to /mnt/fluid/...
Copying uninstaller to /mnt/fluid/...
Copying jars to HDFS...
IBM Fluid Query data movement installed succesfully in /mnt/fluid/!
For details see the log file at /mnt/fluid//var/log/fluidquery_install/fluidquery_install.2017-06-20.15:57:59.log


[root@ip-xxxxxxxxxxxxx fluid]# export HADOOP_CLASSPATH=/mnt/fluid/nzjdbc3.jar
[root@ip-xxxxxxxxxxxx fluid]# hadoop jar /mnt/fluid/nzcodec.jar -conf fq-import-remote-updated.xml
FluidQuery version: 1.7.0.1 [Build 161026-10]

 https://www.ibm.com/support/knowledgecenter/en/SSULQD_1.7.0/com.ibm.nz.hdp.doc/tt_dm_lesson_12.html

https://www.ibm.com/support/knowledgecenter/en/SSULQD_1.7.0/com.ibm.nz.hdp.doc/r_hdp_fdm_ts.html

Tuesday, June 20, 2017

AWS cli query examples

using the query option in aws cli makes it very easy to work with cli

below are few examples of using JMESPath query :
  •   use of contains
 aws emr list-clusters --region us-east-1 --query 'Clusters[?Name==`{{ clustername }}`]|[?contains(Status.State, `TERMINATED`) == `false`].[Id]

  • use of length to get the count
aws emr list-clusters --region us-east-1 --query 'Clusters[?Name==`{{ current_run_reference }}`]|[?contains(Status.State, `TERMINATED`) == `false`].[Id]|length(@)
  • list route53 records of a IP
aws route53 list-resource-record-sets --hosted-zone-id ZXXXXXXXXXXXXX --query 'ResourceRecordSets[?ResourceRecords[0].Value==`10.253.xx.xxx`]' 

  • Filter with more than one condition
aws route53 list-resource-record-sets --hosted-zone-id ZXXXXXXXXXXXXX --query 'ResourceRecordSets[?ResourceRecords[0].Value==`10.253.xx.xx`]|[?Type==`CNAME`]'

Tuesday, April 25, 2017

Postgres column encryption using pgcrypto on RDS

This should work on NON-RDS postgres databases also

Add pycrypto extension

CREATE EXTENSION pgcrypto;

To check if pycrypto is added:

SHOW rds.extensions;

pgsql offers two method of column encryption, Below is example where we need encryption and decrytion as well.
perform the steps to create a public and private before.

create a table:
CREATE TABLE secured_table(property SERIAL PRIMARY KEY, property varchar(100), value bytea);


test for rows :
qa=> select * from secured_table;
 id | property | value
----+----------+-------
(0 rows)

insert encrypted data:


INSERT INTO secured_table(property, value)
SELECT robotccs.property, pgp_pub_encrypt(robotccs.value, keys.pubkey) As value
FROM (VALUES ('awskey', 'xfadgghlrgsadff'),
    ('secretkey', 'aCFDGsfsabasfafdh') ) As robotccs(property, value)
     CROSS JOIN (SELECT dearmor('-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.14 (GNU/Linux)
-----END PGP PUBLIC KEY BLOCK-----') As pubkey) As keys;



check rows after insert , it shows only encrypted values:

qa=> select * from secured_table;
 id | property  | value
----+-----------+----------------------------------------------
  1 | awskey    | \xc1c04c038068b7aeb7cbdfd5010800b10b11ef0fa2
  2 | secretkey | \xc1c04c038068b7aeb7cbdfd50107ff5cc130e2e216
(2 rows)

to find decrypted original values:

SELECT property, pgp_pub_decrypt(value, keys.privkey) As ccdecrypt
FROM secured_table
    CROSS JOIN
    (SELECT dearmor('-----BEGIN PGP PRIVATE KEY BLOCK-----
Version: GnuPG v2.0.14 (GNU/Linux)
-----END PGP PRIVATE KEY BLOCK-----') As privkey) As keys;
 property  |     ccdecrypt
-----------+-------------------
 awskey    | xfadgghlrgsadff
 secretkey | aCFDGsfsabasfafdh
(2 rows)

STEPS TO GENERATE THE KEY:

[user@server ~]$ gpg --gen-key
gpg (GnuPG) 2.0.14; Copyright (C) 2009 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection?
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: admin1
Email address: admin1@company.com
Comment:
You selected this USER-ID:
    "admin1 <admin1@company.com>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
You need a Passphrase to protect your secret key.

can't connect to `/home/user/.gnupg/S.gpg-agent': No such file or directory
You don't want a passphrase - this is probably a *bad* idea!
I will do it anyway.  You can change your passphrase at any time,
using this program with the option "--edit-key".

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
asgdhgjasdhad
asghadfhj


gpg: /home/user/.gnupg/trustdb.gpg: trustdb created
gpg: key 966B48BB marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/966B48BB 2016-09-27
      Key fingerprint = 7243 4195 E241 8B7E A994  6817 8F12 326A 966B 48BB
uid                  admin1 <admin1@company.com>
sub   2048R/B7CBDFD5 2016-09-27


[[user@server ~]$ gpg --list-secret-keys
/home/user/.gnupg/secring.gpg
---------------------------------
sec   2048R/966B48BB 2016-09-27
uid                  admin1 <admin1@company.com>
ssb   2048R/B7CBDFD5 2016-09-27

[user@server ~]$ gpg -a --export B7CBDFD5
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v2.0.14 (GNU/Linux)

mQENBFfqtg0BCADixmjg0xac4ylRMBdQhlu+K1iDNeYJWYh1IqpiTwZEfO2mSpm+
uQVk2jHNs9n3oDTng9ggtkq5gQDvDmsIk4X2LqXEgzKaMIQRZAVTL5CbRtUManbn
ANPOrlPtLQJ301T5faZ7EnosSvEtVyNqapnwkkBG7KSMnUfSGLo2RrdlPMPPvWjg
c35wN3Yqr/U02cxkYyDcmPIpzJD3HdwlDCb2d5PAo89gcu9oTT5k54cBBMx17Izc
ge1rdMyvAKdaoJyHrgjItu9SE81i3ydDZJ8FkStt2coxGJXmN9V6rZCM9YxEjnwr
u0yEne7bNep8XnN8OU1QD62IsJfgX0MFykaLC2ndp0kaEbPfztjTp9keGFiTPsLY
MvNf28LwEoXc0uVhVGnoHOkCbVEVHg75jQxKAqnqTMh78TcqxyxrVf2gKPNhN/T4
YhA7t5x5u0YDy0K2FaRdKiZQv1+mcrOcpvY=
=6cNj
-----END PGP PUBLIC KEY BLOCK-----
[user@server ~]$ gpg -a --export-secret-keys 966B48BB
-----BEGIN PGP PRIVATE KEY BLOCK-----
Version: GnuPG v2.0.14 (GNU/Linux)

lQOYBFfqtg0BCADixmjg0xac4ylRMBdQhlu+K1iDNeYJWYh1IqpiTwZEfO2mSpm+
uQVk2jHNs9n3oDTng9ggtkq5gQDvDmsIk4X2LqXEgzKaMIQRZAVTL5CbRtUManbn
ANPOrlPtLQJ301T5faZ7EnosSvEtVyNqapnwkkBG7KSMnUfSGLo2RrdlPMPPvWjg
c35wN3Yqr/U02cxkYyDcmPIpzJD3HdwlDCb2d5PAo89gcu9oTT5k54cBBMx17Izc
ge1rdMyvAKdaoJyHrgjItu9SE81i3ydDZJ8FkStt2coxGJXmN9V6rZCM9YxEjnwr
WUrCdJ6Vx78b9YfoxkTjYK9xMhG+xj7bZanbABEBAAEAB/0cXIdwrsBiPgAZ2POM
LPGRDBIVft/WhYMNG0PHPcOL5zqVlI474x9Ak2VX057MBcD4FMN/+9cb8igFaH6M
Y7Qvq3qB5Df86VprDmD4HA0azmCwx9PiZMvDSobRNSgyMj+rYEKPm9T5xc758QRL
N9UpgtuqbcglPzpklUEcrqVwcfe3D6fqfrpUPdGqzfdH2LEOF5cydFnRzbuswsAu
ySXnWbuaDTgMxO2w5f/7VodPzK5wu2tsPwvVos1GKqiKPHlfLWhOn3qNpk6ODRIM
/ioZ+hXVu0yEne7bNep8XnN8OU1QD62IsJfgX0MFykaLC2ndp0kaEbPfztjTp9ke
GFiTPsLYMvNf28LwEoXc0uVhVGnoHOkCbVEVHg75jQxKAqnqTMh78TcqxyxrVf2g
KPNhN/T4YhA7t5x5u0YDy0K2FaRdKiZQv1+mcrOcpvY=
=MRtJ
-----END PGP PRIVATE KEY BLOCK-----
[user@server ~]$