Remove Files Older than x Days Unix remove log files older than x days.. Type : find /path/to/files* -mtime +5 -exec rm {} \; Find Size of Directories To find the size of a set of directories :- Type : du -sh . < current location / directory > reports size in MB/ i.e. cd…
Often Hortonworks delivered Hive enabled clusters, are not setup to allow you to easily connect a SQL tool to HIVE. Follow these steps initially to connect a SQL tool such as SQL Developer / Tableau / DV Desktop etc In order to connect to Hive from DVD, we need to configure BDCS-CE’s Hive Thrift Server…
Below some parameterised user management scripts for creating Unix users, and associated HDFS paths. create_user.sh #!/bin/bash # Call this script with 2 parameters, i.e. create_user.sh username password usage() { echo “Usage pass in username and password i.e. : create_user.sh username password” } if [ $# -ne 2 ] ; then usage else echo “Username is…
A simple script below which uses Spark/Scala to generate a graph output, then write the contents of the connected nodes to a Hive table. Command Line – execute Spark job calling Scala file spark-shell –conf “spark.ui.port=1081” –driver-memory 20G –executor-memory 20G -i graph_test.scala <graph_test.scala> import org.apache.spark._ import org.apache.spark.rdd.RDD import org.apache.spark.graphx.Edge import org.apache.spark.graphx.Graph import org.apache.spark.graphx.GraphLoader import org.apache.spark.SparkContext…
Below, a shell script which takes a file from a Object Store Container Copies the file locally Creates a Hive table from the content #!/bin/bash CONTAINER=sc-oc-vm-dev-ace-01 DIRECTORY= PRICINGREPORT=”PriceReport_$(date +”%Y-%m-%d”).csv” LIVEAPPSDATA=”RMSLiveAppsData_$(date +”%Y-%m-%d”).csv” FILELINECOUNT=0 echo “Object Storage Container Name :” $CONTAINER echo “LiveApps Pricing Data File :” $PRICINGREPORT echo “LiveApps Data File :” $LIVEAPPSDATA echo “—————————————————————–“ echo…
When working on a Hadoop Eco system, where you may have tables stored in HIVE, there maybe times when you need to copy/migrate HIVE data files from one cluster to another. Simply copying the files from the HDFS file system, from source to target clusters will not work. A simple solution here is to use…
Use Winscp Command Line You can use the Winscp command line tool on a Windows platform to copy files to a remote location, much like you would with ftp/sftp/scp. From a windows command line prompt enter the following command, this connects as the user “opc” to a remote IP address using a private key file…
Useful Spark Commands You can run Spark jobs via the command line. Some examples below, including just starting and configuring a session. Also how to query a hive table, and write back a Spark Data Frame to Hive table. Some useful hints below:- Start a Spark Session $>spark-shell Welcome to ____ __ / __/__ ___…
Some useful SQL below, to quickly find status and any warnings/error messages from a running OBIEE Agent select i.job_id, i.instance_id, J.name AS Agent_Name, i.status status_code, to_char(i.begin_ts,’DD/MM/YYYY HH24:MI:SS’) begin_ts, i.status as STATUS, case i.status when 0 then ‘Completed’ when 1 then ‘Running’ when 2 then ‘Failed’ when 3 then ‘Cancelled’ when 4 then ‘Timed-out’ when 5…
There are often times when you want to update values presented to a user on a dashboard page, but “writeback” is not a secure solution and offers no validation A workaround, is to embed an APEX form/application and pass values via a java script called with in an OBIEE analysis. Steps to Achieve this :-…