Vipul Pathak

.

EMP and DEPT tables are pretty popular between Oracle users. These tables were very handy in quickly trying new queries. Also, there exists a DUAL table in Oracle that was pretty useful in evaluate expressions, like- “Select (SYSDATE + 1/24) as OneHourFromNow  FROM DUAL“. These tables doesn’t exists in Hive, but we can create them on our own.

I have created these  tables for trying the Hive queries and even for Pig scripts. I created the following two data files –

EMP

7369,SMITH,CLERK,7902,17-DEC-1980,800,\000,20
7499,ALLEN,SALESMAN,7698,20-FEB-1981,1600,300,30
7521,WARD,SALESMAN,7698,22-FEB-1981,1250,500,30
7566,JONES,MANAGER,7839,2-APR-1981,2975,\000,20
7654,MARTIN,SALESMAN,7698,28-SEP-1981,1250,1400,30
7698,BLAKE,MANAGER,7839,1-MAY-1981,2850,\000,30
7782,CLARK,MANAGER,7839,9-JUN-1981,2450,\000,10
7788,SCOTT,ANALYST,7566,09-DEC-1982,3000,\000,20
7839,KING,PRESIDENT,\000,17-NOV-1981,5000,\000,10
7844,TURNER,SALESMAN,7698,8-SEP-1981,1500,0,30
7876,ADAMS,CLERK,7788,12-JAN-1983,1100,\000,20
7900,JAMES,CLERK,7698,3-DEC-1981,950,\000,30
7902,FORD,ANALYST,7566,3-DEC-1981,3000,\000,20
7934,MILLER,CLERK,7782,23-JAN-1982,1300,\000,10

DEPT

10,ACCOUNTING,NEW YORK
20,RESEARCH,DALLAS
30,SALES,CHICAGO
40,OPERATIONS,BOSTON

DUAL

I created a folder in HDFS and then I used HDFS’ copyFromLocal command to upload these text files to HDFS. Next, I created the following Hive table definitions –

EMP Table Creation

CREATE EXTERNAL TABLE IF NOT EXISTS emp (
    emp_no int,
    ename string,
    job string,
    mgr_id int,
    date_of_joining string,
    salary int,
    bonus int,
    dept_no int)
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY ','
    LINES TERMINATED BY '\n'
    NULL DEFINED AS '00'
STORED AS TEXTFILE
LOCATION 'hdfs://localhost:9000/user/vpathak/Data/emp'
;

DEPT Table Creation

CREATE EXTERNAL TABLE IF NOT EXISTS dept (
    dept_no  int,
    d_name   string,
    l_loc    string)
ROW FORMAT DELIMITED
    FIELDS TERMINATED BY ','
    LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 'hdfs://localhost:9000/user/vpathak/Data/dept'
;

DUAL Table Creation

CREATE TABLE IF NOT EXISTS dual ( name  string )
ROW FORMAT DELIMITED LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 'hdfs://localhost:9000/user/vpathak/Data/dual' ;

Usage

Now since the tables in Hive are created, I can use them in a pretty handy way for quick tryouts –

hive> SELECT format_number(100000/6, 2) AS SixParts FROM dual          -- Returns 16,666.67  ;
hive> SELECT pow(2, 8) AS ByteCapacity FROM dual                       -- Returns 256  ;
hive> SELECT from_unixtime(unix_timestamp()) AS CurrentTime FROM dual  -- Returns the current time ;
hive> SELECT 'Vipul Pathak' AS HiveUser FROM dual                      -- Returns a static string ;
hive>
hive> SELECT e.emp_no, e.ename, e.bonus, d.d_name
        FROM emp AS e JOIN dept AS d
          ON (e.dept_no = d.dept_no) ;

Finally for pig scripts, I created the following pig relations and placed them in the ~/.pigbootup file :

emp = LOAD 'hdfs://localhost:9000/user/vpathak/Data/emp'
      USING PigStorage(',')
      AS (emp_no: INT, ename: CHARARRAY, job: CHARARRAY,
          mgr_id: INT, doj: CHARARRAY, salary: FLOAT,
          bonus: FLOAT, dept_no: INT);

dept = LOAD 'hdfs://localhost:9000/user/vpathak/Data/dept'
       USING PigStorage(',')
       AS (dept_no: INT, d_name: CHARARRAY, d_loc: CHARARRAY) ;

These tables are pretty useful for trying out different kind of queries (like back in Oracle days).   🙂

..