Category: PIG

PIG: Testing

Apache PIG analyzes large data sets. There are a variety of ways of processing data using it. As I learn more about it I will put use cases below.

JSON:

REGISTER 'hdfs:///elephant-bird-core-4.15.jar';
REGISTER 'hdfs:///elephant-bird-hadoop-compat-4.15.jar';
REGISTER 'hdfs:///elephant-bird-pig-4.15.jar';
REGISTER 'hdfs:///json-simple-1.1.1.jar';

loadedJson = LOAD '/hdfs_dir/MyFile.json' USING com.twitter.elephantbird.pig.load.JsonLoader('-nestedLoad') AS (json:map[]);
rec = FOREACH loadedJson GENERATE json#'my_key' as (m:chararray);
DESCRIBE rec;
DUMP rec;

--Store the results in a hdfs dir. You can have HIVE query that directory
STORE rec INTO '/hdfs_dir' USING PigStorage();