CCA175 Practice Exam Questions

Solution :

Step 1:

hdfs dfs -mkdir sparksql4

hdfs dfs -put course.txt sparksql4/

hdfs dfs -put fee.txt sparksql4/

Step 2 : Now in spark shell

// load the data into a new RDD

val course = sc.textFile("sparksql4/course.txt")

val fee = sc.textFile("sparksql4/fee.txt")

// Return the first element in this RDD

course.fi rst()

fee.fi rst()

//define the schema using a case class case class Course(id: Integer, name: String) case class Fee(id: Integer, fee: Integer)

// create an RDD of Product objects

val courseRDD = course.map(_.split(",")).map(c => Course(c(0).tolnt,c(1)))

val feeRDD =fee.map(_.split(",")).map(c => Fee(c(0}.tolnt,c(1}.tolnt))

courseRD

D.  

courseRD

D.  

feeRD

D.  

feeRD

D.  

// change RDD of Product objects to a DataFrame val courseDF = courseRD

D.  

D.  

// register the DataFrame as a temp table courseD

F.  

F.  

// Select data from table

val results = sqlContext.sql(......SELECT' FROM course """ )

results. showQ

val results = sqlContext.sql(......SELECT' FROM fee......)

results. showQ

val results = sqlContext.sql(......SELECT * FROM course LEFT JOIN fee ON course.id = fee.id......)

results-showQ

val results ="sqlContext.sql(......SELECT * FROM course RIGHT JOIN fee ON course.id = fee.id "MM )

results. showQ

val results = sqlContext.sql(......SELECT' FROM course LEFT JOIN fee ON course.id = fee.id where fee.id IS NULL"

results. show()

Solution :

A.  

B.  

Solution :

Step 1 : Create all three files in hdfs in directory called spark2 (We will do using Hue). However, you can first create in local filesystem and then upload it to hdfs

Step 2 : Load the Content.txt file

val content = sc.textFile("spark2/Content.txt") //Load the text file

Step 3 : Load the Remove.txt file

val remove = sc.textFile("spark2/Remove.txt") //Load the text file

Step 4 : Create an RDD from remove, However, there is a possibility each word could have trailing spaces, remove those whitespaces as well. We have used two functions here flatMap, map and trim.

val removeRDD= remove.flatMap(x=> x.splitf',") ).map(word=>word.trim)//Create an array of words

Step 5 : Broadcast the variable, which you want to ignore

val bRemove = sc.broadcast(removeRD

D.  

Step 6 : Split the content RDD, so we can have Array of String. val words = content.flatMap(line => line.split(" "))

Step 7 : Filter the RDD, so it can have only content which are not present in "Broadcast Variable". val filtered = words.filter{case (word) => !bRemove.value.contains(word)}

Step 8 : Create a PairRDD, so we can have (word,1) tuple or PairRD

D.  

Step 9 : Nowdo the word count on PairRD

D.  

D.  

Step 10 : Save the output as a Text file.

wordCount.saveAsTextFile("spark2/result.txt")