Code Snippets Scala

Short texts Scala 🧑‍💻 Code snippets Scala ✍️ Reading lists Scala 👀

How to perform Spark Streaming CSV Files from a directory and perform operations using UDF

// Databricks notebook source import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.Column import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType, StringType, StructField, StructType} // Create Spark Session val spark = SparkSession.builder().master(“local”).appName(“Spark Stream UDF”) .getOrCreate() // COMMAND ———- val schema = StructType(List( StructField(“Date”, StringType, true), StructField(“Open”, DoubleType, true), StructField(“High”, DoubleType, true), StructField(“Low”, DoubleType, true), StructField(“Close”, DoubleType, true), StructField(“Adjusted Close”, DoubleType, true), StructField(“Volume”, […]

How to perform Window Operations during Spark Structured Streaming

// Databricks notebook source import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.Column import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType, StringType, StructField, StructType} // Create Spark Session val spark = SparkSession.builder().master(“local”).appName(“File Source”) .getOrCreate() val schema = StructType(List( StructField(“Date”, StringType, true), StructField(“Open”, DoubleType, true), StructField(“High”, DoubleType, true), StructField(“Low”, DoubleType, true), StructField(“Close”, DoubleType, true), StructField(“Adjusted Close”, DoubleType, true), StructField(“Volume”, DoubleType, true) )) def […]

Max difference between increasing elemets

import Math.* def maxDiff(xs: List[Int]): Int = xs.scanLeft(Int.MaxValue)(min) .tail .zip(xs) .map(x => x._2 – x._1) .filter(_ > 0) .maxOption .getOrElse(-1) maxDiff(List(7, 1, 5, 4)) // 4 maxDiff(List(8, 5, 3 ,3)) // -1

Explain about Spark Streaming Checkpoints

// Databricks notebook source import org.apache.spark.sql.SparkSession import org.apache.spark.sql.functions._ import org.apache.spark.sql.Column import org.apache.spark.sql.streaming.Trigger import org.apache.spark.sql.types.{DoubleType, StringType, StructField, StructType} val spark = SparkSession .builder() .master(“local”) .appName(“Check Point”) .getOrCreate() // COMMAND ———- val schema = StructType(List( StructField(“Date”, StringType, true), StructField(“Open”, DoubleType, true), StructField(“High”, DoubleType, true), StructField(“Low”, DoubleType, true), StructField(“Close”, DoubleType, true), StructField(“Adjusted Close”, DoubleType, true), StructField(“Volume”, DoubleType, true) […]

ECIES on Curve25519

// Testing Curve25519 by https://github.com/gherynos/secrete // import coursierapi.MavenRepository interp.repositories.update(interp.repositories() ::: List(MavenRepository.of(s”””file://${System.getenv(“HOME”)}/.m2/repository”””))) import $ivy.`net.nharyes:secrete:1.0.3` import net.nharyes.secrete.ecies._ import java.security.KeyPair import java.util.Random import net.nharyes.secrete.curve.Curve25519KeyPairGenerator val u2 = Curve25519KeyPairGenerator.generateKeyPair() val text = “Text message” val message:ECIESMessage = ECIESHelper.encryptData(u2.getPublic(), text); val data = ECIESHelper.decryptMessage(u2.getPrivate(), message) new String(data, ECIESHelper.ENCODING)