Consume streaming data with Azure Event Hubs connector on Azure DataBricks · GitHub
Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Consume streaming data with Azure Event Hubs connector on Azure DataBricks
# Need to install com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.16 in your cluster
# Used Databricks Runtime 6.6 (includes Apache Spark 2.4.5, Scala 2.11)
## Setup Azure IoT Hub (Event Hub compatible endpoint) connection information
EVENTHUB_CONNECTION_STRING = "Your connection string" # Endpoint=sb://aaa.windows.net/;SharedAccessKeyName=iothubowner;SharedAccessKey=xxx;EntityPath=bbb
CONSUMER_GROUP = "Your consumer group" # If you don't have consumer group, set $Default here
## Setup Event Hubs settings
event_hub_config = {
"eventhubs.connectionString": spark._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(EVENTHUB_CONNECTION_STRING),
"eventhubs.consumerGroup": CONSUMER_GROUP
}
## Load and visualize streaming data
from pyspark.sql.functions import col
raw_streaming_df = (spark.readStream
.format("eventhubs")
.options(**event_hub_config)
.load()
.withColumn("body", col("body").cast("string"))
)
display(raw_streaming_df)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.