全部产品

Redis

本文介绍如何使用Databricks 读写Redis数据

前提条件

打通网络环境

data
  • 登陆Redis控制台添加DDI集群各个机器ip至访问白名单,或者在vpc登陆处设置允许vpc内免密访问。

datadata

使用Databricks 读写Redis数据

  • 链接Redis数据库代码实现

%spark
import com.redislabs.provider.redis._
val redisServerDnsAddress = "your address"
val redisPortNumber = 6379
val redisPassword = "your password"
//获取RedisConfig
val redisConfig = new RedisConfig(new RedisEndpoint(redisServerDnsAddress, redisPortNumber, redisPassword))
  • Redis数据库中相关String,List,Set,Hash的数据类型读写代码如下:

String 读写

%spark
//String 读写
val stringRDD = sc.parallelize(Seq(("000001", "Jack"), ("000002", "Rose")))
sc.toRedisKV(stringRDD)(redisConfig)
val keysRDD = sc.fromRedisKeyPattern("000*", 5)(redisConfig)
val stringRDD2 = keysRDD.getKV
stringRDD2.collect().foreach(println)
data

List读写

%spark
//List 读写
val stringListRDD = sc.parallelize(Seq("dog", "cat", "pig"))
sc.toRedisLIST(stringListRDD, "animal")(redisConfig)
val keysRDD = sc.fromRedisKeyPattern("animal*")(redisConfig)
val listRDD = keysRDD.getList
listRDD.collect().foreach(println)
data

HASH读写

%spark
//Hash读写
val stringHashRDD = sc.parallelize(Seq(("jack","22"), ("rose","23"),("sir","24"),("jack","24")))
sc.toRedisHASH(stringHashRDD, "message")(redisConfig)
val keysRDD = sc.fromRedisKeyPattern("message*")(redisConfig)
val hashRDD = keysRDD.getHash
hashRDD.collect().foreach(println)

dataSET读写

%spark
//Set 读写
val stringHashRDD = sc.parallelize(Seq("hello", "hello", "word"))
sc.toRedisSET(stringHashRDD, "setKey")(redisConfig)
val keysRDD = sc.fromRedisKeyPattern("setKey*")(redisConfig)
val setRDD = keysRDD.getSet
setRDD.collect().foreach(println)
data