提问者:小点点

scala火花盒类通用编码器


怎么才能拿到这个方法编译,奇怪的是,火花隐式已经被导入了。

def loadDsFromHive[T <: Product](tableName: String, spark: SparkSession): Dataset[T] = {
    import spark.implicits._
    spark.sql(s"SELECT * FROM $tableName").as[T]
  }

这是错误:

Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._  Support for serializing other types will be added in future releases.
[error]     spark.sql(s"SELECT * FROM $tableName").as[T]

共1个答案

匿名用户

根据org. apache.lark.sql.SQLImplicits的源代码,您需要类型类TypeTag作为您的类型,以便隐式Encoder存在:

import scala.reflect.runtime.universe.TypeTag
def loadDsFromHive[T <: Product: TypeTag](tableName: String, spark: SparkSession): Dataset[T] = ...