我正在尝试使用本文末尾详述的apache数据流管道:https://cloud.google.com/blog/products/gcp/how-to-process-weather-satellite-data-in-real-time-in-bigquery.我试图下载的goes-16数据集是:https://console.cloud.google.com/storage/browser/gcp-public-data-goes-16?authuser=3
p = beam.Pipeline(runner, options=opts)
(p
| 'events' >> beam.io.ReadStringsFromGoogleCloud(bucketname) <---- looking for this
| 'filter' >> beam.FlatMap(lambda message: g2j.only_infrared(message))
| 'to_jpg' >> beam.Map(lambda objectid:
g2j.goes_to_jpeg(
objectid, lat, lon, bucket,
'goes/{}_{}/{}'.format( lat, lon, os.path.basename(objectid).replace('.nc','.jpg') )
))
)
任何帮助赞赏,如果我去关于这个完全错误的让我知道!
对于流数据,云存储——
如果是批量的,那么您可以从Cloud Storage触发Cloud Function,然后将该消息推送到Pub/Sub。