从python上传到Bigquery
我有一个
Python脚本,它从firebase下载数据,操作它然后将其转储到JSON文件中.我可以通过命令行将它上传到BigQuery,但现在我想将一些代码放入Python脚本中以便将其全部完成.
这是我到目前为止的代码. import json from firebase import firebase firebase = firebase.FirebaseApplication('<redacted>') result = firebase.get('/connection_info',None) id_keys = map(str,result.keys()) #with open('result.json','r') as w: # connection = json.load(w) with open("w.json","w") as outfile: for id in id_keys: json.dump(result[id],outfile,indent=None) outfile.write("n") 解决方法
要使用google-cloud-bigquery Python库加载JSON文件,请使用
Client.load_table_from_file()方法.
bigquery_client = bigquery.Client() dataset = bigquery_client.dataset('mydataset') table = dataset.table('mytable') with open(source_file_name,'rb') as source_file: # This example uses JSON,but you can use other formats. # See https://cloud.google.com/bigquery/loading-data job_config = bigquery.LoadJobConfig() job_config.source_format = 'NEWLINE_DELIMITED_JSON' job = client.load_table_from_file( source_file,table,job_config=job_config) 来自代码示例:https://github.com/GoogleCloudPlatform/google-cloud-python/blob/5f059f006b655970b1ef12977146c64bc9b60894/docs/bigquery/snippets.py#L379-L392 编辑:从Python库的0.28.0版开始,上传到表的方式发生了变化.以下是0.27及更早版本的方法. 要使用google-cloud-bigquery Python库加载JSON文件,请使用Table.upload_from_file()方法. bigquery_client = bigquery.Client() dataset = bigquery_client.dataset('mydataset') table = dataset.table('mytable') # Reload the table to get the schema. table.reload() with open(source_file_name,but you can use other formats. # See https://cloud.google.com/bigquery/loading-data job = table.upload_from_file( source_file,source_format='NEWLINE_DELIMITED_JSON') 来自代码示例:https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigquery/cloud-client/load_data_from_file.py 注意:您必须首先创建表并指定模式(也可以使用Python库完成).遗憾的是,客户端库尚不支持架构自动检测功能:https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2926 (编辑:李大同) 【声明】本站内容均来自网络,其相关言论仅代表作者个人观点,不代表本站立场。若无意侵犯到您的权利,请及时与联系站长删除相关内容! |