Module pipelines.utils.dump_to_gcs.tasks

Tasks for dumping data directly from BigQuery to GCS.

Functions

def download_data_to_gcs(dataset_id: str, table_id: str, project_id: str = None, query: Union[str, jinja2.environment.Template] = None, jinja_query_params: dict = None, bd_project_mode: str = 'prod', billing_project_id: str = None, location: str = 'US', maximum_bytes_processed: float = 5368709120)

Get data from BigQuery.

def get_project_id(project_id: str = None, bd_project_mode: str = 'prod')

Get the project ID.

def trigger_cron_job(project_id: str, dataset_id: str, table_id: str, cron_expression: str)

Tells whether to trigger a cron job.

def update_last_trigger(project_id: str, dataset_id: str, table_id: str, execution_time: datetime.datetime)

Update the last trigger.