Module `pipelines.utils.tasks`

Helper tasks that could fit any pipeline.

Functions

def check_table_exists(dataset_id: str, table_id: str, wait=None) ‑> bool: Check if table exists in staging on GCP
def create_table_and_upload_to_gcs(data_path: Union[str, pathlib.Path], dataset_id: str, table_id: str, dump_mode: str, biglake_table: bool = True, wait=None) ‑> None: Create table using BD+ and upload to GCS.
def get_current_flow_labels() ‑> List[str]: Get the labels of the current flow.
def get_current_flow_mode(labels: List[str]) ‑> str: Get the mode (prod/dev/staging) of the current flow.
def get_now_date(): Returns the current date in YYYY-MM-DD.
def get_now_time(): Returns the HH:MM.
def get_user_and_password(secret_path: str, wait=None): Returns the user and password for the given secret path.
def greater_than(value, compare_to) ‑> bool: Returns True if value is greater than compare_to.
def log_task(msg: Any, level: str = 'info', wait=None): Logs a message to prefect's logger.
def rename_current_flow_run_dataset_table(prefix: str, dataset_id, table_id, wait=None) ‑> None: Rename the current flow run.
def rename_current_flow_run_msg(msg: str, wait=None) ‑> None: Rename the current flow run.
def rename_current_flow_run_now_time(prefix: str, now_time=None, wait=None) ‑> None: Rename the current flow run.