Module pipelines.utils.backfill_flow.tasks
Tasks for the backfill flow
Functions
def create_timestamp_parameters(start: pendulum.datetime.DateTime, end: pendulum.datetime.DateTime, interval: pendulum.duration.Duration, datetime_format: str = 'YYYY-MM-DD', reverse: bool = False)-
Create a list of parameters for a flow that takes timestamp parameters.
Args
start- The start of the range of timestamps to generate
end- The end of the range of timestamps to generate
interval- The interval between timestamps
datetime_format- The format to use for the timestamps
reverse- Whether to reverse the order of the timestamps
Returns
A list of parameters for a flow that takes timestamp parameters
def launch_flow(flow_name: str, parameter: Dict[str, Any], agent_label: str, flow_project: str = 'main', parameter_defaults: Dict[str, Any] = None, help_name: str = None, datetime_start_param: str = None, datetime_end_param: str = None, fetch_flow_run_info_sleep_time: int = 30, prefect_client: prefect.client.client.Client = None)-
Launch a flow with the given parameters.
Args
flow_name- The name of the flow to launch
parameter- The backfill parameters to use when launching the flow
agent_label- The agent label to use when launching the flow
flow_project- The project to use when launching the flow
parameter_defaults- The default parameters to use when launching the flow
help_name- A help name to use when setting the flow run name
datetime_start_param- The name of the start datetime parameter
datetime_end_param- The name of the end datetime parameter
fetch_flow_run_info_sleep_time- The time to sleep between fetching flow run info
prefect_client- The Prefect client to use
Returns
None
def parse_datetime(datetime_string: str, timezone: str = 'America/Sao_Paulo') ‑> pendulum.datetime.DateTime-
Parse a datetime string.
Args
datetime_string- The datetime string to parse
timezone- The timezone to use
Returns
The parsed datetime
def parse_duration(duration_dict: Dict[str, int]) ‑> pendulum.duration.Duration-
Parse a duration dictionary.
Args
duration_dict- The duration dictionary to parse
Returns
The parsed duration