Module pipelines.rj_cor.meteorologia.meteorologia_redemet.tasks

Tasks for meteorologia_redemet

Functions

def check_for_new_stations(dataframe: pandas.core.frame.DataFrame, wait=None)

Check if the updated stations are the same as before. If not, consider flow as failed and call attention to change treat_data task.

def download_data(first_date: str, last_date: str) ‑> pandas.core.frame.DataFrame

Request data from especified date range

def download_stations_data() ‑> pandas.core.frame.DataFrame

Download station information

def get_dates(first_date: str, last_date: str) ‑> Tuple[str, str]

Task to get first and last date. If none date is passed on parameters or we are not doing a backfill the first_date will be yesterday and last_date will be today. Otherwise, this function will return date inputed on flow's parameters.

def save_data(dataframe: pandas.core.frame.DataFrame, partition_column: str = 'data_medicao') ‑> Union[str, pathlib.Path]

Salve dataframe as a csv file

def treat_data(dataframe: pandas.core.frame.DataFrame, backfill: bool = 0) ‑> pandas.core.frame.DataFrame

Rename cols, convert timestamp, filter data for the actual date

def treat_stations_data(dataframe: pandas.core.frame.DataFrame) ‑> pandas.core.frame.DataFrame

Treat station data