Module pipelines.rj_cor.comando.eventos.utils

General purpose functions for the comando project

Functions

def build_redis_key(dataset_id: str, table_id: str, name: str, mode: str = 'prod')

Helper function for building a key where to store the last updated time

def compare_actual_df_with_redis_df(dfr: pandas.core.frame.DataFrame, dfr_redis: pandas.core.frame.DataFrame, columns: list) ‑> pandas.core.frame.DataFrame

Compare df from redis to actual df and return only the rows from actual df that are not already saved on redis.

def format_date(first_date, last_date)

Format date to "dd/mm/yyyy" and add one day to last date because the API has open interval at the end: [first_date, last_date).

def get_redis_output(redis_key, is_df: bool = False)

Get Redis output. Use get to obtain a df from redis or hgetall if is a key value pair.

def get_token()

Get token to access comando's API

def get_url(url, parameters: dict = None, token: str = None)

Make request to comando's API

def treat_wrong_id_pop(dfr)

Create id_pop based on pop_titulo column