@'hj"dZddlmZddlmZmZmZmZddlm Z ddl Z ddl Z ddl mZddlmZGdd eZGd d eZGd d eZy)z;Tabular parser. Contains parsers for tabular data files. )Path)AnyDictListOptional)AbstractFileSystemN) BaseReader)Documentc ^eZdZdZdddedededdffd Z d d ed ee de e fd Z xZ S) CSVReaderzCSV parser. Args: concat_rows (bool): whether to concatenate all rows into one document. If set to False, a Document will be created for each row. True by default. T) concat_rowsargsr kwargsreturnNc2t||i|||_yz Init params.N)super__init__ _concat_rows)selfr rr __class__s l/home/kushmeetdev/Regenta/Chatbot/venv/lib/python3.12/site-packages/llama_index/readers/file/tabular/base.pyrzCSVReader.__init__s $)&)'file extra_infoc ddl}g}t|5}|j|}|D]"}|j dj |$ ddd|j |jd}|ri||}|jrtdj ||gS|D cgc]} t| |c} S#t$r tdwxYw#1swYxYwcc} w)ziParse file. Returns: Union[str, List[str]]: a string or a List of strings. rNz)csv module is required to read CSV files., )filename extension textmetadata) csv ImportErroropenreaderappendjoinnamesuffixrr ) rrrr$ text_listfp csv_readerrowr#r"s r load_datazCSVReader.load_data s K  $Z 12BJ! 1  30 1 1 !%  D 1(1j1H   $))I"6JK KGPQtH$:Q Q KIJ J K 1 1RsB:9C#C:CCN)__name__ __module__ __qualname____doc__rboolrrrrrr r0 __classcell__rs@rr r s\8<(c((s(t( 8<RR&.tnR hRrr c|eZdZdZdddiddededed ed ed ed d ffdZ dde de e de e d e efdZxZS)PandasCSVReaderaPandas-based CSV parser. Parses CSVs using the separator detection from Pandas `read_csv`function. If special parameters are required, use the `pandas_config` dict. Args: concat_rows (bool): whether to concatenate all rows into one document. If set to False, a Document will be created for each row. True by default. col_joiner (str): Separator to use for joining cols per row. Set to ", " by default. row_joiner (str): Separator to use for joining each row. Only used when `concat_rows=True`. Set to "\n" by default. pandas_config (dict): Options for the `pandas.read_csv` function call. Refer to https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html for more information. Set to empty dict by default, this means pandas will try to figure out the separators, table head, etc. on its own. Trr )r col_joiner row_joiner pandas_configrr r;r<r=rrNc\t||i|||_||_||_||_yr)rrr _col_joiner _row_joiner_pandas_config)rr r;r<r=rrrs rrzPandasCSVReader.__init__Ws7 $)&)'%%+rrrfsc|r;|j|5}tj|fij}dddn tj|fij}j fddj }j r+tjj||xsigS|Dcgc]}t||xsic}S#1swYxYwcc}w)z Parse file.Nc|jj|jtj Sr1)r?r)astypestrtolist)r/rs rz+PandasCSVReader.load_data..us)))// 30F0F0HIr)axisr!) r&pdread_csvrAapplyrGrr r@r))rrrrBfdfr,r"s` rr0zPandasCSVReader.load_datags  ;![[:d&9&9: ; ;T9T%8%89BHH IPQ &(    **00;jFVTV LUCGdZ-=2> ! ; ; s!C>C%C"NNr2r3r4r5rr6rFdictrrrrrrr r0r7r8s@rr:r:=s8! ,,, ,  ,  ,, ,&&*+/ TN ' (  h rr:c~eZdZdZddddiddeded ed ed ed ed dffdZ dde de e de e d e efdZxZS)PandasExcelReaderaKCustom Excel parser that includes header names in each row. Parses Excel files using Pandas' `read_excel` function, but formats each row to include the header name, for example: "name: joao, position: analyst". The first row (header) is not included in the generated documents. Args: concat_rows (bool): Determines whether to concatenate all rows into one document. If set to False, one Document is created for each row. Defaults to True. sheet_name (str | int | None): Defaults to None, meaning all sheets. Alternatively, pass a string or an integer to specify the sheet to be read. field_separator (str): Character or string to separate each field. Default: ", ". key_value_separator (str): Character or string to separate the key from the value. Default: ": ". pandas_config (dict): Options for the `pandas.read_excel` function call. Refer to https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html for more details. Defaults to an empty dictionary. TNrz: )r sheet_namefield_separatorkey_value_separatorr=rr rVrWr=rrcjt||i|||_||_||_||_||_y)zInitializes the parameters.N)rrr _sheet_name_field_separator_key_value_separatorrA) rr rUrVrWr=rrrs rrzPandasExcelReader.__init__s? $)&)'% /$7!+rrrrBc tjjd}|n td|rF|j |5}t j ||jfi|j}dddn+t j ||jfi|j}g}tt jr|jd}|jj} g} |jD]R\} } |jj!| D cgc]} | |j"| | c} }| j%|T|j&r1|j%t)dj!| |xsi|S|j+| Dcgc]}t)||xsic}|S|j-D]}|jd}|jj} g} |jD]R\} } |jj!| D cgc]} | |j"| | c} }| j%|T|j&r0|j%t)dj!| |xsi|j+| Dcgc]}t)||xsic}|S#1swY9xYwcc} wcc}wcc} wcc}w)zParses the file.openpyxlNz[Please install openpyxl to read Excel files. You can install it with 'pip install openpyxl'r r!) importlibutil find_specr%r&rK read_excelrYrA isinstance DataFramefillnacolumnsrGiterrowsrZr)r[r(rr extendvalues)rrrrB openpyxl_specrNdfs documentsrOheadersr,_r/header formatted_rowr"s rr0zPandasExcelReader.load_datas"00<  $ m   P!mmAt'7'7O4;N;NO P P--d&6&6N$:M:MNC  c2<< (BBjj'')GI++- 03 $ 5 5 : :'.""(4#<#<"=c&k_M!   / 0    $))I"6AQrRLE  %. !dZ5E2FD5jjl YYr]**++-  kkm4FAs$($9$9$>$>+2 & &ht'@'@&A#f+Q%M $$]34$$$$ dii &:ZEUSUV$$)2 $%$9IrJ' 4G P P*s$,KK K"+K' ,K, KrPrQr8s@rrTrTs.!##' ,,,  , ! ,,, ,*&*+/ TTTNT ' ( T h TrrT)r5pathlibrtypingrrrrfsspecrr_pandasrKllama_index.core.readers.baser llama_index.core.schemar r r:rTrrrxsP ,,%4,)R )RXDjDN{ {r