@'h "dZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z ddl mZmZmZ ddlmZGdd e Zy#e$rdZYwxYw) a Unstructured file reader. A parser for unstructured text files using Unstructured.io. Supports .csv, .tsv, .doc, .docx, .odt, .epub, .org, .rst, .rtf, .md, .msg, .pdf, .heic, .png, .jpg, .jpeg, .tiff, .bmp, .ppt, .pptx, .xlsx, .eml, .html, .xml, .txt, .json documents. N)Path)AnyDictListOptionalSetTuple) BaseReader)DocumentNodeRelationshipTextNode)Elementc@eZdZdZddddddedededeedeed df fd Z e ddedefd Z dd ee d ee dee dee deedeeed eefdZ dd e d ee d eefdZdeedee dee deedeeed eef dZxZS)UnstructuredReaderz8General unstructured text reader for a variety of files.N)api_keyurlallowed_metadata_typesexcluded_metadata_keysargsrrrrreturnct||t td||_t ||_|j r|xsdnd|_|xsttttdf|_ |xsdh|_ y)a Initialize UnstructuredReader. Args: *args (Any): Additional arguments passed to the BaseReader. api_key (str, optional): API key for accessing the Unstructured.io API. If provided, the reader will use the API for parsing files. Defaults to None. url (str, optional): URL for the Unstructured.io API. If not provided and an api_key is given, defaults to "http://localhost:8000". Ignored if api_key is not provided. Defaults to None. allowed_metadata_types (Optional[Tuple], optional): Tuple of types that are allowed in the metadata. Defaults to (str, int, float, type(None)). excluded_metadata_keys (Optional[Set], optional): Set of metadata keys to exclude from the final document. Defaults to {"orig_elements"}. Attributes: api_key (str or None): Stores the API key. use_api (bool): Indicates whether to use the API for parsing files, based on the presence of the api_key. url (str or None): URL for the Unstructured.io API if using the API. allowed_metadata_types (Tuple): Tuple of types that are allowed in the metadata. excluded_metadata_keys (Set): Set of metadata keys to exclude from the final document. NzUUnstructured is not installed. Please install it using 'pip install -U unstructured'.zhttp://localhost:8000 orig_elements)super__init__r ImportErrorrbooluse_apirstrintfloattyperr)selfrrrrr __class__s q/home/kushmeetdev/Regenta/Chatbot/venv/lib/python3.12/site-packages/llama_index/readers/file/unstructured/base.pyrzUnstructuredReader.__init__s2 $ ?g  G} 59\\311t&<'    J A # '=&Q@Q#c|||S)zSet the server url and api key.)clsrrs r$from_apizUnstructuredReader.from_apiFs7C  r%fileunstructured_kwargsdocument_kwargs extra_infosplit_documentsc|r|jni}|jd|jd td|j||}|j |||||S)a Load data using Unstructured.io. Depending on the configuration, if url is set or use_api is True, it'll parse the file using an API call, otherwise it parses it locally. extra_info is extended by the returned metadata if split_documents is True. Args: file (Optional[Path]): Path to the file to be loaded. unstructured_kwargs (Optional[Dict]): Additional arguments for unstructured partitioning. document_kwargs (Optional[Dict]): Additional arguments for document creation. extra_info (Optional[Dict]): Extra information to add to the document metadata. split_documents (Optional[bool]): Whether to split the documents. excluded_metadata_keys (Optional[List[str]]): Keys to exclude from the metadata. Returns: List[Document]: List of parsed documents. r*metadata_filenamezePlease provide a 'metadata_filename' as part of the 'unstructured_kwargs' when loading a file stream.)copyget ValueError_partition_elements_create_documents)r"r*r+r,r-r.relementss r$ load_datazUnstructuredReader.load_dataKs6=P1668UW # #F + 7#''(;<Dw #'":":;NPT"U%%     "   r%c|rt||d<|jr(ddlm}|d|j|j dzd|Sddlm}|di|S)a: Partition the elements from the file or via API. Args: file (Optional[Path]): Path to the file to be loaded. unstructured_kwargs (Dict): Additional arguments for unstructured partitioning. Returns: List[Element]: List of partitioned elements. filenamer)partition_via_apiz/general/v0/general)rapi_url) partitionr')rrunstructured.partition.apir:rrunstructured.partition.autor<)r"r+r*r:r<s r$r4z&UnstructuredReader._partition_elementszsd .1$i  + << D$ #88&  >323 3r%r6c |xsi}|xsit|xs jg} d dtdttdt t tfffd }t|dk(rgS|D cgc]*} djt | j,} } ||d} | jddxs| d } tdd j| | | | d |} |rg}t|D]s\}}|j|}td|j |||||d |}| j#|j$t&j(<|j+|u|S| g}|Scc} w)a0 Create documents from partitioned elements. Args: elements (List): List of partitioned elements. document_kwargs (Optional[Dict]): Additional arguments for document creation. extra_info (Optional[Dict]): Extra information to add to the document metadata. split_documents (Optional[bool]): Whether to split the documents. excluded_metadata_keys (Optional[List[str]]): Keys to exclude from the metadata. Returns: List[Document]: List of parsed documents. Nelementsequence_numberrc i|jj}|jDcic]7\}}|vr.|t|jr|nt j |9}}}|||d<|Scc}}w)NrA)metadatato_dictitems isinstancerjsondumps) r@rAcandidate_metadatakeyvaluerC doc_extras excluded_keysr"s r$_merge_metadataz=UnstructuredReader._create_documents.._merge_metadatas"NG$4$4$<$<$>!M*!M #5":":"<  Cm+ !%)D)DEE*+H*.=*+OsrqsM 884HH7 JJ GsAA  A