A'hGBdZddlmZddlZddlmZmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZdd lmZdd lmZej2eZGd dee ZGddeZGddZGddeeZ GddeeZ!GddeeZ"y)z3Interfaces to be implemented by general evaluators.) annotationsN)ABCabstractmethod)Sequence)Enum)AnyOptionalUnion)warn) AgentAction)BaseLanguageModel)run_in_executor)ChainceZdZdZdZ dZ dZ dZ dZ dZ dZ d Z d Z d Z d Z d Z dZ dZ dZ dZ dZ dZ dZ dZy) EvaluatorTypezThe types of the evaluators.qacot_qa context_qapairwise_string score_stringlabeled_pairwise_stringlabeled_score_string trajectorycriterialabeled_criteriastring_distance exact_match regex_matchpairwise_string_distanceembedding_distancepairwise_embedding_distance json_validity json_equalityjson_edit_distancejson_schema_validationN)__name__ __module__ __qualname____doc__QACOT_QA CONTEXT_QAPAIRWISE_STRING SCORE_STRINGLABELED_PAIRWISE_STRINGLABELED_SCORE_STRINGAGENT_TRAJECTORYCRITERIALABELED_CRITERIASTRING_DISTANCE EXACT_MATCH REGEX_MATCHPAIRWISE_STRING_DISTANCEEMBEDDING_DISTANCEPAIRWISE_EMBEDDING_DISTANCE JSON_VALIDITY JSON_EQUALITYJSON_EDIT_DISTANCEJSON_SCHEMA_VALIDATIONb/home/kushmeetdev/Regenta/Chatbot/venv/lib/python3.12/site-packages/langchain/evaluation/schema.pyrrs& B F%JS'O!L7H1@#VH<)7'OPKIKN9=-M"?;#M.#M=-T5Ir?rc,eZdZdZeeddZy) LLMEvalChainz,A base class for evaluators that use an LLM.c y)z#Create a new evaluator from an LLM.Nr>)clsllmkwargss r@from_llmzLLMEvalChain.from_llmNr?N)rEr rFrreturnrB)r&r'r(r) classmethodrrGr>r?r@rBrBKs622r?rBcpeZdZdZeddZeddZed dZed dZ d d dZ y) _EvalArgsMixinz(Mixin for checking evaluation arguments.cyz2Whether this evaluator requires a reference label.Fr>selfs r@requires_referencez!_EvalArgsMixin.requires_referenceWr?cy)0Whether this evaluator requires an input string.Fr>rOs r@requires_inputz_EvalArgsMixin.requires_input\rRr?c6d|jjdS)z&Warning to show when input is ignored.zIgnoring input in , as it is not expected. __class__r&rOs r@_skip_input_warningz"_EvalArgsMixin._skip_input_warningas$DNN$;$;#<r?r@rLrLTs~2 VV  $(#/ //  /r?rLceZdZdZed dZed dZeddd d dZddd d dZ ddd d dZ ddd d d Z y)StringEvaluatorzcGrade, tag, or otherwise evaluate predictions relative to their inputs and/or reference labels.c.|jjS)zThe name of the evaluation.rXrOs r@evaluation_namezStringEvaluator.evaluation_names~~&&&r?cyrNr>rOs r@rQz"StringEvaluator.requires_referencerRr?Nr_r`c y)a:Evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nr>rP predictionr_r`rFs r@_evaluate_stringsz!StringEvaluator._evaluate_stringsrHr?cTKtd|jf|||d|d{S7w)aIAsynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. It is recommended that the dictionary contain the following keys: - score: the score of the evaluation, if applicable. - value: the string value of the evaluation, if applicable. - reasoning: the reasoning for the evaluation, if applicable. Nrqr_r`)rrrrps r@_aevaluate_stringsz"StringEvaluator._aevaluate_stringssE,%   " " "       s (&(c T|j|||jd|||d|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rnrtr>)rarrrps r@evaluate_stringsz StringEvaluator.evaluate_stringss@$ ##iu#E%t%% !Ye GM  r?cpK|j|||jd|||d|d{S7w)a Asynchronously evaluate Chain or LLM output, based on optional input and label. Args: prediction (str): The LLM or chain prediction to evaluate. reference (Optional[str], optional): The reference label to evaluate against. input (Optional[str], optional): The input to consider during evaluation. kwargs: Additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. rnrtNr>)rarurps r@aevaluate_stringsz!StringEvaluator.aevaluate_stringssN$ ##iu#E,T,, !Ye GM    s -646rdrb) rqzUnion[str, Any]r_Optional[Union[str, Any]]r`rzrFrrIdict) rqrer_rfr`rfrFrrIr{) r&r'r(r)rhrlrQrrrrurwryr>r?r@rjrjs; '' 04+/  $ -  )       604+/  $ -  )      F$(#   !        6$(#   !        r?rjceZdZdZeddd ddZddd ddZddd ddZddd ddZy) PairwiseStringEvaluatorzDCompare the output of two models (or two outputs of the same model).Nrnc y)1Evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nr>rPrq prediction_br_r`rFs r@_evaluate_string_pairsz.PairwiseStringEvaluator._evaluate_string_pairsrHr?cVKtd|jf||||d|d{S7w)@Asynchronously evaluate the output string pairs. Args: prediction (str): The output string from the first model. prediction_b (str): The output string from the second model. reference (Optional[str], optional): The expected output / reference string. input (Optional[str], optional): The input string. kwargs: Additional keyword arguments, such as callbacks and optional reference strings. Returns: dict: A dictionary containing the preference, scores, and/or other information. Nrqrr_r`)rrrs r@_aevaluate_string_pairsz/PairwiseStringEvaluator._aevaluate_string_pairssH(%   ' ' "%      )')c V|j|||jd||||d|S)rrnrr>)rarrs r@evaluate_string_pairsz-PairwiseStringEvaluator.evaluate_string_pairs0sG( ##iu#E*t** !%     r?crK|j|||jd||||d|d{S7w)rrnrNr>)rarrs r@aevaluate_string_pairsz.PairwiseStringEvaluator.aevaluate_string_pairsMsU( ##iu#E1T11 !%       .757) rqrerrer_rfr`rfrFrrIr{) r&r'r(r)rrrrrr>r?r@r}r}s8N $(#     !        4$(#     !       F$(#     !       D$(#     !       r?r}ceZdZdZed dZedd d dZdd d dZdd d dZ dd d dZ y) AgentTrajectoryEvaluatorz,Interface for evaluating agent trajectories.cy)rTTr>rOs r@rUz'AgentTrajectoryEvaluator.requires_inputnsr?N)r_c y)Evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. Nr>rPrqagent_trajectoryr`r_rFs r@_evaluate_agent_trajectoryz3AgentTrajectoryEvaluator._evaluate_agent_trajectorysrHr?cVKtd|jf||||d|d{S7w)Asynchronously evaluate a trajectory. Args: prediction (str): The final predicted response. agent_trajectory (List[Tuple[AgentAction, str]]): The intermediate steps forming the agent trajectory. input (str): The input to the agent. reference (Optional[str]): The reference answer. Returns: dict: The evaluation result. N)rqrr_r`)rrrs r@_aevaluate_agent_trajectoryz4AgentTrajectoryEvaluator._aevaluate_agent_trajectorysH*%   + + "-      rc V|j|||jd||||d|S)rrnrqr`rr_r>)rarrs r@evaluate_agent_trajectoryz2AgentTrajectoryEvaluator.evaluate_agent_trajectorysG* ##iu#E.t.. !-     r?crK|j|||jd||||d|d{S7w)rrnrNr>)rarrs r@aevaluate_agent_trajectoryz3AgentTrajectoryEvaluator.aevaluate_agent_trajectorysU* ##iu#E5T55 !-       rrb) rqrerz!Sequence[tuple[AgentAction, str]]r`rer_rfrFrrIr{) r&r'r(r)rhrUrrrrrr>r?r@rrks@6 $(   <    !      8$(   <    !     J$(   <    !     H$(   <    !     r?r)#r) __future__rloggingabcrrcollections.abcrenumrtypingrr r warningsr langchain_core.agentsr langchain_core.language_modelsr langchain_core.runnables.configrlangchain.chains.baser getLoggerr&loggerrerrBrLrjr}rr>r?r@rs9"#$''-<;'   8 $3JC3Jl252././bq ncq ho nco dx ~sx r?