A'hE*dZddlmZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddZddZGddeeeZGddeeeZ Gdde Z!y)z-LLM Chains for evaluating question answering.) annotationsN)Sequence)AnyOptional) Callbacks)BaseLanguageModel)PromptTemplate) ConfigDict)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYctjd|jtj}|rF|j dj dk(ry|j dj dk(ry |jj djtjddtj}|j dk(ry|j dk(ry|jj d jtjddtj}|j dk(ry|j dk(ry y#t$rYywxYw) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translatestr maketransstring punctuation IndexError)textmatch first_word last_words i/home/kushmeetdev/Regenta/Chatbot/venv/lib/python3.12/site-packages/langchain/evaluation/qa/eval_chain.py _get_scorer*sC II5tzz|R]] SE ;;q>   !Y . [[^ ! ! #{ 2! JJL    # - -cmmBFDVDV.W X      *    ; .! JJL UWR  Ys}}RV-?-?@ A  ??  ) __ + -!.     s&=A&E5$E58A&E5E55 FFc^|j}t|}|d\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. )NN) reasoningvaluescore)rr*)r%r, parsed_scoresr-r.s r)_parse_string_eval_outputr02sD Iy)M! u$ u c2eZdZUdZdZded<edZeddZ e ddZ e dd Z e dd Z e d dd Z dd d  ddZddZd d d dd ddZd d d dd ddZy ) QAEvalChainz,LLM Chain for evaluating question answering.resultsr output_keyignoreextracyNFclss r)is_lc_serializablezQAEvalChain.is_lc_serializableQr1cy)N correctnessr;selfs r)evaluation_namezQAEvalChain.evaluation_nameUsr1cyNTr;rBs r)requires_referencezQAEvalChain.requires_referenceYr1cyrFr;rBs r)requires_inputzQAEvalChain.requires_input]rHr1Nc |xst}hd}|t|jk7rtd|d|j|d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. >queryanswerresultInput variables should be , but got llmpromptr;)rsetinput_variables ValueError)r=rRrSkwargsexpected_input_varss r)from_llmzQAEvalChain.from_llmask,!6; #f&<&<"= =,-@,AB!1124 4s64V44r1 callbacksct|Dcgc]\}}|||||||d} }}|j| |Scc}}w)5Evaluate question answering examples and predictions.rLrMrNrZ enumerateapply) rCexamples predictions question_key answer_keyprediction_keyr[iexampleinputss r)evaluatezQAEvalChain.evaluatesd"(1  7 !.!*-%a.8   zz&Iz66 Acft||j}t|vr|t|t<|SNr0r5rrCrN parsed_results r)_prepare_outputzQAEvalChain._prepare_output11&2IJ f %+G_M' "r1F referenceinputr[include_run_infoc B||||d||}|j|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r^r[rvrqrC predictionrtrur[rvrWrNs r)_evaluate_stringszQAEvalChain._evaluate_stringss70#$   - ##F++r1cpK|j|||d||d{}|j|S7w)Nr^rir[rvacallrqrzs r)_aevaluate_stringszQAEvalChain._aevaluate_stringssHzz"i:N-"   ##F++  646returnboolrr rm)rRrrSOptional[PromptTemplate]rWrrr3r^)rbSequence[dict]rcrrdr rer rfr r[rr list[dict]rNdictrrr{r rt Optional[str]rurr[rrvrrWrrr)__name__ __module__ __qualname____doc__r5__annotations__r model_config classmethodr>propertyrDrGrJrYrjrqr|rr;r1r)r3r3Hs6JL,05 5)5 5  55D$"& 7 $7 7$7 7  7  77 7,$(##!&!,!,! !,  !,  !,!,!, !,N$(##!&,,! ,  ,  ,,, ,r1r3c4eZdZdZeddZeddZeddZe dZ eddZ eddZ e d dd Z dd d  dd Zdd Zd d d dd ddZd d d dd ddZy )ContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextcyr:r;r<s r)r>z%ContextQAEvalChain.is_lc_serializabler?r1cy)z.Whether the chain requires a reference string.Tr;rBs r)rGz%ContextQAEvalChain.requires_referencer1cy)z+Whether the chain requires an input string.Tr;rBs r)rJz!ContextQAEvalChain.requires_inputrr1r6r7crhd}|t|jk7rtd|d|jy)N>rLrNcontextrOrP)rTrUrV)r=rSrXs r)_validate_input_varsz'ContextQAEvalChain._validate_input_varssM< #f&<&<"= =,-@,AB!1124  >r1cy)NzContextual Accuracyr;rBs r)rDz"ContextQAEvalChain.evaluation_names$r1Nc N|xst}|j||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rQr;)r rr=rRrSrWs r)rYzContextQAEvalChain.from_llms1,)>   (4s64V44r1rZct|Dcgc]\}}|||||||d} }}|j| |Scc}}w)r]rLrrNrZr_) rCrbrcrd context_keyrfr[rgrhris r)rjzContextQAEvalChain.evaluate sd"(1  7 !.";/%a.8   zz&Iz66 rkcft||j}t|vr|t|t<|Srmrnros r)rqz"ContextQAEvalChain._prepare_output#rrr1Frsc B||||d||}|j|S)Nrrxryrzs r)r|z$ContextQAEvalChain._evaluate_strings)s7$$   - ##F++r1cpK|j|||d||d{}|j|S7w)Nrr~rrzs r)rz%ContextQAEvalChain._aevaluate_strings>sHzz"yJO-"   ##F++  rr)rSr rNonerrm)rRrrSrrWrrrr)rbrrcrrdr rr rfr r[rrrrr)rrrrrr>rrGrJr rrrDrYrjrqr|rr;r1r)rrs=L%%,05 5)5 5  55:$$& 7 $77 7 7  7  77 7,$(##!&,,! ,  ,  ,,, ,2$(##!&,,! ,  ,  ,,, ,r1rcXeZdZdZeddZeddZe d d dZy) CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.cyr:r;r<s r)r>z!CotQAEvalChain.is_lc_serializableSr?r1cy)NzCOT Contextual Accuracyr;rBs r)rDzCotQAEvalChain.evaluation_nameWs(r1Nc N|xst}|j||d||d|S)zLoad QA Eval Chain from LLM.rQr;)r rrs r)rYzCotQAEvalChain.from_llm[s1%:   (4s64V44r1rrrm)rRrrSrrWrrr) rrrrrr>rrDrYr;r1r)rrPsjG)),0 5  5) 5 5  5 5r1r)r%r rzOptional[tuple[str, int]])r%r rr)"r __future__rrr"collections.abcrtypingrr langchain_core.callbacks.managerrlangchain_core.language_modelsrlangchain_core.promptsr pydanticr langchain.chains.llmr #langchain.evaluation.qa.eval_promptr r rlangchain.evaluation.schemarrlangchain.schemarr*r0r3rrr;r1r)rso3" $ 6<1)RRE$:,F,(O\F,R|,?L|,~5'5r1