>'hڇdZddlmZddlZddlZddlZddlmZddlm Z m Z ddl m Z m Z mZmZmZmZddlmZddlmZdd lmZ dd lmZmZmZmZddlZdd l m!Z!dd l"m#Z#m$Z$m%Z%m&Z&ejNe(Z)Gd deZ*GddedZ+GddeZ,GddedZ-GddZ.ee,e-e/fZ0GddeZ1ee1e/fZ2Gdde.Z3 d(dZ4dZ5d)dZ6d*d Z7Gd!d"Z8 d+d#Z9 d,d$Z: d-d%Z; d.d&Zd/d'Z?y#e$rdd lmZmZmZmZY-wxYw)0z?This module contains the evaluator classes for evaluating runs.) annotationsN)abstractmethod) AwaitableSequence)AnyCallableLiteralOptionalUnioncast) TypedDict run_helpers)schemas) BaseModelFieldValidationError validator)wraps) SCORE_TYPE VALUE_TYPEExampleRunc(eZdZUdZded< ded<y)Categoryz$A category for categorical feedback.Optional[Union[float, int]]valuestrlabelN__name__ __module__ __qualname____doc____annotations__e/home/kushmeetdev/Regenta/Chatbot/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.pyrr/s. &&C J&r'rc@eZdZUdZded< ded< ded< ded<y ) FeedbackConfigziConfiguration to define a type of feedback. Applied on on the first creation of a feedback_key. z0Literal['continuous', 'categorical', 'freeform']typerminmaxz%Optional[list[Union[Category, dict]]] categoriesNr r&r'r(r*r*8s* ;: $$; $$A55r'r*F)totalceZdZUdZded< dZded< dZded< dZd ed < dZd ed < e e Z ded< dZ ded< dZ ded< dZded< dZd ed< GddZedddZy)EvaluationResultzEvaluation result.rkeyNrscorerrz Optional[str]commentzOptional[dict] correction)default_factorydictevaluator_infoz%Optional[Union[FeedbackConfig, dict]]feedback_configOptional[Union[uuid.UUID, str]] source_run_id target_run_idextraceZdZdZdZy)EvaluationResult.ConfigzPydantic model configuration.FN)r!r"r#r$ allow_extrar&r'r(Configr?bs + r'rAT)prectd|vs|d.t|ttfrtj d||S)z$Check that the value is not numeric.r3zJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluess r(check_value_non_numericz(EvaluationResult.check_value_non_numericgsE & F7O$;!c5\*C! r')r!r"r#r$r%r3rr4r5rr7r8r9r;r<r=rArrLr&r'r(r1r1Gs H@E:0E:8!G]!2!%J%: 6ND65=AO:A;59M29659M29!E> ) wD! " r'r1ceZdZUdZded<y)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zlist[EvaluationResult]resultsNr r&r'r(rNrNvs $#!r'rNcReZdZdZe d ddZ d ddZy) RunEvaluatorzEvaluator interface class.Ncy)zEvaluate an example.Nr&)selfrunexampleevaluator_run_ids r( evaluate_runzRunEvaluator.evaluate_runsr'cKtjfd}tjj d|d{S7w)z#Evaluate an example asynchronously.ctjdi5jcdddS#1swYyxYw)Nr&)rhtracing_contextrW)current_contextrVrUrTrSsr(_run_with_contextz5RunEvaluator.aevaluate_run.._run_with_contexts@##6o6 I((g7GH I I Is4=N)rZget_tracing_contextasyncioget_running_looprun_in_executor)rSrTrUrVr]r\s```` @r( aevaluate_runzRunEvaluator.aevaluate_runsH002 I I--/??FWXXXXsAA A ANNrTrrUOptional[Example]rVOptional[uuid.UUID]return*Union[EvaluationResult, EvaluationResults])r!r"r#r$rrWrbr&r'r(rQrQs}$&*04 # ###. # 4 ##&*04 Y  Y# Y. Y 4 Yr'rQcHeZdZUdZded< ded< dZded< dZd ed <y) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr2z'dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr:r;z6Optional[Union[str, dict[Union[uuid.UUID, str], str]]]r4)r!r"r#r$r%r;r4r&r'r(rjrjs8 H@ 33459M296FJG CJ:r'rjceZdZdZ d d dZ d ddZ ddZ ddZeddZ d ddZ d dfd Z d dd Z dd Z xZS)DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. ct|}|r t|}t||ddlm}|.|j |t |_t|dd|_tj|r<| td|j |t |_t|dd|_y|j ttttt gt"f|t |_t|dd|_y)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. rrNprocess_inputsr!rmFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.)_normalize_evaluator_funcr langsmithrensure_traceable_serialize_inputsafuncgetattr_nameinspectiscoroutinefunction TypeErrorr rrr r_RUNNABLE_OUTPUTfuncrSr}rvrs r(__init__zDynamicRunEvaluator.__init__s(). -e4Ed D)  $55&76DJ! 4IJDJ  & &t , 3 %55%66DJ!z3HIDJ#44XsHW$568HHI4P05DI!z3HIDJr'c<ttrjs|_S stddvr|r|jd<t fddDrtdtdid|iS#t $r}td|d}~wwxYw) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r2c3&K|]}|v ywNr&).0kresults r( z@DynamicRunEvaluator._coerce_evaluation_result..sJq1F?J)r3rr4zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r;z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r&)rDr1r; ValueErrorrxallr)rSrr; allow_no_keyes ` r(_coerce_evaluation_resultz-DynamicRunEvaluator._coerce_evaluation_results f. /'''4$M  FFLXOF"| $ u J,IJJ OOUhX$Q &P&PQ Q 44:8=  sAA>> BBBcd|vrB|j}|dDcgc]}|j||c}|d<tdi|S|jtt||dScc}w)NrO)r;T)r;rr&)copyrrNr r7)rSrOr;cprs r(_coerce_evaluation_resultsz.DynamicRunEvaluator._coerce_evaluation_resultss  B!+..q .NByM%*r* *-- w }4.  sA)ct|tr|js||_|St|}|j ||Sr)rDr1r;_format_evaluator_resultr)rSrr;s r(_format_resultz"DynamicRunEvaluator._format_result!sB f. /'''4$M)&1..v}EEr'ct|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. rvhasattrrSs r(is_asynczDynamicRunEvaluator.is_async/tW%%r'ct|dsPtj}|jr t d|j |j ||S|tj}d|ji}t|ddrt|j|d<|j||||d}|j||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. r}tCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.Nr< session_id experimentrun_idmetadatalangsmith_extra)rr_get_event_loop is_running RuntimeErrorrun_until_completerbuuiduuid4idrwrrr}r)rSrTrUrV running_looprrs r(rWz DynamicRunEvaluator.evaluate_run8s"tV$"113L&&("R $66t7I7I#w7WXX  ##zz| $3SVV#< 3 d +%(%8H\ "  '7XN  ""6+;<rxrs r(__repr__zDynamicRunEvaluator.__repr__s&tzzl!44r'r)r}XCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]rvzIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])F)rzUnion[EvaluationResult, dict]r; uuid.UUIDrboolrgr1)rOzUnion[dict, EvaluationResults]r;rrgrh)rzMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]r;rrgrhrgrrcrd)rTrrUrerVrf)rTrrUrergrhrgr)r!r"r#r$rrrrpropertyrrWrbrr __classcell__)rs@r(rmrms;, 0J 0J 0Jl# -!   < / !  4  " F  F ! F 4 F&&&*04 $= $=#$=. $= 4 $=R&*04 = =#=. =D6://!2/ 3/"5r'rmct|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rmr}s r( run_evaluatorrs t $$r'i'c^t|}t|tkDr|dtdz dz}|S)Nz...))reprlen_MAXSIZE)objss r( _maxsize_reprrs1 S A 1v n1  & Hr'ctt|jd}t|jd}||dS)NrTrU)rTrU)rget)inputs run_truncatedexample_truncateds r(rurus5!&**U"34M%fjj&;< -> ??r'ceZdZdZ d d dZed dZ d ddZ d ddZ d ddZ ddZ e dd Z dd Z y)DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.Nct|}|r t|}t||ddlm}|.|j |t |_t|dd|_tj|r<| td|j |t |_t|dd|_y|j ttttt t"gt$f|t |_t|dd|_y)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rrNror!rmrq)$_normalize_comparison_evaluator_funcrrsrrtrurvrwrxryrzr{r rrrr r_COMPARISON_OUTPUTr}r~s r(rz&DynamicComparisonRunEvaluator.__init__s(4D9 8?Ed D)  $55&76DJ! 4IJDJ  & &t , 3 %55%66DJ!z3HIDJ#44!#(9:*,  15 DI!z3HIDJr'ct|dSrrrs r(rz&DynamicComparisonRunEvaluator.is_asyncrr'cXt|dsPtj}|jr t d|j |j ||Stj}|j|}|j||||d}|j|||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. r}rrtagsr) rr_rrrr acompare_runsrr _get_tagsr}_format_results)rSrunsrUrr;rrs r( compare_runsz*DynamicComparisonRunEvaluator.compare_runsstV$"113L&&("R $66&&tW5 ~~d#  '4dC  ##FM4@@r'cKt|ds|j||Stj}|j |}|j ||||dd{}|j |||S7w)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rvrrN)rrrrrrvr)rSrrUr;rrs r(rz+DynamicComparisonRunEvaluator.acompare_runss tW%$$T73 3 ~~d#zz  '4dC"   ##FM4@@  sAA: A8!A:c&|j||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rSrrUs r(rz&DynamicComparisonRunEvaluator.__call__0s  w//r'c"d|jdS)rz>r'cg}|D]^}|jdt|jzt|dds8|jdt|jz`|S)zExtract tags from runs.zrun:rNz experiment:)appendrrrwr)rrrTs r(rz'DynamicComparisonRunEvaluator._get_tagsEs` AC KKSVV, -sL$/ MC,??@ A r'ct|tr|js||_|St|tr9t ||Dcic]\}}|j |c}}|j |d}n4t|trd|vr |j |d<nd|}t| tdid|i|Scc}}w#t$r}td||d}~wwxYw)N)rkr2r;r2zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r;zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r&) rDrjr;listziprrxr7rr)rSrr;rrTr3msgrs r(rz-DynamicComparisonRunEvaluator._format_resultsPs f8 9'''4$M  %;>tV;LMZS%3665=Mzz!.F  %F" $ u -%+I/ S/ ! -"M 6J 6J 6Jp&&AEA!A,=A #A@AEA!A,=A #A:AE0!0,=0 #0"?"="!" " $ "r'rct|S)z.Create a comaprison evaluator from a function.)rrs r(comparison_evaluatorrus ) ..r'ct d tj jjDcgc]!\}}|j|j k7s |#}}} jjDcgc]-\}}|j tjjur|/c}}|r6t fd|Ds2t|Dcgc] }|vs| c}dk7rd d}t|t fd|Dr|ddgk(rStjr? d fd }td rtd |_|S|j|_|Sd fd }td rtd |_|S|j|_|Scc}}wcc}}wcc}w) N)rTrUroutputsreference_outputs attachmentsc32K|]}|vxs|vywrr&rpnameargs_with_defaultssupported_argss r(rz,_normalize_evaluator_func..) GLE^ # Bu0B'B B UInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc32K|]}|vxs|vywrr&rs r(rz,_normalize_evaluator_func..)CH>5,>#>>rrTrUcK|||r |jni|jxsi|r|jxsini|r|jxsinid}i}g}jj D]K\}}||vs |j |j |jfvr|j||D||||<M|i|d{S7wN)rTrUrrrr rrr parametersitemskindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYr rTrUarg_mapkwargsargs param_nameparamr}sigs r(awrapperz+_normalize_evaluator_func..awrappers&07gnnR"{{0b@G7#6#6#<"RBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. E"4262222sA2C 6AC C C r!c~|||r |jni|jxsi|r|jxsini|r|jxsinid}i}g}jj D]K\}}||vs |j |j |jfvr|j||D||||<M|i|Srrrs r(wrapperz*_normalize_evaluator_func..wrappers&07gnnR"{{0b@G7#6#6#<"RBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. ET,V,,r')rTrrUrergr|ry signaturerrr VAR_KEYWORDdefault Parameteremptyrrrrzrrwr! r}rpall_argsarr r rrrs ` @@@r(rrrrs N   D !C&)nn&:&:&<X(%!--@WXHX,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@AF G oLT     & &t , 3 3#4 3! 364,j)   O&&   O -44,j)   N%%   N}YEs!F)F)?2F/ F5F5c d tj jjDcgc]!\}}|j|j k7s |#}}} jjDcgc]-\}}|j tjjur|/c}}|r6t fd|Ds2t|Dcgc] }|vs| c}dk7rd d}t|t fd|Dr|ddgk(rStjr? d fd }td rtd |_|S|j|_|S d fd }td rtd |_|S|j|_|Scc}}wcc}}wcc}w) NrrUrrrc32K|]}|vxs|vywrr&rs r(rz7_normalize_comparison_evaluator_func..rrrrrc32K|]}|vxs|vywrr&rs r(rz7_normalize_comparison_evaluator_func.. rrrrUcK|||r |jni|Dcgc]}|jxsic}|r|jxsinid}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M|i|d{Scc}w7 wNrrrrrrrrr rrUrTrrrrrr}rs r(r z6_normalize_comparison_evaluator_func..awrappers!&07gnnR=ABc 1r 1BBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. E"426222 C3s#C C AC 1AC ?C  C r!c~|||r |jni|Dcgc]}|jxsic}|r|jxsinid}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M|i|Scc}wrrrs r(r z5_normalize_comparison_evaluator_func..wrapper7s!&07gnnR=ABc 1r 1BBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. ET,V,,! CsB: )rrrUrergrr rs ` @@@r(rrs SN   D !C&)nn&:&:&<X(%!--@WXHX,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@AF G oLT    & &t , 3# 3.? 3# 344,j)   O&&   O -# -.? -# -64,j)   N%%   N}YEs!F/F/?2F5 F;F;c@t|tttfrd|i}|S|st d|t|t r't d|Dst d|dd|i}|St|trd|i}|St|tr |St d|) Nr3zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got c3<K|]}t|tywr)rDr7)rxs r(rz+_format_evaluator_result..cs71:a&7sz8Expected a list of dicts or EvaluationResults. Received .rOrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rDrrFrErrrrr7)rs r(rrXs&4,-6"* M);;A( D   FD !777J6(RST V$ M FC 6" M FD ! M &&,X /  r'c d tjjjDcgc]\}}| }}}jjDcgc]-\}}|jtj j ur|/c}}|r6t fd|Ds=t|Dcgc] }|vs| c}dk7rd d}|r |d|dz }t|t fd|Dr|dd gk(rS d fd }td rtd |_ |S|j|_ |Scc}}wcc}}wcc}w) Nrexamplesrrrc32K|]}|vxs|vywrr&rs r(rz/_normalize_summary_evaluator..rrrrr"z Received arguments c3&K|]}|v ywrr&)rrrs r(rz/_normalize_summary_evaluator..s?U.(?rrr%c |||Dcgc]}|jc}|Dcgc]}|jxsic}|Dcgc]}|jxsic}d}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M |i|} t| tr| St| Scc}wcc}wcc}w)Nr$) rrrrrrrrrDr1r) rr%rUrTrrrrrrr}rs r(r z-_normalize_summary_evaluator..wrappers $9ABg7>>B9=>#CKK-2->KS%Tgoo&;&;%T GFD%(^^%9%9%; A! E(zz33--& GJ$78-4Z-@z* A4*6*F&"23 +F3 3)C>%TsC+C0 C5 r!)rzSequence[schemas.Run]r%zSequence[schemas.Example]rgrh) ryr rrrrrrrrrrwr!) r}rrrrrr rrrs ` @@@r(_normalize_summary_evaluatorr)sSN   D !C&)nn&:&:&<=(%=H=,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@ C   )(15 5Co?h? ?8PD  4' 43L 4 7 4:*1z)BGD* % IPHXHX w>Es E %2E; EE)r}r)rr)rr7rgr7)r}rrgr)r}rrgz|Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])r}rrgzUnion[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]])rz;Union[EvaluationResults, dict, str, int, bool, float, list]rgzUnion[EvaluationResults, dict])r}rrgSUMMARY_EVALUATOR_T)@r$ __future__rr_ryrabcrcollections.abcrrtypingrrr r r r typing_extensionsr rsrrZr pydantic.v1rrrr ImportErrorpydanticlogging functoolsrlangsmith.schemasrrrr getLoggerr!rGrr*r1rNrQr7r|rjrrmrrrrurrrrrrrr*r)r&r'r(r7sE" /(' BB   8 $'y' 6Ye 6,y,^" "YY8)+Cs E++F?F