%GgۇdZddlmZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZddlmZddlmZddlmZ dd lmZmZmZmZddl Z dd l!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'e jPe)Z*Gd d eZ+GddedZ,GddeZ-GddedZ.GddZ/ee-e.e0fZ1GddeZ2ee2e0fZ3Gdde/Z4 d'dZ5dZ6d(dZ7d)dZ8Gd d!Z9 d*d"Z: d+d#Z; d,d$Z< d-d%Z=ee eejNeejLgee-e.ffe e ejNe ejLgee-e.fffZ>d.d&Z?y#e$rdd lmZmZmZmZY-wxYw)/z?This module contains the evaluator classes for evaluating runs.) annotationsN)abstractmethod) Any AwaitableCallableDictListLiteralOptionalSequenceUnioncast) TypedDict run_helpers)schemas) BaseModelFieldValidationError validator)wraps) SCORE_TYPE VALUE_TYPEExampleRunc(eZdZUdZded< ded<y)Categoryz$A category for categorical feedback.Optional[Union[float, int]]valuestrlabelN__name__ __module__ __qualname____doc____annotations__{/home/kushmeetdev/apache_webroot/langgraph_flaskproject/venv/lib/python3.12/site-packages/langsmith/evaluation/evaluator.pyrr2s. &&C J&r)rc@eZdZUdZded< ded< ded< ded<y ) FeedbackConfigziConfiguration to define a type of feedback. Applied on on the first creation of a feedback_key. z0Literal['continuous', 'categorical', 'freeform']typerminmaxz%Optional[List[Union[Category, dict]]] categoriesNr"r(r)r*r,r,;s* ;: $$; $$A55r)r,F)totalceZdZUdZded< dZded< dZded< dZd ed < dZd ed < e e Z ded< dZ ded< dZ ded< dZded< dZd ed< GddZedddZy)EvaluationResultzEvaluation result.r keyNrscorerrz Optional[str]commentzOptional[Dict] correction)default_factoryrevaluator_infoz%Optional[Union[FeedbackConfig, dict]]feedback_configOptional[Union[uuid.UUID, str]] source_run_id target_run_idextraceZdZdZdZy)EvaluationResult.ConfigzPydantic model configuration.FN)r#r$r%r& allow_extrar(r)r*Configr@es + r)rBT)prectd|vs|d.t|ttfrtj d||S)z$Check that the value is not numeric.r5zJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluess r*check_value_non_numericz(EvaluationResult.check_value_non_numericjsE & F7O$;!c5\*C! r))r#r$r%r&r'r5rr6r7rdictr9r:r<r=r>rBrrMr(r)r*r3r3Js H@E:0E:8!G]!2!%J%: 6ND65=AO:A;59M29659M29!E> ) wD! " r)r3ceZdZUdZded<y)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zList[EvaluationResult]resultsNr"r(r)r*rPrPys $#!r)rPcReZdZdZe d ddZ d ddZy) RunEvaluatorzEvaluator interface class.Ncy)zEvaluate an example.Nr()selfrunexampleevaluator_run_ids r* evaluate_runzRunEvaluator.evaluate_runsr)cKtjfd}tjj d|d{S7w)z#Evaluate an example asynchronously.ctjdi5jcdddS#1swYyxYw)Nr()rhtracing_contextrY)current_contextrXrWrVrUsr*_run_with_contextz5RunEvaluator.aevaluate_run.._run_with_contexts@##6o6 I((g7GH I I Is4=N)r\get_tracing_contextasyncioget_running_looprun_in_executor)rUrVrWrXr_r^s```` @r* aevaluate_runzRunEvaluator.aevaluate_runsH002 I I--/??FWXXXXsAA A ANNrVrrWOptional[Example]rXOptional[uuid.UUID]return*Union[EvaluationResult, EvaluationResults])r#r$r%r&rrYrdr(r)r*rSrSs}$&*04 # ###. # 4 ##&*04 Y  Y# Y. Y 4 Yr)rScHeZdZUdZded< ded< dZded< dZd ed <y) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. r r4z'Dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr;r<z6Optional[Union[str, Dict[Union[uuid.UUID, str], str]]]r6)r#r$r%r&r'r<r6r(r)r*rlrls8 H@ 33459M296FJG CJ:r)rlceZdZdZ d d dZ d ddZ ddZ ddZeddZ d ddZ d dfd Z d dd Z dd Z xZS)DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. ct|}|r t|}t||ddlm}|.|j |t |_t|dd|_tj|r<| td|j |t |_t|dd|_y|j ttttt gt"f|t |_t|dd|_y)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. rrNprocess_inputsr#roFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.)_normalize_evaluator_funcr langsmithrensure_traceable_serialize_inputsafuncgetattr_nameinspectiscoroutinefunction TypeErrorrrrr r_RUNNABLE_OUTPUTfuncrUrrxrs r*__init__zDynamicRunEvaluator.__init__s(). -e4Ed D)  $55&76DJ! 4IJDJ  & &t , 3 %55%66DJ!z3HIDJ#44XsHW$568HHI4P05DI!z3HIDJr)c<ttrjs|_S stddvr|r|jd<t fddDrtdtdid|iS#t $r}td|d}~wwxYw) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r4c3&K|]}|v ywNr().0kresults r* z@DynamicRunEvaluator._coerce_evaluation_result..sJq1F?J)r5rr6zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r<z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r()rEr3r< ValueErrorrzallr)rUrr< allow_no_keyes ` r*_coerce_evaluation_resultz-DynamicRunEvaluator._coerce_evaluation_results f. /'''4$M  FFLXOF"| $ u J,IJJ OOUhX$Q &P&PQ Q 44:8=  sAA>> BBBcd|vrB|j}|dDcgc]}|j||c}|d<tdi|S|jtt||dScc}w)NrQ)r<T)r<rr()copyrrPrrN)rUrQr<cprs r*_coerce_evaluation_resultsz.DynamicRunEvaluator._coerce_evaluation_resultss  B!+..q .NByM%*r* *-- w }4.  sA)ct|tr|js||_|St|}|j ||Sr)rEr3r<_format_evaluator_resultr)rUrr<s r*_format_resultz"DynamicRunEvaluator._format_result$sB f. /'''4$M)&1..v}EEr)ct|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. rxhasattrrUs r*is_asynczDynamicRunEvaluator.is_async2tW%%r)ct|dsPtj}|jr t d|j |j ||S|tj}d|ji}t|ddrt|j|d<|j||||d}|j||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.Nr= session_id experimentrun_idmetadatalangsmith_extra)rraget_event_loop is_running RuntimeErrorrun_until_completerduuiduuid4idryr rrr)rUrVrWrX running_looprrs r*rYz DynamicRunEvaluator.evaluate_run;s"tV$"113L&&("R $66t7I7I#w7WXX  ##zz| $3SVV#< 3 d +%(%8H\ "  '7XN  ""6+;<rzrs r*__repr__zDynamicRunEvaluator.__repr__s&tzzl!44r)r)rXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]rxzIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])F)rzUnion[EvaluationResult, dict]r< uuid.UUIDrboolrir3)rQzUnion[dict, EvaluationResults]r<rrirj)rzMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]r<rrirjrirrerf)rVrrWrgrXrh)rVrrWrgrirjrir )r#r$r%r&rrrrpropertyrrYrdrr __classcell__)rs@r*roros;, 0J 0J 0Jl# -!   < / !  4  " F  F ! F 4 F&&&*04 $= $=#$=. $= 4 $=R&*04 = =#=. =D6://!2/ 3/"5r)roct|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rors r* run_evaluatorrs t $$r)i'c^t|}t|tkDr|dtdz dz}|S)Nz...))reprlen_MAXSIZE)objss r* _maxsize_reprrs1 S A 1v n1  & Hr)ctt|jd}t|jd}||dS)NrVrW)rVrW)rget)inputs run_truncatedexample_truncateds r*rwrws5!&**U"34M%fjj&;< -> ??r)ceZdZdZ d d dZed dZ d ddZ d ddZ d ddZ ddZ e dd Z dd Z y)DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.Nct|}|r t|}t||ddlm}|.|j |t |_t|dd|_tj|r<| td|j |t |_t|dd|_y|j ttttt t"gt$f|t |_t|dd|_y)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rrNrqr#rors)$_normalize_comparison_evaluator_funcrrurrvrwrxryrzr{r|r}rrr rr r_COMPARISON_OUTPUTrrs r*rz&DynamicComparisonRunEvaluator.__init__s(4D9 8?Ed D)  $55&76DJ! 4IJDJ  & &t , 3 %55%66DJ!z3HIDJ#44!#(9:*,  15 DI!z3HIDJr)ct|dSrrrs r*rz&DynamicComparisonRunEvaluator.is_asyncrr)cXt|dsPtj}|jr t d|j |j ||Stj}|j|}|j||||d}|j|||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rrrtagsr) rrarrrr acompare_runsrr _get_tagsr_format_results)rUrunsrWrr<rrs r* compare_runsz*DynamicComparisonRunEvaluator.compare_runsstV$"113L&&("R $66&&tW5 ~~d#  '4dC  ##FM4@@r)cKt|ds|j||Stj}|j |}|j ||||dd{}|j |||S7w)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rxrrN)rrrrrrxr)rUrrWr<rrs r*rz+DynamicComparisonRunEvaluator.acompare_runss tW%$$T73 3 ~~d#zz  '4dC"   ##FM4@@  sAA: A8!A:c&|j||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rUrrWs r*rz&DynamicComparisonRunEvaluator.__call__3s  w//r)c"d|jdS)rz>r)cg}|D]^}|jdt|jzt|dds8|jdt|jz`|S)zExtract tags from runs.zrun:rNz experiment:)appendr rryr)rrrVs r*rz'DynamicComparisonRunEvaluator._get_tagsHs` AC KKSVV, -sL$/ MC,??@ A r)ct|tr|js||_|St|tr9t ||Dcic]\}}|j |c}}|j |d}n4t|trd|vr |j |d<nd|}t| tdid|i|Scc}}w#t$r}td||d}~wwxYw)N)rmr4r<r4zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r<zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r() rErlr<listziprrzrNrr)rUrr<rrVr5msgrs r*rz-DynamicComparisonRunEvaluator._format_resultsSs f8 9'''4$M  %;>tV;LMZS%3665=Mzz!.F  %F" $ u -%+I/ S/ ! -"M 6J 6J 6Jp&&AEA!A,=A #A@AEA!A,=A #A:AE0!0,=0 #0"?"="!" " $ "r)rct|S)z.Create a comaprison evaluator from a function.)rrs r*comparison_evaluatorrxs ) ..r)ct d tj jjDcgc]!\}}|j|j k7s |#}}} jjDcgc]-\}}|j tjjur|/c}}|r6t fd|Ds2t|Dcgc] }|vs| c}dk7rd d}t|t fd|Dr|ddgk(rStjr? d fd }td rtd |_|S|j|_|Sd fd }td rtd |_|S|j|_|Scc}}wcc}}wcc}w) N)rVrWroutputsreference_outputs attachmentsc32K|]}|vxs|vywrr(rpnameargs_with_defaultssupported_argss r*rz,_normalize_evaluator_func..) GLE^ # Bu0B'B B UInvalid evaluator function. Must have at least one argument. Supported arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc32K|]}|vxs|vywrr(rs r*rz,_normalize_evaluator_func..)CH>5,>#>>rrVrWcK|||r |jni|jxsi|r|jxsini|r|jxsinid}i}g}jj D]K\}}||vs |j |j |jfvr|j||D||||<M|i|d{S7wN)rVrWrrrr rrr parametersitemskindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYr rVrWarg_mapkwargsargs param_nameparamrsigs r*awrapperz+_normalize_evaluator_func..awrappers&07gnnR"{{0b@G7#6#6#<"RBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. E"4262222sA2C 6AC C C r#c~|||r |jni|jxsi|r|jxsini|r|jxsinid}i}g}jj D]K\}}||vs |j |j |jfvr|j||D||||<M|i|Srrrs r*wrapperz*_normalize_evaluator_func..wrappers&07gnnR"{{0b@G7#6#6#<"RBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. ET,V,,r))rVrrWrgrir~r{ signaturerrr VAR_KEYWORDdefault Parameteremptyrrrr|rryr# rrpall_argsarr r rr rs ` @@@r*rtrts N   D !C&)nn&:&:&<X(%!--@WXHX,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@AF G oLT     & &t , 3 3#4 3! 364,j)   O&&   O -44,j)   N%%   N}YEs!F)F)?2F/ F5F5c d tj jjDcgc]!\}}|j|j k7s |#}}} jjDcgc]-\}}|j tjjur|/c}}|r6t fd|Ds2t|Dcgc] }|vs| c}dk7rd d}t|t fd|Dr|ddgk(rStjr? d fd }td rtd |_|S|j|_|S d fd }td rtd |_|S|j|_|Scc}}wcc}}wcc}w) NrrWrrrc32K|]}|vxs|vywrr(rs r*rz7_normalize_comparison_evaluator_func..rrrrrc32K|]}|vxs|vywrr(rs r*rz7_normalize_comparison_evaluator_func..rrrrWcK|||r |jni|Dcgc]}|jxsic}|r|jxsinid}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M|i|d{Scc}w7 wNrrrrrrrrr rrWrVrrrrr rr s r*r z6_normalize_comparison_evaluator_func..awrappers!&07gnnR=ABc 1r 1BBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. E"426222 C3s#C C AC 1AC ?C  C r#c~|||r |jni|Dcgc]}|jxsic}|r|jxsinid}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M|i|Scc}wrrrs r*r z5_normalize_comparison_evaluator_func..wrapper:s!&07gnnR=ABc 1r 1BBI)>Br ),)=)=)? E%J!W, ::!77!11*!KK (;<181DF:. ET,V,,! CsB: )rrrWrgrirrrs ` @@@r*rrs SN   D !C&)nn&:&:&<X(%!--@WXHX,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@AF G oLT    & &t , 3# 3.? 3# 344,j)   O&&   O -# -.? -# -64,j)   N%%   N}YEs!F/F/?2F5 F;F;c@t|tttfrd|i}|S|st d|t|t r't d|Dst d|dd|i}|St|trd|i}|St|tr |St d|) Nr5zdExpected a non-empty dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got c3<K|]}t|tywr)rErN)rxs r*rz+_format_evaluator_result..fs71:a&7sz8Expected a list of dicts or EvaluationResults. Received .rQrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rErrGrFrrrr rN)rs r*rr[s&4,-6"* M);;A( D   FD !777J6(RST V$ M FC 6" M FD ! M &&,X /  r)c d tjjjDcgc]\}}| }}}jjDcgc]-\}}|jtj j ur|/c}}|r6t fd|Ds=t|Dcgc] }|vs| c}dk7rd d}|r |d|dz }t|t fd|Dr|dd gk(rS d fd }td rtd |_ |S|j|_ |Scc}}wcc}}wcc}w) Nrexamplesrrrc32K|]}|vxs|vywrr(rs r*rz/_normalize_summary_evaluator..rrrrr$z Received arguments c3&K|]}|v ywrr()rrrs r*rz/_normalize_summary_evaluator..s?U.(?rrr'c |||Dcgc]}|jc}|Dcgc]}|jxsic}|Dcgc]}|jxsic}d}i}g} jjD]K\}}||vs |j|j |j fvr|j||D||||<M |i|} t| tr| St| Scc}wcc}wcc}w)Nr&) rrrrrrrrrEr3r) rr'rWrVrrrrr rrr s r*r z-_normalize_summary_evaluator..wrappers $9ABg7>>B9=>#CKK-2->KS%Tgoo&;&;%T GFD%(^^%9%9%; A! E(zz33--& GJ$78-4Z-@z* A4*6*F&"23 +F3 3)C>%TsC+C0 C5 r#)rzSequence[schemas.Run]r'zSequence[schemas.Example]rirj) r{rrrrrrrrrrryr#) rrrrrrr rr rs ` @@@r*_normalize_summary_evaluatorr+sSN   D !C&)nn&:&:&<=(%=H=,,. E1 99G--33 3    PX   HDq1C(CD E J11?0@ C   )(15 5Co?h? ?8PD  4' 43L 4 7 4:*1z)BGD* % IPHXHX w>Es E %2E; EE)rr)rr)rrNrirN)rrrir)rrriz|Union[Callable[[Run, Optional[Example]], _RUNNABLE_OUTPUT], Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])rrrizUnion[Callable[[Sequence[Run], Optional[Example]], _COMPARISON_OUTPUT], Callable[[Sequence[Run], Optional[Example]], Awaitable[_COMPARISON_OUTPUT]]])rz;Union[EvaluationResults, dict, str, int, bool, float, list]rizUnion[EvaluationResults, dict])rrriSUMMARY_EVALUATOR_T)@r& __future__rrar{rabcrtypingrrrrr r r r r rtyping_extensionsrrurr\r pydantic.v1rrrr ImportErrorpydanticlogging functoolsrlangsmith.schemasrrrr getLoggerr#rHrr,r3rPrSrNr~rlrrorrrrwrrrtrrr,r+r(r)r*r8sE"    (' BB   8 $'y' 6Ye 6,y,^" "YY8)+Cs E++F?F