From 20ef271aa2ec4e7e051ddc722bbf4c1d5021d833 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Mon, 10 Mar 2025 12:49:30 -0700 Subject: [PATCH 01/90] Uploading Michael's notebook before testing --- .../validation_of_application_scorecard.ipynb | 1448 +++++++++++++++++ 1 file changed, 1448 insertions(+) create mode 100644 notebooks/code_sharing/credit_risk/validation_of_application_scorecard.ipynb diff --git a/notebooks/code_sharing/credit_risk/validation_of_application_scorecard.ipynb b/notebooks/code_sharing/credit_risk/validation_of_application_scorecard.ipynb new file mode 100644 index 000000000..17a1007e7 --- /dev/null +++ b/notebooks/code_sharing/credit_risk/validation_of_application_scorecard.ipynb @@ -0,0 +1,1448 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Validation of an application scorecard model \n", + "\n", + "As a model validator, your task is to independently assess the application scorecard model developed using the ValidMind Library, which is based on Kaggle's [Lending Club](https://www.kaggle.com/datasets/devanshi23/loan-data-2007-2014/data. Your role focuses on evaluating the model developer's work by conducting thorough testing and validation, potentially including the use of challenger models to benchmark performance.\n", + "\n", + "An application scorecard model is a type of statistical model used in credit scoring to evaluate the creditworthiness of potential borrowers by generating a score based on various characteristics of an applicant such as credit history, income, employment status, and other relevant financial data.\n", + " - This score assists lenders in making informed decisions about whether to approve or reject loan applications, as well as in determining the terms of the loan, including interest rates and credit limits.\n", + " - Effective validation of application scorecard models ensures that lenders can manage risk efficiently while maintaining a fast and transparent loan application process for applicants.\n", + "\n", + "This interactive notebook provides a step-by-step guide for:\n", + "\n", + " - Loading the developer model and verifying the data quality steps performed by the model developer.\n", + " - Independently replicating the model's results and conducting additional tests to assess performance, stability, and robustness.\n", + " - Setting up test inputs and challenger models for comparative analysis.\n", + " - Running validation tests, analyzing results, and logging findings to ValidMind." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "## About ValidMind\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.\n", + "\n", + "\n", + "\n", + "### Before you begin\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language.\n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html).\n", + "\n", + "\n", + "\n", + "### New to ValidMind?\n", + "If you haven't already seen our [Get started with the ValidMind Library](https://docs.validmind.ai/developer/get-started-validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "\n", + "
C`lx3fq=AU2DMz>ASdDO} z73F?C+W&UP{ByBQ_H6rJ7z2?C1S|-$hy?k~LEoS(oC9nFP(RcOG3 9^hK=XiOGXffYjR2>KrcVAd0PQ12DI9jEl;Mv &gC8n$c3-!^I|NCy+~mXFtQAxRV5zP96cJP8r29?ufJcPNBvZY9En{_lln zUMk`9>NNhHrL|aL=XzmqyE0+^^P&9mi|@GC`#K5hClAHC*I(m~5659EBOhLWaT;FQ zrW5{PpNTtdtIJ&-(SlE#YAN*iq$jLjJXQEQ=e`iR&JHhY^AyK4br4eT`w30HgyTam zN(EJNTf8)-wQxGyjo<#QnETvm7|zUoBHWwS4}TbLkFVBUBseJz1oOqW`Irxp97!@j z=dQKj)K&Ki%?t#=#e602z ++J-ai9U!@y@-d%0TNj)3$l~*7f-J}YB7&${&`{ong zGG!MwI;F PhwRHr?YIRXwVrLxd9)#Deu_>Rse>t%&g*bcOk}OZZQOsL?NSbcj zgB jBTGi&d Vw6W2xP`b*jB~y-Dps>CEk#R#J*yl8=5y z4>JDzH75JK3;W)9iPT7;uSiPi&o1lcmvH zn6`C>5~fZGxs&OmT3E)AH7`1_R?Wi{RlaqXj9bhnNeerXHkX>KZZ>z4>a42HI(!^Q zS{3wDESR;CBusj#n()#|b@Y;vZ|<75q_je9X4|$ds&_htOrBVklqR=i{D)1I^lZ+O zv`MeX$d4mc*~6czTIAGWOtRWbHC{AUz3rnSx}M_{s(bH=qgj#SblsVX)qB#IG4v~~ zV<#Uc%k$0`PszW_Fp*J=TeTXhJ1@5TF8olact&5Vt XSy* ely&bQkP;+sYhu*aiTP&xKiMEyy3UI;rF_mQQ&vE;kUTq_qX9U zw_)9;q;M2W5h<-GZ7A*F>7B*_ln#_mlrEI6l $18;Y{#1F6{7_m;;U5Veyz}RNT;3@|&fsuIu11alDs0PgwQKc7UQ$1$ zj=@H>^HeJ8zUd>HbEG|*=syu%oY9u6YRD=_uP#B#OGsI-B2tdZor`+!EkSKl@{!^A z3(EKFwjrg%86*zkk?X^cXaK2!UQP@|2fVwX?#b!s{WVWzQ3a$tRS%)K)BTX?%Q$4b z;tEo3?8k-;>rJns4sy)why=UmsLGWnG-#y{^6q>axwKFzAM72gJYKOrnldC@SyJ~s z8gJ1JHSC;*{JTv-UC1Sb1A8eS9& v(3?Ts4GT!r|jJy98I z4Q>34L7o+VB&q%SlIZ`x&me#~1e~>iX#`xffSCjuX#tZ7z)$}JU_JqOlNkV03cx1; z05Gcnd>#M*6AQqn0st_#0DLw80MiS=Cj 15CJt<&J`cnE)LMh>t{*-}~L6j&;3?+^d zPf4T ;^KKrXBEugSP0t z#wU&*tMQxD_7l=HG8?>PO3P@D2e}N6(Y7f|aM0Gz{=oiY=?n~p5e1BdA?5o3=@7t} z$mr2+U5fl{YCyLuYxf~dp8^45C 9$0UP|af}a@h*hhGLWU%nfb%2nU=_wvRyIuGgy%jHC zFiPmC9ECqtb4Ops2Z}9>MJ#<=EnIsV&0oEK56`~YQ@H4HLeL3y5Zu zwSqriGS>%>-~JQ_^u2(eBzDEm5*!2@vw>Xf$4_X-kt}XRVS7H;wmEmF&MjWiZwucb zwE+99OvfYI*$C}Qh6xqhSz%+_M%;z%ru@n&Cvo3x^Ki3-0&INqHGj2hBwjS|CN}7_ zLHX@%A^&;Q0ny>O3r@-N=UYzC!fT4sg_%9_M7__c%93N75Q)y^Q}PG!xjiTH=Zp-n zIzv~8diWWis&QAI6#ur58~n?@B$}dno%c)%=$WQ6KAtHh2302SO1hCi7eB?j)IuqL zO_0>R&JZcOjXk;0V;Y%jH$>$xrVvvDU52e=C2iF&lzJ6sNb?WemzsQzmfqU-lMJp{ zOLe^3kd#KP$<62O33FjEsae^EvGGnO+f8RMsmt7UUO61WWF0C|SL$Ie<%LZpGivvj z_AXFL*B?HU#@sz6`99vp*0LWe<;=s<)CbAZtqMyO^P4x3hE>@?%uo0#KAl=X&goy0 zID>Cw#px81^T^X@L992)J*lh0dzbkN>VvG~0#|86wasMQ=QKru_feLg;-J{wP=|!< zyvA5m9?6>6?~!7yu1N-y^oet|f#l7zEN0o_aAITYD%m6l6+1e0lsw!D$!_z{l8;js znK;3Y#NQsHD(W31RjT@mxiI%5`TD9hvo-9H)aLbi-&+f7Fkz2M$?m1~$eJ(y#67>3 z!mfG{>2C8;)opzmo7%91ytr~iHR(wj*;X=F^~LTg8TRsqYQcerq@8K5YI;F7u?@CT zOb@TAI(@mA88df*YWEf=MZ=f5s;$OeiglhR*}4Yz7@hkki2kIbQr$|+iR0Bz#kF%& zRT*(9iiNKSG4BqiDb6G fBhshIBXGTe&;hfvp*fx9$<%DtKLO= zi{2x}8B^ui_QO##Cnai8@4a%NzAu`8xeNtOk5k^-Sb{Q6uR@oru0-AYvB+bzf@^9q zR5|i&8S45qRXOL89kO0>M!BM}qw@5v_sa2}^xL0vpD8zXnht*s%1 ud4&lhLd|y&}TbjD0M{l0*)c&qt7U!U@IEYdJ>w@`4bxBm50VMw#t2X z{7{d@H&N(C2_0G75)}_PgwC{g{)hk6{>uu=v*MRf_W#cJ?3YmXZxYJr-=;<+)80OQ z31$BeZXbW@YnB3^D**8be6%#86ZIMUO-!Pql-9osL?_u_OZFiHK}zFSrWL_J=z o~|@@f5WE|3Pt|LhO!e=d~C0+}3xumnLCrC=Wf zCRxyu1tz(ioCiV|2ySp32vrcvb;u&x_ktMI1N(41)CI@NA{gw;_0Y}VwGGF~eS%{l z$Z3!cbwc}c4E=&_$dl`bH2GM%`Ev#N$>J|6ludiNoPMp-&oA}PRbk!O$AaT@rTFUo z5WmXD(=nOl310){7ta46n3H@l@2Hp9qFcOhe)lHv;`$(*k<^#JGPVbvZtEj7TW`;; z9?_T|@TC`?*f0~%>$Foec)w3{Uc6R#J@lavH~5V(@$NRE=Jq#Y1wB7;_PXYLD%%PB zAD+p_n^x!j-Y(= 0XxrK0PX%*3uvV1PbJ(#z>y%yK4lEy!rUW?z` z<~ pskQHO~7|u@8Vin zRfO%_IKe){T lbFNYq34K8cE;uYY z?7z#S>|Z97C8##NN|$P#>{NX0K%g|L#ucgJ1}oLsfrA*Uf@M;4Ag8uAT`E b{^SN6hTTq1}Q7@!^vU# zTV;ihhN&J|lqinY^!9Pvoyw#w`K-9OXdZLY?UK5l?rtgV(_5(^!$wk$NSD%PG$-q- zJ(h-t#Y#q9H%ftStH?6vm+GdzXBeB^Na}rEN9psyTk;KmPR2KWO!fxt^yy#Pj-1;) zPPO3PD^-uZr dL`;B=62}vfcX(Y15)JY17k2RWi7eWVn4H<2!nw@7O&V zq-MzwrfB~IY4VgBs{3|LnWKx+*`yH{Rr}rJB#V2WNe6dVQe)^+pZB((ijTW1h{w{0 zs)6&v6tylGk)g&pzRia(QRRMk@2%gXgy?>&?b~6_Ek#S?`Hb uQ`LXN;U6gOxaEpdpH9U^MztvTxm{80p7L@7~D@sjD zZHg_W+&@gazGy(xoG7rq1IGt&b^r$laB2WY25?>ghXrs_0LKJ|Qtl%rKbq4tk >&M#-_+J zC|vn0F%@Oqe!!XDxq|pZiOBnqi}FmG4%@Y?pEBXADU)$Dm0l4YRO#F_<+wa+B(AG~ za)xz9=Z7bwJ8uJ()9>Cu-UXM?z#Uec!=sldzxx=}ae^_sr(v9Olc6rcbNef$Byjc6 z1m)y7H)Tdg19ZI64pe=cpYp3`vT~Wv3CL6i 72QPe6_A6H)zzu;Tr%4D!|fBbh2; z44vEGoi+T41MsviKmEg#IXsO6@TU*RD+e$G06eJ!U=jdWDF9#|0Anp+Du8NQz-$1f zTEK+JW&}(L_zMAGQULHB0>Hcg)>^>S0JXG$*#W@62mliVfUgk%<_NIUVozhhK}!P~ z1CCn2Q~}Of8qyeW(b9;<0GKhLi54(vfSVRDZ-9puFm-^Z7BG823oT#*0r(>Tz#IZT zTEH{{;0L9UFLbr|$N>WrgTn?!Muo>m#C-pyodjAAt}9$ixQ1}e;5xzefolTS!JGnP z4&w~t3u6jn2;*i)fibENCXvQ4J}@RQ4siZ(-f*sPesE539?);-FZ7Y6APP?rpbiLh z+mF(k(v||J(Von+JcGfs}5PAWBb4Z%SWEKT0SioYJ2%kTQr8MTw!rQQ|d?{1^TY z1`5KT%1Q|!EB(Enl}1*g@{R2ORPM6`at-y@Qc~Y6jZ8z!Xrv|ZaiWj6koKEjRaYBX z>XD@?kanP~<`qIJ?Z`1mNg7`%ka@I^x0=@xL5c!szQ*s1Zo}hkxuit*HFZ$AO7j{d z)I((^c%894o%Rd*4*MFp367z@UjOZnw?CGP;13)OFa+6XLkvNW0>&B)I0RXKk&QIu zK^X|J4FO^>5@0Ok
#n ;eu;ZV fRoMr|LW!lxEi;ObW~572H6D=EouV|k=XEl4 zY;zKqpI^+Tw3HUd)+U8IW68$6evGr_dexJVG-i~Oi(+p#%<8>)Ev-6WCK(ydV|N^_ zBQZCfrRG&05TisVGNsKD_UapjlxSflg#|2RtBvR;*+$$YuX|P`ibMy+jA7eJ?BZr> z`*V&`^wZv?%GJ}-v^+zmQN78M#{~z~l2RMCalPJT^Zl8sgkhIS@yWW()*6LHEsmXG z>@A9jX>DuP-q%a2z2YsIJmsOPU;H-7`|&U$G%aI&RPC9K*+yhvy;tPQ40qCM!Wcz? zK~*w!>seCKBh9x@j{>R59Se5by->-#`WIry93l-y4`;4-IZD x~q7D>6vm*B6TQ)2-;;yP9H0Rt4ggR+qfU7)oj{yujEbc@iY( zOEz^+ky^9*GcDS=OP1X?5Nq;UQNciu=+qj;JTOc5O?60TjB1`HCY_tBF5h%1n)dPn z2~uol{BM^j()*oee3$oBB{{Y9ef(yGYTS?tiYd<-)jXvId#v<2llA%;qx(p=e7gRn z#GjOZw+a7Ud-9lp3}h(p{8$Ep&VGOEh1)SaZonf1tm|McC$HaNy#{MESeL jiK5`X1KSqn5K0wB=bWrQZS;(`r8#pC*W9E! $nE1M$w{Ef<0i+7=rJWplh#MVkaLO{6Z3+1kElTcL)f8@Gs zhca-N4(j-MIm$6CL5*YgBfO_bd9w_mx|4RH9o+|`%#KmYPhCo({+}_(v*M3rrC+ZJ z{<*va<`1Z^1xz6Tath$A1xzF0q6N$(&`1lIOaNpT0L&-wZ;_j5J7AIlkahmjuL;ur zx7Xf2I@0a`Ysrb$kw_Uz8BQ5V`SqIMKlGX)NKzmNX{0IeUZRqZM()wdNE*oqJd)to zv#9(d|B{u)O9{NE!25|xUQiCCDb2CqX{42ne)~17zsXk`&nRu(AaBVsmd3k@N>7jm z5A2$9Adf-2;Q0i7)8@;bRPbwBnmpM{i yYIoh@q_925if9 zLw!&e1jvVNNP__7pe_iI2itN!v 2m?2-^K k zIJN2)yu{+UVCB43oO?D?EWW-9x7@GhJ8u~+G*~f4&=09XJ(G^$p|%&qx;1VHUk#1; ziG8o()MRg*ld}-dQ5_MJolgn<4W5b1mEDELdNZ+Jye;3g^8_w)d?UWv^ZSDSoJ?%c z@|vJ~WenOaHs@?sjN{LJzRb_RJsvx)$;DS<97X*Jw!)5gu|k!4Ex8VB4)Fa>9mZ`< zUh%yaf8t-hbHGj4t-&3S8u9yj)D#4#V6k1DkGM)LCjqBN2;Y1oME|l+{F!bG1@mEQ z-kTZDuM|h~EtYpjZ$f6H%?5;PQSl>syYv)S<~5JE=o^4v9}xL%$=~>9cShk2cB^qi zw^gW#jtlBGQORBAAMp8&?D=Yj4!o=SJ|8+sm%nD_g7dAu@h^fF2>uo;g~bLD!f4$M z_}--uT(8Dcyu0W<=T$sFxjp?GU%lioH>*Bf9!8GgXZqjg&AR(?Ik{=PqERHDQ}&cA z-Xvk`mI=7hB7Humw+Y{1i@=9}eZn7bu@+o;8+_VR5Z!hR;ZN+l$yd5^5q~~C1W$DD zDYzQf7FKNy5)3!A7hRH^_>*7Fm9y$u;?x-|UfKuAlj7eclzIHigt7#c@90Fb+AqDh z?}aqUW=l^pso^7{pK7EqeHbK}9G{}r`OsI=Ej>tjcRxg)?1)oI6E+cNrja7&Vc{hiX^Zpjn0%EL~ z5O7#BKXy&g-_?>tTO=vgju|0M+!rBr8qkzYwi!&;1nnlVE=nfi)fiQaoky6_mh^YN zS~Mf6YxK$Wv!8r>ES;@t*vwY3OYB^nX?>BIVKGRmeCG+%%C&v*xG#+eo+lFf_ID)4 zxPtGJo5|9k%k>mb4`z|FWAmf|P3|bN_6}t_E$KsE4{~KTH_0Yd#s;Tmr;i~Y( BT3nVy084ii#>V zs;Oe1Tu_W(|5$Onz>DmAe}fb_%x4y74pg03^ig3vaT)P6J x@?a (s^KSnid*=aGMY867P@ -yM6?4Lf0T4t5vv9gwRLnW&oH1b5QNgd? z<7q$l?l61r?%kQ4aq4+~R(Ex!4!l+M_w7DtMtcfdE1IC|k9kbhypQhpN<^DSUB*pj zH$|`8xZr_rcH%1Cub~O|HltS?R$$%Yd*n|3hk$ iq;BIDmx~@Wuf^Q3v2A1E8)0;H3l9(E=Vj z0I2Q&JPd%XmU<)xK#d3NwAhmvaM0pNVgMcr0Hi43pOvFz8OoEC0lc&{Co#}MOG^?1 zt+cczG2pGmhr|H ZeI|tJ1Tz9$cevJYec_tI zb%bjN*UOFo*QOz%F#)a#TniTh$Zv4IaGr2(a6WJj&~NB3^br$S0#ES$M>9&&{0IRA zcxKxXIuL>hp@hy_^W9Y&cPD8*2)zh>2>l5C2?GcN34;jHgrS7tguk2@6kH?-|D5*| z$TjlIW*{wTq#^hPLXdvQF_3v=35g~jeFqsw _AzlhwW*@pf>8^p^>Q=V-9Rj0QV%SG@94iXOpT>|`}!yh@!*C2q@0`oUG z25DdfAb=65qy>%vP!94!88`;pa)5dv9Rk!L&k3O(xh$0b#~`->F_eQk1JJgd2I+D+IVMSejUfMlS0Dqyz5cy}+@*@dpHaTEzEtBM7%F`ex_EJ`27B^i z^l(+$)@Ht*<+j|? wF-A;R0vmXWHBd{Oyef4 zU&|*3*HsNTlC1LX?7~O(Ojqf;UT2P(N8&rDkFjqf1a{$#olLu?5zHO~JJz~Pft!Sx zGxmc{u*o%UG0r2MSw+WJc!MKiN*WumPU~y3JEzy@%#SB=LC>c$6Qq7z)h^y#%CG{q zJW$CVKjX>;L^WWZjcd!!Wy{z(mj-jQ)f4%V^EUJId&aZOo!VT$u#SAfkt4kM!$9sJ z $DY`Od6id8m; zEV)~IBe)m8T;-odd2?>-%H^B=dx<*#KJTXl!PsGiaIIdfQjxn;Trek7SaEQK(8j4) z`Fej1QP{s!)M NML?^rRYB#)lIx@h-;^Pkm~QlqHsI2|E9Xc)EF$4okuTqHi4 z+*v$X E*ma D7C# z7#dn%k<{t7RLk>-(qe6{n0T@lwe|f8rP1)d!Ut)Q`a$Ygs#?@&^_nfi#0 S$3tK}eV=bQokr4IVI2suSBv44Uvo3i3QE zR41a)*vgsIyxikrk#B;!xWxvkjmcc9)q?_xX_rMmDY6otPOGAbYoVu@TYm!;ls;FO zEbXPP>opZFEpk*GZuV7ubH1fAr* b9}@>K&tsH(hEu^ 63!ChTX;{L@3ISKZrf?L0q-#F*H|yA9nPfizhg|#!EY-Bf%vfcS({^$;U+8 z`dTwQ#&t5f_3SVXd9VfRJlujOz1xVZzn_i=mh{2>HYA~e)0*Jv@o&++#vRb99vg6d z$LqM&=w|rvzIc@KCJp;dxj~+gpE}4-$FB)a2?^wH<%g>yuQlX#0hkj2=J)`t1prtp z0AOZ*ocn{x0btz$fcXJTw174ZfVcsGP7Q$20f1%=fan2OXaNBP_`L K#bCeK19sYh(sV5X+L)k{zxE7X~Zrf zDrp2J5WE2JVS-odi0}lW64U|50T8V~MAFm;`9N?2NM4PY1tJ%u10asUF(N=|`T>HM z_DfJ8q=EmFrafqr^i3mR5#j2)sP r?Fp$^00RRe z4+t=jKmdaRh6MIyQ3 _ v;*Eema;$J@G2d3@d z_EkH?&!~Nv_pnIh^SsxyS;=*{nOm-~N!$YVtL=04!3rxb+1rTe7!ioyXTIduQEB|M zs*&uGwcfmq)f_%0-kiJNFrGg;?Fu*PXbE3!@NzEDumRKTLNSx$Ofk>v1+nw2j&XHs zWN=x^Ih@;>a`x C2Z4rMvQt%XJ(b@9`@Y2wru3YE6n=Kact~>-h8EYTT~5u z`fvv(jpg+({lbO?PvWxH#c*TZn)02urt-$Nj$HjK;e4J^06H+zgN+_Cn%V0;fwN4v z;VKm`W~(n~%Dj2>1;3v^hb`S+%&vLo&Kd@tW=E7TT+oL^y!e?PqhIQX`?qd}lcrzc zo*1v^JHGG9Kk{&8>$Wi=-ePyzR%_>RlNv4Nc2!N`;;irT!7T=JCmYOU-3?dr+nUzo zeBRq|=9?a|cTQd7#C7_d&%l0cqocN*{lXRU!|{8GZJw><7R^RQ#YXgwj_5m}_t=Q= z(ZfcD$VYTNCQgi#cTD>A9Tm|#CIUXKFe rQwUld;z4wALkF+~?w9p$>|n0jrX5a%49rJf+Xr-aql z1?-t4K7DpmY-8S>dg-1ZjPKfn4ry0KN=)w}KI#2Y*=Js?n2R%&=c -oZ>?i^I}}6$z5hh&GZt*!S<>2kcS;;+v=;N z$1c;vkQAQMKS$F=r>&)F9=mDk{ZOf3`XlLvVH?ry!X6>j{RY*y#V5(()O2dc+nv dKy!03Yv?>PGhMaZ5N3B^)6AX-b@qT zCy!7cF;1h;)LTh)-%w3#etQ%Z_TZ+F7MLk1(jE%WA3RkojeRMk)f%kK3#dyEVO~=! zLuyd3hqjSwEbdI3c0VY&pKK$pe_caLzLhGa9GxLf>zGEZ>$*y~z#db+*zG9 o&DJDwen2vPI6Z@36ZSRU-w!fvD z)U2r#7=JpaUuWU?rbSez{R`xWpvvE3+rKch$X5--HWPX0Z?WxfvF#t1L}XunJbpuL zi@l9-y%+ZA+JebA{dz9$qlm_>$D3n*fPfE}X5mv4KcZ8gvav(=nK*4k5!UP636%r} z;sIt4@kPT4ND#;4VeCgdsDCnE*sdcU`$@!u$_r3otzOu+N@u+0;zjJ+t0OA2-G>dk zvuK<8DDK|Y2Dhnt6|ZP@7w_*g8>30{P|1`Fc<$C^cxLSqTz#s5lk5BA)nV?~*-Zyq zy^6<9FGiuv(nM6RiGX%H_Cz_gV^Fv9acJZ0V!XVh5$-*@2R5Cw5glHXgY8!2qfaiK zkw7`1ds~iT>o;ZSka+~!x27-d(d95MPFjYWj%e^N{L%d-v&v7$-(uUp?%f3s5Ae6x z_W#$de~WFuVuJsQNwVSiFT^$?Mrl4@t`W+<32q>6$zqu7-33CKMnof?Wgxi80-8n? zBl%?!?z`6(5%InWa70KW0-DBSi-=$Hx3@s#gR(!?`8_{~Z$ub_I%RQ9Q$G>#K%mp) zBXw!SKG+A(Fc8v+7zg#s>EB+Q{8pEyJ*}skMxY~t+uu|BZzJY!i)|oAfgmRHt;{#U z4FoA!!~%gz7OWtTEP{a`_G6IqfQToHT5?_xzd*19;5d|(^OMc*`Jr7%hX8q@90XaQ zgL)wi`V4vHdSo$9PKP$+x?mgn0YKeQ$4?z(1M=q;+o~S)^_#z4waz%tw@$4lD(ba6 ze{Zt4Dm0i?-A}Vp)on7IFIUv% $mo3(7fTryzb`t^1ta(hsWU4m79dJB;g9{|X;a zJc3Uy8OmSx6I2lq@myAq6TEYYk~1FW%V#dmVs8{(=XNKy;DzCNtoKoCuA%2=#(LCi z?wR#`ZrjQM%=)iw@w1c7nGA=??An>Vn7t;w*}FBS@V$ad`9@c-u(#6huw0Y6TttMQ zs?o70?B=TJY>P_q%=;akIppuk<$U?TJ)K#?JvsM)uT}Ddc{_*YFON-NHoc#t>Tf)c z8**|ED_*qZ>>I4-E3XM=Ypxm29 TV^W#DOy?8l*h8(3aJDT1_*VKWIRn$hyk6%w z>}ubKOj&pd)A&?lw$A8auEW_;Ovd6IF3o1T%KUIwyruRrcI_tz-1Xip)^W5iQ#2zH zU(W5u85ZiYy_7q?KfAxz$J+mGVq2Vgk>^*Ds MJrdOnbGkH?gX_tgc zug=n@TegZPsZCM~i;v>i=uyZ`DiTuiCklZ=XXVX(orOjABb9{%FG_Pp-=p)oEZk!o z<1ZH8nJvC=v0gN+Jb-Ssxz3)t57pA3-ihMa>5oL$?M}+edl(@%(}FHYcu%$Cx9%}N z(OdeQanjqoldja}Ymrcxe_gEIY&%LR&68dyeki;@aGqFWf3To?B3PLm){j1SqfAV? z6Q=xn)k5fU?=)q*pj^29E?jDVXSo<1)?e~_8%Eb2)j)h#GmvV3qODkdq@Eb`?y17r zd#(6p%OHC2nC;@B=)qE)^9JftXo %^y6or#R+CU5@YpN=~h2LadwfpXmH6zc+_iD;m&!k>RtJTKKDis5T*sZ zP!3L{B!{@Av~Pb-cwRQ1;=6^brFusx!@Hu;Xui2}L46 gIt=(QA%KA$26~t@fZGVTihz3vxP^c#2)t_q?- 0o*RY)dJirz^wvYDd7Dcc&A4}fcJI~tSm^(5IBO0(1zeo2qXj% z+7mhwLI|A*6?(F1>xv+GA_-B1!Gs}%VT2Kc{~)kI{+}txk3)q*S{$~`4nbZz+p+o6 zAT+ d9p9aT#+(_6R%Yj8{`FWiuR|$%qWFx5+;Boe zN9xdn-pxm^To^P{YKxSsbx>KH8M;xa2%mN*!rSY8Sig2TUU~I33Y^*!cL`XAeWTvs zpbT?VZ-8bhOr 8 sb)CpZu2JM_#&lKQ-LikN#EDk)-KnT-(tqWd>1c3=& zTO{6G8V@nz5hi {rKyK_zG?E1vYJ;T!848Y>ANQz*#@yo<7Eb41S9!1({T85mn`yuPzL=W z`%q3JvXOeA45a_8)9{LgFfb5c5Fmizhrt122rw{!!H|K0L^dLj2Mi0OK|Ux4z`l%J z4$|ei$fl+~h@lR-E!c;;<+2dVeSxxIz@R?ZhIRqihQSGr!8t&F2rxjwHUQ~zT~G#& zK|N3g(toNT8;A;oFL)OIs33Q(A`i1T-_#u&_>6sBeJ4jHbCKP8^3yIg<5v{A_>NE% zGsnILazR%{@e{^2XP546%vatM#<$pB?prw`PIYbLe5O`RCnj (l!D^6E2HhpZT$Mbnt|cGr2A zanpE*=i~XyVJWJZj<@mAj!)3N_;|E^r!7-+4~0%on1!RK7BGwTNgL%)*&4mBurD7s z#K!bPbnoynys*kVJlw1`-sEM>CcGWYMz21{Ts<|7x4Rk1Sl& IAOmczAe zZNWZYK)&%iyaY|k{K_O1O~E-;F7amWtJ!zwUUL^M5A%WM+f=;`Z>!vztmB4G%;pVN z+v1oz1x!lAOulx#N$iHx9^4^xpPSgyhR^;qj}xa{RasAQ;|o^yl^=%R&7;gqZt;KW zQ5L7}+h(G0vul92Q_L! )(9u8NOw3bAt@)n}u^F@UV z75-EmYd7`uC;g?V_Z4)PR;6N#%bv)w^){+tRuy5%!qyUU@)J(1*($EMS5)}kNKog5 zkk=C522e#yd=$~82I7=WQA*csabga$P|EPgp+8odA}xpvqn3~9Ax%BL81+BYPz*Vd zCe3T%LD%{`fJ$BeR`jlpXy0z{r4`lF#ADAAy%#bK#ZMobimj5*iG!0%sCqVWg6*2u z6q9y|Qme;FJ+t3Y(KbPJtKrsSt>+!6)^r`o&@D)~wmgIC-R+>d*1ZAZBx zy4hT6(ReTQKG#{|iNF}Kap4^G?C5EV^gFF7#~wOriz> 7yerWz&9%TO)tUQQZ)#c0ge^}%qqaj1!2r5Dw zSWS@FUmFLK?I1#XLPx@XDRhxM5rjxW6k#x72w@mu1mSOyi(G_?MJ`l4_#tlA{v$dX zuY-GB^+MTp9_X~iS=`e9EcQND4|hyoh%GN2#Lg)hxMd4H^tyU8yn4byJTZ(znL1Y3 z*!mN8ZI^&PZJ3So+*Rnq`dC+Oj|MUn;rl4hR>N+3>Y%4og2;F!h}i{#a`0pA`Y zURxlrf%v2m#E57G^};k9%EL4q#4e2&8hC1H+k;QrYw9O$!?xyA_aK0Qcr`&aM CUtlTO2bTjU~tQ{P1)5P?Aa0l^7GA`pW>9D)G*5I~rM zJRlZ<$OOkB4a&kkhwdJWxNBf#dQv>_hn< z`vc{m4736H FRM-8h za#fGMW)1ot bl<=OLXs+OW zy7c3=ZkoaC7TfbWb=vao_8(OFDf(RL x@UZ#Wg@ry z-G07WLKYv;{*uZ$xQN+3b^!hy?7$55jbK-24aYGifjHf08}7V40EwMnGkvE pTk&aj;1T;NO?Ho}mW_tUor?+S? z)CxJGoSccIGN(*>#_W+&&h?$5MfpHM|CJ;zR+rEwWk%9Lh3y`@oTg&Z!!bgbT1E9= zWhw4R{o>Q@R-7=y^QH3a$d%%u+r@jZSDDmxwLenWZk8tZTrEB^z9GKI@fZ5p+!VIG z((^G6D^fO3H aN3Tg6Qdj}@Ie zr_nmzf;dt7Ah{X~%7d}xlwHY4(V%r7$!uq#V%2LmDsA9E<^20Ll5yH-di32gk-w8J zO>1^cyf*0 1ZM0T(r z)qAroRbNa~3@cO!do%l}XN{X9=54$t3>LA78uV4H?08%eQ1cDtx!TSr@APG56#nJ= zWBI4@+=}^w#y9zDfuLn7?^G;kK{sJ=hRFjA(DHx{19dfmDWL|zf&gw2wF%(ya3@d@z-<5#7?@O(nA658vfYN@PY5Jb zC}_!(agryD(38-c(3cQSh#*7~q6mWt|4G=A%l}nl`Sw>RV8x)BJ5$ljM*1k&vNbwf zyCq6*UKbs`(HG@s^~Kf)$D*C}+MqRe6j)Jd2yUd?9&PLM0lS=Ahdy^-k1lU?$0`>S z^oXm1x-3dS)i?LUm%6V-`IDdFXR}|Tnw2bZ6`$w0aaKHbFPn~ghp6!QIf-~wXl?XB z#}c=kR}VQh00$2pyskDlN$8*__VL&%ayDL}%0@SDx}vj++IaE&DR^e&KHR=I0qsAi zpfmbmT-EG7
u~RZIeT=6^VQFaZF}-2pHM0IWj*Fbx3A>j5wmz>o8P znD4`y008p=m}sd+VxYPfFe`wW7BDdYc>VxjZU9RyV0wUBTEGke@bCa&k^t}&0>C^0 z;57t*sRF=*2&k{cj>Le2mWCt-8fj@vV!%;L6A}Yp;s7Tt&Ljq0v@|0z;Hm{oAmFJ5 z%pvec2~;tpZMe^HZ{dEzeT4f4_Y3Y3+!weHaQ)$$!!?HM3fB^@pB(|N8C)y4K5$Lo zTEO|kIlB|!T;cp$5?T{{2ucD?zyub|E{S<<>`S)&2myq)gm#1ugkVA_p);W?p*!S< zK(>1jdJ*~%`VsmQ1`q}k1`(nOLkYtPBMGAkW51RD>;4ND55jK>QZX72DfoCjyy`~; zB#o#;ystp?(ReZu!Ac|g5K#w2CXGiH$*1v>BI1sw9J~?;o>c%5hBRp)RDl2mACo5{ zmFAU4t>^>-7yu%XMra~aW_XPdrpy}eC?YI@hm~Abe*I8W2WdmDOP(@o#3#}}%}4mj zxybv_M~w&tF{vN=4kDN)jr8qjnKD-_K!E`Oga4049E?7sK@0(m0_@8H3 g zmiHkI3=06p1?mCdIJ774!(ahyxTgXLD2@y7PQ3 zKfPtL7uayGqgJSveYRuY-L>GBc->Oft6NhQ>a<<8=k{!7@}2#7@qxZfJe|*##AK=F zHD1T9Y;%ph9eJ2>KHC{TTQ`_nFsY-;*s_4j_PoQ1y*sLu@kjW)^@XYh?^olShbu9) zdQU~Uu{Uv-6QOwQmM#pQ-<>^bFoxUp-k85VC0ONIbp;pgOg^lCID`o(RI`W7M&P?= zH{z%aU7Q>2iYu*+ +jo(pWMhr&o>-LdrPll53hK}QWeV{=pBPC zZ|z~-b~D_Xue$uv%!ZuntFe6fjBb2jojsi6vkPp>#9>TCV+%I1!94s+<~F?iR59~) z>jZXMgR?lJ%2xEb{u4CY5n+e9eOc34Yq^eTyIBi|r%d4WA=q)}5$ lJj^K#xCf+h~3b?lv|@B!e2}q?txnYKj-l?cFm&s zY cK@hI2l z $+<*!6%!3!C7Uwb~bLzC>@fS}qqqM8S(ejMaCeKQE#xcrsi&qL; zd%G)A+TWoZFOQL&m+TVgSto?kdLGKuq;9l$%TLnXR3x3+(hXJf8Guwy9cj0w
J zbfqJk??|(ne-*v)9ranJdq_ik V5f z3SV~zQ%S?x)5{ Kic>S&wO%Hb{F(d|BrlJrk)5=MHKN?jj#(}uN=Qq$Zv(WBBV>FRIJ(i>`T zr`H~Ntb976xftncrK~e+hR>84NNV(Qt{8amgjBPdITh<{Bv!rQCq4&S z3Zk0drD{HjSJcn36z#JvQ2ULXm9Z0hDZAG0L8o4##q|q1tJj-17rb-MDwc(pQx;yW zh5SQZq?9IE!h)3A>PL+pDA&!MEI&MdE>Eh7e}q~8-llxbKzcHhcPf^iptJIn0ET!N z!eRJ_;TqP8zl-!l+TjQ)LK}iVA&?M6Xiw-!2qAPLbRl#jR46rR*CbIS&tSq3!Z5-J z!YINR0t~(532}ej)yHq2nfzE(C`nC3{(7I#i5*5LrNJC*l(7#p-BQri>dR5%OjF$2 z@hiTe7{wUct;Hc7&!FcQx})Pb5?#Fe4rThCMThujs1w%5C+tgc@u#iW!Y~adHX4b| zR7I%3HW@vy9FL9?S?j_ybDZKb8Y?Uspbdq6(8_RcY;o}%>bVqSJpMG+38{w4;&X6^ z+5&d+%EMOtYaEufnqJ%y C;1Y|b&`IdRgOfNo zd;wBNZ$RtgFOs{hgR&Ekqmq (+a$6H|cOPuPu4$3BL< zKVy)eieHneCMJ+4>W7O0CPSXZ!%SXYQNW4;08;|M%pZ_f2C&)yz{CKs8UVoD047?% z^Z+1(0APjykVJqQTFgldSZJ{%G4Q`FRoRm^00%7%NDMU8(ul-BV=ay(2AXJTN@BoC zi!+G<7cI?547h4>BQfBv1!OG1QwzvgKnpD(_W-T5fb;`+YXKPuP-p>32%xlpJoL}J zNC?9R+<&;|aG&Ac!u^E%2=@){72F@VC$$Jxgnzq)Me5`Uz63u)00Exmc7zUuU_vON zGeIL=k+?$13JfPXK8O%a7)lsU7)cmS7)uyOh$T#bygy@riwog5rLBpw{G<62705$G z;`-*Nw5bgIus~jdeUPF+8j@utke)PB7LmEYp9%6n3^JJJYgc3+WG=`r`!<1ert!hj z%21Fl%Tb#8z( 7@_+&{ KfYxdU%T#b?lpH!HCY+O@0{9>&&a#TpJU&uthzks_xGyI z)u&C^3wynJhh2mC{nId8X}1S^Gc*q0-{j4@`!~hc&Xlos-b=W+HHK_*s1G}LsSSJJ z)p*Xr)S3?+)sp*sFPS|O6vZ8E7QyCsyQrG%d==}()Z`=gPvQ0bt$F>S7kTdGMW$8l z4g8t&2ULdbj&dpf=IkwFiQh3KlC$(3!nlq9j4$c!Vh?wCj#nBR^Y# $nwG%lUb)Em*6PnL{E&d-*T=AzEGKir>x|-; z4Eiox{a!+uXAAi;{GSSC356H->Iu{RU3{9}&JhdFtAv)f{MACdTJfT&j$}?_>gqEe zrNdfhu~PRYf`@O4a%28MVf~io%6`u!sm0~7w85G6^pg9%B;V+4O7D(Rv^_XPs?tAQ zSfA@Fwy(O9eAjA{IJUGh)yHw2^62iDV*4(w)ZQo6$^+zkQ>E+oNLi ^@(z!X~q|@#4sl$DTDyuxS61Obzpf)xS5$89y*<%*DUfjHIkov*VOX7lz z+ti4_V5J`wE^fWwf^MPq65rRqq ujaHr!b9MwMU1Nuf@5jYamS=m>eB=l*Y(uH!F?c_9W%V?w-o$9hb=Ylch08;F zia|TNp=FS$dSWF_zgI5Wj&3bZ?LC`n7O6)yJbhoi*gRcKDCLEY#&^U*kH^#ii)mtG zD?_SpN{Fz*w4oHfW&riXDpIVub&`6Uej4?_sFUFMMiARQ3|6nI>r2f%wxV$Q?5|2o z@0DV$Ic0^pwpS>dK8bXiiHEqtiWah~@C9S4+@KV@T2ZIcrzlR Cf$U% zGhO6|=+7mXCH!&5U(>RD)j%?rRwNY=4qU? zdkqpjkE2khJhX4)GF&--D_XR;2YU9YGg|+81!~#M1TBo}g{t(JhjJEl!fvC|(Wy#m zl%4SgPe>kvj+tFXi%M3L`dx6JI~(wb*JilU;SiKPZ9A^s_B{5vT7Y%SkK%@$K2BR_ zhX*G+W47rET&Kli?D1hG`lZYXjnyBH?ijdY>y4kjU7MfxU)gy4nuPZEFGYSceRkyW z0KfzS4qCt*0u8l*X#{?639UJ46ZpNQG#FN3g$OIgzl4-V&f}kw(MZ0(e<_k&1PH$= zsU?8a1TvP!J4=3jkw{P=If0a=k;sTF1|DDFK?Zpt)_9Y_Ylz@=MI M9nrxbVMSP%WLE}BGG+&CGy*^PikZ|QeN}RmZXoe zlqP$Bfm{Z%pC%vF4PI*6kJ*23leD4vwih|qzo+)!o{);AH83C$DiibxFdx+=fD|Pg z7Lc&uH~{ljIfiXGCQDSZd y|}rK(|ypKH9Q&5PkrOePtOrlrw3f-SN3SY@7!Kjwbx?> z`=~`K(?9DHleO>`6Kv+gx9(WV>#X9& zl*Y0}#Vjis8#C>rM{rFGQ~02VmHA2WZMeYM`l@*kD4upX!7k|7f?MEjz`KmR$@!an z;GR%N*=H-}a0BR6-agBQO)RRVG95BOWqm!8&u*N<+9y9_oiBZ0Zp_<@=R4%$?WyZ= zQm=)$^?^_5RM}NrbwD&WPMXSadq?1uu!h*4E)jQPt4jL*TPT)pb`c|XMzW=H#X zypKsdH}AF`S1s9?9oOKMs_xVpDkFX%uOC~&^*c11O?|4yZS$PjwdQtQr_NvaDwlV2 z1_RUBC5mwN)6E&&h0PsUo48Y~RsX8&>Os@lL~jGGv4b)D_RavlvT-SEH?JPE_ogk^ z$iJFO-|8v9Y5yC3+S_Y9?drmt*KaI89KV~`_V4?6ec{HE6N<{iuwZ}pqk8O!-pUD< zUxf5C>y<-mHV~FrpZAIBx0Z5Q8(O$`Qkc+Tk++z#I#ck+F;x3kuj#CfSA=c0>5@h7 z_X?X^OQ@{dIm#vPCkv-d##72G2ZhL5t?A@9FX*#}rid9~+k_`>QFMRT!PJpc_mm0V z%cNEvk0G;DO~npDDL#oO+Ddr^q13h1w~9+`y9-^nVe0MYuF97yzEG8xFO}#@!F1ze z>B5Im$0dICC~9K$+QRi59mU=unNn&-6Z+x)`P8w(6{vCpJ83~>9np6cB}Glo7Mk{2 zMBl&ifcm)ekoxI67dmM8Kx)8};o{Um{iX0RXOdWgHt%jn>_&qcixF9fFhEFqw1 znX=B6Ws?48LuycSXUUqSsVfdMl$piFK8t6Eiq2K<7oPEZrt}I3pt^Q(6Q?z^R&F1> zg|c0$P$+!gD4sJNrO+8q=vfDbD;_`67m{jCF5GsZrEv1_S!w+2+M@9y2jwpAvM{`D zUv-t`n-$4UdBWl)tLfsj7u4#KNYOoXG`&h^h}fK|BRLp9SL}9m5%t5Ti(5{8q&m&X zqwJiE1wU*i&O5$_9(Cz} Q27>waXNxRat{6T?g!2vJ_34eh@di^&0KkI1h(8l+btXS)tjRSL23OO0;{46GmOe zp`azZaBRc6cx6XD-0Iw9?4EZP&9C(syRT`37Ce24P7hv#ZcSDqJ6#ua@ZLxCi^~_h z)OsTde_RjkkaEyIyAHT%oxSKyxQL6d=HcYr=4e V}dAjKkHX+n9Sk1@it(L4F#3 zO*o57Ab%S_TnsP~0IU%JSPuXooB&`t09aedYYlk~^5Z%L))fG(9RL$8U}^yHUjjhS z2f&vJ0Bs)tza{{5e!x--X#N2BI{~$|fbSCk{}%v&9~4kmiw%hZ_&)*wUhxC!Yq29S zV6Vl2#6SZrzn5_4N!kOvwEW&e84MF}-~ZG?8JK%=4*dxOwA%y8_8>wup+dhi$on$| z*)aU3Fc$YstkQn73tl%Q-`0|S!$9x?;MWp0!kN}j3j`<-x-?=F5u(2Pc7g8~*_Zvt zWIr|dt`-PzkPfeHLOKXw8ox4<2I3b8X|kZD$qV^Gv?HRLMr70anQ6o@Z9g 3_@QHx_$9*+vljPbndci0sIs peBp*6BF48!-JeOu^$_2f0uVNEMbN> zFJ#TbTe6=%Msd2;vbo6F=h;VT)v?k1R(SrwFjN#1z^KkF;_5}s wh^$m7dz3&5CHmuRrI^RZa-wuz4hFl4{NcysOMLYPFpA za<9xb=-|L4wA+E#70zJ|{I>J{@q2jh4)58fZaMgn(GE7yY6kDxCyVdbSmI;#?(oyS zQ`wI#yP$mcP=@k|XH0aKqTQJ+Yni@@Sxr7h-@Nt^&hK7lw%aEKqqAxi`(Z;P-po7` zXMF0z*zTHv3Xb(>O%`5b+xVBVOF~AnA8y>o>n`TAjSe5;Xq$CJ4C}zQ6XWEE;dc|c z{%zm&N+_IuJ4u+)VWYQ~Yq~hOcd($>) *5^ zdxgg#A%$mNURE4%%n~!_-lP&g-j?1+RTXouoRJn&Zd7io%LTdyeMQrE3k7HUN|cHF zT5&=29mVR_m6UTzOoU4Liv;_G5~by^`I7Oh!F0`5o$2F!mI^ruH$>5J@19yOo=C?h zIxE*&XYZ-=fTQE;-4xlk#dO=0XraaWOzOzBlj6?a2b57;Z3^>R9hV$yhe&me{VC7M z`y|WR)kX7L2dUm44vFngT^EmcjaTleHd3g5-k8qI%Ty*k+%C?r>ZM4U@PMk+CrsSv zpjUY7{d?N$=v=W~j0L^0&_vl?Xh!S2l7x+=;Sw{k4qaJqn3QO*Cms%dC54U{K;`N` zrflp2q{` L#U*N` z@Rm}FM+)i-YmM}%$nMV-hIj~-Y_?Y@$vmVS%Iu&Gj?55!9gZmsJ^CpRWQ ~xCHzq) zU)!#H)j;^NkasE;zM#7>L4b(^4C3 7!PopP``xZld#f}oI@H*$4CY3`wD*WIo3*|?ABa_>O2^sFZ} zt@CQSUn5(zrf4lT?lB#i?|qG(jc231Uc<1v>KvM1Z!>mVeH}Haeh&$a>!Pn7199jz zOB^@f9qXn 75>dRY<^yibO=cf+x)A4J<*xw#y ze|wbaR38}?IdF9E@PSd${YOR&4Qm+PKz?RmSozZnW#k _}in5 zTm%TeDV!z9AEnoPPmBmpninA9*Axlhk)=tOJ-uL`2xcI70U)yd=s5<06r_VtCW}-W zu}o8s_N7nq5~s#%41_T7A|qlQpm}i;>LkLO#zPFsXuZO|_lfmjFWkVp1pgL-6f zP2*9f73M&Y(+Gj^5+?{}&>rcV+{WKi`)^N3#X=ew5D>4xh`|4($U+kcRRGLa<#fm+ z&tD-1U``8jS(wwxKsuz!b-}irpKN|_4~|0_s2_j;`5?%1WjU7n0A&D32O!Ax$ZbNJ zETF+TKt7mf1JEAigCLjxw)YeMD~}=-2y5_c{E @hCfx!LRn- lq%Zbt*_tYP_BO247SPE_U!Dg=lKb?$5fS+kN9&7tyMe9+pDe_bZ7UJ&tnsQ zF=E=z%VnO{J&d5ZEaVj_{4lCvlr=E#? i?dw$q!oKu)2~-p$B#|%{bB2vX4iA^gdUd}osyb( z?)W^GdXj(&KlDL2>=xm%`DJ)l@ijcZ%N11qQpu$0yu+h6A4R)@n&QegC!*w$W+>%g zAydzKJd-o-6cgp0!lXwBvv1BQxMvIQvpuKoU~J<*vPlbDvWN1vb5%1_*rtnjGX8$o z_?g{DvL1y`na!P&nS~=G81>*MoY$L;yrXG%j%_fCRgB)xU0!sSjaa{mD|TAQK3Lp| zTQOu3SIsm4AJ+3^N3GQ5r!HN>Et{-VO |f15|yn8FCJvp%=1W~tvVwoofB=qvpVKdPe( zPKYKCKML~#4;8p2J1N$?I8cs#8wwfm_mykwnA3U9JBcn9tE6-*M!+ZXseHdrh2iHG ziEj4imHw6D3NupXQRf spwjavk(-@bjL zPCNb!UH^<7UFH4`Av$uMxJi{QEa@{^TJ}T|;@i@~&eiv*PUkw)Ij2>U|IQ`Ug#oXm zROj)O HWJ|7CM=_1s=H#e; zb?znAPw%78p6n08pvOa$#pdzilVd^R%%D(ZS @v2 z?i;rYPYVJSJ9`<>gR`QAV@)P2H>WG8tXGtBtIY%IO71wR>gV@LJ@QRA^UoiIHT$~= zGnR--N6(3L`}_IgjVm35vM(2vOF!r-u55cE)y{rFIZfT5_;`AwIOj@N8ZF)_R`O^= zUH!CFylz&NvN`5R&?_EoNA-h`A%jSk-u &bKIXZB>^+Ij+4I&Fo=btEr7 z9?ZsWrYdxMizzAyJ&OFSU*P eG k_O^qyGn57Sw+}y(pB8`f+3#WIUDl_@8eFifO7X;!q2vD z#a?wfptIrd^y(nXv5nAKbrGIYeIU9w$pg~<%0a&Czb18!Ng&Vr4<`?Q@&K$l
xCnb4KcozR2Oi_nMAkI &2t!*MzcP@uNFRTeiEzb| z6a2x0AqT)98);)O_CJP7WCxBx41jTm0EQpRfpM4fKv_5r$lFjB3<~6dARhzc0%;IK zUg!q|s1Ja2xnHmk?SXNE@MGIhKkP%0%Rw4!L-?^Dkp7nr@*jK!Qqmv0(5Og28sj^2 zM qABTbj(wS+JZjzJ8crkrBuqyEclA92bk5>^#KhSRGgGEvd?N zTlSRmc{_}sadH8FDDesoYu<}{c%(XeYCs|vY}A7b-+7#CHDwxKyZ#iu=&BVPGUYYL zOo`$0jui2WJ47?ORbo)NK`~>oA(U-hcAiUIu$xOFTEXO*Nqwt~NL~M1^o?p=V z6uWrWK4!s~O>9=V5wpu>p0f`<570Un| JGJVfDm)mrlOv!LkA9@5* lLlJV6zBJ~r6os0$*IdGD%MS3kv+kZRxR2k)-hPB?i-XRj(A^6n|9qU z^||o>+B+AxnzFu+OU0-ZDHVxGZlMgN&e?mdeK|+DBoYz2sU#_ ePwvL0hFmi)<(4}S#x3bx|1)PBGw+z0=Xu_b=N)E!KHt{f>%P|B{r6hG&)WM$ zP#q&5s5aOAfvR6(D10^2NopuFR*gD(nwm)sQ_ep#MkuYmC*13=q%vAA7RrsnXalG5 zLjU5g=&aE#CBpYx>76%E2+y9ZrPjSDp&W|((;g9Rlt`~k9ldwCFmAh2oy1L1_uM~- zT2wGy$l7z4%Dr4mx2Bp3rB5~qDQ8bA+s#`;RdeP-A?B3ZswYdGH*sq7uC>&{X)BfO z7VZ)zZ5${$8t_bcbx^tBWVnQCVp&BMWFqD1?@TD;%w586yMxl(P4}v)P5lL<4Jt|T z pY#o8QU;^*CRRAQ z*F97@vlVXAbrf=qZHAVQxs84-G{vSmk8z25m!^?lz#sAI|CVI*SNU~CFgXCo7XX+a z;6Hs`5KI=(S__yj0HhEAOc?;O2+&STd*TCN-hhr;z|;W_TEOf9Afo_a0s)Xz05FFD z$SVMtMgXK10L&!dtOZOa&{GSTPvGB`j^asPSjoZa3|4xu%7WFK5y6 (7*<1ijD2GD2dC-e>a1O0%uzn6}*=`jgo2|Phga3}O8z`g59@FEN(3?d9B_!9n< zbVSmPBSaGvgjm97|Er3k>4(7s_q!5OJjgB@iG|2LAjg1=1CowLTGH$;^0C%PJVXN0 z>|7$dk~HaIrx3_Ou=9!RL;`6?vm*+o%tTrezk8*TcZhTZa*igyR!Y*ugPcRA%$m9& zjdoX(raq9VK=J{ZibzG8G+NmRUQ+}KOCv9lFijk^1JXhna7@BAJF7^!;A (#DQUfc_gHVW3g<= z1Lpw5frI>T4rPE3^@(|*EJFgM5sx7r9E3xd*e*DRGN4>IhBBZ$@S)9O969{_ooE0) zA}NX2{! GEO%92P4LMo(H2kk;}$DD&bt6)cBI+P&7Ah4Vyf766bDk zo-ZA@RpvEq0=Ir%DQ7cRDGL%(WednIp}KAv{7zhh{pSVYV}mQ18M7q#mem$!K;!)^ z-ZY$j9QvG{{&o&t;CTz1EU01~< P zAIR>18p~GP64;($ db5__hOU@RXzO z>@4ia-CR71U7D@rmuL3n_pjZ=*_?@Cv({f>Jxm+0Ze>@Qjh7g9S5#-VizSQgp1$T5 zyJhp8laBM-_YB|{Z{N)-mOteq5@yO;^`F6~z6xW5$_Dd=$KtuL1C6+%7@Du^{F1wF zznkmP_c*ukySw7W|5Uq<{_lQWF|o+?+&tB?%z=CB8cBuDO{P-v#gkO+&$M>yM42g9 z4^0xDhWXO%9O?@TnuQ3KFA9}gJGYcHy(3i?uR(OvtmlI9L{w}RI!MS~7B0Q>I+cFt zlCHLhHAK<@-zw+%ey?u*Vi)E7xU;IslyyaiPQ^;DOJ1syqZHB=X-Cwj8#~eqTg?^* zKmD5SVBVfu#CjEXx7bA)WW7;$xObcO=C7!850xO-!7f6Co3%RW@J*`qtpa-PIS=Ka z+FR;vTOLT6z&GmB{=)>HsswdWo5s}IJReE)xSc}%$mJ5(zWY>OFJhIZISZx91sQ_X z$R$$UF{_mA94=BxgVTkEb8o8KTUn_%<5Ifb@i^&~m>*R~rr4_;uoZO>uMn=E+OMA3 zd?&55Z>Z|Q5LWHHznLoYNfrII<1p1f9`}%*f0Clc5pSuZ@BT@vSc(o_6et{Ax>q=! z|CZX>3FwtPyORY*Bm1R?zK-HCELWq)C&WY*qK@=PYp5oOIyP6T-ewPF0gwsdnFf zM%A$F2IXCDT6FSpKWXWVOv!|rN>wf1p{!iPOWN1HS@d}1MnUhvGs!CF*~+|=>C)c% z^_4vupHd}6-jd3{_ExPZ&y(s64-@R1S4yUZ^p}P{DW}RK`n|tB|574T;-@y@@9PuC z3uLCI;>pKlCTJ);hQ&uYJeFZv0FUv81QSAILK6b4_bdojgpbNhB(4Ji)?Bd8g0&T_ zr(g{Q>n0ZhtdBGj6Y+l`FKO2vy-2tRp%0-Sp+8{&!JFVi7$VN6$?;Iaza=%1eDPvA zpSv&K1|OB15@~LY9QoU>rj7bVp)3O*^wxVd-64Jt?U+9R mGo6m3rFhE@gjz~1)*vHbp4w087i $h z fL9fmT9k$5t5@U5 zZHd? _{G n3+xE`Grx?O zNYcUehHDJh6|N;*Ke%pio#6Vw^?>UD;|^mDV+> W`AI7Wfz0&N3wBzmikw3ljT}Vk(8y3ANrAj2?vNsrYmk;;rxO5@ z5~SD2QBWuOd_1HTC)Z*e)GzLi(nwuI!h$x6yRI}}*MfGDwn92hc_4FrVUz2R%TZtu zz=06}M8jTJd;Z~r0*nO!4+3~tfP?f92Ir6#9vW~CK9m7zMMDGUVtJ4c&LIwfV*ngH zNFW~oh6?hCbwE90I&%1nI7km^Aq^ahY2g_924w(nF8bu~^LL^__=qee-UFY0AjF1! zOqfcPr)Bx@2GTTnGC!Ux+1rO}I5msU9{o=Kjo$$-bx1Z>@8Bpl(_tl>T9(1l_U8Pw zbrQMz)VH!eQI_o2zEUo%yfxFFYRbNv+LrZrQ_ke84zfw^-{HLFK3vndMy$;tGj?uu z7C(NYuIz{8MzWm=RWjpzUp~fAhcTG8f^D?>9Osk2fteqC39XJk!kBe=#|DpxVrQR< zWb0>|Ff-3(F+ryG_)+OW)^6NJW{>O?o_1^pTGo3y7G76yjb<@ScdJn5#y4?z*>aYh z!IbefQG2*tMV4%3`h5P`t!kOuR12A-VmH6EaT-6PmmS|)Kb^}lxQ&PGiNzc3YZ lYWn_{(CMRg;79 zUEdvy*}Kz>(RLTSNw|y(eX8(*L=U!cxe;EJe+Mrl-?Uyh;VjPRB1gqtMziMrUhMf{ z<+$)t9Ny(SgxS0z4qr>q#RFDNLnpfFGtW+2vqsCB^2U{^Y{K;$T<^y_IGrC8_)yyj z*|cM$_?wo|?Awd^eC{J>zSQJ3ze8Qb=iBe(XWZgAry$G?9h}U2^~_~c+VtZaRoZb& zy`q_T&&%RX@Tml nGW4c3o(BBdDk Z&6$w+h34Fs1Vwoe>)9bd;W+vrCm&Z-!+228Gmm&wX`^qO)r2!j;JWWGb~P ze7<@@Qkd%4@T1a-Zv&`(wVhPe=Spd%MXp2=k*`X+&{|>>RU{Pd>nUB{dAAUg(vY^I zU8PGF8LMtJeI*FTt*N{l_o#v7)jGSZXm!&1MM9U9W`g&DXVkEi7u1|od)3#~2~_4h zbJgB$ro#0_Wh#Hq7eb$7snTy;rRwJ8>4ICc8uhG+70Lx$i)q9BL&A)e_f>WGrU~kJ ziFDSReXbL;a+IY__DBvbTPWF7U{F+Aubpb%?RC;tltNXu;H;2eryDiP&03f{q5(D8 zem@;~*@K#QXp%JBSuQ*&TqwPC-$G&>QzGbQ&yjxLZ?W?6i(>I+`D^`p;eX$zgFs~= z4&MJ$feKav f*_K-< zF8M8;TvP|^IWNFE0h3S<7lh1Dd*e `(PVyX%l*wuwoNtc;g95P0-AQQsp#W6y;0pfg dI#Dr*@fK4o1lx^XX2C= z!*N{H9kjIlx8zFHK}$9)#5#90$MoB>JX90tfiGpGV(Ydo=tD{0K^ly51L$KGr_kdy z2hg*nW3b}&5gfs}ptGaO(AiK2Y;mJIHr(QcQl^r2UB$>{Xaef7 uL_5d6MCrFmrz_B;{s1_UJ$jnD(a6bL&+L;_I<#3KmT>_sABAW&)cDv@*`25H1D zcvX+=W6}VxxRE#zg*1Dqw6Eo9Ue6<6nS#l+MuZ~56HKn*bvP&=ehW|(r(kjo!j#BQ zb^r8*C`}tlJ49g(_F`$i0wwPAB5_bZlm~mLG=iG8J&^88n_T~~KqU%GAl`_G0uP8e zqR0f|5rl(q1YsaFLD;VaCJ=-G5R4!W%7lE7R?GuFI4BF^z Os4`=Xmj6As)cfMlG z7KgI>4HTT4$u{o3Y6usS)tWtX!Bb`!RwHkG^#?xPZ30^nlgFDM9L!g2-irqw%Vw^G zU&TL0&Bk^Y>hQfhv$^?;>amL0Dcr^E1Z=Cj317^6!Cbqz3FqCZViyFqXDih`IG6rm zT&CGtuIb%}> M%2 z+1jLhwpFKBT!M`U>#NLX_fODerukX16LsBjvzDFN`L^enlc%2J&3c` D)us`Mp+V8Wa^36YC$Uh>G+NoDiZ24Es5Q&ihmi#>GYY2agSmiFw}~9Y?WYh7cJV z5*QozL1lHFx{2`y5FZ<;2n&h*Ik(gMel&;*8WR#68$KcAed~YOag(6%ae<0Nx3N)S z;W4q{!D6|_ijXk>u~9KGzv_FVnBc&%A^y=(G5+x(@V(LxYn}YQ-uk4sG2v0;{Mv{G N>W0@1*9q5~{6Chflf(c3 literal 0 HcmV?d00001 From 17b5da82a9fc6542ec670fade8d046500927c781 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 11 Mar 2025 12:16:44 -0700 Subject: [PATCH 03/90] 111 draft --- .../model_validation/111-draft.ipynb | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 notebooks/tutorials/model_validation/111-draft.ipynb diff --git a/notebooks/tutorials/model_validation/111-draft.ipynb b/notebooks/tutorials/model_validation/111-draft.ipynb new file mode 100644 index 000000000..f4372fd53 --- /dev/null +++ b/notebooks/tutorials/model_validation/111-draft.ipynb @@ -0,0 +1,20 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# ValidMind for model validation — 111 Import the champion model\n", + "\n", + "Learn how to use ValidMind for your end-to-end model validation process based on common scenarios with our series of four introductory notebooks. In this first notebook, you'll set up the ValidMind Library and import a champion model for review." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} From bc4e985a3618fb46128b5f7f17b1a66d3acaff03 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 11 Mar 2025 12:20:31 -0700 Subject: [PATCH 04/90] 111 notes --- .../tutorials/model_validation/111-draft.ipynb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/notebooks/tutorials/model_validation/111-draft.ipynb b/notebooks/tutorials/model_validation/111-draft.ipynb index f4372fd53..178d0bb4f 100644 --- a/notebooks/tutorials/model_validation/111-draft.ipynb +++ b/notebooks/tutorials/model_validation/111-draft.ipynb @@ -8,6 +8,24 @@ "\n", "Learn how to use ValidMind for your end-to-end model validation process based on common scenarios with our series of four introductory notebooks. In this first notebook, you'll set up the ValidMind Library and import a champion model for review." ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- About ValidMind block\n", + "- Prereq block\n", + " - Access to champion model export\n", + " - Registered model in platform, assigned as validator\n", + "- Install & initialize block\n", + " - move LLM context + add explanatory section\n", + " - Adjust initialize with validator instructions > need to remove yourself as a developer!\n", + "- Load the champion model\n", + "- Load sample model development dataset\n", + "- Upgrade ValidMind\n", + "- In summary\n", + "- Next steps\n" + ] } ], "metadata": { From 29a6a075dffb1d9ed5f55b8cbc561009a7dbdc68 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 11 Mar 2025 12:25:21 -0700 Subject: [PATCH 05/90] 111 adjusting --- notebooks/tutorials/model_validation/111-draft.ipynb | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/notebooks/tutorials/model_validation/111-draft.ipynb b/notebooks/tutorials/model_validation/111-draft.ipynb index 178d0bb4f..ae2e3b960 100644 --- a/notebooks/tutorials/model_validation/111-draft.ipynb +++ b/notebooks/tutorials/model_validation/111-draft.ipynb @@ -14,14 +14,14 @@ "metadata": {}, "source": [ "- About ValidMind block\n", - "- Prereq block\n", - " - Access to champion model export\n", - " - Registered model in platform, assigned as validator\n", "- Install & initialize block\n", " - move LLM context + add explanatory section\n", " - Adjust initialize with validator instructions > need to remove yourself as a developer!\n", - "- Load the champion model\n", - "- Load sample model development dataset\n", + "- Getting to know ValidMind\n", + " - preview_template()\n", + "- Importing the champion model\n", + " - Load the champion model\n", + " - Load sample model development dataset\n", "- Upgrade ValidMind\n", "- In summary\n", "- Next steps\n" From 2ab9c2683776932d3724aae98174e953144e79ea Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 11 Mar 2025 13:10:42 -0700 Subject: [PATCH 06/90] 111 +intro --- .../tutorials/model_validation/111-draft.ipynb | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/notebooks/tutorials/model_validation/111-draft.ipynb b/notebooks/tutorials/model_validation/111-draft.ipynb index ae2e3b960..9c0361f57 100644 --- a/notebooks/tutorials/model_validation/111-draft.ipynb +++ b/notebooks/tutorials/model_validation/111-draft.ipynb @@ -6,7 +6,23 @@ "source": [ "# ValidMind for model validation — 111 Import the champion model\n", "\n", - "Learn how to use ValidMind for your end-to-end model validation process based on common scenarios with our series of four introductory notebooks. In this first notebook, you'll set up the ValidMind Library and import a champion model for review." + "Learn how to use ValidMind for your end-to-end model validation process based on common scenarios with our series of four introductory notebooks. In this first notebook, you'll set up the ValidMind Library and import a champion model for review.\n", + "\n", + "These notebooks use an application scorecard model as an example, but the same principles shown here apply to other model types." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "An *application scorecard model* is a type of statistical model used in credit scoring to evaluate the creditworthiness of potential borrowers by generating a score based on various characteristics of an applicant such as credit history, income, employment status, and other relevant financial data.\n", + "\n", + " - This score assists lenders in making informed decisions about whether to approve or reject loan applications, as well as in determining the terms of the loan, including interest rates and credit limits.\n", + " - Effective validation of application scorecard models ensures that lenders can manage risk efficiently while maintaining a fast and transparent loan application process for applicants.\n", + "\n", + " Model validation aims to independently assess *champion models* created by model developers by conducting thorough testing and analysis, potentially including the use of challenger models to benchmark performance." ] }, { From c7fcb9e30bebe69b742f869ff59ad5f92ff96e73 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 11 Mar 2025 14:04:07 -0700 Subject: [PATCH 07/90] 111 + intro --- notebooks/tutorials/model_validation/111-draft.ipynb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/notebooks/tutorials/model_validation/111-draft.ipynb b/notebooks/tutorials/model_validation/111-draft.ipynb index 9c0361f57..15d449298 100644 --- a/notebooks/tutorials/model_validation/111-draft.ipynb +++ b/notebooks/tutorials/model_validation/111-draft.ipynb @@ -17,18 +17,21 @@ "source": [ "## Introduction\n", "\n", + "Model validation aims to independently assess the compliance of *champion models* created by model developers with regulatory guidance by conducting thorough testing and analysis, potentially including the use of challenger models to benchmark performance. Assessments, presented in the form of a validation report, typically include *model findings* and recommendations to address those issues.\n", + "\n", "An *application scorecard model* is a type of statistical model used in credit scoring to evaluate the creditworthiness of potential borrowers by generating a score based on various characteristics of an applicant such as credit history, income, employment status, and other relevant financial data.\n", "\n", " - This score assists lenders in making informed decisions about whether to approve or reject loan applications, as well as in determining the terms of the loan, including interest rates and credit limits.\n", " - Effective validation of application scorecard models ensures that lenders can manage risk efficiently while maintaining a fast and transparent loan application process for applicants.\n", - "\n", - " Model validation aims to independently assess *champion models* created by model developers by conducting thorough testing and analysis, potentially including the use of challenger models to benchmark performance." + "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "# WIP BLOCK\n", + "\n", "- About ValidMind block\n", "- Install & initialize block\n", " - move LLM context + add explanatory section\n", From d942bdf66f9b61968d93da3f57dc0548e73e96c3 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Tue, 11 Mar 2025 16:42:03 -0700 Subject: [PATCH 08/90] 102 accidentally deleted a header somehow --- .../102-start_development_process.ipynb | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/notebooks/tutorials/model_development/102-start_development_process.ipynb b/notebooks/tutorials/model_development/102-start_development_process.ipynb index 4cdfa0247..b64c31426 100644 --- a/notebooks/tutorials/model_development/102-start_development_process.ipynb +++ b/notebooks/tutorials/model_development/102-start_development_process.ipynb @@ -70,6 +70,15 @@ "Refer to the first notebook in this series: 101 Set up ValidMind\n" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "## Setting up" + ] + }, { "cell_type": "markdown", "metadata": {}, From 511520266a3801ae52c497b75f20ecc45c03e35b Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 12 Mar 2025 09:49:49 -0700 Subject: [PATCH 09/90] 111 new draft --- .../111-set_up_validmind.ipynb | 256 ++++++++++++++++++ 1 file changed, 256 insertions(+) create mode 100644 notebooks/tutorials/model_validation/111-set_up_validmind.ipynb diff --git a/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb b/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb new file mode 100644 index 000000000..f3bdf68c9 --- /dev/null +++ b/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb @@ -0,0 +1,256 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "34e60911", + "metadata": {}, + "source": [ + "# ValidMind for model validation — 111 Set up the ValidMind Library\n", + "\n", + "Learn how to use ValidMind for your end-to-end model validation process based on common scenarios with our series of four introductory notebooks. In this first notebook, set up the ValidMind Library in preparation for validating a champion model.\n", + "\n", + "These notebooks use an application scorecard model as an example, but the same principles shown here apply to other model types." + ] + }, + { + "cell_type": "markdown", + "id": "0426ef4f", + "metadata": {}, + "source": [ + "## Introduction\n", + "\n", + "Model validation aims to independently assess the compliance of *champion models* created by model developers with regulatory guidance by conducting thorough testing and analysis, potentially including the use of challenger models to benchmark performance. Assessments, presented in the form of a validation report, typically include *model findings* and recommendations to address those issues.\n", + "\n", + "An *application scorecard model* is a type of statistical model used in credit scoring to evaluate the creditworthiness of potential borrowers by generating a score based on various characteristics of an applicant such as credit history, income, employment status, and other relevant financial data.\n", + "\n", + " - This score assists lenders in making informed decisions about whether to approve or reject loan applications, as well as in determining the terms of the loan, including interest rates and credit limits.\n", + " - Effective validation of application scorecard models ensures that lenders can manage risk efficiently while maintaining a fast and transparent loan application process for applicants." + ] + }, + { + "cell_type": "markdown", + "id": "about-intro-e542b60e-207d-49de-a380-4a6f99f1c392", + "metadata": {}, + "source": [ + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "about-begin-5260219d-b4ce-4a0b-924d-7333fe5d7be3", + "metadata": {}, + "source": [ + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "about-signup-9ffa73ae-a0c8-4c3d-b307-08dbb5c54194", + "metadata": {}, + "source": [ + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "\n", + " For access to all features available in this notebook, create a free ValidMind account.\n", + "" + ] + }, + { + "cell_type": "markdown", + "id": "about-concepts-fa466c67-c594-4e29-aaba-72e793f905b6", + "metadata": {}, + "source": [ + "### Key concepts\n", + "\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "\n", + "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "\n", + "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", + "\n", + "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/run_tests_that_require_multiple_datasets.html))\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures.\n", + "\n", + "**Test suites**: Collections of tests designed to run together to automate and generate model documentation end-to-end for specific use-cases.\n", + "\n", + "Example: the [`classifier_full_suite`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html#ClassifierFullSuite) test suite runs tests from the [`tabular_dataset`](https://docs.validmind.ai/validmind/validmind/test_suites/tabular_datasets.html) and [`classifier`](https://docs.validmind.ai/validmind/validmind/test_suites/classifier.html) test suites to fully document the data and model sections for binary classification model use-cases." + ] + }, + { + "cell_type": "markdown", + "id": "install-library-2450d0dd-fd90-407c-8cd7-53ef0973ffcf", + "metadata": {}, + "source": [ + "## Install the ValidMind Library\n", + "\n", + "
\n", + "Signing up is FREE — Register with ValidMindRecommended Python versions\n", + "\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-python-7a733859-9daf-4528-8128-c5d23a0fd6ef", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "install-initialize-0221421a-9308-4d3d-93d3-0514d5fbe196", + "metadata": {}, + "source": [ + "## Initialize the ValidMind Library\n", + "\n", + "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook." + ] + }, + { + "cell_type": "markdown", + "id": "install-snippet-7b7726eb-26bd-43bd-b1ef-504895bf9bd0", + "metadata": {}, + "source": [ + "### Get your code snippet\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + " For example, to register a model for use with this notebook, select:" + ] + }, + { + "cell_type": "markdown", + "id": "install-variables-b606b626-0791-458d-811a-1b3228ce85a2", + "metadata": {}, + "source": [ + " - Documentation template: `Credit Risk Scorecard`\n", + " - Use case: `Credit Risk — CECL`" + ] + }, + { + "cell_type": "markdown", + "id": "install-credentials-90676cb9-9fd8-4484-8804-bff9cc2796f6", + "metadata": {}, + "source": [ + " You can fill in other options according to your preference.\n", + " \n", + "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "\n", + "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-init-0c822102-b258-4404-b21e-26b8540f611a", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "upgrade-vm-4b09d40c-edad-4f40-8884-c0f0eb014875", + "metadata": {}, + "source": [ + "## Upgrade ValidMind\n", + "\n", + "
\n", + "Python 3.8 <= x <= 3.11After installing ValidMind, you’ll want to periodically make sure you are on the latest version to access any new features and other enhancements.\n", + "\n", + "Retrieve the information for the currently installed version of ValidMind:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "upgrade-show-752d8d8d-55f7-4465-aed0-34c5a3be8663", + "metadata": { + "vscode": { + "languageId": "plaintext" + } + }, + "outputs": [], + "source": [ + "%pip show validmind" + ] + }, + { + "cell_type": "markdown", + "id": "upgrade-version-d64591ca-3073-4b3e-9586-d3577adda203", + "metadata": {}, + "source": [ + "If the version returned is lower than the version indicated in our [production open-source code](https://github.com/validmind/validmind-library/blob/prod/validmind/__version__.py), restart your notebook and run:\n", + "\n", + "```bash\n", + "%pip install --upgrade validmind\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "upgrade-restart-3825c878-7301-47b2-92b3-2bab240f343b", + "metadata": {}, + "source": [ + "You may need to restart your kernel after running the upgrade package for changes to be applied." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "name": "python", + "version": "3.10" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} From fbab0ee2b37caf9167397529e89847d95d587824 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 12 Mar 2025 09:56:13 -0700 Subject: [PATCH 10/90] 111 editing intro to be validator-focused --- notebooks/templates/about-validmind.ipynb | 2 +- .../tutorials/model_validation/111-set_up_validmind.ipynb | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/notebooks/templates/about-validmind.ipynb b/notebooks/templates/about-validmind.ipynb index 1c135f268..e982e60d0 100644 --- a/notebooks/templates/about-validmind.ipynb +++ b/notebooks/templates/about-validmind.ipynb @@ -31,7 +31,7 @@ "source": [ "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb b/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb index f3bdf68c9..afd60e0bc 100644 --- a/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb +++ b/notebooks/tutorials/model_validation/111-set_up_validmind.ipynb @@ -34,9 +34,9 @@ "source": [ "## About ValidMind\n", "\n", - "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n", "\n", - "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + "You use the ValidMind Library to automate comparison and other validation tests, and then use the ValidMind Platform to submit compliance assessments of champion models via comprehensive validation reports. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model developers." ] }, { @@ -58,7 +58,7 @@ "source": [ "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", From a9e1ddf34da8971f0c9a983d7b4c328c1721dc22 Mon Sep 17 00:00:00 2001 From: Beck <164545837+validbeck@users.noreply.github.com> Date: Wed, 12 Mar 2025 09:58:09 -0700 Subject: [PATCH 11/90] Edited the About ValidMind section to be more role agnostic --- .../capital_markets/quickstart_option_pricing_models.ipynb | 2 +- .../quickstart_option_pricing_models_quantlib.ipynb | 2 +- .../code_samples/credit_risk/application_scorecard_demo.ipynb | 2 +- .../credit_risk/application_scorecard_executive.ipynb | 2 +- .../credit_risk/application_scorecard_full_suite.ipynb | 2 +- .../credit_risk/application_scorecard_with_bias.ipynb | 2 +- .../credit_risk/application_scorecard_with_ml.ipynb | 2 +- .../code_samples/custom_tests/implement_custom_tests.ipynb | 2 +- notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb | 2 +- notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb | 2 +- .../application_scorecard_ongoing_monitoring.ipynb | 2 +- .../quickstart_customer_churn_ongoing_monitoring.ipynb | 2 +- .../time_series/quickstart_time_series_full_suite.ipynb | 2 +- .../time_series/quickstart_time_series_high_code.ipynb | 2 +- .../operational_deposit/operational_deposit_poc.ipynb | 2 +- .../customizing_tests_with_output_templates.ipynb | 2 +- notebooks/how_to/configure_dataset_features.ipynb | 2 +- .../how_to/document_multiple_results_for_the_same_test.ipynb | 2 +- notebooks/how_to/explore_test_suites.ipynb | 2 +- notebooks/how_to/load_datasets_predictions.ipynb | 2 +- notebooks/how_to/log_metrics_over_time.ipynb | 2 +- notebooks/how_to/run_documentation_sections.ipynb | 2 +- notebooks/how_to/run_documentation_tests_with_config.ipynb | 2 +- notebooks/how_to/run_tests/1_run_dataset_based_tests.ipynb | 2 +- notebooks/how_to/run_tests/2_run_comparison_tests.ipynb | 2 +- notebooks/how_to/run_tests_that_require_multiple_datasets.ipynb | 2 +- notebooks/how_to/run_unit_metrics.ipynb | 2 +- notebooks/how_to/use_dataset_model_objects.ipynb | 2 +- notebooks/tutorials/intro_for_model_developers.ipynb | 2 +- .../tutorials/model_development/101-set_up_validmind.ipynb | 2 +- 30 files changed, 30 insertions(+), 30 deletions(-) diff --git a/notebooks/code_samples/capital_markets/quickstart_option_pricing_models.ipynb b/notebooks/code_samples/capital_markets/quickstart_option_pricing_models.ipynb index b019c7b45..6da5cffde 100644 --- a/notebooks/code_samples/capital_markets/quickstart_option_pricing_models.ipynb +++ b/notebooks/code_samples/capital_markets/quickstart_option_pricing_models.ipynb @@ -82,7 +82,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb b/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb index e25191847..a8548826b 100644 --- a/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb +++ b/notebooks/code_samples/capital_markets/quickstart_option_pricing_models_quantlib.ipynb @@ -120,7 +120,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/credit_risk/application_scorecard_demo.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_demo.ipynb index 5bb5985f4..1bc8aa931 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_demo.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_demo.ipynb @@ -86,7 +86,7 @@ "\n", "\n", "### New to ValidMind?\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb index 349cfd30c..8025ece67 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_executive.ipynb @@ -37,7 +37,7 @@ "\n", "\n", "### New to ValidMind?\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb index 2c91302c1..77e5b2ba0 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_full_suite.ipynb @@ -37,7 +37,7 @@ "\n", "\n", "### New to ValidMind?\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb index 0d6f4e270..0b909ddda 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_with_bias.ipynb @@ -75,7 +75,7 @@ "\n", "\n", "### New to ValidMind?\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n" + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n" ] }, { diff --git a/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb b/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb index ca1bdb4e3..961a92c5c 100644 --- a/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb +++ b/notebooks/code_samples/credit_risk/application_scorecard_with_ml.ipynb @@ -37,7 +37,7 @@ "\n", "\n", "### New to ValidMind?\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb b/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb index 34ed27e57..80393d5f3 100644 --- a/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb +++ b/notebooks/code_samples/custom_tests/implement_custom_tests.ipynb @@ -78,7 +78,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb b/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb index 0f88228e1..055028eca 100644 --- a/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/prompt_validation_demo.ipynb @@ -66,7 +66,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb b/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb index f6942033e..dfc9ccc15 100644 --- a/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb +++ b/notebooks/code_samples/nlp_and_llm/rag_documentation_demo.ipynb @@ -33,7 +33,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb b/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb index e4e48884d..04ca7aa1c 100644 --- a/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb +++ b/notebooks/code_samples/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb @@ -33,7 +33,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb b/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb index 156f3fb14..5396ee1aa 100644 --- a/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb +++ b/notebooks/code_samples/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb @@ -74,7 +74,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb b/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb index edd3ca9b5..c674ffd42 100644 --- a/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb +++ b/notebooks/code_samples/time_series/quickstart_time_series_full_suite.ipynb @@ -77,7 +77,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb b/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb index 8873b8524..8418dcd0a 100644 --- a/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb +++ b/notebooks/code_samples/time_series/quickstart_time_series_high_code.ipynb @@ -77,7 +77,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb b/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb index 9fc708325..b98aa9d67 100644 --- a/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb +++ b/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb @@ -37,7 +37,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb b/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb index 980529dd4..c787a8831 100644 --- a/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb +++ b/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb @@ -85,7 +85,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/how_to/configure_dataset_features.ipynb b/notebooks/how_to/configure_dataset_features.ipynb index 9bf927740..2e381bd89 100644 --- a/notebooks/how_to/configure_dataset_features.ipynb +++ b/notebooks/how_to/configure_dataset_features.ipynb @@ -67,7 +67,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "For access to all features available in this notebook, create a free ValidMind account.\n", "
\n", diff --git a/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb b/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb index 8dc4ab10d..2529727af 100644 --- a/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb +++ b/notebooks/how_to/document_multiple_results_for_the_same_test.ipynb @@ -82,7 +82,7 @@ "\n", "### New to ValidMind?\n", "\n", - "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models, find code samples, or read our developer reference.\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", "\n", "