From 56437dd5d183c417e4df6a2635b6c3c15ecd0db2 Mon Sep 17 00:00:00 2001 From: sitaowang1998 Date: Sun, 13 Jul 2025 22:17:10 -0400 Subject: [PATCH 1/3] Add architecture doc --- docs/src/dev-docs/arch.png | Bin 0 -> 25822 bytes docs/src/dev-docs/architecture.md | 79 ++++++++++++++++++++++++++++++ docs/src/dev-docs/index.md | 8 +++ 3 files changed, 87 insertions(+) create mode 100644 docs/src/dev-docs/arch.png create mode 100644 docs/src/dev-docs/architecture.md diff --git a/docs/src/dev-docs/arch.png b/docs/src/dev-docs/arch.png new file mode 100644 index 0000000000000000000000000000000000000000..ec64a0cac3692797db3e7ab133d10d343f4ec11d GIT binary patch literal 25822 zcmeIb2Ut{FvNjBe2m(Tr5lKx0N)AfSHlfK;K#&|I=O`JZ$vG%Nk_43?N|Kxv1qCID zWRM_|BqzVs3g>vv+;3+7e`fBzGtc*^(7pBwdso$}_10UpACaocviRpOo<~DN!$-)$ z)zQ$<gw!Z4?co+V@q@UlM6}?-qyCZ#*92N*EqPqrOQgjrq=eZ4lY)V zyi(vh!rs-~7W@Xy;HQcP_@M*-adR1RUo+&t0zOJQI@+3Ro0}+DyP{^{<>%n#1I?Ee z1L)2JR^>q5(se6m2RqP7hMPwKH1Y9q z^Kx(tpeFkL)@68kICwa&fi?|e3u9;NUzS5n;co8iVhtYg#{-{EjJn`rYHVx%vzhy+ z08AX5&CH#DHi4(+VdRx&hV!3iJDt)MYS07CKl*}^Zt z|MMm-oln;}Sr*lxVeNhTC@14A-K@>bT~22EInvd^!PeE<@sEwB4)*rI2>#H~*xA{^ zDB7m7j7vm8P?U1DNj@rTu#Ow3D6Zb#q&k@t(@@WF25P|KG>{ zCJpD`_Rc>t9k9zEHfwC_b|PIy9zI*}Oi~sOU~dAWb@e(iT0SQ?)WiO`!0z%x|0q-C z7H|YM1or6Zb@0g&)lbRX)!59~73}Gs6N1P6Ip@#Ir_E;8?x!P8oBrXYHC!E>QRed3 zR|YrqhsA&F(cd1I2bh5)N+zaWw$}DwoAQEZ^02aYHP>)7Hbq_V02>LkS-INTf^Xd5 zqltqX=xL^C@>_$ksg0#Gs#V3!6-x;KN*L#v9qN)*k>p&C-}4JpJ6c0=C;PJ*6t`n1(Tk5oBt8*pFF4gU%Dn2 zM`M5>{sGqn|KXOn{?bkSocYv2{SP?=V_R!W6nroR9v)?I|1BQks=&!kM_DS+{J)2X z03hs7O8TGi5dU-D-~^=o>J0?`6TJa|;Qvq4C<*+k>HkQ-e`3-9h2Q_jaiOV|xtW_S zaH{_|WcXKY>~EmIpB4(jzkeZ+<2{X9PTlmW+x`W}ai4;=Uy6EGJz)$cF*JXP`k8g-tyo&TrK--4mjIE(jGS0~%^-$I?-C&=#)`uvk4 z1X0SL;uaTh5V`)T?Uz97k2nk<*^_YTN5pd)Ed30^egsIrMP7dhmwqhrOL+8m1xvr4 zkSB-9KO-{wWk;TpO26*N)3(1mfcop<@Xu)e=TB5PB*VrcxQ ze&f`?|BM(*|75uH9 ze?cj5pG36(dQRb#IryCjKoJZmb^XQ>{1^NIiX8f}S$-e(hYj)L+Fz0pf8Lyb*FO1` zR5&@If89DK@%KMt>-^n&;_orje@y>iEBwi|{n1nX(>SGnd;OnC>OV~X|81P&znA_` zAl?|5Jyri#i$pCLsD(`TzfdM*7e6|0j&$ zAF%E}DdBb9)*7hiKZaVk#(#qMegVG!`=|x(e|usO1VjGJBA5TVcjH@QfH*(ra3DkrWK@J zH2W11)?DzVM@jPcv@TkyLUJ{+@p<|#Cgl(&bCYcN`@L|OSn-(vf3^jEvMc)pZNy{G zt-k1*xM_K`{n3VC?*fg8XPf{x#sx@sDra_EMh5$1>~aM5quHQE4>6{AqW}V%M|2QA z9^+(#ZNEz2O4D{hXJ|95^Pj0_-1Ls(hKGBIH~oTK0O;+ z(ox@dk>wXI(>Ke=;FP{Kx7Z6)HVgDfL%_fOsPImjEIRTeGCe^S3Ng92VeY zo_EvX803;QLd`Nv`WxzB*?Y7}xqJIytwIm}(OWd;<`F;IBz|)9dBg?bWv-GNPc9~N z8fNWobT8MDk?mQ~;>@q%qBB97;$q~4tEkf7<;f0g4H>Ekd#t<;++8fpR%VcidLZPs z1oJbfFp*wg-TE>V*%V32DpR14`tZXeZL0Xe!6$mwL#4M|H{Zz)h-I z?g+KmEWXV-qm;%aIXg4QR+up5Yb+IjfGPGreNe`i&>cm|TV&poB8XQ-Tm)wKq^B#G zrHFW3>Goyts@=z{6~6ADoVsvSFku+63*8K{BqPVi2M({Y#hY{n69RphAk6AhBF5Lj zgtkPuHBDf`kI!w2<&#<7mm6Q--~UoZ&5%TnjG`7!sdHU~rc_+Kb~`DrSJ35i_%K$= z<9mE|EOPmt>(e+-*7#s1TS7eDdxPw2r!vCoqgv3-v|q&;{f@S@Z`QdM4?p$kM=&FUqYa2sT2*Cva=eOFEJwJywdXx+IaaaD znWw4SbUpyj`f=J~Pe!kiNu~>Cg7E=+=%ox(YpZnJRYNhz!Bji8fOG(&{fc_VYW9{2 zA5Z)sW{N|8hmDz|B2>mG+P=WnESFU;3kKJ#dWf3f{7Cp<4~$5y<=8=oZNjQ&sC^*~ zuK3{Ml+Z{wT&Jd9pVDsX;zjDjaImd!vM0)EG9L8R=rq1dDMP;FMy5$zYjt}Xk1Tgg zhv%~EU(Yio;>NVqd}L09EHN7)Mpl`4}?LDr86*Z75)ATGl)Pc7oG3#}NMhqB;$jdikJuv{v&Qce==#|91e z$T$r$W(W*e+cj!-gRAA%_-SHWNmsgGSiQ@I9}_Xk!!};=cwR^)W+3FpOoLA|t*1qk zJ4EFtoIN(I_gvS9sWy1KbuIXQKia8X>cEXiP)+#Wd9a#65m*F{RwAvJosRmRI}Fsu z`Mk>o^2z-%s&BvA&=P8QV+#O_Y)6;N=n%uqXD$cc4%4Z7?s6>&<4XoP6}k_^2?{1J zdyL(d-a}}jBACsx;We^OKr^t?1cH|}O6;;{MSTq};NHae)W`vD%$5G-mZuEeN{^%y zZIk8M>GK$ppfBNY?YsBs0x1RMmy+v9Ld~=2QuBgRG=)^~6ijq3B=#6NkHn<51*O~& z8MP!r7Jj-z<))dM=X-|h`t|xtWX~{b?jX^?DILUz=_x2fl~l@Na(JtyGgJgT+BBrO zhHO6FQ2^GLWh5|5^4Wj~8KUbgwwW0nM?iP(Ev5ssPAWo-hv8WWciwTPkMj+DSf7+o)=yBF%Z1c|@f%zym!tSZIsq{{+M#x4$LB#>1*$a=8N{e} zMJcos1lneKZs>9QdgyTNx4=v&b`)t_;$>;lXGlfZ5bLr=Iu!bM%m|xdYJono zwoU|^|MwM4Vm+d%xyv5s+X^TG1?Zc5j7JKny@(JuFuaOzN?EhWmBwkp{l7VDx`$3( zib7+MNE&jcutjX}_Nn=g3#!9fS1$XHgLXm^45#PfIgAj4nDa{-Q|^@LGR;>ooOCW! zX2CrLUlrFmi^&8)uyHZeZeI8n4IeBsRIwR+oCDBGf1mhw#ru4Axo__C6YERZJ{her zJ@@tF^GNsAaV4Lfm4Z75+wZ`kvM|{aEX8?2aF($3qe)B9*~%XH-kXy+Y+ru?GKEHjVD<(i|FwD(s|5_ zG>g@t-S|?M)$_yAu;J*<8}ogfW&~W4m;H~n$iI$M$`azEbzM8x^!-@aZschwIOEAb zmK&cz7#{)u@7fNLeT&1KApt;IJ~)9RjkJTY@E6M4z}73QxdVh0PKV+2%s&f(`QXm} zV&PTOPP}`fK73E~YZET=*~$TwNd6-G_JaSt!hueP^5Ux%EKU%_`d~ zQOh*vCLMY7Ix2P(JI;fNNXPjrfUsF8*R$i za8`us$Xw*Lz-!nV)HA6q(I`}IT>v1?+R@&8_Eqy&B=-cIpwU$PG0K8-By|h9 zkqb|IIA+Yt%Fibsyur4Tq2t&l=s6-c^sPuZ>XC;I%O4zm>l^5fX9z8HS}aIapYP4Q z7TKM(H10J|?fv!nP)K!uo-9G#O5GQDc@|2&V3XC1GEE{ES(g&UoEiie7t)u_yCz#A}o$a)W{BUO#=^bId4b zc6_+^zCeK-pdqTQ)d_K-dpTY(QJ-xRD%|px2PPd6_iW~uD9~l-uu*q+T{QEE7#0-l zHJppPu`u57qVl6j25!&I;WyU+0(5#1kCH|RIeL}u2zJ^Ii>jiAD7(mP0 zX9fAeq&k6UDFq|PwZx&%{P%=h7aZ4GaXCrm#RZD#(dG^SdhBSt{X>dkn2tvGzGe&8 zA`Vi^z)m^mR83;vGM68QHmF{K7p_1n$c+&tu}j#DKWY2@(@Uj~O}^jt6At>^zGS?I zG+%oYl`Uki9|Xa^i0)hMX%f_AxRVfodgf};VPds&L_>Y4tlBZjs?E@L}+9AI6UCcZ{%1oQCa99)ijcG!X%xt8-!YFtK$u4 zttXSb=UydCTQET^UvT`#Qtn(z7ClyV|K7XuspjFBe zNoP__lbGp@&eKp9l-BQdf617tnj;nZbX(v)k;Rh$UM^rAM%V&J;6V$>)uulZe`s^Y zmY908j!0xeRub8LdE4}z-p#jnY%+Uj7_7W*CT zYgXIe0*WO2P^1*cU@EOwd~btRw{Jm#@Xpi+UvBYrx z!!-ES$u2E$w4=Zy{v9TW!!9?{syX(N5oQn^Kw+wVMdakwB=cmEz z5#gvisbGv3prpiSAu3^+w$Pm6GZ=f-4JLe7bH(^m=OamJDJ|(wvsBgVOE+YgAo$^( zN;sg;&+beUHPPDDJOCg{T(y^{s9wLV8n3Da;os}DtFKFr2ltTcq|plc1V@CFMAW?G zx1ag&Bnmt_Vf1|>(C4BiOjD~}snLn5%--A#Jebv|BreeB0W_xsYQGUZ>gmSHkQj~y zpLrMp`e2dbXpuT4I)K~KqXYn#?jk>9)TrmRBDtcH@cX|1{9BhwoEZb)?Fop2P@sp! zuu%j!v*;LsX>9W=UniQfkN3ZD3LUPsUGYCYd^MLzwhO|4YvAi#hmG&P0|eL7v$PL; zim!$`1G|7k%t6?Tl4}lfePyKbZWtb=6ySg)IFO$J`}4lZ-ya~ia#QOcXZg|A&_IpT z%)%#^J~w^kYcqw}e)7OqfG@ z1%#!S!IK<`hr|?Qh`LX7k76}h+W-LoQSScMmjP03eMuC~M4{CI6>%j%H9Z8B);n;j zUcYC^TjgHUo3JdvHNv8Z?V_6&(I%qUE5s( z6+JTE>#r}xi|sEk2+c(CUfRLO`aYlSPu4@isF32S&rud-Zy6%APsGpoUesWrmAmWG zb?Hd50@PWyCIGOOau=Q<5Bj7rfvfP#9MYT$ik zi0(lj@SC~7gJ=MDO&{O+%ixn|HbWdj`wMwz3}xQTAz!C)2czI1R#@!*&V+y82Sh!i zTCqAKfCJh3@z}4l9_z$&)XoxJ6!m__N~nR0AX3YOEGjBQ8_kh5= zJzP$I2dJgi`w*}{13s$_vfqzZ{TIIMylD;f{q0ue6`o5mK{l-A0y z8Pj7y5tzJ3w0^siv%!z=U$vG3w;`@oW*{mn%-M9v6Ac)u6pmZ+0Gkv5-Uqi2Dwq`% zOV(qTW<{UVhoV0@Zq88ol-5Ow=N5xB1l<_z)@7v@%(dNBK&PXPzBqdZz8dnD_5~%NmlrUwl}a`ErKML zfunbJ!{vc2nniKDMTC0YJ=diVF|oW~CNQ=Jx119KU|Wr5d|hDy$ z;>80b$057QO)2r%9suqT{~qLv0s1fiA*<|Iu`we=ooBl*nSL3q<`e}R81^kcLC&D^ z3R+k?-En6ZHcx9Ee-wHO&zj_eMnJ3f17qPhrdtcI=MhYYKpm>~T{q8uO;eXDJU{0l zMm&QG7CP|sm^>MG%O?emik|@pfSf>FKg#}oDC>DSJh6GrG#zdH8;Cs?Kny`{fYT4Q z?iD%9o)q@hVW}0s+U8;5Uwr$%SYz=uZy!n1Ipbyl(`r&N$Azg79O5E?{Bs=^80urR z4ZK3ZAu89C%30VMO@(`XG8vw+eLVrY^Q;QqIC&#==F1RYw`~=VTMs+$lOv7r)^sZJ zLy4fiYgY{CBBVQjFD0-%7Fdb}wXtx%JA&oH3O>@WVWN$sV7Y#3m zu~wIv3JZyh&mu6E$rCo8&V^vrVF*XO#I;ZGN{hI4P^+*w{r)oTY>cj$88)X*DXr;ts5v@i;c&HsVdsO~ zC}guuGsorh*}Iplg37G?1r^&%J$8KRZzVS z(C|RR_n47J9)nI@)XY_MQeN=@1~aK!*E*QoGx3=E6lPpi5auea*J~9*1BWZfGSrJSr5_S z@9ZZ|iFe#5M^|C9yz}i1zezD22mpE?@|+!XlHWsRD!}Q2yk<|BtVjo0E^-oLfY;h? z8EXfRHo{iJxe7w+LZ3Vm8=L7cJFB3Yj?wNsPc6n)v)Mlh{j61GJJMcmls47U?Bn<4 zNp(LBpCTh<4|09JoK=Cz^ZHw-#|YCs`7D(YkOKmsn73K z?NF1YX0)&!Ru2bKMhavq74Uvy)hPWW>hPmy#WeDO5+=7PR6 zyJUR$Gv4yzw9CebOYp!NIwD&-TPRas8pWJ6l*QbQAq^-wvPiw)e5U{Lo+ik`T=GfO z_+enwNTXB?ch+qdHW_V5`V=P0ImwMn(+8NhPYZ}11{A~5lxaWS4}iP9hnSaQX%X;(f*aJ2KW3DdnS;0jaPyXz9dMtrrTiOsz@vqRfLznXkC+Cve4XH=mI!ZI@$z`JTUYnn0ck%FXoHcK5F9==D#Bu zK>jj`Xo;+bK0ESmA~>N<3IWMxr0v)Nj{4 zDGkSOC2l3#0jvR{M#-KU4UP5{susYuM=`bh1ui$~a!m_Q*}&{GTIDY;iGBILW1C&B zlo~V1upnNQ4K@N-(nLq=AuhpGVp6X`oSD$CZLbJKoD?s`aY0&czqwMF0ma7F>v5tZ zbBDk%ImZ#YW1gR(5HAlh3pX7W33;Y-Y?N{y)Xc#eeMo_Hr5xor68HOb=$aH}%L2{r zCh2wVUs0lam}jS^u|Z|=lSx&RK*Mg5Yv(@`V?V1Wa`8;Wvy=QHFG;4OF-gqWs>`fKHXrE z8pG6+fRKEJ@JIDdHjErD?Mg|27$GYogXUaXYWpzhX^K&eJwwl?FMx&7W^U@4ie6nj zP0w&xyE`2zt;aJ}rMP)kMPd4@ZGG@qy_d^+)tSjoln zNh`JQT~&oLU&2S1;V!N$&LAJ0*C!w#kOVlJU9XB{XKJRR zNGkL^qhG4MAg*@P%tyEStw5i3Nea&orKVjmt(DEZ#X25Iu%K6t>^w9CLFBw{OPWI8 zUh{@BuLGEk-FKEBYX+Sq4u6_AnMI38dw9Zq)NcA6^iu}`RL&1yB_aGVfdLNonfy=> zI>#-M{otK?r!D8ISvXGRhnA*lTyznhQ1O@J=jw?Rv)1lvxDq;|Xfg#(m#!Y&%Q4j2 zo>wk7pqcJ53%2k!9}Ic6ow-7doJ7jjs|oMy|=MVdE0jU#Qo=OFlFEVMtOLRqZPrUWuW8L?U zb$QnnSy$HkQZiVKtGsu2-09GcFpUEwE?%{YThM%%J^IdOB~%YH!>`Hq`_(u(XVNva zHLmkfBGY)Dm(E)E=ObLVM=a-|5NwG~3J`*(dXi(B`+Yz15LznABT`wFCyHOa!CW0# zdBwBJ6%@CaELwvw4{6BU)E2Xgy}?unG8Ni!p)kl^2uJsMC*t(U=zt``Gp$5=RkNBv zRBcVwX9DX*Vq9q=s2e^T)JQkK0qR^ed`9g9<$4Grc47rlM){;}M$XzJdblV$bbo$e{O^JGRpJq<@9k9!;8H&xbB4$DOO&aLsd2KkEG0`_2;@}RL zQTj)XLTas#lO4mFA*$;<0%AlpHP75mt($HkI#V%Rc5>abT_S;4`7r~Na~HKocoQSC zTj1PA&}9h}z81?40zJ?g^)BZ%XS zZjmQPOH7R3jyy<#T#XCQ*fPC4P^6aU`~ka7%ymJvwfAy*d8T$J31NtA2F5ueNvyR? zX06v)u)P_w0x^RL5?YB!FF;;}XygqaU2~dlZ|_4L?xjYOkFnfLyv}2xqx*YaBt^@aOaB66`@&S4z)$!7Ga50jlr%k{1439__zlECS~;O_Qvj>}cPQv&3f zIXeT58sz=KR<9R(;}~`0F8@?#B+faZ^@V~ZRQ96Kpau#V)xtHvtEXhEpdXG6v1`|f z3AV%tYLS$^h0wxrd)~9#=kU{J|$)i`Rq|1aVz} z-)NtU9EPOSvnmeB5?*6`9~${)D=pkSt!TgkFR#D)vR{+mXp>w*c=mZ3X~s=r@upFp z#cKTyPDG z@J>l9fl5n$;S8~2#qF>rLx*V#z;e?9y7=(0q_r zit(*{@Ff@olFmPg{cI$QZN}c{EMQHv~Z{R3?^2$)RD33{{u+gl?WMNP@!RfX^Et69P$)UzeZ;`N!6TUlXXsH`T@GJ?h$)f0gdmP z=Lwn7TtW zyfGzcPy~1Ngx`H|@HKOtjgm;S+%n$C!XT?${|vH{!2+2_3USyaIxpPW#~j=6O?pPl zjoTJLX36!23!XEo^3{81g}v0zAC)Tx1sB{D&&RIg`S{W=A64rKzxV0d=H#{T8fm%& zgFf;EG?i6z06Ns_Q<tppT)%q>$Uob5Ivhykk!XbX>(~xa6&{%$%zrvIhe2?*0gs@P#!jEs5Ac9f zR=&%8ud+F$&KCe7#%2F3L6?WaW^X~AK?%s7Ep$TWF337<)0{11i4X#h^)uzk4bIc+ zrkzocyI(%Pxi1TFG4#@V>xo8D(#xuJ&-N=FCdBTEcs&*160rXy(ApK{LYu1Zj6=$z zdHLB)#OuO#4l+!(nyL)}+CUD>bn-E+QK@5#-s zQy%cSHA1e((XQ~0FV1%W3ETzq;f_x7mj1x9Z7OU3h=}aK?!FB+HZF%$W{XLB*Cd!sxJsKoGi<*M>-X`p|?A4sY}z@uLpqyofd6t z+640G40;O|pgOak`F}sYR7F61Tk$b>OD7!MDcQk16|#m@B>Xc-YeARqVblg(l@Rhg z_rU$@$JVxH+C2wC{6Z^w!mf0$43RP`Yn$zN0d(c^m9C5G>(z(AG{0w-JJ8`S2LUyf zrO}OMJZ6qJTYIQx}Sfn*1aQjnT?^j3n9g($uV)CaEy$*rA(VZ22b`}D#U zFV=bad58=%G4DY@wVd#CJXIM zLC41(6S3)*li#P`(4;rjG3yPOGt9n1kkrS>C5D%HBp}!uUh@R0Z7n>1RT1!s$6WFC zQY~^-EW$w6R<8XCxUQ7>S}wtqWBGZ__ern8OiJPH-cMh&N<88(F87kWP(4b?TXAcT zfhU~H5MpiiHoLtH*OFFxZJ>ETSa4W>oLzmDlJ=V2&bNcB#)j`@sPtF5+u<2fY1Wph zrMDo<&jpgMCXYGe@q8s_o@o5ONhB;Hl^{RT`%l zx7jM(O8%5Q>E_q@#^O$O3r)=c;dgm%q*=4jh_GbJ1C=D1YlB2vl+RzP2-?zQ*o{l` zm7TeZVYj=jv=kRpmkz)5!0MZA`)6H!KC7Afs%(6!!lr7{Hm;?DHxsoQcd()0SjSkZ z@?tiQc^+a%_T&Lmqa}X-ecVWo7K6Kj&&`OOkaiT7>{YKX2ws?0U*sYAhO45tDSOV_ z_|VTSY7|bb2&GZOkMuA}93q;1J(5@RguQg2688mac+>4yuw2OM<*{Vb+k@6$E;hFIm4TsK{kd&SCVO?)3kAK`2L3Z28RTV+}I0`!dXilEr=Nmv*NNBGm}NDkmb@tv$UHIad+IOarQ%lZwoKG6+y`q z{lFVEjvnuCr!^2a4b3X`Em7k{h1)L5K1@p*aKcQFZHS?Jr|obb0jK7jPzZS=Tw%U# z(|?9bw;Jax?-c3MGP4i0(7VlQ4kn2I`u3-LU&j#?gHu`!XFnWfh1x#A(3@*DxQlLl zpwE~X-WTxYcJGbV_H9ZBro$;+HvO7709F-2){qocR|pF1FS{#D`AUPp5tYq+4_;fi zJyc2!UP4hh6j6LhMiAc+!)2|s-N=J9>+0UWROQL+*Cknx-C69R_E3@XW4#wIJHSvx zg-yXp5FcuTtApcD6`|X&WNIxXy_u_CJ8MYtLSAsG6+SH&C0QHLGb?Kgejufugn+?< z)W1H=9!Ii&`KsK2drAMXYEBSKzV5ekV}64djyZ#AH%!Xfim$e2$0A^dJg;3a{lL<_ zw%o&9UKmx|A42B44P@J+5n)r@$ac9Erh4K5n`Ptlu}3BG_%r>C!rMZc1+-7V8$|CC z3iIAAY$&_-TG(Y)2xDk!Jf!1t{388yCioJr0~+l{o6vban~0>IYN%_-o`srDZ;H6?ggOa zmE~Et?o0@%YBBHcyxa4V3ga9B&E}23mVP$F)ya?GM;@*rz9KZ`C7w)}IMI$&1-1C- z!90zZzdVoM@#N$F^NHc-Z;H*%#kNy*SIt=Ze6P=OXuLXONn`qfEDf#+;mybvCvrd= zjmiFc^ewD$uRki=D@sexh)7NPwWHs$Ts^w2uwT_qAJzV61zx|;+&An35pYuOw!J-u zb*%EQyCdV`4h#1=-|VJ$={!Hn1kq~8o1X`ZHUCF0roAV(mPR-pxbUf#w*gf733`Wp z`RMe-1MojM+I7r^FOO$p#nl1ol0F6{Tal=$IB(4j^Lp@-OqxBwszX}~EJZ=trF>dT zYKycz{QOT@c8j}1ztzctQYKM%8Tk@l7%Gd6BK-+F0PECoK)~Y$;CyEcHS6&K03jNQ zOiDyeJ|LsFI33CP5LLGe^5de>rsMTqsCRt4*ILeXP%%g{lO6^wJd*#e)2IlQTqwGI zUlCdoPT5;#V&-V>3p{`o@WGp3t0($F9xQc@{`tPcLkNS+ z+qXH_i&e74vSiv}xmdJ4+W2MfKe6n3*&r2!f4=V_Hm7kUhYB_fa(qC+?BYfbCwW5# zC}qy{+h1(uZMJb+BWVN`LG9S6YG&3OkLd^{0LjwcXXv+QzHSKwhkEZ#Z7dEx}}m!FpOydP~>B^@~PH!XUt;& zRH<-o#9X`tOn;xW(E~bNO7&TBV8tP=&;Qic+WV{0)>jeOp^FdKkbe+elGI;$5 z>z3;~X;eM-Z4jc+TWk#KIUs$z4J$2_rds66&|sK67?LeL@vxwMwe1+;6~eiM)JnqH zRLr0`UbZ}XnzCI{P43vMseX*LFP_F=1joRnQ(s;p4saWJnrVG`pW9m^t1-Zz# z!ijEN$Tawn^jL`umQwTefcS|Tr##P_)xeDbyJYqTDQ>xy_SUD^DUU$CCXM%G;5pQL za)h%%a}T>{e4=Rl51oQ!p#xc>K1GgGt*vXT!xap71HXfP)V>-ALk4I*RLSNLA&evF zmSk!-B)nU@`0ze6L9s;bj20|&-%_h^;QZU4uVIo$~FhY9}#mb%Y&%OnQR_RSfp#QuhyTvkQQ0)N zLYwQ-aQiil;t3lA{vQJJ{2-h-mkw9V{gP|XFY_5ytmM)ss?ElYsuC)Xi78d#T~B33p)try)iS$H1ZAo`iuN@f6v6wMTvAcrd* zS1`fm#UWjn^;X3v&}> zB}N)mMbS0F9xHIvh5)jJedz_r&bA8@vDu7y&O{5UqK{k>a14;paTa8Kj!#N4Hd(0N zG*oZD-?d>*v|i%D)WiGJOzfi+pUMXMn~p96mVxMrDH7L=j<_Df_Jg#P1QqiehnguY zP~uQXWE9pR}^q8N>z zVg$hY2E`)xQ{QjM8d_!_@Q!MQELkqIJt7~(Z7BLd?VLz#Wv>uxS9iJ160k2a@zH12 ziNoxn&s^q}E80VvB$>j~AL~}icoOVV_D0k3O((%snU!BLxCfc@X2?nsHlk?X5j;yJ z@UHCTl(Dy#anZ}1iEUkc=xv-yzELhP0Q2bBKd$u!AQPzEFA_6C+vcV$rarE zKwie(bVWMKO$Bf1^pC2Lc)df-k824-c0ZN6*=q(^ydO$^67Td*GX{FFvr6tn;&}$_ zt;fT)$-dTkL;GqpJ%lzxrKDz{tdF}yyNrg>urD_+`oWsZ$DId2LvAeey@e3~W3ZTn zeLH7vhDXp-y=2$bw|%_FbU78D+m9oVS0@V5PFG`J@;SC0|MvHkpD+Xe&bInnfuT39+1ds_e?>E_Le529z$_D@OxBQ0p)_G$q zZsYzul=^6q`~g4K^SVCaYu(u=(pkdJg)D+v2#EW`#?h*NV48o&#bRSa68SN0{Tzbk oz9>~)wyX4Qh~@CTb|@|nufTpAx@rpk!xA)vv@*O@$|&gn066YQ>;M1& literal 0 HcmV?d00001 diff --git a/docs/src/dev-docs/architecture.md b/docs/src/dev-docs/architecture.md new file mode 100644 index 000000000..362f42a17 --- /dev/null +++ b/docs/src/dev-docs/architecture.md @@ -0,0 +1,79 @@ +# Architecture + +## Spider Architecture +`Spider` consists of several components that work together to provide a scalable, low-latency and +fault-tolerant distributed task execution system. + +```{image} ./arch.png + :width: 80% + :align: center + :alt: Spider Architecture +``` + +### Storage +`Spdier` relies on a fault-tolerant and ACID storage, e.g. MariaDB, to store all the states of the +system. +The storage stores the following information: +- Tasks metadata, including: + - Task ID + - Task inputs/outputs type and values + - Task status +- Job metadata, including + - Job ID + - Task graph + - Job status +- Data objects, including: + - Data object ID + - Data object type + - Data object value + - References from tasks and clients +- Client/Scheduler/Worker metadata, including: + - Client ID + - Scheduler ID + - Worker ID + - Heartbeat timestamps + +### Scheduler +Scheduler is responsible for: +- Allocating tasks to idle workers on their request +- Failure detection and recovery +- Garbage collection +- Straggler detection and task replication +For now `Spider` only supports a single scheduler, and we plan to support multiple schedulers if it +becomes the bottleneck of the system. + +### Worker +A worker executes tasks allocated by the scheduler. It runs the following steps in loop: +1. Request a task from the scheduler +2. Fetch task inputs from the storage +3. Spawn a process to execute the task +4. Store task outputs in the storage and update task and job states +Each worker only executes one task at a time. + +### Client +Client communicates only with the storage to submit jobs and query job status and fetch job +results. + +## Data Abstraction +`Spider` provides a simple data abstraction for task inputs and outputs, which encapsulates the +- locality of the data, i.e. the addresses of the data +- checkpointed or not, i.e. whether the data is persisted +This abstraction allows `Spider` to support: +- locality-aware task scheduling +- fine-grained failure recovery +- garbage collection in background + +## Fault Tolerance +`Spider` is designed to be fault-tolerant. The system can recover from failures of a scheduler or a +worker. + +Schedulers, workers and clients send periodic heartbeats to the storage to indicate their liveness. +If a scheduler fails, the host can restart a new scheduler instance and fetch the latest state from +the storage. +If a worker fails while executing a task, the scheduler will detect the failure and perform +recovery of the job. +- Identify all the failed tasks inside a job +- Compute the minimum subgraph that contains the fail tasks where all inputs to the subgraph are + available +- Invalidate all the tasks in the subgraph, set the tasks on the input boundary as ready and the + rest as waiting diff --git a/docs/src/dev-docs/index.md b/docs/src/dev-docs/index.md index 637fdad1d..8731af5cb 100644 --- a/docs/src/dev-docs/index.md +++ b/docs/src/dev-docs/index.md @@ -6,6 +6,13 @@ sidebar (if it's hidden, click the icon) to navigate ::::{grid} 1 1 2 2 :gutter: 2 +:::{grid-item-card} +:link: architecture +Architecture +^^^ +Architecture overview of Spider. +::: + :::{grid-item-card} :link: testing Testing @@ -17,5 +24,6 @@ How to test Spider. :::{toctree} :hidden: +architecture testing ::: From b2236b4fe7a8c1a3aacfce9bc5a596663bfc0484 Mon Sep 17 00:00:00 2001 From: sitaowang1998 Date: Sun, 13 Jul 2025 22:39:53 -0400 Subject: [PATCH 2/3] Improve readability --- docs/src/dev-docs/architecture.md | 4 ++-- docs/src/dev-docs/index.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/src/dev-docs/architecture.md b/docs/src/dev-docs/architecture.md index 362f42a17..05e00c74c 100644 --- a/docs/src/dev-docs/architecture.md +++ b/docs/src/dev-docs/architecture.md @@ -11,7 +11,7 @@ fault-tolerant distributed task execution system. ``` ### Storage -`Spdier` relies on a fault-tolerant and ACID storage, e.g. MariaDB, to store all the states of the +`Spdier` relies on a fault-tolerant and ACID storage, e.g. MariaDB, to persist all the states of the system. The storage stores the following information: - Tasks metadata, including: @@ -72,7 +72,7 @@ If a scheduler fails, the host can restart a new scheduler instance and fetch th the storage. If a worker fails while executing a task, the scheduler will detect the failure and perform recovery of the job. -- Identify all the failed tasks inside a job +- Identify all the failed tasks within the job - Compute the minimum subgraph that contains the fail tasks where all inputs to the subgraph are available - Invalidate all the tasks in the subgraph, set the tasks on the input boundary as ready and the diff --git a/docs/src/dev-docs/index.md b/docs/src/dev-docs/index.md index 8731af5cb..eb5806f17 100644 --- a/docs/src/dev-docs/index.md +++ b/docs/src/dev-docs/index.md @@ -10,7 +10,7 @@ sidebar (if it's hidden, click the icon) to navigate :link: architecture Architecture ^^^ -Architecture overview of Spider. +Overview of Spider's architecture. ::: :::{grid-item-card} From 872dc0099d48867abb182c1ce94ee4188b0c266e Mon Sep 17 00:00:00 2001 From: sitaowang1998 Date: Mon, 14 Jul 2025 09:54:19 -0400 Subject: [PATCH 3/3] Reformat markdown --- docs/src/dev-docs/architecture.md | 68 +++++++++++++++++-------------- 1 file changed, 38 insertions(+), 30 deletions(-) diff --git a/docs/src/dev-docs/architecture.md b/docs/src/dev-docs/architecture.md index 05e00c74c..1fcdfa2e9 100644 --- a/docs/src/dev-docs/architecture.md +++ b/docs/src/dev-docs/architecture.md @@ -1,6 +1,7 @@ # Architecture ## Spider Architecture + `Spider` consists of several components that work together to provide a scalable, low-latency and fault-tolerant distributed task execution system. @@ -11,39 +12,43 @@ fault-tolerant distributed task execution system. ``` ### Storage + `Spdier` relies on a fault-tolerant and ACID storage, e.g. MariaDB, to persist all the states of the system. The storage stores the following information: -- Tasks metadata, including: - - Task ID - - Task inputs/outputs type and values - - Task status -- Job metadata, including - - Job ID - - Task graph - - Job status -- Data objects, including: - - Data object ID - - Data object type - - Data object value - - References from tasks and clients -- Client/Scheduler/Worker metadata, including: - - Client ID - - Scheduler ID - - Worker ID - - Heartbeat timestamps +* Tasks metadata, including: + * Task ID + * Task inputs/outputs type and values + * Task status +* Job metadata, including + * Job ID + * Task graph + * Job status +* Data objects, including: + * Data object ID + * Data object type + * Data object value + * References from tasks and clients +* Client/Scheduler/Worker metadata, including: + * Client ID + * Scheduler ID + * Worker ID + * Heartbeat timestamps ### Scheduler + Scheduler is responsible for: -- Allocating tasks to idle workers on their request -- Failure detection and recovery -- Garbage collection -- Straggler detection and task replication +* Allocating tasks to idle workers on their request +* Failure detection and recovery +* Garbage collection +* Straggler detection and task replication For now `Spider` only supports a single scheduler, and we plan to support multiple schedulers if it becomes the bottleneck of the system. ### Worker + A worker executes tasks allocated by the scheduler. It runs the following steps in loop: + 1. Request a task from the scheduler 2. Fetch task inputs from the storage 3. Spawn a process to execute the task @@ -51,19 +56,22 @@ A worker executes tasks allocated by the scheduler. It runs the following steps Each worker only executes one task at a time. ### Client + Client communicates only with the storage to submit jobs and query job status and fetch job results. ## Data Abstraction + `Spider` provides a simple data abstraction for task inputs and outputs, which encapsulates the -- locality of the data, i.e. the addresses of the data -- checkpointed or not, i.e. whether the data is persisted +* locality of the data, i.e. the addresses of the data +* checkpointed or not, i.e. whether the data is persisted This abstraction allows `Spider` to support: -- locality-aware task scheduling -- fine-grained failure recovery -- garbage collection in background +* locality-aware task scheduling +* fine-grained failure recovery +* garbage collection in background ## Fault Tolerance + `Spider` is designed to be fault-tolerant. The system can recover from failures of a scheduler or a worker. @@ -72,8 +80,8 @@ If a scheduler fails, the host can restart a new scheduler instance and fetch th the storage. If a worker fails while executing a task, the scheduler will detect the failure and perform recovery of the job. -- Identify all the failed tasks within the job -- Compute the minimum subgraph that contains the fail tasks where all inputs to the subgraph are +* Identify all the failed tasks within the job +* Compute the minimum subgraph that contains the fail tasks where all inputs to the subgraph are available -- Invalidate all the tasks in the subgraph, set the tasks on the input boundary as ready and the +* Invalidate all the tasks in the subgraph, set the tasks on the input boundary as ready and the rest as waiting