Kev ntsuas kev siv hluav taws xob thiab qhov zoo sib xws hauv cov khoom siv hluav taws xob hluavtaws: Ib qho kev sib sib zog nqus dej thiab kev sib piv
Introduction
Nyob rau niaj hnub digital era, kev paub txog cov ntaub ntawv ntiag tug tau nce siab. Cov neeg siv nce ntxiv paub txog lawv cov ntaub ntawv raws li tus cim digital ntiv tes, ua rau muaj kev pheej hmoo rau lawv tus kheej thaum muaj ntaub ntawv ua txhaum cai. Qhov kev txhawj xeeb no tau nthuav dav ntxiv los ntawm cov cai xws li GDPR, uas txhawb cov neeg siv los thov kom tshem tawm lawv cov ntaub ntawv. Thaum xav tau ntau, txoj cai no tuaj yeem raug nqi ntau rau cov tuam txhab vim tias kev nkag mus rau cov ntaub ntawv raug txo qis; Cov kev txwv uas feem ntau yog lub sijhawm- thiab siv peev txheej los kov yeej.
Table of Contents
Dab tsi yog cov ntaub ntawv hluavtaws hluavtaws?
Nkag mus rau cov ntaub ntawv hluavtaws, kev daws teeb meem rau qhov teeb meem no. Synthetic data generators tsim datasets uas ua raws li cov neeg siv cov ntaub ntawv tiag tiag thaum khaws cia tsis qhia npe thiab tsis pub lwm tus paub. Txoj hauv kev no tau txais kev sib tw thoob plaws kev lag luam, los ntawm kev kho mob mus rau nyiaj txiag, qhov twg tsis pub lwm tus paub yog qhov tseem ceeb.
Yuav ua li cas xaiv txoj cai hluavtaws cov ntaub ntawv tsim qauv?
Nyob rau hauv ntau haiv neeg toj roob hauv pes ntawm hluavtaws cov ntaub ntawv tiam, muaj ib tug abundance ntawm txoj kev muaj, txhua vying rau xim nrog nws cov peev xwm tshwj xeeb. Xaiv txoj kev tsim nyog tshaj plaws rau ib daim ntawv thov yuav tsum muaj kev nkag siab zoo txog kev ua tau zoo ntawm txhua qhov kev xaiv. Qhov no yuav tsum muaj kev ntsuam xyuas dav dav ntawm ntau yam khoom siv hluav taws xob cov ntaub ntawv raws li cov txheej txheem ntsuas tau zoo los txiav txim siab paub.
Dab tsi hauv qab no yog kev sib piv nruj nruj ntawm Syntho Cav nrog rau qhov kev paub zoo qhib qhov chaw, Synthetic Data Vault (SDV). Hauv qhov kev tshuaj ntsuam no, peb tau siv ntau qhov kev ntsuas feem ntau xws li statistical fidelity, kwv yees qhov tseeb thiab kev sib raug zoo sib txawv.
Synthetic Data Evaluation Metrics
Ua ntej qhia txog ib qho kev ntsuas tshwj xeeb, peb yuav tsum lees paub tias muaj ntau lub tswv yim hais txog kev ntsuas cov ntaub ntawv hluavtaws, txhua qhov uas muab kev nkag siab rau qee yam ntawm cov ntaub ntawv. Nrog rau qhov no hauv siab, peb pawg hauv qab no sawv tawm tias yog qhov tseem ceeb thiab dav. Cov kev ntsuas no muab kev nkag siab rau ntau yam ntawm cov ntaub ntawv zoo. Cov pawg no yog:
- Statistical Fidelity Metrics: Kev tshuaj xyuas cov yam ntxwv tseem ceeb ntawm cov ntaub ntawv, xws li txhais tau tias thiab qhov sib txawv, kom ntseeg tau tias cov ntaub ntawv hluavtaws sib haum nrog cov thawj dataset tus lej profile.
- Predictive Accuracy: Kev tshuaj xyuas cov ntaub ntawv hluavtaws tsim qauv kev ua tau zoo, kev cob qhia nrog cov ntaub ntawv qub, thiab tshuaj xyuas cov ntaub ntawv hluavtaws (Train Real – Test Synthetic, TRTS) thiab vice versa (Train Synthetic – Test Real, TSTR)
- Inter-Variable Relationships: Qhov kev sib koom ua ke no suav nrog:
- Feature Correlation: Peb ntsuam xyuas qhov zoo ntawm cov ntaub ntawv hluavtaws tswj kev sib raug zoo ntawm qhov sib txawv uas siv cov coefficients sib raug zoo. Ib qho kev ntsuas zoo xws li Propensity Mean Squared Error (PMSE) yuav yog hom no.
- Cov ntaub ntawv sib nrig: Peb ntsuas qhov kev sib nrig sib dependences ntawm lub variables to taub qhov tob ntawm cov kev sib raug zoo tshaj li correlations.
Kev Tshawb Fawb Sib Piv: Syntho Cav vs. Open-Source Alternatives
Kev sib piv kev soj ntsuam tau ua los ntawm kev siv cov qauv ntsuas ntsuas thiab cov txheej txheem kev sim zoo tib yam thoob plaws txhua tus qauv, suav nrog Syntho Cav thiab SDV qauv. Los ntawm kev sib txuas cov ntaub ntawv los ntawm cov peev txheej zoo ib yam thiab muab lawv mus rau tib qhov kev xeem txheeb cais thiab cov qauv kev kawm tshuab, peb ua kom muaj kev sib piv ncaj ncees thiab tsis ncaj ncees. Cov seem uas ua raws cov ntsiab lus qhia txog kev ua tau zoo ntawm txhua lub tshuab hluav taws xob cov ntaub ntawv sib txuas thoob plaws ntau qhov kev ntsuas uas tau hais los saum toj no.
Raws li cov ntaub ntawv siv rau kev ntsuam xyuas, peb siv lub UCI Adult's Census Dataset uas yog ib tug zoo-paub dataset nyob rau hauv lub machine learning zej zog. Peb ntxuav cov ntaub ntawv ua ntej txhua qhov kev cob qhia thiab tom qab ntawd muab cov ntaub ntawv faib ua ob pawg (kev cob qhia thiab kev tuav pov hwm rau kev sim). Peb siv cov kev cob qhia teeb tsa los tsim 1 lab cov ntaub ntawv tshiab nrog txhua tus qauv thiab ntsuas ntau yam kev ntsuas ntawm cov ntaub ntawv tsim tawm no. Rau kev ntsuas kev kawm txuas ntxiv ntawm lub tshuab, peb siv cov khoom tuav pov hwm los ntsuas cov ntsuas xws li cov cuam tshuam nrog TSTR thiab TRTS.
Txhua lub tshuab hluav taws xob tau khiav nrog qhov tsis raug. Raws li qee tus qauv, zoo li Syntho, tuaj yeem ua haujlwm tawm ntawm cov ntaub ntawv tabular, tsis muaj kev kho kom zoo. Kev tshawb nrhiav txoj cai hyperparameters rau txhua tus qauv yuav siv sijhawm ntau, thiab Table 2 twb tau qhia lub sijhawm loj sib txawv ntawm Syntho tus qauv thiab cov uas tau sim tawm tsam.
Nws yog ib qho tseem ceeb uas tsis yog lwm tus qauv hauv SDV, Gaussian Copula Synthesizer yog raws li cov txheej txheem txheeb cais. Nyob rau hauv sib piv, tus so yog raws li neural tes hauj lwm xws li Generative Adversarial Networks (GAN) qauv thiab variational auto-encoders. Qhov no yog vim li cas Gaussian Copula tuaj yeem pom raws li lub hauv paus rau txhua tus qauv sib tham.
tau
Cov Ntaub Ntawv Zoo
Daim duab 1. Kev pom cov txiaj ntsig zoo rau txhua tus qauv
Cov kev sib tham yav dhau los ua raws li cov qauv thiab cov sawv cev hauv cov ntaub ntawv tuaj yeem pom hauv daim duab 1 thiab Table 1. Ntawm no, txhua qhov kev ntsuas kev siv tuaj yeem txhais tau raws li hauv qab no:
- Zuag qhia tag nrho Cov qhab nia zoo: Kev ntsuam xyuas tag nrho ntawm cov ntaub ntawv hluavtaws zoo, sib txuas ntau yam xws li kev txheeb cais thiab cov yam ntxwv ntawm cov ntaub ntawv.
- Kem Shapes: Ntsuam xyuas seb cov ntaub ntawv hluavtaws muaj tib yam kev faib tawm raws li cov ntaub ntawv tiag tiag rau txhua kab.
- Kem Pair Trends: Ntsuam xyuas kev sib raug zoo lossis kev sib raug zoo ntawm ob pawg kab hauv cov ntaub ntawv hluavtaws piv rau cov ntaub ntawv tiag.
Zuag qhia tag nrho, nws tuaj yeem pom tias Syntho ua tiav cov qhab nia siab dhau ntawm pawg thawj coj. Yuav pib nrog, thaum saib tag nrho cov ntaub ntawv zoo (kuaj nrog SDV metrics tsev qiv ntawv) Syntho tuaj yeem ua tiav cov txiaj ntsig nce siab ntawm 99% (nrog rau kab duab adherence ntawm 99.92% thiab kem khub zoo ntawm 99.31%). Qhov no yog thaum SDV tau txais qhov txiaj ntsig ntawm qhov ntau tshaj 90.84% (nrog Gaussian Copula, muaj cov kab ke sib txuas ntawm 93.82% thiab kab ke sib txuas ntawm 87.86%).
Table 1. Ib tug tabular sawv cev ntawm cov qhab nia zoo ntawm txhua cov ntaub ntawv tsim tawm rau ib tus qauv
Cov ntaub ntawv npog
Lub Diagnosis Report module ntawm SDV coj los rau peb paub tias SDV-tsim cov ntaub ntawv (hauv txhua kis) ploj lawm ntau dua 10% ntawm cov lej ntau; Nyob rau hauv rooj plaub ntawm Triplet-Based Variational Autoencoder (TVAE), tib tus nqi ntawm cov ntaub ntawv categorical tseem ploj lawm thaum piv rau cov ntaub ntawv qub. Tsis muaj cov lus ceeb toom no tau tsim nrog cov txiaj ntsig tau los ntawm kev siv Syntho.
Daim duab 2. Kev pom pom ntawm qhov nruab nrab ntawm kab ke kev ntsuas kev ua tau zoo rau txhua tus qauv
Hauv kev sib piv, zaj duab xis ntawm daim duab 2 qhia tias SDV cov ntaub ntawv tau txais txiaj ntsig zoo dua hauv cov kev pab cuam nrog qee yam ntawm lawv cov qauv (xws li GaussianCopula, CopulaGAN, thiab Conditional Tabular GAN - CTGAN). Txawm li cas los xij, nws yog ib qho tseem ceeb uas yuav tsum tau ua kom pom tseeb tias qhov kev ntseeg siab ntawm Syntho cov ntaub ntawv tshaj li ntawm SDV qauv, vim tias qhov sib txawv ntawm cov kev pab cuam thoob plaws ntau pawg thiab ntau yam yog qhov tsawg, nthuav tawm tsuas yog 1.1% variance. Nyob rau hauv sib piv, SDV qauv qhia ib tug ntau variation, xws li los ntawm 14.6% mus rau 29.2%.
Cov metrics sawv cev ntawm no, tuaj yeem txhais raws li hauv qab no:
- Pawg Kev Pabcuam: Ntsuas qhov muaj nyob ntawm txhua pawg hauv cov ntaub ntawv hluavtaws piv rau cov ntaub ntawv tiag.
- Kev Pabcuam Ntau Yam: Ntsuas seb qhov ntau npaum li cas ntawm qhov tseem ceeb hauv cov ntaub ntawv hluavtaws sib xws hauv cov ntaub ntawv tiag.
Table 2. Ib tug tabular sawv cev ntawm qhov nruab nrab kev pab them nqi ntawm ib tug muab attribute hom rau ib tug qauv
Utility
Tsiv mus rau lub ntsiab lus ntawm kev siv hluav taws xob ntawm cov ntaub ntawv hluavtaws, qhov teeb meem ntawm kev cob qhia cov qauv ntawm cov ntaub ntawv yuav cuam tshuam. Txhawm rau kom muaj kev sib npaug thiab kev sib piv ntawm txhua lub hauv paus peb tau xaiv lub neej ntawd Gradient Boosting Classifier los ntawm lub tsev qiv ntawv SciKit Learn, pom tias nws tau txais kev ncaj ncees raws li tus qauv ua tau zoo nrog kev teeb tsa tawm ntawm lub thawv.
Ob qho qauv sib txawv tau kawm, ib qho ntawm cov ntaub ntawv hluavtaws (rau TSTR) thiab ib qho ntawm cov ntaub ntawv qub (rau TRTS). Cov qauv kev cob qhia ntawm cov ntaub ntawv hluavtaws raug soj ntsuam los ntawm kev siv cov khoom siv hluav taws xob (uas tsis tau siv thaum lub sijhawm tsim cov ntaub ntawv hluavtaws) thiab cov qauv kev cob qhia ntawm cov ntaub ntawv qub raug sim ntawm cov ntaub ntawv hluavtaws.
Daim duab 3. Visualization of Area Under the Curve (AUC) cov qhab nias rau ib txoj hauv kev rau ib tus qauv
Cov txiaj ntsig pom pom saum toj no qhia tau hais tias qhov zoo tshaj ntawm cov ntaub ntawv hluavtaws tsim los ntawm Syntho lub cav piv rau lwm txoj hauv kev, pom tsis muaj qhov sib txawv ntawm cov txiaj ntsig tau los ntawm cov txheej txheem sib txawv ( taw qhia rau qhov sib xws ntawm cov ntaub ntawv hluavtaws thiab cov ntaub ntawv tiag). Tsis tas li ntawd, cov kab liab dotted tam sim no nyob rau hauv zaj duab xis yog qhov tshwm sim tau los ntawm kev ntsuam xyuas lub hauv paus kev ua tau zoo ntawm Tsheb ciav hlau tiag, Test Real (TRTR) kev xeem los muab cov hauv paus rau cov ntsuas ntsuas. Cov kab no sawv cev rau tus nqi 0.92, uas yog Cheeb Tsam Hauv qab qhov Curve qhab nia (AUC qhab nia) ua tiav los ntawm tus qauv kawm ntawm cov ntaub ntawv tiag tiag thiab sim ntawm cov ntaub ntawv tiag.
Table 3. Ib daim duab sawv cev ntawm AUC cov qhab nia ua tiav los ntawm TRTS thiab TSTR raws tus qauv.
Lub sijhawm sib piv
Lawm, nws yog ib qho tseem ceeb los xav txog lub sij hawm nqis peev los tsim cov txiaj ntsig no. Lub visualization hauv qab no qhia txog qhov no xwb.
Daim duab 5. Kev pom lub sij hawm coj mus cob qhia thiab ua cov ntaub ntawv hluavtaws tsim ntawm ib lab datapoints nrog tus qauv nrog thiab tsis muaj GPU.
Daim duab 5 qhia txog lub sijhawm siv los tsim cov ntaub ntawv hluavtaws hauv ob qhov chaw sib txawv. Thawj qhov (ntawm no hu ua Tsis muaj GPU), tau sim khiav ntawm qhov system nrog Intel Xeon CPU nrog 16 cores khiav ntawm 2.20 GHz. Cov kev ntsuam xyuas tau cim tias "khiav nrog GPU" nyob rau ntawm lub kaw lus nrog AMD Ryzen 9 7945HX CPU nrog 16 cores khiav ntawm 2.5GHz thiab NVIDIA GeForce RTX 4070 Laptop GPU. Raws li pom hauv daim duab 2 thiab hauv Table 2 hauv qab no, nws tuaj yeem pom tias Syntho tau nrawm dua ntawm kev tsim cov ntaub ntawv hluavtaws (hauv ob qho xwm txheej) uas yog qhov tseem ceeb hauv kev ua haujlwm zoo.
Table 5. Tabular sawv cev ntawm lub sij hawm coj mus cov ntaub ntawv hluavtaws tsim ntawm ib lab datapoints nrog txhua tus qauv nrog thiab tsis muaj GPU
Cov kev tshawb pom no qhia txog qhov tseem ceeb ntawm kev ntsuam xyuas zoo hauv kev xaiv txoj kev tsim cov ntaub ntawv hluavtaws. Syntho's Cav, nrog nws txoj hauv kev AI-tsav, ua kom pom qhov ua tau zoo hauv qee qhov kev ntsuas, thaum qhib cov cuab yeej zoo li SDV ci hauv lawv cov kev ua tau zoo thiab kev txhim kho hauv zej zog.
Raws li thaj chaw ntawm cov ntaub ntawv hluavtaws txuas ntxiv mus, peb xav kom koj siv cov kev ntsuas no hauv koj cov haujlwm, tshawb nrhiav lawv cov kev xav tsis zoo, thiab qhia koj cov kev paub dhau los. Nyob twj ywm rau cov ntawv tshaj tawm yav tom ntej uas peb yuav nkag mus tob rau hauv lwm cov kev ntsuas thiab qhia txog cov piv txwv tiag tiag ntawm lawv daim ntawv thov.
Thaum kawg ntawm hnub, rau cov uas tab tom nrhiav kev sim cov dej ntawm cov ntaub ntawv hluavtaws, qhov kev nthuav qhia qhib qhov chaw tuaj yeem yog qhov kev xaiv tsim nyog tau txais kev nkag mus tau; Txawm li cas los xij, rau cov kws tshaj lij siv cov thev naus laus zis niaj hnub no rau hauv lawv txoj kev txhim kho, txhua txoj hauv kev ntawm kev txhim kho yuav tsum raug coj los thiab zam txhua qhov kev cuam tshuam. Yog li ntawd nws tseem ceeb heev uas yuav tau xaiv qhov kev xaiv zoo tshaj plaws uas muaj. Nrog rau cov kev txheeb xyuas tau muab los saum toj no nws dhau los ua qhov pom tseeb tias Syntho thiab nrog rau qhov Syntho Cav yog lub cuab yeej muaj peev xwm rau cov kws kho mob.
Txog Syntho
Syntho muab lub ntse hluavtaws cov ntaub ntawv tsim platform, leveraging ntau cov ntaub ntawv hluavtaws cov ntaub ntawv thiab tiam txheej txheem, txhawb cov koom haum kom txawj ntse hloov cov ntaub ntawv mus rau hauv kev sib tw. Peb cov AI-tsim cov ntaub ntawv hluavtaws ua piv txwv cov qauv ntawm cov ntaub ntawv qub, ua kom muaj tseeb, kev ceev ntiag tug, thiab nrawm, raws li kev soj ntsuam los ntawm cov kws tshaj lij sab nraud xws li SAS. Nrog cov ntse de-identification nta thiab daim ntawv qhia zoo ib yam, cov ntaub ntawv rhiab raug tiv thaiv thaum khaws cia kev ncaj ncees rau kev xa mus. Peb lub platform tso cai rau kev tsim, kev tswj hwm, thiab kev tswj cov ntaub ntawv xeem rau qhov chaw tsis tsim khoom, siv txoj cai raws li cov ntaub ntawv tsim cov txheej txheem rau cov xwm txheej. Tsis tas li ntawd, cov neeg siv tuaj yeem tsim cov ntaub ntawv hluavtaws programmatically thiab tau txais cov ntaub ntawv sim tiag tiag los tsim cov kev sim thiab kev txhim kho kom yooj yim.
Koj puas xav kawm ntxiv txog kev siv cov ntaub ntawv hluavtaws? Xav tias dawb rau teem caij demo!
Txog ntawm tus kws sau ntawv
Software Engineering Intern
Roham yog ib tug tub ntxhais kawm bachelor ntawm Delft University of Technology thiab yog Software Engineering Intern ntawm Syntho.
Machine Engineer Tswvyim
Mihai ua tiav nws PhD los ntawm University of Bristol ntawm lub ncauj lus ntawm Hierarchical Reinforcement Learning thov rau Robotics thiab yog a Machine Learning Engineer at Syntho.
Txuag koj cov ntaub ntawv qhia txog hluavtaws tam sim no!
- Cov ntaub ntawv hluavtaws yog dab tsi?
- Vim li cas cov koom haum siv nws?
- Tus nqi ntxiv cov ntaub ntawv hluavtaws cov neeg siv khoom
- Yuav pib li cas
