AI-tsim Synthetic Data, yooj yim thiab ceev nkag mus rau cov ntaub ntawv zoo?
AI tsim cov ntaub ntawv hluavtaws hauv kev xyaum
Syntho, tus kws tshaj lij hauv AI-tsim cov ntaub ntawv hluavtaws, lub hom phiaj tig privacy by design rau hauv kev sib tw kom zoo dua nrog AI-tsim cov ntaub ntawv hluavtaws. Lawv pab cov koom haum tsim cov ntaub ntawv muaj zog nrog kev nkag tau yooj yim thiab nrawm rau cov ntaub ntawv zoo thiab tsis ntev los no tau yeej Philips Innovation Award.
Txawm li cas los xij, kev tsim cov ntaub ntawv hluavtaws nrog AI yog qhov kev daws teeb meem tshiab uas feem ntau qhia cov lus nug nquag. Txhawm rau teb cov no, Syntho tau pib qhov xwm txheej kawm ua ke nrog SAS, tus thawj coj ua lag luam hauv Advanced Analytics thiab AI software.
Hauv kev koom tes nrog Dutch AI Coalition (NL AIC), lawv tau tshawb xyuas tus nqi ntawm cov ntaub ntawv hluavtaws los ntawm kev sib piv AI-tsim hluavtaws cov ntaub ntawv tsim los ntawm Syntho Engine nrog cov ntaub ntawv qub los ntawm ntau yam kev ntsuas ntawm cov ntaub ntawv zoo, kev cai lij choj thiab kev siv tau.
Puas yog cov ntaub ntawv tsis qhia npe tsis yog kev daws teeb meem?
Classic anonymization cov tswv yim muaj nyob rau hauv ib qho uas lawv tswj cov ntaub ntawv qub txhawm rau txhawm rau taug qab cov tib neeg. Piv txwv yog generalization, suppression, so, pseudonymization, data masking, thiab shuffling ntawm kab & kab. Koj tuaj yeem pom cov piv txwv hauv cov lus hauv qab no.
Cov txheej txheem no qhia 3 qhov kev sib tw tseem ceeb:
- Lawv ua hauj lwm txawv ib hom ntaub ntawv thiab ib dataset, ua rau lawv nyuaj rau scale. Tsis tas li ntawd, txij li lawv ua haujlwm sib txawv, yuav tsum muaj kev sib cav txog cov txheej txheem twg los siv thiab kev sib xyaw ua ke uas xav tau.
- Yeej ib txwm muaj kev sib raug zoo nrog cov ntaub ntawv qub. Qhov no txhais tau hais tias yuav muaj kev pheej hmoo tsis pub lwm tus paub, tshwj xeeb tshaj yog vim tag nrho cov ntaub ntawv qhib thiab muaj cov txheej txheem los txuas cov ntaub ntawv no.
- Lawv tswj cov ntaub ntawv thiab yog li rhuav tshem cov ntaub ntawv hauv cov txheej txheem. Qhov no yog qhov tshwj xeeb tshaj yog kev puas tsuaj rau AI cov haujlwm uas "kev twv twv lub zog" yog qhov tseem ceeb, vim tias cov ntaub ntawv tsis zoo yuav ua rau muaj kev nkag siab tsis zoo los ntawm AI qauv (khoom khib nyiab-hauv yuav ua rau khib nyiab tawm).
Cov ntsiab lus no kuj raug ntsuas los ntawm qhov kev tshawb fawb no.
Kev taw qhia txog qhov kev kawm
Rau cov ntaub ntawv kawm, lub hom phiaj dataset yog telecom dataset muab los ntawm SAS muaj cov ntaub ntawv ntawm 56.600 cov neeg siv khoom. Cov ntaub ntawv muaj 128 kab, suav nrog ib kab ntawv qhia seb tus neeg siv khoom tau tawm hauv lub tuam txhab (piv txwv li 'churned') lossis tsis. Lub hom phiaj ntawm cov ntaub ntawv kawm yog siv cov ntaub ntawv hluavtaws los cob qhia qee cov qauv los kwv yees cov neeg siv khoom churn thiab ntsuas qhov ua tau zoo ntawm cov qauv kev cob qhia. Raws li churn twv ua ntej yog ib qho kev faib ua haujlwm, SAS tau xaiv plaub tus qauv kev faib tawm nrov los ua qhov kev kwv yees, suav nrog:
- Random hav zoov
- Gradient txhawb
- Logistic regression
- Neural network
Ua ntej tsim cov ntaub ntawv hluavtaws, SAS random faib cov ntaub ntawv xov tooj sib txuas rau hauv lub tsheb ciav hlau teeb (rau kev cob qhia cov qauv) thiab cov khoom tuav (rau cov qhab nia cov qauv). Muaj ib qho kev tuav pov hwm sib cais rau cov qhab nia tso cai rau qhov kev ntsuam xyuas tsis ncaj ncees ntawm cov qauv kev faib tawm yuav ua tau zoo npaum li cas thaum siv rau cov ntaub ntawv tshiab.
Siv lub tsheb ciav hlau teeb tsa raws li cov tswv yim, Syntho siv nws lub tshuab Syntho los tsim cov ntaub ntawv hluavtaws. Rau kev ntsuas ntsuas, SAS kuj tau tsim ib qho kev hloov pauv ntawm lub tsheb ciav hlau teeb tsa tom qab siv ntau yam kev qhia tsis qhia npe kom ncav cuag ib qho chaw pib (ntawm k-anonimity). Cov kauj ruam dhau los ua rau plaub cov ntaub ntawv:
- Lub tsheb ciav hlau dataset (piv txwv li thawj dataset rho tawm cov ntaub ntawv tuav tseg)
- Cov ntaub ntawv tuav pov hwm (piv txwv li ib pawg ntawm cov ntaub ntawv qub)
- Ib daim ntawv teev npe tsis qhia npe (raws li lub tsheb ciav hlau dataset)
- Synthetic dataset (raws li lub tsheb ciav hlau dataset)
Cov ntaub ntawv 1, 3 thiab 4 tau siv los cob qhia txhua tus qauv kev faib tawm, ua rau 12 (3 x 4) cov qauv kawm. SAS tom qab siv cov ntaub ntawv tuav pov hwm los ntsuas qhov tseeb uas txhua tus qauv kwv yees cov neeg siv khoom churn. Cov txiaj ntsig tau nthuav tawm hauv qab no, pib nrog qee qhov kev txheeb cais.
Daim duab: Machine Learning pipeline generated in SAS Visual Data Mining and Machine Learning
Cov txheeb cais yooj yim thaum sib piv cov ntaub ntawv tsis qhia npe rau cov ntaub ntawv qub
Anonymization cov txheej txheem rhuav tshem txawm tias cov qauv yooj yim, kev lag luam logic, kev sib raug zoo thiab kev txheeb cais (xws li piv txwv hauv qab no). Kev siv cov ntaub ntawv tsis qhia npe rau kev txheeb xyuas qhov yooj yim yog li tsim cov txiaj ntsig tsis ntseeg. Qhov tseeb, qhov tsis zoo ntawm cov ntaub ntawv tsis qhia npe ua rau nws yuav luag tsis tuaj yeem siv nws rau kev ua haujlwm siab tshaj plaws (xws li AI / ML qauv thiab dashboarding).
Cov txheeb cais yooj yim thaum sib piv cov ntaub ntawv hluavtaws nrog cov ntaub ntawv qub
Kev tsim cov ntaub ntawv hluavtaws nrog AI khaws cov qauv yooj yim, kev lag luam logic, kev sib raug zoo thiab kev txheeb cais (xws li piv txwv hauv qab no). Siv cov ntaub ntawv hluavtaws rau cov kev ntsuas yooj yim yog li tsim cov txiaj ntsig zoo. Cov lus nug tseem ceeb, cov ntaub ntawv hluavtaws puas muaj rau cov haujlwm analytics siab heev (xws li AI / ML qauv thiab dashboarding)?
AI-tsim cov ntaub ntawv hluavtaws thiab cov tshuaj ntsuam xyuas siab heev
Synthetic cov ntaub ntawv tuav tsis tau tsuas yog rau cov qauv yooj yim (raws li qhia nyob rau hauv lub yav dhau los zaj duab xis), nws kuj captures sib sib zog nqus 'pob ntseg' statistical qauv uas yuav tsum tau rau advanced analytics ua hauj lwm. Cov tom kawg tau pom nyob rau hauv daim ntawv qhia hauv qab no, qhia tias qhov tseeb ntawm cov qauv kawm ntawm cov ntaub ntawv hluavtaws piv rau cov qauv kawm ntawm cov ntaub ntawv qub zoo ib yam. Tsis tas li ntawd, nrog rau thaj tsam hauv qab ntawm qhov nkhaus (AUC*) ze rau 0.5, cov qauv kev cob qhia ntawm cov ntaub ntawv tsis qhia npe ua tau qhov phem tshaj plaws. Daim ntawv qhia tag nrho nrog rau tag nrho cov kev ntsuam xyuas siab heev ntawm cov ntaub ntawv hluavtaws sib piv nrog cov ntaub ntawv qub yog muaj nyob rau ntawm kev thov.
* AUC: thaj tsam hauv qab qhov nkhaus yog qhov ntsuas rau qhov tseeb ntawm cov qauv ntsuas siab tshaj plaws, suav nrog qhov tseeb qhov zoo, qhov tsis zoo, qhov tsis zoo tsis tseeb thiab qhov tsis zoo. 0,5 txhais tau hais tias tus qauv kwv yees randomly thiab tsis muaj lub zog kwv yees thiab 1 txhais tau hais tias tus qauv yeej ib txwm raug thiab muaj lub zog twv ua ntej.
Tsis tas li ntawd, cov ntaub ntawv hluavtaws no tuaj yeem siv los nkag siab cov ntaub ntawv cov yam ntxwv thiab cov hloov pauv tseem ceeb uas xav tau rau kev cob qhia tiag tiag ntawm cov qauv. Cov tswv yim xaiv los ntawm cov algorithms ntawm cov ntaub ntawv hluavtaws piv rau cov ntaub ntawv qub tau zoo sib xws. Li no, cov txheej txheem qauv tuaj yeem ua tiav ntawm cov khoom siv hluavtaws no, uas txo cov kev pheej hmoo ntawm cov ntaub ntawv ua txhaum cai. Txawm li cas los xij, thaum inferencing ib tug neeg cov ntaub ntawv (xws li telco neeg siv) retraining ntawm thawj cov ntaub ntawv yog pom zoo kom piav qhia, nce txais los yog vim hais tias ntawm kev cai.
AUC los ntawm Algorithm pawg los ntawm Txoj Kev
Cov lus xaus:
- Cov qauv kev cob qhia ntawm cov ntaub ntawv hluavtaws piv rau cov qauv kev cob qhia ntawm cov ntaub ntawv qub qhia tau tias muaj kev ua tau zoo heev
- Cov qauv kev cob qhia ntawm cov ntaub ntawv tsis qhia npe nrog 'cov txheej txheem tsis qhia npe classic' qhia kev ua haujlwm qis dua piv rau cov qauv kawm ntawm cov ntaub ntawv qub lossis cov ntaub ntawv hluavtaws
- Synthetic cov ntaub ntawv tsim yog ib qho yooj yim thiab ceev vim hais tias cov txheej txheem ua hauj lwm raws nraim tib yam ib dataset thiab ib hom ntaub ntawv.
Tus nqi-ntxiv cov ntaub ntawv hluavtaws siv cov ntaub ntawv
Siv cov ntaub ntawv 1: Synthetic cov ntaub ntawv rau kev tsim qauv thiab kev soj ntsuam siab heev
Muaj cov ntaub ntawv muaj zog nrog kev nkag tau yooj yim thiab nrawm rau kev siv tau, cov ntaub ntawv zoo yog qhov tseem ceeb los tsim cov qauv (xws li dashboards [BI] thiab cov tshuaj ntsuam xyuas qib siab [AI & ML]). Txawm li cas los xij, ntau lub koom haum raug kev txom nyem los ntawm cov ntaub ntawv zoo tshaj plaws uas ua rau muaj 3 qhov teeb meem tseem ceeb:
- Kev nkag mus rau cov ntaub ntawv siv sijhawm ntev vim yog (kev ceev ntiag tug) cov cai, txheej txheem sab hauv lossis cov ntaub ntawv silos
- Classic anonymization cov txheej txheem rhuav tshem cov ntaub ntawv, ua cov ntaub ntawv tsis tsim nyog rau kev tsom xam thiab kev txheeb xyuas qib siab (khoom khib nyiab hauv = khib nyiab tawm)
- Cov kev daws teeb meem uas twb muaj lawm tsis tuaj yeem ntsuas tau vim tias lawv ua haujlwm sib txawv ntawm cov ntaub ntawv thiab ib hom ntaub ntawv thiab tsis tuaj yeem tuav cov ntaub ntawv loj ntau lub rooj.
Synthetic cov ntaub ntawv mus kom ze: tsim qauv nrog li-zoo-zoo li-cov ntaub ntawv hluavtaws tiag tiag rau:
- Txo kev siv cov ntaub ntawv qub, tsis muaj kev cuam tshuam koj cov neeg tsim khoom
- Xauv cov ntaub ntawv tus kheej thiab muaj kev nkag tau mus rau cov ntaub ntawv ntau uas yav tas los txwv (piv txwv li vim yog kev ceev ntiag tug)
- Cov ntaub ntawv yooj yim thiab nrawm nkag mus rau cov ntaub ntawv cuam tshuam
- Scalable solution uas ua haujlwm zoo ib yam rau txhua cov ntaub ntawv, datatype thiab rau cov ntaub ntawv loj heev
Qhov no tso cai rau lub koom haum tsim kom muaj cov ntaub ntawv muaj zog nrog kev nkag tau yooj yim thiab nrawm rau cov ntaub ntawv siv tau, cov ntaub ntawv zoo los qhib cov ntaub ntawv thiab siv cov ntaub ntawv muaj peev xwm.
Siv cov ntaub ntawv 2: cov ntaub ntawv ntsuas hluavtaws ntse rau kev sim software, txhim kho thiab xa khoom
Kev sim thiab kev txhim kho nrog cov ntaub ntawv xeem zoo yog qhov tseem ceeb los xa cov kev daws teeb meem software hauv lub xeev. Siv cov ntaub ntawv tsim khoom qub zoo li pom tseeb, tab sis tsis tso cai vim (kev ceev ntiag tug) cov cai. Lwm txoj Test Data Management (TDM) cov cuab yeej qhia "legacy-by-design” nyob rau hauv tau txais cov ntaub ntawv xeem txoj cai:
- Tsis txhob cuam tshuam cov ntaub ntawv tsim khoom thiab kev lag luam logic thiab kev ncaj ncees referential tsis khaws cia
- Ua haujlwm qeeb thiab siv sijhawm
- Yuav tsum tau ua haujlwm ntawm tes
Synthetic cov ntaub ntawv mus kom ze: Ntsuam xyuas thiab txhim kho nrog AI-tsim cov ntaub ntawv hluavtaws sim kom xa cov kev daws teeb meem ntawm lub xeev-ntawm-the-art software ntse nrog:
- Cov ntaub ntawv zoo li ntau lawm nrog kev khaws cia ua lag luam logic thiab kev ncaj ncees referential
- Tsim cov ntaub ntawv yooj yim thiab nrawm nrog lub xeev-of-the-art AI
- Tsis pub twg paub-los-tsim
- Yooj yim, ceev thiab agile
Qhov no tso cai rau lub koom haum sim thiab txhim kho nrog cov ntaub ntawv xeem qib tom ntej kom xa cov kev daws teeb meem zoo tshaj plaws hauv lub xeev!
Xav paub ntau ntxiv
txaus siab? Yog xav paub ntxiv txog cov ntaub ntawv hluavtaws, mus saib ntawm Syntho lub vev xaib lossis hu rau Wim Kees Janssen. Yog xav paub ntxiv txog SAS, mus saib www.sas.com los yog hu rau kees@syntho.ai.
Hauv qhov kev siv no, Syntho, SAS thiab NL AIC ua haujlwm ua ke kom ua tiav cov txiaj ntsig tau npaj tseg. Syntho yog tus kws tshaj lij hauv AI-tsim cov ntaub ntawv hluavtaws thiab SAS yog tus thawj coj hauv kev lag luam hauv kev tshuaj xyuas thiab muab software rau kev tshawb nrhiav, tshuaj xyuas thiab pom cov ntaub ntawv.
* Kwv yees xyoo 2021 - Cov ntaub ntawv thiab cov tswv yim tsom xam rau kev tswj hwm, ntsuas thiab hloov pauv kev lag luam digital, Gartner, 2020.
Txuag koj cov ntaub ntawv qhia txog hluavtaws tam sim no!
- Cov ntaub ntawv hluavtaws yog dab tsi?
- Vim li cas cov koom haum siv nws?
- Tus nqi ntxiv cov ntaub ntawv hluavtaws cov neeg siv khoom
- Yuav pib li cas
