Purpose: The purpose of this study was to measure sensitivity of virtual reality (VR) in detecting biomarkers of neurovascular remodeling suboptimally evaluated in digital subtraction angiography (DSA) of treated unruptured, intracranial aneurysms.

Methods: The sensitivity of virtual reality and digital subtraction angiography in detection of neurovascular biomarkers in aneurysms treated with flow diversion, coiling, and clipping were evaluated. Validated grading scales were integrated into a standardized rating platform. The respective novel and conceptual measures of minimal imaging important difference (MIID) and number needed to image (NNI) were calculated for each biomarker.

Results: In flow diversion, coiling, and clipping, minimal imaging important difference and number needed to image were associated with virtual reality in detection of abnormal biomarkers, with the exception of stasis phase associated with digital subtraction angiography. Number needed to image was associated with flow diversion stent stenosis (RR: 7.00, 95% CI 0.37 to 131.97; OR: 7.46, 95% CI 0.38 to 148.49). Minimal imaging important difference was greatest in residual aneurysm filling (25%±66, 95% CI) in flow diversion and Meyer score in coiling (42.5%±17.69, 95% CI) and clipping (22.2%±13.58, 95% CI). Regression models demonstrated minimal imaging important difference and number needed to image shared a significant correlation (R²0.99, 95% CI, p<0.001).

Conclusion: Virtual reality adds value to digital subtraction angiography in evaluation of aneurysms treated with flow diversion, coiling, and clipping. Larger, prospective studies are warranted to increase statistical power and validate clinical significance.

Keywords: sphenoid bone, craniofacial abnormalities, malocclusion, megalencephaly, facial asymmetry, exophthalmos

Unruptured, intracranial aneurysms, have a prevalence of 3% worldwide, global incidence of 700,000 person-years, and rupture rate of 0.3% per year.¹ As neuroimaging advances, more asymptomatic aneurysms are detected incidentally, which does not correlate with increased risk of rupture.¹ Why then, is management of aneurysms a highly involved process warranting a multidisciplinary approach?

The reason is that while aneurysmal SAH has a mortality of 35%, treatment carries risk of poor neurological outcome in <1% to 10% of cases depending on treatment type and device generation, which may be higher than risk of rupture.^2,3 In addition to patient-specific modifiable and non-modifiable risk factors, the International Study of Unruptured Intracranial Aneurysms,⁴ American Heart Association/American Stroke Association guidelines,⁵ validated grading scales.^5,6 clinical trials,⁷ and meta-analyses⁶ have also focused on aneurysm-specific neuroimaging biometrics, such as location, size, and aspects ratios (dome: neck, height:neck). These biometrics are macroscopic results of microscopic neurovascular changes, or biomarkers, mainly studied in basic science research, but important in clinical decision-making.^8–11 And so, we looked at what translational tools exist to fill this unmet need in the clinical research setting using virtual reality (VR).¹² Currently, some neurovascular biomarkers in flow diverted, coiled, and clipped aneurysms are suboptimally evaluated with standard imaging and diagnostic tools making their validated clinical significance unknown.^13–19 The use of VR in treated aneurysms has not been explored.

DSA, the technology used to extract images from diagnostic cerebral angiogram, is the gold standard for surveillance of treated aneurysms. The purpose is to evaluate evolving characteristics of aneurysms at prospective intervals over time.^8,9 The clinician is then more adept at predicting clinical outcomes that guide management and understanding an aneurysm’s natural history, the cumulative aim of which is to develop novel diagnostic tools and biomarkers of prognostication.^{10,11,20–26}

Diagnostic limitations

In flow diversion, limitations of DSA are suboptimal visualization of stent structure and alignment, free-hanging edges, and stent stenosis.²⁷ In coiled aneurysms, complications may arise from aberrancies in coil-thrombus complex.^25,28 Predictive biomarkers include thrombus organization²⁹ coil packing³⁰ and coil orientation.³¹ In clipping of extradural, petrous or cavernous segment aneurysms, DSA is unable to assess bony erosion or peripheral nerve pathology.^32,33 In CTA, which is part of the surveillance algorithm, clip and bone artifact are limiting factors.^27,33

All data and materials have been made publicly available in Open Science Framework and can be accessed at https://osf.io/6kwyc/. VR reconstructions of unruptured, intracranial aneurysms treated with flow diversion,³⁴ coiling, and clipping from March 2010 to September 2018 at our university-affiliated, community-based comprehensive stroke center were retrospectively built using Surgical Theater³⁵ (Precision VR^® by Surgical Theater Inc., Cleveland, OH). Four institutional endovascular surgical neuroradiologists independently evaluated cases. Variables were inputted into a standardized rating platform to facilitate comparison of VR and DSA sensitivity in detecting biomarkers of neurovascular remodeling. Validated grading scales were integrated to increase intra- and interrater agreement and generalizability.

Validated grading of neurovascular biomarkers

Flow diversion cases were evaluated through adaptation of variables from the following validated grading scales: O’Kelly-Marotta,²⁰ SMART,³⁶ Kamran-Byrne,³⁷ and flow-diverting stent scale.³⁸ In our study, parameters of residual aneurysm filling were: none 0%, entry remnant <5%, subtotal 5-95%, and total >95%. Parameters of stasis phase were: arterial clearance before capillary phase, clearance before venous phase, or stagnation in venous phase. Parameters of branch filling defined by TICI³⁹ were: complete (TICI 3), partial I (TICI 2b >50%), partial II (TICI 2a <50%), or insufficient. Endoleak was identified as: absent or present. Free-hanging edge was identified as: absent, present without thrombus, or present with thrombus. Parameters of stent stenosis adapted from NASCET criteria40 were: minimal 0-29%, mild 30-49%, moderate 50-69%, severe 70-94%, critical 95-99%, or occluded 100%. Coiling cases were evaluated using variables adapted from modified Raymond Roy classification⁴¹ (class I: complete obliteration, class II: residual neck, class IIIa: residual aneurysm with contrast opacification within coil, class IIIb: contrast opacification outside coil) and Meyer scale⁴² (grade 1 > or = 90%, grade 2 70–89%, grade 3 50–69%, grade 4 26–49%, or grade 5 < or = 25% occlusion; modifier I: coil opacification; modifier G: independent aneurysm growth). Clipping cases were evaluated using variables adapted from Raymond Roy occlusion classification⁴¹ (class I: complete obliteration, class II: residual neck, class III: residual aneurysm) and Meyer scale. Qualitative information was included, if applicable.

360° VR model rendering and manipulation

The VR model can be manipulated in any axis or magnitude to examine global neuroanatomy and neuropathology. VR software controls scan intensities to visualize selected systems and facilitate treatment planning for neurovascular conditions.¹² This technology is also able to highlight specific components in flow diverters, coils, and clips as well as clip-in through any aspect of aneurysm or device.

Each aneurysm was reconstructed as a 360°VR model using patient-specific CT volumetric scans (3DCT) derived from DSA. Pre-contrast images, or mask,^43,44 and contrast-enhanced images that subtract the mask, or sub, were the two sequences used. These sequences are established protocols in our institution’s neuroimaging department. There are specific resolution parameters to optimize scans for VR reconstruction. The dimensions are 512, 512, 502 on an x, y, z plane with 0.25mm spacing per slice collecting 498 to 502 files per sequence. Sub and mask 3DCTs were loaded into VR software to render the 360°VR model, which was later edited to highlight critical components. For each case, sub and mask 3DCTs from the most recent DSA were merged.

The sub sequence only showed vessels from DSA. Specific digital presets were created to show parent vessels and distal branches around aneurysms. The mask sequence showed bone and other radiopaque elements, such as metal in flow diverters (cobalt-chromium and platinum-tungsten wires), coils (platinum), and clips (titanium or Phynox^® alloy⁴⁵). A mesh generation tool produced a 3D object of each device using high-intensity variations per voxel from mask sequence. This extraction allowed raters to interact with devices dependently and independently of sub sequence.

Data analysis

Relative risk (RR)^46,47 (95% CI) and OR^{46, 47} (95% CI) for each neurovascular biomarker was determined. Haldane-Anscombe correction was used for zero values in contingency tables,⁴⁸ if applicable. Pearson’s coefficient was used to measure association of RR and OR size with CI width.⁴⁹

The novel concept of minimal imaging important difference was used to define the smallest amount a neuroimaging finding changed to be meaningful. Sensitivity of VR in detecting a neuroimaging abnormality to a greater degree than DSA was defined as a meaningful difference. The diagnostic relevance of minimal imaging important difference was based on principles of clinical relevance in minimal clinically important difference.^50,51 The conceptual measure of number needed to image⁵² (NNI) was defined as number of aneurysms with VR needed to detect an abnormality to a greater degree than DSA. The diagnostic relevance of NNI was based on principles of clinical relevance in number needed to treat.^52,53 Minimal imaging important difference was expressed as a percentage with associated margin of error (95% CI).

In NNI, to account for potentially insignificant CI given dependence on significant absolute risk reduction, CI were expressed as: NNI for an additional harmful outcome to ∞ to NNI for an additional beneficial outcome. In this method, CI ranges from NNI (harm)=-1 to NNI (benefit)=1 via infinity, whereby these are theoretically unattainable values (absolute risk reduction of -100% and 100%, respectively) and NNI=∞ represents midpoint (absolute risk reduction=0%).⁵⁴

All neurovascular biomarkers were analyzed weighted and unweighted. For example, in flow diversion cases, normal designation (i.e. TICI 3) was assigned weight of 1, TICI 2b weight of 2, TICI 2a weight of 3, and insufficient weight of 4. Cases were measured unweighted in a dichotomous manner (TICI 3 versus non-TICI 3) and weighted (TICI 3 versus summation of non-TICI 3) for minimal imaging important difference and NNI. Two-tailed t-test was performed to evaluate for significant differences between unweighted and weighted samples.^46,47,49

Correlation scatter models were made using ordinary least-squares regression⁵⁵ (quadratic regression equation: y = a + bx +cx², 95% confidence and probability) where R, or coefficient of determination, determines strength of association between x (minimal imaging important difference) and y (NNI) and variability in y (NNI) explained by model in weighted and unweighted analysis. Outlier detection was performed prior to regression analysis to account for heterogeneity and limited sample size. Statistical power was calculated using dichotomous endpoint, two-independent sample study model⁴⁹. Statistical software used was MedCalc Software Ltd.⁵⁶

From March 2010 to September 2018, 86 flow diversion, 62 coiling, and 96 clipping cases for treatment of unruptured, intracranial aneurysms were performed. Of these cases, 48 flow diversion, 15 coiling, and 18 clipping cases met technical parameters required to build a VR model.

Measure of association between VR and neuroimaging abnormality, or OR (Table 1), and probability of VR detecting neuroimaging abnormality, or RR (Table 2), were greater than 1 in all neurovascular biomarkers (n=10) and greatest in stent stenosis (RR: 7.00, 95% CI 0.40 to 132.00; OR: 7.50, 95% CI 0.40 to 149.00). Wide CI shared a positive correlation with high RR (Pearson correlation coefficient 0.98, p<0.0001, 95% CI 0.90 to 0.99) and high OR (Pearson correlation coefficient 0.97, p<0.0001, 95% CI 0.89 to 0.99).

Footrnotes:
¹Expressed as percentage with margin of error

²OR <1 and negative minimal imaging important difference represent greater association with DSA

³Modified Raymond Roy classification

Footnotes:
¹Rounded to nearest whole number

²RR <1 and negative NNI represent higher probability in DSA

³Modified Raymond Roy classification

Figures 1 and 2 provided visual representation of aberrant neurovascular biomarkers in VR, which were quantified by minimal imaging important difference (Table 1) and NNI (Table 2). In NNI, CI included infinity with the exception of weighted stent stenosis in VR that had significant absolute risk reduction (11.76%, 95% CI 2.35-21.18). There were significant differences between unweighted and weighted minimal imaging important difference (test statistic t=3.70, P=0.006) and NNI (test statistic t=-2.29, P=0.05).

Figure 1 Biomarker detection in flow diversion. VR model of treated paraophthalmic aneurysm with endoleak (A) is not visualized on DSA (B). VR shows ophthalmic artery branch filling (C) not visualized on DSA (D).

Figure 2 Device abnormalities detected in VR. Irregularly-shaped coil of treated posterior communicating artery aneurysm (A) protrudes into parent vessel increasing coil herniation risk. Stent-stenosis of flow diverted paraophthalmic aneurysm (B) is shown. Residual neck of clipped PICA aneurysm in VR (C) is difficult to visualize on DSA (D).

Scatterplot regression model (Figure 3) demonstrated NNI (n=9) and minimal imaging important difference (n=9) had significant correlation (unweighted: R² 0.99, 95% CI, p<0.001) and variability in y (NNI) was explained by model. In weighted analysis (Figure 4), minimal imaging important difference was greater and NNI lower in the same biomarkers (weighted: R² 0.92, 95% CI, p<0.001). Outlier detection did not show significant outliers (NNI: coefficient of skewness 1.27, P=0.08; coefficient of Kurtosis 0.20, P=0.74) (minimal imaging important difference: coefficient of skewness 0.46, P=0.51; coefficient of Kurtosis -1.50, P=0.21). Subgroup analysis of flow diversion, coiling, and clipping was not performed to prevent bias given small sample size and study design limitations. Of note, stasis phase was the only biomarker more strongly associated with DSA (OR 0.04, 95% CI 0.002 to 0.70; RR 0.05, 95% CI 0.002 to 0.80). Negative minimal imaging important difference and NNI for an additional harmful outcome were reported. Values were included in all analyses except regression model.

Figure 3 Regression analysis of effect of minimal imaging important difference on NNI in unweighted analysis.

Figure 4 Regression analysis of effect of minimal imaging important difference on NNI in weighted analysis.

To our knowledge, this is the first study comparing VR to DSA in evaluation of unruptured, intracranial aneurysms following flow diversion, coiling, and clipping. Results suggested a conservative degree of internal validity in VR conferring an advantage in detection of neuroimaging abnormalities representative of aberrant neurovascular remodeling. These limitations are the result of small sample size with suboptimal precision and generalizability. A modest estimation of minimal sample size (alpha type I error: 0.05, beta type II error: power 80%) assuming small effect size (5% increased incidence) of VR in detecting a neuroimaging abnormality was 304. Prospective, multicenter studies and concurrent advances in existing technologies, standardization protocols, and resource allocation to reliably meet parameters to build a VR model from existing cases are needed.

Clinical relevance of statistical significance

Our study proposed a novel concept: minimal imaging important difference based on minimal clinically important difference, which prioritizes objective measures of beneficence, rather than statistical distribution-based approaches⁵⁷ The conceptual measure of NNI⁵² was evaluated based on number needed to treat, a type of effect size⁵⁸ vital to interpretation of results associated with, yet independent of, statistical significance.

There were challenges in determining clinical significance in neuroimaging-dependent measures of minimal imaging important difference and NNI as compared to minimal clinically important difference and number needed to treat. As such, we attempted to test if a statistically significant association existed between minimal imaging important difference and NNI that would strengthen their independent reliability and synergistically validate their integration into diagnostic and predictive tools with external validity. Correlation and regression models suggested a significant association existed between minimal imaging important difference and NNI (unweighted: R² 0.99, weighted: R² 0.92), whereby values nearing R=1 or 100% of variance theoretically presume fitted values always equal observed values and all data points fall on a fitted regression line. However, residual plots of regression models showed marginal bias in unweighted analysis and marked bias in weighted analysis. We posit that further validation and rigorous critical appraisal of minimal imaging important difference and NNI as interdependent variables is warranted to potentially reveal a symbiotic relationship that achieves statistical significance. The redistribution of weight assignments in weighted analysis is also needed.

It follows that although neuroimaging biomarkers, both aberrant and adaptive, reflect a discrete stage of the neurovascular remodeling process, their dynamic interplay is unknown. Validated grading scales in this study were adapted as static measures of clinical significance for abnormalities that reflect these dynamic processes. In comparison to coiling and clipping, flow diversion is a more recent technology undergoing rapid evolution of device design, indications for use, and, for our purposes, grading scale development.^20,36–38 As such, grading scales that integrate principles of hemodynamics and endothelialization in aneurysmal thrombosis into standard biometrics will continue to be redefined to accurately and reliably reflect clinical significance.⁵⁹

Future directions of virtual reality

Although biomarkers suboptimally evaluated on DSA were appreciable in VR, the technology was unable to identify other fundamental biomarkers like organization of thrombus-coil complex. Presently, neither neuroscience-based VR nor standard tools are designed to evaluate histological and molecular components that are, in effect, the primordial soup of the remodeling process.^60–62 Pathology-based augmented-reality models⁶³ in the pipeline facilitate high-resolution imaging, microscopic evaluation, and real-time pathology-radiology correlations. A merger of these models may be the interface between mechanistic, proof-of-concept data and technologies oriented towards imaging, diagnostic, and clinical domains. Although promising, clinical equipoise is necessary as these technologies advance in breadth and scope. Moreover, from a practical standpoint, it is important to be mindful of logistics and cost-effectiveness of VR.^52,64 The further development of validated measures of clinical significance, such as those presented, are of value in justifying widespread implementation of VR.

Virtual reality serves an adjunctive role in enhancing the sensitivity of DSA in detecting aberrant biomarkers of neurovascular remodeling in treated unruptured, intracranial aneurysms. The perspective from which our study results are examined should be prefaced by a closing remark. In domains that are clinically unexplored such as the role of VR in the present study, compelling but principled arguments that aim to explain rather than quantify observations are part of the natural history of clinically relevant, statistically significant findings.^65–69 By building upon recent preliminary findings,^66–69 a limited technology investigating evidence-based, yet clinically non-validated, neurovascular biomarkers with a proposed standardized methodology has been able to provide an exploratory answer to an interesting clinical question. Inherent statistical limitations in magnitude of effect, articulation of precision, and generalizability of results were partially mitigated by applied importance and provisional credibility based on theoretical coherence, methodological soundness, and results of limited statistical and yet to be determined clinical significance.^70–73

Conflict of interest: The authors declare that they have no conflict of interest.

Informed consent: Informed consent was obtained from all individual participants included in the study.

Ethical approval: All procedures performed in the studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Acknowledgments: None

Funding: No funding was received for this study.

1. Hackenberg K, Hänggi D, Etminan N. Unruptured intracranial aneurysms. Stroke. 2018;49(9):2268–2275.
2. Thompson BG, Brown RD, Amin-Hanjani S, et al. Guidelines for management of patients with unruptured intracranial aneurysms. Stroke. 2015;46(8):2368–2400.
3. Bhatia KD, Kortman H, Orru E, et al. Periprocedural complications of second-generation flow diverter treatment using Pipeline Flex for unruptured intracranial aneurysms: systematic review and meta-analysis. J Neurointerv Surg. 2019;11(8):817–824.
4. Wiebers DO, Whisnant JP, Huston J, et al. Unruptured intracranial aneurysms: natural history, clinical outcome, risks of surgical and endovascular treatment. Lancet. 2003;362(9378):103–110.
5. Backes D, Rinkel GJE, Greving JP, et al. ELAPSS score for prediction of risk of growth of unruptured intracranial aneurysms. Neurology. 2017;88(17):1600–1606.
6. Greving JP, Wermer MJH, Brown RD, et al. Development of PHASES score for prediction of risk of rupture of intracranial aneurysms: pooled analysis of six prospective cohort studies. Lancet. 2014;13(1):59–66.
7. ClinicalTrials.gov. Bethesda, MD: National Library of Medicine; 2017. Identifier NCT04286503. Acetylsalicylic acid plus intensive blood pressure treatment in patients with unruptured intracranial aneurysms (PROTECT-U). Updated January 25, 2020.
8. Carroll CP, Tackla RD, Vuong SM, et al. Surveillance and Screening for Intracranial Aneurysms. In: Ringer AJ. Intracranial Aneurysms. Cambridge, MA: Academic Press; 2018.
9. Stafa A, Leonardi M. Role of neuroradiology in evaluating cerebral aneurysms. Interv Neuroradiol. 2008;14 Suppl 1:23–37.
10. Meng H, Tutino VM, Xiang J, et al. High WSS or low WSS? Complex interactions of hemodynamics with intracranial aneurysm initiation, growth, and rupture: toward a unifying hypothesis. AJNR Am J Neuroradiol. 2014;35(7):1254–1262.
11. Castro MA. Understanding the role of hemodynamics in the initiation, progression, rupture, and treatment outcome of cerebral aneurysm from medical image-based computational studies. ISRN Radiol. 2013;2013:602707.
12. Chugh AJ, Jonathan RP, Justin S, et al. Use of a surgical rehearsal platform and improvement in aneurysm clipping measures: results of prospective, randomized trial. J Neurosurg. 2017;126(3):838–844.
13. Cebral JR, Mut F, Raschi M, et al. Analysis of hemodynamics and aneurysm occlusion after flow-diverting treatment in rabbit models. AJNR Am J Neuroradiol. 2014;35(8):1567–1573.
14. Byrne G, Mut F, Cebral J. Quantifying the large-scale hemodynamics of intracranial aneurysms. AJNR Am J Neuroradiol. 2014;35(2):333–338.
15. Bhogal P, Ganslandt O, Bäzner H, et al. The fate of side branches covered by flow diverters–results from 140 patients. World Neurosurg. 2017;103:789–798.
16. Zhu Y-Q, Li M-H, Lin F, et al. Frequency and predictors of endoleaks and long-term patency after covered stent placement for treatment of intracranial aneurysms: prospective, non-randomised multicentre experience. Euro Radiol. 2013;23(1):287–297.
17. Jagadeesan BD, Sandhu D, Hong KJ, et al. Salvage of herniated flow diverters using stent and balloon anchoring techniques: a technical note. Interv Neurol. 2017;6(1-2):31–35.
18. Darsaut TE, Bing F, Makoyeva A, et al. Flow diversion to treat aneurysms: the free segment of stent. J Neurointerv Surg. 2013;5(5):452–457.
19. Ravindran K, Salem MM, Enriquez-Marulanda A, et al. Quantitative assessment of in-stent stenosis after Pipeline embolization device treatment of intracranial aneurysms: single-institution series and systematic review. World Neurosurg. 2018;120:e1031–e1040.
20. O'Kelly CJ, Krings T, Fiorella D, et al. A novel grading scale for angiographic assessment of intracranial aneurysms treated using flow diverting stents. Interv Neuroradiol. 2010;16(2):133–137.
21. Mascitelli JR, Moyle H, Oermann EK, et al. An update to Raymond–Roy occlusion classification of intracranial aneurysms treated with coil embolization. J Neurointerv Surg. 2015;7(7):496–502.
22. Luo CB, Chang FC, Teng MMH, et al. Stent management of coil herniation in embolization of internal carotid aneurysms. AJNR Am J Neuroradiol. 2008;29(10):1951–1955.
23. Kim BM, Park SI, Kim DJ, et al. Endovascular coil embolization of aneurysms with branch incorporated into the sac. AJNR Am J Neuroradiol. 2010;31:145–151.
24. Killer-Oberpfalzer M, Aichholzer M, Weis S, et al. Histological analysis of clipped human intracranial aneurysms and parent arteries with short-term follow-up. Cardiovasc Pathol. 2012;21(4):299–306.
25. Brinjikji W, Kallmes DF, Kadirvel R. Mechanisms of healing in coiled intracranial aneurysms: review of the literature. AJNR Am J Neuroradiol. 2015;36(7):1216–1222.
26. Costalat V, Sanchez M, Ambard D, et al. Biomechanical wall properties of human intracranial aneurysms resected following surgical clipping. J Biomech. 2011;44(15):2685–2691.
27. Soize S, Gawlitza M, Raoult H, et al. Imaging follow-up of intracranial aneurysms treated by endovascular means. Stroke. 2016;47(5):1407–1412.
28. Fujimura S, Takao H, Suzuki T, et al. Hemodynamics and coil distribution with changing coil stiffness and length in intracranial aneurysms. J Neurointerv Surg. 2018;10(8):797–801.
29. Haworth KJ, Weidner CR, Abruzzo TA, et al. Mechanical properties and fibrin characteristics of endovascular coil–clot complexes: relevance to endovascular cerebral aneurysm repair paradigms. J Neurointerv Surg. 2015;7(4):291–296.
30. Otani T, Shindo T, Ii S, et al. Effect of local coil density on blood flow stagnation in densely coiled cerebral aneurysms: computational study using Cartesian grid method. J Biomech Eng. 2018;140(4):041013.
31. Schirmer CM, Malek AM. Critical influence of framing coil orientation on intra-aneurysmal and neck region hemodynamics in sidewall aneurysm model. Neurosurgery. 2010;67(6):1692–1702.
32. James KL, Oren NG, Amin A, et al. Aneurysms of petrous internal carotid artery: anatomy, origins, and treatment. Neurosurg Focus. 2004;17(5):1–9.
33. Christopher SE, Michael CH, Bernard RB, et al. Cavernous carotid aneurysms: to treat or not to treat? Neurosurg Focus. 2009;26(5):E4.
34. Pipeline™ Embolization/Flex Embolization Device. Micro Therapeutics Inc. d/b/a Medtronic© subsidiary: ev3 Neurovascular. Irvine, CA. Food and Drug Administration pre-market approval April 2011-December 2019.
35. Surgical Theater© Precision Virtual Reality®. Cleveland, OH.
36. Grunwald IQ, Kamran M, Corkill RA, et al. Simple measurement of aneurysm residual after treatment: SMART scale for evaluation of intracranial aneurysms treated with flow diverters. Acta Neurochir. 2012;154(1):21–26.
37. Kamran M, Yarnold J, Grunwald IQ, et al. Assessment of angiographic outcomes after flow diversion treatment of intracranial aneurysms: a new grading schema. Neuroradiology. 2011;53(7):501–508.
38. Park MS, Mazur MD, Moon K, et al. An outcomes-based grading scale for evaluation of cerebral aneurysms treated with flow diversion. J Neurointerv Surg. 2017;9(11):1060–1063.
39. Zaidat Osama O, Yoo AJ, Khatri P, et al. Recommendations on angiographic revascularization grading standards in ischemic stroke. Stroke. 2013;44(9):2650–2663.
40. Barnett HJM, Taylor DW, Haynes RB, et al. North American symptomatic carotid endarterectomy trial: beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. New Engl J Med. 1991;325:445–453.
41. Stapleton CJ, Torok CM, Rabinov JD, et al. Validation of modified Raymond–Roy classification for intracranial aneurysms treated with coil embolization. J Neurointerv Surg. 2016;8(9):927–933.
42. Rouchaud A, Brinjikji W, Gunderson T, et al. Validity of Meyer scale for assessment of coiled aneurysms and aneurysm recurrence. AJNR Am J Neuroradiol. 2016;37(5):844-848.
43. Gyánó M, Góg I, Óriás VI, et al. Kinetic imaging in lower extremity arteriography: comparison to digital subtraction angiography. Radiology. 2018;290(1):246–253.
44. Khajuria R, Gross BA, Du R. Image-Guided Open Cerebrovascular Surgery. Golby AJ. In: Image-Guided Neurosurgery. Boston, MA: Academic Press; 2015.
45. Aneurysm Clips for Neurosurgery. Aesculap®.
46. Shafer DS, Zhang Z. Introductory Statistics. Washington DC: Saylor Foundation; 2012.
47. Norton EC, Dowd BE, Maciejewski ML. Odds ratios current best practice and use. JAMA. 2018;320(1):84–85.
48. Ruxton GD, Neuhäuser M. Review of alternative approaches to calculation of confidence interval for odds ratio of 2 × 2 contingency table. Methods Ecol Evol. 2013;4:9–13.
49. Rosner B. Fundamentals of Biostatistics. Boston, MA; Cengage Learning: 2015.
50. McGlothlin AE, Lewis RJ. Minimal clinically important difference: defining what really matters to patients. JAMA. 2014;312(13):1342–1343.
51. Cranston JS, Kaplan BD, Saver JL. Minimal clinically important difference for safe and simple novel acute ischemic stroke therapies. Stroke. 2017;48(11):2946–2951.
52. Haller S. The concept of number needed to image. AJNR Am J Neuroradiol. 2017;38(10):E79–E80.
53. Saver JL, Lewis RJ. Number needed to treat: conveying the likelihood of a therapeutic effect. JAMA. 2019;321(8):798–799.
54. Altman DG. Confidence intervals for number needed to treat. BMJ. 1998;317(7200):1309–1312.
55. Bath PMW, Lees KR, Schellinger PD, et al. Statistical analysis of primary outcome in acute stroke trials. Stroke. 2012;43(4):1171–1178.
56. Medcalc® Easy-to-Use Statistical Software. 2020© MedCalc Software Ltd.
57. Bernstein JA, Mauger DT. The minimally clinically important difference (MCID): what difference does it make? J Allergy Clin Immunol. 2016;4(4):689–690.
58. Laupacis A, Sackett DL, Roberts RS. An assessment of clinically useful measures of consequences of treatment. New Engl J Med. 1988;318(26):1728–1733.
59. Kirmani JF, Fourcand F, Kulhari A, et al. Abstract wp129: The HEAT grading scale as a neurovascular remodeling based score in aneurysms treated with flow diversion. Stroke.51:AWP129-AWP129.
60. Wang DS, Dake MD, Park JM, et al. Molecular imaging: primer for interventionalists and imagers. J Vasc Interv Radiol. 2006;17(9):1405–1423.
61. Usami Y, Hirokawa N, Saitoh M, et al. Histopathological differences of experimental aneurysms treated with bare platinum, fibered, and bioactive coils. Minim Invasive Ther Allied Technol. 2019;28(3):172–177.
62. Stiver SI, Porter PJ, Willinsky RA, et al. Acute human histopathology of intracranial aneurysm treated using Guglielmi detachable coils: case report and review of the literature. Neurosurgery. 1998;43(5):1203–1207.
63. Hanna MG, Ahmed I, Nine J, et al. Augmented reality technology using Microsoft Hololens in anatomic pathology. Arch Pathol Lab Med. 2018;142(5):638–644.
64. WHO. Cost-Effectiveness Analysis for Health Interventions. World Health Organization. 2020
65. Abelson RP. Statistics as Principled Argument. New York, NY: Taylor & Francis Group; 1995.
66. Ellis PD. The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and Interpretation of Research Results. New York, NY: Cambridge University Press; 2010.
67. Lipsey MW. Translating Statistical Representation of Effects of Education Interventions Into More Readily Interpretable Forms. Washington, DC. National Center for Special Education Research. November 2012.
68. Sawilowsky SS. New effect size rules of thumb. J Mod Appl Stat Methods. 2009;8(2):597–599.
69. Cohen J. Statistical Power Analysis for Behavioral Sciences. New York, NY: Routledge; 2013.
70. Fourcand F, Kulhari A, Singh A, et al. Validation of Surgical Theater in evaluating residual aneurysm after coiling. Society of Vascular and Interventional Neurology 2019 Annual Meeting. Atlanta, GA. November 20-23, 2019.
71. Fourcand F, Barbery D, Kulhari A, et al. Does virtual reality add value to DSA in analyzing aneurysms post Pipeline, coiling and clipping? Society of Vascular and Interventional Neurology 2019 Annual Meeting. Atlanta, GA. November 20-23, 2019.
72. Fourcand F, Kulhari A, Singh A, et al. Validation of Surgical Theater in evaluating aneurysms after Pipeline flow diversion. Society of Vascular and Interventional Neurology 2019 Annual Meeting. Atlanta, GA. November 20-23, 2019.
73. Shaikh A, Barbery D, Kulhari A, et al. Validation of Surgical Theater in evaluating aneurysms after clipping. Society of Vascular and Interventional Neurology 2019 Annual Meeting. Atlanta, GA. November 20-23, 2019.