Quantifying Operator Subjectivity within Flow Cytometry Data Analysis as a Source of Measurement Uncertainty and the Impact of Experience on Results

References

1. 1.↵
   1. Williams D. J.,
   2. Thomas R. J.,
   3. Hourd P. C.,
   4. Chandra A.,
   5. Ratcliffe E.,
   6. Liu Y.,
   7. Rayment E. A.,
   8. Archer J. R.

   Precision Manufacturing for Clinical-Quality Regenerative Medicines. Philos. Trans. R. Soc., A. 2012, 370 (1973), 3924–3949.

   OpenUrlCrossRefPubMed
2. 2.↵
   1. Hourd P.,
   2. Williams D. J.

   Scanning the Horizon for High Value-Add Manufacturing Science: Accelerating Manufacturing Readiness for the Next Generation of Disruptive, High-Value Curative Cell Therapeutics. Cytotherapy 2018, 20 (5), 759–767.

   OpenUrl
3. 3.↵
   1. Harrison R. P.,
   2. Ruck S.,
   3. Medcalf N.,
   4. Rafiq Q. A.

   Decentralized Manufacturing of Cell and Gene Therapies: Overcoming Challenges and Identifying Opportunities. Cytotherapy 2017, 19 (10), 1140–1151.

   OpenUrl
4. 4.↵
   1. Thurman-Newell J. A.,
   2. Petzing J. N.,
   3. Williams D. J.

   Quantification of Biological Variation in Blood-Based Therapy - a Summary of a Meta-Analysis to Inform Manufacturing in the Clinic. Vox Sang. 2015, 109 (4), 394–402.

   OpenUrl
5. 5.↵
   1. Thurman-Newell J. A.,
   2. Petzing J. N.,
   3. Williams D. A.

   Meta-Analysis of Biological Variation in Blood-Based Therapy as a Precursor to Bio-Manufacturing. Cytotherapy 2016, 18 (5), 686–694.

   OpenUrl
6. 6.↵
   1. Thurman-Newell J.,
   2. Petzing J.,
   3. Williams D.

   Analysis of Biological Variation in Blood-Based Therapy as a Precursor to Biomanufacturing. Presented at the MIT Center for Biomedical Innovation’s 2015 BioMAN Summit, Boston, MA, Nov 18–19, 2015.
7. 7.↵
   1. Wang L.,
   2. Hoffman R. A.

   Standardization, Calibration, and Control in Flow Cytometry. Curr. Protoc. Cytom. 2017, 32 (1), 1.3.1–1.3.27.

   OpenUrl
8. 8.↵
   1. van Dongen J. J. M.,
   2. Lhermitte L.,
   3. Böttcher S.,
   4. Almeida J.,
   5. van der Velden V. H. J.,
   6. Flores-Montero J.,
   7. Rawstron A.,
   8. Asnafi V.,
   9. Lécrevisse Q.,
   10. Lucio P.,
   11. Mejstrikova E.,
   12. Szczepański T.,
   13. Kalina T.,
   14. de Tute R.,
   15. Brüggemann M.,
   16. Sedek L.,
   17. Cullen M.,
   18. Langerak A. W.,
   19. Mendonça A.,
   20. Macintyre E.,
   21. Martin-Ayuso M.,
   22. Hrusak O.,
   23. Vidriales M. B.,
   24. Orfao A

   , on behalf of the EuroFlow Consortium (EU-FP6, LSHB-CT-2006-018708). EuroFlow Antibody Panels for Standardized n-Dimensional Flow Cytometric Immunophenotyping of Normal, Reactive and Malignant Leukocytes. Leukemia 2012, 26 (9), 1908–1975.

   OpenUrlCrossRefPubMedWeb of Science
9. 9.↵
   1. Kalina T.,
   2. Flores-Montero J.,
   3. van der Velden V. H. J.,
   4. Martin-Ayuso M.,
   5. Böttcher S.,
   6. Ritgen M.,
   7. Almeida J.,
   8. Lhermitte L.,
   9. Asnafi V.,
   10. Mendonça A.,
   11. de Tute R.,
   12. Cullen M.,
   13. Sedek L.,
   14. Vidriales M. B.,
   15. Pérez J. J.,
   16. Te Marvelde J. G.,
   17. Mejstrikova E.,
   18. Hrusak O.,
   19. Szczepański T.,
   20. van Dongen J. J. M.,
   21. Orfao A

   , on behalf of the EuroFlow Consortium (EU-FP6, LSHB-CT-2006-018708). EuroFlow Standardization of Flow Cytometer Instrument Settings and Immunophenotyping Protocols. Leukemia 2012, 26 (9), 1986–2010.

   OpenUrlCrossRefPubMedWeb of Science
10. 10.↵
    1. Brando B.,
    2. Gatti A.,
    3. Chianese R.,
    4. Gratama J. W.

    Twenty Years of External Quality Assurance in Clinical Cell analysis - A Tribute to Jean-Luc D′Hautcourt. Cytometry, Part B 2007, 72B (1), 2–7.

    OpenUrl
11. 11.↵
    1. Whitby L.,
    2. Whitby A.,
    3. Fletcher M.,
    4. Barnett D.

    Current Laboratory Practices in Flow Cytometry for the Enumeration of CD 4+ T-Lymphocyte Subsets. Cytometry, Part B 2015, 88 (5), 305–311.

    OpenUrl
12. 12.↵
    1. Bainbridge J.,
    2. Rountree W.,
    3. Louzao R.,
    4. Wong J.,
    5. Whitby L.,
    6. Denny T. N.,
    7. Barnett D.

    Laboratory Accuracy Improvement in the UK NEQAS Leucocyte Immunophenotyping Immune Monitoring Program: An Eleven-Year Review via Longitudinal Mixed Effects Modeling. Cytometry, Part B 2018, 94 (2), 250–256.

    OpenUrl
13. 13.↵
    1. Barnett D.,
    2. Granger V.,
    3. Storie I.,
    4. Peel J.,
    5. Pollitt R.,
    6. Smart T.,
    7. Reilly J. T.

    Quality Assessment of CD34+ Stem Cell Enumeration: Experience of the United Kingdom National External Quality Assessment Scheme (UK NEQAS) Using a Unique Stable Whole Blood Preparation. Br. J. Haematol. 1998, 102 (2), 553–565.

    OpenUrlCrossRefPubMedWeb of Science
14. 14.↵
    UK NEQAS. Flow Cytometry Programmes, 2020. http://www.ukneqasli.co.uk/eqa-pt-programmes/flow-cytometry-programmes/ (accessed March 10, 2020).
15. 15.↵
    International Conference for Harmonisation, Quality Guideline Q7: Good Manufacturing Practice Guide for Active Pharmaceutical Ingredients. ICH: Geneva, 2000.
16. 16.↵
    1. Haeckel R.,
    2. Wosniok W.,
    3. Gurr E.,
    4. Peil B.

    Permissible Limits for Uncertainty of Measurement in Laboratory Medicine. Clin. Chem. Lab. Med. 2015, 53 (8), 1161–1171. doi:10.1515/cclm-2014-0874.

    OpenUrl
17. 17.↵
    1. Grant R.,
    2. Coopman K.,
    3. Medcalf N.,
    4. Silva-Gomes S.,
    5. Campbell J. J.,
    6. Kara B.,
    7. Braybrook J.,
    8. Petzing J.

    Understanding the Contribution of Operator Measurement Variability within Flow Cytometry Data Analysis for Quality Control of Cell and Gene Therapy Manufacturing. Measurement 2020, 150, 106998. doi:10.1016/j.measurement.2019.106998.

    OpenUrl
18. 18.↵
    1. O'Neill K.,
    2. Aghaeepour N.,
    3. Špidlen J.,
    4. Brinkman R.

    Flow Cytometry Bioinformatics. PLoS Comput. Biol. 2013, 9 (12), e1003365–10. doi:10.1371/journal.pcbi.1003365.

    OpenUrlCrossRefPubMed
19. 19.↵
    1. Mitsialis V.,
    2. Shouval D.,
    3. Wall S.,
    4. Snapper S. B.,
    5. Konnikova L.

    High Dimensional Immune Phenotyping Using Mass Cytometry Reveals Unique IL1B Signatures in Peripheral Blood and GI Tissue of Ulcerative Colitis, Crohn′s Disease and IL10R Deficient Patients. Gastroenterology 2018, 154 (6 Suppl. 1), S-90–S-91.

    OpenUrl
20. 20.↵
    1. Behbehani G. K.

    Applications of Mass Cytometry in Clinical Medicine: The Promise and Perils of CyTOF. Clin. Lab. Med. 2017, 37 (4), 945–964.

    OpenUrl
21. 21.↵
    1. Leipold M. D.,
    2. Obermoser G.,
    3. Fenwick C.,
    4. Kleinstuber K.,
    5. Rashidi N.,
    6. McNevin J. P.,
    7. Nau A. N.,
    8. Wagar L. E.,
    9. Rozot V.,
    10. Davis M. M.,
    11. DeRosa S.,
    12. Pantaleo G.,
    13. Scriba T. J.,
    14. Walker B. D.,
    15. Olsen L. R.,
    16. Maecker H. T.

    Comparison of CyTOF Assays across Sites: Results of a Six-Center Pilot Study. J. Immunol. Methods 2018, 453, 37–43.

    OpenUrlCrossRef
22. 22.↵
    1. Bruggner R. V.,
    2. Bodenmiller B.,
    3. Dill D. L.,
    4. Tibshirani R. J.,
    5. Nolan G. P.

    Automated Identification of Stratifying Signatures in Cellular Subpopulations. Proc. Natl. Acad. Sci. U.S.A. 2014, 111 (26), E2770–E2777.

    OpenUrlAbstract/FREE Full Text
23. 23.↵
    1. Aghaeepour N.

    , Finak G, Hoos H, Mosmann TR, Brinkman R, Gottardo R, et al. Critical Assessment of Automated Flow Cytometry Data Analysis Techniques. Nat. Methods 2013, 10 (3), 228–238.

    OpenUrlCrossRefPubMedWeb of Science
24. 24.↵
    1. Brinkman R. R.,
    2. Aghaeepour N.,
    3. Finak G.,
    4. Gottardo R.,
    5. Mosmann T.,
    6. Scheuermann R. H.

    Automated Analysis of Flow Cytometry Data Comes of Age. Cytometry, Part A 2016, 89 (1), 13–15.

    OpenUrl
25. 25.↵
    1. Down M.,
    2. Czubak F.,
    3. Gruska G.,
    4. Stahley S.,
    5. Benham D.

    Measurement System Analysis; 2010.
26. 26.↵
    1. Josephson R.,
    2. Ording C. J.,
    3. Liu Y.,
    4. Shin S.,
    5. Lakshmipathy U.,
    6. Toumadje A.,
    7. Love B.,
    8. Chesnut J. D.,
    9. Andrews P. W.,
    10. Rao M. S.,
    11. Auerbach J. M.

    Qualification of Embryonal Carcinoma 2102 Ep as a Reference for Human Embryonic Stem Cell Research. Stem Cells 2007, 25 (2), 437–446.

    OpenUrlCrossRefPubMedWeb of Science
27. 27.↵
    BD Bioscience. BD Human and Mouse Pluripotent Stem Cell Analysis Kit, 2014. http://www.bdbiosciences.com/ds/pm/others/23-10865.pdf (accessed August 11, 2019).
28. 28.↵
    BD Bioscience. BD CS & T Beads, 2016. http://www.bdbiosciences.com/ds/europe/tds/23-14666.pdf (accessed April 11, 2018).
29. 29.↵
    Flowjo Home page, 2018. https://www.flowjo.com (accessed January 10, 2019).
30. 30.↵
    1. Saldaña J.

    The Coding Manual for Qualitative Researchers, 3rd ed; Sage Publications Ltd: London, 2016.
31. 31.↵
    International Organization for Standardization, ISO/IEC Guide 98-3:2008 Evaluation of Measurement Data — Guide to the Expression of Uncertainty in Measurement. ISO: Geneva, 2008.
32. 32.↵
    Certificate of Expert Practice, 2018. Institute of Biomedical Science Web site. https://www.ibms.org/education/higher-and-expert-qualifications/certificate-of-expert-practice/ (accessed August 11, 2018).
33. 33.↵
    Certification in Cytometry, 2018. ISAC Web site. http://isac-net.org/Resources/Certification-in-Cytometry.aspx (accessed August 11, 2018).
34. 34.↵
    Joint Committee for Guides in Metrology. Vocabulaire International de Métrologie; 2012; Vol. 3.
35. 35.↵
    1. Likeman M.,
    2. Anderson V. M.,
    3. Stevens J. M.,
    4. Waldman A. D.,
    5. Godbolt A. K.,
    6. Frost C.,
    7. Rossor M. N.,
    8. Fox N. C.

    Visual Assessment of Atrophy on Magnetic Resonance Imaging in the Diagnosis of Pathologically Confirmed Young-Onset Dementias. Arch. Neurol. 2005, 62 (9), 1410–1415.

    OpenUrlCrossRefPubMedWeb of Science
36. 36.↵
    1. Kather J. N.,
    2. Weidner A.,
    3. Attenberger U.,
    4. Bukschat Y.,
    5. Weis C.-A.,
    6. Weis M.,
    7. Schad L. R.,
    8. Zöllner F. G.

    Color-Coded Visualization of Magnetic Resonance Imaging Multiparametric Maps. Sci. Rep. 2017, 7 (1), –11.

    OpenUrlCrossRef
37. 37.↵
    1. Barnett D.,
    2. Janossy G.,
    3. Lubenko A.,
    4. Matutes E.,
    5. Newland A.,
    6. Reilly J. T.

    Guideline for the Flow Cytometric Enumeration of CD34+ Haematopoietic Stem cells. Prepared by the CD34+ Haematopoietic Stem Cell Working Party. Clin. Lab. Haematol. 1999, 21 (5), 301–308.

    OpenUrlCrossRefPubMedWeb of Science
38. 38.↵
    1. Whitby L.,
    2. Granger V.,
    3. Storie I.,
    4. Goodfellow K.,
    5. Sawle A.,
    6. Reilly J. T.,
    7. Barnett D.

    Quality Control of CD4+ T-Lymphocyte Enumeration: Results from the Last 9 Years of the United Kingdom National External Quality Assessment Scheme for Immune Monitoring (1993-2001). Cytometry 2002, 50 (2), 102–110.

    OpenUrlCrossRefPubMedWeb of Science
39. 39.↵
    Kruskal-Wallis H Test in SPSS Statistics | Procedure, Output and Interpretation of the Output Using a Relevant Example. Laerd Statistics, 2013. https://statistics.laerd.com/spss-tutorials/kruskal-wallis-h-test-using-spss-statistics.php#:˜:text=The%20Kruskal%2DWallis%20H%20test,continuous%20or%20ordinal%20dependent%20variable (accessed August 11, 2018)
40. 40.↵
    1. Ang CHan.,
    2. Low J.,
    3. Shen JYi.,
    4. Cai E. Z. Y.,
    5. Hing E. C. H.,
    6. Chan Y.,
    7. Sundar G.,
    8. Lim T.

    A Protocol to Reduce Interobserver Variability in the Computed Tomography Measurement of Orbital Floor Fractures. Craniomaxillofac. Trauma Reconstr. 2015, 8 (4), 289–298.

    OpenUrl
41. 41.↵
    1. Lockhart C.,
    2. MacDonald L.,
    3. Alessio A.,
    4. McDougald W.,
    5. Doot R.,
    6. Kinahan P.

    Quantifying and Reducing the Effect of Calibration Error on Variability of PET/CT Standardized Uptake Value Measurements. J. Nucl. Med. 2011, 52 (2), 218–224.

    OpenUrlAbstract/FREE Full Text