Skip to main content

Main menu

  • Home
  • Content
    • Current Issue
    • Past Issues
    • Accepted Articles
    • Email Alerts
    • RSS
    • Terms of Use
  • About PDA JPST
    • JPST Editors and Editorial Board
    • About/Vision/Mission
    • Paper of the Year
  • Author & Reviewer Resources
    • Author Resources / Submit
    • Reviewer Resources
  • JPST Access and Subscriptions
    • PDA Members
    • Institutional Subscriptions
    • Nonmember Access
  • Support
    • Join PDA
    • Contact
    • Feedback
    • Advertising
    • CiteTrack
  • .
    • Visit PDA
    • PDA Letter
    • Technical Reports
    • news uPDATe
    • Bookstore

User menu

  • Register
  • Subscribe
  • My alerts
  • Log in
  • My Cart

Search

  • Advanced search
PDA Journal of Pharmaceutical Science and Technology
  • .
    • Visit PDA
    • PDA Letter
    • Technical Reports
    • news uPDATe
    • Bookstore
  • Register
  • Subscribe
  • My alerts
  • Log in
  • My Cart
PDA Journal of Pharmaceutical Science and Technology

Advanced Search

  • Home
  • Content
    • Current Issue
    • Past Issues
    • Accepted Articles
    • Email Alerts
    • RSS
    • Terms of Use
  • About PDA JPST
    • JPST Editors and Editorial Board
    • About/Vision/Mission
    • Paper of the Year
  • Author & Reviewer Resources
    • Author Resources / Submit
    • Reviewer Resources
  • JPST Access and Subscriptions
    • PDA Members
    • Institutional Subscriptions
    • Nonmember Access
  • Support
    • Join PDA
    • Contact
    • Feedback
    • Advertising
    • CiteTrack
  • Follow pda on Twitter
  • Visit PDA on LinkedIn
  • Visit pda on Facebook
Research ArticleResearch

Quantifying Operator Subjectivity within Flow Cytometry Data Analysis as a Source of Measurement Uncertainty and the Impact of Experience on Results

Rebecca Grant, Karen Coopman, Nicholas Medcalf, Sandro Silva-Gomes, Jonathan J. Campbell, Bo Kara, Julian Braybrook and Jon Petzing
PDA Journal of Pharmaceutical Science and Technology January 2021, 75 (1) 33-47; DOI: https://doi.org/10.5731/pdajpst.2019.011213
Rebecca Grant
1Wolfson School of Mechanical, Electrical and Manufacturing Engineering, Loughborough University, Leicestershire, LE11 3TU, United Kingdom;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: r.grant@lboro.ac.uk
Karen Coopman
2Chemical Engineering, Loughborough University, Leicestershire, LE11 3TU, United Kingdom;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Nicholas Medcalf
1Wolfson School of Mechanical, Electrical and Manufacturing Engineering, Loughborough University, Leicestershire, LE11 3TU, United Kingdom;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Sandro Silva-Gomes
3GlaxoSmithKline, Gunnels Wood Road, Stevenage, United Kingdom; and
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jonathan J. Campbell
4LGC, Queen’s Road, Teddington, Middlesex, TW11 0LY, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bo Kara
3GlaxoSmithKline, Gunnels Wood Road, Stevenage, United Kingdom; and
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Julian Braybrook
4LGC, Queen’s Road, Teddington, Middlesex, TW11 0LY, United Kingdom
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Jon Petzing
1Wolfson School of Mechanical, Electrical and Manufacturing Engineering, Loughborough University, Leicestershire, LE11 3TU, United Kingdom;
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Article
  • Figures & Data
  • References
  • Info & Metrics
  • PDF
Loading

References

  1. 1.↵
    1. Williams D. J.,
    2. Thomas R. J.,
    3. Hourd P. C.,
    4. Chandra A.,
    5. Ratcliffe E.,
    6. Liu Y.,
    7. Rayment E. A.,
    8. Archer J. R.
    Precision Manufacturing for Clinical-Quality Regenerative Medicines. Philos. Trans. R. Soc., A. 2012, 370 (1973), 3924–3949.
    OpenUrlCrossRefPubMed
  2. 2.↵
    1. Hourd P.,
    2. Williams D. J.
    Scanning the Horizon for High Value-Add Manufacturing Science: Accelerating Manufacturing Readiness for the Next Generation of Disruptive, High-Value Curative Cell Therapeutics. Cytotherapy 2018, 20 (5), 759–767.
    OpenUrl
  3. 3.↵
    1. Harrison R. P.,
    2. Ruck S.,
    3. Medcalf N.,
    4. Rafiq Q. A.
    Decentralized Manufacturing of Cell and Gene Therapies: Overcoming Challenges and Identifying Opportunities. Cytotherapy 2017, 19 (10), 1140–1151.
    OpenUrl
  4. 4.↵
    1. Thurman-Newell J. A.,
    2. Petzing J. N.,
    3. Williams D. J.
    Quantification of Biological Variation in Blood-Based Therapy - a Summary of a Meta-Analysis to Inform Manufacturing in the Clinic. Vox Sang. 2015, 109 (4), 394–402.
    OpenUrl
  5. 5.↵
    1. Thurman-Newell J. A.,
    2. Petzing J. N.,
    3. Williams D. A.
    Meta-Analysis of Biological Variation in Blood-Based Therapy as a Precursor to Bio-Manufacturing. Cytotherapy 2016, 18 (5), 686–694.
    OpenUrl
  6. 6.↵
    1. Thurman-Newell J.,
    2. Petzing J.,
    3. Williams D.
    Analysis of Biological Variation in Blood-Based Therapy as a Precursor to Biomanufacturing. Presented at the MIT Center for Biomedical Innovation’s 2015 BioMAN Summit, Boston, MA, Nov 18–19, 2015.
  7. 7.↵
    1. Wang L.,
    2. Hoffman R. A.
    Standardization, Calibration, and Control in Flow Cytometry. Curr. Protoc. Cytom. 2017, 32 (1), 1.3.1–1.3.27.
    OpenUrl
  8. 8.↵
    1. van Dongen J. J. M.,
    2. Lhermitte L.,
    3. Böttcher S.,
    4. Almeida J.,
    5. van der Velden V. H. J.,
    6. Flores-Montero J.,
    7. Rawstron A.,
    8. Asnafi V.,
    9. Lécrevisse Q.,
    10. Lucio P.,
    11. Mejstrikova E.,
    12. Szczepański T.,
    13. Kalina T.,
    14. de Tute R.,
    15. Brüggemann M.,
    16. Sedek L.,
    17. Cullen M.,
    18. Langerak A. W.,
    19. Mendonça A.,
    20. Macintyre E.,
    21. Martin-Ayuso M.,
    22. Hrusak O.,
    23. Vidriales M. B.,
    24. Orfao A
    , on behalf of the EuroFlow Consortium (EU-FP6, LSHB-CT-2006-018708). EuroFlow Antibody Panels for Standardized n-Dimensional Flow Cytometric Immunophenotyping of Normal, Reactive and Malignant Leukocytes. Leukemia 2012, 26 (9), 1908–1975.
    OpenUrlCrossRefPubMedWeb of Science
  9. 9.↵
    1. Kalina T.,
    2. Flores-Montero J.,
    3. van der Velden V. H. J.,
    4. Martin-Ayuso M.,
    5. Böttcher S.,
    6. Ritgen M.,
    7. Almeida J.,
    8. Lhermitte L.,
    9. Asnafi V.,
    10. Mendonça A.,
    11. de Tute R.,
    12. Cullen M.,
    13. Sedek L.,
    14. Vidriales M. B.,
    15. Pérez J. J.,
    16. Te Marvelde J. G.,
    17. Mejstrikova E.,
    18. Hrusak O.,
    19. Szczepański T.,
    20. van Dongen J. J. M.,
    21. Orfao A
    , on behalf of the EuroFlow Consortium (EU-FP6, LSHB-CT-2006-018708). EuroFlow Standardization of Flow Cytometer Instrument Settings and Immunophenotyping Protocols. Leukemia 2012, 26 (9), 1986–2010.
    OpenUrlCrossRefPubMedWeb of Science
  10. 10.↵
    1. Brando B.,
    2. Gatti A.,
    3. Chianese R.,
    4. Gratama J. W.
    Twenty Years of External Quality Assurance in Clinical Cell analysis - A Tribute to Jean-Luc D′Hautcourt. Cytometry, Part B 2007, 72B (1), 2–7.
    OpenUrl
  11. 11.↵
    1. Whitby L.,
    2. Whitby A.,
    3. Fletcher M.,
    4. Barnett D.
    Current Laboratory Practices in Flow Cytometry for the Enumeration of CD 4+ T-Lymphocyte Subsets. Cytometry, Part B 2015, 88 (5), 305–311.
    OpenUrl
  12. 12.↵
    1. Bainbridge J.,
    2. Rountree W.,
    3. Louzao R.,
    4. Wong J.,
    5. Whitby L.,
    6. Denny T. N.,
    7. Barnett D.
    Laboratory Accuracy Improvement in the UK NEQAS Leucocyte Immunophenotyping Immune Monitoring Program: An Eleven-Year Review via Longitudinal Mixed Effects Modeling. Cytometry, Part B 2018, 94 (2), 250–256.
    OpenUrl
  13. 13.↵
    1. Barnett D.,
    2. Granger V.,
    3. Storie I.,
    4. Peel J.,
    5. Pollitt R.,
    6. Smart T.,
    7. Reilly J. T.
    Quality Assessment of CD34+ Stem Cell Enumeration: Experience of the United Kingdom National External Quality Assessment Scheme (UK NEQAS) Using a Unique Stable Whole Blood Preparation. Br. J. Haematol. 1998, 102 (2), 553–565.
    OpenUrlCrossRefPubMedWeb of Science
  14. 14.↵
    UK NEQAS. Flow Cytometry Programmes, 2020. http://www.ukneqasli.co.uk/eqa-pt-programmes/flow-cytometry-programmes/ (accessed March 10, 2020).
  15. 15.↵
    International Conference for Harmonisation, Quality Guideline Q7: Good Manufacturing Practice Guide for Active Pharmaceutical Ingredients. ICH: Geneva, 2000.
  16. 16.↵
    1. Haeckel R.,
    2. Wosniok W.,
    3. Gurr E.,
    4. Peil B.
    Permissible Limits for Uncertainty of Measurement in Laboratory Medicine. Clin. Chem. Lab. Med. 2015, 53 (8), 1161–1171. doi:10.1515/cclm-2014-0874.
    OpenUrl
  17. 17.↵
    1. Grant R.,
    2. Coopman K.,
    3. Medcalf N.,
    4. Silva-Gomes S.,
    5. Campbell J. J.,
    6. Kara B.,
    7. Braybrook J.,
    8. Petzing J.
    Understanding the Contribution of Operator Measurement Variability within Flow Cytometry Data Analysis for Quality Control of Cell and Gene Therapy Manufacturing. Measurement 2020, 150, 106998. doi:10.1016/j.measurement.2019.106998.
    OpenUrl
  18. 18.↵
    1. O'Neill K.,
    2. Aghaeepour N.,
    3. Špidlen J.,
    4. Brinkman R.
    Flow Cytometry Bioinformatics. PLoS Comput. Biol. 2013, 9 (12), e1003365–10. doi:10.1371/journal.pcbi.1003365.
    OpenUrlCrossRefPubMed
  19. 19.↵
    1. Mitsialis V.,
    2. Shouval D.,
    3. Wall S.,
    4. Snapper S. B.,
    5. Konnikova L.
    High Dimensional Immune Phenotyping Using Mass Cytometry Reveals Unique IL1B Signatures in Peripheral Blood and GI Tissue of Ulcerative Colitis, Crohn′s Disease and IL10R Deficient Patients. Gastroenterology 2018, 154 (6 Suppl. 1), S-90–S-91.
    OpenUrl
  20. 20.↵
    1. Behbehani G. K.
    Applications of Mass Cytometry in Clinical Medicine: The Promise and Perils of CyTOF. Clin. Lab. Med. 2017, 37 (4), 945–964.
    OpenUrl
  21. 21.↵
    1. Leipold M. D.,
    2. Obermoser G.,
    3. Fenwick C.,
    4. Kleinstuber K.,
    5. Rashidi N.,
    6. McNevin J. P.,
    7. Nau A. N.,
    8. Wagar L. E.,
    9. Rozot V.,
    10. Davis M. M.,
    11. DeRosa S.,
    12. Pantaleo G.,
    13. Scriba T. J.,
    14. Walker B. D.,
    15. Olsen L. R.,
    16. Maecker H. T.
    Comparison of CyTOF Assays across Sites: Results of a Six-Center Pilot Study. J. Immunol. Methods 2018, 453, 37–43.
    OpenUrlCrossRef
  22. 22.↵
    1. Bruggner R. V.,
    2. Bodenmiller B.,
    3. Dill D. L.,
    4. Tibshirani R. J.,
    5. Nolan G. P.
    Automated Identification of Stratifying Signatures in Cellular Subpopulations. Proc. Natl. Acad. Sci. U.S.A. 2014, 111 (26), E2770–E2777.
    OpenUrlAbstract/FREE Full Text
  23. 23.↵
    1. Aghaeepour N.
    , Finak G, Hoos H, Mosmann TR, Brinkman R, Gottardo R, et al. Critical Assessment of Automated Flow Cytometry Data Analysis Techniques. Nat. Methods 2013, 10 (3), 228–238.
    OpenUrlCrossRefPubMedWeb of Science
  24. 24.↵
    1. Brinkman R. R.,
    2. Aghaeepour N.,
    3. Finak G.,
    4. Gottardo R.,
    5. Mosmann T.,
    6. Scheuermann R. H.
    Automated Analysis of Flow Cytometry Data Comes of Age. Cytometry, Part A 2016, 89 (1), 13–15.
    OpenUrl
  25. 25.↵
    1. Down M.,
    2. Czubak F.,
    3. Gruska G.,
    4. Stahley S.,
    5. Benham D.
    Measurement System Analysis; 2010.
  26. 26.↵
    1. Josephson R.,
    2. Ording C. J.,
    3. Liu Y.,
    4. Shin S.,
    5. Lakshmipathy U.,
    6. Toumadje A.,
    7. Love B.,
    8. Chesnut J. D.,
    9. Andrews P. W.,
    10. Rao M. S.,
    11. Auerbach J. M.
    Qualification of Embryonal Carcinoma 2102 Ep as a Reference for Human Embryonic Stem Cell Research. Stem Cells 2007, 25 (2), 437–446.
    OpenUrlCrossRefPubMedWeb of Science
  27. 27.↵
    BD Bioscience. BD Human and Mouse Pluripotent Stem Cell Analysis Kit, 2014. http://www.bdbiosciences.com/ds/pm/others/23-10865.pdf (accessed August 11, 2019).
  28. 28.↵
    BD Bioscience. BD CS & T Beads, 2016. http://www.bdbiosciences.com/ds/europe/tds/23-14666.pdf (accessed April 11, 2018).
  29. 29.↵
    Flowjo Home page, 2018. https://www.flowjo.com (accessed January 10, 2019).
  30. 30.↵
    1. Saldaña J.
    The Coding Manual for Qualitative Researchers, 3rd ed; Sage Publications Ltd: London, 2016.
  31. 31.↵
    International Organization for Standardization, ISO/IEC Guide 98-3:2008 Evaluation of Measurement Data — Guide to the Expression of Uncertainty in Measurement. ISO: Geneva, 2008.
  32. 32.↵
    Certificate of Expert Practice, 2018. Institute of Biomedical Science Web site. https://www.ibms.org/education/higher-and-expert-qualifications/certificate-of-expert-practice/ (accessed August 11, 2018).
  33. 33.↵
    Certification in Cytometry, 2018. ISAC Web site. http://isac-net.org/Resources/Certification-in-Cytometry.aspx (accessed August 11, 2018).
  34. 34.↵
    Joint Committee for Guides in Metrology. Vocabulaire International de Métrologie; 2012; Vol. 3.
  35. 35.↵
    1. Likeman M.,
    2. Anderson V. M.,
    3. Stevens J. M.,
    4. Waldman A. D.,
    5. Godbolt A. K.,
    6. Frost C.,
    7. Rossor M. N.,
    8. Fox N. C.
    Visual Assessment of Atrophy on Magnetic Resonance Imaging in the Diagnosis of Pathologically Confirmed Young-Onset Dementias. Arch. Neurol. 2005, 62 (9), 1410–1415.
    OpenUrlCrossRefPubMedWeb of Science
  36. 36.↵
    1. Kather J. N.,
    2. Weidner A.,
    3. Attenberger U.,
    4. Bukschat Y.,
    5. Weis C.-A.,
    6. Weis M.,
    7. Schad L. R.,
    8. Zöllner F. G.
    Color-Coded Visualization of Magnetic Resonance Imaging Multiparametric Maps. Sci. Rep. 2017, 7 (1), –11.
    OpenUrlCrossRef
  37. 37.↵
    1. Barnett D.,
    2. Janossy G.,
    3. Lubenko A.,
    4. Matutes E.,
    5. Newland A.,
    6. Reilly J. T.
    Guideline for the Flow Cytometric Enumeration of CD34+ Haematopoietic Stem cells. Prepared by the CD34+ Haematopoietic Stem Cell Working Party. Clin. Lab. Haematol. 1999, 21 (5), 301–308.
    OpenUrlCrossRefPubMedWeb of Science
  38. 38.↵
    1. Whitby L.,
    2. Granger V.,
    3. Storie I.,
    4. Goodfellow K.,
    5. Sawle A.,
    6. Reilly J. T.,
    7. Barnett D.
    Quality Control of CD4+ T-Lymphocyte Enumeration: Results from the Last 9 Years of the United Kingdom National External Quality Assessment Scheme for Immune Monitoring (1993-2001). Cytometry 2002, 50 (2), 102–110.
    OpenUrlCrossRefPubMedWeb of Science
  39. 39.↵
    Kruskal-Wallis H Test in SPSS Statistics | Procedure, Output and Interpretation of the Output Using a Relevant Example. Laerd Statistics, 2013. https://statistics.laerd.com/spss-tutorials/kruskal-wallis-h-test-using-spss-statistics.php#:˜:text=The%20Kruskal%2DWallis%20H%20test,continuous%20or%20ordinal%20dependent%20variable (accessed August 11, 2018)
  40. 40.↵
    1. Ang CHan.,
    2. Low J.,
    3. Shen JYi.,
    4. Cai E. Z. Y.,
    5. Hing E. C. H.,
    6. Chan Y.,
    7. Sundar G.,
    8. Lim T.
    A Protocol to Reduce Interobserver Variability in the Computed Tomography Measurement of Orbital Floor Fractures. Craniomaxillofac. Trauma Reconstr. 2015, 8 (4), 289–298.
    OpenUrl
  41. 41.↵
    1. Lockhart C.,
    2. MacDonald L.,
    3. Alessio A.,
    4. McDougald W.,
    5. Doot R.,
    6. Kinahan P.
    Quantifying and Reducing the Effect of Calibration Error on Variability of PET/CT Standardized Uptake Value Measurements. J. Nucl. Med. 2011, 52 (2), 218–224.
    OpenUrlAbstract/FREE Full Text
PreviousNext
Back to top

In This Issue

PDA Journal of Pharmaceutical Science and Technology: 75 (1)
PDA Journal of Pharmaceutical Science and Technology
Vol. 75, Issue 1
January/February 2021
  • Table of Contents
  • Index by Author
  • Complete Issue (PDF)
Print
Download PDF
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word on PDA Journal of Pharmaceutical Science and Technology.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Quantifying Operator Subjectivity within Flow Cytometry Data Analysis as a Source of Measurement Uncertainty and the Impact of Experience on Results
(Your Name) has sent you a message from PDA Journal of Pharmaceutical Science and Technology
(Your Name) thought you would like to see the PDA Journal of Pharmaceutical Science and Technology web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
1 + 3 =
Solve this simple math problem and enter the result. E.g. for 1+3, enter 4.
Citation Tools
Quantifying Operator Subjectivity within Flow Cytometry Data Analysis as a Source of Measurement Uncertainty and the Impact of Experience on Results
Rebecca Grant, Karen Coopman, Nicholas Medcalf, Sandro Silva-Gomes, Jonathan J. Campbell, Bo Kara, Julian Braybrook, Jon Petzing
PDA Journal of Pharmaceutical Science and Technology Jan 2021, 75 (1) 33-47; DOI: 10.5731/pdajpst.2019.011213

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Share
Quantifying Operator Subjectivity within Flow Cytometry Data Analysis as a Source of Measurement Uncertainty and the Impact of Experience on Results
Rebecca Grant, Karen Coopman, Nicholas Medcalf, Sandro Silva-Gomes, Jonathan J. Campbell, Bo Kara, Julian Braybrook, Jon Petzing
PDA Journal of Pharmaceutical Science and Technology Jan 2021, 75 (1) 33-47; DOI: 10.5731/pdajpst.2019.011213
Twitter logo Facebook logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • 1. Introduction
    • 2. Materials and Methods
    • 3. Discussion
    • 4. Conclusions
    • Conflict of Interest Statement
    • Acknowledgements
    • References
  • Figures & Data
  • References
  • Info & Metrics
  • PDF

Related Articles

  • No related articles found.
  • PubMed
  • Google Scholar

Cited By...

  • ConvexGating infers gating strategies from clusters in single cell cytometry data
  • MetaGate: Interactive Analysis of High-Dimensional Cytometry Data with Meta Data Integration
  • Systematic Design, Generation, and Application of Synthetic Datasets for Flow Cytometry
  • Google Scholar

More in this TOC Section

  • A proof-of-concept study on a universal standard kit to evaluate the risks of inspectors for their foundational ability of visual inspection of injectable drug products
  • Understanding Alignment in the Execution of Extractable Screening Studies Between Laboratories: Results of the ELSIE Lab Practices Sub-Team Industry Surveys
  • Definition of Particle Visibility Threshold in Parenteral Drug Products—Towards Standardization of Visual Inspection Operator Qualification
Show more Research

Similar Articles

Keywords

  • Flow cytometry
  • Variation
  • Operators
  • Gauge R&R
  • Uncertainties
  • Measurement uncertainty

Readers

  • About
  • Table of Content Alerts/Other Alerts
  • Subscriptions
  • Terms of Use
  • Contact Editors

Author/Reviewer Information

  • Author Resources
  • Submit Manuscript
  • Reviewers
  • Contact Editors

Parenteral Drug Association, Inc.

  • About
  • Advertising/Sponsorships
  • Events
  • PDA Bookstore
  • Press Releases

© 2025 PDA Journal of Pharmaceutical Science and Technology Print ISSN: 1079-7440  Digital ISSN: 1948-2124

Powered by HighWire