{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,22]],"date-time":"2026-04-22T05:38:19Z","timestamp":1776836299617,"version":"3.51.2"},"reference-count":60,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2020,2,29]],"date-time":"2020-02-29T00:00:00Z","timestamp":1582934400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Information"],"abstract":"<jats:p>This manuscript outlines a viable approach for training and evaluating machine learning systems for high-stakes, human-centered, or regulated applications using common Python programming tools. The accuracy and intrinsic interpretability of two types of constrained models, monotonic gradient boosting machines and explainable neural networks, a deep learning architecture well-suited for structured data, are assessed on simulated data and publicly available mortgage data. For maximum transparency and the potential generation of personalized adverse action notices, the constrained models are analyzed using post-hoc explanation techniques including plots of partial dependence and individual conditional expectation and with global and local Shapley feature importance. The constrained model predictions are also tested for disparate impact and other types of discrimination using measures with long-standing legal precedents, adverse impact ratio, marginal effect, and standardized mean difference, along with straightforward group fairness measures. By combining interpretable models, post-hoc explanations, and discrimination testing with accessible software tools, this text aims to provide a template workflow for machine learning applications that require high accuracy and interpretability and that mitigate risks of discrimination.<\/jats:p>","DOI":"10.3390\/info11030137","type":"journal-article","created":{"date-parts":[[2020,3,2]],"date-time":"2020-03-02T06:34:23Z","timestamp":1583130863000},"page":"137","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":23,"title":["A Responsible Machine Learning Workflow with Focus on Interpretable Models, Post-hoc Explanation, and Discrimination Testing"],"prefix":"10.3390","volume":"11","author":[{"given":"Navdeep","family":"Gill","sequence":"first","affiliation":[{"name":"H2O.ai, Mountain View, CA 94043, USA"}]},{"given":"Patrick","family":"Hall","sequence":"additional","affiliation":[{"name":"H2O.ai, Mountain View, CA 94043, USA"},{"name":"Department of Decision Sciences, The George Washington University, Washington, DC 20052, USA"}]},{"given":"Kim","family":"Montgomery","sequence":"additional","affiliation":[{"name":"H2O.ai, Mountain View, CA 94043, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8340-8145","authenticated-orcid":false,"given":"Nicholas","family":"Schmidt","sequence":"additional","affiliation":[{"name":"BLDS, LLC, Philadelphia, PA 19103, USA"}]}],"member":"1968","published-online":{"date-parts":[[2020,2,29]]},"reference":[{"key":"ref_1","unstructured":"Rudin, C. (2018). Please Stop Explaining Black Box Models for High Stakes Decisions and Use Interpretable Models Instead. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1811.10154.pdf."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Feldman, M., Friedler, S.A., Moeller, J., Scheidegger, C., and Venkatasubramanian, S. (2015, January 10\u201313). Certifying and Removing Disparate Impact. Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, Australia. Available online: https:\/\/arxiv.org\/pdf\/1412.3756.pdf.","DOI":"10.1145\/2783258.2783311"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R. (2012, January 8\u201310). Fairness Through Awareness. Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, Cambridge, MA, USA. Available online: https:\/\/arxiv.org\/pdf\/1104.3913.pdf.","DOI":"10.1145\/2090236.2090255"},{"key":"ref_4","unstructured":"Buolamwini, J., and Gebru, T. (2018, January 23\u201324). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the Conference on Fairness, Accountability and Transparency, New York, NY, USA. Available online: http:\/\/proceedings.mlr.press\/v81\/buolamwini18a\/buolamwini18a.pdf."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"121","DOI":"10.1007\/s10994-010-5188-5","article-title":"The Security of Machine Learning","volume":"81","author":"Barreno","year":"2010","journal-title":"Mach. Learn."},{"key":"ref_6","unstructured":"Tram\u00e8r, F., Zhang, F., Juels, A., Reiter, M.K., and Ristenpart, T. (2016, January 10\u201312). Stealing Machine Learning Models via Prediction APIs. Proceedings of the 25th USENIX Security Symposium, Austin, TX, USA. Available online: https:\/\/www.usenix.org\/system\/files\/conference\/usenixsecurity16\/sec16_paper_tramer.pdf."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017, January 25). Membership Inference Attacks Against Machine Learning Models. Proceedings of the 2017 IEEE Symposium on Security and Privacy (SP), San Jose, CA, USA. Available online: https:\/\/arxiv.org\/pdf\/1610.05820.pdf.","DOI":"10.1109\/SP.2017.41"},{"key":"ref_8","unstructured":"Shokri, R., Strobel, M., and Zick, Y. (2019). Privacy Risks of Explaining Machine Learning Models. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1907.00164.pdf."},{"key":"ref_9","unstructured":"Williams, M. (2017). Interpretability, Fast Forward Labs. Available online: https:\/\/www.cloudera.com\/products\/fast-forward-labs-research.html."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Friedman, J.H. (1979). A Tree-structured Approach to Nonparametric Multiple Regression. Smoothing Techniques for Curve Estimation, Springer. Available online: http:\/\/inspirehep.net\/record\/140963\/files\/slac-pub-2336.pdf.","DOI":"10.1007\/BFb0098488"},{"key":"ref_11","first-page":"1","article-title":"Multivariate Adaptive Regression Splines","volume":"19","author":"Friedman","year":"1991","journal-title":"Ann. Stat."},{"key":"ref_12","unstructured":"(2020, February 24). Mortgage Data (HMDA), Available online: https:\/\/www.consumerfinance.gov\/data-research\/hmda\/."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Friedman, J.H. (2001). Greedy Function Approximation: a Gradient Boosting Machine. Ann. Stat., 1189\u20131232. Available online: https:\/\/statweb.stanford.edu\/~jhf\/ftp\/trebst.pdf.","DOI":"10.1214\/aos\/1013203451"},{"key":"ref_14","unstructured":"Friedman, J.H., Hastie, T., and Tibshirani, R. (2001). The Elements of Statistical Learning, Springer. Available online: https:\/\/web.stanford.edu\/~hastie\/ElemStatLearn\/printings\/ESLII_print12.pdf."},{"key":"ref_15","unstructured":"Recht, B., Re, C., Wright, S., and Niu, F. (2020, February 26). HOGWILD: A Lock-free Approach to Parallelizing Stochastic Gradient Descent. Available online: https:\/\/papers.nips.cc\/paper\/4390-hogwild-a-lock-free-approach-to-parallelizing-stochastic-gradient-descent.pdf."},{"key":"ref_16","unstructured":"Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.R. (2012). Improving Neural Networks by Preventing Co-adaptation of Feature Detectors. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1207.0580.pdf."},{"key":"ref_17","unstructured":"Sutskever, I., Martens, J., Dahl, G., and Hinton, G. (2013, January 16\u201321). On the Importance of Initialization and Momentum in Deep Learning. Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA. Available online: http:\/\/proceedings.mlr.press\/v28\/sutskever13.pdf."},{"key":"ref_18","unstructured":"Zeiler, M.D. (2012). ADADELTA: An Adaptive Learning Rate Method. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1212.5701.pdf."},{"key":"ref_19","unstructured":"A\u00efvodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., and Tapp, A. (2019). Fairwashing: The Risk of Rationalization. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1901.09749.pdf."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. (2019). Fooling LIME and SHAP: Adversarial Attacks on Post-hoc Explanation Methods. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1911.02508.pdf.","DOI":"10.1145\/3375627.3375830"},{"key":"ref_21","unstructured":"Vaughan, J., Sudjianto, A., Brahimi, E., Chen, J., and Nair, V.N. (2018). Explainable Neural Networks Based on Additive Index Models. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1806.01933.pdf."},{"key":"ref_22","unstructured":"Yang, Z., Zhang, A., and Sudjianto, A. (2019). Enhancing Explainability of Neural Networks Through Architecture Constraints. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1901.03838.pdf."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Goldstein, A., Kapelner, A., Bleich, J., and Pitkin, E. (2015). Peeking Inside the Black Box: Visualizing Statistical Learning with Plots of Individual Conditional Expectation. J. Comput. Graph. Stat., 24, Available online: https:\/\/arxiv.org\/pdf\/1309.6392.pdf.","DOI":"10.1080\/10618600.2014.907095"},{"key":"ref_24","unstructured":"Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems (NIPS), Curran Associates, Inc.. Available online: http:\/\/papers.nips.cc\/paper\/7062-a-unified-approach-to-interpreting-model-predictions.pdf."},{"key":"ref_25","unstructured":"Lundberg, S.M., Erion, G.G., and Lee, S.I. (2017, January 10). Consistent Individualized Feature Attribution for Tree Ensembles. Proceedings of the 2017 ICML Workshop on Human Interpretability in Machine Learning (WHI 2017), Sydney, Australia. Available online: https:\/\/openreview.net\/pdf?id=ByTKSo-m-."},{"key":"ref_26","unstructured":"Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates. Available online: http:\/\/www.utstat.toronto.edu\/~brunner\/oldclass\/378f16\/readings\/CohenPower.pdf."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"155","DOI":"10.1037\/0033-2909.112.1.155","article-title":"A Power Primer","volume":"112","author":"Cohen","year":"1992","journal-title":"Psychol. Bull."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Zafar, M.B., Valera, I., Gomez Rodriguez, M., and Gummadi, K.P. (2017, January 3\u20137). Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification Without Disparate Mistreatment. Proceedings of the 26th International Conference onWorldWideWeb, Perth, Australia. Available online: https:\/\/arxiv.org\/pdf\/1610.08452.pdf.","DOI":"10.1145\/3038912.3052660"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Lou, Y., Caruana, R., Gehrke, J., and Hooker, G. (2013, January 11\u201314). Accurate Intelligible Models with Pairwise Interactions. Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Chicago, IL, USA. Available online: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.352.7682&rep=rep1&type=pdf.","DOI":"10.1145\/2487575.2487579"},{"key":"ref_30","unstructured":"Apley, D.W. (2016). Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Models. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1612.08468.pdf."},{"key":"ref_31","unstructured":"Shapley, L.S., and Roth, A.E. (1988). The Shapley value: Essays in Honor of Lloyd S. Shapley, Cambridge University Press. Available online: http:\/\/www.library.fa.ru\/files\/Roth2.pdf."},{"key":"ref_32","unstructured":"Hall, P. (2020, February 26). On the Art and Science of Machine Learning Explanations. Available online: https:\/\/arxiv.org\/pdf\/1810.02909.pdf."},{"key":"ref_33","unstructured":"Molnar, C. (2020, February 26). Interpretable Machine Learning. Available online: https:\/\/christophm.github.io\/interpretableml-book\/."},{"key":"ref_34","unstructured":"Hu, X., Rudin, C., and Seltzer, M. (2019). Optimal Sparse Decision Trees. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1904.12847.pdf."},{"key":"ref_35","unstructured":"Chen, C., Li, O., Barnett, A., Su, J., and Rudin, C. (2019, January 8\u201314). This Looks Like That: Deep Learning for Interpretable Image Recognition. Proceedings of Neural Information Processing Systems (NeurIPS), Vancouver, BC, Canada. Available online: https:\/\/arxiv.org\/pdf\/1806.10574.pdf."},{"key":"ref_36","doi-asserted-by":"crossref","first-page":"916","DOI":"10.1214\/07-AOAS148","article-title":"Predictive Learning Via Rule Ensembles","volume":"2","author":"Friedman","year":"2008","journal-title":"Ann. Appl. Stat."},{"key":"ref_37","first-page":"3790","article-title":"Monotonic Calibrated Interpolated Lookup Tables","volume":"17","author":"Gupta","year":"2016","journal-title":"J. Mach. Learn. Res."},{"key":"ref_38","unstructured":"Wilkinson, L. (2020, February 26). Visualizing Big Data Outliers through Distributed Aggregation. Available online: https:\/\/www.cs.uic.edu\/~wilkinson\/Publications\/outliers.pdf."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1561\/2200000055","article-title":"Generalized Low Rank Models","volume":"9","author":"Udell","year":"2016","journal-title":"Found. Trends\u00ae Mach. Learn."},{"key":"ref_40","unstructured":"Holohan, N., Braghin, S., Mac Aonghusa, P., and Levacher, K. (2019). Diffprivlib: The IBM Differential Privacy Library. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1907.02444.pdf."},{"key":"ref_41","unstructured":"Ji, Z., Lipton, Z.C., and Elkan, C. (2014). Differential Privacy and Machine Learning: A Survey and Review. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1412.7584.pdf."},{"key":"ref_42","unstructured":"Papernot, N., Song, S., Mironov, I., Raghunathan, A., Talwar, K., and Erlingsson, \u00da. (2018). Scalable Private Learning with PATE. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1802.08908.pdf."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., and Zhang, L. (2016, January 24\u201328). Deep Learning with Differential Privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria. Available online: https:\/\/arxiv.org\/pdf\/1607.00133.pdf.","DOI":"10.1145\/2976749.2978318"},{"key":"ref_44","unstructured":"Pearl, J., and Mackenzie, D. (2020, February 26). The Book of Why: The New Science of Cause and Effect. Available online: http:\/\/cdar.berkeley.edu\/wp-content\/uploads\/2017\/04\/Lisa-Goldberg-reviews-The-Book-of-Why.pdf."},{"key":"ref_45","first-page":"841","article-title":"Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GPDR","volume":"31","author":"Wachter","year":"2017","journal-title":"Harv. JL Tech."},{"key":"ref_46","unstructured":"Ancona, M., Ceolini, E., Oztireli, C., and Gross, M. (May, January 30). Towards Better Understanding of Gradient-based Attribution Methods for Deep Neural Networks. Proceedings of the 6th International Conference on Learning Representations (ICLR 2018), Vancouver, BC, Canada. Available online: https:\/\/www.research-collection.ethz.ch\/bitstream\/handle\/20.500.11850\/249929\/Flow_ICLR_2018.pdf."},{"key":"ref_47","doi-asserted-by":"crossref","unstructured":"Wallace, E., Tuyls, J., Wang, J., Subramanian, S., Gardner, M., and Singh, S. (2019). AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1909.09251.pdf.","DOI":"10.18653\/v1\/D19-3002"},{"key":"ref_48","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/s10115-011-0463-8","article-title":"Data Preprocessing Techniques for Classification Without Discrimination","volume":"33","author":"Kamiran","year":"2012","journal-title":"Knowl. Inf. Syst."},{"key":"ref_49","doi-asserted-by":"crossref","unstructured":"Zhang, B.H., Lemoine, B., and Mitchell, M. (2018, January 2\u20133). Mitigating Unwanted Biases with Adversarial Learning. Proceedings of the 2018 AAAI\/ACM Conference on AI, Ethics, and Society, New Orleans, LA, USA. Available online: https:\/\/arxiv.org\/pdf\/1801.07593.pdf.","DOI":"10.1145\/3278721.3278779"},{"key":"ref_50","unstructured":"Zemel, R., Wu, Y., Swersky, K., Pitassi, T., and Dwork, C. (2013, January 16\u201321). Learning Fair Representations. Proceedings of the International Conference on Machine Learning, Atlanta, GA, USA. Available online: http:\/\/proceedings.mlr.press\/v28\/zemel13.pdf."},{"key":"ref_51","doi-asserted-by":"crossref","unstructured":"Kamiran, F., Karim, A., and Zhang, X. (2012, January 10). Decision Theory for Discrimination-aware Classification. Proceedings of the 2012 IEEE 12th International Conference on Data Mining, Brussels, Belgium. Available online: http:\/\/citeseerx.ist.psu.edu\/viewdoc\/download?doi=10.1.1.722.3030&rep=rep1&type=pdf.","DOI":"10.1109\/ICDM.2012.45"},{"key":"ref_52","unstructured":"Rauber, J., Brendel, W., and Bethge, M. (2017). Foolbox: A Python Toolbox to Benchmark the Robustness of Machine Learning Models. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1707.04131.pdf."},{"key":"ref_53","unstructured":"Papernot, N., Faghri, F., Carlini, N., Goodfellow, I., Feinman, R., Kurakin, A., Xie, C., Sharma, Y., Brown, T., and Roy, A. (2018). Technical Report on the CleverHans v2.1.0 Adversarial Examples Library. arXiv, Available online: https:\/\/arxiv.org\/pdf\/1610.00768.pdf."},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Amershi, S., Chickering, M., Drucker, S.M., Lee, B., Simard, P., and Suh, J. (2015, January 18\u201323). Modeltracker: Redesigning Performance Analysis Tools for Machine Learning. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Seoul, Korea. Available online: https:\/\/www.microsoft.com\/en-us\/research\/wp-content\/uploads\/2016\/02\/amershi.CHI2015.ModelTracker.pdf.","DOI":"10.1145\/2702123.2702509"},{"key":"ref_55","doi-asserted-by":"crossref","unstructured":"Papernot, N. (2018, January 19). A Marauder\u2019s Map of Security and Privacy in Machine Learning: An overview of current and future research directions for making machine learning secure and private. Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, Toronto, ON, Canada. Available online: https:\/\/arxiv.org\/pdf\/1811.01134.pdf.","DOI":"10.1145\/3270101.3270102"},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., and Gebru, T. (2019, January 29\u201331). Model Cards for Model Reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA. Available online: https:\/\/arxiv.org\/pdf\/1810.03993.pdf.","DOI":"10.1145\/3287560.3287596"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Bracke, P., Datta, A., Jung, C., and Sen, S. (2020, February 26). Machine Learning Explainability in Finance: An Application to Default Risk Analysis. Available online: https:\/\/www.bankofengland.co.uk\/-\/media\/boe\/files\/working-paper\/2019\/machine-learning-explainability-in-finance-an-application-to-default-risk-analysis.pdf.","DOI":"10.2139\/ssrn.3435104"},{"key":"ref_58","doi-asserted-by":"crossref","unstructured":"Friedler, S.A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E.P., and Roth, D. (2019, January 29\u201331). A Comparative Study of Fairness-enhancing Interventions in Machine Learning. Proceedings of the Conference on Fairness, Accountability, and Transparency, Atlanta, GA, USA. Available online: https:\/\/arxiv.org\/pdf\/1802.04422.pdf.","DOI":"10.1145\/3287560.3287589"},{"key":"ref_59","first-page":"130","article-title":"An Introduction to Artificial Intelligence and Solutions to the Problems of Algorithmic Discrimination","volume":"73","author":"Schmidt","year":"2019","journal-title":"Conf. Consum. Financ. Law Q. Rep."},{"key":"ref_60","unstructured":"Hoare, C.A.R. (1981). The 1980 ACM Turing Award Lecture. Communications, Available online: http:\/\/www.cs.fsu.edu\/~engelen\/courses\/COP4610\/hoare.pdf."}],"container-title":["Information"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/3\/137\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T09:02:45Z","timestamp":1760173365000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2078-2489\/11\/3\/137"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2020,2,29]]},"references-count":60,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2020,3]]}},"alternative-id":["info11030137"],"URL":"https:\/\/doi.org\/10.3390\/info11030137","relation":{},"ISSN":["2078-2489"],"issn-type":[{"value":"2078-2489","type":"electronic"}],"subject":[],"published":{"date-parts":[[2020,2,29]]}}}