{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,15]],"date-time":"2026-02-15T21:19:54Z","timestamp":1771190394611,"version":"3.50.1"},"reference-count":36,"publisher":"Walter de Gruyter GmbH","issue":"3","license":[{"start":{"date-parts":[[2022,8,1]],"date-time":"2022-08-01T00:00:00Z","timestamp":1659312000000},"content-version":"unspecified","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,8,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:sec>\n                  <jats:title>Purpose<\/jats:title>\n                  <jats:p>Patent classification is one of the areas in Intellectual Property Analytics (IPA), and a growing use case since the number of patent applications has been increasing worldwide. We propose using machine learning algorithms to classify Portuguese patents and evaluate the performance of transfer learning methodologies to solve this task.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Design\/methodology\/approach<\/jats:title>\n                  <jats:p>We applied three different approaches in this paper. First, we used a dataset available by INPI to explore traditional machine learning algorithms and ensemble methods. After preprocessing data by applying TF-IDF, FastText and Doc2Vec, the models were evaluated by cross-validation in 5 folds. In a second approach, we used two different Neural Networks architectures, a Convolutional Neural Network (CNN) and a bi-directional Long Short-Term Memory (BiLSTM). Finally, we used pre-trained BERT, DistilBERT, and ULMFiT models in the third approach.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Findings<\/jats:title>\n                  <jats:p>BERTTimbau, a BERT architecture model pre-trained on a large Portuguese corpus, presented the best results for the task, even though with a performance of only 4% superior to a LinearSVC model using TF-IDF feature engineering.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Research limitations<\/jats:title>\n                  <jats:p>The dataset was highly imbalanced, as usual in patent applications, so the classes with the lowest samples were expected to present the worst performance. That result happened in some cases, especially in classes with less than 60 training samples.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Practical implications<\/jats:title>\n                  <jats:p>Patent classification is challenging because of the hierarchical classification system, the context overlap, and the underrepresentation of the classes. However, the final model presented an acceptable performance given the size of the dataset and the task complexity. This model can support the decision and improve the time by proposing a category in the second level of ICP, which is one of the critical phases of the grant patent process.<\/jats:p>\n               <\/jats:sec>\n               <jats:sec>\n                  <jats:title>Originality\/value<\/jats:title>\n                  <jats:p>To our knowledge, the proposed models were never implemented for Portuguese patent classification.<\/jats:p>\n               <\/jats:sec>","DOI":"10.2478\/jdis-2022-0015","type":"journal-article","created":{"date-parts":[[2022,8,12]],"date-time":"2022-08-12T09:49:38Z","timestamp":1660297778000},"page":"49-70","source":"Crossref","is-referenced-by-count":8,"title":["A Use Case of Patent Classification Using Deep Learning with Transfer Learning"],"prefix":"10.2478","volume":"7","author":[{"given":"Roberto","family":"Henriques","sequence":"first","affiliation":[{"name":"Campus de Campolide , Lisboa , Portugal"}]},{"given":"Adria","family":"Ferreira","sequence":"additional","affiliation":[{"name":"Campus de Campolide , Lisboa , Portugal"}]},{"given":"Mauro","family":"Castelli","sequence":"additional","affiliation":[{"name":"Campus de Campolide , Lisboa , Portugal"}]}],"member":"374","published-online":{"date-parts":[[2022,8,12]]},"reference":[{"key":"2023041406491823279_j_jdis-2022-0015_ref_001","doi-asserted-by":"crossref","unstructured":"Abdelgawad, L., Kluegl, P., Genc, E., Falkner, S., & Hutter, F. (2020). Optimizing Neural Networks for Patent Classification. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). volume 11908 LNAI. doi:10.1007\/978-3-030-46133-1{\\_}41.","DOI":"10.1007\/978-3-030-46133-1"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_002","doi-asserted-by":"crossref","unstructured":"Aristodemou, L., & Tietze, F. (2018). The state-of-the-art on Intellectual Property Analytics (IPA): A literature review on artificial intelligence, machine learning and deep learning methods for analysing intellectual property (IP) data. World Patent Information, 55, 37\u201351. doi:10.1016\/J.WPI.2018.07.002.","DOI":"10.1016\/j.wpi.2018.07.002"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_003","doi-asserted-by":"crossref","unstructured":"Bispo, T.D., Macedo, H.T., Santos, F.D.O., Da Silva, R.P., Matos, L.N., Prado, B.O., Da Silva, G.J., & Guimar\u00e3es, A. (2019). Long short-term memory model for classification of english-PtBR cross-lingual hate speech. Journal of Computer Science, 15. doi:10.3844\/jcssp.2019.1546.1571.","DOI":"10.3844\/jcssp.2019.1546.1571"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_004","doi-asserted-by":"crossref","unstructured":"Quinta de Castro, P.V., F\u00e9lix Felipe da Silva, N., & da Silva Soares, A. (2018). Portuguese Named Entity Recognition Using LSTM-CRF. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). volume 11122 LNAI. doi:10.1007\/978-3-319-99722-3{\\_}9.","DOI":"10.1007\/978-3-319-99722-3_9"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_005","unstructured":"De Castro, P.V.Q., Da Silva, N.F.F., & Da Silva Soares, A. (2019). Contextual representations and semi-supervised named entity recognition for Portuguese language. In CEUR Workshop Proceedings. volume 2421."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_006","unstructured":"Derieux, F., Bobeica, M., Pois, D., & Raysz, J.P. (2010). Combining semantics and statistics for patent classification. In CEUR Workshop Proceedings. volume 1176."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_007","unstructured":"Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL HLT 2019\u20132019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\u2014Proceedings of the Conference. volume 1."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_008","unstructured":"Espacenet (2021). Espacenet Patent search. URL: https:\/\/lp.espacenet.com\/?locale=pt_LP."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_009","doi-asserted-by":"crossref","unstructured":"Feldman, R., & Sanger, J. (2006). The Text Mining Handbook. doi:10.1017\/cbo9780511546914.","DOI":"10.1017\/CBO9780511546914"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_010","doi-asserted-by":"crossref","unstructured":"Gomez, J.C., & Moens, M.F. (2014). A survey of automated hierarchical classification of patents. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8830. doi:10.1007\/978-3-319-12511-4.","DOI":"10.1007\/978-3-319-12511-4_11"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_011","doi-asserted-by":"crossref","unstructured":"Gon\u00e7alves, T., Silva, C., Quaresma, P., & Vieira, R. (2006). Analysing part-of speech for Portuguese text classification. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). volume 3878 LNCS.","DOI":"10.1007\/11671299_57"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_012","doi-asserted-by":"crossref","unstructured":"Hu, J., Li, S.B., Hu, J.J., & Yang, G.C. (2018). A hierarchical feature extraction model for multi-label mechanical patent classification. Sustainability (Switzerland), 10. doi:10.3390\/su10010219.","DOI":"10.3390\/su10010219"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_013","unstructured":"Instituto Nacional da Propriedade Intelectual (2018). C\u00f3digo da Propriedade Industrial. URL: https:\/\/inpi.justica.gov.pt\/Portals\/6\/PDF%20INPI\/Legisla%C3%A7%C3%A3o%20e%20outros%20documentos\/CPI%20-%202018.pdf?ver=2019-06-28-153157-733."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_014","unstructured":"IP5 (2019). IP5 Statistics Report 2018 Edition. URL: https:\/\/www.fiveipoffices.org\/statistics\/statisticsreports\/2019edition"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_015","doi-asserted-by":"crossref","unstructured":"Kowsari, K., Meimandi, K.J., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. doi:10.3390\/info10040150.","DOI":"10.3390\/info10040150"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_016","doi-asserted-by":"crossref","unstructured":"Krestel, R., Chikkamath, R., Hewel, C., & Risch, J. (2021). A survey on deep learning for patent analysis. World Patent Information, 65, 102035.","DOI":"10.1016\/j.wpi.2021.102035"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_017","doi-asserted-by":"crossref","unstructured":"Lai, K., & Wu, S.J. (2005). Using the patent co\u2013citation approach to establish a new patent classification system. Information Processing and Management, 41(2), 313\u2013330","DOI":"10.1016\/j.ipm.2003.11.004"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_018","doi-asserted-by":"crossref","unstructured":"Lee, J.S., & Hsiang, J. (2020). Patent classification by fine-tuning BERT language model. World Patent Information, 61. doi:10.1016\/j.wpi.2020.101965.","DOI":"10.1016\/j.wpi.2020.101965"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_019","doi-asserted-by":"crossref","unstructured":"Li, S.B., Hu, J., Cui, Y.X., & Hu, J.J. (2018). DeepPatent: patent classification with convolutional neural networks and word embedding. Scientometrics, 117. doi:10.1007\/s11192-018-2905-5.","DOI":"10.1007\/s11192-018-2905-5"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_020","unstructured":"Liddy, E.D. (2001). Natural Language Processing. In Encyclopedia of Library and Information Science. Encyclopedia of Library and Information Science."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_021","doi-asserted-by":"crossref","unstructured":"Manning, C.D., Raghavan, P., & Schutze, H. (2008). Introduction to Information Retrieval. doi:10.1017\/cbo9780511809071.","DOI":"10.1017\/CBO9780511809071"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_022","doi-asserted-by":"crossref","unstructured":"Pan, S.J., & Yang, Q. (2010). A survey on transfer learning. doi:10.1109\/TKDE.2009.191.","DOI":"10.1109\/TKDE.2009.191"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_023","doi-asserted-by":"crossref","unstructured":"Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In NAACL HLT 2018\u20142018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\u2014Proceedings of the Conference. volume 1. doi:10.18653\/v1\/n18-1202.","DOI":"10.18653\/v1\/N18-1202"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_024","doi-asserted-by":"crossref","unstructured":"Risch, J., & Krestel, R. (2019). Domain-specific word embeddings for patent classification. Data Technologies and Applications, 53. doi:10.1108\/DTA-01-2019-0002.","DOI":"10.1108\/DTA-01-2019-0002"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_025","doi-asserted-by":"crossref","unstructured":"Rodrigues, R.C., Rodrigues, J., de Castro, P.V.Q., da Silva, N.F.F., & Soares, A. (2020). Portuguese language models and word embeddings: Evaluating on semantic similarity tasks. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). volume 12037 LNAI. doi:10.1007\/978-3-030-41505-1{\\_}23.","DOI":"10.1007\/978-3-030-41505-1_23"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_026","doi-asserted-by":"crossref","unstructured":"dos Santos, C., & Guimar\u00e3es, V. (2015). Boosting Named Entity Recognition with Neural Character Embeddings. doi:10.18653\/v1\/w15-3904.","DOI":"10.18653\/v1\/W15-3904"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_027","doi-asserted-by":"crossref","unstructured":"Silva, C., & Ribeiro, B. (2010). Inductive Inference for Large Scale Text Classification: Kernel Approaches and Techniques. volume 255. doi:10.1007\/978-3-642-04533-2.","DOI":"10.1007\/978-3-642-04533-2"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_028","unstructured":"Souza, F., Nogueira, R., & Lotufo, R. (2019). Portuguese Named Entity Recognition using BERT-CRF. arXiv. URL: https:\/\/arxiv.org\/abs\/1909.10649v2."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_029","doi-asserted-by":"crossref","unstructured":"Trappey, A.J., Hsu, F C., Trappey, C.V., & Lin, C.I. (2006). Development of a patent document classification and search platform using a back-propagation network. Expert Systems with Applications, 31. doi:10.1016\/j.eswa.2006.01.013.","DOI":"10.1016\/j.eswa.2006.01.013"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_030","doi-asserted-by":"crossref","unstructured":"Trappey, A.J., Trappey, C.V., Chiang, T.A., & Huang, Y.H. (2013). Ontology-based neural network for patent knowledge management in design collaboration. International Journal of Production Research, 51. doi:10.1080\/00207543.2012.701775.","DOI":"10.1080\/00207543.2012.701775"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_031","doi-asserted-by":"crossref","unstructured":"Trappey, A.J.C., Trappey, C.V., Wu, C.-Y., & Lin, C.-W. (2012). A patent quality analysis for innovative technology and product development. Advanced Engineering Informatics, 26, 26\u201334. doi:10.1016\/j.aei.2011.06.005.","DOI":"10.1016\/j.aei.2011.06.005"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_032","unstructured":"Wagner Filho, J.A., Wilkens, R., Idiart, M., & Villavicencio, A. (2019). The BRWAC corpus: A new open resource for Brazilian Portuguese. In LREC 2018\u201411th International Conference on Language Resources and Evaluation."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_033","unstructured":"World Intellectual Property Organization (2008). WIPO Intellectual Property Handbook: Policy, Law and Use. doi:1."},{"key":"2023041406491823279_j_jdis-2022-0015_ref_034","doi-asserted-by":"crossref","unstructured":"Wu, J.L., Chang, P.C., Tsao, C.C., & Fan, C.Y. (2016). A patent quality analysis and classification system using self-organizing maps with support vector machine. Applied Soft Computing Journal, 41. doi:10.1016\/j.asoc.2016.01.020.","DOI":"10.1016\/j.asoc.2016.01.020"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_035","doi-asserted-by":"crossref","unstructured":"Zhang, X.Y. (2014). Interactive patent classification based on multi-classifier fusion and active learning. Neurocomputing, 127. doi:10.1016\/j.neucom.2013.08.013.","DOI":"10.1016\/j.neucom.2013.08.013"},{"key":"2023041406491823279_j_jdis-2022-0015_ref_036","doi-asserted-by":"crossref","unstructured":"Zhuang, F.Z., Qi, Z.Y., Duan, K.Y., Xi, D.B., Zhu, Y.C., Zhu, H.S., Xiong, H., & He, Q. (2021). A comprehensive survey on transfer learning, in Proceedings of the IEEE, 109(1), Jan. 2021. doi:10.1109\/JPROC.2020.3004555.","DOI":"10.1109\/JPROC.2020.3004555"}],"container-title":["Journal of Data and Information Science"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.sciendo.com\/pdf\/10.2478\/jdis-2022-0015","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,14]],"date-time":"2023-04-14T11:07:30Z","timestamp":1681470450000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.sciendo.com\/article\/10.2478\/jdis-2022-0015"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,8,1]]},"references-count":36,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,8,12]]},"published-print":{"date-parts":[[2022,8,1]]}},"alternative-id":["10.2478\/jdis-2022-0015"],"URL":"https:\/\/doi.org\/10.2478\/jdis-2022-0015","relation":{},"ISSN":["2543-683X"],"issn-type":[{"value":"2543-683X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,8,1]]}}}