藥品核實的正確性對於病患的用藥安全至關重要，否則配錯藥時甚至會危害
到病患的健康與生命。然而，以一般大型醫院而言，對於沒有 barcode 或 RFID
Tag 等額外識別輔具的藥排，若藥師只憑藉藥排擺放位置，或是只看外觀包裝，
很容易出錯。因此，本論文的目的是設計一個基於藥排外觀的辨識系統，在手持
藥排的過程中，透過兩台相對的攝影機，同時捕捉藥排的正、反面影像，並且實
時地辨識出該藥排。據此，本論文提出了一個基於深度學習的「手持藥排之空中
辨識技術」(Hand-held Blister Identification network，HBIN)，能在開放的環境中對
手持藥排進行辨識。結構上，HBIN 是由「藥排框取網路」(Blister pack cropping
network，BCN) 以及「藥排辨識網路」(RTT identification network，RIN) 這兩個子
網路所組合而成，其中 BCN 負責對輸入影像中的藥排進行框取，並將框取後的藥
排正、反面拼整為固定方向與大小的影像。接著，再將該影像輸入 RIN 得到辨識
結果。為了實現 HBIN，本論文以馬偕醫院成人錠劑區，共計 230 類藥排，建置
了三種不同類型的資料集，並在不同場景下對 HBIN 進行了本雛型的辨識測試：
在熟悉場景中，HBIN 的辨識率 F1-score 分別達到了 97.52% 和 94.33%。另外，依
據背景相減後的影像進行框取以及基於特徵分塊分類器進行辨識而近一步改良的
HBIN。於陌生場景中，辨識率 F1-score 達到了 90%，其與原先的 HBIN 相比提高
了 15% 的辨識效能。

Correct medicine identification is critical to medicine dispensing. Failure to do so
may result harm to the health and life of the patient who takes the medicine. Without addi-
tional identification aids, such as bar-code or RFID Tag, dispensing medicine can be error
prone. The purpose of this study is to design an image-based easy-to-use deep-learning
identification system for blister packages: with a pair of two oppositely situated cameras,
when a fetched blister package passes through, both front and back sides of the hand-held
blister package will be captured and identified in real time. Accordingly, this paper pro-
poses a Hand-held Blister Identification Network (HBIN), a deep learning-based identifi-
cation technology that identifies hand-held blister packages in an open environment. The
proposed HBIN consists of two sub-networks: the Blister pack cropping network (BCN)
and the RTT identification network (RIN). The BCN is to crop hand-held blister packages
in the input image, as well as to juxtapose both sides of the cropped blister packages into a
fixed direction and size image template, RTT (Rectified Tow-Sided Template). Then, the
RTT is input into RIN for final identification result. To train and test HBIN, three datasets
have been constructed that are based on the 230 types of blister packages dispensed at the
adult lozenge station in Mackay Memorial Hospital. Ground truth of cropped contours
and types of blister packages are also labeled, with which BCN and RIN are trained. As
a whole, HBIN is tested in different scenes in this paper. In two similar scenes, the F1-
score of HBIN reaches 97.52% and 94.33%, respectively. The HBIN is further improved
by two measures: one by cropping on background subtracted images instead of the origi-
nal images, and the other by part-level classifier that discount the influence by the partial
occlusion by hand on blister packages. The F1-score thus obtained is about 90% or un-
constrained background environment, which is about 15% improvement over the baseline
solution.

References

[1] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection
and semantic segmentation,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition,
pp. 580–587, June 2014.

[2] R. Girshick, “Fast r-cnn,” in Proceedings of the IEEE International Conference on Computer Vision,
pp. 1440–1448, 2015.

[3] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region
proposal networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6,
pp. 1137–1149, 2017.

[4] J. S. Wang, A. Ambikapathi, Y. Han, S. Chung, H. Ting, and C. Chen, “Highlighted deep learning
based identification of pharmaceutical blister packages,” in 2018 IEEE 23rd International Conference
on Emerging Technologies and Factory Automation (ETFA), vol. 1, pp. 638–645, Sep. 2018.

[5] J. Rivera-Rubio, S. Idrees, I. Alexiou, L. Hadjilucas, and A. Bharath, “Small hand-held object recogni-
tion test (short),” in 2014 IEEE Winter Conference on Applications of Computer Vision, WACV 2014,
pp. 524–531, 2014.

[6] X. Lv, S.-Q. Jiang, L. Herranz, and S. Wang, “Rgb-d hand-held object recognition based on hetero-
geneous feature fusion,” Journal of Computer Science and Technology, vol. 30, no. 2, pp. 340–352,
2015.

[7] X. Lv, X. Liu, X. Li, X. Li, S. Jiang, and Z. He, “Modality-specific and hierarchical feature learning
for rgb-d hand-held object recognition,” Multimedia Tools and Applications, vol. 76, no. 3, pp. 4273–
4290, 2017.

[8] A. Krizhevsky, I. Sutskever, and G. Hinton, “Imagenet classification with deep convolutional neural
networks,” in Advances in Neural Information Processing Systems, vol. 2, pp. 1097–1105, 2012.

[9] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recogni-
tion,” arXiv preprint arXiv:1409.1556, 2014.

[10] C. Su, J. Li, S. Zhang, J. Xing, W. Gao, and Q. Tian, “Pose-driven deep convolutional model for person
re-identification,” in 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3980–
3989, Oct 2017.

[11] H. Yao, S. Zhang, R. Hong, Y. Zhang, C. Xu, and Q. Tian, “Deep representation learning with part
loss for person re-identification,” IEEE Transactions on Image Processing, vol. 28, pp. 2860–2871,
June 2019.

[12] Y. Sun, L. Zheng, Y. Yang, Q. Tian, and S. Wang, “Beyond part models: Person retrieval with refined
part pooling (and a strong convolutional baseline),” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11208
LNCS, pp. 501–518, 2018.

[13] P. Isola, J. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial
networks,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–
5976, July 2017.

[14] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image seg-
mentation,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intel-
ligence and Lecture Notes in Bioinformatics), vol. 9351, pp. 234–241, 2015.

[15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, June 2016.

[16] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. Alemi, “Inception-v4, inception-resnet and the impact
of residual connections on learning,” in 31st AAAI Conference on Artificial Intelligence, AAAI 2017,
pp. 4278–4284, 2017.

[17] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in 2018 IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pp. 7132–7141, June 2018.