Building Desktop Passport Scanner with Qt and USB Camera
If you search for passport scanner software or MRZ reader software, you will find many of them are only available for mobile devices. For police officers, scanning passports by mobile devices is convenient when they are patrolling. However, for customs and immigration officers, they usually use desktop system and professional passport scanner or reader, which cost a lot, to check passengers’ passport information. Dynamsoft’s OCR SDK is available for both mobile and desktop scenarios. In this article, I will demonstrate an economic way that uses a cheap USB web camera (less than $20), Qt, and Dynamsoft MRZ SDK to build a desktop passport scanner application for Windows and Linux.
This article is Part 1 in a 8-Part Series.
- Part 1 - Building Desktop Passport Scanner with Qt and USB Camera
- Part 2 - Making Web Passport MRZ Reader and Scanner in HTML5 and JavaScript
- Part 3 - How to Build Python MRZ Scanner SDK and Publish It to PyPI
- Part 4 - How to Recognize MRZ from Passport and ID Card with Node.js
- Part 5 - How to Make Java MRZ Detector with Dynamsoft Label Recognizer for Windows and Linux
- Part 6 - How to Create a Flutter plugin of Passport MRZ Recognition for Windows, Linux, Android, iOS and Web
- Part 7 - MRZ Recognition on Android Using Dynamsoft Label Recognizer SDK
- Part 8 - Developing a Desktop MRZ Scanner for Passports, IDs, and Visas with Dynamsoft C++ Capture Vision SDK
Prerequisites
- Dynamsoft Capture Vision Trial License
- Qt 5.12.11
- Windows
-
Linux
sudo apt-get install qt5-default
The Skeleton of Qt C++ Project for Desktop Passport Scanner
Before getting started, let’s get the codebase of the barcode scanning application that I implemented recently.
git clone https://github.com/yushulx/Qt-desktop-barcode-reader.git
The codebase has implemented the file loading and camera video streaming functions. What I need to do is to replace barcode recognition SDK with MRZ recognition SDK. In addition, the project needs to import extra character models (trained by Caffe) for OCR and a template file for providing MRZ recognition parameters.
Character model files
MRZ.caffemodel
MRZ.prototxt
MRZ.txt
Template file
{
"CharacterModelArray" : [
{
"DirectoryPath": "model",
"FilterFilePath": "",
"Name": "MRZ"
}
],
"LabelRecognizerParameterArray" : [
{
"BinarizationModes" : [
{
"BlockSizeX" : 0,
"BlockSizeY" : 0,
"EnableFillBinaryVacancy" : 1,
"LibraryFileName" : "",
"LibraryParameters" : "",
"Mode" : "BM_LOCAL_BLOCK",
"ThreshValueCoefficient" : 15
}
],
"CharacterModelName" : "MRZ",
"LetterHeightRange" : [ 5, 1000, 1 ],
"LineStringLengthRange" : [30, 44],
"MaxLineCharacterSpacing" : 130,
"LineStringRegExPattern" : "([ACI][A-Z<][A-Z<]{3}[A-Z0-9<]{9}[0-9][A-Z0-9<]{15}){(30)}|([0-9]{2}[(01-12)][(01-31)][0-9][MF<][0-9]{2}[(01-12)][(01-31)][0-9][A-Z<]{3}[A-Z0-9<]{11}[0-9]){(30)}|([A-Z<]{0,26}[A-Z]{1,3}[(<<)][A-Z]{1,3}[A-Z<]{0,26}<{0,26}){(30)}|([ACIV][A-Z<][A-Z<]{3}([A-Z<]{0,27}[A-Z]{1,3}[(<<)][A-Z]{1,3}[A-Z<]{0,27}){(31)}){(36)}|([A-Z0-9<]{9}[0-9][A-Z<]{3}[0-9]{2}[(01-12)][(01-31)][0-9][MF<][0-9]{2}[(01-12)][(01-31)][0-9][A-Z0-9<]{8}){(36)}|([PV][A-Z<][A-Z<]{3}([A-Z<]{0,35}[A-Z]{1,3}[(<<)][A-Z]{1,3}[A-Z<]{0,35}<{0,35}){(39)}){(44)}|([A-Z0-9<]{9}[0-9][A-Z<]{3}[0-9]{2}[(01-12)][(01-31)][0-9][MF<][0-9]{2}[(01-12)][(01-31)][0-9][A-Z0-9<]{14}[A-Z0-9<]{2}){(44)}",
"MaxThreadCount" : 4,
"Name" : "locr",
"TextureDetectionModes" :[
{
"Mode" : "TDM_GENERAL_WIDTH_CONCENTRATION",
"Sensitivity" : 8
}
],
"ReferenceRegionNameArray" : [ "DRRegion" ],
"Timeout": 20000
}
],
"LineSpecificationArray" : [
{
"Name":"L0",
"LineNumber":"",
"BinarizationModes" : [
{
"BlockSizeX" : 30,
"BlockSizeY" : 30,
"Mode" : "BM_LOCAL_BLOCK"
}
]
}
],
"ReferenceRegionArray" : [
{
"Localization" : {
"FirstPoint" : [ 0, 0 ],
"SecondPoint" : [ 100, 0 ],
"ThirdPoint" : [ 100, 100 ],
"FourthPoint" : [ 0, 100 ],
"MeasuredByPercentage" : 1,
"SourceType" : "LST_MANUAL_SPECIFICATION"
},
"Name" : "DRRegion",
"TextAreaNameArray" : [ "DTArea" ]
}
],
"TextAreaArray" : [
{
"LineSpecificationNameArray" : ["L0"],
"Name" : "DTArea",
"FirstPoint" : [ 0, 0 ],
"SecondPoint" : [ 100, 0 ],
"ThirdPoint" : [ 100, 100 ],
"FourthPoint" : [ 0, 100 ]
}
]
}
Desktop Passport Scanner for Windows and Linux
Since we have already got the codebase, it won’t take too much time to get the application work.
Library Linking
We extract the library files from the downloaded archive and put them into the corresponding folders:
-
Windows
Copy
DynamsoftLabelRecognizerx64.libtoplatform/windows/lib.Copy
DynamicPdfx64.dll,DynamsoftLabelRecognizerx64.dll,DynamsoftLicenseClientx64.dllandvcomp140.dlltoplatform/windows/bin. -
Linux
Copy
libDynamicPdf.so,libDynamsoftLabelRecognizer.so, andlibDynamsoftLicenseClient.sotoplatform/linux.
After that, we update the CMakeLists.txt file to link libraries and copy model and template files:
if (CMAKE_HOST_WIN32)
target_link_libraries(${PROJECT_NAME} PRIVATE Qt5::Widgets Qt5::MultimediaWidgets "DynamsoftLabelRecognizerx64")
elseif(CMAKE_HOST_UNIX)
target_link_libraries(${PROJECT_NAME} PRIVATE Qt5::Widgets Qt5::MultimediaWidgets "DynamsoftLabelRecognizer")
endif()
# Copy DLLs
if(CMAKE_HOST_WIN32)
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_directory
"${PROJECT_SOURCE_DIR}/platform/windows/bin/"
$<TARGET_FILE_DIR:${PROJECT_NAME}>)
endif()
# Copy template
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_directory
"${PROJECT_SOURCE_DIR}/template/"
$<TARGET_FILE_DIR:${PROJECT_NAME}>)
# Copy model files
add_custom_command(TARGET ${PROJECT_NAME} POST_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_directory
"${PROJECT_SOURCE_DIR}/model"
$<TARGET_FILE_DIR:${PROJECT_NAME}>/model)
Steps to Modify the Code for MRZ Recognition
Next, we import DynamsoftLabelRecognizer.h and DynamsoftCore.h to mainwindow.h:
#include "DynamsoftLabelRecognizer.h"
#include "DynamsoftCore.h"
In mainwindow.cpp, we search for the line of invoking DBR_DecodeFile() method and then replace the line with:
int errorCode = DBR_DecodeFile(reader, fileName.toStdString().c_str(), "");
Add the following code block to get passport information:
DLR_Result **results = handler->results;
for (int ri = 0; ri < handler->resultsCount; ++ri)
{
DLR_Result* result = handler->results[ri];
int lCount = result->lineResultsCount;
for (int li = 0; li < lCount; ++li)
{
DM_Point *points = result->lineResults[li]->location.points;
int x1 = points[0].x, y1 = points[0].y;
int x2 = points[1].x, y2 = points[1].y;
int x3 = points[2].x, y3 = points[2].y;
int x4 = points[3].x, y4 = points[3].y;
}
if (lCount < 2)
{
continue;
}
string line1 = result->lineResults[0]->text;
string line2 = result->lineResults[1]->text;
if (line1.length() != 44 || line2.length() != 44)
{
continue;
}
if (line1[0] != 'P')
continue;
else {
// Type
string tmp = "Type: ";
tmp.insert(tmp.length(), 1, line1[0]);
out += QString::fromStdString(tmp) + "\n";
// Issuing country
tmp = "Issuing country: "; line1.substr(2, 5);
tmp += line1.substr(2, 3);
out += QString::fromStdString(tmp) + "\n";
// Surname
int index = 5;
tmp = "Surname: ";
for (; index < 44; index++)
{
if (line1[index] != '<')
{
tmp.insert(tmp.length(), 1, line1[index]);
}
else
{
break;
}
}
out += QString::fromStdString(tmp) + "\n";
// Given names
tmp = "Given Names: ";
index += 2;
for (; index < 44; index++)
{
if (line1[index] != '<')
{
tmp.insert(tmp.length(), 1, line1[index]);
}
else
{
tmp.insert(tmp.length(), 1, ' ');
}
}
out += QString::fromStdString(tmp) + "\n";
// Passport number
tmp = "Passport number: ";
index = 0;
for (; index < 9; index++)
{
if (line2[index] != '<')
{
tmp.insert(tmp.length(), 1, line2[index]);
}
else
{
break;
}
}
out += QString::fromStdString(tmp) + "\n";
// Nationality
tmp = "Nationality: ";
tmp += line2.substr(10, 3);
out += QString::fromStdString(tmp) + "\n";
// Date of birth
tmp = line2.substr(13, 6);
tmp.insert(2, "/");
tmp.insert(5, "/");
tmp = "Date of birth (YYMMDD): " + tmp;
out += QString::fromStdString(tmp) + "\n";
// Sex
tmp = "Sex: ";
tmp.insert(tmp.length(), 1, line2[20]);
out += QString::fromStdString(tmp) + "\n";
// Expiration date of passport
tmp = line2.substr(21, 6);
tmp.insert(2, "/");
tmp.insert(5, "/");
tmp = "Expiration date of passport (YYMMDD): " + tmp;
out += QString::fromStdString(tmp) + "\n";
// Personal number
if (line2[28] != '<')
{
tmp = "Personal number: ";
for (index = 28; index < 42; index++)
{
if (line2[index] != '<')
{
tmp.insert(tmp.length(), 1, line2[index]);
}
else
{
break;
}
}
out += QString::fromStdString(tmp) + "\n";
}
}
}
DLR_FreeResults(&handler);
So far, the static image recognition is completed. In the following, we will implement real-time passport scanning by camera video stream.
To store MRZ recognition information and share it between threads, we create a new class MRZInfo:
#ifndef MRZINFO_H
#define MRZINFO_H
#include <QString>
class MRZInfo
{
public:
MRZInfo() = default;
~MRZInfo(){};
bool isNull();
public:
QString text;
int x1, y1, x2, y2, x3, y3, x4, y4, xx1, yy1, xx2, yy2, xx3, yy3, xx4, yy4;
};
#endif // MRZINFO_H
Open work.h to add a new slot function detectMRZ(), which works in a worker thread for recognizing MRZ:
void Work::detectMRZ()
{
while (m_bIsRunning)
{
QImage image;
m_mutex.lock();
// wait for QList
if (queue.isEmpty())
{
m_listIsEmpty.wait(&m_mutex);
}
if (!queue.isEmpty())
{
image = queue.takeFirst();
}
m_mutex.unlock();
if (!image.isNull())
{
// Detect MRZ
}
}
}
To recognize MRZ and extract passport information, we firstly convert QImage to ImageData and then call DLR_RecognizeByBuffer() method:
// Convert QImage to ImageData
ImageData data;
data.bytes = (unsigned char *)image.bits();
data.width = image.width();
data.height = image.height();
data.stride = image.bytesPerLine();
data.format = IPF_ARGB_8888;
data.bytesLength = image.byteCount();
QDateTime start = QDateTime::currentDateTime();
int errorCode = DLR_RecognizeByBuffer(recognizer, &data, "locr");
QDateTime end = QDateTime::currentDateTime();
DLR_ResultArray *handler = NULL;
DLR_GetAllResults(recognizer, &handler);
std::vector<MRZInfo> all;
QString out = "Elapsed time: " + QString::number(start.msecsTo(end)) + "ms\n\n";
DLR_Result **results = handler->results;
for (int ri = 0; ri < handler->resultsCount; ++ri)
{
DLR_Result* result = handler->results[ri];
int lCount = result->lineResultsCount;
if (lCount < 2)
{
continue;
}
DLR_LineResult *l1 = result->lineResults[0];
DLR_LineResult *l2 = result->lineResults[1];
string line1 = l1->text;
string line2 = l2->text;
if (line1.length() != 44 || line2.length() != 44)
{
continue;
}
if (line1[0] != 'P')
continue;
MRZInfo info;
DM_Point *points = l1->location.points;
int x1 = points[0].x, y1 = points[0].y;
int x2 = points[1].x, y2 = points[1].y;
int x3 = points[2].x, y3 = points[2].y;
int x4 = points[3].x, y4 = points[3].y;
DM_Point *points2 = l2->location.points;
int xx1 = points2[0].x, yy1 = points2[0].y;
int xx2 = points2[1].x, yy2 = points2[1].y;
int xx3 = points2[2].x, yy3 = points2[2].y;
int xx4 = points2[3].x, yy4 = points2[3].y;
// Type
string tmp = "Type: ";
tmp.insert(tmp.length(), 1, line1[0]);
out += QString::fromStdString(tmp) + "\n";
// Issuing country
tmp = "Issuing country: "; line1.substr(2, 5);
tmp += line1.substr(2, 3);
out += QString::fromStdString(tmp) + "\n";
// Surname
int index = 5;
tmp = "Surname: ";
for (; index < 44; index++)
{
if (line1[index] != '<')
{
tmp.insert(tmp.length(), 1, line1[index]);
}
else
{
break;
}
}
out += QString::fromStdString(tmp) + "\n";
// Given names
tmp = "Given Names: ";
index += 2;
for (; index < 44; index++)
{
if (line1[index] != '<')
{
tmp.insert(tmp.length(), 1, line1[index]);
}
else
{
tmp.insert(tmp.length(), 1, ' ');
}
}
out += QString::fromStdString(tmp) + "\n";
// Passport number
tmp = "Passport number: ";
index = 0;
for (; index < 9; index++)
{
if (line2[index] != '<')
{
tmp.insert(tmp.length(), 1, line2[index]);
}
else
{
break;
}
}
out += QString::fromStdString(tmp) + "\n";
// Nationality
tmp = "Nationality: ";
tmp += line2.substr(10, 3);
out += QString::fromStdString(tmp) + "\n";
// Date of birth
tmp = line2.substr(13, 6);
tmp.insert(2, "/");
tmp.insert(5, "/");
tmp = "Date of birth (YYMMDD): " + tmp;
out += QString::fromStdString(tmp) + "\n";
// Sex
tmp = "Sex: ";
tmp.insert(tmp.length(), 1, line2[20]);
out += QString::fromStdString(tmp) + "\n";
// Expiration date of passport
tmp = line2.substr(21, 6);
tmp.insert(2, "/");
tmp.insert(5, "/");
tmp = "Expiration date of passport (YYMMDD): " + tmp;
out += QString::fromStdString(tmp) + "\n";
// Personal number
if (line2[28] != '<')
{
tmp = "Personal number: ";
for (index = 28; index < 42; index++)
{
if (line2[index] != '<')
{
tmp.insert(tmp.length(), 1, line2[index]);
}
else
{
break;
}
}
out += QString::fromStdString(tmp) + "\n";
}
info.text = out;
info.x1 = x1;
info.y1 = y1;
info.x2 = x2;
info.y2 = y2;
info.x3 = x3;
info.y3 = y3;
info.x4 = x4;
info.y4 = y4;
info.xx1 = xx1;
info.yy1 = yy1;
info.xx2 = xx2;
info.yy2 = yy2;
info.xx3 = xx3;
info.yy3 = yy3;
info.xx4 = xx4;
info.yy4 = yy4;
all.push_back(info);
}
DLR_FreeResults(&handler);
surface->appendResult(all);
How to Build the Qt CMake Project
The CMake build commands are a bit different between Windows and Linux:
# Windows
mkdir build
cd build
cmake -G "MinGW Makefiles" ..
cmake --build .
MRZRecognizer.exe
# Linux
mkdir build
cd build
cmake ..
cmake --build .
./MRZRecognizer
Running Passport Scanner
When running the program, you need to enter a valid license key:
Then you can scan passport information from static images or webcam.
Source Code
https://github.com/yushulx/cmake-cpp-barcode-qrcode-mrz/tree/main/examples/qt_mrz