Tutorial: CraftAR On-device Image Recognition for iOS

Dec 22, 2020

Catchoom Team

This article applies only to the On-device Image Recognition SDK.

In order to add on-device collections to your iOS app, it is necessary to:

create an on-device collection in CraftAR;
generate a collection bundle for your SDK version (On-device Image Recognition SDK or Augmented Reality SDK v4 +;
download the bundle of that collection from CraftAR and add it to the app;
you are ready to go and start using the on-device collection with the SDK

Once the collection is added to the device, you can start using the on-device Image Recognition SDK to perform visual search queries on the device. You can find more details for the first step in the tutorial about how to create an on-device collection in CraftAR. And for the second and third steps, please read the tutorial about how to Manage on-device collections for the iOS SDKs.

An Image Recognition app using the native On-device iOS Image Recognition SDK can be implemented by following three steps: first, set up the UIVIewController, then load the collection for on-device Image Recognition and, finally run image recognition to get the results for each item that is recognized.

If you want to see an example that implements On-device Image Recognition, take a look at the open source samples available in our Github repository

1. Set up the SDK in your UIViewController

Adopt CraftARSDKProtocol and SearchProtocol in your UIViewController

@interface MyViewController () <CraftARSDKProtocol, SearchProtocol> {
    CraftARSDK_IR *_sdk;
    CraftAROnDeviceIR *_oir;
}
@end

@interface MyViewController () <CraftARSDKProtocol, SearchProtocol> {

CraftARSDK_IR *_sdk;

CraftAROnDeviceIR *_oir;

}

@end

Set up the camera capture

The CraftARSDK_IR class manages the camera capture. Once the view is loaded, get the instance of the CraftARSDK_IR and start the camera capture with a UIView to show the video feed from the camera.

- (void) viewWillAppear:(BOOL) animated {
    [super viewWillAppear:animated];
    // setup the CraftAR SDK
    _sdk = [CraftARSDK sharedCraftARSDK];
    // Become delegate of the SDK to receive capture initialization callbacks
    _sdk.delegate = self;
    // Start Video Preview for search
    [_sdk startCaptureWithView: self._preview];
}

- (void) viewWillAppear:(BOOL) animated {

[super viewWillAppear:animated];

// setup the CraftAR SDK

_sdk = [CraftARSDK sharedCraftARSDK];

// Become delegate of the SDK to receive capture initialization callbacks

_sdk.delegate = self;

// Start Video Preview for search

[_sdk startCaptureWithView: self._preview];

}

Loading collections into memory

Once the collection is added to the local database, we can set the collection that will be used for on-device image recognition using the CraftAROnDeviceIR instance. Loading collections can take a few seconds which depend on the amount of images. The SDK provides progress feedback for this process as well.

- (void) loadDemoCollection: (CraftARCollection*) collection {
    MainScreenViewController* myself = self;
    // Load the collection before doing any searches
    [mOnDeviceIR setCollection:collection setActive:YES withOnProgress:^(float progress) {
        NSLog(@"Load collection progress: %f", progress);
    } onSuccess:^{
        // Now the collection is ready for recognition
    } andOnError:^(CraftARError *error) {
        NSLog(@"Error adding collection: %@", [error localizedDescription]);
    }];
}

- (void) loadDemoCollection: (CraftARCollection*) collection {

MainScreenViewController* myself = self;

// Load the collection before doing any searches

[mOnDeviceIR setCollection:collection setActive:YES withOnProgress:^(float progress) {

NSLog(@"Load collection progress: %f", progress);

} onSuccess:^{

// Now the collection is ready for recognition

} andOnError:^(CraftARError *error) {

NSLog(@"Error adding collection: %@", [error localizedDescription]);

}];

}

Prepare the SDK to process searches

The CraftARSDK_IR class also manages the search processes by sending search events to the searchControllerDelegate, in this case the CraftAROnDeviceIR. The CraftAROnDeviceIR class performs visual searches in the collection of images that is previously set for this app. You can find more details about setting the bundles in the separate tutorial about how to manage collection bundles with the On-device Image Recognition SDK.

Once the VideoCapture is ready, it performs a callback to didStartCapture. We suggest to set the CraftAROnDeviceIR's search controller instance as the searchControllerDelegate, and the UIViewController as the delegate of the CraftAROnDeviceIR to receive the responses from the search process.

- (void) didStartCapture {
    self._previewOverlay.hidden = NO;
    // Get the CraftAROnDeviceIR instance (for on-device searches)
    // and set it as the search controller delegate for the SDK
    _oir = [CraftAROnDeviceIR sharedOnDeviceIR];
    _sdk.searchControllerDelegate = _oir.mSearchController;
    // Set the view controller as delegate of the CraftAROnDeviceIR to receive the
    // search results
    _oir.delegate = self;
}

- (void) didStartCapture {

self._previewOverlay.hidden = NO;

// Get the CraftAROnDeviceIR instance (for on-device searches)

// and set it as the search controller delegate for the SDK

_oir = [CraftAROnDeviceIR sharedOnDeviceIR];

_sdk.searchControllerDelegate = _oir.mSearchController;

// Set the view controller as delegate of the CraftAROnDeviceIR to receive the

// search results

_oir.delegate = self;

}

2. Implementing the searches and parsing the results

Once the ViewController has the necessary protocols and instance of CraftAROnDeviceIR, it’s time to add code to start scanning the real world.

Using the CraftARSDK_IR class, we can search the on-device image database in two modes: Finder Mode or Single Shot Mode.

Option A. Use Single Shot Mode to take a single picture

Call singleShotSearch to perform a search with a single image. By calling this method, the SDK triggers the still image capture from the camera and forwards the captured image to the CraftAROnDeviceIR.

A common approach consists in triggering the search when the user performs an action on the UI:

- (IBAction)snapPhotoToSearch:(id)sender {
   [_sdk singleShotSearch];
}

- (IBAction)snapPhotoToSearch:(id)sender {

[_sdk singleShotSearch];

}

The response to a query produces a callback to didGetSearchResults with the results available in an array that is ready to parse.

- (void) didGetSearchResults:(NSArray *)results {
    // Parse the results
    for (CraftARItem* item in results) {
        // Obtain information/experience for each item
    }
    // Restart the VideoCapture that the singleShotSearch freezes.
   [[_sdk getCamera] restartCapture];
}

- (void) didGetSearchResults:(NSArray *)results {

// Parse the results

for (CraftARItem* item in results) {

// Obtain information/experience for each item

}

// Restart the VideoCapture that the singleShotSearch freezes.

[[_sdk getCamera] restartCapture];

}

Option B. Use Finder Mode for continuous scanning

Call startFinder to start searching continuously without user intervention. This method forwards the camera frames to the CraftAROnDeviceIR instance.

- (void) mySearchFunction {    
    // Start scanning
    [_sdk startFinder];
}

- (void) mySearchFunction {

// Start scanning

[_sdk startFinder];

}

For every frame that is processed, the SDK generates a query and searches for matches in the local collection.