Writing an image classifier on .Net Maui

Spanish version: https://medium.com/@rretamal.dev/escribiendo-un-clasificador-de-im%C3%A1genes-en-net-maui-7e3b0da6da88

In this opportunity I wanted to do something different and among other things I wanted to write a small app to classify dogs.

So now that .Net Maui was recently released the question was: Why not?

The first to do in this case was to choose the technology to train the model, I did some initial testing using ML.Net, and I generated a preliminary classifier model but I couldn’t use it in a mobile app, for that the suggestion was to use an API but I wanted to have the processing of the images on the mobile side so finally I decided to use Tensorflow Lite.

Because I didn’t want to reinvent the wheel I applied transfer learning to an existing model based on MobileNetV2 that is oriented to mobile devices, but what is the cool thing about this? with the proper quantity of images you can retrain the model to classify and recognize any kind of objects with an acceptable level of efficiency and performance.

In case you want to get more information about the model you can go to the following link:

For the next step you need dog images, a lot of dogs, so I used the following dataset:

Once you decided the model to use and with the dataset prepared with the images we need to do the transfer learning and the result of this operation is going to be a .tflite file and a labels.txt file with the name of the dog's classes.

The following is a tutorial that I used for the transfer learning, in my case I’m getting the images from a zip file on my Google Drive, training the model and generating the files:

So once you have the files the next step is to create the Maui project on Visual Studio 2022.

Next, you have to set a super cool name for your project and select a proper location:

So perfect! we have our project created!

The next thing was to get the libraries to use Tensorflow on Xamarin, I found a couple that were outdated, so in this case was a better option to create a Java Binding project to use directly the Tensorflow library (aar).

We are going to structure the app in the following way:

• Maui.DogClassifier: Maui project with the UI of the app and is where we are going to capture the images
• Maui.DogClassifier.Core: the project with the logic to load the model and classify the images
• Maui.DogClassifier.Lib: the binding project to use the aar library of Tensorflow

Before anything as you can remember the training process is going to generate two files: a tflite and labels.txt file, those files must be in your Raw folder of the project like in the image, at the moment of adding the files they will be added as a MauiAsset automatically.

Now lets execute the project as it is:

Perfect! we have a Maui app running. Now we need to change the interface to capture the image, show it and show the result of the classification process.

To keep the example simple we are only going to change the button, add a picture control to show the photo and an extra label to show the result, the code is going to look like the following example:

<?xml version="1.0" encoding="utf-8" ?>
<ContentPage xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             x:Class="Maui.DogClassifier.MainPage">
			 
    <ScrollView>
        <VerticalStackLayout 
            Spacing="25" 
            Padding="10,0" 
            VerticalOptions="Center">


            <Image
                Source="dotnet_bot.png"
                SemanticProperties.Description="Cute dot net bot waving hi to you!"
                HeightRequest="150"
                HorizontalOptions="Center" />
                
            <Label 
                Text="Hello, Dog!"
                SemanticProperties.HeadingLevel="Level1"
                FontSize="30"
                HorizontalOptions="Center" />
            
            <Label 
                Text="Please take a picture of your dog"
                SemanticProperties.HeadingLevel="Level2"
                SemanticProperties.Description="Please take a picture of your dog"
                FontSize="18"
                HorizontalOptions="Center" />


            <Button Text="Take picture" x:Name="btnCapture" />


            <Image x:Name="imgCapture" IsVisible="false" HeightRequest="300"/>


            <Label 
                x:Name="lblResult"
                Text="It's a !"
                SemanticProperties.HeadingLevel="Level2"
                FontSize="18"
                HorizontalOptions="Center" IsVisible="false"/>




        </VerticalStackLayout>
    </ScrollView>
 
</ContentPage>

Next in our MainPage.xaml.cs file we have to do two things, create our Classifier (we will create that after this) and handle the camera actions like this:

public partial class MainPage : ContentPage
{	int count = 0;
	Classifier classifier;


	public MainPage()
	{
		InitializeComponent();
		classifier = new Classifier();


    }


    private async void BtnCapture_Clicked(object sender, EventArgs e)
    {
        var photo = await MediaPicker.CapturePhotoAsync();


        await LoadPhotoAsync(photo);


        byte[] image = File.ReadAllBytes(photo.FullPath);


        var result = classifier.ClassifyImage(image);


        lblResult.IsVisible = true;


        if (result != null)
        {
            if (result.Percentage > 0.7)
            {
                lblResult.IsVisible = true;
                lblResult.Text = $"It's a {result.Name}! ({result.Percentage.ToString()})";
                return;
            }
        }


        lblResult.Text = "Dog not recognized, please try again";
    }


    async Task LoadPhotoAsync(FileResult photo)
    {
        // save the file into local storage
        var newFile = Path.Combine(FileSystem.CacheDirectory, photo.FileName);
        using (var stream = await photo.OpenReadAsync())
        using (var newStream = File.OpenWrite(newFile))
        {
            await stream.CopyToAsync(newStream);
        }


        imgCapture.Source = newFile;
        imgCapture.Aspect = Aspect.Center;
    }
}

As you can see we created the classifier object and when the user clicked the button we will open the camera calling the method MediaPicker.CapturePhotoAsync to get the image to send it to the classifier.

The classifier in this case is going to return the best result with the class name and the probability.

Now we are going to create our classifier, for that in the library project we will see a folder for each platform, on each we need to create the class to be invoked dynamically by the Maui project based on the platform that is running (that is so cool!), so we will create the file Classifier.cs, the result will look like the following image:

For the moment we are going to focus on the Android project. The first step is to load our model from the Raw folder:

private MappedByteBuffer LoadModel()
{
    var assets = Android.App.Application.Context.Assets;


    var assetDescriptor = assets.OpenFd("model.tflite");
    var inputStream = new FileInputStream(assetDescriptor.FileDescriptor);


    var mappedByteBuffer = inputStream.Channel.Map(FileChannel.MapMode.ReadOnly, assetDescriptor.StartOffset, assetDescriptor.DeclaredLength);


    return mappedByteBuffer;
}

Second, we will load the labels cleaning a little bit the text of the labels:

private List<string> LoadLabels()
{
    var assets = Android.App.Application.Context.Assets;


    var streamReader = new StreamReader(assets.Open("labels.txt"));


    //Transform labels.txt into List<string>
    var labels = streamReader.ReadToEnd().Split('\n').Select(c => c.Split('-')[1]).ToList();


    return labels;
}

And third, we will classify the image using the interpreter of Tensorflow:

public ImageResult ClassifyImage(byte[] image)
{
    var tensor = interpreter.GetInputTensor(0);
    var shape = tensor.Shape();


    var width = shape[1];
    var height = shape[2];


    var byteBuffer = GetPhotoAsByteBuffer(image, width, height);


    var outputLocations = new float[1][] { new float[labels.Count] };
    var outputs = Java.Lang.Object.FromArray(outputLocations);


    interpreter.Run(byteBuffer, outputs);
    var classificationResult = outputs.ToArray<float[]>();


    //Map the results
    var classificationModelList = new List<ImageResult>();


    for (var i = 0; i < labels.Count; i++)
    {
        var label = labels[i]; classificationModelList.Add(new ImageResult()
        {
            Name = label,
            Percentage = classificationResult[0][i]
        });
    }


    var top = classificationModelList.OrderByDescending(c => c.Percentage).FirstOrDefault();


    return top;
}

So what do we need to do now? pray it works! haha!

The initial screen is going to look like this:

On the second screen I’m going to introduce my dog Oky!

The classifier result of the photo was a Cocker Spaniel with a 99% of probability which is very good!

You can find the full code in the following repo:

https://github.com/rretamal/DogClassifier

Final comments:

The training of the model was a project itself, I tried different models receiving better or worse results. You can check articles that explain how to improve the models (there is always space to improve). The important thing here is the infinite application for Machine Learning models and how easy is to implement those models on mobile apps. .Net Maui also has the best things of the Xamarin apps pushed to a whole new level with performance improvements, a new ecosystem of controls, the multiplatform management that is so cool and more.

These are some things that I want to mention:

• You need to have a good amount of images, a lot! so you could train properly the model otherwise you will get weir results.
• Google Colab has excellent resources for free and with GPU use.
• Although I used a pre-trained model and did a transfer learning you can build your own models and improve them continuously, combine different models, increase your training data, play with the epochs, etc. This article is very interesting on how they improved a classifier: https://towardsdatascience.com/building-the-hotdog-not-hotdog-classifier-from-hbos-silicon-valley-c0cb2317711f

Happy Coding!