Send and run the model

Now that the model is prepared and exported in ONNX Format. We will see how to use BlindAI to deploy and run DistilBERT. 
Step 1: Install the client SDK
BlindAI Client is a python package that provides a simple and straightforward way to connect with BlindAI Server.
You can install the latest version of the client using pip.
1
pip install blindai 
Copied!
For now, the library is only compatible with Linux. We are working on a native Windows version, but If you are using Windows, you can still use the library with Windows Subsystem for Linux in the meantime. 
 Step 2: Send the model
You can run the following script in order to send the model to the server:
1
from blindai.client import BlindAiClient, ModelDatumType
2
​
3
client = BlindAiClient()
4
​
5
client.connect_server(addr="localhost", simulation=True)
6
​
7
client.upload_model(
8
    model="./distilbert-base-uncased.onnx", 
9
    shape=inputs.shape, 
10
    dtype=ModelDatumType.I64
11
    )
Copied!
The client is straightforward, we require an address, so if you have loaded the inference server on the same machine, simply mention "localhost" as we did. For simplicity, in the simulation mode, connect_server simply creates an insecure channel with the server. This is meant as a quick way to test without requiring specific hardware, so do not use the simulation mode in production.
For the upload_model method, we need to specify the ONNX file, the shape of the inputs, and the type of data. Here because we run a BERT model, the inputs would be integers to represent the different tokens sent to the model.
For more details about the client API, check the API reference.
Step 3: Run the inference
The process is as straightforward as before, simply tokenize the input you want before sending it. As of now, the tokenization must happen at the client-side, but we will implement it shortly in the server-side, so that the client interface remains lightweight.
1
from transformers import DistilBertTokenizer
2
​
3
# Prepare the inputs
4
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased")
5
sentence = "I love AI and privacy!"
6
inputs = tokenizer(sentence, padding = "max_length", max_length = 8)["input_ids"]
Copied!
Now we simply have to create our client, connect and send data to be analyzed. In the same fashion as before, we will create a client in simulation, and simply send data to be analyzed with the proper communication channel.
1
from blindai.client import BlindAiClient
2
​
3
# Load the client
4
client = BlindAiClient()
5
client.connect_server("localhost", simulation=True)
6
​
7
# Get prediction
8
response = client.run_model(inputs)
Copied!
For more details about the client API, check the API reference.
​

Prepare the model

Next - Getting started

Cloud deployment

Last modified 1mo ago

Copy link

Contents

Step 1: Install the client SDK

Step 2: Send the model

Step 3: Run the inference