Python TypeError: Descriptors cannot not be created directly

While working with Python’s machine learning and artificial intelligence packages, you may encounter an error as follows:

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.

If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

This error occurs because the protobuf library introduced breaking changes in version 4.21.0, specifically:

Python upb requires generated code that has been generated from protoc 3.19.0 or newer.

The following tutorial shows why this error occurs and ways you can fix it.

Why TypeError: Descriptors cannot not be created directly occurs

This error may happen when you install or run any Python code that uses the protobuf package.

The protobuf package is a dependency for many machine learning and AI packages, so the package can be installed without your command.

For example, the tensorflow, ray, google-api, and azureml libraries all has the protobuf package as a dependency.

If you use any one of these libraries, then you need to fix the error in one of two ways:

• Downgrade protobuf to version 3.20.x
• If your ML and AI libraries have fixed the error, upgrade them to the latest version
• Export the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python variable

Let’s see how to fix this error next.

How to fix TypeError: Descriptors cannot not be created directly

To easiest way to fix this error is to downgrade the protobuf library installed on your Python system to version 3.20.x

Use one of the following commands to install protobuf version 3.20.x:

pip install 'protobuf==3.20.*'

# For pip3:
pip3 install 'protobuf==3.20.*'

Once done, you can check the protobuf package version by running the pip show command as follows:

pip show protobuf
#or
pip3 show protobuf

Notice the version number in the output below:

Name: protobuf
Version: 3.20.3
Summary: Protocol Buffers
Home-page: https://developers.google.com/protocol-buffers/
Author: 
Author-email: 
License: BSD-3-Clause
Location: /opt/homebrew/lib/python3.10/site-packages
Requires: 
Required-by: tensorboard

If you have a requirements.txt file, you can specify a dependency to protobuf version 3.20.x as follows:

###### requirements.txt ######
protobuf==3.20.*

This way, the protobuf version should be compatible with your ML libraries.

Another way to fix this error is to upgrade your ML libraries.

For example, TensorFlow version 2.7.3 fixed this error by specifying the max version range for the protobuf package.

Upgrade the library with one of the following commands:

pip install tensorflow --upgrade
# or:
pip3 install tensorflow --upgrade

Other libraries might have a similar fix released, but you need to check their websites for more information.

The third solution to this error is to export the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python variable.

You need to run the following command from the terminal:

# Linux / macOS
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

# Windows Command Prompt
setx PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION python

# PowerShell
$Env:PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION="python"

If you’re deploying to a cloud-based service such as Azure or Google Cloud, then find out how to add an environment variable to your deployment to add the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION variable.

But keep in mind that this solution uses a pure Python implementation for protocol parsing, so processing your code might be slower.

Conclusion

The TypeError: Descriptors cannot not be created directly is caused by a breaking change in protobuf version 4.

To fix this error, you can downgrade protobuf to version 3.20.x or set the environment variable to PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python.

Some libraries that depend on protobuf have addressed this error by adding the upper bound spec for protobuf to only install version 3.20.x.

I hope this tutorial was helpful. Until next time! 🙏