Python data services

All Python software is contained in the crowd namespace, and the modules are in subfolders named accordingly. E.g., the module to calculate engagement score for community members resides in crowd-members-score and provides its functionality in a members_score folder under the crowd namespace.

The requirements.txt file contains the list of internal modules to install. The modules specify their requirements in their setup.py's install_requires.

The requirements.dev.txt contains dev requirements for formatting, linting, testing, and pre-commit-hooks.

Setup

The environment needs to be s to develop, and some hooks must be configured.

Requirements

  • Python 3.8

Steps

The Python modules should be installed in a venv-crowd virtual environment. There are two ways to do this: installing the whole crowd.dev backend suite, including the Node.js modules, or setting it up manually.

Install the whole suite

Installing the whole suite will automatically create the virtual environment and add all necessary packages. This is the more straightforward and recommended approach.

cd crowd.dev/backend && npm ci

Set up Python only manually

  • Navigate to backend
  • Create a virtual environment with python3.8 -m venv venv-crowd
  • Activate it on unix/Windows with Git Bash source venv-crowd/bin/activate or on Windows venv-crowd\Scripts\activate.bat
  • Install all crowd modules (they will be installed in editable mode): pip install -r requirements.txt
  • Install the dev modules pip install -r requirements.dev.txt

Hooks

  • Setup the pre-commit hook for formatting and linting pre-commit install
  • Setup the pre-push testing hook on Unix make or on Unix/Windows cp hooks/pre-push .git/hooks
  • verify that everything works with running pytest

Structure

The Python code can be split into: a simple backend module and the microservices that make use of that module.

The backend performs most of the necessary operations, such as querying, sending messages to write to the DB, and managing the Serverless infrastructure; and microservices that perform business logic and features using that backend.

This module contains several microservices:

  • crowd-check-merge-members: The check merge microservice is the main component of the cross-platform matching feature; it is responsible for detecting duplicate members across various platforms to merge them automatically.
  • crowd-members-score: This is the main microservice used for calculating the engagement level of the community members.
  • crowd-github: this performs the GitHub integration's onboaring. It will be moved to Node.js soon.

Each of these services has a setup.py file which we use to install the module and its corresponding required packages.

Testing

For unit testing, you can either:

  • Run pytest in the root of python (in backend/src/serverless/microservices/python/) to run all python microservices-related tests.
  • cd into a specific microservice and run pytest there to run unit tests related only to that specific microservice.

Note:
If you set up pre-push correctly, all tests must succeed before pushing your changes to remote.