All Python software is contained in the crowd namespace, and the modules are in subfolders named accordingly. E.g., the module to calculate engagement score for community members resides in
crowd-members-score and provides its functionality in a
members_score folder under the
requirements.txt file contains the list of internal modules to install. The modules specify their requirements in their
requirements.dev.txt contains dev requirements for formatting, linting, testing, and
The environment needs to be s to develop, and some hooks must be configured.
- Python 3.8
The Python modules should be installed in a
venv-crowd virtual environment. There are two ways to do this: installing the whole crowd.dev backend suite, including the Node.js modules, or setting it up manually.
Installing the whole suite will automatically create the virtual environment and add all necessary packages. This is the more straightforward and recommended approach.
cd crowd.dev/backend && npm ci
- Navigate to
- Create a virtual environment with
python3.8 -m venv venv-crowd
- Activate it on unix/Windows with Git Bash
source venv-crowd/bin/activateor on Windows
- Install all crowd modules (they will be installed in editable mode):
pip install -r requirements.txt
- Install the dev modules
pip install -r requirements.dev.txt
- Setup the pre-commit hook for formatting and linting
- Setup the pre-push testing hook on Unix
makeor on Unix/Windows
cp hooks/pre-push .git/hooks
- verify that everything works with running
The Python code can be split into: a simple backend module and the microservices that make use of that module.
The backend performs most of the necessary operations, such as querying, sending messages to write to the DB, and managing the Serverless infrastructure; and microservices that perform business logic and features using that backend.
This module contains several microservices:
- crowd-check-merge-members: The check merge microservice is the main component of the cross-platform matching feature; it is responsible for detecting duplicate members across various platforms to merge them automatically.
- crowd-members-score: This is the main microservice used for calculating the engagement level of the community members.
- crowd-github: this performs the GitHub integration's onboaring. It will be moved to Node.js soon.
Each of these services has a
setup.py file which we use to install the module and its corresponding required packages.
For unit testing, you can either:
pytestin the root of python (in
backend/src/serverless/microservices/python/) to run all python microservices-related tests.
cdinto a specific microservice and run
pytestthere to run unit tests related only to that specific microservice.
If you set up
pre-push correctly, all tests must succeed before pushing your changes to remote.
Updated 4 months ago