Writing integrations

Goals and naming

For each integration, given a set of endpoints, get all information from those endpoints. Endpoints are, for example:

  • twitter: mentions (will get Tweet mentions from an account)
  • discord: stringify({channel: t1234, channelName: general, thread: true}) (get messages from a Thread)

It can be run in two ways:

  • onboarding: it will aim to get all past activities. This has to be triggered manually.
  • scheduled: it will get all activities up to a stopping point. This stopping point is defined per integration and/or per endpoint.

Main entities

  • Coordinators: scheduled functions that run every 20 minutes. They start a worker for each workspace that has set up the integration
  • Workers: workers initiate the iterators. They fetch the endpoints and get or refresh tokens.
  • Iterators: where the main functionality happens. They get all the activities and members from the target source, parse them and add them to the database.

Base iterator

The base iterator is an abstract class that contains high-level functionality. iterate is the main function. All integration-specific functionality is implemented in child classes. In a nutshell, iterate

  • checks we are not over the global limit
    * end

  • checks we are not at an end state

  • gets the activities from the source, parses them and adds them to the DB

  • next: gets the next state next page in endpoint, next endpoint or end
    next page in endpoint
    endpoint
    * end

  • isLimitReached: check if we are reaching the rate limit. This is implemented in the child classes.
    * call a new worker with a sleep parameter. Send the state.

  • check if we are close to the λ's 15min maximum runtime.
    * call a new worker without sleep. Send the state.

next

The next function obtains the next state given a current state.

  • If we are done we return end
  • Check if the endpoint is finished. It is finished if there is no next page, or with an integrationSpecificIsEndpointFinished (only called when onboarding=false) implemented in the child classes.
    end: if the endpoint is finished and we are at the last endpoint
    next endpoint: if the endpoint is finished and there are more endpoints
    * next page: otherwise

Discord

Endpoints: members, channels and threads. Channels and threads:

      {
        name,
          thread: true | false,
        new: true | false
      }

integrationSpecificIsEndpointFinished

  • endpoint is members: did the newest member returned join more than 20mins ago?
  • otherwise:
    is the thread/channel new? false
    did the newest message come more than 20mins ago?

Slack

Endpoints: members, channels and threads.
Channels:

      {
        name,
          new: true | false
      }

Threads:

      {
        name,
          threadId,
          channel: channelName,
          channelId,
          placeholder,
          new: true | false
      }

integrationSpecificIsEndpointFinished

  • endpoint is members: false (difference with Discord is that we do not get when they joined)
  • otherwise:
    is the thread/channel new? false
    did the newest message come more than 20mins ago?

Twitter

Endpoints: follows, mentions, hashtags

integrationSpecificIsEndpointFinished

  • endpoint is follows: have we seen that member before in this endpoint? (we do not get a timestamp for follows, but they are returned sorted)
  • otherwise:
    * did the newest tweet come more than 20mins ago?

Did this page help you?