Group Chat Backend

A group chat backend

Learning challenges: #

• Real time messages processing ( broadcasting messages to users of the same channel ).
• Saving messages to a database for persistence as a decoupled service ( Processing Redis Streams )
• Managing database between two different service ( ChatApp and MessageSyncApp)
• Clean Architecture

Key skills built: #

• Express, Sequeilze, PostgresSQL
• Handling websocket connections
• Error handling, Typescript

About the Project #

You can find the github repository here

The goal of the project is to build a group chat backend, where

• After a user is authenticated, he/she can create a channel and other users can join that channel.
• When a message is sent to the server, it is broadcasted to the users online and saved to a database for persistence ( saving messages will be a separate app to avoid bogging down the chatApplication )
• channels are long lasting web socket servers with a unique ID.

The project has three main parts:

• chatService ( handles CRUD operations for users and channels also displays messages of a channel )
• messageSyncingService ( saves messages to database for persistence )
• websocketServer ( handles messages, also broadcasts messages to connected users )

I’ll be using express for the api and Sequeilze as an ORM with PostgresSQL as the database, (mention reids and docker here)

I’ll be following the Router-Service-Controller architecture with the following folder structure:

|---chatService
|---.env
|   |--src
|      |---config
        |---db
        |---controller
        |---service
        |---router

• Routes: Declares API endpoints and maps them to respective controllers.

• Controllers: Handle the logic behind validating the request parameters, query and sending appropriate responses to those requests by calling the appropriate service methods.

• Services: Responsible for business logic and data processing. It acts as a bridge between the controller layer and the data access layer ( sequeilze in my case ), providing an interface for the controller to interact with the data access layer.

ChatService #

This handles the CRUD operations for users and channels also displays message history of a channel.

The user CRUD is divided into signup and login:

Signup: #

• User sends their credentials to the /signup route.
• The credentials are validated against constraints ( valid email, unique username, etc ).
• A hashed password is stored in the Database and a JWT identifying the user is created and the created user is sent as a response.

Login: #

• User sends their credentials to the /login route.
• The credentials are validated and verified.
• Upon successful verification, a JWT identifying the user is set as cookie.

The CRUD operation for the channels is:

• Authenticated user sends channel credentials to /create-channel
• The credentials are validated ( unique name ).
• Upon successful creation, a uuid of the server is returned to the user.
• Other users can join the server using that uuid as a url parameter in the /join-channel route.

Before I discuss how the messages are saved and displayed, let’s take a closer look at the database design, websocketServer, and chatSyncingService.

Database #

The database schema is as follows:

As you can see the main tables are servers, users and messages. The messages and server_users tables are JOIN tables who establish a many-to-many relationship between users and servers, tracking what users are in which servers and what messages belong to what users and servers, both have server_id and user_id which are primary keys ( id ) in the servers and users tables respectively.

The JOIN tables are created through sequelize associations:

DB.user.belongsToMany(DB.server, {
  through: DB.serverUser,
  foreignKey: "user_id",
  otherKey: "server_id",
});
DB.server.belongsToMany(DB.user, {
  through: DB.serverUser,
  foreignKey: "server_id",
  otherKey: "user_id",
});
DB.messages.hasOne(DB.user, { foreignKey: "id" });
DB.messages.hasOne(DB.server, { foreignKey: "id" });

WebsocketServer #

The Websocket server is responsible for real time full-duplex communication between the users and the servers.

When a user connects to a server,the messages he sends are broadcasted to all other users connected to the same server and the messages are pushed into a redis stream which is read by the messages syncing service and stored in the database.

Here’s the breakdown of the broadcasting logic:

• User specifies which server he want’s to send messages to through the query param ws:localhost:3000/:channelId
• The Websocket server verifies if the channelId exists and if the user is a part of that channel through a method in the serverService.
• Upon successful validation, the user is added in a in memory object which tracks the connections of servers.
• If the user sends a message, it is broadcasted to ever other active user in that channel via the object.

Let’s talk about the object which tracks connections: #

• It is a map of sets:

const connections: Map<string, Set<WebSocket>> = new Map();

• The logic goes as follows:

if (connections.get(channelId)) {
    connections.get(channelId)!.add(ws);
} else {
    const wsSet: Set<WebSocket> = new Set([ws]);
    connections.set(channelId, wsSet);
}

Basically, upon connection if the channelId already exists in the object then add the connection to the map and if the channelId does not exist in the object then add it in the object along with a set containing the current websocket connection.

MessageSyncingService #

This is responsible for writing messages to the database for persistence, I’ve made this a separate service to decouple it from the chat service to avoid slowing it down.

The problem with two services sharing the same database table #

The MessageSyncingService would write the messages to the messages table and the ChatService would read form it, which means conflicts and data inconsistencies are not likely to occur.

But there’s still a problem. If I track the same table ( messages ) across two services, both of the models should exactly be the same.

Sequelize offers model synchronization, where a model can be synchronized with the database by calling model.sync(options) but still this is not a viable option because:

• If only one of the service has that option set then, if the state of the model is changed in that service, the other service would error out essentially meaning one of the service is the dominant one.

• If both the models have this option, there is a possibility that they keep changing the database to fit their model, possibly breaking the app.

Possible solutions #

• Have a separate service that handles the database, if any other service wants to access the database it would do it through that service.

• Have an external mechanism to manage migrations to enforce a schema that both the models will follow.

Solution #

For my current use-case, having external migrations seems to be the best option because it’s relatively simple and get’s the job done.

I’m using a library called SlonikMigrator to handle the migrations. Here are some migrations form the project.

• Creating the users table:

CREATE TABLE IF NOT EXISTS users
(
    id         BIGSERIAL    NOT NULL PRIMARY KEY,
    username   VARCHAR(50)  NOT NULL UNIQUE,
    email      VARCHAR(50)  NOT NULL UNIQUE,
    password   VARCHAR(200) NOT NULL,
    created_at DATE DEFAULT NOW(),
    updated_at DATE DEFAULT NOW()

);

• Creating the messages table:

CREATE TABLE IF NOT EXISTS messages
(
    id        BIGSERIAL NOT NULL,
    user_id    BIGINT REFERENCES users(id),
    server_id  BIGSERIAL REFERENCES servers(id),
    value     TEXT DEFAULT NULL,
    created_at DATE DEFAULT NOW(),
    updated_at DATE DEFAULT NOW()
);

These are the up migrations, the down migrations would simply be:

• Users:

DROP TABLE IF EXISTS public.users CASCADE;

• Messages:

DROP TABLE IF EXISTS public.messages CASCADE;

Conclusion #

So that’s it, This article covers all the major decision I’ve made when building this project, for the entire code you can visit the github repository here