Practical Experiences From Full Stack Development of A Single Page Web App
Background
About 2 years ago, I summarized my several years’ software development experience and proposed some architectural design ideas in my article “Single Page Web App UI Development Thoughts”. However, these ideas were just rough ideas without being practiced anywhere. In the last 2 years, though in my formal job, I have worked on both frontend and backend, I only contributed features to different components of these systems and didn’t get a chance to design and implement a whole system using my ideas. So I built a side project from scratch to practice my ideas.
The side project is called Lingxijiao (demo: www.lingxijiao.com , github: https://github.com/swortaljuju/lingxijiao/tree/master/app ). It is a tiny dating oriented social app mainly for Chinese people. The core idea is to help strangers to connect with each other only when they confirm mutual interest to each other through simple but sharp posts and responses.
At the very beginning, I thought this project would only take 1 - 2 months of my spare time. But it actually took about 4 months of my spare time to develop it and another 4 months to let the target user group know the existence of this app. The time was spent on solving technical challenges encountered when using the latest technologies and on dealing with human problems related to that user group’s founder. In this article, I will talk about the technical stack and challenges of my side project, my experience of practicing my architectural ideas in a web app and lessons learned from human problems.
Technical Stack and Challenges
Language: TypeScript.
TypeScript is used as the only programming language in my project in build system, backend and frontend.
It is typed JavaScript and has many features of object oriented design such as class, interface, type check while still preserving JavaScript’s flexible nature. So it has good readability and is good for large team projects.
It also has good JavaScript compatibility. JavaScript library can be directly used in TypeScript projects with only an additional type declaration file. However, there are several technical challenges a developer may encounter when using TypeScript:
Not all JavaScript libraries have a declaration file for TypeScript or the declaration file could be outdated. A developer may need to add the declaration file by himself.
Due to JavaScript’s flexible types, some types in TypeScript declaration files can be very complicated, full of generics and hard to use.
Build system: Webpack
Webpack is used as the build tool for both frontend and backend since I use TypeScript for both frontend and backend. It compiles TypeScript, SASS and tsx, adds library dependencies into the built files, injects environment constants into code and copies resources to the deployment folder. For the backend, there is no need to minify the compiled file because the server has enough memory for the code, doesn’t need to download the code and some backend libraries such as Typegoose which relies on reflection will have problems after minification.
The advantage is Webpack contains both compiler feature and task executor feature which is equivalent to Bower + Grunt/Gulp. It also has many plugins, covering both build and custom tasks. And it supports both frontend and backend.
The challenge is some backend NodeJs libraries don’t support webpack. For example, node-pre-gyp, a cpp compiler and executor for NodeJs, doesn’t support webpack. It will throw errors when building any NodeJs library relying on node-pre-gyp. The solution is making the NodeJs library as external dependency and make it accessible to built executables instead of merging it into the executables.
Environment Config: Dotenv
Dotenv supports loading different configuration files based on the environment. It is used to load environment variables such as production db credentials, logging level, service addresses, constants and so on.
I18N framework: I18Next
Both frontend and backend needs I18n support.
There are multiple I18N libraries such as I18Next and Globalizejs.
I18Next has following major characteristics
Simple: easy to configure.
Support various message formatting and interpolation;
Popular;
Plenty of middleware such as locale detectors which determine locale on both server side and client side.
Support both frontend and backend.
Globalizejs is more comprehensive supporting more types of localization such as date, currency and number while I18Next only supports message localization. But because of that, GlobalizeJs requires downloading and configuring more localization metadatas such as currency, number, dates i18n data. So the initialization logic is complicated. My app is a social media app which only needs message localization. So I picked the simpler one I18Next.
Database: Mongodb
Mongodb’s denormalized data model is suitable for social media use cases. I will discuss this later.
Backend: NodeJs, ExpressJs
I picked NodeJs because of Mongodb. There are many server frameworks on NodeJs such as ExpressJs. I picked ExpressJs because it is most popular and unopinionated so that I can try my own server infrastructure ideas.
Database Driver: Typegoose
Mongodb has its own NodeJs driver. But there is a more powerful Mongodb NodeJs driver called Mongoose. Mongoose stands out because it supports schema definition and validation. Mongoose is in JavaScript and its schema is defined in JSON format. Typegoose is a TypeScript library built on Mongoose. In Typegoose, a schema is defined as a TypeScript class with annotations to indicate properties of a field such as validation method which is clearer and more readable than JSON format.
Backend logger: Winston
Winston supports both logging and profiling with many options.
Process Management: Pm2
Pm2 manages the NodeJs process. It could automatically restart NodeJs app on failure. It could also automatically cluster the NodeJs app so that I don’t need to add custom clustering logic. Third, It supports zero downtime reload so that the reload process is smoother after app upgrades.
Protocol between frontend and backend: Protobuf
One common data format between frontend and backend is JSON. But JSON doesn’t have type definition which could cause misunderstanding and ambiguity between frontend and backend developers. Also if the backend uses typed language such as Java, developers need to create converter and parser for each JSON object which is dummy.
Protobuf solves these problems. Developers define object and field types in a proto file which is like a declaration file. Then the proto file can be compiled to different languages including JavaScript, Java, python and so on. In each language, we just need to call the serialize/deserialize method and it will convert Object from/to string/JSON with validation. One drawback of Protobuf is that it looks like it was originally invented for communication between backend rpc calls instead of between frontend and backend. So its JavaScript compiler doesn’t work and integrates well into the Webpack build system. We might need to build our app 2 times. First convert proto to .js file. Secondly, build the whole app with .js files.
Another option is GraphQL. It supports more flexible query from frontend but requires more work on backend to support such flexibility.
Frontend: React, Redux, Bootstrap, SASS, CSS module
React + Redux enforces a good model - view pattern and makes it easy to reuse components. Bootstrap provides common UI components with beautiful UX design and supports styles on different devices. SASS enables developers to generate CSS dynamically. CSS module restricts a CSS style’s scope to only a component so that a component could be easily packaged and reused with its own CSS styles without polluting the global CSS namespace.
Server: AWS Lightsail
It is a cheap simple cloud virtual machine with different operating systems similar to DigitalOcean’s product. At first I chose an instance with 1 GB memory. But the memory was not enough for building the app. Npm and NodeJs consume a lot of memory. So I switched to the 4 GB instance.
Architectural Ideas
Backend
Database:
One interesting database schema design method I learned was to design schema based on possible queries. Instead of designing schema based on business models and their relationships, write some possible database queries in your application first and then figure out the best schema for these queries.
Why nosql over sql?
Nosql is suitable for Web-scale applications, including social networks, gaming, media sharing, and Internet of Things (IoT) while sql is suitable for Ad hoc queries; data warehousing; OLAP (online analytical processing). Our app is social media, so Nosql is more suitable.
Flexible schema structure, we can easily update post and answer templates or use different templates.
After writing all possible queries, I find that most queries are around the post. So a tree structure of posts with multiple children responses is a better fit for our case. SQL join is more complicated and less performant for our case.
We don’t have strong consistency and ACID requirements. So SQL doesn’t help here.
Why Mongodb?
Mongodb supports multi-level indexes. We can index posts and answers and improve related query performance.
Mongodb supports text indexing.
Mongodb is JavaScript based which is easy for beginners.
Mongodb uses JSON data as schema type which is easy to integrate with the frontend.
Mongodb has a sophisticated aggregation pipeline which can be used to join different schemas in case we need them in the future.
Mongodb’s text index is a great feature. It facilitates developers to add generic text search features to UI. For example, by indexing a post’s contents and comments, a generic text search over these contents and comments can be easily implemented. The way Mongodb index works is
1. Tokenize text content
2. Index by text tokens.
For example, when a sentence is stored in a database, it is first tokenized into words and phrases. Then Mongodb indexes the row by these words and phrases tokens.
One downside of this feature is that it only supports a limited number of languages not including Chinese because tokenization algorithms could be hard to implement for some languages such as Chinese. One solution to this limitation is to integrate with ElasticSearch. ElasticSearch is like a database with much stronger indexing features and faster lookup. The drawbacks are
Much more extra system resources including disk space and memories are needed. The data needs to be copied to ElasticSearch so that it could build indexes. It also requires a lot of memory.
Query might be complicated. When you query ElasticSearch, it only returns the data stored in ElasticSearch, not the data stored in your original database. If you want to get the whole object from your original database, you may need to join ElasticSearch database with your original database such as Mongodb.
There is no perfect driver or tool to connect ElasticSearch, Mongodb and NodeJs TypeScript. One great library is called Mongoosatic. It supports synchronizing Mongodb’s data with ElasticSearch, querying, joining and converting annotated Mongodb schema to ElasticSearch schema. But the owner is looking for other people to maintain and doesn’t have a TypeScript declaration file. Another tool is ElasticSearch’s own NodeJs driver. But it doesn’t have Mongoosatic’s great features mentioned above.
Another solution is to tokenize Chinese text in server code and store the tokens into a separate database field which has text index enabled. For example, in server code, I tokenize Chinese post content with Nodejieba library. Then I store post content into another field called postContentTokens. And I only add a text index to the postContentTokens field.
Server code
ExpressJs is a unopinionated NodeJs app framework. “Unopinionated” means it doesn’t enforce any design pattern which gives developers more freedom to create and experiment their own design pattern.
A simple NodeJs app usually has all code in one big index.js file. But that’s not scalable and maintainable. So we need to divide the code into files across multiple folders. My current server code directory structure is:
I18n folder contains i18n translation messages. Resources folder contains other static contents such as images and the Chinese text tokenization library’s data model. Schema folder contains Mongodb schema. Services folder contains code implementing business logic. Each file in it contains a category of business logic. Environment.ts contains setup code for global dependencies such as logger, database connection, i18n, mailer and so on so that they can be used in other files. Index.ts contains major app setup and some middlewares.
Middleware is a very important pattern. It can contain common non-business logic such as logger, parser, error handling and profiler across all services. Business logic should be implemented explicitly in each service’s handler.
NodeJs’s asynchronous nature is also a great feature. It is suitable for I/O intensive tasks so that one routine doesn’t block other routines when it waits for I/O to finish. To avoid too many nested levels of callbacks, we can use the new async/await syntax.
Frontend
I proposed many design patterns in my Single Page Web App UI Development Thoughts posted 2 years ago. In my app, I applied my design patterns and obtained some hands-on experiences. I will discuss them in the following part.
Modularize components to code clean, reusable and with low maintenance cost
React’s Component base class and jsx/tsx tags make it super easy to build reusable web components. Developers just need to create a class extending React.Component and add jsx/tsx rendering logic into it. With the CSS Module library, CSS style sheets can be restricted to local scope without polluting the global css namespace. With SASS, CSS variable and relative CSS size units such as vw and vh, CSS style sheets can be configured by the parent component or container component. And together with React Component’s props and state, we can easily make a component broadly configurable to any specs and initializable to a lot of states.
When two components are similar, there are 2 options to share code between them.
Composition: Extract common features into a separate component and let the 2 components contain that smaller component.
Inheritance: Create a base component and make the 2 components extend it.
React documentation recommends the first solution because with React props and composition, it can cover all cases that inheritance supports. Reference Also as a design pattern, composition is recommended over inheritance because inheritance breaks encapsulation. Reference So overall, composition is preferred over inheritance.
Architectural Pattern
React and Redux are usually used together with React rendering dom and Redux managing global states and talking to backend.
In my previous article, I proposed the following architectural patterns.
Organize and group the code into 3 phases:
Phases:Update data model with backend API: load data from backend. The data should contain any data MAY BE used after a user action.
Update UI states: Javascript code update all related UI states with data from the model
Render UI into HTML: Call any UI rendering callback. It shouldn’t request any new data to be loaded. All data should have been loaded in phase 1.
Decrease number of dependencies among views. Use view –> model -> model -> view pattern instead.
These 2 patterns can be achieved with React + Redux. For “1 Organize and group the code into 3 phases”, the React + Redux realization flow is:
In React component’s user action handler, trigger Redux’s dispatch() to change Redux’s states. The dispatch() could either call a Redux asynchronous Thunk to talk to the backend first or trigger synchronous client side only state update.
Note that to avoid the messy React UI and Redux state change chain, in Reach components, we should restrict Redux’s dispatch() to be called only from user action handlers so that only user handlers could update Redux states.
After talking to backend and updating Redux’s states, React component’s Redux state change listeners will be triggered and React component’s props will be updated based on Redux’s new state.
React component’s props updates will trigger a rerender.
For “2. Decrease number of dependencies among views”, React + Redux achieves it in the following ways:
React + Redux is like view - model pattern where React component is view while Redux states is model.
A React component couldn’t directly trigger state/props change in sibling or parent components. It could only change child components’ props to rerender them which decreases the number of dependencies among views. If it wants to change sibling components, it needs to update Redux’s state first and then Redux triggers other components’ listeners which is a cleaner approach.
Another advantage of React + Redux is that it doesn’t have observer pattern and event listeners only exist between React components and Redux states instead of inside React components or Redux states. This advantage matches “1.3.1. Never use observer” and “1.3.2. Event dispatcher - listener pattern is not recommended”‘s proposal in my previous article.
Human Problems
I finished and launched my dating app early February 2021. But it was advertised to its target user 4 months later in late May 2021. This delay is caused by human problems.
The brief story:
Part of my dating app’s ideas come from a local voluntary Chinese dating group’s offline event and I had planned to build such an app as my side project in 202. In September 2020, after a chat with the group founder, we found we both wanted to build a dating app and agreed to work together. In early October, I presented my design of the dating app to the founder and reached consensus on the design. From October to January 2021, I was developing the dating app myself and kept the founder informed of the process every week. In early 2021, I demoed my app to the founder and he said it was great without any constructive feedback and promised to show this app to other volunteers in a week to collect some feedback. But nothing happened. Then in the following 4 months, I kept asking him how they want to advertise the app in the dating group and he kept telling me they want to change the app to a different direction and kept promising me they would have some ideas in one or two weeks. But nothing happened, not even an active followup from the founder. In the process, I also heard that other volunteers may not know my app yet. Finally, at the end of May, after the founder missed the promise one more time, I lost my patience with the founder and contacted a sister Chinese entertaining group to advertise my app on their platform. That sister group member contacted the founder and finally the founder agreed to advertise my app in the dating group. This time he fulfilled his promise but still not all of his promises. If anyone is interested in a more detailed version of this story, please go to this post https://www.1point3acres.com/bbs/thread-761581-1-1.html (written in Chinese).
In the whole communication process, I stated my goal and idea clearly, explicitly asked the founder’s standing and opinion and negotiated professionally while the founder’s words don’t seem to consort with his behavior. Here are some major takeaways from these experience:
Dedication is much harder to enforce inside a project than in work. So pick your side project partner carefully.
Pick someone who shares the same project goal, passion and preferences with you. Otherwise you may encounter some people who pretend to agree with you, hide their real intention and just want to take advantage of you.
Trust is established on good communication and mutual understanding which is crucial to collaboration. It is a red flag if your partner hides his real goal or misses promises without reasonable excuses.
Respect is also very important. It is not a moral requirement. It is just a tool to help you evaluate how important you are in your partner’s mind. The more important you are to each other, the more reciprocal your collaboration will be.
One red flag when picking a partner is that the potential partner has strong preferences and bias over a lot of things and people and doesn’t know how to give constructive feedback. Some people may have strong preferences over things and people which could result in bias. No matter if the preferences are correct or not, that is fine. But if he doesn’t know how to provide constructive feedback for others to meet his preference, then it is highly possible that others couldn’t meet his preference and he reacts badly to that.
If you have already picked a wrong partner, started the work for some time and the partner becomes less dedicated to the project, ask him for his expected timeline and check with him on each milestone.
