Swortal

  Goal And Target Audience This is a summary and highlights of 6 software engineering and architecture related books I recently read. All the 6 books are very heavy and interesting. Reading real contents of them is always recommended. The target audience is experienced software engineers who want to learn more in larger system design and architecture. The goal of this article is to provide summaries of the 6 books as well as brief highlights of interesting or important contents from the 6 books so that the target audience could learn the 6 books’ important ideas quickly and read only important chapters and save time. The highlights and summaries will be divided into several general categories. Each category contains a highlight section and a related chapters section. The related chapters section contains all chapter references which are related to the category’s topic. The highlight section contains a brief summary of interesting or important or enlightening contents from the r...

  Highlights Beyoncé Rule: If a product experiences outages or other problems as a result of infrastructure changes, but the issue wasn’t surfaced by tests in our Continuous Integration (CI) system, it is not the fault of the infrastructure change. SWG ch 1 what is software engineering? Test Goal If a fault is present in a system, then we want it to fail during testing as quickly as possible. More emphasis should be placed on finding the most severe faults than on finding other faults. Test writing guideline Control and observe system state record and playback store states in a single place so that it is easy to start a system, subsystem, or component in an arbitrary state for a test, Preprocessor macros, when activated, can expand to state-reporting code or activate probe statements that return or display information, or return control to a testing console. limit nondeterminism Replace real implementation with test mocks usin...

  Dependency Management Highlights Shared Library vs Shared Service The shared library technique is a good approach for homogeneous environments where shared code change is low to moderate. One distinguishing factor about the shared service technique is that the shared code must be in the form of composition, not inheritance. The shared service technique is good to use in highly polyglot environments (those with multiple heterogeneous languages and platforms), and also when shared functionality tends to change often. SAH ch 8 Reuse Patterns Sidecar pattern and service mesh Share common operational lib such as logging, monitoring, service discovery, auth… and connect these lib among services. SAH ch 8 Reuse Patterns Source control vs Dependency-management All else being equal, prefer source control problems over dependency-management problems. Source control problems are a lot easier to think about and a lot cheaper to deal with than dependency-management ones. SWG C...

  Code Base Management Highlights Version controls and branch management branches are a drag on productivity. In many cases we think complex branch and merge strategies are a perceived safety crutch—an attempt to keep trunk stable. As we see throughout this book, there are other ways to achieve that outcome. Long-Lived dev Branches are expensive to maintain and sync developers must not have a choice when adding a dependency onto some library that is already in use in the organization. One native advantage of a many repo approach is that separate repositories are obviously capable of having different sets of authorized developers, visibility, permissions, and so on. Stitching that feature into a monorepo can be done but implies some ongoing carrying costs in terms of customization and maintenance. VCSs will focus on allowing larger repositories with better performance scaling, but also removing the need for larger repositories by providing better mechanisms to stitch them toge...

  C loud Highlights Container vs VMs: container is smaller so easier to transfer on the internet container layers make it lighter to move from one machine to another by only moving the changed layer e.g. an app could contain 4 layers and images of a container: pure linux distribution, apache server, db, php server; if only php server is changed, then only that layer of image need to be moved and others don’t need to be moved which has smaller network throughput one service per container VM can host more different types of software because it doesn’t rely on any OS SAP ch 16 Virtualization Autoscaler Configurations to add and remove an instance The new VM image configurations and settings The CPU utilization threshold The network I/O bandwidth thresholds The minimum and maximum number of instances you want in this group process of autoscaler removing an instance autoscaler notify the load balancer to stop sending request to that instance autoscaler notify t...

  Scaling Highlights Ways to scale up business model integration: Open host service When a subsystem has to be integrated with many others , customizing a translator for each can bog down the team. Define a shared protocol with shared a shared domain language Use a one-off translator to augment the protocol for that special case so that the shared protocol can stay simple and coherent. Published Language Use a well-documented shared language that can express the necessary domain information as a common medium of communication, translating as necessary into and out of that language. DDD ch 14 Maintaining Model Integrity Large scale domain model evolving order Too complicated and restrictive architecture based on up-front design assumption could prohibit developers from switching to better architecture based on better domain model. Don’t overconstrain the detailed design and model decisions that must be made with detailed knowledge. Let this conceptual ...

  Data And Database Highlights When to break database into multiple one? Change control How many services are impacted by a database table change? if one table is depended by too many services, then it is hard to decompose that table Better to refactor database schema to bounded context first so that one table can only be directly accessed by only one service Connection management breaking into multiple services may increase number of connections to db. One solution is to assign connection quota to each service either evenly or based on different services Scalability, Fault tolerant breaking into multiple database can help increase scalability and fault tolerant Database type optimization Can put some data type in a more suitable type of database SAH Ch 6 Pulling Apart Operational Data How to split table ownership among multiple services Several scenarios: single ownership scenario: one table is written by only one service; straight forward… common owner...

  Highlights What make a good service? loose coupling and high cohesion MSV Ch 3 How to model service Integration Guidelines service change shouldn’t break consumer Keep Your APIs Technology-Agnostic Make Your Service Simple for Consumers: providing a client lib makes client easy to use but increase coupling Hide Internal Implementation Detail MSV Ch 4 Integration Integrability It is about how to integrate a dependency into a system and its difficulty. type of services distances Syntactic distance: data type difference e.g. int vs float data semantic distance: data meaning difference e.g. integer means latitude or altitude behavioral semantic distance: e.g. states and modes of the system. temporal distance: e.g. rates, order of events, latency resource distance: assumptions about shared resources, devices, memories Tactics: Limiting dependency: provide a simple interface to externals Use an Intermediary : e.g. pub-sub bus Restrict Communication Paths Ad...

  Interface and Module Design  Highlights Programming Paradigms Dijkstra showed that sequential statements could be proved correct through simple enumeration. The goal is to eliminate GOTO statement Structured programming imposes discipline on direct transfer of control. Allows modules to be recursively decomposed into provable units, which in turn means that modules can be functionally decomposed。 Object-oriented programming imposes discipline on indirect transfer of control. Object oriented design is the ability, through the use of polymorphism, to gain absolute control over every source code dependency in the system. It allows the architect to create a plugin architecture, in which modules that contain high-level policies are independent of modules that contain low-level details. e.g. use interface to decouple caller and implement and achieve dependency inversion. Functional programming imposes discipline upon variable assignment. Has no mutable variable. ...

  Reliability Highlights Distributed Database eventual consistency pattern Background sync process to sync different service’s databases. Cons: break bounded context among dbs. Because background process also has access to many dbs Suitable when there is no other communication among related services orchestration Pros: Atomic business request Cons: Slower responsiveness event based: Sync through events pros: Fast responsiveness Cons: Complex error handling SAH Ch 9 Data Ownership and Distributed Transactions Distributed Transactions the various patterns used to implement distributed transactionality (such as compensating transactions) succumb to a wide variety of failure modes and boundary conditions, along with adding inherent complexity via undo operations. Distributed transactions present a host of difficulties and thus are best avoided if possible. SAH Ch 12 Transactional Sagas Transaction distributed transactions are hard to get right and can actually ...

  Availability Highlights How to detect faults? many ways, interesting one: voting (3 ways) replication functioning redundancy: same function implemented by different pieces of code analytics redundancy: One function’s one input can come from different sources so that it can pick most reliable source to use SAP Ch 4 Availability How to recover from faults? preparation and repair: many ways, some interesting ones: ignore fault behavior. e.g. ignore bad sensor output graceful degradation: This tactic maintains the most critical system functions in the presence of component failures, while dropping less critical functions. reconfiguration resources Reintroduction tactics: Re-sync a previous failed component in shadow for a predefined duration state re-sync escalating restart: different level of restarts nonstop forwarding if a router’s supervisor fails, router can continue forwarding information based on known path SAP Ch 4 Availability How to Prevent fau...

  Team Work And Communication  Highlights Talk to domain experts The best way is to let developer collaborate with domain experts leading by developer so that can gain quick feedback on domain model force domain expert to refine their own understanding by being forced to distill what they know to essentials DDD ch1 crunching knowledge How to interview stakeholders to gather core requirements: Stakeholders are sometimes not clear about QA requirements Architect can play dumb. e.g. if they don’t know what time requirement is needed, just ask them if 1hr is ok, 30 minutes is ok …. until they seem to be happy with the number. The Quality Attribute Workshop Stakeholders present business requirements including known QA requirements Architect presents current design and plan Identification of architectural drivers. The facilitators will share their list of key architectural drivers that they assembled in the prior two steps, and ask the stakeholders for clarificat...