Hard to keep up

I’m learning a lot from this project, and most of all how hard it is to keep up with the documentation. Initial priority was hitting the market, and making the first customer. It was later changed to showing the product to investors, and then back to making customers again. In only two months.

I’m having trouble keeping the documented architecture up to date, since the speed of development is so high. We have architectural discussions all the time in the team, and decisions are made, but the documentation is not keeping up.

We’ve made a Linux container for our Report Generator, and we have it always on, with an API we can call to generate a report. The API returns the generated report, instead of storing it to a Blob Storage. This solves the problem that the user wants to create more than one report from the same Crawl Session. The always on linux container also generates the report in a matter of seconds.

I’ve set up a Dashboard in Azure, showing a selection of interesting metrics, like database load, number of Crawl Container instances and backend load. This will help us know when we need to scale.

We need to watch the number of Crawl Container instances since there’s a hard limit on how many we can have running simultaneously. We have the application check that limit before trying to create a new instance, and we let the user know if the “queue” is full, but having that happening more than just every now and then would suggest we need to address that issue. Possible solutions are asking Microsoft to raise the limit, or distributing the Crawl Sessions across multiple regions, since the limit is per region.

We also need to monitor the load of our databases. We have two for now. One that holds the users and the agencies, and their relations. Another that holds the agency specific information, like the SiteGroups, permissions and Crawl Results. We’ve prepared for letting the Agencies have separate databases, and/or connection strings, to balance the load. For now, all of them use the same database. We’ll monitor the load to know when we need to scale vertically. The User database however will primarily scale horizontally (manually) when needed.

Next, I will need to catch up om that FMEA.