Public Management App (Betha Sistemas)

The project

I joined the team in 2017 to help develop this new web project. The previous project had been developed in PowerBuilder and functioned as a desktop application, but at the time, the company was migrating all of its applications to the web. This project manages the City Hall's accounts, allowing users to process them, create dynamic reports, and access information that was previously unavailable to them, thereby enabling the public administration to better understand the resources it could collect and optimize its processes.

The problem

As you can imagine, this application handles a huge amount of data, and since one of the project’s goals was to improve the presentation of the counters and display millions of data points summarized on the screen, most of the pages experienced various performance issues at first. Since I was working there as a full-stack engineer, I was able to step in and investigate any related issues that might arise.

Approach

The two and a half years I spent working on this project led me to address various issues and make improvements along the way, but there are three situations worth highlighting:

1. Coordinating with other engineers and managers on how to improve the performance of the debt screen.
2. Manually implementing the front end of the debt screen.
3. Working on generating reports with Kafka and maintaining the consumer project.

Demo

1. The performance bottleneck

Some town halls had a huge amount of debt data, millions of records sometimes, and the screen was fast for small entities, but took several seconds to load when we started to migrate large town hall entities to this new project. After tests with the client and the API, it was clear the problem was with the back end and how the DB (PostgreSQL at that time) was handling the queries. We first used ORMs to access the DB and to ensure "cleaner code", but the fix was obvious after we compared the results with a query written by hand. The approach I took to solve this problem was, first, to remove the ORM from the endpoints related to the debts, mainly for endpoints that need to deal with the entire dataset per entity. Second, I reviewed the tables and suggested improvements to ensure we had less payload and better indexes based on some filters available on the client side. After these changes, the screen was able to load in less than 1-2 seconds for almost all entities.

2. The front end of the debt screen

As you can see in the demo above, the debt screen is a complex page with many filters and a table that can display millions of records. Although it feels like a simple table today, it was not that easy back then. All the logic needed to be implemented by hand, and the team/managers were very happy with the results, mainly considering all the data complexity around it, performance, and how quickly I could deliver that page without hurting performance when expanding or selecting them. No libs were used or hurt in this process. I tried to use some, but I couldn't find a lib that would solve the problem entirely.

3. Generating reports with Kafka

The users can emit up to 3 reports for each debt, each one related to a part of the process of generating the debts and sending them to the citizens. We had a Spring Boot project that was the consumer, dealing with the messages sent by the app, connecting to a Kafka topic and generating reports in PDF format. The main challenge with this project was to deal with failed messages, understand the business requirements when something failed, how to keep that project up and running (for this part, we had a really good DevOps person that helped with that, but since he was a shared resource, I needed to jump into many of the problems and only reach out to him as a last resort). Note that we didn't have any limit on how many PDFs the user could emit, so there were several cases where they selected all debts at once (remember the millions of records I mentioned earlier) and simply hit to emit them. The app was fantastic; it was easy to use and allowed such a thing that would have been unimaginable years before in a public management system, but it brought technical challenges that I would carry with me forever in the tech industry.

Conclusion

This was my first experience as a full-stack engineer, dealing with a huge amount of data and confidential information. I learned a lot about performance, DB optimization, and how to deal with complex business requirements. The project was a success, and the team was able to deliver a product that improved the public administration's ability to manage its resources and processes. At the time I left the company for my next opportunity, the project was being used by more than 12 town halls in Brazil and impacted more than 500,000 citizens, also increasing the company's revenue by 15% from one year to another.