Eventu: The history behind my first big project

Part 1: The most stressful day of my professional life.

Back in 2016, in my second year of studying Mechanical Engineering in University, one of the projects I was involved in was the yearly “Engineering Week” (Semana da Engenharia)¹. The decision to be part of the team came after attending its 12th edition in my freshman year in 2015. The conference attracts on average 500 students every year from the major engineering colleges in my home state and is organized entirely by volunteer students. And this fact generates one of it’s biggest challenges: the staff changes almost entirely every year as students come in and out of the project and up to that point, when it came to the ticket selling system and its other features, the new team always wanted or needed to do it from scratch.

At that time, the system for the 12th edition was built in C# and the C# guy left. And for the 13th edition the new team had two CS students that knew PHP and WORDPRESS. So instead of reusing the C# code from the previous year, the new website where the attendees would buy or validate their tickets and choose what lectures and workshops they want to go to, would now be all coded as a Wordpress website with custom plugins.²

DISCLAIMER: As what follows can be understood as a series of major errors in software engineering, operations and basic risk assessment, I want to make clear that my intention here is not to speak ill of any of the people that worked in this project at the time. I think it’s wonderful that University allows for a safe space to try and fail and learn, and these were not professionals, but students with little to no guidance that were doing their best and nowadays have brilliant careers as engineers.

The conference in theory takes a whole year to organize. On the Thursday and Friday of the event week the team starts collecting the names for next year’s organization, but in practice work only starts in January (the event is usually in August) and everything usually piles up to the very last minute³. The 13th edition was scheduled for August 22 to 26, 2016. Therefore in the 8 months prior, the multiple teams get together, make an action plan and execute. The leadership positions are called “Managers” with the main position being the “General Manager”. In the 13th edition I got the opportunity to be the Programming Manager, that has nothing to do with software development but with the booking of speakers, arranging workshops and activities. Essentially the Programming Manager handles the content of the conference.

The two main features that the Engineering Week requires for it’s system are: Selling Tickets / Validating tickets that were sold offline. Store all the multiple available activities so when the “Schedule release date” comes, around 2 weeks before the event, students can pick and choose. Each student ends up with their own personalized schedule.

These features are released on the website at different dates, as sales start much earlier, while some speakers and activities are still being arranged.

While the sales feature worked quite well and was released on schedule, as time progressed the “Schedule release date” was getting postponed and postponed, because of bugs that the team wasn’t able to fix in the process of validating and saving the options a user chose to the Wordpress Database.

After multiple deadlines were missed, we released the schedule selection as it was even as we knew it was buggy, because the starting date of the event was coming and we had to know who and how many people would be in some activities.

On the starting night of the event, the team was working late to fix the remaining bugs. While we were all watching the inaugural lecture. During one of the test runs, connected to the Production database, they ran an SQL query to clear out the data for the test user.

But in their rush and tiredness, they made a big mistake: they forgot to add the “WHERE” part in their UPDATE query. Instead of just clearing the test user’s data, the query erased everyone’s schedules, replacing them with empty fields. To make things worse, they didn’t have any backup. Testing in production, directly on the database, running these things manually instead of a script, ALL of this would be unthinkable in a professional environment nowadays, but I want to remind you that these were students that did it all by themselves without the supervision of someone experienced, 10 years ago.

I was at home and switched my suit for pijamas, when the team called me in panic and I and seven other team leaders raced back to the university around midnight. We knew we had to act fast to come up with a plan, not only technical but communications as well. Our only choice was to go back to last year’s system and manually add all the activities for this year’s event.

We worked all night, recreating the entire schedule. We were so tired that we made some mistakes – some activities got the wrong place or time. But we kept going, knowing we couldn’t fail.

As morning came, we had to tell all the attendees that they needed to choose their activities again on the first day of the event. We quickly set up a local server and LAN at the university and organized counters where people could make their choices in person, instead of doing it from home like we originally planned.

That day, after working for almost 24 hours straight, I finally got to bed at 10 AM. I was so exhausted that when I woke up at 3 PM, I had sleep paralysis for the first and only time in my life. It really showed how hard this whole situation had been on all of us.

The first day of the event was crazy. There were long lines of people waiting to redo their schedules. It was really stressful, but somehow, we got through it.

Part 2: The other origin story

The Engineering Week as a whole ended up being great and with very positive feedback, after the initial frustration (something about Kahneman’s Peak–end rule)⁴. But I really understood how much we needed a good, easy-to-use system, developed in a professional manner. And seeing all that happened, I wanted to have something good running it, and I wanted to be the one that built it.

That’s where Gustavo Stork comes in. He was another student who later became my partner in starting Eventu. Gustavo and a friend had made an app for a conference organized by PET (Programa de Educação Tutorial), another student group in the Computer Engineering department. While our needs and problems were more related to selling tickets and signing up to lectures, their app solved many of the display and content features we wanted for Engineering Week.

Gustavo liked what they’d made and reached out to me. He wanted my help to turn their app into a real product that other people could use. He thought it could work for more than just their one event. I was really interested in the idea, since this was a big challenge to tackle by myself as inexperienced as I was and having a partner would help me stay engaged.

As we talked about the possibilities, Gustavo’s original partner decided to leave the project. Seeing a chance to use what I’d learned from organizing Engineering Week, I joined Gustavo. This is how Eventu was born.

We wanted Eventu to be a complete system for managing conferences that would fix the problems we’d both experienced. We aimed to create a platform that could handle selling tickets, managing schedules, and keeping attendees engaged, all while being flexible enough for different types of events. But the main thing was the apps: we wanted a system where we had a template for an Android and iOS app that we could build natively and publish in the app store, instead of having a generic “Eventu” app up.

We used our different experiences – my backend skills I learned from Laracasts⁵ and my event management skills from Engineering Week and Gustavo’s mobile app skills from the PET conference app – to build Eventu from scratch. Our goal was to make a system that would not only prevent the kind of disaster I’d been through but also make things better for both event organizers and attendees.

Part 3: Putting it to test

After six intense months of coding overnight at Gustavo’s house, we were finally ready to put Eventu to the test. Our plan was to progressively release the features for the upcoming 14th Engineering Week, starting with content, then scheduling, and finally ticket sales. This was the only way we were able to keep up with schedule as we were releasing features as soon as they were available.

The content feature rolled out smoothly, giving us a confidence boost. However, the real problems started when we reached the scheduling phase.. Despite our best efforts, we encountered significant optimization issues that we hadn’t anticipated. Local, dev environment tests work fine but all is different when there are 300 users pressing the refresh button. As the week before the 14th Engineering Week approached, I found myself back at the university, pulling all-nighters with the team to manage the infrastructure and fix the problems.

The night we were set to release the schedule feature became a true test of our system - and our nerves. At 7 PM, we had to halt everything and reset the system. The load was so intense that it was using 100% of our server’s resources - all 24 cores and 128 GB of RAM..

The culprit? A lot of n+1 database queries⁶. By the way, this was the day I learned that n+1 queries were a thing. It was a race against time, fueled by determination not to repeat the disaster of the previous year, I wrote all the cache logic to use Redis in three hours and at 10 PM, we were ready to try again. We held our breath as we brought the system back online.

To our immense relief, the optimizations worked. Although they were far from optimal, they proved themselves good enough. The server load dropped dramatically, and the system was able to handle the influx of users.

These two episodes, the dropping of the database and the overnight optimization are two foundational experiences in my career. They gave me confidence around the technical skills I’ve harnessed so far and helped me ease my impostor syndrome from having started coding at 18, rather than 12 years old as some of my friends.

We progressed to support not only Engineering Week but also some other two conferences, and Gustavo’s Bachelor Final Project was about Eventu⁷. But with both of us getting jobs, the 2020 pandemic and graduation, we decided that it was time to move to other projects.

Notes: