My summer internship at Sandtable

Close to handing over the projects I’ve worked on during my short stint as a software engineer intern at Sandtable, I find this is a good opportunity to reflect about my experience working here. I was brought in to explore solutions for some inconveniences in the development process, and thanks to the support of my wonderful team I was able to attack some of these directly. I’m glad that before leaving I can see the services I created live in production, simplifying the workflows of our users.

The process started with excellent onboarding, I was very happy that on my first day our CTO Thomas had the good will to sit down with me and talk over all the services that make Sandtable tick. I was in awe by the capacity of our engineers to maintain this massive infrastructure composed by elastic clusters of resources, while making them available to the data scientists with a simple interface. This overview served as a good introduction to implementing distributed systems, and I learned about tools such as Mesos and Marathon.

Thomas also took this opportunity to explain how some development tasks were too many clicks away. He introduced me to some ideas that were floating around, these worked as inspiration for the tasks of my internship.

The ideas were soon formalized with Trello cards for reviewing progress every two weeks at the sprint recap. This contributed to having a shared understanding of what the requirements were, so no misunderstandings could happen down the road. Having daily standups also helped everyone to stay on the same page and to get feedback on my work as soon as possible.

My first task was to build a service that would function as a messenger to the users. Services on the platform were all implementing their own communication channels with Sandtable users so the idea was to create something that would serve as a central dispatcher so all other microservices could easily pass their messages through it, we called it Sandnotify.

I was given one or two days to play with Flask. This was my first time working professionally with Python so I spent this time making sure I knew how to use the testing, debugging and documenting tools. I also had to make sure I was prepared for composing multi-container Docker images so the server could be easily deployed with a database.

I started building Sandnotify right away. I had an idea of what it needed to do: take a request with a payload carrying the representation of a message and it’s target, and pass it along to the specified user, simple as that. However, underestimating the complexity was a mistake.

It didn’t take me long to produce the first version of Sandnotify, it did it’s job. It had a RESTful API and I learned about a very important tradeoff in systems design: failing fast when validating forms versus going through all of the data points and responding with a complete list of errors. We chose failing fast to keep the whole system moving quickly. However, the code felt very fickle. Adding new features started feeling like a lot of work and I was often asking myself if most lines of code were in the right place. This is when I realised the value of architecture design. As someone who had mostly worked with MVC frameworks I was used to having guidance for where things should go but Sandnotify had no such conventions, I unconsciously abused this and ended up with my house of cards.

After an extensive code review with Thomas, we managed to abstract the interfaces I was encountering and identify the layers of interaction that was going on between different subsystems of Sandnotify. With this new understanding, the reorganization was quick and everything felt much more clean after it. Sandnotify now has a clear division of concerns and a way of error handling that makes it relatively easier to debug.

After this, implementing new features was a breeze:

  • I made sure Sandnotify was aware of which channels were valid by keeping a store of channels and keeping it synchronized with Slack. For this, Sandnotify listens to relevant events such as ‘open_channel’ or ‘user_join’. Listening to Slack events required me to reply with HTTP status OK as soon as possible, and only then process the event. This was solved creating new threads.
  • I implemented interactive messages. Services are able to send notifications that call for action with buttons. This required notification persistence, meaning I had to store the notifications temporarily in a database, so it could remember which notification a button callback belonged to even after the server went down. This button is now being used to terminate clusters from Slack.

Sandnotify now lives in production happily integrated with services such as Sandjs, Sandscheduler and Sandcommander. Integrating proved an additional challenge because as soon as we started exposing the features for them to be used,  we discovered some edge case requirements we were not initially aware of. At some point, I had to change the application’s database from MongoDB to Postgres. Thankfully, everything was abstracted in a way that this costed little work but it was still tricky to bend code without making it smell.

Sandcommander is a second service I built. The idea was to trigger scripts from Slack. For example: team members can now compare docker-images for all of our services on different environments. We can also run Jenkins jobs like dev-to-qa deployment just by calling Sandcommander in Slack. I used all of the lessons learned with Sandnotify, but it had its own set of problems. For example, getting information about jobs from the Jenkins API was extremely slow so I had to dig around Jenkins to find a more efficient way.

I have enjoyed building these applications and working with Sandtable. I have learned a lot, not only from my code but from others too. I also had a very good insight into the workflow of data scientists, who made interesting presentations explaining the reasoning behind their models. I also can’t avoid mentioning how content I was with the amount of board games we’ve played not only in the after hours but also during lunch time!
I want to thank everybody at Sandtable for making my internship such a pleasing experience! Thank you for your kindness team Sandtable, and God speed.

Leave a comment

Please prove that you are human: