Python + Upsolver: Simplified Realtime Data Workflows

Notes

One of the powerful things about Python is its ability to connect disparate tools into one common integrated development experience. In this talk, we’ll explore how to create and run a near real-time pipeline where we consume events from a Kafka topic and transform the data before landing them in the lake, using Upsolver through our Python SDK. In this way, we get exactly-once processing, strong ordering and automatic schema evolution out of the box thanks to the powerful Upsolver engine, but without having to switch to a different UI and building in SQL only.

Join my newsletter to learn about Python and Django with 1,600+ other developers!

Newer

#Python
#Django
#models
#graphviz
#admin

First Major Model - Building SaaS with Python and Django #163

In this episode, we got to work on the core modeling for the application. I started by adding some visualization tooling to see the models in the system, then got to modeling of the primary …

Older

#Python
#Django
#SaaS
#email
#templates
#testing
#CI

Finish Signup and CI - Building SaaS with Python and Django #162

In this episode, I continued on the signup flows that started in the previous stream. We configured the email backend and verified that the end to end signup flow worked. Then I set up the …

If you like my work and you want to say thanks, you can become a patron on Patreon. Thanks for supporting me!

Check out my Patreon Superhero supporters!

Django Riffs is my podcast devoted entirely to learning the Django web framework. Give it a listen!

All the shows are on the Show Notes page.

Book Recommendations

My library is full of technical books. Many of these were hugely influencial in helping me be a better software engineer.

Check out my favorite software books on the Book Recommendations page.

If you enjoyed this article, you may also be interested in…

Accounts and Email - Building SaaS with Python and Django #164

In this episode, I planned to do the work of sending email prompts for the journal to users. Along the path, we realized that the Account model was missing, so I stopped to build that out before we could proceed. By the end of the stream, we had a working background job that would send email and was 100% unit tested.

DNS Strikes Back - Building SaaS with Python and Django #172

In this episode, we returned to our DNS configuration from the previous stream and worked on testing email from end to end on the production site. On the stream, we spent a lot of time looking through SendGrid documentation, digging into layers of DNS, and troubleshooting the challenges that can arise when getting a site online.

Deploy to Heroku - Building SaaS with Python and Django #171

In this episode, we did the work to deploy the site to Heroku to make the site live. We configured the production site, fixed some missing settings, and connected to the relevant DNS configuration.

Deployment Checklist - Building SaaS with Python and Django #168

In this episode, I added the deployment checklist and improved the security of the app. Then we moved to work to set up the database to use DATABASE_URL and prepare to use Postgres.

Ellen's Alien Game / Display Related / Neovim config - Building SaaS with Python and Django #157

In this episode, I did another Exercism problem in Python that focused on Python classes. After the exercise, I deployed my model change from last stream and discussed deployment strategy. Then we worked on some of the display pieces for the new feature. Once the feature was added, I discussed performance testing and showed some performance problems that the new feature created that I then fixed. We ended the stream with some Neovim configuration to correct some LSP formatting problems.