Secure your spot!

Ever wonder how LLMs grasp context? Dive into the world of Vector Embeddings with Ben Church! Discover how Airbyte extracts, transforms, and loads data into a Vector Database, arming your LLM with rich, domain-specific context. Ready to supercharge your business's next phase? Join the evolution! 🚀📊

Kubernetes + MLOps = Generative AI's dream team! 🚀 Annie Talvasto from VSHN is set to unravel the symphony of scalable AI training and top-notch model management. Ready to unlock generative AI's full might with seamless scalability? It's a tech fusion you won't want to miss! 🎵🤖

Spam-fighting at scale is where real-time data & top-tier ML collide! 🚫📧 Louis Brandy, with experience from Facebook & Rockset, unveils the challenges of low-latency demands, hybrid queries, and swift defense against digital tricksters. Join us on this tech-packed journey against spam! 💡🔍

Houseplants giving you the blues? 🌱 Rock on with Danica Fine from Confluent as she jams out a tune with Raspberry Pi & Apache Kafka. From soil sensors to Telegram alerts, it's a symphony of tech! With ksqlDB & Kafka Connect, we're turning plant woes into rock shows. Let's turn up the volume on plant care!

Indulge in a feast of delectable reports at our conference. Bring your appetite, it's going to be a culinary delight! 🍽️ 📊

Register!

🚀 @Meta's Supriya supercharges with PyTorch 2.1's prowess.

💡 Paul Chiusano reshapes microservices - simpler, sleeker.

🔍 Nathan Marz's Rama: redefining backend landscapes.

🌊 Santona Tuli untangles the streaming maze.

Register Now!

🚀 PyTorch 2.1 is a game-changer in deep learning! Packed with features, it's optimized for both speed & memory: achieving 8.5x faster rates for 'segment anything' & 5.6x for 'llama2'. Supriya Rao from Meta will break down its groundbreaking capabilities & how to supercharge your models natively! 🧠🔧

Rethinking microservices? Paul Chiusano's set to shake things up #bythebay with Unison Computing! Imagine services as seamless as local functions & without the serialization hassle. into this fresh perspective & gear up for a demo that showcases a new era of service development!

Presenting Rama by Nathan Marz of Red Planet Labs: The future of programming platforms is here! Elevating backend development, this "programmable datastore" promises scalability & resilience, outshining all DBs & stream processors. Code smarter, not harder with Rama!

Streaming data challenges? Santona Tuli of Upsolver is set to demystify at #bythebay! Unpack upstream data validation, speed vs. state intricacies, & the art of microbatching. Learn how to achieve data consistency and precision in analytics. A transformative session awaits!

Consider this the Ctrl+C on today's tech narrative. Return for the update!

Ctrl+V

So, dive into the ocean of insights and learn how we can contribute to Hydroflow and be a part of this revolutionary project aimed at redefining the landscape of distributed systems.

1. Let's start with your background. Can you tell us about your career journey so far?

I got my start programming pretty early, when I was 6, but I started seriously building systems a bit later when I got introduced to Scala, around 2012. From there, I was primarily a hobbyist programmer, building games and simple websites (using then cutting-edge technologies like Scala.js!), which eventually led into several internships at startups like Coursera and Apollo GraphQL where I started focusing more on distributed systems and developer tools. That and my experience contributing to open-source projects like Scala Native got me quite excited about working on programming languages, which I decided to pursue in a PhD program at UC Berkeley. I'm now in the third year of my PhD, where I've been working on programming languages and optimizers for distributed systems as part of the Hydro project!

2. What potential downsides need to be considered and addressed when developing new technologies in software engineering in academia?

The biggest challenge with developing systems in academia is the disconnect between the goals of a research project and the needs of real-world developers. It's often impossible to satisfy both in a single project, since academia is more interested in identifying elegant solutions and rigorous proofs/evaluations while industry is looking for easy-to-adopt solutions that can provide an immediate business benefit. Academic projects tend to introduce a lot of new concepts, which ends up hurting the user experience since developers now have to study papers to understand how to use your library. On the other hand, focusing too much on industrial use cases can lock your work down to incremental improvements and limits the discovery of paradigm shifts. To be fair, my own research is stuck between these two worlds, as it is grounded in academic foundations but focused on delivering meaningful improvements to real-world developers. But I've found that a tick-tock model of developing core research and then working on making it accessible to a wider audience has helped balance these two sides.

3. Tell us about Hydroflow. What problems is it solving? How does one contribute to it?

Hydroflow is the foundation of the Hydro programming stack, and provides a low-level IR for programming distributed dataflow systems with an optimizer that aims to automatically discover more scalable versions of your logic and a type system that helps ensure that developers guard against sources of non-determinism in a distributed system. The second piece in particular builds on a lot of past research from our group on ways to reason about "application consistency" -- making sure that your business logic executes correctly regardless of failures or network latency. We are applying many theoretical results such as the CALM (Consistency As Logical Monotonicity) theorem to ensure that Hydroflow programs can be automatically optimized without impacting the correctness of the underlying logic. We've even used Hydroflow to auto-optimize protocols like Paxos, since it can rewrite general distributed protocols without needing users to reason about the correctness of the transformations. The hope is that by programming backend logic in Hydroflow, developers can more efficiently utilize the availability of cloud-scale compute by focusing on their business logic and letting our optimizer figure out how to distribute it. Hydroflow is open-source at https://github.com/hydro-project/hydroflow and we encourage anyone interested to contribute optimization opportunities, challenging protocols, and new deployment targets!

4. You are a prolific OSS contributor, on your website, you say: "I explore the boundaries of programming". In your opinion, how important OSS is for pushing the boundaries of programming?

OSS software has a unique role in the development of new systems, especially those that affect how other developers build their own applications. While industry often leads in the development of systems with higher performance and cost efficiency, these systems are often specialized for specific use cases and deployment scenarios. The beauty of open-source is that it forces the system developers to create the abstractions necessary for a wide audience to use the system, and opens up the software to contributions from the community that increase its flexibility by aggregating use cases from around the world. I think for modern systems, being open-source (or at least source-available) is quite critical if you want adoption from a wide range of developers. It naturally creates a community of people excited about the technology and making it more accessible, which is the first step towards stretching the boundaries of what we can do with programming!

If you've found this conversation with Shadaj Laddad as enlightening as we have, you won't want to miss the chance to learn even more. Shadaj will be speaking at the upcoming SBTB conference this November. It's a golden opportunity to delve further into the intricacies of Hydroflow, distributed systems, and the intersection of academia and industry. You'll even have the chance to engage with Shadaj personally, asking questions and discussing the future of programming languages and distributed systems.

As a special offer for our readers, if you share this blog post on your social media platforms and send an email with the subject "Shadaj" and the link to your post to sbtb@konfy.care, you'll receive a 15% discount on your SBTB ticket.

Don't miss this unique opportunity to be at the forefront of revolutionary changes in the software engineering landscape.

See you by the Bay!

🌟 We're thrilled to host a Workshop "Building production-ready LLM-powered applications" with Josh Tobin, CEO of Gantry & co-creator of Full Stack Deep Learning.

A must-attend for anyone interested in #AI & #LLM! 🌟

👉 Secure your spot

About the course The way AI-powered apps are built has changed:

• Before LLMs, an idea would bottleneck on training models from scratch, and then it'd bottleneck again on scalable deployment.

• Now, a compelling MVP based on pretrained LLM models and APIs can be configured and serving users in an hour.

An entirely new ecosystem of techniques, tools, and tool vendors is forming around LLMs. Even ML veterans are scrambling to orient themselves to what is now possible and figure out the most productive techniques and tools.

In this course, we'll teach you how to build AI-powered applications from scratch, while following the best practices that will allow you to balance shipping quickly with building high-quality, production-ready applications your users trust. We'll walk you through a structured approach to AI app development loosely based on the test-driven development methodology used in traditional software engineering.

About your Instructor Josh Tobin is the cofounder and CEO of Gantry, co-creator of Full Stack Deep Learning, and a former OpenAI / UC Berkeley AI researcher.

Gantry is building product testing and analytics for AI-powered applications. Testing and analytics are essential tools when building any product, but they’re even more essential for AI-based applications. That’s because these applications fail in harder-to-detect ways, and those failures erode user trust over time, eventually leading to churn. Gantry helps you build AI your users trust through powerful observability, analytics, and evaluation for your AI-powered products.

About The Full Stack Building an AI-powered product is much more than just training a model or writing a prompt.

The Full Stack brings people together to learn and share best practices across the entire lifecycle of an AI-powered product: from defining the problem and picking a GPU or foundation model to production deployment and continual learning to user experience design.

We've taught courses in building deep learning-powered applications at UC Berkeley, in-person, and online. More recently, we hosted the first LLM bootcamp focused on teaching practitioners how to build LLM-powered applications, from prompt engineering to retrieval augmentation and AI-first design. Our courses have featured contributed lectures from Pieter Abbeel (Professor at UC Berkeley and co-founder of Covariant), Richard Socher (co-founder of You.com and former Chief Scientist / EVP at Salesforce), and Andrej Karpathy (OpenAI researcher and former Director of AI at Tesla).

Our courses have been described as "high quality tokens" by Andrej Karpathy, "the most comprehensive and interesting class I ever attended" by Boris Dayma (creator of Craiyon and Dall-E mini), and "I can't believe they made this available for free" by Jo Kristian Bergum (Distinguished Engineer at Yahoo and Co-Creator of Vespa).

Register now

Have you ever wondered if there's a parallel universe where software developers are the superheroes of scientific research? Well, guess what? That universe exists, and it's not an alternate reality. It's right here, in our world of Open Source Science, or OSSci for the cool kids!

Imagine a world where a researcher in Boston collaborates with a software engineer in Bangalore, using a tool built by a developer in Berlin, to solve a complex scientific problem. That's the power of open source science - a global community of bright minds working together, breaking through traditional boundaries of knowledge, and creating groundbreaking solutions.

This isn't a fantasy; it's happening right now, in a movement called open science. One of the leaders in this field is a group known as Open Source Science (OSSci), steered by Tim Bonnemann at IBM, who we had the chance to talk to recently.

So, Tim let's journey to the roots, unraveling the enigma from the start: What's the concept of Open-Source Science?

Open-Source Science (OSSci) is a new initiative under NumFOCUS that aims to accelerate scientific research and discovery by improving any and all aspects of how open source is being applied in science – from better onboarding for students and young scientists on their journey towards becoming empowered OSS contributors and advocates, to encouraging collaboration across scientific disciplines, all the way to supporting ongoing efforts by some of the biggest funders of research globally to drive OSS best practices through the grants they award. OSSci is a community initiative. We bring together scientists who use open source to tackle the biggest challenges our planet is facing, OSS developers who love science, and other stakeholders, to share their ways of working, explore and document the OSS tool landscape, and identify common pain points.

Who's eligible to dive into the world of OSSci?

OSSci is open to anyone passionate about accelerating science through open source. Our interest groups – initially focused on chemistry/material science, life sciences/healthcare, and climate/sustainability – have been getting under way. Just fill out the application form to express your interest. We are starting to show up at conferences and events, in addition to (co-)hosting our own OSSci meetups. More opportunities to plug in and contribute will be announced over the coming months. Make sure to follow OSSci on Medium or subscribe to our monthly newsletter and be the first to know!

We're absolutely thrilled that an OSSci track is gracing our #bythebay conference. Our anticipation is sky-high for the fresh perspectives it will unveil and the new connections we're bound to make! What's your take on this?

Our goal is to elevate stories of successful use of open source in science but also highlight some of the particular challenges when it comes to open source in academia and science. We want to inspire our wider SBTB audience – some of the most capable people in tech today – to take a closer look at science and the unique technological challenges in front of us.

Key points the track will cover:

• Open-source software driving scientific advances

• Scientific research driving the development of innovative OSS tools and approaches

• Up-and-coming science-related OSS tools looking for contributors

• Lessons learned from collaboration in the open source for science space

Whether you’re a researcher, research engineer, OSS developer, administrator, or otherwise have experience at the intersection of open source and science – we want to meet you by the Bay!

We also invite you to watch Tim's interview with Doc Searls and Dan Lynch on their FLOSS Weekly show :

Eager to learn more about Open Source Science? Here are some USEFUL LINKS:

https://chief.sc/ossci-join

https://opensource.science/

http://eepurl.com/imrbTc

So, we extend our thanks to Tim for his valuable time and illuminating perspectives!

Open Source Science presents an exciting opportunity for everyone in the software development and scientific research fields. Each line of code can bring us closer to groundbreaking discoveries. We invite you to bring your passion, expertise, and curiosity to the SBTB conference. Remember, we are all contributors, innovators, and key players in this shared journey. Let's explore this vast realm of Open Source Science together.

Can't wait to see you there!

Welcome back to our ongoing series of blog posts where we spotlight the key aspects of our various tracks. We aim to inspire you with past presentations and provide an idea of what we anticipate seeing on stage this year.

You have a last few days before we close our CFP, take your chance and submit your best talk!

Today, we're focusing on Cloud Architectures. Cloud Architectures looks at the ways full stack systems are built these days — from APIs to multiplatform clients to LLM clients. Everything old is new again with the async programming required to query LLM oracles at scale. With multimodal AI apps shifting vast volumes of data—texts, images, videos, logs—we rely on tried-and-tested systems like Kafka and Spark to keep the data flowing. But, with the emergence of a new generation of tools, and the expanding of cloud spaces, and it has become obvious that we need robust methods to test, deploy, and monitor these tools.

So, we are eager to learn about deployment of LLM workflows in the cloud, how to leverage existing tools and even build new ones.

Join us for retrospective of the Cloud Architectures track. We are retracing past successes for you to get inspired and share your innvovations with us.

Ryan Knight: "Data Consistency Patterns in Cloud Native Applications"

In Cloud Native Architectures, services need to stay loosely connected and not share data, making data consistency hard to achieve. With many NoSQL databases favoring eventual consistency, the problem grew. We've seen solutions like the popular Distributed Saga pattern. Ryan Knight explored these complexities and shared alternative solutions using the database for consistency.

We like it for the exploration of popular and alternative patterns: it provides a fresh perspective and new solutions to a common problem.

Charles Pretzer: "Linkerd and the Service Mesh"

Charles Pretzer shared an insightful overview of Linkerd, a service mesh for Kubernetes. He traced Linkerd's evolution from its Scala, Finagle, Netty, and JVM roots in 1.x, to its modern 2.x version in Go and Rust. His talk covered the service mesh model, Linkerd's implementation, and valuable lessons from four years of production use worldwide.

Understanding of this model is crucial nowadays as service mesh architectures are a key component in modern distributed systems.

Mary Grygleski: "Deploying a Modern Serverless Reactive Container to the Cloud"

Mary Grygleski's session offered a deep dive into reactive systems and their benefits for managing concurrency in distributed systems. She discussed Vert.x's unique polyglot nature, event loop, and the Vertical model. Through live coding, Mary demonstrated building a use case with JVM languages like Java and Kotlin, dockerizing it, and deploying it as a serverless container using Knative to a Kubernetes cluster.

Live coding, if done well, is a killer. Mary illustrated how to code a simple use case using multiple JVM languages, and then built, dockerized, and deployed it as a serverless container. 😍

Annie Talvasto: "Top new CNCF projects to look out for"

Annie Talvasto's talk introduced the most exciting tools and projects from the Cloud Native Computing Foundation (CNCF), known for hits like Kubernetes & Prometheus. Her compact, demo-rich presentation aimed to inspire attendees with relevant tech for the projects and keep them ahead of the curve with the trends.

This overview helps attendees stay up-to-date with the technologies and tools.

And the demo format makes it an engaging, fast-paced presentation.

Eynav Mass: "Prepare Your System To Scale (OR Why Auto-Scaling Is Not Enough)"

The talk delved into the often underestimated topic of scale, highlighting its importance for growing businesses. Eynav Mass shared insights on how to best prepare systems for anticipated future load, discussing key considerations often overlooked in the initial phases of preparing for scaling.

This talk is valuable for us as it focuses on how to plan for scale - a crucial but often overlooked aspect.

Viktor Gamov: "Proxies, Gateways, and Meshes: Cloud Connectivity 101 for Developers"

Viktor Gamov's talk delved into the evolution of API gateway technology, from managing network runtime to full lifecycle API management. He highlighted how service mesh emerged as a new pattern for microservice communication. The session explored the differences and similarities between API gateways and service mesh, providing practical advice on when to employ each, and clarifying their roles in the communication layer.

This talk is appealing to the audience as it covers essential elements of modern app development: API gateways and service meshes.

While you've been pondering, we've extended our CFP. So, dust off your keyboard, summon your expertise, and step onto our stage.

Join us!

Petr Zapletal is a Principal Engineer at Disney Streaming who specializes in the design and implementation of highly scalable, reactive and resilient distributed systems. He is a functional programming and open source evangelist and has expertise in the area of big data and machine classification techniques. Petr participates in the whole software delivery lifecycle; from requirement analysis & design, through to maintaining systems in production. During his career, Petr has worked for various companies from start-ups to large international corporations.

Petr's current interests are Reactive Systems, Distributed Streaming, and Deep Learning. Petr is also an author of ThisWeekInScala and a seasoned conference speaker.

1. In your 2019 presentation, you talked about Change Data Capture. Can you explain how your perspective on CDC has evolved over the years and why?

In my 2019 presentation, I discussed Change Data Capture (CDC). Over the years, I've witnessed its growing importance and impact on data integration, real-time analytics, and event-driven architectures. I believe CDC has become an essential tool for capturing and propagating changes in data systems efficiently and reliably.

Learn more here:

2. Do you have any projects you're currently involved in? What is its primary focus, and what unique challenges does it pose?

Currently, I am involved in a project that explores predictive scaling for distributed systems, utilizing artificial intelligence (AI) to efficiently manage resources. This project focuses on proactively anticipating workload changes and making real-time decisions for optimal resource allocation. The main challenge is to get accurate predictions and be able to project them into actual cloud deployments.

3. In the context of AI, what potential do you see for Scala, and how might it compete or collaborate with other languages popular in AI like Python or smth?

Scala's role in the era of AI is interesting, and I think complementary to Python. While Python has emerged as the go-to language for AI due to its extensive libraries and ecosystem, Scala offers its strengths and opportunities.

Scala's functional programming paradigm, static typing, and seamless integration with existing Java libraries make it a powerful tool for data manipulation and processing, which are crucial steps in AI workflows. Scala's expressive type system presents potential opportunities for developers to enhance data consistency, improve code readability, and mitigate runtime errors while building complex AI systems. While Scala has the potential to complement Python and provide a strong foundation for data manipulation, it is important to note that adopting Scala in AI comes with its own set of challenges and considerations.

The integration of Scala with existing AI frameworks and libraries may require additional effort, and the learning curve for developers transitioning from Python to Scala could pose initial hurdles.

Nevertheless, exploring Scala's typesafety for building reliable and scalable AI models could be an avenue worth considering.

Embarking on a fresh chapter of our series "Meanwhile, We Continue to Cover our Tracks"

Up next: the powerful tandem of Data & AI. Guided by our theme: "Code and Data the Age of AI", we're sharing insights & best practices from past editions, offering a tantalizing preview of this year's journey.

Submit a talk! ONLY 1 DAY TO GO!

Always dreamt of AI that gorges on all the Data it can get its hands on? Welcome home!

We've always looked at AI fed by all the data you could give it, and consider it a art of distribuited systems and data pipelines. We're continuing to embed the Python stack in distributed systems and data pipelines but taking it a step further. Get ready for new ways to achieve scale. Thanks to LLM programming, all programming paradigm ways and frameworks for big and small data are back in action.

Consider this track as the grand finale of software engineering, diving into all possible ways to program with data and AI.

Chris Fregly: "Continuous ML Applications in Production"

Cris introduces continuous "VOTE" techniques (Validation, Optimizing, Training, Explainability) to improve machine learning pipelines using the open-source tool, PipelineAI.

The talk matters to us because it addresses a significant gap in real world enterprises, which is the ability to adapt and optimize AI models in real-time.

Anima Anandkumar: "Next-generation frameworks for Large-scale Machine Learning"

Anima showed how simple gradient compression (SignSGD) leads to communication savings while preserving accuracy, consequently explaining that with better algorithmic design, obtaining “free lunches” and better efficiency in ML is possible.

The talk addresses current challenges in the field of deep learning, such as the need for larger datasets and models, and the demand for more computing infrastructure

Antje Barth: "Put Your Machine Learning on Autopilot"

Antje Barth focuses on the concept of Automated Machine Learning (AutoML), live-demoing how it can automate tasks like data analysis, feature engineering, model training, and tuning in the ML workflow.

The significance of the talk lies in AutoML's potential to save time and effort. It can accelerate AI adoption and make the technology more accessible to a broader audience.

Adam Gibson: "Deploying and serving hardware-optimized ML pipelines using GraalVM"

GraalVM combined with the eclipse deeplearning4j framework to create and deploy ML pipelines combining Python scripts and hardware-optimized ML pipelines into one binary.

Hardcore optimizations down to hardware - can't get enough of it!

Adi Polak: "Rethinking scalable machine learning with Spark ecosystem"

Adi guides us through a typical ML workflow powered by Spark: data ingestion, data cleanup, feature extraction, model training, model serving, and scoring.

Apache Spark is a widely used tool for large-scale data processing and machine learning, making this talk highly relevant for many attendees.

Alexander O'Connor: "Transformers End-to-End: Experience training and deploying large language models in production"

Alexander peports his team's experience of leveraging LLMs for the kinds of problems they, and many other data science teams, face in customer support, ecommerce, and more.

Transformers and large language models are a significant trend in the field of AI and ML which we could not miss.

Alon Gubkin: "Building an ML Platform from Scratch"

Alon taught us how to set up an ML platform using open-source tools like Cookiecutter, DVC, MLFlow, KFServing, Pulumi, GitHub Actions and more.

Practical and Hands-On, Use of OSS, Relevance to ML, Experience, and Expertise - a canonical example of a good fit for our conference.

Together, let's aim for the celestial heights!

Grab your hot ticket!

Don't miss out on our social media updates! Keep an eye on our platforms for the latest and most exciting content. Stay connected and stay tuned!

Facebook

LinkedIn

Twitter

We've heard your request for more details on the talks we want to see and their priority on this track. In order to provide illustrative insights into our expectations, we have curated presentations from 2018-2021 conferences that exemplify it.

But if you're ready, wait no more and submit your talk to inspire others or grab your ticket to dive into the latest news from the field.

Let's ignite innovation and embrace the future together!

CFP

Tickets

Let's start!

Drawing from his extensive experience and exploration of Rust, Bryan Cantrill's talk aligns well with our focus on diverse programming languages and industry insights.

Chris Thalinger's talk on "Graal: How to use the new JVM JIT compiler in real life" perfectly complements our track, as it explains how to leverage Graal, highlighting its unique properties and usage in production.

Neha Narkhede underscores the critical role of event-driven architectures and microservices, and illustrates the use of Apache Kafka for real-time stream processing and unifying data. Everything we love - innovative data handling and software development practices!

Ólafur Geirsson introduces Metals, a language server enhancing code editing in various editors, and emphasizes the need for extending the Language Server Protocol (LSP) with key protocols for a comprehensive IDE experience. One of our favorite category of talks - make your life easier with tools while enhancing coding practices.

Themba Fletcher's talk delves into the technical nuances of a metadata-driven platform at Crunchbase, highlighting its agility, granular permission control, and the challenges of field type modifications and A/B testing implications.

Optimizing GraphQL query performance is key. With query plans, we can execute queries efficiently. Analyze cost, visualize, and even modify plans based on client knowledge. We care about business value and love reducing costs. And Greg Kesler unlocked the full potential of the queries with smart execution strategies.

Adam Warski discover Project Loom, the game-changer in JVM concurrency. Lightweight threads, continuations, and disruptive potential. Expolation of the future in tech is one of our preferred topics and concurrent programming on the JVM.

We are curious! Stewart Stewart will leave you with an increased appreciation for the symbiosis between math and FP when it comes to problem-solving, and the ability able to spot use cases for groups in future problems.

We know managing complexity in large applications is crucial and encourage our speakers to talk about that. Microservices offer a solution, but debugging issues can be challenging without proper tools. Tim Condon explores distributed tracing, understands its benefits, and implements it in Swift.

We invite top book authors to share threir experience reports. Sergey Winitzki delved into functional programming and decided to write a book. Discover his motivations and unique approach, leaving no stone unturned. Through practical working code, not mere theory, he unravels surprises and offers a fresh perspective on the practicality of functional programming.

SEE YOU BY THE BAY!

🌟 In the swift current of tech, we navigate by staying tuned into relevant blogs and new discoveries. Our compass 🧭? Continuous learning and exploration!

Can't imagine a week without our tech fix from 🚀Petr Zapletal's 'This Week in Scala'. For years, Petr's covered it with insightful reviews, relevant news, and captivating talks!

As a speaker, he has a knack for choosing topics that are not only relevant now but through the years. Back in 2019, we were fortunate to have him at our SBTB conference. His talk on 'Change Data Capture in Distributed Systems' was enlightening.

Modern systems are typically designed as a collection of cooperating micro-services, each with its dedicated data store to cater to individual needs. Given the various requirements, corresponding data are often stored in data stores of very different characteristics and use cases.

Petr's talk addressed the advantages and disadvantages of various CDC approaches, guiding in this often complex area. He generously shared his experiences, including samples, and offered recommendations that resonated with the audience.

Enjoy the video:

Delighted to invite Petr Zapletal to our CFP!

And we're opening the stage for all of you. 📢

Got a groundbreaking presentation? It's time to share it with the world.

Apply now!

🌟 The heart of #OSSci: Community, Mapping all scientific domains to corresponding OSS projects, and Development. We stick together, find gaps in science, and fill them with OSS. Because nothing beats the joy of communication and discovery!

Some details may age, but the principles are forever young!

Let's look back at OSS and delve into some of the timeless tips that are always up to date.

💡 From creating #ScalikeJDBC to observing the #Scala community's rise. Kazuhiro Sera shared five vital tips from his OSS experience: 1) Find your lifework project 2) Be careful about adding Scala dependencies 3) Stick with binary compatibility 4) Provide cross builds 5) Have effective CI builds

If you'd like to know more about the tips, this video is for you:

In 2018, Pathikrit Bhowmick brought a game-changer - mutable ArrayDeque, that outperforms most current mutable Scala collections like Lists, Buffers, Stacks and Queues. It has been a while since then, and things have changed, but it is still worth learning, especially because it encourages the audience to contribute other useful data structures like Ropes, Zippers, Disjoint Sets etc. to Scala. And these data structures are universal and not specific to Scala so it'll be useful for anyone.

If you are 🌟 curious about contributing to open-source #Scala projects on GitHub, Seth Tisue's talk in 2018 is your perfect guide! It navigated through the open-source landscape and advised on contributing in areas from documentation to the Scala compiler. Get involved today!

💐 We sincerely thank our wonderful community for your constant efforts and desire to improve things! We can't wait to witness the unveiling of something extraordinary at our 10th SBTB conference in November.

Let's keep the spirit alive!

Early Bird Tickets

CFP

We're thrilled to announce our first keynote speaker!

Meet Anthony J. Annunziata 👏👏👏!

Anthony Annunziuata started the IBM Quantum Network, that now includes hundreds of partners around the world learning Quantum Computing on IBM pioneering hardware and software. He then built a team that used OSS tools to help science at Accelerated Discovery, and Open-Source Science is a key community strategy that started and continues, at NumFOCUS, as one of the results.

Anthony and Alexy launched OSSci at SciPy 2022 in Austin, and now we're continuing the OSSci idea at our SBTB Conference!

Anthony J. Annunziata will present with the talk "AI, Quantum, and AI + Quantum: What’s Real, What’s Coming, and What’s the Point?"

The world keeps marching forward in two foundation-changing technologies: AI, and Quantum Computing. AI is and has been here, and rapidly advancing; foundation models trained on multi-modal data and useful for a multiplicity of tasks are advancing by the day. Real use cases for generative AI are taking off, for consumers, for developers, and for businesses. Applications in science and health are some of the most promising. At IBM and many other places, researchers and developers have come together to use AI to find new materials and drug candidates, to assist doctors with diagnosis and care delivery, and beyond. Meanwhile, quantum computing has been advancing steadily from dream to experiment and now getting close to practical reality for some select applications. We’re through the hype of the 2018-2021 period in quantum, and with recent announcements we are marching steadily toward useful-scale machines integrated as first class infrastructure in the cloud that developers can build on, using serverless functions powered by QPUs alongside CPUs, GPUs, and other novel chips and systems. And now we are starting to think about orchestration among quantum and classical resources for highly automated and accelerated workflows for complex problems like automated scientific discovery and synthesis. While much of this is still squarely in the future category, so was an AI that could pass the Bar exam only a year ago. In my talk I’ll provide both highlights and a clear point of view across this progress, highlight key challenges at the data, compute, platform, and application levels today and opportunities to solve them in the near future to enable AI + Quantum progress.

If you want to get more, 🗓️ mark your calendars for Scale by the Bay this November!

We can't wait to meet you all.

Register now!

Trust us, it will be an epic event you won't want to miss!

He laid the groundwork for Spark in the community, and together with Alexy Khrabrov, they launched the world's first Spark meetup. 🏆

Let's say that Spark is in our DNA here in the Communities By the Bay 🧬

Today's a great day to revisit Matei Zaharia's insightful talk at SBTB 2020.

In his talk Matei Zaharia discussed how they built Databricks' large-scale cloud service, managing millions of VMs and exabytes of data using Scala, Kubernetes, Envoy, and Prometheus.

Explored the design patterns and engineering processes they've adopted, and the changes were made to improve data analytics systems in the cloud, including ACID storage system and Apache Spark's auto-scaling and auto-shutdown features.

Please, enjoy the whole video:

It's full of valuable insights and may inspire you for something great!

Meanwhile, join us in November as we continue the adventure of Spark at SBTB 2023! ❤️

Get your pass

Blast from the past!

🎉 Announcing Retrospective of Spark at SBTB week!

Today we suggest you dive into 2019's last in-person gathering, unlock some valuable insights, and explore lessons learned.

Let's relive the magic of the Spark! ⚡️

While preparing for the 2019 edition, we interviewed Russell Spitzer and asked him what's the biggest thing that is misunderstood about Spark? He belived that the key misconception is that Spark guarantees instant speed boosts, but its true strength lies in effectively processing distributed data; for single-machine tasks, stick to one machine. And later, in his talk Russell Spitzer discussed three trends shaping the industry's future and whether #Spark may be the right fit for one's data problems.

So, "To Spark or not to spark" you can learn from his video and also enjoy the whole interview here.

Two incredible engineers Prashant Sharma & Nick Pentreath from IBM explored how to deploy an end-to-end ML pipeline with Apache Spark Streaming and Kubernetes, utilizing Structured Streaming for large-scale, real-time machine learning. Discovered how to preprocess data, host models using IBM MAX, and scale pipelines with Spark and Kubernetes. This talk included a live demo of real-time object detection in images. Key takeaways: reusing ML models with IBM MAX, and scaling online ML applications with Spark and Kubernetes.

Demo code

Spark & Delta Lake? Sure! Michael Armbrust showed that Delta Lake open-source storage layer offers ACID transactions, scalable metadata handling, and unifies streaming and batch data processing and explored Delta Lake Advantages: - ACID transactions on Spark: Ensuring consistent data for readers. - Scalable metadata handling: Managing petabyte-scale tables with ease. - Streaming & batch unification: Simplifying data ingest, backfill, and queries. - Schema enforcement: Preventing bad record insertion during ingestion. - Time travel: Allowing rollbacks, audit trails, and reproducible ML experiments. Learn more here:

A gentle reminder: If you've got awesome Spark ideas to share, don't forget to submit a CFP for Scale By The Bay!

Get Early Bird tickets 🐦 while they're hot!

See you by the Bay!

As promised, we have some incredible news to share with you - get ready for an exciting journey into the world of open-source science!

Today, we are thrilled to present our brand new track Open-Source Science and an exclusive interview with Alexy, the founder of the renowned Scale by the Bay Conference.

While we were writing this post, Alexy attended Stripe Sessions, where John Collison, President and Cofounder of Stripe, had a fireside chat with Sam Altman, CEO and cofounder of OpenAI.

Fresh from an insightful event, Alexy is eager to share this takeaway:

John asked Sam, when is AGI (Artificial General Intelligence) coming?

Sam said it depends on the definition of AGI and asked for his own, said: when AI can solve really hard scientific problems, and create new knowledge. OSSci is inspired by the same vision.

OSSci is inspired by the same vision. Learn more in this interview as Alexy sheds light on groundbreaking research, collaborations, and the significance of making scientific knowledge accessible to all.

• What is Open-Source Science?

Open-Source Science (OSSci) is an open-source community initiative at NumFOCUS, the home of Jupyter, pandas, NumPy, SciPy, scimitar-learn, the bulk of the Python Data Science Stack. It's cofounded by IBM Research and its Accelerated Discovery team, and started by yours truly as the OSS strategy for science. We bring together OSS developers and researchers to accelerate science and solve the hardest challenges facing humanity — cure cancer, reverse climate change, develop new materials for the growing humankind.

You can read more at opensource.science.

• Why did we add the Open-Source Science track to the conference program?

We need to spot light OSSci allies — those scientists, labs, companies who lead with OSS tools, who collaborate and contribute, and the OSS developers inspired by the cool problems science asks.

• Who should submit a talk?

A researcher using OSS tools that are instrumental or have tangible impact on your work, and want to share your experience and/or invite collaborators. A developer of OSS tools used in research, a research engineer supporting scientists, who understands what makes a difference for your lab, and can share your findings and enable others enable their science colleagues to accelerate discovery.

• What is the main takeaway for the audience?

You will expand your horizon beyond the usual e-commerce topics prevalent in the Valley, and see an enormous world powered by national and nonprofit funding dedicated to solving long-term, challenging problems while setting up educational and community engagement ecosystem. Expect to be inspired and find ways to contribute to human progress!

Please remember that we're looking for the best talks, and if you have something to share about Open-Source Science, we want to listen!

Submit your best talk now

We are also looking for the best attendees ;). so if you want to know the latest updates from the OSS field, secure your Early Bird ticket now!

Join SBTB 2023

Hope to see you by the Bay! ❤❤❤

Let’s dive into the tracks that we are preparing for you! One of our classic tracks is Thoughtful #SoftwareEngineering. It embraces best practices in coding, testing, and maintaining software, now augmented by AI copilots.

Let's look at some of our best performances from the previous editions of the Scale By the Bay Conference.

The 1st one we want to share is terrific @li_haoyi, and in his talk, he explored how far one could stretch the Scala REPL, performing complex tasks that you may associate with greater effort.

The next, @Oli_kitty and @andygscott guided you through recursion schemes fundamentals and Droste, a recursion library for Scala. Oli first joined SBTB as an attendee then she became a speaker. Now, she is the producer of the conference for the past 3 years.

We also adore panels!

And in 2020, we had the Programming Languages in the Age of the Cloud panel with @runarorama & @rakyll, who explored how developers’ life changes with the ascent of ubiquitously distributed systems.

And if you want to learn the latest news from the Software Engineering, you'd better buy your early bird ticket to SBTB'23 now!

Get your Ticket

See you by the Bay!

Subscribe now

The 10th Anniversary SBTB is coming back for its best year ever!

The theme of this edition is Code and Data in the Age of AI.  The LLMs can help us code, but the coding will become more interesting, not less. Data is feeding the AI and it's ever more important to make it fast and treat it safely.  And AI becomes Socratic and Reasonable -- with dialogue programming exercising all the skills we honed at SBTB for years, processing billions of messages asynchronously, augmeting workflows with retrieval from vector databases, and so on.  

1. CFP is open

Submit your best talk by May 31 for the first cut of the program, and by June 15 for a chance to present at SBTB 2023!

Submit a talk

2. Book a super early bird ticket!

Enjoy early bird prices while they last  -- it’ll go up soon!

Register

3. We need you!

Would your company or your meetup group be a partner of Scale By the Bay? Sponsoring, guest blogging, meetup participation, you name it. If so, please email sponsors@bythebay.io and let us know who we could engage with

In the next newsletters

We have a lot to share with you, please stay tuned for:

• How iconic conference tracks Thoughtful Software Engineering, Cloud Architecture and Data evolved in the Age of AI?

• What’s the topic of a brand new track we added this year and why it matters?

• Get ready to dive into the world of AI like never before. Our fire side chat with a special guest will blow your mind!