Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 5 additions & 17 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,24 +17,12 @@ of Texas and beyond.

## Upcoming PyTexas Meetups

### February Meetup - February 4, 2025
### March Meetup - March 4, 2025

#### Celery and its challenges in production - Dishant Sethi
#### Accelerate your pandas application with FireDucks; A multi-threaded DataFrame library with JIT Compiler - Sourav Saha

"Asynchronous task queues like Celery have become essential in handling time-consuming or background jobs and optimising performance in production.
In this talk, I will introduce [FireDucks](https://fireducks-dev.github.io), that comes with a multithreaded kernel backend written in C++, a JIT compiler to auto-detect and optimize the existing performance issues involved in a user program, and a pure python frontend highly compatible with pandas allowing a native pandas application to be executed as it is without any manual changes. With the promise of highly compatible pandas APIs and the revved-up performance, FireDucks can serve the demands of this digital age and transform the arduous task of data wrangling into a more efficient and less taxing endeavor. It can be simply installed on any Linux system using pip.

Running Celery at scale comes with unique challenges. In this talk, will explore topics such as:
![Sourav Saha Avatar](https://drive.google.com/file/d/1rmVRsmsUVzBO3HUy81o33vV4DcoWauNb/view?usp=sharing){: style="height:150px;width:150px" align=left}

- maintaining worker reliability
- optimising tasks
- handling errors
- handling retries effectively
- task prioritisation
- monitoring
- autoscaling based on number of tasks under heavy workload"

![Dishant Sethi Avatar](https://avatars.githubusercontent.com/u/31399916){: style="height:150px;width:150px" align=left}

*I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions.*

*When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking.*
*Sourav has 11 years of professional experience at NEC Corporation, Japan in the diverse fields of High-Performance Computing, Distributed Programming, and Data Science. He holds a bachelor’s degree in Computer Science and is currently affiliated with the Digital Technology Development Laboratories at NEC as a Senior Research Engineer. Blending the mixture of different niche technologies related to compiler framework, high performance computing, and multi-threaded programming, his team is developing a high-performance compiler accelerated DataFrame library, named FireDucks. He is mostly passionate about program optimization, public speaking and has been speaking in several technical conferences, meetups etc.*
4 changes: 4 additions & 0 deletions docs/past_meetups/.authors.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,4 +67,8 @@ authors:
name: Jonathan Scofield
description: Jonathan Scofield is a data analyst and engineer in the music publishing industry. He regularly uses Pandas, Polars, and PySpark to build pipelines for asset metadata and transaction reports. Jonathan's journey with Polars began a year ago when he needed to migrate data between an on-premise database and a cloud-based solution. His entire team was awed by its rare combination of speed, flexibility, and simplicity. Since then, Polars has been a vital tool in his data toolbox and will continue to be for the foreseeable future. When he is not geeking out about data, he enjoys playing the piano and attempting to learn Mandarin.
avatar: https://jonathanscofield1.github.io/jonathanscofield/images/profile.png
dishantsethi:
name: Dishant Sethi
description: I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions. When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking.
avatar: https://avatars.githubusercontent.com/u/31399916

34 changes: 34 additions & 0 deletions docs/past_meetups/posts/2025-02-04.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: "Celery and Its Challenges in Production"
description: Dishant talked about Celery and the unique challenges of running it at scale.
date: 2025-02-04
categories:
- Celery
authors:
- dishantsethi
---

# December 2024 Meetup

Dishant talked about Celery and the unique challenges of running it at scale.

<!-- more -->
![Dishant Sethi Avatar](https://avatars.githubusercontent.com/u/31399916){: style="height:150px;width:150px" align=left}

#### Celery and its challenges in production - Dishant Sethi

Asynchronous task queues like Celery have become essential in handling time-consuming or background jobs and optimising performance in production.

Running Celery at scale comes with unique challenges. In this talk, will explore topics such as:

- maintaining worker reliability
- optimising tasks
- handling errors
- handling retries effectively
- task prioritisation
- monitoring
- autoscaling based on number of tasks under heavy workload."

*I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions.*

*When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking.*
13 changes: 8 additions & 5 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# This file was autogenerated by uv via the following command:
# uv pip compile requirements.in -o requirements.txt
babel==2.16.0
babel==2.17.0
# via mkdocs-material
cairocffi==1.7.1
# via cairosvg
cairosvg==2.7.1
# via mkdocs-material
certifi==2024.12.14
certifi==2025.1.31
# via requests
cffi==1.17.1
# via cairocffi
Expand All @@ -15,7 +15,10 @@ charset-normalizer==3.4.1
click==8.1.8
# via mkdocs
colorama==0.4.6
# via mkdocs-material
# via
# click
# mkdocs
# mkdocs-material
cssselect2==0.7.0
# via cairosvg
defusedxml==0.7.1
Expand Down Expand Up @@ -48,7 +51,7 @@ mkdocs==1.6.1
# mkdocs-redirects
mkdocs-get-deps==0.2.0
# via mkdocs
mkdocs-material==9.5.49
mkdocs-material==9.6.1
# via -r requirements.in
mkdocs-material-extensions==1.3.1
# via
Expand All @@ -72,7 +75,7 @@ pycparser==2.22
# via cffi
pygments==2.19.1
# via mkdocs-material
pymdown-extensions==10.14
pymdown-extensions==10.14.3
# via mkdocs-material
python-dateutil==2.9.0.post0
# via ghp-import
Expand Down