diff --git a/docs/index.md b/docs/index.md index 26b926a..3ac79bf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -17,24 +17,12 @@ of Texas and beyond. ## Upcoming PyTexas Meetups -### February Meetup - February 4, 2025 +### March Meetup - March 4, 2025 -#### Celery and its challenges in production - Dishant Sethi +#### Accelerate your pandas application with FireDucks; A multi-threaded DataFrame library with JIT Compiler - Sourav Saha -"Asynchronous task queues like Celery have become essential in handling time-consuming or background jobs and optimising performance in production. +In this talk, I will introduce [FireDucks](https://fireducks-dev.github.io), that comes with a multithreaded kernel backend written in C++, a JIT compiler to auto-detect and optimize the existing performance issues involved in a user program, and a pure python frontend highly compatible with pandas allowing a native pandas application to be executed as it is without any manual changes. With the promise of highly compatible pandas APIs and the revved-up performance, FireDucks can serve the demands of this digital age and transform the arduous task of data wrangling into a more efficient and less taxing endeavor. It can be simply installed on any Linux system using pip. -Running Celery at scale comes with unique challenges. In this talk, will explore topics such as: +![Sourav Saha Avatar](https://drive.google.com/file/d/1rmVRsmsUVzBO3HUy81o33vV4DcoWauNb/view?usp=sharing){: style="height:150px;width:150px" align=left} -- maintaining worker reliability -- optimising tasks -- handling errors -- handling retries effectively -- task prioritisation -- monitoring -- autoscaling based on number of tasks under heavy workload" - -![Dishant Sethi Avatar](https://avatars.githubusercontent.com/u/31399916){: style="height:150px;width:150px" align=left} - -*I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions.* - -*When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking.* +*Sourav has 11 years of professional experience at NEC Corporation, Japan in the diverse fields of High-Performance Computing, Distributed Programming, and Data Science. He holds a bachelor’s degree in Computer Science and is currently affiliated with the Digital Technology Development Laboratories at NEC as a Senior Research Engineer. Blending the mixture of different niche technologies related to compiler framework, high performance computing, and multi-threaded programming, his team is developing a high-performance compiler accelerated DataFrame library, named FireDucks. He is mostly passionate about program optimization, public speaking and has been speaking in several technical conferences, meetups etc.* diff --git a/docs/past_meetups/.authors.yml b/docs/past_meetups/.authors.yml index d144a45..8ffa093 100644 --- a/docs/past_meetups/.authors.yml +++ b/docs/past_meetups/.authors.yml @@ -67,4 +67,8 @@ authors: name: Jonathan Scofield description: Jonathan Scofield is a data analyst and engineer in the music publishing industry. He regularly uses Pandas, Polars, and PySpark to build pipelines for asset metadata and transaction reports. Jonathan's journey with Polars began a year ago when he needed to migrate data between an on-premise database and a cloud-based solution. His entire team was awed by its rare combination of speed, flexibility, and simplicity. Since then, Polars has been a vital tool in his data toolbox and will continue to be for the foreseeable future. When he is not geeking out about data, he enjoys playing the piano and attempting to learn Mandarin. avatar: https://jonathanscofield1.github.io/jonathanscofield/images/profile.png + dishantsethi: + name: Dishant Sethi + description: I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions. When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking. + avatar: https://avatars.githubusercontent.com/u/31399916 \ No newline at end of file diff --git a/docs/past_meetups/posts/2025-02-04.md b/docs/past_meetups/posts/2025-02-04.md new file mode 100644 index 0000000..0e44740 --- /dev/null +++ b/docs/past_meetups/posts/2025-02-04.md @@ -0,0 +1,34 @@ +--- +title: "Celery and Its Challenges in Production" +description: Dishant talked about Celery and the unique challenges of running it at scale. +date: 2025-02-04 +categories: + - Celery +authors: + - dishantsethi +--- + +# December 2024 Meetup + +Dishant talked about Celery and the unique challenges of running it at scale. + + +![Dishant Sethi Avatar](https://avatars.githubusercontent.com/u/31399916){: style="height:150px;width:150px" align=left} + +#### Celery and its challenges in production - Dishant Sethi + +Asynchronous task queues like Celery have become essential in handling time-consuming or background jobs and optimising performance in production. + +Running Celery at scale comes with unique challenges. In this talk, will explore topics such as: + +- maintaining worker reliability +- optimising tasks +- handling errors +- handling retries effectively +- task prioritisation +- monitoring +- autoscaling based on number of tasks under heavy workload." + +*I’m Dishant! I’m a computer science graduate with over 5 years of experience in software development, cloud infrastructure, and AI/ML pipelines. I’ve had the opportunity to work with a few early-stage startups in Boston and New York, providing software consultancy. These days, I’m focused on building AI applications, helping teams solve technical challenges and build scalable solutions.* + +*When I’m not working on tech, you’ll probably find me in the kitchen, experimenting with new recipes. Feel free to reach out if you would like to talk about product engineering, AI, or cooking.* diff --git a/requirements.txt b/requirements.txt index e863092..213494b 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,12 +1,12 @@ # This file was autogenerated by uv via the following command: # uv pip compile requirements.in -o requirements.txt -babel==2.16.0 +babel==2.17.0 # via mkdocs-material cairocffi==1.7.1 # via cairosvg cairosvg==2.7.1 # via mkdocs-material -certifi==2024.12.14 +certifi==2025.1.31 # via requests cffi==1.17.1 # via cairocffi @@ -15,7 +15,10 @@ charset-normalizer==3.4.1 click==8.1.8 # via mkdocs colorama==0.4.6 - # via mkdocs-material + # via + # click + # mkdocs + # mkdocs-material cssselect2==0.7.0 # via cairosvg defusedxml==0.7.1 @@ -48,7 +51,7 @@ mkdocs==1.6.1 # mkdocs-redirects mkdocs-get-deps==0.2.0 # via mkdocs -mkdocs-material==9.5.49 +mkdocs-material==9.6.1 # via -r requirements.in mkdocs-material-extensions==1.3.1 # via @@ -72,7 +75,7 @@ pycparser==2.22 # via cffi pygments==2.19.1 # via mkdocs-material -pymdown-extensions==10.14 +pymdown-extensions==10.14.3 # via mkdocs-material python-dateutil==2.9.0.post0 # via ghp-import