Skip to content

Comments

add my first version of ATLAS scraper#32

Merged
pierrepo merged 10 commits intomainfrom
atlas_scraper
Jan 26, 2026
Merged

add my first version of ATLAS scraper#32
pierrepo merged 10 commits intomainfrom
atlas_scraper

Conversation

@sheikhPHD
Copy link
Collaborator

No description provided.

@pierrepo
Copy link
Member

pierrepo commented Nov 6, 2025

Thanks @sheikhPHD
Could you please reorganize all files into a single ATLAS folder?

@sheikhPHD
Copy link
Collaborator Author

Hi @pierrepo , here is the updated pydantic validated ATLAS scraper.
Could you please review? Thanks

@pierrepo
Copy link
Member

pierrepo commented Jan 7, 2026

Thanks @sheikhPHD
Could please remove all unnecessary files (in yellow bellow):
image
and keep only the scrap_atlas.py and .gitignore files ?

Also could you store collected metadata in two parquet files: atlas_datasets.parquet and atlas_files.parquet?

@pierrepo
Copy link
Member

Thanks @sheikhPHD 🎉

@pierrepo pierrepo merged commit 3d1c511 into main Jan 26, 2026
1 check passed
@pierrepo pierrepo deleted the atlas_scraper branch January 26, 2026 11:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants