Shell imports by magrigsdev · Pull Request #17 · unflores/stats_visualiser

magrigsdev · 2025-02-19T02:12:27Z

add themes script
create themes.json
save on .json themes
no yet save on database
ParentId issue no yet fix

Deploy will require gd ext-zip extensions to be installed into php

…to string.

magrigsdev · 2025-02-20T02:02:55Z

ericlouz c'est mon deuxième compte git. vraiment desolé poir le melange de compte

unflores · 2025-02-20T11:28:00Z

Mes retours sont pour le plupart structural et sur le nommage des fichiers et variables. Sinon, c'est cool que tu as quelquechose qui nous ammène bien vers la but. Bravo.

Voila mes retours:

Faut garder les parentId en integer et pas varchar. C'est mieux pour les jointures et aussi pour la structure de ne pas dependre d'une donnée de domaine.
Faut enlever les DS_Store
Ne garde pas les fichiers data dans le repo. Déjà, git n'est pas fait pour garder des larges fichiers bin. Normalement, tu rajoutes un dossier data dans gitignore et on garde les flux d'imports ailleurs.
Globalement, on ne va pas faire l'intermediaire du fichier json, et on va laisser les ids etre set par la bdd.
Fait attention de ne pas commettre des functions qui ne sont pas utilisées.
Fait attention de ne pas definir des functions en public si elles ne sont pas utilisées publiquement.
Attention à l'usage des mots comme data ou utils ou tout mot qui ne renseigne pas sur le but de la fonction ou variable.

Au niveau d'objets je pense qu'il y a plus de sens à avoir les objets suivants:
CommandIngestTheme
Script/IngestTheme => Imports\Themes\ExtractService
Script/SaveThemes => Imports\Themes\CreateService
CommandSaveTheme => redundant. à la poubelle

Au niveau de responsabilites:
CommandIngestTheme

agit comme le point d'entree
gere les arguments
inject le nom du fichier import

Imports\Themes\ExtractService

agit comme un service de transformation des données
retourne un tableau d'objets pret à etre persisté

Imports\Themes\CreateService

Fait la persistence
Gere une validation eventuelle

unflores · 2025-02-21T09:51:50Z

.gitignore

 /public/assets/
 /assets/vendor/
 ###< symfony/asset-mapper ###
+.DS_Store


Il faut effacer tout les fichiers .DS_Store

unflores · 2025-02-21T09:55:41Z

migrations/Version20250221003103.php

+    public function up(Schema $schema): void
+    {
+        // this up() migration is auto-generated, please modify it to your needs
+        $this->addSql('CREATE TABLE theme (id INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, parent_id INTEGER DEFAULT NULL, code VARCHAR(255) NOT NULL, is_section BOOLEAN NOT NULL, external_id VARCHAR(255) NOT NULL)');


Je pense que ta bdd est en mauvaise etat comme la mienne ne necessite pas cette migration. En plus la fonctionne down suprimme la bdd 😅

quand je vais merger je vais pas merger les migrations

unflores · 2025-02-21T09:56:06Z

src/Command/CommandIngestTheme.php

+
+    protected function configure(): void
+    {
+        $this


Cela n'a pas besoin d'exister.

unflores · 2025-02-21T10:12:48Z

src/Command/CommandIngestTheme.php

+        if ($arg1) {
+            $io->note(sprintf('You passed an argument: %s', $arg1));
+            $excel_file = $this->projectDir.'/public/File/emissions_GES_structure.xlsx';
+
+            if (!file_exists($excel_file)) {
+                $io->error('file does not exist');
+
+                return Command::FAILURE;
+            }
+            try {
+                $themes = $ingestTheme->PrepareThemesForDatabase($ingestTheme->GetThemesFromExcelFile($excel_file));
+                $themes = json_decode($themes);
+
+                // $io->success('Résultat: '.$ingestTheme->checkThemesEmpty());
+            } catch (\Exception $e) {
+                $io->error('Erreur lors de la lecture du fichier : '.$e->getMessage());
+
+                return Command::FAILURE;
+            }
+
+            return Command::SUCCESS;
+        }


Pour ce fichier, j'attend voir cet interface: https://github.com/unflores/stats_visualiser/blob/936714b32b9904ae1b1f646e6d125c7d4f7b31fa/src/Command/CommandIngestTheme.php

Si tu veux plus en discuter j'ai des dispos. Quand on a discuté de la sortie des données, on a dit que ce sera plutot en form de php dictionary et pas json. Changes l'output s'il te plait.

oui je suis entrain de le modifier

unflores · 2025-02-21T10:15:15Z

src/Controller/Api/ThemesController.php

+    private $entityManager;
+    private $projectDir;
+


ces changements n'ont pas besoin d'etre ici

unflores · 2025-02-21T10:16:17Z

src/Import/IngestTheme.php

+    private $projectDir;
+    private $entityManager;
+    private $themeRepository;
+


le nom, signature de certaines methodes et le dossier de ce fichier doivent changer. Regarde mon feedback d'avant pour plus d'info.

unflores · 2025-02-21T10:23:39Z

src/Command/CommandIngestTheme.php

+                return Command::FAILURE;
+            }
+            try {
+                $themes = $ingestTheme->PrepareThemesForDatabase($ingestTheme->GetThemesFromExcelFile($excel_file));


La methode GetThemesFromExcelFile doit retourner qqch de ce format ci:

[ 'P.0' => [ 'code' => 'emission_ges', 'parent_id' => null, 'is_section' => true ], 'P.0.1' => [ 'code' => 'emission_ges.some_child', 'parent_id' => null, 'is_section' => true ], ... ]

dictionnaire de donnée.

je vais finir les changement et tu vas les voir.

unflores

il y a encore pas mal de changements à faire.

magrigsdev · 2025-02-25T10:20:05Z

Could you take a moment to review this? I’ve finished everything, and it's all completed.

unflores · 2025-02-25T13:48:49Z

src/Imports/Themes/ExtractService.php

+        $newSpreadsheet = new Spreadsheet();
+        $newSheet = $newSpreadsheet->getActiveSheet();


Je pense que ceci doit etre effacé.

src/Command/CommandExtractService.php

unflores · 2025-02-25T14:02:52Z

src/Imports/Themes/ExtractService.php

+    private function getParentIdByparentExternalId(string $parentExternalId): ?int
+    {
+        $result = $this->entityManager->getRepository(Theme::class)
+        ->createQueryBuilder('t')
+        ->select('t.id')
+        ->where('t.externalId = :externalId')
+        ->setParameter('externalId', $parentExternalId)
+        ->getQuery()
+        ->getOneOrNullResult();
+
+        return $result['id'] ?? null;
+    }


Normalement ce genre de code dois vivre dans tes repositories, donc un theme repo.

unflores · 2025-02-25T14:03:37Z

src/Imports/Themes/ExtractService.php

+    public function checkAllParentIdNotNull(): bool
+    {
+        $nullCount = $this->themeRepository->createQueryBuilder('t')
+        ->select('COUNT(t.id)')
+        ->where('t.id >= :startId')
+        ->andWhere('t.parentId IS NOT NULL')
+        ->setParameter('startId', 2)
+        ->getQuery()
+        ->getSingleScalarResult();
+
+        return 0 === $nullCount ? false : true;
+    }


Je pense que ceci doit etre defini dans ton theme repository 🤔

unflores · 2025-02-25T14:05:20Z

tests/Imports/Themes/ExtractServiceTest.php

+    {
+        $ExtractServices = new ExtractService($this->entityManager, $this->projectDir);
+        $saveThemes = $ExtractServices->SaveThemesOnDatabase();
+        $this->assertTrue($saveThemes, 'themes are not saved');


les themes doivent etre sauvgardés normalement non?

unflores · 2025-03-03T09:05:21Z

migrations/Version20250116221307.php

@@ -1,27 +0,0 @@
-<?php


celle-la elle n'est pas à effacer.

…is not null, except for the first theme.

…ctService.php file.

…RepositoryTest.phand ThemeTest.php file

ericlouz added 9 commits February 17, 2025 09:42

add file .Ds_store

73ae94c

phpoffice/phpspreadshee

4f8dc41

creation des fonctions pour la lecture et gestions de donnees excel

fa05a8c

save themes on themes.json file

3f0c7ed

fix code

33598d3

update composer and fix code IngestTheme

9dc1744

Modify the parentId property in the Theme entity and change its type …

02db447

…to string.

run testSaveImportedThemes on dataBase

52b7f22

Import completed and theme saved to the database.

85ec700

magrigsdev self-assigned this Feb 20, 2025

magrigsdev requested a review from unflores February 20, 2025 02:01

magrigsdev added 3 commits February 21, 2025 00:51

prepare theme for saving database

e0253af

it works ingest theme command

8fc9f03

The test works correctly for the first theme added to the database.

644a987

unflores reviewed Feb 21, 2025

View reviewed changes

src/Command/CommandIngestTheme.php

protected function configure(): void

{

$this

Copy link

Owner

unflores Feb 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cela n'a pas besoin d'exister.

magrigsdev requested a review from unflores February 21, 2025 09:56

unflores reviewed Feb 21, 2025

View reviewed changes

unflores requested changes Feb 21, 2025

View reviewed changes

magrigsdev added 2 commits February 21, 2025 12:09

Some changes in the naming of directories and functions

ec90a56

Shell imports has successfully completed.

31770ed

magrigsdev requested a review from unflores February 25, 2025 10:17

unflores reviewed Feb 25, 2025

View reviewed changes

unflores reviewed Mar 3, 2025

View reviewed changes

migrations/Version20250116221307.php

@@ -1,27 +0,0 @@

<?php

Copy link

Owner

unflores Mar 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

celle-la elle n'est pas à effacer.

magrigsdev and others added 22 commits March 3, 2025 23:05

Several functions in extraService.php have been modified and improved.

e7d3647

The isFirstThemeParentIdNull test completed successfully

ec167aa

The test was completed successfully, and the parentId for all themes …

e91cd7d

…is not null, except for the first theme.

The CommandExtractService command was completed successfully.

6fed003

The migration Version20250116221307 has been successfully added.

9a9a964

entity theme modify code by name

2310e5e

creation migrations/Version20250306062918.php

2efe854

Every occurrence of 'code' has been replaced with 'name' in the Extra…

612b04a

…ctService.php file.

Every occurrence of 'code' has been replaced with 'name' in the Theme…

75775b4

…RepositoryTest.phand ThemeTest.php file

Remove unused code

ebcb682

Remove unused parameter

f4be30b

Remove non-needed functions for theme processing

6b229be

Add ability to upsert themes

3b35115

Remove xlsx de repo

0b5a359

Update code to use built in repo methods

ab97ee9

Update names of array keys

7091bc6

Add gd extension for ci

ad25c1b

Add another lib

ad7bdfa

Remove prefix from extension

f0eeb49

Fix linter

5e489cc

Add test file

3949ed4

Add comment

1229fd8

unflores force-pushed the shell-imports branch from 6a2a5ab to 1229fd8 Compare March 10, 2025 09:11

unflores approved these changes Mar 10, 2025

View reviewed changes

Update themes location

870798b

unflores force-pushed the shell-imports branch from e0af9a3 to 870798b Compare March 10, 2025 09:33

unflores marked this pull request as ready for review March 10, 2025 09:37

unflores merged commit 95ecae0 into main Mar 10, 2025
1 check passed

unflores deleted the shell-imports branch March 10, 2025 09:40

		$newSpreadsheet = new Spreadsheet();
		$newSheet = $newSpreadsheet->getActiveSheet();

Conversation

magrigsdev commented Feb 19, 2025 • edited by unflores Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

magrigsdev commented Feb 20, 2025

Uh oh!

unflores commented Feb 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

unflores left a comment

Choose a reason for hiding this comment

Uh oh!

magrigsdev commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

magrigsdev commented Feb 19, 2025 •

edited by unflores

Loading

magrigsdev commented Feb 25, 2025 •

edited

Loading