Use sparse checkout for components

[kk-chesse]

This PR reworks how components are pulled from the remote. Which would fail, if you tried to pull files from a private project with http(s)

Old way

1. check if file exists on remote, to determine which directory structure to use (this steps would fail on private repositories)
2. "convert" component to project include
3. download as project include
4. handle as project include

New way

1. download components with a sparse checkout, download both possible locations in the same call (this way we avoid login requirements)
2. check if the included component exists
3. handle as component include

Additional changes
I had to change how parseIncludeComponent fetches to the tags. From try/catch to if/else, because if a firewall blocks ssh it takes a few minutes before the call timeouts...

TODOs

• I'm calling parseIncludeComponent twice, which is expensive, due to fetching tags
  Done, the result is cached now
• I'm not sure if the inner expands done for projects are neccessary for components
  Done, code is refactored so it can be used for project and component includes
• This PR remove the only use of Utils.remoteFileExist, but I don't know if you want to keep it around
  Done, function is removed

[ANGkeith]

using git sounds good to me. That way the storing of creds(like PAT) can be handled by git

these refactoring won't be easy, we have a couple of integration test, make sure those passes to prevent regression bugs

[kk-chesse]

I worked on the next version to. Changes;

• fixed the integration tests
• fixed the missing expands in inner local includes
• added a cache for the parsed components

The tests are working on my end and I can use it with our private Gitlab instance.

[ANGkeith]

nice work man, thanks for the contribution. in general lgtm, but will take another look when i have more spare time.
good to see all the integration test passes without much tweaking

I see you have TODOs in your pr description.

1. I'm calling parseIncludeComponent twice, which is expensive, due to fetching tags
2. I'm not sure if the inner expands done for projects are neccessary for components
3. This PR remove the only use of Utils.remoteFileExist, but I don't know if you want to keep it around

re. no.1 i think you have resolved it right?
re. no.2 what do you mean here?
re. no.3 i'm ok with removing that Utils.remoteFileExist

---

On a seperate topic,

i also see that you've considered using git archive.

a while back, i did some profiling and found that git archive can be more performant on elephant size repo like gitlab.com

interestingly, i face different error from you when attempting git archive via https.
getting the same error same as this but yeah, guess we have no choice but to go with sparse checkout

[kk-chesse]

I did a last round of refactoring and removed Utils.remoteFileExist. I think it's ready for review now.

re. no.1 i think you have resolved it right?

Yes, now the result is cached

re. no.2 what do you mean here?

That TODO was mostly for myself. I wasn't sure what some code did, but now I know and refactored the code accordingly.

re. no.3 i'm ok with removing that Utils.remoteFileExist

Done :)

I will update the opening post

---

Yeah, it's a shame git archive isn't working for https.

a while back, i did some profiling and found that git archive can be more performant on elephant size repo like gitlab.com

Super interesting profiling data, thank you for showing me that.

[ANGkeith]

i see that this is inspired from downloadIncludeProjectFile , wondering if we should clean up the temporary folder (${cwd}/${target}.${ext})

---

Not sure if i am missing something. I dont think --ignore-missing-args is required right ? since the templates folder should always exist

[kk-chesse]

IMHO, we should always cleanup in both downloadIncludeProjectFile and downloadIncludeComponent

---

The /template directory won't be created by the checkout if neither templates/component.yml nor templates/component/template.yml exists. This was a measure to defer the error handling for missing files to a later time.

We can either do the check later (as is done currently) or duplicate the check here. I'm open to anything.

[ANGkeith]

if you don't mind, could you update to perform the clean up .

other than that i think we're good to go ! thanks, nice work

[kk-chesse]

Sure, I added the cleanup code to both functions

Thank you for your review!

[kk-chesse]

Sorry forgot to run the linter. Im not using TS often.