Optimization/large csv import and memory leakage job

[usama20]

Task 1:
I have optimized csv reading task by using techniques

Solution 1: (When user does not wants to process csv file in the background)

I have first converted CSV import logic to a Service to follow single responsability pattern
I have then read csv file one by one instead of loading the whole file in memory
I have then make two arrays for storing validated blogs and invalidate blogs to keep track of record as well so that we can see which records was validated and which are not
I have then bulk inserted record in db using insert_all instead of inserting. records one by one to db for each batch size
I have then created a file in logs directory for storing invalidated records.
In this way I have solved the problem of memory overflow and too much queries.
Solution 2 : (When user is comfortable in running the file in the background)

I have stored the file temporarily in the tmp folder of our application
I have then created a background job and perform_later to run it in the background
This job will call the same service with a batch size and process the csv file
Summary:
I have taken care optimizing service to ensure memory overflow does not occur and to check if our file is processed successfully or not I have handled both valid and invalid record in CSV files.

==========================================================================================

Task 2:
I have done optimization in this task by focusing on 3 problems

1. Instead of loading all blogs in memory I have used batch processing
2. Instead of calling api one by one and storing response one by one for each block I have created a separate job that will take a batch of valid blog_ids and fetch response for that batch
3. Lastly I have stored the responses of each blog in an array and bulk insert it in the api responses table

I have also removed all extra variables like instance variables of @processed_blogs do avoid memory leakage and allow garbage collector of ruby to do its task.