Import Configuration

Import

Overview

The main project for the import is in the solution Import folder, CTI.Import.Key project files: Program class is the startup, and calls separate classes for each type of import. RegistryImport includes the code to retrieve data from the registry - in the registry envelope container The key app.config settings will be described below. Import.Services NOTE: We are using the V3 import methods and the related MappingHelperV3 class. The original import methods have not been removed yet Handles mapping from the JSON-LD format from the registry to POCO classes After mapping the data successfully the appropriate related method from the workIT.Services project will be called to handle saving the data, marking data to be updated in elastic, and any caching methods. Example CredentialServices.Import( ThisEntity entity, ref SaveStatus status ) workIT.Services Has all the business service layer code. There will be code used just by the import as well as by the credential finder All data CRUD code is in the workIt.Factories project We mostly use entity frameworks (database first) for interfacing with the database. workIT.Data contains the related edmx files. We used separate edmx files for tables and views. There are a few stored procedures in use, mostly for retrieving data to load into the elastic indices Most DTO classes are in the workIT.Models project

App.Config

There will be many of the same keys used by the web project.

Primary Key values

Application Key	Value/Notes
GeoNamesUserName	Provide your custom user name for google geo services
scheduleType	Options: weekly, (any updates in the last 7 days) daily, (any updates in the last 24 hours) sinceLastRun, (gets the last run date for the import from the activity log and uses as the starting date) hourly, (any updates in the last hour) adhoc, (manually set the starting and ending dates) or integer - (interpreted as meaning minutes. Could use to schedule for 7:00 am and minutes since 8pm previous night. The maximum is 1440 - one day.)
DoingDownloadOnly	Normally false, set to true if say only want to download and save the registry document and not import data
startingDate	If doing an adhoc run, provide the starting date. NOTE: start/end dates will be converted to UTC before calling the registry searches. If the dates will be provided in UTC, be sure to set usingUTC_ForTime to false.
endingDate	If doing an adhoc import and the endDate is to be less than the current date, then provide an end date.
usingUTC_ForTime	If true (default), the import will convert any calculated dates to UTC.
Import Entity options. Importing_credential Importing_assessment_profile Importing_learning_opportunity_profile importing_organization Importing_cost_manifest_schema importing_condition_manifest_schema importing_competency_frameworks	Normally these are all true. Set to false if wish to only import particular entities.
Skipping options - normally false	special codes for imports to handle special cases. Typically due to a change to data format in the registry
deleteAction	Actions: 0-normal; 1-DeleteOnly; 2-SkipDelete
doingExpandOfRegion	If true expand state abbreviations to the full name
Keys that are needed due to common code used by the finder, but not applicable directly to the import	<add key="credentialCacheMinutes" value="1" />     <add key="organizationCacheMinutes" value="1" />     <add key="learningOppCacheMinutes" value="1" />     <add key="maxKeywordLength" value="200" />     <add key="maxReferenceTextLength" value="900" />     <add key="maxReferenceUrlLength" value="600" />
CR registry	Keys related to the target credential registry environment
MyCredentialEngineAPIKey	Provide your API Key from the CE accounts site
usingGraphDocuments	Always true now.
sandbox	If downloading from the sandbox:
	<add key="credentialRegistryUrl" value="https://sandbox.credentialengineregistry.org" />     <add key="credentialRegistryGet" value="https://sandbox.credentialengineregistry.org/ce-registry/envelopes/{0}" />     <add key="credentialRegistrySearch" value="https://sandbox.credentialengineregistry.org/ce-registry/search?" />     <add key="credentialRegistryResource" value="https://sandbox.credentialengineregistry.org/graph/{0}" />
production	If downloading from production:
	<add key="credentialRegistryUrl" value="https://credentialengineregistry.org" /> <add key="credentialRegistryGet" value="https://credentialengineregistry.org/ce-registry/envelopes/{0}" /> <add key="credentialRegistrySearch" value="https://credentialengineregistry.org/ce-registry/search?" /> <add key="credentialRegistryResource" value="https://credentialengineregistry.org/resources/{0}" />
Elastic search related
elasticSearchUrl	Url to the instance of elastic being used
credentialRebuildPageSize	Was used in prototyping, typically set to a large size like 9999. The import typically imports a small amount daily, or hourly. More pertinent for a full rebuild.
credentialCollection organizationCollection assessmentCollection learningOppCollection	Name of elastic collection that will be updated by the import. Can use different collections for different sources. For example one for a test environment and another for production copies.
Options to use or not use Elastic for the searches usingElasticCredentialSearch usingElasticOrganizationSearch usingElasticAssessmentSearch usingElasticLearningOppSearch	These are always true, unless using sql server for searching. Note that the sql search has been set as the default, so these keys will all be set to false.
delayingAllCacheUpdates	False - will update caches, and elastic on a per record basis, True - store requests in the SearchPendingReindex table, and handle at end of import.
updateCredIndexAction	Action to take when updating an elastic index. 0-none; 1- Use Index; 2-use Bulk (default)
Section for email	The user would update this section for providers for sending emails. There is an email manager with code to Mailgun, an api service and localhost (not recently verified). The email service is disabled by default.
	<add key="sendEmailFlag" value="FALSE" />
Section for tracing and logging	The utilities project has code for tracing, logging errors, and saving files (example files downloaded from registry). The same approach is used in the finder project. The default folder for logs is c:\@logs. Create the latter folder and give modify access as needed for your environment, or just select Everyone.
appTraceLevel	Set to the max level of tracing to display. The DoTrace methods start with a display level. The actual method that outputs trace messages to a file (and console) will only output message with a display level less than or equal to the value of appTraceLevel. LoggingHelper.DoTrace( 8,”some message to only show in rare circumstances” ); Or to messages to always log. LoggingHelper.DoTrace( 1, string.Format( " - Updates since: {0} ", startingDate ) );
path.error.log	Path and filename for file to store error messages. Uses a pattern so only a month of files are kept, and then overwritten. C:\@logs\[date]_FinderImport_ErrorLog.txt
path.trace.log	Path and filename for file to store trace messages. Uses a pattern so only a month of files are kept, and then overwritten. C:\@logs\[date]_FinderImport_TraceLog.txt
path.log.file	Path and pattern for saving files such as registry downloads. C:\@logs\RegistryDownload\[date]_[filename].json If using the latter default value, be sure to create the RegistryDownload folder in the C:\@logs folder.
path.email.log	Path to file for logging emails, if logAllEmail=yes
Connection strings
DefaultConnection AccountEntities	These are not enabled for this project - yet. Authentication is not used by the import. The finder uses SSO via the accounts site.
MainConnection workIT_RO workITEntities workITViews	These are set up for sql server. User should only have to provide the database name (if not using the default), server name/data source, and password.

Next: Import Workflow