diff --git a/docs/scorecards/concepts-and-structure.md b/docs/scorecards/concepts-and-structure.md index 0e5b96b5a4..eac12d6e6d 100644 --- a/docs/scorecards/concepts-and-structure.md +++ b/docs/scorecards/concepts-and-structure.md @@ -11,18 +11,26 @@ import TabItem from "@theme/TabItem" # Concepts and structure -## Scorecard structure table +In your [Builder](https://app.getport.io/settings/data-model) page scorecards are represented by three blueprints: +- [`Scorecard`](#scorecard-structure) - Represents a collection of rules and levels for evaluating entities. +- [`Rule`](#rule-blueprint) - Defines specific criteria for evaluation. +- [`Rule Result`](#rule-result-blueprint) - Stores the evaluation results for each entity. + +## Scorecard structure A single scorecard defines a category to group different checks, validations and evaluations. -Below is the structure of a single scorecard: +Below is the structure of a single `scorecard` blueprint: + +| Name | Type | Description | +|------|------|-------------| +| `Blueprint` | String (format: blueprints) | The target blueprint whose entities will be evaluated. | +| [`Levels`](#levels) | Array of objects | An array of levels with titles and colors (e.g., Bronze, Silver, Gold). | +| [`Filter`](#filter-elements) | Object | Optional query to filter which entities should be evaluated. | +| `Rules tested` | Number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of [rule](#rule-elements) evaluations performed. | +|` Rules passed` | Number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of successful [rule](#rule-elements) evaluations. | +| `% of rules passed` | Number ([calculation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/calculation-property)) | Calculated percentage of passed rules. | -| Field | Type | Description | -|------------------------------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------| -| `title` | `String` | Scorecard name that will be shown in the UI | -| `identifier` | `String` | The unique identifier of the `Scorecard`. The identifier is used for API calls, programmatic access and distinguishing between different scorecards | -| [`filter`](#filter-elements) | `Object` | Optional set of [conditions](#conditions) to filter entities that will be evaluated by the scorecard | -| [`levels`](#levels) | `Array` | The levels that we define for the scorecard, for example `Basic`, `Bronze`, `Silver`, `Gold` | -| [`rules`](#rule-elements) | `Object` | The rules that we create for each scorecard to determine its level | +Relations: The scorecard blueprint doesn't have any relations by default. A scorecard contains and groups multiple rules that are relevant to its specific category, for example a scorecard for _service maturity_ can contain 3 rules, while the _production readiness_ scorecard can contain 2 completely different rules. @@ -49,6 +57,7 @@ If the entity didn't pass any rule, it will be at the `Basic` level, and thus ca + ```json showLineNumbers { "identifier": "Ownership", @@ -96,7 +105,8 @@ If the entity didn't pass any rule, it will be at the `Basic` level, and thus ca -```json + +```json showLineNumbers { "identifier": "monitoringMaturity", "title": "Monitoring Maturity", @@ -123,7 +133,8 @@ If the entity didn't pass any rule, it will be at the `Basic` level, and thus ca -```json + +```json showLineNumbers { "identifier": "ProductionReadiness", "title": "Production Readiness", @@ -159,15 +170,7 @@ If the entity didn't pass any rule, it will be at the `Basic` level, and thus ca Rules enable you to generate checks inside a scorecard only for entities and properties. -A scorecard rule is a single evaluation consisting of multiple checks, each rule has a level which directly translates to how important it is for the check to pass (the more basic the check, the lower its level): - -| Field | Type | Description | -|---------------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `title` | `String` | `Rule` name that will be shown in the UI | -| `description` | `String` | Description that will be shown in the UI when the rule is expanded. Value that contains markdown is also supported and will be displayed in a markdown format | -| `identifier` | `String` | The unique identifier of the `Rule` | -| `level` | `String` | One of the levels defined in the scorecard [levels key](#levels) | -| `query` | `Object` | The query is built from a [`combinator`](#combinator) (or / and) and an array of [`conditions`](#conditions) | +A scorecard rule is a single evaluation consisting of multiple checks, each rule has a level which directly translates to how important it is for the check to pass (the more basic the check, the lower its level). ### Combinator @@ -236,23 +239,64 @@ Conditions are small boolean checks that help when determining the final status #### Available operators -| Operator | Supported Types | Description | -|---------------------|--------------------------------------------------|-----------------------------------------------------------------------| -| `=` | `String`, `Number`, `Boolean` | checks if the rule value is equal to the entity value | -| `!=` | `String`, `Number`, `Boolean` | checks if the rule value is not equal to the entity value | -| `<=` | `Number` | checks if the rule value is less than or equal to the entity value | -| `>=` | `Number` | checks if the rule value is greater than or equal to the entity value | -| `<` | `Number` | checks if the rule value is less than the entity value | -| `>` | `Number` | checks if the rule value is greater than the entity value | -| `contains` | `String`, `Number` | checks if the rule value is contained within the entity value | -| `containsAny` | `Array` | checks if any of the specified strings exist in the target array | -| `doesNotContains` | `String`, `Number` | checks if the rule value is not contained within the entity value | -| `endsWith` | `String`, `Number` | checks if the rule value ends with the entity value | -| `doesNotEndsWith` | `String`, `Number` | checks if the rule value does not end with the entity value | -| `beginsWith` | `String`, `Number` | checks if the rule value begins with the entity value | -| `doesNotBeginsWith` | `String`, `Number` | checks if the rule value does not begin with the entity value | -| `isEmpty` | `String`, `Number`, `Boolean`, `Array`, `Object` | checks if the rule value is an empty string, array, or object | -| `isNotEmpty` | `String`, `Number`, `Boolean`, `Array`, `Object` | checks if the rule value is not an empty string, array, or object | +| Operator | Supported Types | Description | +|---------------------|--------------------------------------------------|------------------------------------------------------------------------| +| `=` | `String`, `Number`, `Boolean` | checks if the rule value is equal to the entity value. | +| `!=` | `String`, `Number`, `Boolean` | checks if the rule value is not equal to the entity value. | +| `<=` | `Number` | checks if the rule value is less than or equal to the entity value. | +| `>=` | `Number` | checks if the rule value is greater than or equal to the entity value. | +| `<` | `Number` | checks if the rule value is less than the entity value. | +| `>` | `Number` | checks if the rule value is greater than the entity value. | +| `contains` | `String`, `Number` | checks if the rule value is contained within the entity value. | +| `containsAny` | `Array` | checks if any of the specified strings exist in the target array. | +| `doesNotContains` | `String`, `Number` | checks if the rule value is not contained within the entity value. | +| `endsWith` | `String`, `Number` | checks if the rule value ends with the entity value. | +| `doesNotEndsWith` | `String`, `Number` | checks if the rule value does not end with the entity value. | +| `beginsWith` | `String`, `Number` | checks if the rule value begins with the entity value. | +| `doesNotBeginsWith` | `String`, `Number` | checks if the rule value does not begin with the entity value. | +| `isEmpty` | `String`, `Number`, `Boolean`, `Array`, `Object` | checks if the rule value is an empty string, array, or object. | +| `isNotEmpty` | `String`, `Number`, `Boolean`, `Array`, `Object` | checks if the rule value is not an empty string, array, or object. | + +### Rule blueprint + +The `Rule` blueprint contains the following properties: +| Name | Type | Description | +|------|------|-------------| +| `Level` | String (enum) | The required level for this rule (must be one of the scorecard's defined levels). | +| `Query` | Object | The evaluation criteria for entities. | +| `Rule description` | String | Optional explanation of the rule's logic. | +| `Entities tested` | Number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of entities evaluated by this rule. | +| `Entities passed` | Number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of entities that passed this rule. | +| `% of entities passed` | Number ([calculation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/calculation-property)) | Calculated percentage of passed entities. | + +Relations: +| Name | Target Blueprint | Required | Many | Description | +|:----:|:----------------:|:---------:|:-----:|:-----------:| +| Scorecard | Scorecard | true | false | The scorecard this rule belongs to | + +### Rule result blueprint + +The `Rule result` blueprint contains the following properties: +| Name | Type | Description | +|------|------|-------------| +| `Result` | String (enum) | Whether the entity passed the rule ("Passed" or "Not passed"). | +| `Entity` | String | The identifier of the evaluated entity. | +| `Entity link` | String (url) | Calculated URL to the evaluated entity. | +| `Result last change` | Date-Time (mirror) | Last time the rule result changed. | +| `Scorecard` | String (mirror) | Mirror property showing the parent scorecard title. | +| `Blueprint` | String (mirror) | Mirror property showing the target blueprint. | +| `Level` | String (mirror) | Mirror property from the related rule. | + +Relations: +| Name | Target Blueprint | Required | Many | Description | +|------|-----------------|----------|-------|-------------| +| Rule | Rule | true | false | The rule that generated this result. | +| [Blueprint Identifier] | [Dynamic] | false | false | Automatically created relation to the target blueprint when a new scorecard is created. | +| Owning Teams | Teams | false | true | The relation to the Team blueprint is created by default. | + +:::info Dynamic Relations +When a new scorecard is created, Port automatically creates a relation in the Rule Result blueprint to the scorecard's target blueprint. For example, if you create a scorecard for the "service" blueprint, a new relation named "service" will be added to the Rule Result blueprint. +::: ## Scorecard total level calculation @@ -309,41 +353,66 @@ A scorecard filter is used to make sure only relevant entities are evaluated, on | Field | Description | |-----------------------------|-----------------------------------------------------------| -| [`combinator`](#combinator) | Defines the logical operation to apply to the query rules | -| [`conditions`](#conditions) | An array of boolean conditions to filter entities with | +| [`combinator`](#combinator) | Defines the logical operation to apply to the query rules.| +| [`conditions`](#conditions) | An array of boolean conditions to filter entities with. | + +## Important Notes + +1. The scorecard blueprints are protected and their core structure cannot be modified: + - Default properties cannot be changed or deleted. + - Required relations cannot be modified. + - The blueprints themselves cannot be deleted. + +2. You can extend the blueprints with: + - New properties. + - New non-required relations. + - Additional configurations that don't affect the core functionality. + +3. Rule Results are automatically generated and managed by Port: + - They cannot be created, deleted, or modified directly. + - You can update the custom properties you created for the rule results. + - Rule results are not searchable in the global search. + - They are updated automatically when rules are evaluated. + +## Validation Rules + +The system enforces several validation rules to maintain data integrity: + +1. Rule levels must match one of the levels defined in their parent scorecard. +2. Scorecard blueprint built-in relations cannot be renamed or modified. +3. Rule results maintain immutable core properties while allowing updates to custom properties. ## Scorecard UI indications After configuring scorecards for the blueprint, each entity created from it will have a `Scorecards` tab in its [entity page](/customize-pages-dashboards-and-plugins/page/entity-page), detailing the different checks and their results: -![Developer Portal Scorecards Tab](../../static/img/software-catalog/scorecard/tutorial/ScorecardsTab.png) + Additionally, the [catalog page](/customize-pages-dashboards-and-plugins/page/catalog-page) of each blueprint will automatically have a column for each scorecard rule. -For example, this `service` blueprint has 4 rules configured, and we can see a column for each of them in the catalog: +For example, this `Microservice` blueprint has five rules configured, and we can see a column for each of them in the catalog: -![catalogPageScorecardColumns](../../static/img/software-catalog/scorecard/catalogPageScorecardColumns.png) + ### Customizing views You can use table operations (sort, edit, group-by, etc.) to create various helpful views of your scorecards. -For example, here are the scores of all `Services` in an organization grouped by team: +For example, here are the scores of all `Microservice` in an organization grouped by team: -![catalogViewScorecardsByTeam](../../static/img/software-catalog/scorecard/catalogViewScorecardsByTeam.png) + Note that every column (scorecard metric) in the table has an aggregation in the bottom, hover over it to see the compliance of this metric across all entities in the table. ### Rule result summaries Scorecard rules are automatically added as columns in the relevant catalog page, and each such column is summarized on the bottom. -For example, these services have some rules defined in their scorecards, and we can see that: +For example, these microservices have some rules defined in their scorecards, and we can see that: -- 100% of `Team Batman's` services have an on-call defined, but only 67% of them have a PR cycle time shorter than 1500 - minutes. -- The bottom of the table contains an aggregation of the results of each rule for all services (across all teams). 11 - out of 18 services in total have a build success rate that is higher than 70%. +- 100% of `Ecosystem team's` microservices have an on-call defined, but only 67% of them have domain configured. +- The bottom of the table contains an aggregation of the results of each rule for all microservices (across all teams). Six + out of eight microservices in total have a domain configured. -![catalogRuleSummaries](../../static/img/software-catalog/scorecard/catalogRuleSummaries.png) + ## Next steps diff --git a/docs/scorecards/examples.md b/docs/scorecards/examples.md index 80a9808b17..649bef53f3 100644 --- a/docs/scorecards/examples.md +++ b/docs/scorecards/examples.md @@ -4,6 +4,7 @@ sidebar_position: 4 # Examples + ### Ownership scorecard The following example demonstrates an ownership scorecard. @@ -14,15 +15,17 @@ It has one filter defined: It has two rules: -- Check that a defined on-call exists and that the number of `open_incidents` is lower than 5; +- Check that a defined on-call exists and that the number of `open_incidents` is lower than 5. - Check if a team exists. +
+Ownership scorecard definition (click to expand) + ```json showLineNumbers -[ - { - "title": "Ownership", - "identifier": "ownership", - "filter": { +{ + "identifier": "ownership", + "title": "Ownership", + "filter": { "combinator": "and", "conditions": [ { @@ -32,7 +35,7 @@ It has two rules: } ] }, - "rules": [ + "rules": [ { "title": "Has on call?", "identifier": "has_on_call", @@ -61,15 +64,33 @@ It has two rules: "conditions": [ { "operator": "isNotEmpty", - "property": "$team" + "relation": "team" } ] } } - ] - } -] + ], + "levels": [ + { + "color": "paleBlue", + "title": "Basic" + }, + { + "color": "bronze", + "title": "Bronze" + }, + { + "color": "silver", + "title": "Silver" + }, + { + "color": "gold", + "title": "Gold" + } + ] +} ``` +
___ ### Ensure relation existence @@ -77,10 +98,13 @@ ___ Say we have a `service` blueprint that has a relation to another blueprint named `domain`. We can define a scorecard that checks that all of our services have a related domain. Services with empty `domain` relations will fail this check: +
+Ensure relation existence scorecard definition (click to expand) + ```json showLineNumbers { - "title": "Domain definition", "identifier": "domain_definition", + "title": "Domain definition", "rules": [ { "identifier": "hasDomain", @@ -96,9 +120,28 @@ We can define a scorecard that checks that all of our services have a related do ] } } + ], + "levels": [ + { + "color": "paleBlue", + "title": "Basic" + }, + { + "color": "bronze", + "title": "Bronze" + }, + { + "color": "silver", + "title": "Silver" + }, + { + "color": "gold", + "title": "Gold" + } ] } ``` +
___ @@ -109,10 +152,13 @@ To assess the deployment frequency of a `service`, simply checking the `deployme - Add an [aggregation property](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property.md) to the `service` blueprint that counts the number of related `deployment` entities. - Add a scorecard with a rule based on the new aggregation property: +
+DORA metrics scorecard definition (click to expand) + ```json showLineNumbers -{ - "title": "DORA Metrics", +{ "identifier": "dora_metrics", + "title": "DORA Metrics", "rules": [ { "identifier": "deployFreqBronze", @@ -144,6 +190,235 @@ To assess the deployment frequency of a `service`, simply checking the `deployme ] } } + ], + "levels": [ + { + "color": "paleBlue", + "title": "Basic" + }, + { + "color": "bronze", + "title": "Bronze" + }, + { + "color": "silver", + "title": "Silver" + }, + { + "color": "gold", + "title": "Gold" + } ] } ``` +
+ +___ + +## Automation examples + +### Send a Slack/Teams message whenever a scorecard rule result is updated + +#### Automation definition + +By using the `ENTITY_UPDATED` trigger type, we can run custom logic whenever an entity of a specific type is updated. + +The following configuration will cause a message to be sent whenever a scorecard rule result is updated: + +
+Automation configuration (click to expand) + +**Remember to change `github-org-name` and `github-repo-name` to your GitHub organization name and repository in the highlighted lines.** + +```json showLineNumbers +{ +"identifier": "ruleResultUpdated", +"title": "Rule result updated", +"trigger": { + "type": "automation", + "event": { + "type": "ENTITY_UPDATED", + "blueprintIdentifier": "_rule_result" + } +}, +"invocationMethod": { + "type": "GITHUB", + # highlight-start + "org": "github-org-name", + "repo": "github-repo-name", + # highlight-end + "workflow": "notify-rule-result-updated.yaml", + "workflowInputs": { + "rule_result_name": "{{ .event.context.entityIdentifier }}", + "entity_link": "{{ .event.diff.after.properties.entity_link }}" + }, + "reportWorkflowStatus": true +}, +"publish": true +} +``` +
+ +* `invocationMethod.workflowInputs` is the payload to be passed to the GitHub workflow upon every execution. In this example, we pass the rule result's identifier and the link to the evaluated entity. + +* `invocationMethod.reportWorkflowStatus` is set to `true` to automatically update the action run in Port with the status of the GitHub workflow. + +#### Backend - GitHub workflow + +The `notify-rule-result-updated.yaml` workflow will contain the logic to send a Slack/Teams message. + +

Prerequisite - set up webhooks

+ +The workflow requires a Slack webhook URL and/or a Microsoft Teams webhook URL to send the message. + +Slack: +1. To set up a Slack webhook, follow the instructions [here](https://api.slack.com/messaging/webhooks). +2. Once you have the webhook URL, add it as a secret in your GitHub repository named `SLACK_WEBHOOK_URL`. + +Microsoft Teams: +1. To set up a Microsoft Teams webhook, follow the instructions [here](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook). +2. Once you have the webhook URL, add it as a secret in your GitHub repository named `TEAMS_WEBHOOK_URL`. + +
+GitHub workflow (click to expand) + +This workflow includes steps to send a message via **Slack** and **Microsoft Teams**. +**Use only the step(s) that apply to your use case.** + +```yaml showLineNumbers title="notify-rule-result-updated.yaml" +name: Notify when rule result is updated + +on: + workflow_dispatch: + inputs: + # Note that the inputs are the same as the payload (workflowInputs) defined in the automation + rule_result_name: + description: "The rule result's name" + required: true + type: string + entity_link: + description: "A link to the evaluated entity" + required: true + type: string + +jobs: + send_message: + runs-on: ubuntu-latest + steps: + - name: Send message to Slack + env: + SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} + run: | + curl -X POST -H 'Content-type: application/json' --data '{"text":"The rule result ${{ inputs.rule_result_name }} has been updated. See evaluated entity: https://app.port.io${{ inputs.entity_link }}"}' $SLACK_WEBHOOK_URL + + - name: Send message to Microsoft Teams + env: + TEAMS_WEBHOOK_URL: ${{ secrets.TEAMS_WEBHOOK_URL }} + run: | + curl -H 'Content-Type: application/json' -d '{"text":"The rule result ${{ inputs.rule_result_name }} has been updated. See evaluated entity: https://app.port.io${{ inputs.entity_link }}"}' $TEAMS_WEBHOOK_URL +``` +
+ +--- + +### Create a GitHub issue whenever a scorecard rule result is degraded + +#### Automation definition + +By using the `ENTITY_UPDATED` trigger type, we can run custom logic whenever an entity of a specific type is updated. + +The following configuration will create a GitHub issue whenever a scorecard rule result's `Result` property changes from `Passed` to `Not passed`: + +
+Automation configuration (click to expand) + +**Remember to change `github-org-name` and `github-repo-name` to your GitHub organization name and repository in the highlighted lines.** + +```json showLineNumbers +{ + "identifier": "ruleResultDegraded", + "title": "Rule result degraded", + "trigger": { + "type": "automation", + "event": { + "type": "ENTITY_UPDATED", + "blueprintIdentifier": "_rule_result" + }, + "condition": { + "type": "JQ", + "expressions": [ + ".diff.before.properties.result == \"Passed\"", + ".diff.after.properties.result == \"Not passed\"" + ], + "combinator": "and" + } + }, + "invocationMethod": { + "type": "GITHUB", + # highlight-start + "org": "github-org-name", + "repo": "github-repo-name", + # highlight-end + "workflow": "create-issue-on-rule-degradation.yaml", + "workflowInputs": { + "rule_result_name": "{{ .event.context.entityIdentifier }}", + "entity_link": "{{ .event.diff.after.properties.entity_link }}" + }, + "reportWorkflowStatus": true + }, + "publish": true +} +``` +
+ +* `trigger.condition` checks the rule result's `Result` property before and after the update. The automation will only run for rule results that have been degraded. + +* `invocationMethod.workflowInputs` is the payload to be passed to the GitHub workflow upon every execution. In this example, we pass the rule result's identifier and the link to the evaluated entity. + +* `invocationMethod.reportWorkflowStatus` is set to `true` to automatically update the action run in Port with the status of the GitHub workflow. + +#### Backend - GitHub workflow + +The `create-issue-on-rule-degradation.yaml` workflow will contain the logic to create a GitHub issue. + +
+GitHub workflow (click to expand) + +```yaml showLineNumbers title="create-issue-on-rule-degradation.yaml" +name: Create issue when rule is degraded + +on: + workflow_dispatch: + inputs: + # Note that the inputs are the same as the payload (workflowInputs) defined in the automation + rule_result_name: + description: 'The rule result name' + required: true + type: string + entity_link: + description: 'A link to the evaluated entity' + required: true + type: string + +# Grant write access to issues so the workflow can create them +permissions: + contents: read + issues: write + +jobs: + send_message: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: create an issue + uses: dacbd/create-issue-action@main + with: + token: ${{ github.token }} + # By default, the issue will be created in the same repository as the workflow + repo: ${{ github.context.repo.repo}} + title: '${{ inputs.rule_result_name }} - degraded rule result' + body: | + The rule result ${{ inputs.rule_result_name }} has been degraded. + See evaluated entity: https://app.port.io${{ inputs.entity_link }} +``` +
\ No newline at end of file diff --git a/docs/scorecards/manage-scorecards.md b/docs/scorecards/manage-scorecards.md index ce0c468cb4..46a01e1beb 100644 --- a/docs/scorecards/manage-scorecards.md +++ b/docs/scorecards/manage-scorecards.md @@ -240,7 +240,7 @@ Here are some request examples that will create the Scorecard of Ownership on th After creating the Scorecards, you will see a new tab in the profile Entity page of each of your Blueprint's Entities, showing the various scorecards levels. -For example, we can [create the entity below](../build-your-software-catalog/sync-data-to-catalog/sync-data-to-catalog.md#creating-entities) +For example, we can [create the entity below](/build-your-software-catalog/sync-data-to-catalog/sync-data-to-catalog.md#entities): ```json showLineNumbers { @@ -255,13 +255,13 @@ For example, we can [create the entity below](../build-your-software-catalog/syn } ``` -And then look at the [specific page](https://app.getport.io/MicroserviceEntity?identifier=cart-service&activeTab=3) of this entity, on the scorecards tab +And then look at the [specific page](https://app.getport.io/MicroserviceEntity?identifier=cart-service&activeTab=3) of this entity, on the scorecards tab. -![Developer Portal Scorecards Tab](../../static/img/software-catalog/scorecard/tutorial/ScorecardsTab.png) + We can see that the `hasSlackChannel` rule passed because we provided one to that entity, while the `hasTeam` failed because we didn't provide any team. -Therefore the level of the entity is `Bronze` because it passed all the rules in the `Bronze` level (hasSlackChannel) +Therefore the level of the entity is `Bronze` because it passed all the rules in the `Bronze` level (hasSlackChannel). **Update scorecards** @@ -272,26 +272,31 @@ To update a scorecard you can use 2 different URLs: The request body will include the existing body of the Scorecard, after the desired updates to the existing Scorecard have been applied. +:::note +When using the multiple update Scorecards `https://api.getport.io/v1/blueprints/{blueprint_identifier}/scorecards` PUT request, keep in mind that you will see a new `id` property. This is used via Port to identify the Scorecard in order to be able to update its properties. +::: + **Delete scorecards** :::danger A Scorecard cannot be restored after deletion! ::: -- Make an HTTP PUT request and remove it from the array of the scorecards via the URL `https://api.getport.io/v1/blueprints/{blueprint_identifier}/scorecards` -- Make an HTTP DELETE request to the URL `https://api.getport.io/v1/blueprints/{blueprint_identifier}/scorecards/{scorecard_identifier}` the `scorecard_identifier` is the identifier of the scorecard we want to delete +- Make an HTTP PUT request and remove it from the array of the scorecards via the URL `https://api.getport.io/v1/blueprints/{blueprint_identifier}/scorecards`. +- Make an HTTP DELETE request to the URL `https://api.getport.io/v1/blueprints/{blueprint_identifier}/scorecards/{scorecard_identifier}` the `scorecard_identifier` is the identifier of the scorecard we want to delete. -:::note -When using the multiple update Scorecards `https://api.getport.io/v1/blueprints/{blueprint_identifier}/scorecards` PUT request, keep in mind that you will see a new `id` property. This is used via Port to identify the Scorecard in order to be able to update its properties +:::info Using entities API +Scorecards can be managed either with the dedicated **Scorecards API** mentioned above, or by using the standard **Entity API** endpoints: [Create](/api-reference/create-an-entity), [Update](/api-reference/update-an-entity), [Delete](/api-reference/delete-an-entity). ::: + **Create scorecards** -In order to create a scorecard from the [Terraform provider](../../build-your-software-catalog/custom-integration/iac/terraform/) , you will need to use the `port_scorecard` resource. +In order to create a scorecard from the [Terraform provider](/build-your-software-catalog/custom-integration/iac/terraform/) , you will need to use the `port_scorecard` resource. Here is an example of how to create an Ownership scorecard with the Terraform provider: @@ -353,13 +358,6 @@ In order to delete a scorecard using the Terraform provider, use the `terraform -## Scorecards as Blueprints - -Port allows you to manage scorecards as blueprints. -This feature enables you to create and manage scorecards using dedicated blueprints, allowing for rich customization, visualization and automation of your scorecards. - -To learn more, refer to the [scorecards as blueprints](/scorecards/scorecards-as-blueprints) page. - ## Next steps [Dive into advanced operations on Scorecards with our API ➡️ ](/api-reference/port-api) \ No newline at end of file diff --git a/docs/scorecards/scorecards-as-blueprints.md b/docs/scorecards/scorecards-as-blueprints.md deleted file mode 100644 index b3afc15caa..0000000000 --- a/docs/scorecards/scorecards-as-blueprints.md +++ /dev/null @@ -1,352 +0,0 @@ ---- -sidebar_position: 5 -title: Scorecards as blueprints -sidebar_label: Scorecards as blueprints ---- - -import PortTooltip from "/src/components/tooltip/tooltip.jsx" - -import Tabs from "@theme/Tabs" -import TabItem from "@theme/TabItem" - -:::info Closed beta feature -This capability is currently in closed beta, and is not yet generally available. -If you would like to join the beta, please reach out to us. -::: - -Port allows you to manage scorecards as blueprints. -This feature enables you to create and manage scorecards using dedicated blueprints, allowing for rich customization, visualization and automation of your scorecards. - -## Why manage scorecards as blueprints? - -With this powerful feature you can accomplish the following: - -1. **RBAC Management** - - Control who can create and modify scorecards and rules. - - Define which teams can view specific scorecard results. - - Manage permissions for rule creation and modification at a granular level. - -2. **Automations and Actions** - - Create workflows that trigger based on rule results. - - Set up automated notifications when entities change compliance levels. - - Build self-service actions to remediate failed rules. - - Integrate with external systems to sync scorecard data. - -3. **Reports and Dashboards** - - Build custom dashboards to visualize compliance across your organization. - - Track progress of entities through different levels. - - Generate reports on rule effectiveness and entity compliance. - - Monitor trends in rule results over time. - -## Overview - -After enabling `scorecards as blueprints`, three new blueprints will be created in your [data model](https://app.getport.io/settings/data-model): -- `Scorecard` - Represents a collection of rules and levels for evaluating entities. -- `Rule` - Defines specific criteria for evaluation. -- `Rule Result` - Stores the evaluation results for each entity. - -## Blueprint Structure - - - - - -The `Scorecard` blueprint contains the following properties: -| Name | Type | Description | -|------|------|-------------| -| Blueprint | string (format: blueprints) | The target blueprint whose entities will be evaluated | -| Levels | array of objects | An array of levels with titles and colors (e.g., Bronze, Silver, Gold) | -| Filter | object | Optional query to filter which entities should be evaluated | -| Rules Tested | number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of rule evaluations performed | -| Rules Passed | number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of successful rule evaluations | -| % of Rules Passed | number ([calculation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/calculation-property)) | Calculated percentage of passed rules | - -Relations: -| Name | Target Blueprint | Required | Many | Description | -|------|-----------------|----------|-------|-------------| -| - | - | - | - | No default relations | - - - - - -The `Rule` blueprint contains the following properties: -| Name | Type | Description | -|------|------|-------------| -| Level | string (enum) | The required level for this rule (must be one of the scorecard's defined levels) | -| Query | object | The evaluation criteria for entities | -| Rule Description | string | Optional explanation of the rule's logic | -| Entities Tested | number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of entities evaluated by this rule | -| Entities Passed | number ([aggregation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/aggregation-property)) | Number of entities that passed this rule | -| % of Entities Passed | number ([calculation](/build-your-software-catalog/customize-integrations/configure-data-model/setup-blueprint/properties/calculation-property)) | Calculated percentage of passed entities | - -Relations: -| Name | Target Blueprint | Required | Many | Description | -|------|-----------------|----------|-------|-------------| -| scorecard | Scorecard | true | false | The scorecard this rule belongs to | - - - - - -The `Rule result` blueprint contains the following properties: -| Name | Type | Description | -|------|------|-------------| -| Result | string (enum) | Whether the entity passed the rule ("Passed" or "Not passed") | -| Entity | string | The identifier of the evaluated entity | -| Result Last Change | string (date-time) | Timestamp of the last result change | -| Level | string (mirror) | Mirror property from the related rule | -| Scorecard | string (mirror) | Mirror property showing the parent scorecard title | -| Blueprint | string (mirror) | Mirror property showing the target blueprint | -| Entity Link | string (url) | Calculated URL to the evaluated entity | - -Relations: -| Name | Target Blueprint | Required | Many | Description | -|------|-----------------|----------|-------|-------------| -| rule | Rule | true | false | The rule that generated this result | -| [Blueprint Identifier] | [Dynamic] | false | false | Automatically created relation to the target blueprint when a new scorecard is created | - -:::info Dynamic Relations -When a new scorecard is created, Port automatically creates a relation in the Rule Result blueprint to the scorecard's target blueprint. For example, if you create a scorecard for the "service" blueprint, a new relation named "service" will be added to the Rule Result blueprint. -::: - - - - -## Important Notes - -1. The scorecard blueprints are protected and their core structure cannot be modified: - - Default properties cannot be changed or deleted. - - Required relations cannot be modified. - - The blueprints themselves cannot be deleted. - -2. You can extend the blueprints with: - - New properties. - - New non-required relations. - - Additional configurations that don't affect the core functionality. - -3. Rule Results are automatically generated and managed by Port: - - They cannot be created, deleted, or modified directly. - - You can update the custom properties you created for the rule results. - - Rule results are not searchable in the global search. - - They are updated automatically when rules are evaluated. - -## Validation Rules - -The system enforces several validation rules to maintain data integrity: - -1. Rule levels must match one of the levels defined in their parent scorecard. -2. Scorecard blueprint built-in relations cannot be renamed or modified. -3. Rule results maintain immutable core properties while allowing updates to custom properties. - -## Scorecard automation examples - -### Send a Slack/Teams message whenever a scorecard rule result is updated - -#### Automation definition - -By using the `ENTITY_UPDATED` trigger type, we can run custom logic whenever an entity of a specific type is updated. - -The following configuration will cause a message to be sent whenever a scorecard rule result is updated: - -
-Automation configuration (Click to expand) - -**Remember to change `github-org-name` and `github-repo-name` to your GitHub organization name and repository in the highlighted lines.** - -```json showLineNumbers -{ -"identifier": "ruleResultUpdated", -"title": "Rule result updated", -"trigger": { - "type": "automation", - "event": { - "type": "ENTITY_UPDATED", - "blueprintIdentifier": "_rule_result" - } -}, -"invocationMethod": { - "type": "GITHUB", - # highlight-start - "org": "github-org-name", - "repo": "github-repo-name", - # highlight-end - "workflow": "notify-rule-result-updated.yaml", - "workflowInputs": { - "rule_result_name": "{{ .event.context.entityIdentifier }}", - "entity_link": "{{ .event.diff.after.properties.entity_link }}" - }, - "reportWorkflowStatus": true -}, -"publish": true -} -``` -
- -* `invocationMethod.workflowInputs` is the payload to be passed to the GitHub workflow upon every execution. In this example, we pass the rule result's identifier and the link to the evaluated entity. - -* `invocationMethod.reportWorkflowStatus` is set to `true` to automatically update the action run in Port with the status of the GitHub workflow. - -#### Backend - GitHub workflow - -The `notify-rule-result-updated.yaml` workflow will contain the logic to send a Slack/Teams message. - -

Prerequisite - set up webhooks

- -The workflow requires a Slack webhook URL and/or a Microsoft Teams webhook URL to send the message. - -Slack: -1. To set up a Slack webhook, follow the instructions [here](https://api.slack.com/messaging/webhooks). -2. Once you have the webhook URL, add it as a secret in your GitHub repository named `SLACK_WEBHOOK_URL`. - -Microsoft Teams: -1. To set up a Microsoft Teams webhook, follow the instructions [here](https://learn.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/add-incoming-webhook). -2. Once you have the webhook URL, add it as a secret in your GitHub repository named `TEAMS_WEBHOOK_URL`. - -
-GitHub workflow (Click to expand) - -This workflow includes steps to send a message via **Slack** and **Microsoft Teams**. -**Use only the step(s) that apply to your use case.** - -```yaml showLineNumbers title="notify-rule-result-updated.yaml" -name: Notify when rule result is updated - -on: - workflow_dispatch: - inputs: - # Note that the inputs are the same as the payload (workflowInputs) defined in the automation - rule_result_name: - description: "The rule result's name" - required: true - type: string - entity_link: - description: "A link to the evaluated entity" - required: true - type: string - -jobs: - send_message: - runs-on: ubuntu-latest - steps: - - name: Send message to Slack - env: - SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} - run: | - curl -X POST -H 'Content-type: application/json' --data '{"text":"The rule result ${{ inputs.rule_result_name }} has been updated. See evaluated entity: https://app.port.io${{ inputs.entity_link }}"}' $SLACK_WEBHOOK_URL - - - name: Send message to Microsoft Teams - env: - TEAMS_WEBHOOK_URL: ${{ secrets.TEAMS_WEBHOOK_URL }} - run: | - curl -H 'Content-Type: application/json' -d '{"text":"The rule result ${{ inputs.rule_result_name }} has been updated. See evaluated entity: https://app.port.io${{ inputs.entity_link }}"}' $TEAMS_WEBHOOK_URL -``` -
- ---- - -### Create a GitHub issue whenever a scorecard rule result is degraded - -#### Automation definition - -By using the `ENTITY_UPDATED` trigger type, we can run custom logic whenever an entity of a specific type is updated. - -The following configuration will create a GitHub issue whenever a scorecard rule result's `Result` property changes from `Passed` to `Not passed`: - -
-Automation configuration (Click to expand) - -**Remember to change `github-org-name` and `github-repo-name` to your GitHub organization name and repository in the highlighted lines.** - -```json showLineNumbers -{ - "identifier": "ruleResultDegraded", - "title": "Rule result degraded", - "trigger": { - "type": "automation", - "event": { - "type": "ENTITY_UPDATED", - "blueprintIdentifier": "_rule_result" - }, - "condition": { - "type": "JQ", - "expressions": [ - ".diff.before.properties.result == \"Passed\"", - ".diff.after.properties.result == \"Not passed\"" - ], - "combinator": "and" - } - }, - "invocationMethod": { - "type": "GITHUB", - # highlight-start - "org": "github-org-name", - "repo": "github-repo-name", - # highlight-end - "workflow": "create-issue-on-rule-degradation.yaml", - "workflowInputs": { - "rule_result_name": "{{ .event.context.entityIdentifier }}", - "entity_link": "{{ .event.diff.after.properties.entity_link }}" - }, - "reportWorkflowStatus": true - }, - "publish": true -} -``` -
- -* `trigger.condition` checks the rule result's `Result` property before and after the update. The automation will only run for rule results that have been degraded. - -* `invocationMethod.workflowInputs` is the payload to be passed to the GitHub workflow upon every execution. In this example, we pass the rule result's identifier and the link to the evaluated entity. - -* `invocationMethod.reportWorkflowStatus` is set to `true` to automatically update the action run in Port with the status of the GitHub workflow. - -#### Backend - GitHub workflow - -The `create-issue-on-rule-degradation.yaml` workflow will contain the logic to create a GitHub issue. - -
-GitHub workflow (Click to expand) - -```yaml showLineNumbers title="create-issue-on-rule-degradation.yaml" -name: Create issue when rule is degraded - -on: - workflow_dispatch: - inputs: - # Note that the inputs are the same as the payload (workflowInputs) defined in the automation - rule_result_name: - description: 'The rule result name' - required: true - type: string - entity_link: - description: 'A link to the evaluated entity' - required: true - type: string - -# Grant write access to issues so the workflow can create them -permissions: - contents: read - issues: write - -jobs: - send_message: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@v4 - - name: create an issue - uses: dacbd/create-issue-action@main - with: - token: ${{ github.token }} - # By default, the issue will be created in the same repository as the workflow - repo: ${{ github.context.repo.repo}} - title: '${{ inputs.rule_result_name }} - degraded rule result' - body: | - The rule result ${{ inputs.rule_result_name }} has been degraded. - See evaluated entity: https://app.port.io${{ inputs.entity_link }} -``` -
\ No newline at end of file diff --git a/static/img/scorecards/custom-scorecards-example.png b/static/img/scorecards/custom-scorecards-example.png index e737b6c660..1df33a6c42 100644 Binary files a/static/img/scorecards/custom-scorecards-example.png and b/static/img/scorecards/custom-scorecards-example.png differ diff --git a/static/img/software-catalog/scorecard/catalogPageScorecardColumns.png b/static/img/software-catalog/scorecard/catalogPageScorecardColumns.png index 9f3d6cae07..cb4db9bdf8 100644 Binary files a/static/img/software-catalog/scorecard/catalogPageScorecardColumns.png and b/static/img/software-catalog/scorecard/catalogPageScorecardColumns.png differ diff --git a/static/img/software-catalog/scorecard/catalogRuleSummaries.png b/static/img/software-catalog/scorecard/catalogRuleSummaries.png index 469d659369..e9e3eb4aa3 100644 Binary files a/static/img/software-catalog/scorecard/catalogRuleSummaries.png and b/static/img/software-catalog/scorecard/catalogRuleSummaries.png differ diff --git a/static/img/software-catalog/scorecard/catalogViewScorecardsByTeam.png b/static/img/software-catalog/scorecard/catalogViewScorecardsByTeam.png index 9f1b0c1c5d..517ac26f55 100644 Binary files a/static/img/software-catalog/scorecard/catalogViewScorecardsByTeam.png and b/static/img/software-catalog/scorecard/catalogViewScorecardsByTeam.png differ diff --git a/static/img/software-catalog/scorecard/tutorial/ScorecardsTab.png b/static/img/software-catalog/scorecard/tutorial/ScorecardsTab.png index 5490ca53a8..55835d96b3 100644 Binary files a/static/img/software-catalog/scorecard/tutorial/ScorecardsTab.png and b/static/img/software-catalog/scorecard/tutorial/ScorecardsTab.png differ